### Rationale for this change
We have many deprecated APIs in Python we should slowly remove.
### What changes are included in this PR?
Several deprecation messages are acted on.
### Are these changes tested?
By the existing test suite.
### Are there any user-facing changes?
Yes, some deprecated user-facing APIs are.removed.
### Was AI used for this PR?
In accordance to the [AI generation guidelines](https://arrow.apache.org/docs/dev/developers/overview.html#ai-generated-code), please disclose below whether and how AI was used in this PR.
**PR code and description written by:**
- [ ] Human
- [x] AI
**Reviewed before submission by:**
- [x] Human
- [x] AI
- [ ] Not reviewed
* GitHub Issue: #49153
Authored-by: Rok Mihevc <rok@mihevc.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Arrow is currently missing remainder and modulo kernels.
### What changes are included in this PR?
Add the kernels `remainder`, `remainder_checked`, `modulo`, and `modulo_checked`, following the terminology in the [divmod proposal](https://lists.apache.org/thread/h3t6nz1ys2k2hnbrjvwyoxkf70cps8sh):
* `remainder` uses truncated (C/C++) semantics — the result has the sign of the dividend, e.g. `remainder(-7, 3) == -1`.
* `modulo` uses floored (Python/R) semantics — the result has the sign of the divisor, e.g. `modulo(-7, 3) == 2`.
Both are supported for integer, floating-point and decimal inputs, and are exposed as `compute::Remainder` / `compute::Modulo` in `api_scalar.h`.
Decimal arguments are promoted to a common scale like `add`, but the result type is resolved by a dedicated resolver rather than reusing `ResolveDecimalAdditionOrSubtractionOutput`: a remainder is always smaller in magnitude than the divisor, so no extra digit is needed for a carry, and the result is `precision = max(p1, p2)`, `scale = s1`. This keeps maximum-precision inputs (`decimal128(38, 0)`, `decimal256(76, 0)`) from overflowing the decimal precision range.
Left as follow-ups: combined divmod #27909, floor division #39386, and duration support.
### Are these changes tested?
Yes. New tests cover truncated vs. floored semantics across signed/unsigned integer, floating-point and decimal types, division by zero, `INT_MIN % -1` overflow, and decimal result-type resolution (same type in/out, mixed precisions and scales, Decimal128/Decimal256 promotion, decimal/integer promotion, and maximum-precision decimals).
### Are there any user-facing changes?
Yes — four new compute functions (`remainder`, `remainder_checked`, `modulo`, `modulo_checked`), the corresponding `compute::Remainder` / `compute::Modulo` C++ APIs, and a new `BasicDecimal256 operator%`. These are additions only; no existing API is changed or removed.
* GitHub Issue: #46901
* xref: https://github.com/apache/arrow/pull/46902,
* xref: https://github.com/apache/arrow/pull/28497
Lead-authored-by: Fangchen Li <fangchen.li@outlook.com>
Co-authored-by: Rok Mihevc <54589+rok@users.noreply.github.com>
Co-authored-by: tadeja <864005+tadeja@users.noreply.github.com>
Signed-off-by: Rok Mihevc <rok@mihevc.org>
### Rationale for this change
NumPy 2.0 added `StringDType`, a variable-width UTF-8 string dtype. `pa.array` currently rejects it with `Unsupported numpy type 2056`. Now that pyarrow requires NumPy 2.0 the conversion can use the public `NpyString_*` C API.
This supersedes #50951. The first commit is @ alippai's first commit from that PR, unchanged. His later commits added batching and new bulk builder APIs in Arrow C++; those can come later in their own PR with benchmarks, as discussed there. Nothing outside the NumPy conversion code changes here.
### What changes are included in this PR?
- `NumPyDtypeToArrow` maps `StringDType` to `string`, so it is also the inferred type for `pa.array`, `pa.infer_type`, `pa.from_numpy_dtype` and lists of such arrays. `large_string` and `string_view` can be requested with `type`. Any other type raises a `TypeError` naming the requested type.
- The conversion holds the dtype's allocator lock for the whole array and never holds the GIL while taking it.
- A null entry in a StringDType array becomes an Arrow null unless the dtype's `na_object` is a `str`. NumPy reports such entries as that string from `__getitem__`, in ufuncs and comparisons, and in casts, and there is no way to tell them apart from a regular entry holding the same string, so the conversion writes the string too. NaN-like sentinels such as `np.nan` and `pd.NA`, `None`, and arbitrary objects all become arrow nulls.
- A short section in `docs/source/python/numpy.rst` describing the support and the semantics for converting missing data.
If left unfixed, https://github.com/apache/arrow/issues/51156 also becomes possible to trigger from Python by creating a StringDType array with a very large entry.
### Are these changes tested?
Yes. Tests cover the three target types, strings crossing NumPy's short, medium and long storage thresholds, non-ASCII input, strided input, `mask`, `None`, NaN and string sentinels, the rejected target type, and list inference. I also ran a threaded test that converts an array while other threads hit NumPy's null-comparison error path on the same array to check the lock ordering.
### Are there any user-facing changes?
Yes. `pa.array` and the inference functions accept `StringDType` arrays.
AI disclosure: I used an AI model to work on the follow-up commits on top of @ alippai's first commit and for code review.
* GitHub Issue: #42018
* GitHub Issue: #51652
Lead-authored-by: Nathan Goldbaum <nathan.goldbaum@gmail.com>
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Co-authored-by: Adam Lippai <adam@rigo.sk>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Summary:
- Document that explicit dataset schemas for partitioned datasets must include partition fields used in filters or projections.
- Add a doctest-backed example showing a Hive partition field used with `count_rows`.
Validation:
- `/tmp/arrow-doccheck-30800/bin/python -m pytest --doctest-glob="*.rst" docs/source/python/dataset.rst -q` — passed, `1 passed`
- `git diff --check HEAD~1 HEAD` — passed
* GitHub Issue: #30800
Authored-by: kalyanamdewri <kalyanampriyam@gmail.com>
Signed-off-by: Nic Crane <thisisnic@gmail.com>
### Rationale for this change
Addresses #51349
### What changes are included in this PR?
* Adds a new `read_kms_url` member to `DecryptionConfiguration` and a parameter with the same name to `rotate_master_keys`. Both are false by default.
* Adds these to the PyArrow bindings.
### Are these changes tested?
Yes, this includes new unit tests.
### Are there any user-facing changes?
Yes, this adds a new user-facing option. Users that relied on this behaviour previously will now need to opt-in and enable the flag.
* GitHub Issue: #51349
Authored-by: Adam Reeve <adreeve@gmail.com>
Signed-off-by: Adam Reeve <adreeve@gmail.com>
### Rationale for this change
The Python memory/IO docs list `pyarrow.HdfsFile` as a `NativeFile` subclass, but that class no longer exists (legacy HDFS was deprecated in 2.0 and later removed). The `:class:` reference has no target, and HDFS access now uses `pyarrow.fs.HadoopFileSystem`, which returns regular `NativeFile` objects. It's the only unresolved pyarrow cross-reference on the page.
### What changes are included in this PR?
Removes the stale `pyarrow.HdfsFile` bullet in docs/source/python/memory.rst. One line, docs only.
### Are these changes tested?
Confirmed `pyarrow.HdfsFile` isn't in pyarrow, the ref appears only once, and all remaining `:class:` refs on the page resolve after removal.
### Are there any user-facing changes?
Documentation only.
Authored-by: Ritika shrestha <87307821+ritsth@users.noreply.github.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
There are already utilities to import tensor data from NumPy.
With DLPack standard, it will work across many tensor providers.
### What changes are included in this PR?
- Bind `FixedShapedTensorArray::FromTensor` in Python
- Add `ImportArrayVersionedFromDLPack` and `ImportTensorVersionedFromDLPack` in C++
- Add in Python `Array.from_dlpack` and `Tensor.from_dlpack`
Similar to the export, multidimensional tensor import require an explicit step through `Tensor`.
- No `Array::FromTensor` or even `FixedSizeListArray::FromTensor`. IMHO it does not feel as necessary in this direction but open to anyone's take one it.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes, new APIs.
* GitHub Issue: #39295
Lead-authored-by: AntoinePrv <AntoinePrv@users.noreply.github.com>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
The current parquet section is one-page and covering lots of topics. As per https://github.com/apache/arrow/issues/46601 the parquet section is most visited and would therefore need a bit of maintenance/content update.
### What changes are included in this PR?
- Parquet documentation is split in 4 different subpages (index.rst, parquet.rst, parquet_datasets.rst and parquet_encryption.rst).
- Some additional content is added
### Are these changes tested?
Via the docs preview
### Are there any user-facing changes?
No
* GitHub Issue: #47686
Lead-authored-by: AlenkaF <frim.alenka@gmail.com>
Co-authored-by: Alenka Frim <AlenkaF@users.noreply.github.com>
Co-authored-by: Adam Reeve <adreeve@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
Require Numpy 2.0 to support newer features, allow newer numpy API/ABI
### What changes are included in this PR?
- Numpy 2.0+ requirement
- Pandas 2.2.2+ requirement (first wheel compatible with numpy 2.0)
- Pyodide 0.27.1 usage (numpy 2.0 api)
### Are there any user-facing changes?
Yes, in contrast to pyarrow 25, after merging this PR pyarrow 26 can't be used with Numpy 1.x or pandas 2.0-2.2.1
**This PR includes breaking changes to public APIs.**
This PR was created using GPT-5.6-Sol-xhigh, every line read & reviewed by me.
* GitHub Issue: #48473
Lead-authored-by: Adam Lippai <adam@rigo.sk>
Co-authored-by: Ádám Lippai <adam@rigo.sk>
Co-authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
Summary:
- Document that quoted numeric CSV values are still subject to type inference.
- Add a doctest-backed example showing how `ConvertOptions.column_types` preserves leading zeros.
- AI assistance was used for issue triage and drafting this documentation example.
Validation:
- `/tmp/arrow-doccheck-46853/bin/python -m pytest --doctest-glob="*.rst" docs/source/python/csv.rst -q` — passed, `1 passed`
- `git diff --check HEAD~1 HEAD` — passed
* GitHub Issue: #46853
Authored-by: kalyanamdewri <kalyanampriyam@gmail.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
Closes#50801
As title, this exposes RecordBatchReaderSourceNodeOptions to Python. With it, a hash join can build its hash table once and stream the probe side through as a generator-backed RecordBatchReader, releasing each input chunk as it is consumed, instead of materializing the full probe table up front (table_source) or pinning all fragments for the plan's lifetime (dataset scan).
### Are these changes tested?
Yes
### Are there any user-facing changes?
Yeah, users can pass RecordBatchReaderSourceNodeOptions into Declaration, but its not a breaking change
* GitHub Issue: #50801
Lead-authored-by: You-Cheng Lin <106612301+owenowenisme@users.noreply.github.com>
Co-authored-by: You-Cheng Lin <mses010108@gmail.com>
Signed-off-by: Rossi Sun <zanmato1984@gmail.com>
### Rationale for this change
Follow-up to https://github.com/apache/arrow/issues/49232#issuecomment-5105290467 / #49590
### What changes are included in this PR?
Remove warnings from `write_feather()`, `read_feather()`, `read_table()`, and `FeatherDataset`.
Warn **only** on writing with `version=1` / reading Feather V1 file.
Update docs to V1-only and keep IPC migration guide.
Additionally corrected notes like `deprecated as of 24.0.0` to `25.0.0` instead.
`DeprecationWarning` in place of current `FutureWarning`.
### Are these changes tested?
Yes, by CI.
### Are there any user-facing changes?
Yes!
Feather V2 APIs no longer emit deprecation warnings.
Reading/writing the legacy Feather V1 format emits `DeprecationWarning` in place of `FutureWarning`.
* GitHub Issue: #50808
Lead-authored-by: Tadeja Kadunc <tadeja.kadunc@gmail.com>
Co-authored-by: tadeja <tadeja@users.noreply.github.com>
Co-authored-by: Rok Mihevc <rok@mihevc.org>
Signed-off-by: Rok Mihevc <rok@mihevc.org>
### Rationale for this change
[Pandas 2.0.0](https://pypi.org/project/pandas/2.0.0/) was released on April 3rd 2023.
The last version of Pandas 1 was [Pandas 1.5.3](https://pypi.org/project/pandas/1.5.3/) and it was released on January 19th 2023. That version only supports up to Python 3.11.
We are currently supporting up to Pandas 1.5.2 (first version supporting 3.11).
On the issue we discussed to bump our Pandas support to 2.0.0 but given 2.0.3 is the latest in the 2.0.x, was released on June 2023, only supports up to Python 3.11 and bumping to it allows use to improve even further some of the conditionals we are bumping to Pandas >= 2.0.3
### What changes are included in this PR?
Update minimal support check to Version(2.0.3) and remove all test skips, conditionals and scenarios where we are checking for Pandas < 2.0.3.
We found a couple of minor things
### Are these changes tested?
Yes via CI
### Are there any user-facing changes?
Yes, Pandas < 2 won't be supported.
* GitHub Issue: #48808
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Python 3.10 is End of Support on October 2026. We can drop support on PyArrow 26.0.0.
See: https://endoflife.date/python
### What changes are included in this PR?
Remove support for 3.10. Update minimum required Python version and update some CI jobs to use the minimum Python supported version.
Some CI jobs using default Python 3.10 (Ubuntu 20.04) have been bumped.
### Are these changes tested?
They will be tested via CI and Archery
### Are there any user-facing changes?
Yes in terms of not supporting older versions of Python. This will be a breaking change for some users but not for the API itself
* GitHub Issue: #50410
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Arrow C++ is deprecating the Feather reader/writer (#49231) so we should update the Python functions too. Feather V2 is exactly the Arrow IPC file format, making the separate `pyarrow.feather` module redundant. Users should migrate to `pyarrow.ipc`.
### What changes are included in this PR?
- Added `FutureWarning` deprecation warnings to all four public APIs in `pyarrow.feather`: `write_feather()`, `read_feather()`, `read_table()`, and `FeatherDataset`
- Extracted `_read_table_internal()` so that `read_feather()` calling `read_table()` internally does not produce double warnings
- Updated `FeatherDataset.read_table()` to use the internal function for the same reason
- Added `.. deprecated:: 24.0.0` directives to all four public API docstrings
- Added module-level `pytestmark` filter in `test_feather.py` to suppress warnings in existing tests
- Added 5 new tests verifying deprecation warnings are emitted correctly (including a no-double-warning test)
- Added `@ pytest.mark.filterwarnings` to 5 test functions in `test_dataset.py` that use feather as a utility
- Updated `feather.rst` with a deprecation notice and migration guide (including a note that `pyarrow.ipc` does not compress by default unlike `feather.write_feather`)
- Updated `formats.rst` to mark the Feather API section as deprecated
### Are these changes tested?
Yes. Five new tests verify that each deprecated function emits exactly one `FutureWarning`, and existing tests continue to pass with module/function-level warning filters.
### Are there any user-facing changes?
Yes — calling `pyarrow.feather.write_feather()`, `read_feather()`, `read_table()`, or `FeatherDataset()` now emits a `FutureWarning` directing users to `pyarrow.ipc` equivalents. Existing functionality is unchanged.
* GitHub Issue: #49232
Lead-authored-by: Piyush Kanti Chanda <piyush.chanda@databricks.com>
Co-authored-by: Isaac
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
Arrow has `atan2` and `logb` but no `hypot`. Computing a Euclidean norm currently needs `sqrt(add(multiply(x, x), multiply(y, y)))`, which can overflow at the `multiply` step even when the result is representable.
### What changes are included in this PR?
Add a `hypot` kernel backed by `std::hypot` (float32/float64, with integer/decimal inputs promoted to float64), the `Hypot()` C++ function, a `FunctionDoc` and `compute.rst` entry, and a Python expression test. `pc.hypot` is auto-exposed from the registry.
### Are these changes tested?
Yes, C++ tests for the basic cases, null/NaN/Inf handling, and an overflow case confirming it doesn't fall back to a naive `sqrt(x*x + y*y)`.
### Are there any user-facing changes?
Yes, a new `hypot` function in C++ and Python. Additive only.
* GitHub Issue: #50197
Lead-authored-by: Shrivas Shankar <shrivasshankar2019@gmail.com>
Co-authored-by: Shrivas Shankar <86219405+shrivasshankar@users.noreply.github.com>
Co-authored-by: Rok Mihevc <rok@mihevc.org>
Signed-off-by: Rok Mihevc <rok@mihevc.org>
### Rationale for this change
See https://github.com/apache/arrow/issues/47435.
### What changes are included in this PR?
Adds direct encryption / decryption Python API
### Are these changes tested?
Yes, see PR.
### Are there any user-facing changes?
Yes, new Python bindings.
* GitHub Issue: #47435
Authored-by: Sreesh Maheshwar <smaheshwar@palantir.com>
Signed-off-by: Adam Reeve <adreeve@gmail.com>
### Rationale for this change
Add an optional default_column_type parameter to the CSV reading API (C++ and Python) to provide a fallback type when per-column types aren’t specified, improving schema consistency and complementing the existing column_types logic.
### What changes are included in this PR?
- c++: new convert option "default_column_type" to augment logic around column_types parameter
- 3 reader tests: DefaultColumnTypePartialDefault, DefaultColumnTypeAllStringsWithHeader, DefaultColumnTypeAllStringsNoHeader). The last two tests are inspired by https://github.com/pandas-dev/pandas/pull/62242 and https://github.com/pandas-dev/pandas/issues/57666
- python: corresponding changes to make cpp change consumable from python
- python: extended test_convert_options test - include. default_column_type
- python: added new test "test_default_column_type" which tests how the field impacts schema; also test implicitly verifies leading zero preservation
- relevant documentation update for python component;
### Are these changes tested?
Yes. Existing and new tests are passing.
C++:
> [==========] Running 3 tests from 1 test suite.
> [----------] Global test environment set-up.
> [----------] 3 tests from ReaderTests
> [ RUN ] ReaderTests.DefaultColumnTypePartialDefault
> [ OK ] ReaderTests.DefaultColumnTypePartialDefault (3 ms)
> [ RUN ] ReaderTests.DefaultColumnTypeAllStringsWithHeader
> [ OK ] ReaderTests.DefaultColumnTypeAllStringsWithHeader (0 ms)
> [ RUN ] ReaderTests.DefaultColumnTypeAllStringsNoHeader
> [ OK ] ReaderTests.DefaultColumnTypeAllStringsNoHeader (0 ms)
> [----------] 3 tests from ReaderTests (4 ms total)
>
> [----------] Global test environment tear-down
> [==========] 3 tests from 1 test suite ran. (4 ms total)
> [ PASSED ] 3 tests.
All:
> [==========] 264 tests from 46 test suites ran. (452 ms total)
> [ PASSED ] 264 tests.
pyarrow:
New tests are passing.
### Are there any user-facing changes?
I believe this change is backward compatible. Parameter is optional and its default value doesn't change the existing behavior; All the existing rests are passing.
Maybe relevant: https://github.com/apache/arrow/issues/22232
Relates to https://github.com/apache/arrow/issues/47502
* GitHub Issue: #47502
* GitHub Issue: #22232
Lead-authored-by: Vlad Borovtsov <borovcov.vlad@gmail.com>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
`ColumnChunkMetaData.to_dict()` method omits `bloom_filter_offset` and `bloom_filter_length` even when a bloom filter is written to the Parquet file. Users cannot programmatically verify bloom filter presence via the Python metadata API without resorting to file size comparison.
### What changes are included in this PR?
1. `python/pyarrow/includes/libparquet.pxd`: Declare `bloom_filter_offset()` and `bloom_filter_length()` (both optional[int64_t]) on `CColumnChunkMetaData`. This is to expose the existing C++ methods to Cython.
2. `python/pyarrow/_parquet.pyx`: Add `bloom_filter_offset` and `bloom_filter_length` properties to `ColumnChunkMetaData` (returns int when set, None otherwise). Add both fields to `to_dict()` and `__repr__`.
3. `python/pyarrow/tests/parquet/test_metadata.py`: Add `test_bloom_filter_offset_in_metadata` verifying that columns with a bloom filter expose non-None integer values and that `to_dict()` contains the keys, while columns without a bloom filter return None.
### Are these changes tested?
Yes. `test_bloom_filter_offset_in_metadata` in test_metadata.py covers:
- Column with bloom filter: bloom_filter_offset and bloom_filter_length are non-None integers
- Column without bloom filter: both return None
- Both keys present in to_dict() output
<img width="863" height="215" alt="image" src="https://github.com/user-attachments/assets/d465d6bd-55d1-4c5f-9f11-6a60b3bf1cbe" />
Here is closer look on the logic output:
<img width="464" height="424" alt="image" src="https://github.com/user-attachments/assets/6e4810f2-c1c0-41ea-b559-00f99d42e2c4" />
output:
```python
col_a bloom_filter_offset: 10699
col_a bloom_filter_length: 1040
col_b bloom_filter_offset: None
col_b bloom_filter_length: None
col_a to_dict(): {'file_offset': 0, 'file_path': '', 'physical_type': 'BYTE_ARRAY', 'num_values': 1000, 'path_in_schema': 'a', 'is_stats_set': True, 'statistics': {'has_min_max': True, 'min': 'id_0', 'max': 'id_999', 'null_count': 0, 'distinct_count': None, 'num_values': 1000, 'physical_type': 'BYTE_ARRAY'}, 'geo_statistics': None, 'compression': 'SNAPPY', 'encodings': ('PLAIN', 'RLE', 'RLE_DICTIONARY'), 'has_dictionary_page': True, 'dictionary_page_offset': 4, 'data_page_offset': 4035, 'total_compressed_size': 5336, 'total_uncompressed_size': 11208, 'bloom_filter_offset': 10699, 'bloom_filter_length': 1040}
```
* GitHub Issue: #49927
Lead-authored-by: Haziq Hakimi Mazlisham <95128744+haziqishere@users.noreply.github.com>
Co-authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
NumPy's `datetime64` type does not support timezones. When converting a timezone-aware Arrow timestamp array to NumPy via `to_numpy()`, the timezone information is silently dropped. This behaviour is expected but undocumented, which can surprise users (see #45644).
### What changes are included in this PR?
Adds a "Timezone-aware Timestamps" subsection to `docs/source/python/numpy.rst` that:
- Explains the timezone loss when calling `to_numpy()` on tz-aware timestamp arrays
- Shows a code example demonstrating the behavior
- Documents two alternatives: `to_pandas()` for tz-aware Series, and `to_pylist()` for Python `datetime` objects with `tzinfo`
### Are these changes tested?
Documentation-only change. All code examples were verified against pyarrow 24.0.0 and `sphinx-lint` passes clean.
### Are there any user-facing changes?
No behaviour changes. This adds documentation for existing behaviour.
### AI-generated code disclosure
This PR was developed with assistance from an AI coding tool (Claude, Anthropic). All changes have been reviewed, understood, and verified.
* GitHub Issue: #45644Closes#45644
Authored-by: Alexandros Anastasiou <anastasioyaa@gmail.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
The [Python / AMD64 Conda Python 3.11 Sphinx & Numpydoc](https://github.com/apache/arrow/actions/runs/24404394331/job/71295731710?pr=49742) job is failing due to the version number change.
### What changes are included in this PR?
- Use ellipsis so that all version-dependent bytes are covered.
- Add `docs/source/python/**` to the python workflow path triggers to run Sphinx & Numpydoc CI on documentation changes.
### Are these changes tested?
Yes, with Conda Python 3.11 Sphinx & Numpydoc job.
### Are there any user-facing changes?
No.
* GitHub Issue: #49745
Authored-by: AlenkaF <frim.alenka@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Pandas is moving the dataframe interchange protocol out of the repo in the next major release (4.0.0).
### What changes are included in this PR?
Filterwarnings are added and the pandas example is removed from the interchange documentation.
### Are these changes tested?
Yes, via CI.
### Are there any user-facing changes?
No.
* GitHub Issue: #49628
Authored-by: AlenkaF <frim.alenka@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Please see #32007, currently, neither arrays nor scalars support Python-native arithmetic operations, such as `array + array`, it has to be done via `pyarrow.compute` API. This PR strives to fix this with custom dunder methods.
### What changes are included in this PR?
Implemented dunder methods
### Are these changes tested?
Yes
### Are there any user-facing changes?
Possibility to use Python operators directly instead of calling the `pyarrow.compute` API.
* GitHub Issue: #32007
Authored-by: Bogdan Romenskii <rmnsk@seznam.cz>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
Closes#49509
### What changes are included in this PR?
Docs formatting/typos corrected.
### Are these changes tested?
Yes, on fork branch.
### Are there any user-facing changes?
No, just corrected formatting/typos in docs.
* GitHub Issue: #49509
Authored-by: Tadeja Kadunc <tadeja.kadunc@gmail.com>
Signed-off-by: Rok Mihevc <rok@mihevc.org>
### Rationale for this change
This closes#48241, #44224 and #43855.
Currently uuid.UUID objects are not inferred/converted automatically in PyArrow, requiring users to explicitly specify the type.
### What changes are included in this PR?
Adding support for Python's uuid.UUID objects in PyArrow's type inference and conversion.
### Are these changes tested?
Yes, added test_uuid_scalar_from_python() and test_uuid_array_from_python() in `test_extension.py`.
### Are there any user-facing changes?
Users can now pass Python uuid.UUID objects directly to PyArrow functions like pa.scalar() and pa.array() without specifying the type;
```python
import uuid
import pyarrow as pa
pa.scalar(uuid.uuid4())
```
<pyarrow.UuidScalar: UUID('958174b9-3a5c-4cdd-8fc5-d51a2fc55784')>
```python
pa.array([uuid.uuid4()])
```
<pyarrow.lib.UuidArray object at 0x1217725f0>
[
73611FD81F764A209C8B9CDBADDA1F53
]
* GitHub Issue: #48241
Lead-authored-by: Tadeja Kadunc <tadeja.kadunc@gmail.com>
Co-authored-by: tadeja <tadeja@users.noreply.github.com>
Co-authored-by: Rok Mihevc <rok@mihevc.org>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Signed-off-by: Rok Mihevc <rok@mihevc.org>
### Rationale for this change
Switch to std::chrono for MSVC to be able to use the system-provided timezone automatically on Windows.
### What changes are included in this PR?
This adds `chrono_internal.h` that uses C++20 std::chrono timezone/calendar APIs on compilers with support (MSVC only for now) and falls back to vendored `date.h` otherwise.
### Are these changes tested?
Partially tested locally and partially to be tested on CI.
### Are there any user-facing changes?
Yes, Windows users will no longer need to install the IANA tzdb (see [instructions here](https://arrow.apache.org/docs/cpp/build_system.html#runtime-dependencies) and [here](https://arrow.apache.org/docs/python/install.html#tzdata-on-windows)). We possibly have tzdb download set up in CI too and should update it appropriately.
* GitHub Issue: #48593
Authored-by: Rok Mihevc <rok@mihevc.org>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Sparse tensor classes (SparseCOOTensor, SparseCSRMatrix, SparseCSCMatrix, SparseCSFTensor) lack API documentation.
### What changes are included in this PR?
* Add class docstrings with format descriptions and examples for all sparse tensor classes
* Add sparse tensor classes to `tables.rst` for API reference generation
### Are these changes tested?
Yes, result example:
<img width="1765" height="816" alt="截圖 2026-02-04 23 49 49" src="https://github.com/user-attachments/assets/5dbdc25c-0645-4dd4-b476-c4b028278677" />
### Are there any user-facing changes?
No.
* GitHub Issue: #43352
Authored-by: Chilin <chilin.cs07@nycu.edu.tw>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
Closes https://github.com/apache/arrow/issues/41863
### Rationale for this change
Other tools in the parquet ecosystem distinguish between `LZ4` and `LZ4_RAW`, matching the specification: https://parquet.apache.org/docs/file-format/data-pages/compression/
`LZ4` (framing) is of course deprecated. PyArrow does not support it, and instead simplifies the user-facing API, using `LZ4` as an alias for the `LZ4_RAW` codec.
However, PyArrow does not accept `LZ4_RAW` as a valid alias for the `LZ4_RAW` codec:
```
ArrowException: Unsupported compression: lz4_raw
```
This is a friction issue, and confusing for some users who are aware of the differences.
### What changes are included in this PR?
- Adding `LZ4_RAW` to the acceptable codec names list.
- Modifying the `LZ4->LZ4_RAW` mapping to also accept `LZ4_RAW->LZ4_RAW`.
- Adding a test
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes, an additive change to the accepted codec names.
* GitHub Issue: #41863
Authored-by: Nick Woolmer <29717167+nwoolmer@users.noreply.github.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
Fixes: #49150
See https://github.com/apache/arrow/pull/48619#issuecomment-3823269381
### Rationale for this change
Fix CI failures
### What changes are included in this PR?
Tests are made more general to allow for Pandas 2 and Pandas 3 style string types
### Are these changes tested?
By CI
### Are there any user-facing changes?
No
* GitHub Issue: #49150
Authored-by: Rok Mihevc <rok@mihevc.org>
Signed-off-by: Rok Mihevc <rok@mihevc.org>
### Rationale for this change
As discussed on the issue we don't seem to have run asv benchmarks on Python for the last years. It is probably broken.
### What changes are included in this PR?
Remove asv benchmarking related files and docs.
### Are these changes tested?
No, Validate CI and run preview-docs to validate docs.
### Are there any user-facing changes?
No
* GitHub Issue: #46008
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
In many places in the Python User Guide the code exampels are written with IPython directive (elsewhere code-block is used). IPython directives are converted to IPython format (`In` and `Out` during the doc build). This can lead to slower builds.
### What changes are included in this PR?
IPython directives are converted to runnable code-block (with `>>>` and `...`) and pytest doctest support for `.rst` files is added to the `conda-python-docs` CI job. This means the code in the Python User Guide is tested separately to the building of the documentation.
### Are these changes tested?
Yes, with the CI.
### Are there any user-facing changes?
Changes to the Python User Guide examples will have to be tested with `pytest --doctest-glob='*.rst' docs/source/python/file.rst`
* GitHub Issue: #28859
Lead-authored-by: AlenkaF <frim.alenka@gmail.com>
Co-authored-by: Alenka Frim <AlenkaF@users.noreply.github.com>
Co-authored-by: tadeja <tadeja@users.noreply.github.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
To close or discuss #48167
`inverse_permutation` and `scatter` functions got implemented via #44393, PR #44394.
### What changes are included in this PR?
Python tests for `scatter`, `inverse_permutation` kernels and bindings for `InversePermutationOptions` and `ScatterOptions`.
### Are these changes tested?
Yes, tests added in test_compute.py.
### Are there any user-facing changes?
Bindings for `InversePermutationOptions` and `ScatterOptions` are added.
#### This PR includes breaking changes to public APIs.
Options `InversePermutationOptions` changed from accepting parameter
`std::shared_ptr<DataType> output_type = NULLPTR`
to
`std::optional<std::shared_ptr<DataType>> output_type = std::nullopt`
* GitHub Issue: #48167
Lead-authored-by: Tadeja Kadunc <tadeja.kadunc@gmail.com>
Co-authored-by: Rossi Sun <zanmato1984@gmail.com>
Co-authored-by: tadeja <tadeja@users.noreply.github.com>
Signed-off-by: Rossi Sun <zanmato1984@gmail.com>
### Rationale for this change
We decided to migrate Arrow C++ to C++20 in this discussion: https://lists.apache.org/thread/48zlj0dn2y0f53y2k37qsr90y781wfnj
### What changes are included in this PR?
1. Build configuration updates (CMake files etc.) to build with C++20 instead of C++17
2. C++-level fixes to ensure compilation succeeds:
* Workarounds for deprecation of atomic access on `std::shared_ptr<T>` (the replacement `std::atomic<std::shared_ptr<T>>` is unfortunately not supported in all standard library implementations)
* Workaround for [Abseil ABI issues](https://github.com/abseil/abseil-cpp/issues/1624)
* Trivial update to `arrow/util/string.h` to call a C++20 API (to validate that C++20 is actually enabled)
3. CI configuration updates to get enough C++20 support on the various platforms:
* Bump default clang and LLVM versions to 18
* Use AlmaLinux 10 for release verification (AlmaLinux 8 is still tested on other CI builds)
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes, Arrow C++ will now require a C++20-compliant compiler.
* GitHub Issue: #45885
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
There is no sponsor to maintain EOL-ed CentOS 7.
### What changes are included in this PR?
Remove CentOS 7 packages related configurations.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* GitHub Issue: #40735
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Columns can b encrypted with individual keys. For this, the column name have to be set in `EncryptionConfiguration::column_keys`. This poses the following challenges for columns with nested fields like `MapType`, `ListType`, and `StructType`. Encrypting a column of such type requires providing an encryption key for all nested (leaf) fields. Ideally, the column name should be sufficient (as it is for any other data type) to encrypt all nested fields.
### What changes are included in this PR?
The column name can be used to encrypt all nested fields of `StrutType`, `MapType`, and `ListType` columns with the same encryption key. The current column naming scheme can still be used for backward compatibility.
### Are these changes tested?
Tested in C++ and Python.
### Are there any user-facing changes?
Column encryption can be configured with less code and more intuitive naming.
Documentation and examples updated.
Fixes#41246.
* GitHub Issue: #41246
Authored-by: Enrico Minack <github@enrico.minack.dev>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
No more VD, no more VD S3 bucket!
### What changes are included in this PR?
Move references to S3 bucket to the new Arrow one, update a few references to regions and things.
### Are these changes tested?
Yeah, for the most part.
### Are there any user-facing changes?
No
* GitHub Issue: #48260
Authored-by: Nic Crane <thisisnic@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
Issue #48095 and #45322
### Rationale for this change
Missing API doc entries.
### What changes are included in this PR?
I've added strings for functions and options to the API doc page.
### Are these changes tested?
I've built docs locally.
### Are there any user-facing changes?
Docs only.
* GitHub Issue: #48095
* Also covers: #45322
Lead-authored-by: Tadeja Kadunc <tadeja.kadunc@gmail.com>
Co-authored-by: tadeja <tadeja@users.noreply.github.com>
Co-authored-by: Alenka Frim <AlenkaF@users.noreply.github.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
I have noticed I forgot to add support for Python 3.14 to the install docs. The support for Python 3.14 has been added in this PR: https://github.com/apache/arrow/pull/47616.
### What changes are included in this PR?
Minor documentation change in Python Compatibility section.
### Are these changes tested?
No, no need.
### Are there any user-facing changes?
No.
Authored-by: Alenka Frim <AlenkaF@users.noreply.github.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Python 3.9 is End of Support on 31st October 2025. We can drop support for pyarrow 22.0.0.
See: https://endoflife.date/python
### What changes are included in this PR?
Remove support for 3.9. Update minimum required Python version and update some CI jobs to use the minimum Python supported version
### Are these changes tested?
They will be tested via CI and Archery
### Are there any user-facing changes?
Yes in terms of not supporting older versions of Python. This will be a breaking change for some users but not for the API itself
* GitHub Issue: #47443
Lead-authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
The structure of the Python build and development documentation has become unclear due to multiple incremental changes over time. This PR aims to improve clarity and organization.
### What changes are included in this PR?
- Introduces tabs to improve navigation.
- Splits the existing documentation into two focused pages: Building and Developing PyArrow.
### Are these changes tested?
Yes. With docs preview build.
### Are there any user-facing changes?
No. Only docs changes.
* GitHub Issue: #20125
Authored-by: AlenkaF <frim.alenka@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Please see Github Issue #47123
### What changes are included in this PR?
Added public Type Enums that mimic the original private variable groups used for internal type checking.
### Are these changes tested?
Yes. Partly for now.
### Are there any user-facing changes?
No, just additional features were added: they will now able to access the underlying types directly via the Type Enums.
* GitHub Issue: #47123
Lead-authored-by: Bogdan Romenskii <rmnsk@seznam.cz>
Co-authored-by: Alenka Frim <AlenkaF@users.noreply.github.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
I originally started this patch because the title of https://arrow.apache.org/docs/python/extending_types.html is "Extending pyarrow..." which immediately jumped out as me as not quite right: It should be "Extending PyArrow..." since PyArrow is the name of the software. I think we try to use the all-lowercase form only in code snippets or when specifically referring to the spelling of PyArrow various distributions.
### What changes are included in this PR?
Changed docs/source/python/extending_types.rst to hopefully use the various forms correctly
I didn't do a deep-dive into other docs.
### Are these changes tested?
Yes, rendered locally.
### Are there any user-facing changes?
No.
Authored-by: Bryce Mecum <petridish@gmail.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
Make filesystem operations more convenient to users by supporting:
- `fsspec` filesystems explicitly by passing `fsspec+` prefix to the filsystem scheme
- explicitly support `hf://` scheme by loading the huggingface fsspec filesystem adapter
### What changes are included in this PR?
`_resolve_filesystem_and_path()` is now able to recognize `fsspec+{protocol}` and `hf://` scheme in which case it attempts to load the specific fsspec filesystem implementation. Made some additional refactoring to consolidate the filesystem from string logic.
### Are these changes tested?
Yes. `fsspec+{protocol}` uris are tested using the `fsspec.MemoryFilesystem` whereas the `hf://` scheme is tested by mocking out the `huggingface_hub.HfFileSystem` class. I also tested the huggingface integration using a live huggingface repository.
### Are there any user-facing changes?
Various user facing functions now support more filesystem implementations:
```py
pyarrow.fs.copy_files()
ParquetFile()
ParquetDataset()
dt.dataset()
pq.read_metadata()
pq.write_metadata()
pq.read_table()
pq.read_schema()
orc.read_table()
```
This includes both the path containing additional schemes and the `filesystem=` argument.
* GitHub Issue: #44900
Authored-by: Krisztian Szucs <szucs.krisztian@gmail.com>
Signed-off-by: Antoine Pitrou <pitrou@free.fr>
### What does this PR do?
Adds a new compute kernel `utf8_zero_fill` to Arrow Compute module. This kernel zero-pads UTF-8 strings while preserving leading signs.
### Why is it needed?
Python's `str.zfill()` behavior is useful and expected in many data cleaning scenarios. Arrow lacked a direct equivalent.
### Implementation details
- Introduces `Utf8ZeroFillTransform` with dedicated `ZeroFillOptions`
- Registers `utf8_zero_fill` as a compute kernel
- Adds Python bindings and a clean alias: `utf8_zfill = utf8_zero_fill`
- Includes sign-aware padding logic
### Are these changes tested?
- C++ unit tests: `TestStringKernels, Utf8ZeroFill`
- Python tests: `test_utf8_zfill()`
Ran both tests
* GitHub Issue: #46683
Authored-by: iabhi4 <iamonecool@gmail.com>
Signed-off-by: Antoine Pitrou <pitrou@free.fr>
Thanks for opening a pull request!
### Rationale for this change
When installing pyarrow with pip, it is possible to experience a confusing time zone related error when writing files with datetime data. #46080 discusses the problem and a solution was eventually found. This change provides helpful information to other users who may experience the problem.
### What changes are included in this PR?
Update to python installation instructions that offers guidance on how to handle tzdata related errors
### Are these changes tested?
I was unable to build the documentation locally. I will use the github action to preview the documentation
### Are there any user-facing changes?
No
* GitHub Issue: #46080
Lead-authored-by: Kyle Hemker <k@hemker.us>
Co-authored-by: Alenka Frim <AlenkaF@users.noreply.github.com>
Co-authored-by: kahemker <github@kylehemker.uk>
Co-authored-by: kahemker <kyle.hemker@gmail.com>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Co-authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
The docs don't build cleanly for me locally and some warnings and errors that are consequential are being ignored. Some of these issues ended up being as severe as entirely missing tables.
### What changes are included in this PR?
- Fixes for each issue I ran into. Each change is a commit with whatever extra detail I needed included in the commit body.
- This also includes an auto-update of our Doxyfile which silences a number of deprecation warnings. I took a look at the diff and it seemed pretty safe and the docs look good to me locally. We can do a preview in CI first though.
### Are these changes tested?
Yes, each change was tested locally.
### Are there any user-facing changes?
No.
* GitHub Issue: #46520
Authored-by: Bryce Mecum <petridish@gmail.com>
Signed-off-by: Bryce Mecum <petridish@gmail.com>
### What changes are included in this PR?
1. Move current Acero Developer's Guild under user doc to where under developer docs;
2. Make it the main doc enclosing elaborating docs like Swiss table.
### Are these changes tested?
No need.
### Are there any user-facing changes?
None.
* GitHub Issue: #46475
Lead-authored-by: Rossi Sun <zanmato1984@gmail.com>
Co-authored-by: Bryce Mecum <petridish@gmail.com>
Signed-off-by: Rossi Sun <zanmato1984@gmail.com>
Rok Mihevc