### Rationale for this change
`pq.read_table` has no way to pass a memory pool. `ParquetReader` and `Dataset.to_table` already accept one, it just wasn't added.
### What changes are included in this PR?
I added a `memory_pool` argument to `read_table`, `ParquetFile` and `ParquetDataset`. It is passed to `ParquetReader` and to `Dataset.to_table`, with docstrings updated.
### Are these changes tested?
Yes. I added new tests in `test_basic.py` - It reads a Parquet file with a tracked memory pool and check that the pool was actually used. They cover `read_table`, `ParquetFile` and `ParquetDataset`.
### Are there any user-facing changes?
New optional `memory_pool` keyword. No breaking changes.
### Was AI used for this PR?
**PR code and description written by:**
- [x] Human
- [x] AI
**Reviewed before submission by:**
- [x] Human
- [ ] AI
Github Issue - https://github.com/apache/arrow/issues/54637
* GitHub Issue: #54637
Authored-by: Omkar Kabde <omkarkabde@gmail.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
`python/pyarrow/array.pxi` defines the class as `Decima32Array` (missing `l`).
### What changes are included in this PR?
Renaming of the class in `array.pxi` to `Decimal32Array`.
### Are these changes tested?
I haven't added a special test for this.
### Are there any user-facing changes?
No.
### Was AI used for this PR?
**PR code and description written by:**
- [X] Human
- [ ] AI
**Reviewed before submission by:**
- [X] Human
- [ ] AI
- [ ] Not reviewed
* GitHub Issue: #55285
Authored-by: AlenkaF <frim.alenka@gmail.com>
Signed-off-by: Tadeja Kadunc <tadeja.kadunc@gmail.com>
### Rationale for this change
We have many deprecated APIs in Python we should slowly remove.
### What changes are included in this PR?
Several deprecation messages are acted on.
### Are these changes tested?
By the existing test suite.
### Are there any user-facing changes?
Yes, some deprecated user-facing APIs are.removed.
### Was AI used for this PR?
In accordance to the [AI generation guidelines](https://arrow.apache.org/docs/dev/developers/overview.html#ai-generated-code), please disclose below whether and how AI was used in this PR.
**PR code and description written by:**
- [ ] Human
- [x] AI
**Reviewed before submission by:**
- [x] Human
- [x] AI
- [ ] Not reviewed
* GitHub Issue: #49153
Authored-by: Rok Mihevc <rok@mihevc.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
`FileKeyWrapper` calls `SetDefaultIfEmpty()` on its stored configuration, but passes the original, unmodified configuration to `GetKmsClient`. This allows the KMS client factory to receive empty instance ID and URL values even though Arrow has already filled in defaults on its internal copy.
### What changes are included in this PR?
- Pass the defaulted `kms_connection_config_` member to `GetKmsClient`.
- Update C++ and Python key-rotation tests to expect Arrow's existing default values for empty fields.
- Separate the writer and reader crypto factories in the dataset encryption tests.
The dataset tests give the writer all keys and the reader only some. With a shared factory, the reader can reuse the writer's cached client, which already has all the keys. Separate factories ensure the reader uses its own keys, so the missing-key checks work correctly. The write-path fix exposes this because writing and reading now use the same default instance ID, where empty and `"DEFAULT"` IDs previously selected separate cache entries.
### Are these changes tested?
- Local C++ encryption key-management suite: 24 tests passed.
- Both local Python encryption test files: 62 passed, 1 skipped. The skipped case requires a build without encryption support.
- C++ and Python formatting and lint, and `git diff --check`, passed.
- Fork CI for `537e296384`: [Python](https://github.com/Dricmoy/arrow/actions/runs/38923806919), [Dev](https://github.com/Dricmoy/arrow/actions/runs/38923806888), and [Docs](https://github.com/Dricmoy/arrow/actions/runs/38923806885) passed. The [C++ workflow](https://github.com/Dricmoy/arrow/actions/runs/38852310915) passed for the preceding commit; the follow-up commit changes only Python tests.
### Are there any user-facing changes?
KMS client factories receive Arrow's `"DEFAULT"` markers instead of empty strings for omitted instance ID and URL values. Explicit non-empty values are preserved. These markers are interpreted by the KMS implementation; they are not automatically resolved server addresses.
I reviewed the compatibility tradeoff: custom factories that special-case empty strings may need updating. Defaulting the instance ID can also change client reuse because the cache is indexed by access token and instance ID. Requests with the same token and resolved ID can now share a client where empty and `"DEFAULT"` IDs previously selected separate entries. This makes client creation consistent with Arrow's existing defaulting logic and stored key metadata, but changes behavior for integrations that rely on empty strings. There are no API signature changes.
Closes#51351. Follow-up to #53949.
### Was AI used for this PR?
AI assisted with the implementation, tests, validation, and this description. I reviewed the configuration flow, the default-value test expectations, the reader/writer cache separation, and the compatibility tradeoff described above. The PR overall is fairly small consistent expectation change, had some tests changed to fit the updated expectations.
**PR code and description written by:**
- [x] Human
- [x] AI
**Reviewed before submission by:**
- [x] Human
- [x] AI
- [ ] Not reviewed
* GitHub Issue: #51351
Authored-by: Dricmoy <dricbhattacharjee@gmail.com>
Signed-off-by: Adam Reeve <adreeve@gmail.com>
### Rationale for this change
`pa.show_info()` prints its Compression Codecs section from a hardcoded list:
```python
codecs = ["brotli", "bz2", "gzip", "lz4_frame", "lz4", "snappy", "zstd"]
```
That list has drifted from `_ensure_compression()` in `io.pxi`, which is what
actually decides which codec names work. Two problems follow:
- `lz4` and `lz4_frame` both map to `CCompressionType_LZ4_FRAME`, so the same
codec is reported twice under two names.
- `lz4_raw` maps to `CCompressionType_LZ4` and is a separate codec, but it is
not listed at all, even though `pa.Codec.is_available('lz4_raw')` is True.
So seven lines are printed for six codecs, and the codec that is missing is a
real one.
Before:
```
Compression Codecs:
brotli : Enabled
bz2 : Enabled
gzip : Enabled
lz4_frame : Enabled
lz4 : Enabled
snappy : Enabled
zstd : Enabled
```
After:
```
Compression Codecs:
brotli : Enabled
bz2 : Enabled
gzip : Enabled
lz4 : Enabled
lz4_raw : Enabled
snappy : Enabled
zstd : Enabled
```
### What changes are included in this PR?
One list in `python/pyarrow/__init__.py`: drop the duplicate `lz4_frame` and add
the missing `lz4_raw`. `lz4` is kept as the name for the frame codec since that
is the one the docstrings lead with (`'lz4' (or 'lz4_frame')`).
On deriving this programmatically rather than hardcoding it — agreed, that is
the better fix. The canonical mapping lives in `_ensure_compression()` in
`io.pxi`, and neither `Codec` nor the C++ `compression.h` currently exposes the
set of supported codecs, so it needs a new accessor on the Cython side for
`show_info()` to read. I am happy to do that here, with the caveat that I cannot
compile the Cython extension in my environment and would be relying on CI to
verify it. If you would prefer, I can keep this change minimal and open a
follow-up issue for the programmatic version. Let me know which you would rather
have.
### Are these changes tested?
Not by a new test. `show_info()` has no existing test coverage and its output is
a diagnostic print, so I did not want to pin the exact text without knowing
whether you would want that — happy to add one if you do.
Verified by applying the same change to an installed pyarrow and reading the
output back, and by walking every name `_ensure_compression()` accepts to
confirm which of them map onto the same C++ enum value:
```
bz2 -> BZ2, gzip -> GZIP, brotli -> BROTLI,
lz4 -> LZ4_FRAME, lz4_frame -> LZ4_FRAME, # same codec, two names
lz4_raw -> LZ4, # distinct, was not listed
snappy -> SNAPPY, zstd -> ZSTD
```
### Are there any user-facing changes?
Only the output of `pa.show_info()`, which now lists each supported codec once
and includes `lz4_raw`. No API change.
### Was AI used for this PR?
In accordance to the [AI generation guidelines](https://arrow.apache.org/docs/dev/developers/overview.html#ai-generated-code), please disclose below whether and how AI was used in this PR.
**PR code and description written by:**
- [x] Human
- [ ] AI
**Reviewed before submission by:**
- [x] Human
- [ ] AI
- [ ] Not reviewed
* GitHub Issue: #52024
Authored-by: nishad shabbir <nishadshabbir28@gmail.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Our JSON parser is inherently recursive, and a crafted JSON document with huge nesting can send it into a stack overflow.
Given that real-world JSON files will not have thousands levels of nesting, we limit the JSON nesting to an arbitrary value of 300.
### Are these changes tested?
Yes, by new tests.
### Are there any user-facing changes?
**This PR contains a "Critical Fix".** A crafted JSON document with a large nesting depth could trigger a denial of service through stack overflow.
### Was AI used for this PR?
**PR code and description written by:**
- [x] Human
- [ ] AI
**Reviewed before submission by:**
- [x] Human (self-reviewed)
- [ ] AI
- [ ] Not reviewed
* GitHub Issue: #51757
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
This PR continues the simdjson migration by replacing the RapidJSON-based parsing implementation used by the JSON parser.
The existing parser uses RapidJSON's SAX/handler interface to parse JSON values and populate Arrow builders. This change replaces that implementation with simdjson's ondemand API while retaining the existing builder and type-inference logic.
### Changes
* Replace the RapidJSON parser and handler interface with simdjson's ondemand API.
* Parse JSON documents using `simdjson::ondemand::parser::iterate_many`.
* Use `ResolveSimdjsonResult()` consistently when handling simdjson results.
* Preserve support for nested objects and arrays.
* Preserve explicit-schema and inferred-field behavior.
* Preserve unexpected-field handling for `Error`, `Ignore`, and `InferType`.
* Continue storing numeric values as raw JSON tokens.
* Trim trailing whitespace from numeric raw tokens to preserve existing behavior.
* Preserve JSON parse error propagation through `Status::Invalid`.
* Remove the parser's RapidJSON-specific dependencies.
### Are there any user-facing changes?
No
### Was AI used for this PR?
In accordance to the [AI generation guidelines](https://arrow.apache.org/docs/dev/developers/overview.html#ai-generated-code), please disclose below whether and how AI was used in this PR.
**PR code and description written by:**
- [X] Human
- [X] AI
**Reviewed before submission by:**
- [X] Human
- [X] AI
- [ ] Not reviewed
Fixes: #51037
* GitHub Issue: #51037
Authored-by: Aaditya Srinivasan <aadityasri03@gmail.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Test warnings accumulated in our CI.
### What changes are included in this PR?
The warnings are fixed or filtered if the deprecated functionality is still being tested.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No. Fixing test warnings in our CI.
### Was AI used for this PR?
In accordance to the [AI generation guidelines](https://arrow.apache.org/docs/dev/developers/overview.html#ai-generated-code), please disclose below whether and how AI was used in this PR.
**PR code and description written by:**
- [x] Human
- [ ] AI
**Reviewed before submission by:**
- [x] Human
- [ ] AI
- [ ] Not reviewed
* GitHub Issue: #51301
Lead-authored-by: AlenkaF <frim.alenka@gmail.com>
Co-authored-by: Alenka Frim <AlenkaF@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Rok Mihevc <rok@mihevc.org>
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
The Arrow GDB pretty-printers do not support run-end encoded data types.
### What changes are included in this PR?
Add GDB pretty-printer support and tests for `run-end encoded` types, scalars, and arrays.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
* GitHub Issue: #50658
Authored-by: fenfeng9 <fenfeng9@qq.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Arrow is currently missing remainder and modulo kernels.
### What changes are included in this PR?
Add the kernels `remainder`, `remainder_checked`, `modulo`, and `modulo_checked`, following the terminology in the [divmod proposal](https://lists.apache.org/thread/h3t6nz1ys2k2hnbrjvwyoxkf70cps8sh):
* `remainder` uses truncated (C/C++) semantics — the result has the sign of the dividend, e.g. `remainder(-7, 3) == -1`.
* `modulo` uses floored (Python/R) semantics — the result has the sign of the divisor, e.g. `modulo(-7, 3) == 2`.
Both are supported for integer, floating-point and decimal inputs, and are exposed as `compute::Remainder` / `compute::Modulo` in `api_scalar.h`.
Decimal arguments are promoted to a common scale like `add`, but the result type is resolved by a dedicated resolver rather than reusing `ResolveDecimalAdditionOrSubtractionOutput`: a remainder is always smaller in magnitude than the divisor, so no extra digit is needed for a carry, and the result is `precision = max(p1, p2)`, `scale = s1`. This keeps maximum-precision inputs (`decimal128(38, 0)`, `decimal256(76, 0)`) from overflowing the decimal precision range.
Left as follow-ups: combined divmod #27909, floor division #39386, and duration support.
### Are these changes tested?
Yes. New tests cover truncated vs. floored semantics across signed/unsigned integer, floating-point and decimal types, division by zero, `INT_MIN % -1` overflow, and decimal result-type resolution (same type in/out, mixed precisions and scales, Decimal128/Decimal256 promotion, decimal/integer promotion, and maximum-precision decimals).
### Are there any user-facing changes?
Yes — four new compute functions (`remainder`, `remainder_checked`, `modulo`, `modulo_checked`), the corresponding `compute::Remainder` / `compute::Modulo` C++ APIs, and a new `BasicDecimal256 operator%`. These are additions only; no existing API is changed or removed.
* GitHub Issue: #46901
* xref: https://github.com/apache/arrow/pull/46902,
* xref: https://github.com/apache/arrow/pull/28497
Lead-authored-by: Fangchen Li <fangchen.li@outlook.com>
Co-authored-by: Rok Mihevc <54589+rok@users.noreply.github.com>
Co-authored-by: tadeja <864005+tadeja@users.noreply.github.com>
Signed-off-by: Rok Mihevc <rok@mihevc.org>
### Rationale for this change
NumPy 2.0 added `StringDType`, a variable-width UTF-8 string dtype. `pa.array` currently rejects it with `Unsupported numpy type 2056`. Now that pyarrow requires NumPy 2.0 the conversion can use the public `NpyString_*` C API.
This supersedes #50951. The first commit is @ alippai's first commit from that PR, unchanged. His later commits added batching and new bulk builder APIs in Arrow C++; those can come later in their own PR with benchmarks, as discussed there. Nothing outside the NumPy conversion code changes here.
### What changes are included in this PR?
- `NumPyDtypeToArrow` maps `StringDType` to `string`, so it is also the inferred type for `pa.array`, `pa.infer_type`, `pa.from_numpy_dtype` and lists of such arrays. `large_string` and `string_view` can be requested with `type`. Any other type raises a `TypeError` naming the requested type.
- The conversion holds the dtype's allocator lock for the whole array and never holds the GIL while taking it.
- A null entry in a StringDType array becomes an Arrow null unless the dtype's `na_object` is a `str`. NumPy reports such entries as that string from `__getitem__`, in ufuncs and comparisons, and in casts, and there is no way to tell them apart from a regular entry holding the same string, so the conversion writes the string too. NaN-like sentinels such as `np.nan` and `pd.NA`, `None`, and arbitrary objects all become arrow nulls.
- A short section in `docs/source/python/numpy.rst` describing the support and the semantics for converting missing data.
If left unfixed, https://github.com/apache/arrow/issues/51156 also becomes possible to trigger from Python by creating a StringDType array with a very large entry.
### Are these changes tested?
Yes. Tests cover the three target types, strings crossing NumPy's short, medium and long storage thresholds, non-ASCII input, strided input, `mask`, `None`, NaN and string sentinels, the rejected target type, and list inference. I also ran a threaded test that converts an array while other threads hit NumPy's null-comparison error path on the same array to check the lock ordering.
### Are there any user-facing changes?
Yes. `pa.array` and the inference functions accept `StringDType` arrays.
AI disclosure: I used an AI model to work on the follow-up commits on top of @ alippai's first commit and for code review.
* GitHub Issue: #42018
* GitHub Issue: #51652
Lead-authored-by: Nathan Goldbaum <nathan.goldbaum@gmail.com>
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Co-authored-by: Adam Lippai <adam@rigo.sk>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
### Rationale for this change
Pandas has deprecated certain aspects of `.values` (when it looses information), which our pandas->pyarrow conversion currently relies upon. So updating that code path to avoid this usage.
### What changes are included in this PR?
Removed usage of `.values` for the deprecated cases (period, interval, datetimetz). In the custom `get_values`, we already avoided this for period and interval, added a special case for converting datetimetz to a numpy array. And ensured that other cases were we did a direct `.values` goes through this `get_values` helper.
### Are these changes tested?
Yes, I added more tests first to ensure full coverage of all cases that might run into the deprecation (e.g. the `Schema.from_pandas(df)` or `pa.array(EA)` with period/datetimetz dtypes were not currently covered)
### Are there any user-facing changes?
No
### Was AI used for this PR?
No
* GitHub Issue: #51302
Authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
Use C++ type traits for `is_nested` function to avoid redefining in Python
### What changes are included in this PR?
Expose `is_nested` function
### Are these changes tested?
Yes
### Are there any user-facing changes?
Yes
* GitHub Issue: #40282
Lead-authored-by: NNic Crane <thisisnic@gmail.com>
Co-authored-by: Nic Crane <thisisnic@gmail.com>
Co-authored-by: Hyunseok Seo <hsseo0501@gmail.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
In pyarrow 24.0.0, `Scalar` and `Array` gained arithmetic dunder methods (#32007) that unconditionally dispatch to `pyarrow.compute.call_function`. When the other operand is an unrecognized type, `_pack_compute_args` raises `TypeError` instead of returning `NotImplemented`. This prevents Python from falling back to the other operand's reflected methods (`__radd__`, `__rsub__`, etc.), breaking downstream libraries that relied on this protocol.
### What changes are included in this PR?
- Adds a `_compute_binary_op` helper in `scalar.pxi` that wraps `call_function` in a `try/except TypeError` and returns `NotImplemented` on failure. This follows the same pattern already used by `Scalar.__eq__` and `Array.__eq__`.
- Updates all binary arithmetic and bitwise dunders on both `Scalar` (`scalar.pxi`) and `Array` (`array.pxi`) to use this helper.
- Adds parametrized tests in `test_scalars.py` and `test_array.py` verifying that reflected operators on custom types are correctly invoked.
### Are these changes tested?
Yes. New parametrized tests (`test_dunders_return_notimplemented_for_unknown_types`) cover all 10 binary operators for both `Scalar` and `Array`. The bug was also manually reproduced against pyarrow 24.0.0 to confirm the tests exercise the right code path.
### Are there any user-facing changes?
Yes. `Scalar` and `Array` arithmetic dunders now return `NotImplemented` instead of raising `TypeError` when the other operand is not a recognized Arrow/NumPy type. This restores the pre-24.0.0 behavior where Python would fall back to the other operand's reflected method.
### AI-generated code disclosure
This PR was developed with assistance from an AI coding tool (Claude, Anthropic). All changes have been reviewed, understood, and verified.
* GitHub Issue: #49826Closes#49826
Lead-authored-by: AlenkaF <frim.alenka@gmail.com>
Co-authored-by: Alexandros Anastasiou <anastasioyaa@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Fix#51220 - Parquet `write_table` options are not all accepted in dataset `ParquetFileWriteOptions` so `pyarrow.parquet.write_to_dataset( ...)` raises `TypeError: unexpected parquet write option:`
for `bloom_filter_options` , `write_time_adjusted_to_utc` or `store_schema`
(Similar to past #37469)
### What changes are included in this PR?
Add `bloom_filter_options`, `store_schema`, `write_time_adjusted_to_utc` to `ParquetFileWriteOptions`,
also explicitly expose `bloom_filter_options` in `pyarrow.parquet.ParquetWriter`.
Test writer options in `make_write_options`, and test three newly supported options in `write_to_dataset`
### Are these changes tested?
Yes. CI also passes.
### Are there any user-facing changes?
Yes, users may now use `bloom_filter_options` , `write_time_adjusted_to_utc` and `store_schema`
in `pyarrow.parquet.write_to_dataset(...)` and in `ParquetFileFormat.make_write_options(...)` without raising TypeError.
### Was AI used for this PR?
**PR code and description written by:**
- [x] Human
- [ ] AI
**Reviewed before submission by:**
- [x] Human
- [x] AI
- [ ] Not reviewed
* GitHub Issue: #51220
Authored-by: Tadeja Kadunc <tadeja.kadunc@gmail.com>
Signed-off-by: Rok Mihevc <rok@mihevc.org>
### Rationale for this change
`ObjectType` and `FileStatistics` in io/hdfs.h have been deprecated for a while and can be removed.
### What changes are included in this PR?
`ObjectType` and `FileStatistics` structs are removed and instead FileSystem API in `arrow::fs` is used. Together with this change, the hdfs connected code is moved from `cpp/src/arrow/io` to `cpp/src/arrow/filesystem` merging `FileSystem` and `HadoopFileSystem` classes from `arrow::io` into the public `HadoopFileSystem` class.
### Are these changes tested?
Existing tests should pass.
### Are there any user-facing changes?
Deprecated structs are removed and all hdfs related code is now a part of the filesystem module.
* GitHub Issue: #45747
Also closes: https://github.com/apache/arrow/issues/22457 (not sure about `io/interfaces.h`?)
Lead-authored-by: AlenkaF <frim.alenka@gmail.com>
Co-authored-by: Alenka Frim <AlenkaF@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
pyarrow.parquet.read_parquet does support using `None` as the partioning argument (and is required when not using partitioning).
This adjusts the documentation accordingly.
Authored-by: Marius van Niekerk <marius.v.niekerk@gmail.com>
Signed-off-by: Nic Crane <thisisnic@gmail.com>
### Rationale for this change
`starts_with`, `ends_with` and `match_substring` have no Substrait mapping, so any expression using them fails to serialize:
```python
import pyarrow as pa
import pyarrow.compute as pc
from pyarrow.substrait import serialize_expressions
schema = pa.schema([pa.field("cat", pa.string())])
serialize_expressions([pc.starts_with(pc.field("cat"), "al")], ["f"], schema)
# ArrowNotImplementedError: No conversion function exists to convert the
# Arrow function starts_with to a Substrait call
```
Comparisons, `isin` and arithmetic serialize fine, so this is a per-function gap. It matters for engines that ingest a PyArrow filter through Substrait, where an unmappable function becomes a hard failure rather than a fallback.
See #50988.
### What changes are included in this PR?
Map the three kernels onto `starts_with`, `ends_with` and `contains` from Substrait's `functions_string.yaml`, in both directions.
The signatures do not line up. Substrait passes the pattern as a second argument, while the Arrow kernels are unary and carry it in `MatchSubstringOptions`. So:
* encoding lifts `MatchSubstringOptions::pattern` out into a literal argument
* decoding requires that argument to be a non-null string literal, and returns `NotImplemented` otherwise
Substrait's `case_sensitivity` option maps onto `ignore_case`. `CASE_INSENSITIVE_ASCII` has no Arrow equivalent and returns `NotImplemented`.
Both caveats are documented in `docs/source/cpp/acero/substrait.rst`.
### Are these changes tested?
Yes.
`Substrait.StringMatchExpressionSerialization` round-trips all three functions with and without `ignore_case`. `Substrait.StringMatchExpressionDeserialization` deserializes hand-written Substrait JSON to cover the default (no option) case and the two rejection paths. `test_serializing_string_match_expressions` round-trips each function through `pyarrow.substrait`.
### Are there any user-facing changes?
Yes. `pc.starts_with`, `pc.ends_with` and `pc.match_substring` can now be serialized to Substrait and consumed back, and Acero can consume plans using Substrait's `starts_with`, `ends_with` and `contains`. No existing behaviour changes: these previously raised.
* GitHub Issue: #50988
Authored-by: Jonas Dedden <mail@jonas-dedden.de>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
`Array.slice`, `ChunkedArray.slice`, `RecordBatch.slice`, and `Table.slice`
accept Python and NumPy integer-like values but reject Arrow integer scalars,
even though those scalars implement Python's integer-index protocol. This makes
values produced by Arrow APIs unnecessarily unusable as slice offsets and
lengths.
### What changes are included in this PR?
Normalize non-`None` offsets and lengths with `operator.index()` at the four
public Python wrapper boundaries before the existing validation and C++ calls.
Add regression coverage for all signed and unsigned Arrow integer scalar types
across the four wrappers, plus NumPy compatibility, invalid and null scalars,
negative values, offset clamping, and int64 overflow behavior.
### Are these changes tested?
Yes. Current Arrow C++ and editable PyArrow were built from source in an
isolated Python 3.13 environment. The focused new tests and existing slice
controls passed: 48 passed.
Cython translation, Python test-file compilation, Flake8, and
`git diff --check` also passed.
### Are there any user-facing changes?
Yes. The four Python `.slice()` methods now accept non-null Arrow integer
scalars for offsets and lengths. Existing Python/NumPy integer behavior and
error behavior for unsupported values are preserved.
### AI assistance
AI assisted analysis, implementation, test drafting, build work, and review.
The account holder reviewed and approved the final four-file diff. The account
holder did not personally run the commands reported above.
* GitHub Issue: #51145
Authored-by: YusefSyed <211442445+YusefSyed@users.noreply.github.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
Fixes test collection failing in nightly builds that don't have Parquet encryption enabled.
### What changes are included in this PR?
Don't reference `InMemoryKmsClient` in function signatures, it's not defined if `pyarrow.parquet.encryption` can't be imported.
### Are these changes tested?
Yes, I reproduced the crash collecting tests without encryption enabled and verified this is fixed, and also verified the tests still work with encryption enabled.
### Are there any user-facing changes?
No
### Was AI used for this PR?
In accordance to the [AI generation guidelines](https://arrow.apache.org/docs/dev/developers/overview.html#ai-generated-code), please disclose below whether and how AI was used in this PR.
**PR code and description written by:**
- [x] Human
- [ ] AI
**Reviewed before submission by:**
- [x] Human
- [ ] AI
- [ ] Not reviewed
* GitHub Issue: #51446
Authored-by: Adam Reeve <adreeve@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Resolves#51300, which mentioned that deprecation warnings were being printed when executing the python tests.
### What changes are included in this PR?
This migrates the test suite to use the non-deprecated api for `null_placement`
### Are these changes tested?
Yes, python compute tests are still passing
### Are there any user-facing changes?
No.
Authored-by: Alexander Taepper <alexander.taepper@gmail.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
Addresses #51349
### What changes are included in this PR?
* Adds a new `read_kms_url` member to `DecryptionConfiguration` and a parameter with the same name to `rotate_master_keys`. Both are false by default.
* Adds these to the PyArrow bindings.
### Are these changes tested?
Yes, this includes new unit tests.
### Are there any user-facing changes?
Yes, this adds a new user-facing option. Users that relied on this behaviour previously will now need to opt-in and enable the flag.
* GitHub Issue: #51349
Authored-by: Adam Reeve <adreeve@gmail.com>
Signed-off-by: Adam Reeve <adreeve@gmail.com>
### Rationale for this change
Calling resize on a memory-mapped file that has never been opened takes down the Python process with a SIGSEGV, so an application cannot catch or recover from it.
### What changes are included in this PR?
The resize path now applies the same open-state check the other file operations already use, before it reaches the native resize call. A directly constructed object raises ValueError.
### Are these changes tested?
The regression runs in-process and checks ValueError and its message with pytest.raises. It does not spawn a subprocess.
Separate, 20-second-bounded processes reproduced the original crash and the exception after the fix. Both used native Arrow 26.0.0-SNAPSHOT; the reused patched extension has byte-identical resize source to this PR.
<details>
<summary>Raw logs</summary>
```console
$ python -c 'import pyarrow as pa; pa.MemoryMappedFile().resize(0)'
Before:
returncode=-11
After:
ValueError: I/O operation on closed file
returncode=1
$ cd python
$ python -m pytest pyarrow/tests/test_io.py -k memory_map_resize -q
.. [100%]
2 passed, 161 deselected, 1 warning in 1.02s
```
</details>
### Are there any user-facing changes?
Yes. Misuse of a directly constructed memory-mapped file now raises a Python exception instead of terminating the process.
**This PR contains a "Critical Fix".** It fixes a process crash reachable from ordinary Python-level object state.
* GitHub Issue: #51229
Lead-authored-by: 1fanwang <1fannnw@gmail.com>
Co-authored-by: Stefan Wang <1fannnw@gmail.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
Fixes#51361.
### What changes are included in this PR?
Method `FileReaderImpl::ReadColumn` should derive `records_to_read` from the RowGroup rather than ColumnChunk's `num_values`. Deriving the right ColumnChunk index in `FileReaderImpl::DecodeRowGroups` is not trivial for nested schemas. This simplifies `FileReaderImpl::ReadColumn` and fixes#51361.
This was silently masked for full-schema reads and for columns with identical num_values(), but surfaces as a hard failure when an earlier, unselected column requires decryption: reading only a trailing plaintext column of a partially column-key-encrypted, plaintext-footer Parquet file threw "Cannot decrypt ColumnMetadata" even though the requested column was never encrypted.
This never corrupts data on unencrypted files: ReadColumn's wrong index is only ever used to look up ColumnChunk(i)->num_values(), a count fed into the *already-correct* reader as an upper bound on how many records to decode. Every row contributes at least one definition/repetition-level entry, so num_values() for any column is always >= that row group's true row count, and every column in a row group shares the same row count.
### Are these changes tested?
Yes, in the context of reading a plaintext column of a partially encrypted Parquet file. This cannot be tested with non-encrypted files.
### Are there any user-facing changes?
No.
### Was AI used for this PR?
In accordance to the [AI generation guidelines](https://arrow.apache.org/docs/dev/developers/overview.html#ai-generated-code), please disclose below whether and how AI was used in this PR.
**PR code and description written by:**
- [X] Human
- [X] AI
**Reviewed before submission by:**
- [X] Human
- [ ] AI
- [ ] Not reviewed
* GitHub Issue: #51361
Lead-authored-by: Enrico Minack <enrico.minack@insightsoftmax.com>
Co-authored-by: Enrico Minack <github@enrico.minack.dev>
Signed-off-by: Adam Reeve <adreeve@gmail.com>
### Rationale for this change
Fix three minor issues as discussed in https://github.com/apache/arrow/pull/51358#issuecomment-5710568289
### What changes are included in this PR?
- Fix the punctuation in title-check comment.
- Remove an extra parenthesis from the `pyrunfile` link in the MATLAB documentation.
- Fix the `buffers` parameter name in the `buffers_to_array` documentation.
### Are these changes tested?
These are minor documentation fixes.
### Are there any user-facing changes?
Yes.
### Was AI used for this PR?
In accordance to the [AI generation guidelines](https://arrow.apache.org/docs/dev/developers/overview.html#ai-generated-code), please disclose below whether and how AI was used in this PR.
**PR code and description written by:**
- [X] Human
- [ ] AI
**Reviewed before submission by:**
- [X] Human
- [ ] AI
- [ ] Not reviewed
Authored-by: Aaditya Srinivasan <aadityasri03@gmail.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
The doc page https://arrow.apache.org/docs/python/generated/pyarrow.parquet.write_table.html does not show `use_content_defined_chunking` as an argument of `write_table`:
<img width="1571" height="821" alt="Screenshot From 2025-09-04 14-45-32" src="https://github.com/user-attachments/assets/a9fb4d7f-9b4d-4ce6-92c8-97b781831f33" />
Note that it's documented below in the list of parameters:
<img width="1800" height="1062" alt="Screenshot From 2025-09-04 14-46-20" src="https://github.com/user-attachments/assets/d9d6c289-97c3-476b-820d-eeb30f8f791f" />
### What changes are included in this PR?
Change the API signatures of `pyarrow.parquet.ParquetWriter` and `pyarrow.parquet.write_table`: `use_content_defined_chunking` parameter is explicitly added, with a default value (`False`), as mentioned by the comment:
https://github.com/apache/arrow/blob/512dcd77dcf3910c41a06d2a1620c729d92d6748/python/pyarrow/parquet/core.py#L1953-L1954
Reorder the parameters in docstring `_parquet_writer_arg_docs` to match the method signature.
Add a test.
Also users won't get this error anymore `TypeError: unexpected parquet write option: use_content_defined_chunking` when running with released wheel (since 21.0.0) :
```shell
python -c "import pyarrow.dataset as ds; ds.ParquetFileFormat().make_write_options(use_content_defined_chunking=True)"
```
or when running :
```python
import pyarrow as pa, pyarrow.parquet as pq
pq.write_to_dataset(pa.table({'a':[1, 2, 3]}), 'cdc_dataset', use_content_defined_chunking=True)
```
### Are these changes tested?
Yes
### Are there any user-facing changes?
Yes:
1. `use_content_defined_chunking` parameter is explicitly added to the API signature and documentation of `pyarrow.parquet.ParquetWriter` and `pyarrow.parquet.write_table`
2. `use_content_defined_chunking` is now accepted in `pyarrow.dataset.ParquetFileFormat.make_write_options` and `pyarrow.parquet.write_to_dataset` instead of raising `TypeError`
* GitHub Issue: #47499
Lead-authored-by: Sylvain Lesage <severo@rednegra.net>
Co-authored-by: Tadeja Kadunc <tadeja.kadunc@gmail.com>
Signed-off-by: Tadeja Kadunc <tadeja.kadunc@gmail.com>
### Rationale for this change
This PR continues the simdjson migration by replacing the RapidJSON-based JSON boundary detection used by the JSON chunker.
The existing implementation uses RapidJSON's streaming parser to identify complete JSON values. This change replaces that logic with the simdjson streaming parser.
### Are these changes tested?
Yes, by existing tests.
### Are there any user-facing changes?
No, except perhaps slight differences in error message and timing of error reporting when reading from an invalid JSON stream (i.e. there may be cases where a JSON parse error is reported earlier or later while iterating over RecordBatches).
* GitHub Issue: #50944
Lead-authored-by: Aaditya Srinivasan <aadityasri03@gmail.com>
Co-authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
**Rationale for this change**
In arrow::dataset::MakeReaderProperties, all parquet::ReaderProperties options had to be copied field-by-field from parquet_scan_options->reader_properties because ReaderProperties did not support mutating or setting MemoryPool* after construction.
**What changes are included in this PR?**
Added set_memory_pool to parquet::ReaderProperties in cpp/src/parquet/properties.h.
Added unit test TestReaderProperties.SetMemoryPool in cpp/src/parquet/properties_test.cc.
Are these changes tested?
Yes
Are there any user-facing changes?
No
* GitHub Issue: #51264
Authored-by: Diveyam Mishra <mishradiveyam@gmail.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
`utf8_normalize` with `form=NFC` or `NFKC` returned the decomposed forms (NFD, NFKD). `Utf8NormalizeBase`
builds the right `utf8proc` options for each form but only calls `utf8proc_decompose()`, which
decomposes regardless of `UTF8PROC_COMPOSE`; the composition step lives in
`utf8proc_normalize_utf32()`, which the kernel never called. See #51225.
### What changes are included in this PR?
- After a successful `utf8proc_decompose()`, when the options include `UTF8PROC_COMPOSE`, call
`utf8proc_normalize_utf32()` on the scratch buffer. It composes in place and returns the new code
point count; the existing UTF-8 encode loop is unchanged. NFD and NFKD take the same path as before.
- Fix the `json_composed` fixture in `scalar_string_test.cc`: its bytes were the decomposed form
(`61 CC 81`), the same string as `json_decomposed`, so the compose assertions were comparing a value
with itself and passed with the bug. Thanks to `@ Santoshkumarpuppala` for spotting that on the issue.
- Add composed/decomposed pairs (U+00E9, and a Hangul syllable with its jamo) to
`test_utf8_normalize` in pyarrow; the existing input, U+00B2, is its own NFC.
### Are these changes tested?
Yes. With the corrected fixture, `TestStringKernels.Utf8Normalize` fails on the unpatched kernel and
passes with this change; the pyarrow test covers the composed forms from Python.
### Are there any user-facing changes?
Yes: `utf8_normalize` with `NFC` and `NFKC` now returns composed output. Callers that depended on the
previous (decomposed) result for those forms will see different bytes.
* GitHub Issue: #51225
Authored-by: singhpratech <singh.prateek86@outlook.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
There are already utilities to import tensor data from NumPy.
With DLPack standard, it will work across many tensor providers.
### What changes are included in this PR?
- Bind `FixedShapedTensorArray::FromTensor` in Python
- Add `ImportArrayVersionedFromDLPack` and `ImportTensorVersionedFromDLPack` in C++
- Add in Python `Array.from_dlpack` and `Tensor.from_dlpack`
Similar to the export, multidimensional tensor import require an explicit step through `Tensor`.
- No `Array::FromTensor` or even `FixedSizeListArray::FromTensor`. IMHO it does not feel as necessary in this direction but open to anyone's take one it.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes, new APIs.
* GitHub Issue: #39295
Lead-authored-by: AntoinePrv <AntoinePrv@users.noreply.github.com>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Calling `pyarrow.compute.FunctionOptions.deserialize()` with anything that is not a `Buffer` crashes the interpreter. `None`, an `int`, a `list` and a `bytes` object all segfault, so a caller who passes the wrong object, or who forwards an unvalidated value, loses the process rather than seeing an exception.
`deserialize()` hands its argument to `pyarrow_unwrap_buffer()`, which returns a null pointer for a non-`Buffer`. The `deref()` on the next line then dereferences null.
Before this change the call terminates the process. After it, the call raises `TypeError` naming the type it received.
### What changes are included in this PR?
The static method now declares its parameter as `Buffer buf not None`, which is what its docstring already says it takes. Cython rejects a wrong type before the unwrap runs, so the null pointer is never produced.
### Are these changes tested?
Yes, by a parametrized case over the four inputs above.
<details>
<summary>Red, with this commit's source file reverted to its parent</summary>
```console
$ git checkout HEAD~1 python/pyarrow/_compute.pyx
$ pip install -e python --no-build-isolation
$ python -X faulthandler -c "import pyarrow.compute as pc; pc.FunctionOptions.deserialize(None)"
Fatal Python error: Segmentation fault
Current thread 0x00000001f6fbe180 (most recent call first):
File "<string>", line 1 in <module>
Extension modules: numpy._core._multiarray_umath, numpy.linalg._umath_linalg, pyarrow.lib, pyarrow._compute (total: 4)
```
The same crash occurs for `1`, `[]` and `b''`.
</details>
<details>
<summary>Green, with the fix applied</summary>
```console
$ git checkout HEAD -- python/pyarrow/_compute.pyx
$ pip install -e python --no-build-isolation
$ python -m pytest pyarrow/tests/test_compute.py -k deserialize_rejects_non_buffers -v
collected 607 items / 603 deselected / 4 selected
pyarrow/tests/test_compute.py::test_function_options_deserialize_rejects_non_buffers[None] PASSED [ 25%]
pyarrow/tests/test_compute.py::test_function_options_deserialize_rejects_non_buffers[1] PASSED [ 50%]
pyarrow/tests/test_compute.py::test_function_options_deserialize_rejects_non_buffers[value2] PASSED [ 75%]
pyarrow/tests/test_compute.py::test_function_options_deserialize_rejects_non_buffers[] PASSED [100%]
====================== 4 passed, 603 deselected in 0.06s =======================
```
```console
$ for v in None 1 "[]" "b''"; do python -c "
import pyarrow.compute as pc
try:
pc.FunctionOptions.deserialize($v)
except TypeError as e:
print('$v', '->', e)
"; done
None -> Argument 'buf' has incorrect type (expected pyarrow.lib.Buffer, got NoneType)
1 -> Argument 'buf' has incorrect type (expected pyarrow.lib.Buffer, got int)
[] -> Argument 'buf' has incorrect type (expected pyarrow.lib.Buffer, got list)
b -> Argument 'buf' has incorrect type (expected pyarrow.lib.Buffer, got bytes)
```
</details>
A valid buffer still round-trips, and the rest of `test_compute.py` is unaffected.
<details>
<summary>Round-trip and full suite</summary>
```console
$ python -c "import pyarrow.compute as pc; o = pc.ArraySortOptions(order='descending'); print(pc.FunctionOptions.deserialize(o.serialize()) == o)"
True
$ python -m pytest pyarrow/tests/test_compute.py -q
584 passed, 23 skipped, 11 warnings in 3.09s
```
</details>
### Are there any user-facing changes?
Passing a non-`Buffer` to `FunctionOptions.deserialize()` now raises `TypeError` instead of terminating the process. Callers already passing a `Buffer`, which is what the docstring documents, are unaffected.
* GitHub Issue: #51041
Authored-by: 1fanwang <1fannnw@gmail.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
Reconstructing a nested Schema from the Parquet Thrift metadata implies a recursive call that can blow up the stack on pathologically-nested schemas (with thousands of nesting levels or more).
By adding a limit on the schema nesting depth, we turn a stack overflow-induced crash into a regular Parquet error.
### Are these changes tested?
By additional unit tests; also privately with a proof-of-concept reproducer that induces a stack overflow exhaustion.
### Are there any user-facing changes?
In the unlikely case where a legitimate Parquet file has a deeper schema than the default schema nesting limit in this PR (100), an error will be raised when reading where it used to succeed. The user can bump the limit to circumvent the error.
**This PR contains a "Critical Fix".** It fixes a crash on a deeply nested Parquet schema that would provoke a stack overflow. It is not an exploitable vulnerability except through denial of service.
Thanks to "1K0CT" for the initial report.
* GitHub Issue: #51238
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Expose remaining csv write options to pyarrow.
### What changes are included in this PR?
Adding `eol` and `null_string` to csv.WriteOptions.
### Are these changes tested?
Yes, testing of setters included.
### Are there any user-facing changes?
* GitHub Issue: #34577
Lead-authored-by: Martin Nowak <code@dawg.eu>
Co-authored-by: AlenkaF <frim.alenka@gmail.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
`BufferReader.readinto()` currently segfaults when passed a read-only destination such as `bytes` or a read-only `memoryview`. `py_buffer()` creates an immutable Arrow buffer, `mutable_data()` returns a null pointer, and the C++ read path attempts to copy into that pointer. A Python API misuse should raise a Python exception rather than terminate the interpreter.
### What changes are included in this PR?
- Validate that a `NativeFile.readinto()` destination is mutable before obtaining its writable pointer.
- Raise `TypeError` for immutable destinations.
- Add regression coverage for `bytes` and read-only `memoryview` destinations.
### Are these changes tested?
Yes. The focused `readinto` tests pass against a locally compiled patched `pyarrow.lib`. The stock 25.0.1 wheel exits with status 139 for the issue reproducer, while the patched build raises the expected `TypeError`; the writable `bytearray` control continues to read successfully.
### Are there any user-facing changes?
Yes. Passing a read-only destination to `NativeFile.readinto()` now raises `TypeError` instead of terminating the interpreter. Writable-buffer behavior is unchanged.
Fixes#51044.
This contribution was developed with assistance from OpenAI Codex. I reviewed, tested, and take responsibility for the patch and this description.
* GitHub Issue: #51044
Authored-by: Darren Carreras <283775510+carrerasdarren-cell@users.noreply.github.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
This might not strictly speaking be a valid "minor" issue since it changes code, but it should keep the logic the same and (for me) clarify the intent of it.
### Are these changes tested?
By existing tests
### Are there any user-facing changes?
No
Authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### What changes are included in this PR?
Remove a CUDA test that tries to access an invalid pointer (after the owning resource was deallocated).
On old CUDA setups it used to raise an exception:
```
Cuda error 1 in function 'cuMemcpyDtoH': [CUDA_ERROR_INVALID_VALUE] invalid argument
```
but more recently it has simply started crashing.
### Are these changes tested?
By existing CI tests. This should fix a crash in the CUDA Python CI.
### Are there any user-facing changes?
No.
* GitHub Issue: #12594
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
## Rationale for this change
`Identity<Max>` (the implicit default start of `cumulative_max`) used `std::numeric_limits<T>::min()`, which is the smallest **positive** value for floating-point types. Any input starting with a non-positive value therefore never replaced the start as the new maximum:
```python
>>> import pyarrow as pa, pyarrow.compute as pc
>>> pc.cumulative_max(pa.array([-2.5, 2.5])).to_pylist()
[2.2250738585072014e-308, 2.5] # expected [-2.5, 2.5]
```
## What changes are included in this PR?
- `Identity<Max>` now uses `-infinity` for floating-point types (and half-float), and keeps `lowest()` for integer types. `-infinity` is the only value satisfying the documented identity property `Op(identity, x) = x for all x` — `lowest()` (most negative finite value) fails it for a leading `-inf` input.
- `Identity<Min>` gets the mirror fix: `+infinity` for floating-point types. Previously `cumulative_min([inf, 1.0])` returned `[DBL_MAX, 1.0]` instead of `[inf, 1.0]`.
Closes#51194.
## Are these changes tested?
- Added `TestCumulative.NegativeValues` covering negative integers, negative floats, and `±Inf` inputs for both `cumulative_max` and `cumulative_min`.
- Added a pyarrow regression test reproducing the issue (`test_cumulative_max_min_negative_default_start`).
## Are there any user-facing changes?
Yes: `cumulative_max`/`cumulative_min` now return correct results for floating-point inputs whose first value is non-positive (for max) / non-negative or `inf` (for min). The documented behavior — 'the default start is the minimum/maximum value of input type' — is now actually honored.
* GitHub Issue: #51194
Authored-by: Adarsh <adarsh@neuroad.ai>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Have the same coverage as `to_numpy`.
### What changes are included in this PR?
- `FixedShapeTensorArray.__dlpack__` explicitly calling `to_tensor`
- `FixedShapeTensorArray.__dlpack_device__` (defaulted)
- `FixedShapeTensorScalar.__dlpack__` explicitly calling `to_tensor`
- `FixedShapeTensorScalar.__dlpack_device__` calling C++ device extraction on the underlying array.
If GH-51122 get merged first, I'll add the consuming side here, otherwise if this one gets merged first, I'll add it there.
### Are these changes tested?
Yes a few since this does not introduce new logic.
### Are there any user-facing changes?
Additions only.
* GitHub Issue: #38868
Authored-by: AntoinePrv <AntoinePrv@users.noreply.github.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
`pyarrow.types.is_temporal()` returns True for interval types, but its docstring doesn't say so:
```python
>>> import pyarrow as pa, pyarrow.types as t
>>> t.is_temporal(pa.month_day_nano_interval())
True
>>> t.is_temporal.__doc__.strip().splitlines()[0]
'Return True if value is an instance of type: date, time, timestamp or duration.'
```
That's the documented contract being narrower than the behaviour, and since these docstrings are generated from the `@ doc(is_null, datatype=...)` decorator they're what ends up on the API docs page.
Including interval is deliberate rather than accidental — `_TEMPORAL_TYPES` is built as
```python
_TEMPORAL_TYPES = ({lib.Type_TIMESTAMP,
lib.Type_DURATION} | _TIME_TYPES | _DATE_TYPES |
_INTERVAL_TYPES)
```
and `test_is_temporal_date_time_timestamp` already asserts `types.is_temporal(pa.month_day_nano_interval())`. So this is a docs fix, not a behaviour question; I've only changed the `datatype` string.
After the change the rendered line is
```
Return True if value is an instance of type: date, time, timestamp, duration or interval.
```
which is exactly the set that returns True — I checked each one (date32, time32, timestamp, duration, interval all True; int32 False) by applying the same edit to an installed pyarrow and reading the generated docstring back.
I found this while comparing every predicate set in `types.py` against the corresponding `arrow::is_*()` in `cpp/src/arrow/type_traits.h`, the same way GH-50847 turned up. Worth noting the two `is_temporal` definitions genuinely differ — the C++ one covers only date/time/timestamp, while Python also counts duration and interval — but that's a semantic question rather than something to change quietly in a MINOR PR, so I've left it alone and only made the Python docstring match the Python behaviour.
Authored-by: nishad shabbir <nishadshabbir28@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Fix a typo in the ORC writer `compression` docstring where `LZ0` was incorrectly written instead of `LZO`.
### What changes are included in this PR?
Change `LZ0` to `LZO` in the `python/pyarrow/orc.py` documentation.
### Are these changes tested?
Documentation-only change; no code behavior is affected.
### Are there any user-facing changes?
No
* GitHub Issue: #50897
Authored-by: Maulik Patel <21225464+maupatel@users.noreply.github.com>
Signed-off-by: Nic Crane <thisisnic@gmail.com>
### Rationale for this change
Enable multidimensional DLPack support for Array via `to_tensor`.
### What changes are included in this PR?
- Add virtual `Array::ToTensor`
- Add `NumericArray::ToTensor` for 1D arrays
- Add `FixedSizeListArray::ToTensor` for multidimensional arrays
- Add DLPack error suggestiong tensor convertion
- Add DLPack tests with `arr.to_tensor().__dlpack__()`
Note: Nulls are explicitly supported in `to_tensor` as unspecified data. This was the current behaviour.
### Are these changes tested?
Yes
### Are there any user-facing changes?
New public Array function.
* GitHub Issue: #38868
Authored-by: AntoinePrv <AntoinePrv@users.noreply.github.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
The invalid-operator error prints the column name instead of the operator:
```
>>> pq.filters_to_expression([[('a', '=!', 1)]])
ValueError: "a" is not a valid operator in predicates.
```
The message means `op`, the value that failed to match; `col` was already consumed by `ds.field(col)` above. The mixup came in 992bee2c8, when a `.format()` over the whole `(col, op, val)` tuple was converted to an f-string. First shipped in 21.0.0.
### What changes are included in this PR?
The message interpolates `op`. The existing test already passes an invalid operator and only checked the exception type, so it gains a `match=`.
### Are these changes tested?
The `match=` fails on unpatched main:
```
E AssertionError: Regex pattern did not match.
E Expected regex: '"=<" is not a valid operator'
E Actual message: '"integers" is not a valid operator in predicates.'
```
`pyarrow/tests/parquet/` passes 316 either way.
### Are there any user-facing changes?
Only the message text. Same exception, same trigger.
* GitHub Issue: #47369
Authored-by: Advit Arora <advitarora2@gmail.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
Round-tripping an empty table through `to_struct_array()` and
`Table.from_struct_array()` raises
`ValueError: Must pass schema, or at least one RecordBatch`.
`to_struct_array()` returns a zero-chunk struct `ChunkedArray`, so
`from_struct_array()` has no batch from which to recover the schema.
For a zero-chunk input, `from_struct_array()` now calls `combine_chunks()` and
converts the resulting empty struct array into a `RecordBatch`. Nonempty inputs
keep the existing path. An empty non-struct `ChunkedArray` still raises the
expected `TypeError`.
### What changes are included in this PR?
The empty `ChunkedArray` path creates one empty batch before calling
`Table.from_batches`. The regression coverage checks the round-trip result,
schema preservation, and the invalid non-struct input.
### Are these changes tested?
I ran this script against PyArrow 25.0.1 and the compiled branch:
```bash
python - <<'PY'
import pyarrow as pa
value = pa.chunked_array(
[], type=pa.struct([("ints", pa.int32()), ("floats", pa.float32())])
)
print(f"pyarrow={pa.__version__}, chunks={value.num_chunks}")
try:
result = pa.Table.from_struct_array(value)
print(f"rows={result.num_rows}, schema={result.schema}")
except Exception as error:
print(f"{type(error).__name__}: {error}")
PY
```
Before:
```text
pyarrow=25.0.1, chunks=0
ValueError: Must pass schema, or at least one RecordBatch
```
After:
```text
pyarrow=26.0.0.dev175+ge4ad179ae, chunks=0
rows=0, schema=ints: int32
floats: float
```
The focused regression suite also runs against the compiled branch:
```text
$ python -m pytest python/pyarrow/tests/test_table.py -k from_struct_array -q
....... [100%]
7 passed, 208 deselected in 0.51s
```
### Are there any user-facing changes?
`Table.from_struct_array()` now returns the expected empty table for a
zero-chunk struct `ChunkedArray` instead of raising `ValueError`.
* GitHub Issue: https://github.com/apache/arrow/issues/48344
* GitHub Issue: #48344
Lead-authored-by: 1fanwang <1fannnw@gmail.com>
Co-authored-by: Alenka Frim <AlenkaF@users.noreply.github.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
In `pyarrow.feather.FeatherDataset.read_pandas`, the docstring previously stated:
```
Read multiple Parquet files as a single pandas DataFrame
```
which was copied from Parquet reader utilities.
### What changes are included in this PR?
Updates the docstring to correctly reference Feather files:
```
Read multiple Feather files as a single pandas DataFrame
```
### Are these changes tested?
Docstring update.
Authored-by: Arnav Ahire <arnavahire5@gmail.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
`pyarrow.types.is_nested()` said a run-end encoded type wasn't nested, while the C++ `arrow::is_nested()` says it is:
```python
>>> import pyarrow as pa
>>> t = pa.run_end_encoded(pa.int32(), pa.string())
>>> t.num_fields
2
>>> pa.types.is_nested(t)
False
```
The Python side reads from a hardcoded `_NESTED_TYPES` set in `python/pyarrow/types.py` rather than from the C++ trait, so it has to be kept in step by hand. Lining that set up against `is_nested()` in `cpp/src/arrow/type_traits.h`, run-end encoded was the only type the two still disagreed about — the list-view types and fixed-size list are already there.
It's also out of step with the type itself. Run-end encoded has two children, the run ends and the values, and every other type in pyarrow that has children answers `True` here.
This is the same thing that happened to fixed-size list in #40171, fixed by #40172, and the list-view types were added after that. This looks like the last one missed when run-end encoding went in.
### What changes are included in this PR?
Adds `Type_RUN_END_ENCODED` to `_NESTED_TYPES`, which is the whole fix.
In the test I've added the run-end encoded case, and while I was there also a map and a dictionary. Map was nested already but wasn't asserted anywhere, and dictionary is the interesting negative — it wraps a value type but is deliberately not nested in either implementation, so pinning it means a later change can't quietly sweep it in.
### Are these changes tested?
Yes. `test_is_nested_or_struct` fails on the current code and passes with the change.
I don't have a local C++ build, so I checked this by running the updated `test_types.py` against an installed pyarrow 25.0.0 with the same one-line change applied to its `types.py`. Before the change that file had 87 passing with `test_is_nested_or_struct` failing; after it, 88 passing. The two errors and one failure I see in both runs are environmental on my machine and unrelated — the errors are the `pickle_module` fixture, which comes from a conftest I wasn't loading, and the failure is `test_pytz_timezone_roundtrip`.
### Are there any user-facing changes?
Yes, though it's small. `pa.types.is_nested()` now returns `True` for run-end encoded types where it previously returned `False`. Anything branching on that predicate will take the nested path for these types, which is the intended answer and what the C++ implementation has always given. Nothing inside pyarrow reads `is_nested` or `_NESTED_TYPES`, so the effect is limited to callers.
* GitHub Issue: #50847
Authored-by: nishad shabbir <nishadshabbir28@gmail.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
Require Numpy 2.0 to support newer features, allow newer numpy API/ABI
### What changes are included in this PR?
- Numpy 2.0+ requirement
- Pandas 2.2.2+ requirement (first wheel compatible with numpy 2.0)
- Pyodide 0.27.1 usage (numpy 2.0 api)
### Are there any user-facing changes?
Yes, in contrast to pyarrow 25, after merging this PR pyarrow 26 can't be used with Numpy 1.x or pandas 2.0-2.2.1
**This PR includes breaking changes to public APIs.**
This PR was created using GPT-5.6-Sol-xhigh, every line read & reviewed by me.
* GitHub Issue: #48473
Lead-authored-by: Adam Lippai <adam@rigo.sk>
Co-authored-by: Ádám Lippai <adam@rigo.sk>
Co-authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Pure version bump to enable implementing more features.
### What changes are included in this PR?
- Change the DLPack header file and definition.
- Use `raw_data` for `Tensor` similar to `Array` (instead of `mutable_raw_data` that is not available on immutable tensors).
- Strides may no longer be `NULL`
- Add `ExportArrayVersioned` and `ExportTensorVersioned` C APIs
- Support for `copy`
- Support for `max_version` and `copy` in `__dlpack__`
- Add `dl_device` to `__dlpack__` parameters as the spec requires but only CPU supported at the moment
### Are these changes tested?
Yes, with existing tests.
### Are there any user-facing changes?
* GitHub Issue: #41670
Authored-by: AntoinePrv <AntoinePrv@users.noreply.github.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
Converting a Table with an `arrow.uuid` extension column to pandas currently produces a column of `bytes` instead of `uuid.UUID` objects. Without a dedicated conversion path, `Table.to_pandas()` / `Array.to_pandas()` falls back to the storage type (`fixed_size_binary(16)`) and materializes raw bytes. Downstream callers (and tests such as those landing via pandas UUID Parquet coverage) expect `uuid.UUID` objects, matching what `to_pylist()` / `UuidScalar.as_py()` already return.
Note: the original issue suggested this might be specific to Python 3.14, but the same `bytes` result reproduces on Python 3.10-3.14 with current PyArrow - `UuidType` never had a pandas conversion path.
Closes#50312
### What changes are included in this PR?
Convert UUID extension arrays to `uuid.UUID` objects in the C++ pandas conversion path (integrated from @ rok's proposal in rok/arrow#55), rather than relying on a Python `to_pandas_dtype()` / `__from_arrow__` compat layer:
- Construct `uuid.UUID` objects from storage bytes in `helpers.cc` (`UuidFromBytes`) and use that from `arrow_to_pandas.cc`
- Reuse empty kwargs / shared empty tuple constants so conversion does not allocate per element
- Keep `to_numpy(zero_copy_only=False)` on UUID extension arrays returning storage `bytes` (extension to NumPy should follow storage type)
- Add/extend regression tests for pandas round-trip and NumPy behavior
### Are these changes tested?
Yes.
Acceptance criteria:
- [x] Tests added for the UUID to pandas round-trip (and NumPy storage behavior)
- [x] Relevant local / CI Python tests passing for the touched paths
- [x] Follows project style / review feedback (lint fix included)
- [x] No intentional breaking public API change beyond correcting incorrect `bytes` return values to `uuid.UUID` for `to_pandas()`
Before / after evidence (backend; console):
```text
# Before (repro on Python 3.10-3.14 with unfixed PyArrow)
>>> type(result_df.loc[0, "id"])
<class 'bytes'>
# After (with this PR)
>>> type(result_df.loc[0, "id"])
<class 'uuid.UUID'>
```
Also covered by added unit tests in `python/pyarrow/tests/parquet/test_data_types.py` and `python/pyarrow/tests/test_extension_type.py`.
### Are there any user-facing changes?
Yes. `Table.to_pandas()` / `Array.to_pandas()` now returns `uuid.UUID` for `arrow.uuid` columns instead of `bytes`. `to_numpy(zero_copy_only=False)` continues to expose storage `bytes`.
* GitHub Issue: #50312
Authored-by: parker-cassar <parker.cassar@berkeley.edu>
Signed-off-by: Rok Mihevc <rok@mihevc.org>
### Rationale for this change
Fix the return value of `ParquetLogicalType.type` for geometry/geography types
### Are these changes tested?
Yes
### Are there any user-facing changes?
Not apart from the bug fix
* GitHub Issue: #50849
Authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
Omkar Kabde