mirror of
https://github.com/apache/arrow.git
synced 2026-10-29 00:56:31 +08:00
main
2353
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Omkar Kabde
|
b8079cd294 |
GH-54637: [Python] add MemoryPool support to pq.read_table (#54919)
### Rationale for this change `pq.read_table` has no way to pass a memory pool. `ParquetReader` and `Dataset.to_table` already accept one, it just wasn't added. ### What changes are included in this PR? I added a `memory_pool` argument to `read_table`, `ParquetFile` and `ParquetDataset`. It is passed to `ParquetReader` and to `Dataset.to_table`, with docstrings updated. ### Are these changes tested? Yes. I added new tests in `test_basic.py` - It reads a Parquet file with a tracked memory pool and check that the pool was actually used. They cover `read_table`, `ParquetFile` and `ParquetDataset`. ### Are there any user-facing changes? New optional `memory_pool` keyword. No breaking changes. ### Was AI used for this PR? **PR code and description written by:** - [x] Human - [x] AI **Reviewed before submission by:** - [x] Human - [ ] AI Github Issue - https://github.com/apache/arrow/issues/54637 * GitHub Issue: #54637 Authored-by: Omkar Kabde <omkarkabde@gmail.com> Signed-off-by: AlenkaF <frim.alenka@gmail.com> |
||
|
Rok Mihevc
|
4a6ebc06dc |
GH-49153: [Python] Remove APIs deprecated before 15.0.0 (#53543)
### Rationale for this change We have many deprecated APIs in Python we should slowly remove. ### What changes are included in this PR? Several deprecation messages are acted on. ### Are these changes tested? By the existing test suite. ### Are there any user-facing changes? Yes, some deprecated user-facing APIs are.removed. ### Was AI used for this PR? In accordance to the [AI generation guidelines](https://arrow.apache.org/docs/dev/developers/overview.html#ai-generated-code), please disclose below whether and how AI was used in this PR. **PR code and description written by:** - [ ] Human - [x] AI **Reviewed before submission by:** - [x] Human - [x] AI - [ ] Not reviewed * GitHub Issue: #49153 Authored-by: Rok Mihevc <rok@mihevc.org> Signed-off-by: Antoine Pitrou <antoine@python.org> |
||
|
Dricmoy Bhattacharjee
|
1776578067 |
GH-51351: [C++][Parquet] Use default KMS connection values (#53996)
### Rationale for this change `FileKeyWrapper` calls `SetDefaultIfEmpty()` on its stored configuration, but passes the original, unmodified configuration to `GetKmsClient`. This allows the KMS client factory to receive empty instance ID and URL values even though Arrow has already filled in defaults on its internal copy. ### What changes are included in this PR? - Pass the defaulted `kms_connection_config_` member to `GetKmsClient`. - Update C++ and Python key-rotation tests to expect Arrow's existing default values for empty fields. - Separate the writer and reader crypto factories in the dataset encryption tests. The dataset tests give the writer all keys and the reader only some. With a shared factory, the reader can reuse the writer's cached client, which already has all the keys. Separate factories ensure the reader uses its own keys, so the missing-key checks work correctly. The write-path fix exposes this because writing and reading now use the same default instance ID, where empty and `"DEFAULT"` IDs previously selected separate cache entries. ### Are these changes tested? - Local C++ encryption key-management suite: 24 tests passed. - Both local Python encryption test files: 62 passed, 1 skipped. The skipped case requires a build without encryption support. - C++ and Python formatting and lint, and `git diff --check`, passed. - Fork CI for `537e296384`: [Python](https://github.com/Dricmoy/arrow/actions/runs/38923806919), [Dev](https://github.com/Dricmoy/arrow/actions/runs/38923806888), and [Docs](https://github.com/Dricmoy/arrow/actions/runs/38923806885) passed. The [C++ workflow](https://github.com/Dricmoy/arrow/actions/runs/38852310915) passed for the preceding commit; the follow-up commit changes only Python tests. ### Are there any user-facing changes? KMS client factories receive Arrow's `"DEFAULT"` markers instead of empty strings for omitted instance ID and URL values. Explicit non-empty values are preserved. These markers are interpreted by the KMS implementation; they are not automatically resolved server addresses. I reviewed the compatibility tradeoff: custom factories that special-case empty strings may need updating. Defaulting the instance ID can also change client reuse because the cache is indexed by access token and instance ID. Requests with the same token and resolved ID can now share a client where empty and `"DEFAULT"` IDs previously selected separate entries. This makes client creation consistent with Arrow's existing defaulting logic and stored key metadata, but changes behavior for integrations that rely on empty strings. There are no API signature changes. Closes #51351. Follow-up to #53949. ### Was AI used for this PR? AI assisted with the implementation, tests, validation, and this description. I reviewed the configuration flow, the default-value test expectations, the reader/writer cache separation, and the compatibility tradeoff described above. The PR overall is fairly small consistent expectation change, had some tests changed to fit the updated expectations. **PR code and description written by:** - [x] Human - [x] AI **Reviewed before submission by:** - [x] Human - [x] AI - [ ] Not reviewed * GitHub Issue: #51351 Authored-by: Dricmoy <dricbhattacharjee@gmail.com> Signed-off-by: Adam Reeve <adreeve@gmail.com> |
||
|
nishad shabbir
|
61f24ce27b |
GH-52024: [Python] Fix the codec list in show_info() (#51753)
### Rationale for this change
`pa.show_info()` prints its Compression Codecs section from a hardcoded list:
```python
codecs = ["brotli", "bz2", "gzip", "lz4_frame", "lz4", "snappy", "zstd"]
```
That list has drifted from `_ensure_compression()` in `io.pxi`, which is what
actually decides which codec names work. Two problems follow:
- `lz4` and `lz4_frame` both map to `CCompressionType_LZ4_FRAME`, so the same
codec is reported twice under two names.
- `lz4_raw` maps to `CCompressionType_LZ4` and is a separate codec, but it is
not listed at all, even though `pa.Codec.is_available('lz4_raw')` is True.
So seven lines are printed for six codecs, and the codec that is missing is a
real one.
Before:
```
Compression Codecs:
brotli : Enabled
bz2 : Enabled
gzip : Enabled
lz4_frame : Enabled
lz4 : Enabled
snappy : Enabled
zstd : Enabled
```
After:
```
Compression Codecs:
brotli : Enabled
bz2 : Enabled
gzip : Enabled
lz4 : Enabled
lz4_raw : Enabled
snappy : Enabled
zstd : Enabled
```
### What changes are included in this PR?
One list in `python/pyarrow/__init__.py`: drop the duplicate `lz4_frame` and add
the missing `lz4_raw`. `lz4` is kept as the name for the frame codec since that
is the one the docstrings lead with (`'lz4' (or 'lz4_frame')`).
On deriving this programmatically rather than hardcoding it — agreed, that is
the better fix. The canonical mapping lives in `_ensure_compression()` in
`io.pxi`, and neither `Codec` nor the C++ `compression.h` currently exposes the
set of supported codecs, so it needs a new accessor on the Cython side for
`show_info()` to read. I am happy to do that here, with the caveat that I cannot
compile the Cython extension in my environment and would be relying on CI to
verify it. If you would prefer, I can keep this change minimal and open a
follow-up issue for the programmatic version. Let me know which you would rather
have.
### Are these changes tested?
Not by a new test. `show_info()` has no existing test coverage and its output is
a diagnostic print, so I did not want to pin the exact text without knowing
whether you would want that — happy to add one if you do.
Verified by applying the same change to an installed pyarrow and reading the
output back, and by walking every name `_ensure_compression()` accepts to
confirm which of them map onto the same C++ enum value:
```
bz2 -> BZ2, gzip -> GZIP, brotli -> BROTLI,
lz4 -> LZ4_FRAME, lz4_frame -> LZ4_FRAME, # same codec, two names
lz4_raw -> LZ4, # distinct, was not listed
snappy -> SNAPPY, zstd -> ZSTD
```
### Are there any user-facing changes?
Only the output of `pa.show_info()`, which now lists each supported codec once
and includes `lz4_raw`. No API change.
### Was AI used for this PR?
In accordance to the [AI generation guidelines](https://arrow.apache.org/docs/dev/developers/overview.html#ai-generated-code), please disclose below whether and how AI was used in this PR.
**PR code and description written by:**
- [x] Human
- [ ] AI
**Reviewed before submission by:**
- [x] Human
- [ ] AI
- [ ] Not reviewed
* GitHub Issue: #52024
Authored-by: nishad shabbir <nishadshabbir28@gmail.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
|
||
|
Antoine Pitrou
|
c74c9576f7 |
GH-51757: [C++] Limit max JSON nesting depth in JSON parser (#51758)
### Rationale for this change Our JSON parser is inherently recursive, and a crafted JSON document with huge nesting can send it into a stack overflow. Given that real-world JSON files will not have thousands levels of nesting, we limit the JSON nesting to an arbitrary value of 300. ### Are these changes tested? Yes, by new tests. ### Are there any user-facing changes? **This PR contains a "Critical Fix".** A crafted JSON document with a large nesting depth could trigger a denial of service through stack overflow. ### Was AI used for this PR? **PR code and description written by:** - [x] Human - [ ] AI **Reviewed before submission by:** - [x] Human (self-reviewed) - [ ] AI - [ ] Not reviewed * GitHub Issue: #51757 Authored-by: Antoine Pitrou <antoine@python.org> Signed-off-by: Antoine Pitrou <antoine@python.org> |
||
|
Aaditya Srinivasan
|
bbbc06f8a8 |
GH-51037: [C++] Replace RapidJSON with simdjson in JSON parser (#51038)
### Rationale for this change This PR continues the simdjson migration by replacing the RapidJSON-based parsing implementation used by the JSON parser. The existing parser uses RapidJSON's SAX/handler interface to parse JSON values and populate Arrow builders. This change replaces that implementation with simdjson's ondemand API while retaining the existing builder and type-inference logic. ### Changes * Replace the RapidJSON parser and handler interface with simdjson's ondemand API. * Parse JSON documents using `simdjson::ondemand::parser::iterate_many`. * Use `ResolveSimdjsonResult()` consistently when handling simdjson results. * Preserve support for nested objects and arrays. * Preserve explicit-schema and inferred-field behavior. * Preserve unexpected-field handling for `Error`, `Ignore`, and `InferType`. * Continue storing numeric values as raw JSON tokens. * Trim trailing whitespace from numeric raw tokens to preserve existing behavior. * Preserve JSON parse error propagation through `Status::Invalid`. * Remove the parser's RapidJSON-specific dependencies. ### Are there any user-facing changes? No ### Was AI used for this PR? In accordance to the [AI generation guidelines](https://arrow.apache.org/docs/dev/developers/overview.html#ai-generated-code), please disclose below whether and how AI was used in this PR. **PR code and description written by:** - [X] Human - [X] AI **Reviewed before submission by:** - [X] Human - [X] AI - [ ] Not reviewed Fixes: #51037 * GitHub Issue: #51037 Authored-by: Aaditya Srinivasan <aadityasri03@gmail.com> Signed-off-by: Antoine Pitrou <antoine@python.org> |
||
|
|
1afe206b14 |
GH-51301: Fix deprecation warnings in the tests with numpy/pandas nightly (#51404)
### Rationale for this change Test warnings accumulated in our CI. ### What changes are included in this PR? The warnings are fixed or filtered if the deprecated functionality is still being tested. ### Are these changes tested? Yes. ### Are there any user-facing changes? No. Fixing test warnings in our CI. ### Was AI used for this PR? In accordance to the [AI generation guidelines](https://arrow.apache.org/docs/dev/developers/overview.html#ai-generated-code), please disclose below whether and how AI was used in this PR. **PR code and description written by:** - [x] Human - [ ] AI **Reviewed before submission by:** - [x] Human - [ ] AI - [ ] Not reviewed * GitHub Issue: #51301 Lead-authored-by: AlenkaF <frim.alenka@gmail.com> Co-authored-by: Alenka Frim <AlenkaF@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: Rok Mihevc <rok@mihevc.org> Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com> Signed-off-by: AlenkaF <frim.alenka@gmail.com> |
||
|
fenfeng9
|
6b1ad60ab5 |
GH-50658: [C++][Dev] Add RunEndEncoded support to gdb_arrow.py (#50768)
### Rationale for this change The Arrow GDB pretty-printers do not support run-end encoded data types. ### What changes are included in this PR? Add GDB pretty-printer support and tests for `run-end encoded` types, scalars, and arrays. ### Are these changes tested? Yes. ### Are there any user-facing changes? No. * GitHub Issue: #50658 Authored-by: fenfeng9 <fenfeng9@qq.com> Signed-off-by: Antoine Pitrou <antoine@python.org> |
||
|
|
5183e21386 |
GH-46901: [C++][Compute] Add remainder and modulo kernels (#48914)
### Rationale for this change Arrow is currently missing remainder and modulo kernels. ### What changes are included in this PR? Add the kernels `remainder`, `remainder_checked`, `modulo`, and `modulo_checked`, following the terminology in the [divmod proposal](https://lists.apache.org/thread/h3t6nz1ys2k2hnbrjvwyoxkf70cps8sh): * `remainder` uses truncated (C/C++) semantics — the result has the sign of the dividend, e.g. `remainder(-7, 3) == -1`. * `modulo` uses floored (Python/R) semantics — the result has the sign of the divisor, e.g. `modulo(-7, 3) == 2`. Both are supported for integer, floating-point and decimal inputs, and are exposed as `compute::Remainder` / `compute::Modulo` in `api_scalar.h`. Decimal arguments are promoted to a common scale like `add`, but the result type is resolved by a dedicated resolver rather than reusing `ResolveDecimalAdditionOrSubtractionOutput`: a remainder is always smaller in magnitude than the divisor, so no extra digit is needed for a carry, and the result is `precision = max(p1, p2)`, `scale = s1`. This keeps maximum-precision inputs (`decimal128(38, 0)`, `decimal256(76, 0)`) from overflowing the decimal precision range. Left as follow-ups: combined divmod #27909, floor division #39386, and duration support. ### Are these changes tested? Yes. New tests cover truncated vs. floored semantics across signed/unsigned integer, floating-point and decimal types, division by zero, `INT_MIN % -1` overflow, and decimal result-type resolution (same type in/out, mixed precisions and scales, Decimal128/Decimal256 promotion, decimal/integer promotion, and maximum-precision decimals). ### Are there any user-facing changes? Yes — four new compute functions (`remainder`, `remainder_checked`, `modulo`, `modulo_checked`), the corresponding `compute::Remainder` / `compute::Modulo` C++ APIs, and a new `BasicDecimal256 operator%`. These are additions only; no existing API is changed or removed. * GitHub Issue: #46901 * xref: https://github.com/apache/arrow/pull/46902, * xref: https://github.com/apache/arrow/pull/28497 Lead-authored-by: Fangchen Li <fangchen.li@outlook.com> Co-authored-by: Rok Mihevc <54589+rok@users.noreply.github.com> Co-authored-by: tadeja <864005+tadeja@users.noreply.github.com> Signed-off-by: Rok Mihevc <rok@mihevc.org> |
||
|
|
17be9c7caa |
GH-51652: [Python] Add NumPy StringDType to Arrow conversion (#51157)
### Rationale for this change NumPy 2.0 added `StringDType`, a variable-width UTF-8 string dtype. `pa.array` currently rejects it with `Unsupported numpy type 2056`. Now that pyarrow requires NumPy 2.0 the conversion can use the public `NpyString_*` C API. This supersedes #50951. The first commit is @ alippai's first commit from that PR, unchanged. His later commits added batching and new bulk builder APIs in Arrow C++; those can come later in their own PR with benchmarks, as discussed there. Nothing outside the NumPy conversion code changes here. ### What changes are included in this PR? - `NumPyDtypeToArrow` maps `StringDType` to `string`, so it is also the inferred type for `pa.array`, `pa.infer_type`, `pa.from_numpy_dtype` and lists of such arrays. `large_string` and `string_view` can be requested with `type`. Any other type raises a `TypeError` naming the requested type. - The conversion holds the dtype's allocator lock for the whole array and never holds the GIL while taking it. - A null entry in a StringDType array becomes an Arrow null unless the dtype's `na_object` is a `str`. NumPy reports such entries as that string from `__getitem__`, in ufuncs and comparisons, and in casts, and there is no way to tell them apart from a regular entry holding the same string, so the conversion writes the string too. NaN-like sentinels such as `np.nan` and `pd.NA`, `None`, and arbitrary objects all become arrow nulls. - A short section in `docs/source/python/numpy.rst` describing the support and the semantics for converting missing data. If left unfixed, https://github.com/apache/arrow/issues/51156 also becomes possible to trigger from Python by creating a StringDType array with a very large entry. ### Are these changes tested? Yes. Tests cover the three target types, strings crossing NumPy's short, medium and long storage thresholds, non-ASCII input, strided input, `mask`, `None`, NaN and string sentinels, the rejected target type, and list inference. I also ran a threaded test that converts an array while other threads hit NumPy's null-comparison error path on the same array to check the lock ordering. ### Are there any user-facing changes? Yes. `pa.array` and the inference functions accept `StringDType` arrays. AI disclosure: I used an AI model to work on the follow-up commits on top of @ alippai's first commit and for code review. * GitHub Issue: #42018 * GitHub Issue: #51652 Lead-authored-by: Nathan Goldbaum <nathan.goldbaum@gmail.com> Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com> Co-authored-by: Adam Lippai <adam@rigo.sk> Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com> |
||
|
Joris Van den Bossche
|
6f161d6525 |
GH-51302: [Python] Avoid deprecated .values call in pandas->pyarrow conversion (#51484)
### Rationale for this change Pandas has deprecated certain aspects of `.values` (when it looses information), which our pandas->pyarrow conversion currently relies upon. So updating that code path to avoid this usage. ### What changes are included in this PR? Removed usage of `.values` for the deprecated cases (period, interval, datetimetz). In the custom `get_values`, we already avoided this for period and interval, added a special case for converting datetimetz to a numpy array. And ensured that other cases were we did a direct `.values` goes through this `get_values` helper. ### Are these changes tested? Yes, I added more tests first to ensure full coverage of all cases that might run into the deprecation (e.g. the `Schema.from_pandas(df)` or `pa.array(EA)` with period/datetimetz dtypes were not currently covered) ### Are there any user-facing changes? No ### Was AI used for this PR? No * GitHub Issue: #51302 Authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com> Signed-off-by: AlenkaF <frim.alenka@gmail.com> |
||
|
Alexandros Anastasiou
|
65b7d24736 |
GH-49826: [Python] Return NotImplemented from Scalar/Array arithmetic dunders for unsupported types (#49845)
### Rationale for this change In pyarrow 24.0.0, `Scalar` and `Array` gained arithmetic dunder methods (#32007) that unconditionally dispatch to `pyarrow.compute.call_function`. When the other operand is an unrecognized type, `_pack_compute_args` raises `TypeError` instead of returning `NotImplemented`. This prevents Python from falling back to the other operand's reflected methods (`__radd__`, `__rsub__`, etc.), breaking downstream libraries that relied on this protocol. ### What changes are included in this PR? - Adds a `_compute_binary_op` helper in `scalar.pxi` that wraps `call_function` in a `try/except TypeError` and returns `NotImplemented` on failure. This follows the same pattern already used by `Scalar.__eq__` and `Array.__eq__`. - Updates all binary arithmetic and bitwise dunders on both `Scalar` (`scalar.pxi`) and `Array` (`array.pxi`) to use this helper. - Adds parametrized tests in `test_scalars.py` and `test_array.py` verifying that reflected operators on custom types are correctly invoked. ### Are these changes tested? Yes. New parametrized tests (`test_dunders_return_notimplemented_for_unknown_types`) cover all 10 binary operators for both `Scalar` and `Array`. The bug was also manually reproduced against pyarrow 24.0.0 to confirm the tests exercise the right code path. ### Are there any user-facing changes? Yes. `Scalar` and `Array` arithmetic dunders now return `NotImplemented` instead of raising `TypeError` when the other operand is not a recognized Arrow/NumPy type. This restores the pre-24.0.0 behavior where Python would fall back to the other operand's reflected method. ### AI-generated code disclosure This PR was developed with assistance from an AI coding tool (Claude, Anthropic). All changes have been reviewed, understood, and verified. * GitHub Issue: #49826 Closes #49826 Lead-authored-by: AlenkaF <frim.alenka@gmail.com> Co-authored-by: Alexandros Anastasiou <anastasioyaa@gmail.com> Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com> |
||
|
tadeja
|
dc90604891 |
GH-51220: [Python][Parquet] Accept write_table options in dataset writer (#51461)
### Rationale for this change Fix #51220 - Parquet `write_table` options are not all accepted in dataset `ParquetFileWriteOptions` so `pyarrow.parquet.write_to_dataset( ...)` raises `TypeError: unexpected parquet write option:` for `bloom_filter_options` , `write_time_adjusted_to_utc` or `store_schema` (Similar to past #37469) ### What changes are included in this PR? Add `bloom_filter_options`, `store_schema`, `write_time_adjusted_to_utc` to `ParquetFileWriteOptions`, also explicitly expose `bloom_filter_options` in `pyarrow.parquet.ParquetWriter`. Test writer options in `make_write_options`, and test three newly supported options in `write_to_dataset` ### Are these changes tested? Yes. CI also passes. ### Are there any user-facing changes? Yes, users may now use `bloom_filter_options` , `write_time_adjusted_to_utc` and `store_schema` in `pyarrow.parquet.write_to_dataset(...)` and in `ParquetFileFormat.make_write_options(...)` without raising TypeError. ### Was AI used for this PR? **PR code and description written by:** - [x] Human - [ ] AI **Reviewed before submission by:** - [x] Human - [x] AI - [ ] Not reviewed * GitHub Issue: #51220 Authored-by: Tadeja Kadunc <tadeja.kadunc@gmail.com> Signed-off-by: Rok Mihevc <rok@mihevc.org> |
||
|
Jonas Dedden
|
c75bc9aa9f |
GH-50988: [C++][Python] Substrait: add mappings for starts_with, ends_with and match_substring (#50989)
### Rationale for this change
`starts_with`, `ends_with` and `match_substring` have no Substrait mapping, so any expression using them fails to serialize:
```python
import pyarrow as pa
import pyarrow.compute as pc
from pyarrow.substrait import serialize_expressions
schema = pa.schema([pa.field("cat", pa.string())])
serialize_expressions([pc.starts_with(pc.field("cat"), "al")], ["f"], schema)
# ArrowNotImplementedError: No conversion function exists to convert the
# Arrow function starts_with to a Substrait call
```
Comparisons, `isin` and arithmetic serialize fine, so this is a per-function gap. It matters for engines that ingest a PyArrow filter through Substrait, where an unmappable function becomes a hard failure rather than a fallback.
See #50988.
### What changes are included in this PR?
Map the three kernels onto `starts_with`, `ends_with` and `contains` from Substrait's `functions_string.yaml`, in both directions.
The signatures do not line up. Substrait passes the pattern as a second argument, while the Arrow kernels are unary and carry it in `MatchSubstringOptions`. So:
* encoding lifts `MatchSubstringOptions::pattern` out into a literal argument
* decoding requires that argument to be a non-null string literal, and returns `NotImplemented` otherwise
Substrait's `case_sensitivity` option maps onto `ignore_case`. `CASE_INSENSITIVE_ASCII` has no Arrow equivalent and returns `NotImplemented`.
Both caveats are documented in `docs/source/cpp/acero/substrait.rst`.
### Are these changes tested?
Yes.
`Substrait.StringMatchExpressionSerialization` round-trips all three functions with and without `ignore_case`. `Substrait.StringMatchExpressionDeserialization` deserializes hand-written Substrait JSON to cover the default (no option) case and the two rejection paths. `test_serializing_string_match_expressions` round-trips each function through `pyarrow.substrait`.
### Are there any user-facing changes?
Yes. `pc.starts_with`, `pc.ends_with` and `pc.match_substring` can now be serialized to Substrait and consumed back, and Acero can consume plans using Substrait's `starts_with`, `ends_with` and `contains`. No existing behaviour changes: these previously raised.
* GitHub Issue: #50988
Authored-by: Jonas Dedden <mail@jonas-dedden.de>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
|
||
|
Yusef Syed
|
7a0491f734 |
GH-51145: [Python] Accept Arrow integer scalars in slice methods (#51160)
### Rationale for this change `Array.slice`, `ChunkedArray.slice`, `RecordBatch.slice`, and `Table.slice` accept Python and NumPy integer-like values but reject Arrow integer scalars, even though those scalars implement Python's integer-index protocol. This makes values produced by Arrow APIs unnecessarily unusable as slice offsets and lengths. ### What changes are included in this PR? Normalize non-`None` offsets and lengths with `operator.index()` at the four public Python wrapper boundaries before the existing validation and C++ calls. Add regression coverage for all signed and unsigned Arrow integer scalar types across the four wrappers, plus NumPy compatibility, invalid and null scalars, negative values, offset clamping, and int64 overflow behavior. ### Are these changes tested? Yes. Current Arrow C++ and editable PyArrow were built from source in an isolated Python 3.13 environment. The focused new tests and existing slice controls passed: 48 passed. Cython translation, Python test-file compilation, Flake8, and `git diff --check` also passed. ### Are there any user-facing changes? Yes. The four Python `.slice()` methods now accept non-null Arrow integer scalars for offsets and lengths. Existing Python/NumPy integer behavior and error behavior for unsupported values are preserved. ### AI assistance AI assisted analysis, implementation, test drafting, build work, and review. The account holder reviewed and approved the final four-file diff. The account holder did not personally run the commands reported above. * GitHub Issue: #51145 Authored-by: YusefSyed <211442445+YusefSyed@users.noreply.github.com> Signed-off-by: AlenkaF <frim.alenka@gmail.com> |
||
|
Adam Reeve
|
ac006a4754 |
GH-51446: [CI][Python] Fix test collection errors when Parquet encryption is unavailable (#51457)
### Rationale for this change Fixes test collection failing in nightly builds that don't have Parquet encryption enabled. ### What changes are included in this PR? Don't reference `InMemoryKmsClient` in function signatures, it's not defined if `pyarrow.parquet.encryption` can't be imported. ### Are these changes tested? Yes, I reproduced the crash collecting tests without encryption enabled and verified this is fixed, and also verified the tests still work with encryption enabled. ### Are there any user-facing changes? No ### Was AI used for this PR? In accordance to the [AI generation guidelines](https://arrow.apache.org/docs/dev/developers/overview.html#ai-generated-code), please disclose below whether and how AI was used in this PR. **PR code and description written by:** - [x] Human - [ ] AI **Reviewed before submission by:** - [x] Human - [ ] AI - [ ] Not reviewed * GitHub Issue: #51446 Authored-by: Adam Reeve <adreeve@gmail.com> Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com> |
||
|
Alexander Taepper
|
0fea33b5d2 |
GH-51300: Fix deprecation warnings for null_placement (#51307)
### Rationale for this change Resolves #51300, which mentioned that deprecation warnings were being printed when executing the python tests. ### What changes are included in this PR? This migrates the test suite to use the non-deprecated api for `null_placement` ### Are these changes tested? Yes, python compute tests are still passing ### Are there any user-facing changes? No. Authored-by: Alexander Taepper <alexander.taepper@gmail.com> Signed-off-by: AlenkaF <frim.alenka@gmail.com> |
||
|
Adam Reeve
|
911a7cbf1f |
GH-51349: [C++][Parquet] Add flag to control pass-through of KMS URLs from key material (#51350)
### Rationale for this change Addresses #51349 ### What changes are included in this PR? * Adds a new `read_kms_url` member to `DecryptionConfiguration` and a parameter with the same name to `rotate_master_keys`. Both are false by default. * Adds these to the PyArrow bindings. ### Are these changes tested? Yes, this includes new unit tests. ### Are there any user-facing changes? Yes, this adds a new user-facing option. Users that relied on this behaviour previously will now need to opt-in and enable the flag. * GitHub Issue: #51349 Authored-by: Adam Reeve <adreeve@gmail.com> Signed-off-by: Adam Reeve <adreeve@gmail.com> |
||
|
|
f3b46fb7d4 |
GH-49677 [Python][C++][Compute] Add search sorted compute kernel (#49679)
### Rationale for this change Add the implemenation of the search sorted compute kernel based on the numpy function: https://numpy.org/doc/stable/reference/generated/numpy.searchsorted.html ### What changes are included in this PR? Implementation of the C++ kernel + Python API. Tests in C++ and Python ### Are these changes tested? Yes ### Are there any user-facing changes? No breaking change * GitHub Issue: #49677 Lead-authored-by: Alexis Placet <2400067+Alex-PLACET@users.noreply.github.com> Co-authored-by: Alexis Placet <alexis.placet.dev@pm.me> Co-authored-by: Antoine Pitrou <antoine@python.org> Co-authored-by: Copilot <copilot@github.com> Signed-off-by: Antoine Pitrou <antoine@python.org> |
||
|
Stefan Wang
|
c400f72eb6 |
GH-51229: [Python] Raise instead of crashing on unopened resize (#51246)
### Rationale for this change Calling resize on a memory-mapped file that has never been opened takes down the Python process with a SIGSEGV, so an application cannot catch or recover from it. ### What changes are included in this PR? The resize path now applies the same open-state check the other file operations already use, before it reaches the native resize call. A directly constructed object raises ValueError. ### Are these changes tested? The regression runs in-process and checks ValueError and its message with pytest.raises. It does not spawn a subprocess. Separate, 20-second-bounded processes reproduced the original crash and the exception after the fix. Both used native Arrow 26.0.0-SNAPSHOT; the reused patched extension has byte-identical resize source to this PR. <details> <summary>Raw logs</summary> ```console $ python -c 'import pyarrow as pa; pa.MemoryMappedFile().resize(0)' Before: returncode=-11 After: ValueError: I/O operation on closed file returncode=1 $ cd python $ python -m pytest pyarrow/tests/test_io.py -k memory_map_resize -q .. [100%] 2 passed, 161 deselected, 1 warning in 1.02s ``` </details> ### Are there any user-facing changes? Yes. Misuse of a directly constructed memory-mapped file now raises a Python exception instead of terminating the process. **This PR contains a "Critical Fix".** It fixes a process crash reachable from ordinary Python-level object state. * GitHub Issue: #51229 Lead-authored-by: 1fanwang <1fannnw@gmail.com> Co-authored-by: Stefan Wang <1fannnw@gmail.com> Signed-off-by: AlenkaF <frim.alenka@gmail.com> |
||
|
Enrico Minack
|
70f5c269ae |
GH-51361: [C++][Parquet] Derive records_to_read in FileReaderImpl::ReadColumn from RowGroup (#51362)
### Rationale for this change Fixes #51361. ### What changes are included in this PR? Method `FileReaderImpl::ReadColumn` should derive `records_to_read` from the RowGroup rather than ColumnChunk's `num_values`. Deriving the right ColumnChunk index in `FileReaderImpl::DecodeRowGroups` is not trivial for nested schemas. This simplifies `FileReaderImpl::ReadColumn` and fixes #51361. This was silently masked for full-schema reads and for columns with identical num_values(), but surfaces as a hard failure when an earlier, unselected column requires decryption: reading only a trailing plaintext column of a partially column-key-encrypted, plaintext-footer Parquet file threw "Cannot decrypt ColumnMetadata" even though the requested column was never encrypted. This never corrupts data on unencrypted files: ReadColumn's wrong index is only ever used to look up ColumnChunk(i)->num_values(), a count fed into the *already-correct* reader as an upper bound on how many records to decode. Every row contributes at least one definition/repetition-level entry, so num_values() for any column is always >= that row group's true row count, and every column in a row group shares the same row count. ### Are these changes tested? Yes, in the context of reading a plaintext column of a partially encrypted Parquet file. This cannot be tested with non-encrypted files. ### Are there any user-facing changes? No. ### Was AI used for this PR? In accordance to the [AI generation guidelines](https://arrow.apache.org/docs/dev/developers/overview.html#ai-generated-code), please disclose below whether and how AI was used in this PR. **PR code and description written by:** - [X] Human - [X] AI **Reviewed before submission by:** - [X] Human - [ ] AI - [ ] Not reviewed * GitHub Issue: #51361 Lead-authored-by: Enrico Minack <enrico.minack@insightsoftmax.com> Co-authored-by: Enrico Minack <github@enrico.minack.dev> Signed-off-by: Adam Reeve <adreeve@gmail.com> |
||
|
Sylvain LesageandTadeja Kadunc
|
68610e4c2b |
GH-47499: [Python] Explicitly add use_content_defined_chunking to ParquetWriter and write_table (#47498)
### Rationale for this change The doc page https://arrow.apache.org/docs/python/generated/pyarrow.parquet.write_table.html does not show `use_content_defined_chunking` as an argument of `write_table`: <img width="1571" height="821" alt="Screenshot From 2025-09-04 14-45-32" src="https://github.com/user-attachments/assets/a9fb4d7f-9b4d-4ce6-92c8-97b781831f33" /> Note that it's documented below in the list of parameters: <img width="1800" height="1062" alt="Screenshot From 2025-09-04 14-46-20" src="https://github.com/user-attachments/assets/d9d6c289-97c3-476b-820d-eeb30f8f791f" /> ### What changes are included in this PR? Change the API signatures of `pyarrow.parquet.ParquetWriter` and `pyarrow.parquet.write_table`: `use_content_defined_chunking` parameter is explicitly added, with a default value (`False`), as mentioned by the comment: https://github.com/apache/arrow/blob/512dcd77dcf3910c41a06d2a1620c729d92d6748/python/pyarrow/parquet/core.py#L1953-L1954 Reorder the parameters in docstring `_parquet_writer_arg_docs` to match the method signature. Add a test. Also users won't get this error anymore `TypeError: unexpected parquet write option: use_content_defined_chunking` when running with released wheel (since 21.0.0) : ```shell python -c "import pyarrow.dataset as ds; ds.ParquetFileFormat().make_write_options(use_content_defined_chunking=True)" ``` or when running : ```python import pyarrow as pa, pyarrow.parquet as pq pq.write_to_dataset(pa.table({'a':[1, 2, 3]}), 'cdc_dataset', use_content_defined_chunking=True) ``` ### Are these changes tested? Yes ### Are there any user-facing changes? Yes: 1. `use_content_defined_chunking` parameter is explicitly added to the API signature and documentation of `pyarrow.parquet.ParquetWriter` and `pyarrow.parquet.write_table` 2. `use_content_defined_chunking` is now accepted in `pyarrow.dataset.ParquetFileFormat.make_write_options` and `pyarrow.parquet.write_to_dataset` instead of raising `TypeError` * GitHub Issue: #47499 Lead-authored-by: Sylvain Lesage <severo@rednegra.net> Co-authored-by: Tadeja Kadunc <tadeja.kadunc@gmail.com> Signed-off-by: Tadeja Kadunc <tadeja.kadunc@gmail.com> |
||
|
Aaditya SrinivasanandAntoine Pitrou
|
3cb299408d |
GH-50944: [C++] Replace RapidJSON with simdjson in JSON chunker (#50945)
### Rationale for this change
This PR continues the simdjson migration by replacing the RapidJSON-based JSON boundary detection used by the JSON chunker.
The existing implementation uses RapidJSON's streaming parser to identify complete JSON values. This change replaces that logic with the simdjson streaming parser.
### Are these changes tested?
Yes, by existing tests.
### Are there any user-facing changes?
No, except perhaps slight differences in error message and timing of error reporting when reading from an invalid JSON stream (i.e. there may be cases where a JSON parse error is reported earlier or later while iterating over RecordBatches).
* GitHub Issue: #50944
Lead-authored-by: Aaditya Srinivasan <aadityasri03@gmail.com>
Co-authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
|
||
|
singhpratech
|
79e074ace3 |
GH-51225: [C++] utf8_normalize: compose for NFC and NFKC (#51237)
### Rationale for this change `utf8_normalize` with `form=NFC` or `NFKC` returned the decomposed forms (NFD, NFKD). `Utf8NormalizeBase` builds the right `utf8proc` options for each form but only calls `utf8proc_decompose()`, which decomposes regardless of `UTF8PROC_COMPOSE`; the composition step lives in `utf8proc_normalize_utf32()`, which the kernel never called. See #51225. ### What changes are included in this PR? - After a successful `utf8proc_decompose()`, when the options include `UTF8PROC_COMPOSE`, call `utf8proc_normalize_utf32()` on the scratch buffer. It composes in place and returns the new code point count; the existing UTF-8 encode loop is unchanged. NFD and NFKD take the same path as before. - Fix the `json_composed` fixture in `scalar_string_test.cc`: its bytes were the decomposed form (`61 CC 81`), the same string as `json_decomposed`, so the compose assertions were comparing a value with itself and passed with the bug. Thanks to `@ Santoshkumarpuppala` for spotting that on the issue. - Add composed/decomposed pairs (U+00E9, and a Hangul syllable with its jamo) to `test_utf8_normalize` in pyarrow; the existing input, U+00B2, is its own NFC. ### Are these changes tested? Yes. With the corrected fixture, `TestStringKernels.Utf8Normalize` fails on the unpatched kernel and passes with this change; the pyarrow test covers the composed forms from Python. ### Are there any user-facing changes? Yes: `utf8_normalize` with `NFC` and `NFKC` now returns composed output. Callers that depended on the previous (decomposed) result for those forms will see different bytes. * GitHub Issue: #51225 Authored-by: singhpratech <singh.prateek86@outlook.com> Signed-off-by: Antoine Pitrou <antoine@python.org> |
||
|
Antoine ProuvostandAntoine Pitrou
|
39706ed061 |
GH-39295: [C++][Python] ConsumingDLPack on Arrays and Tensor (#51122)
### Rationale for this change There are already utilities to import tensor data from NumPy. With DLPack standard, it will work across many tensor providers. ### What changes are included in this PR? - Bind `FixedShapedTensorArray::FromTensor` in Python - Add `ImportArrayVersionedFromDLPack` and `ImportTensorVersionedFromDLPack` in C++ - Add in Python `Array.from_dlpack` and `Tensor.from_dlpack` Similar to the export, multidimensional tensor import require an explicit step through `Tensor`. - No `Array::FromTensor` or even `FixedSizeListArray::FromTensor`. IMHO it does not feel as necessary in this direction but open to anyone's take one it. ### Are these changes tested? Yes. ### Are there any user-facing changes? Yes, new APIs. * GitHub Issue: #39295 Lead-authored-by: AntoinePrv <AntoinePrv@users.noreply.github.com> Co-authored-by: Antoine Pitrou <pitrou@free.fr> Signed-off-by: Antoine Pitrou <antoine@python.org> |
||
|
Stefan Wang
|
8c7b08f426 |
GH-51041: [Python] Reject non-Buffer FunctionOptions.deserialize input (#51129)
### Rationale for this change
Calling `pyarrow.compute.FunctionOptions.deserialize()` with anything that is not a `Buffer` crashes the interpreter. `None`, an `int`, a `list` and a `bytes` object all segfault, so a caller who passes the wrong object, or who forwards an unvalidated value, loses the process rather than seeing an exception.
`deserialize()` hands its argument to `pyarrow_unwrap_buffer()`, which returns a null pointer for a non-`Buffer`. The `deref()` on the next line then dereferences null.
Before this change the call terminates the process. After it, the call raises `TypeError` naming the type it received.
### What changes are included in this PR?
The static method now declares its parameter as `Buffer buf not None`, which is what its docstring already says it takes. Cython rejects a wrong type before the unwrap runs, so the null pointer is never produced.
### Are these changes tested?
Yes, by a parametrized case over the four inputs above.
<details>
<summary>Red, with this commit's source file reverted to its parent</summary>
```console
$ git checkout HEAD~1 python/pyarrow/_compute.pyx
$ pip install -e python --no-build-isolation
$ python -X faulthandler -c "import pyarrow.compute as pc; pc.FunctionOptions.deserialize(None)"
Fatal Python error: Segmentation fault
Current thread 0x00000001f6fbe180 (most recent call first):
File "<string>", line 1 in <module>
Extension modules: numpy._core._multiarray_umath, numpy.linalg._umath_linalg, pyarrow.lib, pyarrow._compute (total: 4)
```
The same crash occurs for `1`, `[]` and `b''`.
</details>
<details>
<summary>Green, with the fix applied</summary>
```console
$ git checkout HEAD -- python/pyarrow/_compute.pyx
$ pip install -e python --no-build-isolation
$ python -m pytest pyarrow/tests/test_compute.py -k deserialize_rejects_non_buffers -v
collected 607 items / 603 deselected / 4 selected
pyarrow/tests/test_compute.py::test_function_options_deserialize_rejects_non_buffers[None] PASSED [ 25%]
pyarrow/tests/test_compute.py::test_function_options_deserialize_rejects_non_buffers[1] PASSED [ 50%]
pyarrow/tests/test_compute.py::test_function_options_deserialize_rejects_non_buffers[value2] PASSED [ 75%]
pyarrow/tests/test_compute.py::test_function_options_deserialize_rejects_non_buffers[] PASSED [100%]
====================== 4 passed, 603 deselected in 0.06s =======================
```
```console
$ for v in None 1 "[]" "b''"; do python -c "
import pyarrow.compute as pc
try:
pc.FunctionOptions.deserialize($v)
except TypeError as e:
print('$v', '->', e)
"; done
None -> Argument 'buf' has incorrect type (expected pyarrow.lib.Buffer, got NoneType)
1 -> Argument 'buf' has incorrect type (expected pyarrow.lib.Buffer, got int)
[] -> Argument 'buf' has incorrect type (expected pyarrow.lib.Buffer, got list)
b -> Argument 'buf' has incorrect type (expected pyarrow.lib.Buffer, got bytes)
```
</details>
A valid buffer still round-trips, and the rest of `test_compute.py` is unaffected.
<details>
<summary>Round-trip and full suite</summary>
```console
$ python -c "import pyarrow.compute as pc; o = pc.ArraySortOptions(order='descending'); print(pc.FunctionOptions.deserialize(o.serialize()) == o)"
True
$ python -m pytest pyarrow/tests/test_compute.py -q
584 passed, 23 skipped, 11 warnings in 3.09s
```
</details>
### Are there any user-facing changes?
Passing a non-`Buffer` to `FunctionOptions.deserialize()` now raises `TypeError` instead of terminating the process. Callers already passing a `Buffer`, which is what the docstring documents, are unaffected.
* GitHub Issue: #51041
Authored-by: 1fanwang <1fannnw@gmail.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
|
||
|
Antoine Pitrou
|
df40dd400b |
GH-51238: [C++][Python][Parquet] Limit schema nesting depth when reading (#51239)
### Rationale for this change Reconstructing a nested Schema from the Parquet Thrift metadata implies a recursive call that can blow up the stack on pathologically-nested schemas (with thousands of nesting levels or more). By adding a limit on the schema nesting depth, we turn a stack overflow-induced crash into a regular Parquet error. ### Are these changes tested? By additional unit tests; also privately with a proof-of-concept reproducer that induces a stack overflow exhaustion. ### Are there any user-facing changes? In the unlikely case where a legitimate Parquet file has a deeper schema than the default schema nesting limit in this PR (100), an error will be raised when reading where it used to succeed. The user can bump the limit to circumvent the error. **This PR contains a "Critical Fix".** It fixes a crash on a deeply nested Parquet schema that would provoke a stack overflow. It is not an exploitable vulnerability except through denial of service. Thanks to "1K0CT" for the initial report. * GitHub Issue: #51238 Authored-by: Antoine Pitrou <antoine@python.org> Signed-off-by: Antoine Pitrou <antoine@python.org> |
||
|
dawgandAlenkaF
|
05c1a4bf54 |
GH-34577 [Python] Expose eol and null_string csv WriteOptions (#46976)
### Rationale for this change Expose remaining csv write options to pyarrow. ### What changes are included in this PR? Adding `eol` and `null_string` to csv.WriteOptions. ### Are these changes tested? Yes, testing of setters included. ### Are there any user-facing changes? * GitHub Issue: #34577 Lead-authored-by: Martin Nowak <code@dawg.eu> Co-authored-by: AlenkaF <frim.alenka@gmail.com> Signed-off-by: AlenkaF <frim.alenka@gmail.com> |
||
|
carrerasdarren-cell
|
b3a9c44447 |
GH-51044: [Python] Reject read-only readinto destinations (#51126)
### Rationale for this change `BufferReader.readinto()` currently segfaults when passed a read-only destination such as `bytes` or a read-only `memoryview`. `py_buffer()` creates an immutable Arrow buffer, `mutable_data()` returns a null pointer, and the C++ read path attempts to copy into that pointer. A Python API misuse should raise a Python exception rather than terminate the interpreter. ### What changes are included in this PR? - Validate that a `NativeFile.readinto()` destination is mutable before obtaining its writable pointer. - Raise `TypeError` for immutable destinations. - Add regression coverage for `bytes` and read-only `memoryview` destinations. ### Are these changes tested? Yes. The focused `readinto` tests pass against a locally compiled patched `pyarrow.lib`. The stock 25.0.1 wheel exits with status 139 for the issue reproducer, while the patched build raises the expected `TypeError`; the writable `bytearray` control continues to read successfully. ### Are there any user-facing changes? Yes. Passing a read-only destination to `NativeFile.readinto()` now raises `TypeError` instead of terminating the interpreter. Writable-buffer behavior is unchanged. Fixes #51044. This contribution was developed with assistance from OpenAI Codex. I reviewed, tested, and take responsibility for the patch and this description. * GitHub Issue: #51044 Authored-by: Darren Carreras <283775510+carrerasdarren-cell@users.noreply.github.com> Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com> |
||
|
Antoine Pitrou
|
6b43215b09 |
GH-12594: [Python][GPU] Remove inherently crashy CUDA test (#51206)
### What changes are included in this PR? Remove a CUDA test that tries to access an invalid pointer (after the owning resource was deallocated). On old CUDA setups it used to raise an exception: ``` Cuda error 1 in function 'cuMemcpyDtoH': [CUDA_ERROR_INVALID_VALUE] invalid argument ``` but more recently it has simply started crashing. ### Are these changes tested? By existing CI tests. This should fix a crash in the CUDA Python CI. ### Are there any user-facing changes? No. * GitHub Issue: #12594 Authored-by: Antoine Pitrou <antoine@python.org> Signed-off-by: Antoine Pitrou <antoine@python.org> |
||
|
Adarsh Mishra
|
f95b0015a6 |
GH-51194: [C++] Fix cumulative_max/min default start for floating-point types (#51203)
## Rationale for this change `Identity<Max>` (the implicit default start of `cumulative_max`) used `std::numeric_limits<T>::min()`, which is the smallest **positive** value for floating-point types. Any input starting with a non-positive value therefore never replaced the start as the new maximum: ```python >>> import pyarrow as pa, pyarrow.compute as pc >>> pc.cumulative_max(pa.array([-2.5, 2.5])).to_pylist() [2.2250738585072014e-308, 2.5] # expected [-2.5, 2.5] ``` ## What changes are included in this PR? - `Identity<Max>` now uses `-infinity` for floating-point types (and half-float), and keeps `lowest()` for integer types. `-infinity` is the only value satisfying the documented identity property `Op(identity, x) = x for all x` — `lowest()` (most negative finite value) fails it for a leading `-inf` input. - `Identity<Min>` gets the mirror fix: `+infinity` for floating-point types. Previously `cumulative_min([inf, 1.0])` returned `[DBL_MAX, 1.0]` instead of `[inf, 1.0]`. Closes #51194. ## Are these changes tested? - Added `TestCumulative.NegativeValues` covering negative integers, negative floats, and `±Inf` inputs for both `cumulative_max` and `cumulative_min`. - Added a pyarrow regression test reproducing the issue (`test_cumulative_max_min_negative_default_start`). ## Are there any user-facing changes? Yes: `cumulative_max`/`cumulative_min` now return correct results for floating-point inputs whose first value is non-positive (for max) / non-negative or `inf` (for min). The documented behavior — 'the default start is the minimum/maximum value of input type' — is now actually honored. * GitHub Issue: #51194 Authored-by: Adarsh <adarsh@neuroad.ai> Signed-off-by: Antoine Pitrou <antoine@python.org> |
||
|
Antoine Prouvost
|
9808d818ce |
GH-38868: [Python] Add dlpack producer to FixedShapeTensorArray/Scalar (#51159)
### Rationale for this change Have the same coverage as `to_numpy`. ### What changes are included in this PR? - `FixedShapeTensorArray.__dlpack__` explicitly calling `to_tensor` - `FixedShapeTensorArray.__dlpack_device__` (defaulted) - `FixedShapeTensorScalar.__dlpack__` explicitly calling `to_tensor` - `FixedShapeTensorScalar.__dlpack_device__` calling C++ device extraction on the underlying array. If GH-51122 get merged first, I'll add the consuming side here, otherwise if this one gets merged first, I'll add it there. ### Are these changes tested? Yes a few since this does not introduce new logic. ### Are there any user-facing changes? Additions only. * GitHub Issue: #38868 Authored-by: AntoinePrv <AntoinePrv@users.noreply.github.com> Signed-off-by: Antoine Pitrou <antoine@python.org> |
||
|
Antoine Prouvost
|
987e2312c3 |
GH-38868: [C++][Python] Add Array::ToTensor and fixed size list support (#50929)
### Rationale for this change Enable multidimensional DLPack support for Array via `to_tensor`. ### What changes are included in this PR? - Add virtual `Array::ToTensor` - Add `NumericArray::ToTensor` for 1D arrays - Add `FixedSizeListArray::ToTensor` for multidimensional arrays - Add DLPack error suggestiong tensor convertion - Add DLPack tests with `arr.to_tensor().__dlpack__()` Note: Nulls are explicitly supported in `to_tensor` as unspecified data. This was the current behaviour. ### Are these changes tested? Yes ### Are there any user-facing changes? New public Array function. * GitHub Issue: #38868 Authored-by: AntoinePrv <AntoinePrv@users.noreply.github.com> Signed-off-by: Antoine Pitrou <antoine@python.org> |
||
|
Advit Arora
|
3c5c2279f7 |
GH-47369: [Python][Parquet] Fix wrong variable in the invalid operator error message (#51004)
### Rationale for this change
The invalid-operator error prints the column name instead of the operator:
```
>>> pq.filters_to_expression([[('a', '=!', 1)]])
ValueError: "a" is not a valid operator in predicates.
```
The message means `op`, the value that failed to match; `col` was already consumed by `ds.field(col)` above. The mixup came in
|
||
|
Stefan WangandAlenka Frim
|
f52076ca06 |
GH-48344: [Python] Fix Table.from_struct_array for empty ChunkedArray (#49869)
### Rationale for this change
Round-tripping an empty table through `to_struct_array()` and
`Table.from_struct_array()` raises
`ValueError: Must pass schema, or at least one RecordBatch`.
`to_struct_array()` returns a zero-chunk struct `ChunkedArray`, so
`from_struct_array()` has no batch from which to recover the schema.
For a zero-chunk input, `from_struct_array()` now calls `combine_chunks()` and
converts the resulting empty struct array into a `RecordBatch`. Nonempty inputs
keep the existing path. An empty non-struct `ChunkedArray` still raises the
expected `TypeError`.
### What changes are included in this PR?
The empty `ChunkedArray` path creates one empty batch before calling
`Table.from_batches`. The regression coverage checks the round-trip result,
schema preservation, and the invalid non-struct input.
### Are these changes tested?
I ran this script against PyArrow 25.0.1 and the compiled branch:
```bash
python - <<'PY'
import pyarrow as pa
value = pa.chunked_array(
[], type=pa.struct([("ints", pa.int32()), ("floats", pa.float32())])
)
print(f"pyarrow={pa.__version__}, chunks={value.num_chunks}")
try:
result = pa.Table.from_struct_array(value)
print(f"rows={result.num_rows}, schema={result.schema}")
except Exception as error:
print(f"{type(error).__name__}: {error}")
PY
```
Before:
```text
pyarrow=25.0.1, chunks=0
ValueError: Must pass schema, or at least one RecordBatch
```
After:
```text
pyarrow=26.0.0.dev175+ge4ad179ae, chunks=0
rows=0, schema=ints: int32
floats: float
```
The focused regression suite also runs against the compiled branch:
```text
$ python -m pytest python/pyarrow/tests/test_table.py -k from_struct_array -q
....... [100%]
7 passed, 208 deselected in 0.51s
```
### Are there any user-facing changes?
`Table.from_struct_array()` now returns the expected empty table for a
zero-chunk struct `ChunkedArray` instead of raising `ValueError`.
* GitHub Issue: https://github.com/apache/arrow/issues/48344
* GitHub Issue: #48344
Lead-authored-by: 1fanwang <1fannnw@gmail.com>
Co-authored-by: Alenka Frim <AlenkaF@users.noreply.github.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
|
||
|
nishad shabbir
|
3db7076bad |
GH-50847: [Python] Add Type_RUN_END_ENCODED to _NESTED_TYPES set (#50848)
### Rationale for this change `pyarrow.types.is_nested()` said a run-end encoded type wasn't nested, while the C++ `arrow::is_nested()` says it is: ```python >>> import pyarrow as pa >>> t = pa.run_end_encoded(pa.int32(), pa.string()) >>> t.num_fields 2 >>> pa.types.is_nested(t) False ``` The Python side reads from a hardcoded `_NESTED_TYPES` set in `python/pyarrow/types.py` rather than from the C++ trait, so it has to be kept in step by hand. Lining that set up against `is_nested()` in `cpp/src/arrow/type_traits.h`, run-end encoded was the only type the two still disagreed about — the list-view types and fixed-size list are already there. It's also out of step with the type itself. Run-end encoded has two children, the run ends and the values, and every other type in pyarrow that has children answers `True` here. This is the same thing that happened to fixed-size list in #40171, fixed by #40172, and the list-view types were added after that. This looks like the last one missed when run-end encoding went in. ### What changes are included in this PR? Adds `Type_RUN_END_ENCODED` to `_NESTED_TYPES`, which is the whole fix. In the test I've added the run-end encoded case, and while I was there also a map and a dictionary. Map was nested already but wasn't asserted anywhere, and dictionary is the interesting negative — it wraps a value type but is deliberately not nested in either implementation, so pinning it means a later change can't quietly sweep it in. ### Are these changes tested? Yes. `test_is_nested_or_struct` fails on the current code and passes with the change. I don't have a local C++ build, so I checked this by running the updated `test_types.py` against an installed pyarrow 25.0.0 with the same one-line change applied to its `types.py`. Before the change that file had 87 passing with `test_is_nested_or_struct` failing; after it, 88 passing. The two errors and one failure I see in both runs are environmental on my machine and unrelated — the errors are the `pickle_module` fixture, which comes from a conftest I wasn't loading, and the failure is `test_pytz_timezone_roundtrip`. ### Are there any user-facing changes? Yes, though it's small. `pa.types.is_nested()` now returns `True` for run-end encoded types where it previously returned `False`. Anything branching on that predicate will take the nested path for these types, which is the intended answer and what the C++ implementation has always given. Nothing inside pyarrow reads `is_nested` or `_NESTED_TYPES`, so the effect is limited to callers. * GitHub Issue: #50847 Authored-by: nishad shabbir <nishadshabbir28@gmail.com> Signed-off-by: AlenkaF <frim.alenka@gmail.com> |
||
|
|
b60d43701b |
GH-48473: [CI][Python] Require numpy 2.0 (#50769)
### Rationale for this change Require Numpy 2.0 to support newer features, allow newer numpy API/ABI ### What changes are included in this PR? - Numpy 2.0+ requirement - Pandas 2.2.2+ requirement (first wheel compatible with numpy 2.0) - Pyodide 0.27.1 usage (numpy 2.0 api) ### Are there any user-facing changes? Yes, in contrast to pyarrow 25, after merging this PR pyarrow 26 can't be used with Numpy 1.x or pandas 2.0-2.2.1 **This PR includes breaking changes to public APIs.** This PR was created using GPT-5.6-Sol-xhigh, every line read & reviewed by me. * GitHub Issue: #48473 Lead-authored-by: Adam Lippai <adam@rigo.sk> Co-authored-by: Ádám Lippai <adam@rigo.sk> Co-authored-by: Raúl Cumplido <raulcumplido@gmail.com> Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com> Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com> |
||
|
Antoine Prouvost
|
118892700b |
GH-41670: [C++][Python] Move to DLPack 1.3 (#50827)
### Rationale for this change Pure version bump to enable implementing more features. ### What changes are included in this PR? - Change the DLPack header file and definition. - Use `raw_data` for `Tensor` similar to `Array` (instead of `mutable_raw_data` that is not available on immutable tensors). - Strides may no longer be `NULL` - Add `ExportArrayVersioned` and `ExportTensorVersioned` C APIs - Support for `copy` - Support for `max_version` and `copy` in `__dlpack__` - Add `dl_device` to `__dlpack__` parameters as the spec requires but only CPU supported at the moment ### Are these changes tested? Yes, with existing tests. ### Are there any user-facing changes? * GitHub Issue: #41670 Authored-by: AntoinePrv <AntoinePrv@users.noreply.github.com> Signed-off-by: AlenkaF <frim.alenka@gmail.com> |
||
|
Parker Charles Cassar
|
892c73dc19 |
GH-50312: [Python] Fix UUID extension type round-trip to pandas returning bytes (#50325)
### Rationale for this change Converting a Table with an `arrow.uuid` extension column to pandas currently produces a column of `bytes` instead of `uuid.UUID` objects. Without a dedicated conversion path, `Table.to_pandas()` / `Array.to_pandas()` falls back to the storage type (`fixed_size_binary(16)`) and materializes raw bytes. Downstream callers (and tests such as those landing via pandas UUID Parquet coverage) expect `uuid.UUID` objects, matching what `to_pylist()` / `UuidScalar.as_py()` already return. Note: the original issue suggested this might be specific to Python 3.14, but the same `bytes` result reproduces on Python 3.10-3.14 with current PyArrow - `UuidType` never had a pandas conversion path. Closes #50312 ### What changes are included in this PR? Convert UUID extension arrays to `uuid.UUID` objects in the C++ pandas conversion path (integrated from @ rok's proposal in rok/arrow#55), rather than relying on a Python `to_pandas_dtype()` / `__from_arrow__` compat layer: - Construct `uuid.UUID` objects from storage bytes in `helpers.cc` (`UuidFromBytes`) and use that from `arrow_to_pandas.cc` - Reuse empty kwargs / shared empty tuple constants so conversion does not allocate per element - Keep `to_numpy(zero_copy_only=False)` on UUID extension arrays returning storage `bytes` (extension to NumPy should follow storage type) - Add/extend regression tests for pandas round-trip and NumPy behavior ### Are these changes tested? Yes. Acceptance criteria: - [x] Tests added for the UUID to pandas round-trip (and NumPy storage behavior) - [x] Relevant local / CI Python tests passing for the touched paths - [x] Follows project style / review feedback (lint fix included) - [x] No intentional breaking public API change beyond correcting incorrect `bytes` return values to `uuid.UUID` for `to_pandas()` Before / after evidence (backend; console): ```text # Before (repro on Python 3.10-3.14 with unfixed PyArrow) >>> type(result_df.loc[0, "id"]) <class 'bytes'> # After (with this PR) >>> type(result_df.loc[0, "id"]) <class 'uuid.UUID'> ``` Also covered by added unit tests in `python/pyarrow/tests/parquet/test_data_types.py` and `python/pyarrow/tests/test_extension_type.py`. ### Are there any user-facing changes? Yes. `Table.to_pandas()` / `Array.to_pandas()` now returns `uuid.UUID` for `arrow.uuid` columns instead of `bytes`. `to_numpy(zero_copy_only=False)` continues to expose storage `bytes`. * GitHub Issue: #50312 Authored-by: parker-cassar <parker.cassar@berkeley.edu> Signed-off-by: Rok Mihevc <rok@mihevc.org> |
||
|
Joris Van den Bossche
|
1b004f2cfd |
GH-50849: [Python] Return correct ParquetLogicalType.type for geometry/geography (#50850)
### Rationale for this change Fix the return value of `ParquetLogicalType.type` for geometry/geography types ### Are these changes tested? Yes ### Are there any user-facing changes? Not apart from the bug fix * GitHub Issue: #50849 Authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com> Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com> |
||
|
You-Cheng LinandYou-Cheng Lin
|
7128c9c262 |
GH-50801: [Python] Expose the record_batch_reader_source Acero node (RecordBatchReaderSourceNodeOptions) (#50802)
### Rationale for this change Closes #50801 As title, this exposes RecordBatchReaderSourceNodeOptions to Python. With it, a hash join can build its hash table once and stream the probe side through as a generator-backed RecordBatchReader, releasing each input chunk as it is consumed, instead of materializing the full probe table up front (table_source) or pinning all fragments for the plan's lifetime (dataset scan). ### Are these changes tested? Yes ### Are there any user-facing changes? Yeah, users can pass RecordBatchReaderSourceNodeOptions into Declaration, but its not a breaking change * GitHub Issue: #50801 Lead-authored-by: You-Cheng Lin <106612301+owenowenisme@users.noreply.github.com> Co-authored-by: You-Cheng Lin <mses010108@gmail.com> Signed-off-by: Rossi Sun <zanmato1984@gmail.com> |
||
|
tadejaandRok Mihevc
|
7ef5982635 |
GH-50808: [Python] Narrow Feather deprecation to V1 format (#50685)
### Rationale for this change Follow-up to https://github.com/apache/arrow/issues/49232#issuecomment-5105290467 / #49590 ### What changes are included in this PR? Remove warnings from `write_feather()`, `read_feather()`, `read_table()`, and `FeatherDataset`. Warn **only** on writing with `version=1` / reading Feather V1 file. Update docs to V1-only and keep IPC migration guide. Additionally corrected notes like `deprecated as of 24.0.0` to `25.0.0` instead. `DeprecationWarning` in place of current `FutureWarning`. ### Are these changes tested? Yes, by CI. ### Are there any user-facing changes? Yes! Feather V2 APIs no longer emit deprecation warnings. Reading/writing the legacy Feather V1 format emits `DeprecationWarning` in place of `FutureWarning`. * GitHub Issue: #50808 Lead-authored-by: Tadeja Kadunc <tadeja.kadunc@gmail.com> Co-authored-by: tadeja <tadeja@users.noreply.github.com> Co-authored-by: Rok Mihevc <rok@mihevc.org> Signed-off-by: Rok Mihevc <rok@mihevc.org> |
||
|
snigdha choppa
|
35c5ffd121 |
GH-50395: [C++] Support duration inputs in temporal rounding (#50675)
### Rationale for this change The `ceil_temporal`, `floor_temporal`, and `round_temporal` functions currently support date, time, and timestamp inputs, but not duration inputs. ### What changes are included in this PR? - Add kernel registration for duration values with second, millisecond, microsecond, and nanosecond resolutions - Support rounding duration inputs using physical units through day - Treat week as seven physical days for duration inputs - Reject ambiguous calendar units such as month, quarter, and year - Reject `calendar_based_origin` for duration inputs - Add focused C++ tests covering all four duration resolutions, positive and negative values, null propagation, day and week rounding, and unsupported calendar behavior ### Are these changes tested? Yes. - `arrow-compute-scalar-temporal-test`: 55 tests passed - Applicable pre-commit C++ formatting and lint checks passed ### Are there any user-facing changes? Yes. Users can now pass duration values to `ceil_temporal`, `floor_temporal`, and `round_temporal` for supported physical units. ### AI assistance I used ChatGPT to help navigate the codebase and draft the initial implementation. I reviewed, revised, and tested the changes locally. * GitHub Issue: #50395 Authored-by: snigdhachoppac <snigchoppa@gmail.com> Signed-off-by: Rok Mihevc <rok@mihevc.org> |
||
|
Raúl Cumplido
|
9ec70551b9 |
GH-50684: [Python][FlightRPC] Break the reference cycle between the C++ FlightServerBase and the Python object to avoid leaking server (#50687)
### Rationale for this change PyFlightServer keeps a reference towards the Python server via `OwnedRefNoGIL server_`, the Python server also keeps a reference of the C++ `PyFlightServer` creating a cycle that is never freed during the process lifetime. ### What changes are included in this PR? Create a new `ReleasePythonServerRef` method that is called after any `server.Shutdown` (including at `__exit__`) allowing for the `OwnedRefNoGIL` to be cleared breaking the cycle. This lets normal reference counting free the previously leaked Python object. ### Are these changes tested? Yes, the newly added tests were failing leaking the references before the fix. Currently a test demonstrating a leak when not calling server.Shutdown is added for discussion purposes. ### Are there any user-facing changes? No * GitHub Issue: #50684 Authored-by: Raúl Cumplido <raulcumplido@gmail.com> Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com> |
||
|
Guja
|
84027295a1 |
GH-49305: [Python] Expose RecordBatchFileReader.count_rows (#50646)
### Rationale for this change Resolves [49305](https://github.com/apache/arrow/issues/49305) `RecordBatchFileReader::CountRows` has existed in Arrow C++ (`cpp/src/arrow/ipc/reader.h`) but was never bound in Python, so the only way to get the total number of rows of an IPC file was: ```python num_rows = sum(reader.get_batch(i).num_rows for i in range(reader.num_record_batches)) ``` ### This findings are done in original opened Issue #49305 That deserializes every record batch just to read its length, which is wasteful and becomes expensive on remote filesystems. ### What changes are included in this PR? Adds `RecordBatchFileReader.count_rows()`: ```python with pa.ipc.open_file(source) as reader: reader.count_rows() ``` Three changes: * `python/pyarrow/includes/libarrow.pxd`: declare `CResult[int64_t] CountRows()` on `CRecordBatchFileReader`, which was the missing piece. * `python/pyarrow/ipc.pxi`: add `count_rows()` to `_RecordBatchFileReader`, released GIL around the call, with the same closed reader guard used by the existing `stats` property * `python/pyarrow/tests/test_ipc.py`: tests. On the naming: `count_rows()` follows the C++ method and is consistent with the existing `count_rows()` on `Dataset`, `Scanner` and `Fragment`. To be precise about the benefit, since the issue describes it as reading the count from the metadata: the C++ implementation still walks every block, but reads only each record batch's flatbuffer message header to pick up its length, and never touches the data buffers. So this is a reduction in bytes read rather than in the number of reads, and the gain shows up on remote filesystems and on files with large batches rather than in a local in memory benchmark. This is only added to the file reader. The stream reader has no footer and cannot count rows without consuming the stream. ### Are these changes tested? Yes, two tests in `python/pyarrow/tests/test_ipc.py`: * `test_file_count_rows`: count matches the sum of the written batch lengths, and counting does not consume the reader (count, `read_all()`, count again). * `test_file_count_rows_no_batches`: a file with a schema but no batches counts 0. Locally `test_ipc.py` passes (72 tests) and `test_feather.py` passes (83 passed, 8 skipped, 1 xfailed), the latter because the feather reader sits on the same file reader. The docstring example was run and produces the output shown. ### Are there any user-facing changes? Yes, a new public method `RecordBatchFileReader.count_rows()`. No existing behaviour changes. ### AI usage disclosure I used Claude Code to locate the unbound C++ method and the place where the declaration was missing, and to draft the binding, the docstring and the tests. I reviewed the result, rebuilt PyArrow locally, ran the test suites quoted above, and checked the C++ implementation of `CountRows` myself to confirm what it actually does before describing the benefit here. * GitHub Issue: #49305 Authored-by: Guja <127162872+GujaLomsadze@users.noreply.github.com> Signed-off-by: Antoine Pitrou <antoine@python.org> |
||
|
fenfeng9
|
ee6ff0a1b6 |
GH-50660: [C++][Dev] Add Decimal32 and Decimal64 GDB pretty-printers (#50723)
### Rationale for this change The Arrow GDB pretty-printers support `Decimal128` and `Decimal256`, but do not support `Decimal32` or `Decimal64`. ### What changes are included in this PR? * Add GDB pretty-printer support and tests for `Decimal32` and `Decimal64`. * Remove the unused `max_type_id` variable. ### Are these changes tested? Yes. ### Are there any user-facing changes? No. * GitHub Issue: #50660 Authored-by: fenfeng9 <fenfeng9@qq.com> Signed-off-by: Antoine Pitrou <antoine@python.org> |
||
|
Max Gorbuk
|
bda98c4d8c |
GH-37476: [C++][Python] Preserve unsigned dictionary index types when building from values (#50475)
### Rationale for this change An unsigned dictionary index type is silently replaced by the signed one of the same width: ```python >>> pa.array(["a", "b"], type=pa.dictionary(pa.uint32(), pa.string())).type dictionary<values=string, indices=int32, ordered=0> ``` It also makes `pa.chunked_array(values, dict_type)` fail with `Array chunks must all be same type`, since the chunk built internally comes back a different type than requested. `DictionaryBuilderCase::CreateFor()` reduces the requested index type to its byte width and hands that to `AdaptiveIntBuilder`, so everything but the width is dropped and the type is rebuilt from what the indices builder reports, which is always signed. @ jorisvandenbossche suggested `AdaptiveUIntBuilder` for unsigned types. That returns a different builder class, which `util/converter.h`, `json/from_string.cc` and the R binding all cast to `DictionaryBuilder<T>`, and `TestDictionaryUnifier.ChunkedArrayNestedDict` hits a `checked_pointer_cast` DCHECK. So this keeps one builder class and just reports the requested signedness. Indices are non-negative and same-width signed and unsigned types have the same layout, so it is value-preserving and free, and the width stays adaptive. ### What changes are included in this PR? `DictionaryBuilderBase` and its `NullType` specialization record the requested signedness and map the signed index type to the unsigned one of the same width in `type()`, `FinishInternal()` and `FinishDelta()`; `CreateFor()` passes it through. The mapping helper lives in `builder.cc`, not the header. `util/converter.h`, `python_to_arrow.cc` and `r_to_arrow.cpp` are untouched, so R is fixed rather than broken. uint64 indices also convert to pandas now, mapped to int64 codes. That is safe because the indices are bounds-checked below the dictionary length, so they never reach the int64 limit; `WriteIndicesUniform` handles uint64 like it already handles uint32. ### Are these changes tested? Six tests in `array_dict_test.cc` cover the distinct paths: type and finish preservation across signed and unsigned, the width adapting while staying unsigned, the `NullType` builder, `FinishDelta`, the supplied-dictionary constructor and the exact-index builder. Three parametrized tests in `test_array.py` cover the eight index types, the width adapting and the uint64-to-pandas conversion, and `test_dictionary_with_pandas` now checks that conversion instead of the old error. A `uint8` grows to `uint16` once more than 128 distinct values are added and stays unsigned; the underlying builder is signed, so it widens after 128 rather than the 256 a real uint8 could hold. `arrow-array-test`, `arrow-ipc-read-write-test`, `arrow-compute-scalar-cast-test`, `arrow-c-bridge-test` and the pyarrow `test_array` and `test_pandas` suites pass. ### Are there any user-facing changes? `pa.array` and `pa.chunked_array` return the requested unsigned index type. `dictionary(uint64(), ...)` produces real uint64 indices, and converting such an array to pandas now works where it used to raise. `take`, `filter`, `cast`, `dictionary_decode`, IPC and scalar access all keep the index type. * GitHub Issue: #37476 Authored-by: mkzung <103102868+mkzung@users.noreply.github.com> Signed-off-by: Antoine Pitrou <antoine@python.org> |
||
|
Raúl Cumplido
|
6287272f39 |
GH-50464: [C++][Python] Simplify arrow_to_pandas DateOffset handling for nanoseconds/milliseconds (#50465)
### Rationale for this change Pandas 1.4 fixed the following: - https://github.com/pandas-dev/pandas/issues/43892 Which made us special case microseconds extracting those from our `MonthDayNano` and manually building `microseconds`. `DateOffset` can be created with the following: ```python DateOffset(years=1, months=1, days=1, seconds=1, microseconds=1, minutes=1, hours=1, weeks=1, nanoseconds=1) ``` We currently only roundtrip Month, Day, Microseconds and Nano due to our MonthDayNano data type. We special cased microseconds but we were not special casing years, seconds, minutos, hours or weeks. As the initial issue that made us special case has been solved upstream we can stop roundtripping microseconds and just return nanoseconds simplifying our codebase. ### What changes are included in this PR? Remove the unnecessary code to retrieve microseconds from nanoseconds. Update test as microseconds won't be returned, only nanoseconds. ### Are these changes tested? Yes via CI ### Are there any user-facing changes? Yes, this is a change of the UX. * GitHub Issue: #50464 Authored-by: Raúl Cumplido <raulcumplido@gmail.com> Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com> |
||
|
Guja
|
10cf43bee6 |
GH-50579: [Python] Fix test_categorical_order_survives_roundtrip pandas 3.X deprecation (#50608)
### Rationale for this change Follow-up to #50543. `test_categorical_order_survives_roundtrip` constructs a `pd.Categorical` with value "a" not in categories `["b","c","d"]`. pandas 3.0.3 raises a deprecation for out-of-category values, which #50543's filterwarnings config now turns into an error - failing nightly wheels across platforms. ### What changes are included in this PR? Replace the out-of-category `"a"` value with an explicit `None`, preserving the test's intent (ordered categorical with non-sorted categories survives a Parquet roundtrip, ARROW-6302) plus a null. ### Are these changes tested? Yes. Reproduced the failure with pandas 3.0.3, confirmed the test passes after the change; all categorical tests in the module pass. ### Are there any user-facing changes? No. Closes #50579. * GitHub Issue: #50579 Authored-by: Guja <127162872+GujaLomsadze@users.noreply.github.com> Signed-off-by: Rok Mihevc <rok@mihevc.org> |
||
|
Liang-Chi HsiehandIsaac
|
0083d11f61 |
GH-50326: [Python] Convert arrays to Python objects without per-element Scalars in to_pylist (#50327)
### Rationale for this change `pa.Array.to_pylist()` converts one element at a time through `Array::GetScalar` plus a Python `Scalar` wrapper; for list types each row additionally allocates a Python `Array` wrapper for the row's values slice and a fresh generator before recursing per element. A `sample` profile shows ~20% of runtime in CPython GC (triggered by the per-row GC-tracked allocations), ~25% in `GetScalar`, and only ~7% doing the useful work of creating the output objects — making `to_pylist` several times slower than converting via `to_pandas()` and back, and ~24x slower than `ndarray.tolist()` for plain int64. Details in #50326; this hit Apache Spark's Arrow-serialized Python UDFs (apache/spark#56940, apache/spark#56943). ### What changes are included in this PR? Following review feedback, this adds a general scalar-free conversion mechanism instead of per-type `to_pylist` overrides: - `Array` gains `cdef object _getitem_py(self, int64_t i)`, returning `self[i]` as a Python object. The base implementation is `GetScalar` + `Scalar.as_py`, so any type without a specialization behaves exactly as today (dates, times, timestamps, durations, decimals, dictionary, extension, unions, views, ...). - The baseline `Array.to_pylist` becomes a single loop over `_getitem_py`. `maps_as_pydicts != None` keeps the Scalar-based path, since map→dict conversion has per-entry duplicate-key semantics. - Specializations avoid all per-element Scalar and per-row Array-wrapper allocation: - integers and floats (a `type_id` switch on `NumericArray`; date/time/timestamp subclasses fall through to the exact base), - boolean, - string/binary and large variants (`GetValue` + `PyUnicode_DecodeUTF8` / `PyBytes_FromStringAndSize`, matching `str(buf, 'utf8')` / `to_pybytes()` exactly), - list/large_list/fixed_size_list (each row's list is built from the child's `_getitem_py` over the offset range; the wrapped child is cached on the parent array), - map (association list of key/value tuples, matching `MapScalar.as_py`), - struct (one dict per row; duplicate field names fall back to the Scalar path so they raise `ValueError` like `StructScalar.as_py`). Nested types compose without any per-row wrappers. `ChunkedArray.to_pylist`, `Table.to_pylist` and `ListScalar.as_py` delegate here and speed up automatically. Follow-up candidates: string/binary views, run-end-encoded, dictionary, a fast path for date32. Benchmarks (macOS arm64, M4 Max): | benchmark | before | after | speedup | |---|---|---|---| | flat `int64` with nulls (4M) | 0.39 s | 0.028 s | 14x (~7 ns/element, on par with `ndarray.tolist`) | | flat `string` (4M) | 0.83 s | 0.06 s | 14x | | `list<string>` (2M rows) | 1.93 s | 0.46 s | 4.2x | | `list<list<int32>>` (1M rows) | 2.10 s | 0.40 s | 5.2x | | `struct<int64,string>` (1M rows) | 0.91 s | 0.07 s | 13x | | `map<string,int64>` (1M rows) | 2.77 s | 0.74 s | 3.8x | ### Are these changes tested? `test_to_pylist_bulk_paths` (added here) compares against the per-scalar conversion with exact element types for representative arrays including sliced views. Additionally verified with a randomized differential test against `[x.as_py() for x in arr]` with exact-type equality: all integer widths (incl. values beyond 2^62), floats (NaN/inf), boolean, string/binary (+large, multibyte), all list kinds, nested lists, struct (incl. empty struct, duplicate-field-name `ValueError`), map (incl. strict-mode duplicate-key `KeyError`), dictionary/null fallbacks, sliced/chunked arrays, and both `maps_as_pydicts` modes — no differences. `pytest test_array.py test_scalars.py test_convert_builtin.py test_table.py test_types.py`: 1295 passed. ### Are there any user-facing changes? No behavior changes, only performance. * GitHub Issue: #50326 This pull request and its description were written by Isaac. Lead-authored-by: Liang-Chi Hsieh <liangchi.hsieh@databricks.com> Co-authored-by: Isaac Signed-off-by: Antoine Pitrou <antoine@python.org> |