### Rationale for this change
resolve#39961
### What changes are included in this PR?
When writing a dataset with CsvFileFormat configured with a custom ParseOptions delimiter (e.g. delimiter=">"), the output CSV still used the default "," delimiter. This was because:
- C++ CsvFileFormat::DefaultWriteOptions() always created WriteOptions::Defaults() with delimiter=',', ignoring the parse_options.delimiter stored on the format object.
- Python CsvFileFormat.make_write_options() unconditionally overwrote the C++ write options with a fresh WriteOptions(**kwargs), discarding any C++-side default that might have been set.
Fix the C++ side by propagating parse_options.delimiter into the write options in DefaultWriteOptions(). Fix the Python side by reading the delimiter from the C++ default when the caller does not explicitly specify one, preventing the overwrite from losing the propagated value.
### Are these changes tested?
Yes
### Are there any user-facing changes?
No
* GitHub Issue: #39961
Lead-authored-by: egolearner <lijiliang@outlook.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
`scikit-build-core` 1.0 added `[tool.scikit-build.sdist.force-include]` in order to force the inclusion of files out of the source tree.
This allows us to remove our workaround with a custom build-backend to solve adding our license files from the main Arrow repository.
### What changes are included in this PR?
Add configuration to force-include `LICENSE.txt` and `NOTICE.txt` and remove custom build-backend.
### Are these changes tested?
Yes, CI and local
### Are there any user-facing changes?
No
* GitHub Issue: #50493
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Towards https://github.com/apache/arrow/issues/50398
closes https://github.com/apache/arrow/issues/50431
### What changes are included in this PR?
This is an agent generated PR that makes `FlightError` a non-cdef subclass of `Exception` as Cython does not support building with the Limited API and cdef classes that inherit from Python classes.
It appears before a `cdef CStatus to_status` was defined on each subclass to return a `MakeFlightError` per flight error flavor. This was pull out to a dedicated `_flight_error_to_status` function instead.
### Are these changes tested?
Yes (via existing tests?)
### Are there any user-facing changes?
No
* GitHub Issue: #50398
* GitHub Issue: #50431
Authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com>
Signed-off-by: David Li <li.davidm96@gmail.com>
### Rationale for this change
pandas 3.0.0 deprecated constructing or casting to a `Categorical` with
non-NA values that are not present in the specified `dtype.categories`.
This surfaces as DeprecationWarnings in the PyArrow
test suite when run against pandas >= 3.0 (GH-49255), in test_category
and test_category_implicit_from_pandas.
### What changes are included in this PR?
- test_category: remove the `cat_strings_with_na` column (relied on
silent 'qux' -> NaN; redundant, cat_strings already carries nulls).
- test_category_implicit_from_pandas: use explicit None
(['a','b',None]) instead of out-of-category 'c'.
### Are these changes tested?
Yes - test-only. Verified no DeprecationWarning under pandas 3.0.3.
### Are there any user-facing changes?
No.
Closes#49255.
* GitHub Issue: #49255
Authored-by: GujaLomsadze <lomsadze.guja@gmail.com>
Signed-off-by: Rok Mihevc <rok@mihevc.org>
### Rationale for this change
Fix#50424 `Python Emscripten Crossbow` job with multiple failing points
A) failing to install Chrome - `Requested Chrome major 148, but apt repo currently publishes 150.0.7871.100`,
B) and underlying Abseil failure - `error: expected '(' before 'ABSL_INTERNAL_CONSTEXPR_SINCE_CXX17'` [logB](https://github.com/ursacomputing/crossbow/actions/runs/28963702667/job/85941369345#step:6:2995)
C) pytest 8.1.1 failing on three tests with the message - `E TypeError: importorskip() got an unexpected keyword argument 'exc_type'` [logC](https://github.com/ursacomputing/crossbow/actions/runs/28972366846/job/85971005157#step:6:6354)
The remaining intermittent issue be best dealt with in a follow up PR:
D) intermittent `loadPyodide` / Selenium timeout hang - `selenium.common.exceptions.TimeoutException: Message: script timeout` [logD](https://github.com/ursacomputing/crossbow/actions/runs/28997397611/job/86049819728#step:6:6856) and [hang log with debug](https://github.com/ursacomputing/crossbow/actions/runs/29015193964/job/86108340808#step:6:7248)
### What changes are included in this PR?
A) Changed `chrome_version: "148"` to `chrome_version: "latest"`
B) Disabled zlib for protobuf build of the host in order to avoid conda include path overriding vendor Abseil headers.
C) Added work-around for current Pyodide's bundled pytest version 8.1.1 which lacks `importorskip(..., exc_type=...)`
### Are these changes tested?
Yes, test-conda-python-emscripten job completed successfully on rerun. Some earlier runs hit the intermittent Chrome/Selenium timeout mentioned above.
### Are there any user-facing changes?
No.
* GitHub Issue: #50424
Authored-by: Tadeja Kadunc <tadeja.kadunc@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
#49511 : `Table.join_asof` outputs a non-deterministic number of chunks, so the printed table sometimes appears as one chunk and sometimes as several (rarely).
`default_memory_pool` prints `bytes_allocated=0` except when a preceding (failing) doctest allocated from the process-global pool (also rarely).
### What changes are included in this PR?
Addition of `.combine_chunks()` to `Table.join_asof` (and removal of two trailing spaces) in `table.pxi`
Change to `bytes_allocated=...` from `bytes_allocated=0` in `memory.pxi`
### Are these changes tested?
Yes, local doctest tests OK.
### Are there any user-facing changes?
No, just docstring changes.
* GitHub Issue: #49511
Authored-by: Tadeja Kadunc <tadeja.kadunc@gmail.com>
Signed-off-by: Rok Mihevc <rok@mihevc.org>
### Rationale for this change
[Pandas 2.0.0](https://pypi.org/project/pandas/2.0.0/) was released on April 3rd 2023.
The last version of Pandas 1 was [Pandas 1.5.3](https://pypi.org/project/pandas/1.5.3/) and it was released on January 19th 2023. That version only supports up to Python 3.11.
We are currently supporting up to Pandas 1.5.2 (first version supporting 3.11).
On the issue we discussed to bump our Pandas support to 2.0.0 but given 2.0.3 is the latest in the 2.0.x, was released on June 2023, only supports up to Python 3.11 and bumping to it allows use to improve even further some of the conditionals we are bumping to Pandas >= 2.0.3
### What changes are included in this PR?
Update minimal support check to Version(2.0.3) and remove all test skips, conditionals and scenarios where we are checking for Pandas < 2.0.3.
We found a couple of minor things
### Are these changes tested?
Yes via CI
### Are there any user-facing changes?
Yes, Pandas < 2 won't be supported.
* GitHub Issue: #48808
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Python 3.10 is End of Support on October 2026. We can drop support on PyArrow 26.0.0.
See: https://endoflife.date/python
### What changes are included in this PR?
Remove support for 3.10. Update minimum required Python version and update some CI jobs to use the minimum Python supported version.
Some CI jobs using default Python 3.10 (Ubuntu 20.04) have been bumped.
### Are these changes tested?
They will be tested via CI and Archery
### Are there any user-facing changes?
Yes in terms of not supporting older versions of Python. This will be a breaking change for some users but not for the API itself
* GitHub Issue: #50410
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
`test_fastparquet_cross_compatibility` is not exercised and when exercised with latest versions it fails because it's not compatible with pandas 3 and it won't be as the project is retired, see:
- https://github.com/dask/fastparquet/pull/973
### What changes are included in this PR?
Removal of fixture to mark test as `fastparquet` and remove single test that still required fastarquet.
### Are these changes tested?
Yes and no. CI is exercised as expected but `fastparquet` tests weren't exercised on CI in the first place.
### Are there any user-facing changes?
No
* GitHub Issue: #37853
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
There is currently no method to convert Arrow Table to Arrow Tensor (conversion from columnar format to a contiguous block of memory). This work is a continuation of `RecordBatch::ToTensor` work, see https://github.com/apache/arrow/issues/40058.
### What changes are included in this PR?
This PR:
- implements `Table::ToTensor` conversion
- adds bindings to Python
- adds benchmarks in C++
- removes the code in `RecordBatch::ToTensor` and uses the Table implementation (`RecordBatch::ToTensor` benchmarks checked)
### Are these changes tested?
Yes, in C++ and Python.
### Are there any user-facing changes?
No, it is a new feature.
* GitHub Issue: #40062
Lead-authored-by: AlenkaF <frim.alenka@gmail.com>
Co-authored-by: Alenka Frim <AlenkaF@users.noreply.github.com>
Co-authored-by: tadeja <tadeja@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Rok Mihevc <rok@mihevc.org>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
Casting to `binary_view` or `string_view` could leave a null variadic buffer slot when all values were inline. This could happen for casts from `binary`, `large_binary`, `string`, `large_string`, and `fixed_size_binary`.
The C Data Interface exporter reads every variadic buffer to get its size. Because of that, exporting such an array could crash, for example through PyArrow `_export_to_c`.
Validation also passed for these arrays. For all-inline view arrays, validation never needed to read an out-of-line data buffer.
### What changes are included in this PR?
This PR fixes the cast kernels so all-inline view arrays do not keep a null variadic buffer slot.
It also makes validation reject null variadic buffer slots, and makes C Data export return an error instead of crashing.
C++ and Python regression tests cover the cast, validation, and export paths.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
**This PR contains a "Critical Fix"** Exporting an all-inline view array through the C Data Interface could crash the process while using only public APIs.
* GitHub Issue: #49740
Authored-by: fenfeng9 <fenfeng9@qq.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Arrow C++ is deprecating the Feather reader/writer (#49231) so we should update the Python functions too. Feather V2 is exactly the Arrow IPC file format, making the separate `pyarrow.feather` module redundant. Users should migrate to `pyarrow.ipc`.
### What changes are included in this PR?
- Added `FutureWarning` deprecation warnings to all four public APIs in `pyarrow.feather`: `write_feather()`, `read_feather()`, `read_table()`, and `FeatherDataset`
- Extracted `_read_table_internal()` so that `read_feather()` calling `read_table()` internally does not produce double warnings
- Updated `FeatherDataset.read_table()` to use the internal function for the same reason
- Added `.. deprecated:: 24.0.0` directives to all four public API docstrings
- Added module-level `pytestmark` filter in `test_feather.py` to suppress warnings in existing tests
- Added 5 new tests verifying deprecation warnings are emitted correctly (including a no-double-warning test)
- Added `@ pytest.mark.filterwarnings` to 5 test functions in `test_dataset.py` that use feather as a utility
- Updated `feather.rst` with a deprecation notice and migration guide (including a note that `pyarrow.ipc` does not compress by default unlike `feather.write_feather`)
- Updated `formats.rst` to mark the Feather API section as deprecated
### Are these changes tested?
Yes. Five new tests verify that each deprecated function emits exactly one `FutureWarning`, and existing tests continue to pass with module/function-level warning filters.
### Are there any user-facing changes?
Yes — calling `pyarrow.feather.write_feather()`, `read_feather()`, `read_table()`, or `FeatherDataset()` now emits a `FutureWarning` directing users to `pyarrow.ipc` equivalents. Existing functionality is unchanged.
* GitHub Issue: #49232
Lead-authored-by: Piyush Kanti Chanda <piyush.chanda@databricks.com>
Co-authored-by: Isaac
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
Arrow has `atan2` and `logb` but no `hypot`. Computing a Euclidean norm currently needs `sqrt(add(multiply(x, x), multiply(y, y)))`, which can overflow at the `multiply` step even when the result is representable.
### What changes are included in this PR?
Add a `hypot` kernel backed by `std::hypot` (float32/float64, with integer/decimal inputs promoted to float64), the `Hypot()` C++ function, a `FunctionDoc` and `compute.rst` entry, and a Python expression test. `pc.hypot` is auto-exposed from the registry.
### Are these changes tested?
Yes, C++ tests for the basic cases, null/NaN/Inf handling, and an overflow case confirming it doesn't fall back to a naive `sqrt(x*x + y*y)`.
### Are there any user-facing changes?
Yes, a new `hypot` function in C++ and Python. Additive only.
* GitHub Issue: #50197
Lead-authored-by: Shrivas Shankar <shrivasshankar2019@gmail.com>
Co-authored-by: Shrivas Shankar <86219405+shrivasshankar@users.noreply.github.com>
Co-authored-by: Rok Mihevc <rok@mihevc.org>
Signed-off-by: Rok Mihevc <rok@mihevc.org>
### Rationale for this change
On the hypothesis tests we treat Warnings as errors and the job has been failing with the latest Pytest version (9.1.0). This is related to:
- https://github.com/pytest-dev/pytest/issues/13409
With the latest pytest using generators for parametrization has been deprecated.
### What changes are included in this PR?
Change the generator that was causing the failures and return a list of supported_factories instead.
### Are these changes tested?
Yes both locally and the archery test that was failing.
### Are there any user-facing changes?
No
* GitHub Issue: #50277
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Constructing a fixed-shape-tensor array from a list of individual ndarrays only
worked when each element was 1-D; ≥2-D elements failed with
`ArrowInvalid: Can only convert 1-dimensional array values`. The only workaround
was stacking the list into a single ndarray and using
`FixedShapeTensorArray.from_numpy_ndarray`.
### What changes are included in this PR?
The C++ list converter `PyListConverter::AppendNdarray` now accepts
multi-dimensional ndarray elements for **fixed-size lists** (the storage of a
fixed-shape tensor) by flattening them in C order. The fixed-size-list builder
still validates that the flattened length matches the list width, so wrong sizes
error cleanly. Variable-sized lists remain restricted to 1-D values to avoid
ambiguity. As a side benefit, plain `fixed_size_list` also accepts
multi-dimensional ndarray elements now.
### Are these changes tested?
Yes:
- `test_tensor_array_from_list_of_ndarrays` — construction from 2-D and 3-D
ndarrays, null handling, storage parity with `from_numpy_ndarray`, and the
size-mismatch error, across `int8`/`int64`/`float32`.
- `test_fixed_size_list_from_multidim_ndarray` — plain `fixed_size_list` from
multi-dim arrays, plus a check that variable-sized lists still reject 2-D.
### Are there any user-facing changes?
Yes — `pa.array([multi-dim ndarrays], type=fixed_shape_tensor(...))` (and the
same for `fixed_size_list`) now works instead of raising. Existing 1-D behavior
and variable-sized-list behavior are unchanged.
Scoped to construction only; the reverse `to_numpy` shape-preservation also
raised in the issue is intentionally left as a separate follow-up.
* GitHub Issue: #49644
Lead-authored-by: Samuel Aboderin <aboderinseun01@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Rok Mihevc <rok@mihevc.org>
### Rationale for this change
Since 24.0, pyarrow ships annotations and `py.typed` marker while the type stubs are still incomplete. This makes type checkers trust the partial stubs and report false errors in downstream code (e.g. Module has no attribute "all" for `pyarrow.compute`). Some type checkers also consume bundled `.pyi` files even without `py.typed`, so the stubs need to be withheld from wheels for now. See #49831.
### What changes are included in this PR?
This temporarily omits both `pyarrow/py.typed` and the bundled `pyarrow-stubs` / `.pyi` files from built wheels until the stubs are complete. Wheel-content validation now asserts that neither `py.typed` nor `.pyi` files are present, and wheel build scripts no longer request stub docstring injection while stubs are not installed.
### Are these changes tested?
Wheel-content validation has been updated to check the intended absence of `py.typed` and `.pyi` files.
### Are there any user-facing changes?
Type checkers no longer pick up pyarrow's incomplete stubs from wheels.
* GitHub Issue: #49831
Authored-by: Rok Mihevc <rok@mihevc.org>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
See #38584 for original PR. Will be quoted for this PR description.
### Rationale for this change
> support multi sortkey nulls first.
```
order by i nulls first, j, k nulls first;
```
> The current null sorting only supports all sortkeys, not a certain sortkey, so NullPlacement is extended to the SortKey field. Since the underlying framework is very well written, when modifying this function, you only need to pass the null_placement of each SortKey in. That’s it.
### What changes are included in this PR?
> 1.SortKey structure, NullPlacemnt transfer logic, sorting logic and Ording related, test related
> 2.Substriait related.
> 3.c_glib related.
> 4.SelectK related.
> 5.RankOptions related.
### Are these changes tested?
> yes, I changed the code inside vector_sort_test.cc and performed additional tests.
### Are there any user-facing changes?
> yes, pg database include null sorting of multiple sort keys.
**This PR includes breaking changes to public APIs.** (If there are any breaking changes to public APIs, please explain which changes are breaking. If not, you can remove this.)
I amended the original PR to be less breaking in public APIs.
Still Ordering, SortOptions, RankOptions, and RankQuantileOptions now accept a `std::optional<NullPlacement>` instead of NullPlacement, which did lead to some changes in downstream APIs and bindings. ~I also need some help with fixing the `c_glib` bindings.~
* GitHub Issue: #38558
Lead-authored-by: Alexander Taepper <alexander.taepper@gmail.com>
Co-authored-by: light-city <455954986@qq.com>
Co-authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Some of our CI jobs are failing due to a breaking change on pytest 9.1:
https://github.com/pytest-dev/pytest/pull/14293
### What changes are included in this PR?
Add `exc_type=ImportError` to the `pytest.importorskip("pyarrow.*)` calls
### Are these changes tested?
Via CI
### Are there any user-facing changes?
No
* GitHub Issue: #50176
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
PyArrow adds a `__inex_level_i__` column to the PyArrow table if the pandas dataframe has an unnamed index it wants to preserve. Currently that creates a duplicate, if such a column already exists.
### What changes are included in this PR?
Bumping the integer number in the generated column in order to not get any duplicates.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
* GitHub Issue: #46179
Authored-by: AlenkaF <frim.alenka@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Fix : https://github.com/apache/arrow/issues/50072
### Rationale for this change
The `replace_with_mask` compute function in PyArrow currently has no Python-level tests for regular use cases.
This PR adds test cases for standard usage of `replace_with_mask` to ensure the function works correctly.
### What changes are included in this PR?
Added parameterized test cases in `python/pyarrow/tests/test_compute.py`
### Are these changes tested?
Yes, Manually
### Are there any user-facing changes?
No user-facing changes.
* GitHub Issue: #50072
Authored-by: Ankit.Ahlawat@ibm.com <Ankit.Ahlawat@ibm.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
Sliced union arrays could report incorrect results from `count`.
For sliced union arrays, `span.GetValues<int8_t>(1)` and `span.GetValues<int32_t>(2)` already return offset-adjusted pointers. The logical null count path then indexed those pointers with `span.offset + i`, effectively applying the parent offset twice.
The sparse union path also checked child nullness with `i` instead of `span.offset + i`.
### What changes are included in this PR?
* Fixes logical null counting for sliced sparse and dense union arrays.
* Add C++ and Python regression tests.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
* GitHub Issue: #50113
Authored-by: fenfeng9 <fenfeng9@qq.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Sliced sparse unions could report incorrect nullness during element access.
The type id lookup used the union offset, but the child null check still used the slice-relative index.
### What changes are included in this PR?
* Updates `IsNullSparseUnion` and `ArraySpan::IsNullSparseUnion` in `data.cc` to use the parent offset when checking child nullness.
* Add C++ and Python regression tests.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
* GitHub Issue: #50105
Authored-by: fenfeng9 <fenfeng9@qq.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Fixes#50037
### What changes are included in this PR?
Add `gc.collect()` before `prior_allocation = pa.total_allocated_bytes()` in `test_table_uses_memory_pool`
and also add to `_check_to_pandas_memory_unchanged` helper with the same pattern.
Same fix as #44793
### Are these changes tested?
Not reproduced locally yet.
### Are there any user-facing changes?
No.
* GitHub Issue: #50037
Authored-by: Tadeja Kadunc <tadeja.kadunc@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
When adding support to Python 3.14 the musllinux wheels failed with newer versions of the musllinux quay.io image. In order to prepare for the next Python 3.15 release we will require newer base images and we should fix the issues we originally found.
### What changes are included in this PR?
- Update Manylinux and Musllinux quay.io images to `2026.05.31-3`
- Use mold linker instead of ld on alpine in order to solve import segfault on pyarrow musllinux amd64
- Remove Python 3.13 free-threaded wheels as Python 3.13t is not available anymore on the underlying manylinux image, see:
- https://github.com/apache/arrow/issues/50086
### Are these changes tested?
Yes via archery
### Are there any user-facing changes?
Yes, Python 3.13 free threaded was experimental, PyPA removed cp313t from the underlying manylinux docker image. We won't support wheels for 3.13 free-threaded anymore. We will support for 3.14 on-wards.
* GitHub Issue: #48028
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
`hash_any` and `hash_all` could return incorrect results for sliced nullable boolean arrays.
The validity bitmap used the slice offset, but the boolean values bitmap did not.
### What changes are included in this PR?
Apply the slice offset when reading boolean values in `hash_any` / `hash_all`.
Add C++ and Python regression tests.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
* GitHub Issue: #50043
Authored-by: fenfeng9 <fenfeng9@qq.com>
Signed-off-by: Rok Mihevc <rok@mihevc.org>
### Rationale for this change
Docs render weirdly
### What changes are included in this PR?
Fix extra blank line breaking docs block
### Are these changes tested?
I'll render docs
### Are there any user-facing changes?
No
* GitHub Issue: #20403
Authored-by: Nic Crane <thisisnic@gmail.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
When data exceeds int32 limits, properly wraps each chunk as ExtensionArray
### What changes are included in this PR?
Modified extension type handling to support both Array and ChunkedArray storage types.
### Are these changes tested?
Yes , Manually tested the changes
### Are there any user-facing changes?
No
### This PR contains a "Critical Fix".
This change fixes a crash in list_ storage . when list data exceeds int32 limits, PyArrow automatically creates a ChunkedArray. However, ExtensionArray.from_storage() only accepts Array objects, not ChunkedArray.
* GitHub Issue: #50012
Authored-by: Ankit.Ahlawat@ibm.com <Ankit.Ahlawat@ibm.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
replace_with_mask crashes when called with null type arrays because a code path incorrectly returns Status::OK() from a function with return type Result<int64_t>.
This PR fixes the issue by ensuring the function returns a valid int64_t value instead of Status::OK() for successful execution.
### What changes are included in this PR?
1. Fixed the replace_with_mask kernel implementation for null type arrays.
2. Replaced the invalid Status::OK() return path with a valid int64_t result.
3. Prevented construction of Result<int64_t> with an OK Status.
### Are these changes tested?
Yes, manually run the test case
### Are there any user-facing changes?
No
**This PR contains a "Critical Fix".**
This change fixes a crash in replace_with_mask when operating on null type arrays. The crash occurs due to an invalid successful return path internally constructing Result<T> with Status::OK().
* GitHub Issue: #47447
Authored-by: Ankit.Ahlawat@ibm.com <Ankit.Ahlawat@ibm.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
Fix failing test on CI as reported in https://github.com/apache/arrow/issues/50041, for a test added in https://github.com/apache/arrow/pull/49694
### What changes are included in this PR?
The test relies on the lower case "europe/brussels" time zone name not being recognized by `zoneinfo` (and it generally is by `pytz`). But apparently on some platforms, `zoneinfo` does accept this. Updating the test to first test this, and thus skip the test dynamically.
* GitHub Issue: #50041
Authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
pq.read_schema drops extension types (UUID comes back as fixed_size_binary[16]), while ParquetFile.schema_arrow and read_table preserve them. Schema inspection via metadata should match table/extension behavior.
### What changes are included in this PR?
- Plumb arrow_extensions_enabled into read_schema and return schema_arrow when enabled so extension types are preserved.
- Add regression test ensuring UUID extension types are retained by read_schema and downgraded to binary(16) when extensions are disabled.
### Are these changes tested?
- Yes: added unit test test_read_schema_uuid_extension_type
### Are there any user-facing changes?
- Behavior improvement: read_schema now preserves extension types (e.g., UUID) when extensions are enabled; no API break
Notes:
- I don't know if the fact the column types being returned are now extension<arrow.uuid> instead of fixed_size_binary[16], is considered a breaking change.
- This PR patch was AI generated, but I personally reviewed it, the scope is small, and it looks fine to me.
* GitHub Issue: #48254
Authored-by: Nicolas Vandeginste <n.vandeginste@abc-arbitrage.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
See https://github.com/apache/arrow/issues/47435.
### What changes are included in this PR?
Adds direct encryption / decryption Python API
### Are these changes tested?
Yes, see PR.
### Are there any user-facing changes?
Yes, new Python bindings.
* GitHub Issue: #47435
Authored-by: Sreesh Maheshwar <smaheshwar@palantir.com>
Signed-off-by: Adam Reeve <adreeve@gmail.com>
### Rationale for this change
`zoneinfo` is available starting with Python 3.9, so we can now assume
that it is available, and so we can switch from returning `pytz`
timezones by default to return `zoneinfo` timezones (or
`datetime.timezone` for fixed offsets).
Only keeping pytz as fallback for strings that are not supported by
`zoneinfo` but were supported by `pytz`. Later, we should maybe
deprecate that fallback.
Generally we should move away from using `pytz`, since the core
functionality of having time zones is now available in the standard
library (`zoneinfo`), and because the pytz package has several warts /
incompatibilities with stdlib datetime
(https://blog.ganssle.io/articles/2018/03/pytz-fastest-footgun.html)
### What changes are included in this PR?
Whenever we create a python timezone object, which is when converting to
pandas or when converting to a `datetime.datetime` object:
- always prefer `zoneinfo` for `datetime.datetime` objects
- prefer `zoneinfo` for pandas objects _if_ pandas >= 3, to align with
the change on the pandas side
(https://github.com/pandas-dev/pandas/issues/34916)
In either case, when preferring `zoneinfo`, we still fall back to `pytz`
for named timezones if `zoneinfo` does not recognize the zone name
(apparently pytz can have some common (older) aliases that might not
always work with zoneinfo).
This fallback is something we could deprecate and remove later on (so we
can eventually remove all usage of pytz)
### Are these changes tested?
Yes
### Are there any user-facing changes?
**This PR includes breaking changes to public APIs.**
It is a different object that we return (different class, i.e.
`zoneinfo.ZoneInfo` instead of a `pytz.tzinfo.BaseTzInfo`, both are
still subclasses of `datetime.tzinfo`), which has some differences in
the API, so for people relying on that, this is a breaking change.
For the conversion to pandas, pandas itself has made this breaking
change anyhow, so for those cases it aligns with that change of pandas.
* GitHub Issue: #15047
## Summary
- Update `python/requirements-wheel-test.txt` for Python `<3.11` from `numpy~=1.21.3` to `numpy~=1.23.2`.
(Previously this PR pinned to `numpy~=1.22.0`, which closes the advisory but is one minor short of what the wheel tests need: `numpy.from_dlpack` was added in numpy 1.23. After @ raulcd's crossbow run failed on `AttributeError: module 'numpy' has no attribute 'from_dlpack'`, the floor was bumped to `~=1.23.2`, matching the cp311 line right below it.)
## Why
Dependabot flags `numpy < 1.22` in this requirements file (GHSA-fpfv-jqm9-f5jm / CVE-2021-34141, fixed in 1.22). `~=1.23.2` keeps that advisory closed and gives the wheel-test runtime `numpy.from_dlpack`, which arrow's test suite uses.
## Validation
- `python3 -m pip install -r python/requirements-wheel-test.txt` locally.
- The pin collapses to two distinct numpy floors across the matrix (cp310 + cp311 now share `~=1.23.2`).
- Awaiting another crossbow run for cp310 wheels.
* GitHub Issue: #49967
Authored-by: Arpit Jain <arpitjain099@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Add an optional default_column_type parameter to the CSV reading API (C++ and Python) to provide a fallback type when per-column types aren’t specified, improving schema consistency and complementing the existing column_types logic.
### What changes are included in this PR?
- c++: new convert option "default_column_type" to augment logic around column_types parameter
- 3 reader tests: DefaultColumnTypePartialDefault, DefaultColumnTypeAllStringsWithHeader, DefaultColumnTypeAllStringsNoHeader). The last two tests are inspired by https://github.com/pandas-dev/pandas/pull/62242 and https://github.com/pandas-dev/pandas/issues/57666
- python: corresponding changes to make cpp change consumable from python
- python: extended test_convert_options test - include. default_column_type
- python: added new test "test_default_column_type" which tests how the field impacts schema; also test implicitly verifies leading zero preservation
- relevant documentation update for python component;
### Are these changes tested?
Yes. Existing and new tests are passing.
C++:
> [==========] Running 3 tests from 1 test suite.
> [----------] Global test environment set-up.
> [----------] 3 tests from ReaderTests
> [ RUN ] ReaderTests.DefaultColumnTypePartialDefault
> [ OK ] ReaderTests.DefaultColumnTypePartialDefault (3 ms)
> [ RUN ] ReaderTests.DefaultColumnTypeAllStringsWithHeader
> [ OK ] ReaderTests.DefaultColumnTypeAllStringsWithHeader (0 ms)
> [ RUN ] ReaderTests.DefaultColumnTypeAllStringsNoHeader
> [ OK ] ReaderTests.DefaultColumnTypeAllStringsNoHeader (0 ms)
> [----------] 3 tests from ReaderTests (4 ms total)
>
> [----------] Global test environment tear-down
> [==========] 3 tests from 1 test suite ran. (4 ms total)
> [ PASSED ] 3 tests.
All:
> [==========] 264 tests from 46 test suites ran. (452 ms total)
> [ PASSED ] 264 tests.
pyarrow:
New tests are passing.
### Are there any user-facing changes?
I believe this change is backward compatible. Parameter is optional and its default value doesn't change the existing behavior; All the existing rests are passing.
Maybe relevant: https://github.com/apache/arrow/issues/22232
Relates to https://github.com/apache/arrow/issues/47502
* GitHub Issue: #47502
* GitHub Issue: #22232
Lead-authored-by: Vlad Borovtsov <borovcov.vlad@gmail.com>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
The `134` version fails to be installed from stable releases on chrome due to a new stable version released.
### What changes are included in this PR?
- Update version of latest stable chrome driver to `148` and how we look for version on instrall script.
- Update selenium to newer version to work with newer chrome.
- Fixes to cross-compilation build after upgrading Abseil and Protobuf and moving to FetchContent.
- Updated timeout so Chrome job doesn't time out.
- Skip substrait tests which currently fail on Pyodide.
### Are these changes tested?
Yes, via archery job.
### Are there any user-facing changes?
No
* GitHub Issue: #49522
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
`ColumnChunkMetaData.to_dict()` method omits `bloom_filter_offset` and `bloom_filter_length` even when a bloom filter is written to the Parquet file. Users cannot programmatically verify bloom filter presence via the Python metadata API without resorting to file size comparison.
### What changes are included in this PR?
1. `python/pyarrow/includes/libparquet.pxd`: Declare `bloom_filter_offset()` and `bloom_filter_length()` (both optional[int64_t]) on `CColumnChunkMetaData`. This is to expose the existing C++ methods to Cython.
2. `python/pyarrow/_parquet.pyx`: Add `bloom_filter_offset` and `bloom_filter_length` properties to `ColumnChunkMetaData` (returns int when set, None otherwise). Add both fields to `to_dict()` and `__repr__`.
3. `python/pyarrow/tests/parquet/test_metadata.py`: Add `test_bloom_filter_offset_in_metadata` verifying that columns with a bloom filter expose non-None integer values and that `to_dict()` contains the keys, while columns without a bloom filter return None.
### Are these changes tested?
Yes. `test_bloom_filter_offset_in_metadata` in test_metadata.py covers:
- Column with bloom filter: bloom_filter_offset and bloom_filter_length are non-None integers
- Column without bloom filter: both return None
- Both keys present in to_dict() output
<img width="863" height="215" alt="image" src="https://github.com/user-attachments/assets/d465d6bd-55d1-4c5f-9f11-6a60b3bf1cbe" />
Here is closer look on the logic output:
<img width="464" height="424" alt="image" src="https://github.com/user-attachments/assets/6e4810f2-c1c0-41ea-b559-00f99d42e2c4" />
output:
```python
col_a bloom_filter_offset: 10699
col_a bloom_filter_length: 1040
col_b bloom_filter_offset: None
col_b bloom_filter_length: None
col_a to_dict(): {'file_offset': 0, 'file_path': '', 'physical_type': 'BYTE_ARRAY', 'num_values': 1000, 'path_in_schema': 'a', 'is_stats_set': True, 'statistics': {'has_min_max': True, 'min': 'id_0', 'max': 'id_999', 'null_count': 0, 'distinct_count': None, 'num_values': 1000, 'physical_type': 'BYTE_ARRAY'}, 'geo_statistics': None, 'compression': 'SNAPPY', 'encodings': ('PLAIN', 'RLE', 'RLE_DICTIONARY'), 'has_dictionary_page': True, 'dictionary_page_offset': 4, 'data_page_offset': 4035, 'total_compressed_size': 5336, 'total_uncompressed_size': 11208, 'bloom_filter_offset': 10699, 'bloom_filter_length': 1040}
```
* GitHub Issue: #49927
Lead-authored-by: Haziq Hakimi Mazlisham <95128744+haziqishere@users.noreply.github.com>
Co-authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Py_DECREF(item) in PyList_SetItem will cause Use-After-Free bug if `PyList_SetItem(indptr.obj(), i, item) < 0` is `true`, cause `PyList_SetItem` always steals a reference to the item, even when it fails.
### What changes are included in this PR?
1. Remove Py_DECREF(item) in PyList_SetItem error path.
### Are these changes tested?
By CI.
### Are there any user-facing changes?
No.
* GitHub Issue: #49917
Authored-by: Wang Rui <55612496+wr-web@users.noreply.github.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
In https://github.com/apache/arrow/pull/38637 we protected the `OwnedRef` and `OwnedRefNoGIL` destructors against being called when the Python interpreter is being finalized, but we didn't address other classes with custom destructors, such as `PyBuffer` and `NumPyBuffer`.
### What changes are included in this PR?
Avoid executing Python C API code if the interpreter is finalized while a C++ destructor runs.
### Are these changes tested?
Only manually, using the initial reproducer in https://github.com/apache/arrow/issues/45214.
### Are there any user-facing changes?
No.
* GitHub Issue: #49942
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Default `pre_buffer=True` on parquet APIs should probably be made consistent.
### What changes are included in this PR?
Couple of default parameter changes and an updated docstring.
### Are these changes tested?
By CI.
### Are there any user-facing changes?
Users will see changes to default behavior. It's hard to estimate how disruptive (if at all) this will be.
* GitHub Issue: #49923
Authored-by: Rok Mihevc <rok@mihevc.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
The `count` kernel used `GetNullCount()`, which reports the physical null count. For run-end encoded arrays, this ignored nulls in the encoded values child.
### What changes are included in this PR?
Use `ComputeLogicalNullCount()` in the `count` kernel so run-end encoded arrays are counted correctly. Add C++ and Python tests for this case.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
* GitHub Issue: #49888
Authored-by: fenfeng9 <fenfeng9@qq.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
Fixes#43574 by reverting a small portion of bd444106af
### Rationale for this change
This reverts a change made in pyarrow 17 which means that reading a single file returns different results when that file happens to be located in a path that contains `x=y` segments (i.e. that look like hive partition columns) than when it doesn't. Particularly given the way some higher-level calls wrap this functionality, e.g. by already opening a file before it is passed to `ParquetDataset`, this can lead to confusing results, e.g. that are different when running code on a local vs remote filesystem. For example, for single-file local reads, `pandas.read_parquet` already opens a filehandle to pass to pyarrow, while for remote reads, it passes a single-file path + filesystem, resulting in code that works differently when tested on a local filesystem compared to the deployed cloud filesystem.
The original change was introduced in https://github.com/apache/arrow/pull/39438 and there was a [discussion thread about it](https://github.com/apache/arrow/pull/39438#discussion_r1469251517) (sorry; github's links to resolved discussions don't always work well!) The gist of the discussion thread seems to be that the PR author thought that this code was unused, when in fact the subsequent issue shows that it _is_ used.
Screenshot of the original discussion thread to help you find it:
<img width="699" height="517" alt="image" src="https://github.com/user-attachments/assets/a01618cc-c39d-48fb-9cb8-bd2c1b0c604f" />
### What changes are included in this PR?
Restores special "single file" handling for single-file paths passed to `ParquetDataset` constructor, and analogous to the handling for an open file handle.
This results in the loaded dataset _not_ parsing the full file path for hive partition columns, which results in a different set of columns.
### Are these changes tested?
Added a new unit test. Verified that it fixes the issue I'd been observing, and which I'd commented on in #43574, though I don't have a working reproduction to verify that it fixes the original issue there.
### Are there any user-facing changes?
**This PR includes breaking changes to public APIs.** In particular, it changes the columns returned by single-file calls to `pyarrow.parquet.read_table(...)`, bringing the results back in line with pyarrow<17.
While technically a breaking change, it should be noted that the original PR that introduced this change in pyarrow 17 did not call out this change as a breaking change. However, it's been some time since then, and it's plausible that some applications have developed dependencies on the current behavior.
* GitHub Issue: #43574
Authored-by: Ben Kurtz <github+2026@benkurtz.org>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
When converting a pandas.Categorical with tz-aware datetime categories to a PyArrow array, the timezone information was silently dropped from the dictionary array's value type. This is a silent data loss bug — no warning or error is raised, but the timezone metadata is lost.
### What changes are included in this PR?
In `python/pyarrow/array.pxi`, the Categorical conversion was using `values.categories.values(raw numpy array) `which strips timezone metadata since numpy does not support tz-aware datetimes. Changed to values.categories (pandas Index) and added from_pandas=True so PyArrow uses the pandas conversion path, which correctly preserves timezone metadata.
### Are these changes tested?
Yes. Verified manually
### Are there any user-facing changes?
Yes — this is a bug fix. Users did #49875
This PR contains a **"Critical Fix"** — timezone information was lost silently during conversion without any warning or error.
* GitHub Issue: #49875
Authored-by: Ankit.Ahlawat@ibm.com <Ankit.Ahlawat@ibm.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
The Hypothesis test strategy for timezones (`pyarrow.tests.strategies.timezones`) only generates named IANA timezones (via pytz and zoneinfo). It does not generate fixed-offset timezones like `+05:30` or `-03:00`. As noted in #31318, this is not supported out of the box by Hypothesis, so a custom strategy is needed. Fixed-offset timezones are already used in manual tests and handled correctly by Arrow, but are never exercised via property-based testing.
### What changes are included in this PR?
Adds a `fixed_offset_timezones` strategy to `python/pyarrow/tests/strategies.py` that generates `datetime.timezone` objects with offsets ranging from UTC-12:00 to UTC+14:00, with minute components of 0, 30, or 45 (covering all real-world UTC offsets). This strategy is included in the existing `timezones` strategy across all dependency configurations.
### Are these changes tested?
This change improves test infrastructure itself -- it extends the `timezones` strategy used by existing Hypothesis-based tests. The generated fixed-offset timezones were verified to work correctly with `pa.timestamp()`.
The test command used was:
```sh
export MAMBA_ROOT_PREFIX=$PWD/.micromamba && eval "$(/tmp/bin/micromamba shell hook -s bash)" && micromamba activate pyarrow-dev && export LD_LIBRARY_PATH=$CONDA_PREFIX/lib:$LD_LIBRARY_PATH && python -m pytest pyarrow/tests/test_scalars.py pyarrow/tests/test_array.py -v -k test_dunders
```
### Are there any user-facing changes?
No. This only affects the internal test suite.
### AI-generated code disclosure
This PR was developed with assistance from an AI coding tool (Claude, Anthropic). All changes have been reviewed, understood, and verified.
* GitHub Issue: #31318
Authored-by: Alexandros Anastasiou <anastasioyaa@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
The goal of the MR is to support the low level python raw fds (`os.open()`)
### What changes are in this PR?
As an alternative to the `str` path, the `pa.OSFile` accepts the `int` parameter as well.
### Are these changes tested?
Yes, new end-to-end tests were added
### Are there any user-facing changes?
Yes, the API is extended with a new parameter type
This MR was created by Codex Cloud.
* GitHub Issue: #49751
Authored-by: Ádám Lippai <adam@rigo.sk>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Pandas is moving the dataframe interchange protocol out of the repo in the next major release (4.0.0).
### What changes are included in this PR?
Filterwarnings are added and the pandas example is removed from the interchange documentation.
### Are these changes tested?
Yes, via CI.
### Are there any user-facing changes?
No.
* GitHub Issue: #49628
Authored-by: AlenkaF <frim.alenka@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
