### Rationale for this change
In `pyarrow.feather.FeatherDataset.read_pandas`, the docstring previously stated:
```
Read multiple Parquet files as a single pandas DataFrame
```
which was copied from Parquet reader utilities.
### What changes are included in this PR?
Updates the docstring to correctly reference Feather files:
```
Read multiple Feather files as a single pandas DataFrame
```
### Are these changes tested?
Docstring update.
Authored-by: Arnav Ahire <arnavahire5@gmail.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
`pyarrow.types.is_nested()` said a run-end encoded type wasn't nested, while the C++ `arrow::is_nested()` says it is:
```python
>>> import pyarrow as pa
>>> t = pa.run_end_encoded(pa.int32(), pa.string())
>>> t.num_fields
2
>>> pa.types.is_nested(t)
False
```
The Python side reads from a hardcoded `_NESTED_TYPES` set in `python/pyarrow/types.py` rather than from the C++ trait, so it has to be kept in step by hand. Lining that set up against `is_nested()` in `cpp/src/arrow/type_traits.h`, run-end encoded was the only type the two still disagreed about — the list-view types and fixed-size list are already there.
It's also out of step with the type itself. Run-end encoded has two children, the run ends and the values, and every other type in pyarrow that has children answers `True` here.
This is the same thing that happened to fixed-size list in #40171, fixed by #40172, and the list-view types were added after that. This looks like the last one missed when run-end encoding went in.
### What changes are included in this PR?
Adds `Type_RUN_END_ENCODED` to `_NESTED_TYPES`, which is the whole fix.
In the test I've added the run-end encoded case, and while I was there also a map and a dictionary. Map was nested already but wasn't asserted anywhere, and dictionary is the interesting negative — it wraps a value type but is deliberately not nested in either implementation, so pinning it means a later change can't quietly sweep it in.
### Are these changes tested?
Yes. `test_is_nested_or_struct` fails on the current code and passes with the change.
I don't have a local C++ build, so I checked this by running the updated `test_types.py` against an installed pyarrow 25.0.0 with the same one-line change applied to its `types.py`. Before the change that file had 87 passing with `test_is_nested_or_struct` failing; after it, 88 passing. The two errors and one failure I see in both runs are environmental on my machine and unrelated — the errors are the `pickle_module` fixture, which comes from a conftest I wasn't loading, and the failure is `test_pytz_timezone_roundtrip`.
### Are there any user-facing changes?
Yes, though it's small. `pa.types.is_nested()` now returns `True` for run-end encoded types where it previously returned `False`. Anything branching on that predicate will take the nested path for these types, which is the intended answer and what the C++ implementation has always given. Nothing inside pyarrow reads `is_nested` or `_NESTED_TYPES`, so the effect is limited to callers.
* GitHub Issue: #50847
Authored-by: nishad shabbir <nishadshabbir28@gmail.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
Require Numpy 2.0 to support newer features, allow newer numpy API/ABI
### What changes are included in this PR?
- Numpy 2.0+ requirement
- Pandas 2.2.2+ requirement (first wheel compatible with numpy 2.0)
- Pyodide 0.27.1 usage (numpy 2.0 api)
### Are there any user-facing changes?
Yes, in contrast to pyarrow 25, after merging this PR pyarrow 26 can't be used with Numpy 1.x or pandas 2.0-2.2.1
**This PR includes breaking changes to public APIs.**
This PR was created using GPT-5.6-Sol-xhigh, every line read & reviewed by me.
* GitHub Issue: #48473
Lead-authored-by: Adam Lippai <adam@rigo.sk>
Co-authored-by: Ádám Lippai <adam@rigo.sk>
Co-authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Add [Python 3.15](https://docs.python.org/3.15/whatsnew/3.15.html) to the pyarrow nightly builds, so that issues can be detected early in the development cycles of both cpython & pyarrow.
Note:
- ~~CFFI and~~ Pandas doesn't have nightly 3.15 wheels, and are accordingly excluded.
- Numpy shipped 3.15rc1 in 2.5.2 ~~doesn't ship a 3.15 wheel, but is needed to build. It is building from source.~~
- [3.15 Schedule](https://peps.python.org/pep-0790/) This PR targets 3.15.0rc1. rc2 is expected 2026-09-01 and final 2026-10-01.
### What changes are included in this PR?
- Adding cp315/cp315t to the build matrix
- Excluding ~~cffi/~~ pandas for 3.15, as the wheels aren't yet available.
### 2026-08-13 edit
- Bump libcst to 1.9.0
- Update manylinux image to 2026.08.08-1 (for 3.15rc1)
- Add -rc to PYTHON_IMAGE_TAG
- Disable pandas tests if Pandas isn't installed:
### Are these changes tested?
- Yes, tested on a local Linux build using the `ninja-release-python` preset (not maximal)
### Are there any user-facing changes?
No functional changes, other than allowing the user to use Python 3.15.
* GitHub Issue: #48172
### Progress Notes (2026-08-15)
Each platform uses different sources for Python builds, each with different release schedules and paths:
- [manylinux](https://github.com/pypa/manylinux) is prebuilt with latest Python builds in `/opt/python`, so using the latest manylinux build is sufficient. 3.15 wasn't available until beta 1.
- musllinux: Uses Alpine system python built from https://github.com/docker-library/python/pull/1123. Might need to bump ALPINE_LINUX version in [arrow/dev/tasks/python-wheels/github.linux.yml](https://github.com/apache/arrow/blob/e611f48081ec927e18e5bcb72fbd071c15c76b08/dev/tasks/python-wheels/github.linux.yml#L35).
- musllinux free-threaded: uses [astral's builds](https://github.com/astral-sh/python-build-standalone). Edit `python-free-threaded-wheel-musllinux-test-imports.dockerfile` + `-test-unittests.dockerfile`: set the python_patch_version, and set build_date in `compose.yaml`.
- macos: Downloads official build from python.org, see `ci/scripts/install_python.sh`. For early builds, needed to disable `--only-binary` since at least one dependency needed to be built (no 3.15 wheels available yet)... only macos uses --only-binary
- Windows: installs builds using `choco`, see `python-wheel-windows-test-vs2022.dockerfile` and `python-free-threaded-wheel-windows-test-vs2022.dockerfile`
For 3.15.0rc1 builds, some care was needed because certain paths needed 3.15.0 and others needed the full 3.15.0rc1.
Lead-authored-by: Paul Timmins <paul@iqmo.com>
Co-authored-by: paul@iqmo.com <paul@iqmo.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Pure version bump to enable implementing more features.
### What changes are included in this PR?
- Change the DLPack header file and definition.
- Use `raw_data` for `Tensor` similar to `Array` (instead of `mutable_raw_data` that is not available on immutable tensors).
- Strides may no longer be `NULL`
- Add `ExportArrayVersioned` and `ExportTensorVersioned` C APIs
- Support for `copy`
- Support for `max_version` and `copy` in `__dlpack__`
- Add `dl_device` to `__dlpack__` parameters as the spec requires but only CPU supported at the moment
### Are these changes tested?
Yes, with existing tests.
### Are there any user-facing changes?
* GitHub Issue: #41670
Authored-by: AntoinePrv <AntoinePrv@users.noreply.github.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
Converting a Table with an `arrow.uuid` extension column to pandas currently produces a column of `bytes` instead of `uuid.UUID` objects. Without a dedicated conversion path, `Table.to_pandas()` / `Array.to_pandas()` falls back to the storage type (`fixed_size_binary(16)`) and materializes raw bytes. Downstream callers (and tests such as those landing via pandas UUID Parquet coverage) expect `uuid.UUID` objects, matching what `to_pylist()` / `UuidScalar.as_py()` already return.
Note: the original issue suggested this might be specific to Python 3.14, but the same `bytes` result reproduces on Python 3.10-3.14 with current PyArrow - `UuidType` never had a pandas conversion path.
Closes#50312
### What changes are included in this PR?
Convert UUID extension arrays to `uuid.UUID` objects in the C++ pandas conversion path (integrated from @ rok's proposal in rok/arrow#55), rather than relying on a Python `to_pandas_dtype()` / `__from_arrow__` compat layer:
- Construct `uuid.UUID` objects from storage bytes in `helpers.cc` (`UuidFromBytes`) and use that from `arrow_to_pandas.cc`
- Reuse empty kwargs / shared empty tuple constants so conversion does not allocate per element
- Keep `to_numpy(zero_copy_only=False)` on UUID extension arrays returning storage `bytes` (extension to NumPy should follow storage type)
- Add/extend regression tests for pandas round-trip and NumPy behavior
### Are these changes tested?
Yes.
Acceptance criteria:
- [x] Tests added for the UUID to pandas round-trip (and NumPy storage behavior)
- [x] Relevant local / CI Python tests passing for the touched paths
- [x] Follows project style / review feedback (lint fix included)
- [x] No intentional breaking public API change beyond correcting incorrect `bytes` return values to `uuid.UUID` for `to_pandas()`
Before / after evidence (backend; console):
```text
# Before (repro on Python 3.10-3.14 with unfixed PyArrow)
>>> type(result_df.loc[0, "id"])
<class 'bytes'>
# After (with this PR)
>>> type(result_df.loc[0, "id"])
<class 'uuid.UUID'>
```
Also covered by added unit tests in `python/pyarrow/tests/parquet/test_data_types.py` and `python/pyarrow/tests/test_extension_type.py`.
### Are there any user-facing changes?
Yes. `Table.to_pandas()` / `Array.to_pandas()` now returns `uuid.UUID` for `arrow.uuid` columns instead of `bytes`. `to_numpy(zero_copy_only=False)` continues to expose storage `bytes`.
* GitHub Issue: #50312
Authored-by: parker-cassar <parker.cassar@berkeley.edu>
Signed-off-by: Rok Mihevc <rok@mihevc.org>
### Rationale for this change
Fix the return value of `ParquetLogicalType.type` for geometry/geography types
### Are these changes tested?
Yes
### Are there any user-facing changes?
Not apart from the bug fix
* GitHub Issue: #50849
Authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Closes#50801
As title, this exposes RecordBatchReaderSourceNodeOptions to Python. With it, a hash join can build its hash table once and stream the probe side through as a generator-backed RecordBatchReader, releasing each input chunk as it is consumed, instead of materializing the full probe table up front (table_source) or pinning all fragments for the plan's lifetime (dataset scan).
### Are these changes tested?
Yes
### Are there any user-facing changes?
Yeah, users can pass RecordBatchReaderSourceNodeOptions into Declaration, but its not a breaking change
* GitHub Issue: #50801
Lead-authored-by: You-Cheng Lin <106612301+owenowenisme@users.noreply.github.com>
Co-authored-by: You-Cheng Lin <mses010108@gmail.com>
Signed-off-by: Rossi Sun <zanmato1984@gmail.com>
### Rationale for this change
Follow-up to https://github.com/apache/arrow/issues/49232#issuecomment-5105290467 / #49590
### What changes are included in this PR?
Remove warnings from `write_feather()`, `read_feather()`, `read_table()`, and `FeatherDataset`.
Warn **only** on writing with `version=1` / reading Feather V1 file.
Update docs to V1-only and keep IPC migration guide.
Additionally corrected notes like `deprecated as of 24.0.0` to `25.0.0` instead.
`DeprecationWarning` in place of current `FutureWarning`.
### Are these changes tested?
Yes, by CI.
### Are there any user-facing changes?
Yes!
Feather V2 APIs no longer emit deprecation warnings.
Reading/writing the legacy Feather V1 format emits `DeprecationWarning` in place of `FutureWarning`.
* GitHub Issue: #50808
Lead-authored-by: Tadeja Kadunc <tadeja.kadunc@gmail.com>
Co-authored-by: tadeja <tadeja@users.noreply.github.com>
Co-authored-by: Rok Mihevc <rok@mihevc.org>
Signed-off-by: Rok Mihevc <rok@mihevc.org>
### Rationale for this change
The `ceil_temporal`, `floor_temporal`, and `round_temporal` functions currently support date, time, and timestamp inputs, but not duration inputs.
### What changes are included in this PR?
- Add kernel registration for duration values with second, millisecond, microsecond, and nanosecond resolutions
- Support rounding duration inputs using physical units through day
- Treat week as seven physical days for duration inputs
- Reject ambiguous calendar units such as month, quarter, and year
- Reject `calendar_based_origin` for duration inputs
- Add focused C++ tests covering all four duration resolutions, positive and negative values, null propagation, day and week rounding, and unsupported calendar behavior
### Are these changes tested?
Yes.
- `arrow-compute-scalar-temporal-test`: 55 tests passed
- Applicable pre-commit C++ formatting and lint checks passed
### Are there any user-facing changes?
Yes. Users can now pass duration values to `ceil_temporal`, `floor_temporal`, and `round_temporal` for supported physical units.
### AI assistance
I used ChatGPT to help navigate the codebase and draft the initial implementation. I reviewed, revised, and tested the changes locally.
* GitHub Issue: #50395
Authored-by: snigdhachoppac <snigchoppa@gmail.com>
Signed-off-by: Rok Mihevc <rok@mihevc.org>
### Rationale for this change
PyFlightServer keeps a reference towards the Python server via `OwnedRefNoGIL server_`, the Python server also keeps a reference of the C++ `PyFlightServer` creating a cycle that is never freed during the process lifetime.
### What changes are included in this PR?
Create a new `ReleasePythonServerRef` method that is called after any `server.Shutdown` (including at `__exit__`) allowing for the `OwnedRefNoGIL` to be cleared breaking the cycle. This lets normal reference counting free the previously leaked Python object.
### Are these changes tested?
Yes, the newly added tests were failing leaking the references before the fix.
Currently a test demonstrating a leak when not calling server.Shutdown is added for discussion purposes.
### Are there any user-facing changes?
No
* GitHub Issue: #50684
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Resolves [49305](https://github.com/apache/arrow/issues/49305)
`RecordBatchFileReader::CountRows` has existed in Arrow C++ (`cpp/src/arrow/ipc/reader.h`) but
was never bound in Python, so the only way to get the total number of rows of an IPC file was:
```python
num_rows = sum(reader.get_batch(i).num_rows for i in range(reader.num_record_batches))
```
### This findings are done in original opened Issue #49305
That deserializes every record batch just to read its length, which is wasteful and becomes
expensive on remote filesystems.
### What changes are included in this PR?
Adds `RecordBatchFileReader.count_rows()`:
```python
with pa.ipc.open_file(source) as reader:
reader.count_rows()
```
Three changes:
* `python/pyarrow/includes/libarrow.pxd`: declare `CResult[int64_t] CountRows()` on
`CRecordBatchFileReader`, which was the missing piece.
* `python/pyarrow/ipc.pxi`: add `count_rows()` to `_RecordBatchFileReader`, released GIL around
the call, with the same closed reader guard used by the existing `stats` property
* `python/pyarrow/tests/test_ipc.py`: tests.
On the naming: `count_rows()` follows the C++ method and is consistent with the existing
`count_rows()` on `Dataset`, `Scanner` and `Fragment`.
To be precise about the benefit, since the issue describes it as reading the count from the
metadata: the C++ implementation still walks every block, but reads only each record batch's
flatbuffer message header to pick up its length, and never touches the data buffers. So this is
a reduction in bytes read rather than in the number of reads, and the gain shows up on remote
filesystems and on files with large batches rather than in a local in memory benchmark.
This is only added to the file reader. The stream reader has no footer and cannot count rows
without consuming the stream.
### Are these changes tested?
Yes, two tests in `python/pyarrow/tests/test_ipc.py`:
* `test_file_count_rows`: count matches the sum of the written batch lengths, and counting does
not consume the reader (count, `read_all()`, count again).
* `test_file_count_rows_no_batches`: a file with a schema but no batches counts 0.
Locally `test_ipc.py` passes (72 tests) and `test_feather.py` passes (83 passed, 8 skipped,
1 xfailed), the latter because the feather reader sits on the same file reader. The docstring
example was run and produces the output shown.
### Are there any user-facing changes?
Yes, a new public method `RecordBatchFileReader.count_rows()`. No existing behaviour changes.
### AI usage disclosure
I used Claude Code to locate the unbound C++ method and the place where the declaration was
missing, and to draft the binding, the docstring and the tests. I reviewed the result, rebuilt
PyArrow locally, ran the test suites quoted above, and checked the C++ implementation of
`CountRows` myself to confirm what it actually does before describing the benefit here.
* GitHub Issue: #49305
Authored-by: Guja <127162872+GujaLomsadze@users.noreply.github.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
The Arrow GDB pretty-printers support `Decimal128` and `Decimal256`, but do not support `Decimal32` or `Decimal64`.
### What changes are included in this PR?
* Add GDB pretty-printer support and tests for `Decimal32` and `Decimal64`.
* Remove the unused `max_type_id` variable.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
* GitHub Issue: #50660
Authored-by: fenfeng9 <fenfeng9@qq.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
An unsigned dictionary index type is silently replaced by the signed one of the same width:
```python
>>> pa.array(["a", "b"], type=pa.dictionary(pa.uint32(), pa.string())).type
dictionary<values=string, indices=int32, ordered=0>
```
It also makes `pa.chunked_array(values, dict_type)` fail with `Array chunks must all be same type`, since the chunk built internally comes back a different type than requested.
`DictionaryBuilderCase::CreateFor()` reduces the requested index type to its byte width and hands that to `AdaptiveIntBuilder`, so everything but the width is dropped and the type is rebuilt from what the indices builder reports, which is always signed.
@ jorisvandenbossche suggested `AdaptiveUIntBuilder` for unsigned types. That returns a different builder class, which `util/converter.h`, `json/from_string.cc` and the R binding all cast to `DictionaryBuilder<T>`, and `TestDictionaryUnifier.ChunkedArrayNestedDict` hits a `checked_pointer_cast` DCHECK. So this keeps one builder class and just reports the requested signedness. Indices are non-negative and same-width signed and unsigned types have the same layout, so it is value-preserving and free, and the width stays adaptive.
### What changes are included in this PR?
`DictionaryBuilderBase` and its `NullType` specialization record the requested signedness and map the signed index type to the unsigned one of the same width in `type()`, `FinishInternal()` and `FinishDelta()`; `CreateFor()` passes it through. The mapping helper lives in `builder.cc`, not the header. `util/converter.h`, `python_to_arrow.cc` and `r_to_arrow.cpp` are untouched, so R is fixed rather than broken.
uint64 indices also convert to pandas now, mapped to int64 codes. That is safe because the indices are bounds-checked below the dictionary length, so they never reach the int64 limit; `WriteIndicesUniform` handles uint64 like it already handles uint32.
### Are these changes tested?
Six tests in `array_dict_test.cc` cover the distinct paths: type and finish preservation across signed and unsigned, the width adapting while staying unsigned, the `NullType` builder, `FinishDelta`, the supplied-dictionary constructor and the exact-index builder. Three parametrized tests in `test_array.py` cover the eight index types, the width adapting and the uint64-to-pandas conversion, and `test_dictionary_with_pandas` now checks that conversion instead of the old error.
A `uint8` grows to `uint16` once more than 128 distinct values are added and stays unsigned; the underlying builder is signed, so it widens after 128 rather than the 256 a real uint8 could hold. `arrow-array-test`, `arrow-ipc-read-write-test`, `arrow-compute-scalar-cast-test`, `arrow-c-bridge-test` and the pyarrow `test_array` and `test_pandas` suites pass.
### Are there any user-facing changes?
`pa.array` and `pa.chunked_array` return the requested unsigned index type. `dictionary(uint64(), ...)` produces real uint64 indices, and converting such an array to pandas now works where it used to raise. `take`, `filter`, `cast`, `dictionary_decode`, IPC and scalar access all keep the index type.
* GitHub Issue: #37476
Authored-by: mkzung <103102868+mkzung@users.noreply.github.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
This is part of the work for #35460. Back in January I started working on this change-set, but after discussions came up regarding the use of the `simdjson::dom` vs `simdjson::ondemand` front-end the development went stale. I saw that @ Reranko05 started the work on #35460 and already made great progress, well done!
I still wanted to put up this change-set as a possible view-point, but feel free to disregard @ Reranko05 if you already started your own work.
### What changes are included in this PR?
This changes `from_string.cc` to use simdjson instead of RapidJSON.
### Are there any user-facing changes?
Yes, the API has changed to no longer allow non-utf8 compatible strings. `simdjson` does not support non-compliant inputs and does not intent to support it in the future [[1]](https://github.com/simdjson/simdjson/issues/1691#issuecomment-895286180) [[2]](https://github.com/simdjson/simdjson/blob/8e6bac94877f2d3d026000d36ce81e0aaf38d26f/doc/basics.md?plain=1#L3701) [[3]](https://github.com/simdjson/simdjson/discussions/2125)
All call-sites in tests only used a transformation to non-utf8 json as an intermediate result to easily go from `string -> Array`, so I changed these to use different helper methods instead.
Another difficult problem arises in the handling of `NaN` and `Inf` literals in inputs. `simdjson` follows the json standard, whereas `RapidJSON` implemented an extension where the following is a valid input:
```
{"x": Inf, "y": -Inf, "z": NaN, "n": Infinity, "m": -Infinity}
```
We need to decide whether to be breaking here or not. This PR now contains a non-breaking version of the change. This can be quite elegantly added by using the public `ondemand` api's function to retrieve the raw token string in the `ConvertNumber` function.
Alternatively, we can wait for the next `simdjson` major release, which will ship with support for a `SIMDJSON_ENABLE_NAN_INF` compilation flag as per [this pull request](https://github.com/simdjson/simdjson/pull/2696). (this was only merged to simdjson main)
### Are these changes tested?
Yes
* GitHub Issue: #50627
Authored-by: Alexander Taepper <alexander.taepper@gmail.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Resolves#49046
`python/scripts/test_imports.py` and `python/scripts/test_leak.py` are leftovers from an older
test layout. They are not referenced by CI, packaging or the pytest suite, and are not
documented anywhere. Their `test_`-prefixed names have also caused accidental pytest collection
in the past (ARROW-1033). This follows the suggestion on the issue to remove them.
Deletes `python/scripts/test_imports.py` and `python/scripts/test_leak.py`.
The other two scripts in that directory are untouched and still in use:
`run_emscripten_tests.py` (invoked from `ci/scripts/python_test_emscripten.sh`) and
`update_stub_docstrings.py`.
The issue description raised the option of keeping `test_leak.py` as a documented snippet for
checking leaks. This PR removes it, following the later comment. Happy to restore and document
it under the developer docs instead if reviewers prefer.
### Are these changes tested?
No new tests, this only removes files that nothing references. I grepped `ci/`, `dev/`,
`python/setup.py`, `python/pyproject.toml` and `python/MANIFEST.in` for references and found
none. The only remaining mention in the repository is a historical `CHANGELOG.md` entry for
ARROW-1033, left as is.
### Are there any user-facing changes?
No.
### AI usage disclosure
I used Claude Code to grep the repository for remaining references to the two scripts and to
help draft this description. The change itself is a two file deletion, and I reviewed both the
search results and the diff.
* GitHub Issue: #49046
Authored-by: Guja <127162872+GujaLomsadze@users.noreply.github.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Fix#50597. Nightly Crossbow job `test-conda-python-emscripten` sometimes finishes with success like [2026-07-18 Docker Test conda-python-emscripten](https://github.com/ursacomputing/crossbow/actions/runs/29628907267/job/88038728709#step:6:7261)
But often hangs like [2026-07-19 Docker Test conda-python-emscripten](https://github.com/ursacomputing/crossbow/actions/runs/29672419322/job/88153721397#step:6:6851)
and [2026-07-20 Docker Test conda-python-emscripten](https://github.com/ursacomputing/crossbow/actions/runs/29715959996/job/88269203693#step:6:6846)
```
driver.load_arrow()
...
selenium.common.exceptions.TimeoutException: Message: script timeout
```
Additionally, main nightly started failing during `snappy_ep` configure with CMake 4.4: `CMake Error: The warning category "linkflags" is not known.`
### What changes are included in this PR?
import `TimeoutException`, introduce 300s timeout + 3 attempts around load_arrow() and full Chrome restart on timeout,
ignore harmless `BrokenPipeError` / `ConnectionResetError` when Chrome has to be killed mid-download for retry.
Also remove the redundant standalone `-Wno-error=linkflags` CMake argument from Snappy's Emscripten configure arguments. The flag remains in `CMAKE_SHARED_LINKER_FLAGS`
### Are these changes tested?
Yes, locally by `docker compose run --rm -e SETUPTOOLS_SCM_PRETEND_VERSION=26.0.0.dev0 conda-python-emscripten`.
Example test run hit the PyArrow load timeout once, restarted Chrome, and then completed successfully.
### Are there any user-facing changes?
No.
* GitHub Issue: #50597
Authored-by: Tadeja Kadunc <tadeja.kadunc@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Pandas 1.4 fixed the following:
- https://github.com/pandas-dev/pandas/issues/43892
Which made us special case microseconds extracting those from our `MonthDayNano` and manually building `microseconds`.
`DateOffset` can be created with the following:
```python
DateOffset(years=1, months=1,
days=1, seconds=1, microseconds=1,
minutes=1, hours=1, weeks=1, nanoseconds=1)
```
We currently only roundtrip Month, Day, Microseconds and Nano due to our MonthDayNano data type. We special cased microseconds but we were not special casing years, seconds, minutos, hours or weeks.
As the initial issue that made us special case has been solved upstream we can stop roundtripping microseconds and just return nanoseconds simplifying our codebase.
### What changes are included in this PR?
Remove the unnecessary code to retrieve microseconds from nanoseconds.
Update test as microseconds won't be returned, only nanoseconds.
### Are these changes tested?
Yes via CI
### Are there any user-facing changes?
Yes, this is a change of the UX.
* GitHub Issue: #50464
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Follow-up to #50543. `test_categorical_order_survives_roundtrip` constructs a
`pd.Categorical` with value "a" not in categories `["b","c","d"]`. pandas 3.0.3
raises a deprecation for out-of-category values, which #50543's filterwarnings
config now turns into an error - failing nightly wheels across platforms.
### What changes are included in this PR?
Replace the out-of-category `"a"` value with an explicit `None`, preserving the
test's intent (ordered categorical with non-sorted categories survives a Parquet
roundtrip, ARROW-6302) plus a null.
### Are these changes tested?
Yes. Reproduced the failure with pandas 3.0.3, confirmed the test passes after
the change; all categorical tests in the module pass.
### Are there any user-facing changes?
No.
Closes#50579.
* GitHub Issue: #50579
Authored-by: Guja <127162872+GujaLomsadze@users.noreply.github.com>
Signed-off-by: Rok Mihevc <rok@mihevc.org>
### Rationale for this change
`pa.Array.to_pylist()` converts one element at a time through `Array::GetScalar` plus a Python `Scalar` wrapper; for list types each row additionally allocates a Python `Array` wrapper for the row's values slice and a fresh generator before recursing per element. A `sample` profile shows ~20% of runtime in CPython GC (triggered by the per-row GC-tracked allocations), ~25% in `GetScalar`, and only ~7% doing the useful work of creating the output objects — making `to_pylist` several times slower than converting via `to_pandas()` and back, and ~24x slower than `ndarray.tolist()` for plain int64. Details in #50326; this hit Apache Spark's Arrow-serialized Python UDFs (apache/spark#56940, apache/spark#56943).
### What changes are included in this PR?
Following review feedback, this adds a general scalar-free conversion mechanism instead of per-type `to_pylist` overrides:
- `Array` gains `cdef object _getitem_py(self, int64_t i)`, returning `self[i]` as a Python object. The base implementation is `GetScalar` + `Scalar.as_py`, so any type without a specialization behaves exactly as today (dates, times, timestamps, durations, decimals, dictionary, extension, unions, views, ...).
- The baseline `Array.to_pylist` becomes a single loop over `_getitem_py`. `maps_as_pydicts != None` keeps the Scalar-based path, since map→dict conversion has per-entry duplicate-key semantics.
- Specializations avoid all per-element Scalar and per-row Array-wrapper allocation:
- integers and floats (a `type_id` switch on `NumericArray`; date/time/timestamp subclasses fall through to the exact base),
- boolean,
- string/binary and large variants (`GetValue` + `PyUnicode_DecodeUTF8` / `PyBytes_FromStringAndSize`, matching `str(buf, 'utf8')` / `to_pybytes()` exactly),
- list/large_list/fixed_size_list (each row's list is built from the child's `_getitem_py` over the offset range; the wrapped child is cached on the parent array),
- map (association list of key/value tuples, matching `MapScalar.as_py`),
- struct (one dict per row; duplicate field names fall back to the Scalar path so they raise `ValueError` like `StructScalar.as_py`).
Nested types compose without any per-row wrappers. `ChunkedArray.to_pylist`, `Table.to_pylist` and `ListScalar.as_py` delegate here and speed up automatically. Follow-up candidates: string/binary views, run-end-encoded, dictionary, a fast path for date32.
Benchmarks (macOS arm64, M4 Max):
| benchmark | before | after | speedup |
|---|---|---|---|
| flat `int64` with nulls (4M) | 0.39 s | 0.028 s | 14x (~7 ns/element, on par with `ndarray.tolist`) |
| flat `string` (4M) | 0.83 s | 0.06 s | 14x |
| `list<string>` (2M rows) | 1.93 s | 0.46 s | 4.2x |
| `list<list<int32>>` (1M rows) | 2.10 s | 0.40 s | 5.2x |
| `struct<int64,string>` (1M rows) | 0.91 s | 0.07 s | 13x |
| `map<string,int64>` (1M rows) | 2.77 s | 0.74 s | 3.8x |
### Are these changes tested?
`test_to_pylist_bulk_paths` (added here) compares against the per-scalar conversion with exact element types for representative arrays including sliced views. Additionally verified with a randomized differential test against `[x.as_py() for x in arr]` with exact-type equality: all integer widths (incl. values beyond 2^62), floats (NaN/inf), boolean, string/binary (+large, multibyte), all list kinds, nested lists, struct (incl. empty struct, duplicate-field-name `ValueError`), map (incl. strict-mode duplicate-key `KeyError`), dictionary/null fallbacks, sliced/chunked arrays, and both `maps_as_pydicts` modes — no differences. `pytest test_array.py test_scalars.py test_convert_builtin.py test_table.py test_types.py`: 1295 passed.
### Are there any user-facing changes?
No behavior changes, only performance.
* GitHub Issue: #50326
This pull request and its description were written by Isaac.
Lead-authored-by: Liang-Chi Hsieh <liangchi.hsieh@databricks.com>
Co-authored-by: Isaac
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
We currently leak the list and items in the case of an iterator raising an Exception and size is present.
### What changes are included in this PR?
Use `OwnedRef` instead of relying on manual `Py_DECREF` in order to avoid exception scenario returning early in `RETURN_IF_PYERROR`.
### Are these changes tested?
Yes, I've created a new test that fails previously to the fix and passes after the fix
### Are there any user-facing changes?
No
* GitHub Issue: #50591
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
`table.cast(table.schema)` errored or segfaulted when the schema contained a struct with an extension-typed field whose storage is a list.
This cast takes a zero-copy view, and `ViewDataImpl::MakeDataView` found children using `type->fields()`, which is never populated for extension types since their structure is in `storage_type()`. The same issue exists in `AccumulateLayouts`. The storage's child data gets dropped and leaves an invalid array that downstream code throws/crashes on.
This bug is similar to #37669, except that the fix for #37669 addressed casting *to* an extension with nested storage but not this particular struct-wrapped view path.
### What changes are included in this PR?
- `cpp/src/arrow/array/data.cc` - `AccumulateLayouts` and `ViewDataImpl::MakeDataView` now recurse into `storage_type()->fields()` for `ExtensionType`, instead of the extension type's empty `fields()`.
- `cpp/src/arrow/array/array_view_test.cc`
- `python/pyarrow/tests/test_extension_type.py`
### Are these changes tested?
Yes,
- `cpp/src/arrow/array/array_view_test.cc` - Two tests for viewing an extension array with nested storage for fixed-size and variable-size lists.
- `python/pyarrow/tests/test_extension_type.py` - A test casting a table / struct containing an extension type with list-based storage to its own schema.
- Also, all reproducers from the linked issue were checked against this fix, including the segfault case.
### Are there any user-facing changes?
No API changes. Operations that previously raised errors or crashed relating to casting/viewing a struct or table with an extension type with nested storage now succeed.
* GitHub Issue: #37004
Authored-by: Connor Simms <connorvsimms@gmail.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
resolve#39961
### What changes are included in this PR?
When writing a dataset with CsvFileFormat configured with a custom ParseOptions delimiter (e.g. delimiter=">"), the output CSV still used the default "," delimiter. This was because:
- C++ CsvFileFormat::DefaultWriteOptions() always created WriteOptions::Defaults() with delimiter=',', ignoring the parse_options.delimiter stored on the format object.
- Python CsvFileFormat.make_write_options() unconditionally overwrote the C++ write options with a fresh WriteOptions(**kwargs), discarding any C++-side default that might have been set.
Fix the C++ side by propagating parse_options.delimiter into the write options in DefaultWriteOptions(). Fix the Python side by reading the delimiter from the C++ default when the caller does not explicitly specify one, preventing the overwrite from losing the propagated value.
### Are these changes tested?
Yes
### Are there any user-facing changes?
No
* GitHub Issue: #39961
Lead-authored-by: egolearner <lijiliang@outlook.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
`scikit-build-core` 1.0 added `[tool.scikit-build.sdist.force-include]` in order to force the inclusion of files out of the source tree.
This allows us to remove our workaround with a custom build-backend to solve adding our license files from the main Arrow repository.
### What changes are included in this PR?
Add configuration to force-include `LICENSE.txt` and `NOTICE.txt` and remove custom build-backend.
### Are these changes tested?
Yes, CI and local
### Are there any user-facing changes?
No
* GitHub Issue: #50493
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Towards https://github.com/apache/arrow/issues/50398
closes https://github.com/apache/arrow/issues/50431
### What changes are included in this PR?
This is an agent generated PR that makes `FlightError` a non-cdef subclass of `Exception` as Cython does not support building with the Limited API and cdef classes that inherit from Python classes.
It appears before a `cdef CStatus to_status` was defined on each subclass to return a `MakeFlightError` per flight error flavor. This was pull out to a dedicated `_flight_error_to_status` function instead.
### Are these changes tested?
Yes (via existing tests?)
### Are there any user-facing changes?
No
* GitHub Issue: #50398
* GitHub Issue: #50431
Authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com>
Signed-off-by: David Li <li.davidm96@gmail.com>
### Rationale for this change
pandas 3.0.0 deprecated constructing or casting to a `Categorical` with
non-NA values that are not present in the specified `dtype.categories`.
This surfaces as DeprecationWarnings in the PyArrow
test suite when run against pandas >= 3.0 (GH-49255), in test_category
and test_category_implicit_from_pandas.
### What changes are included in this PR?
- test_category: remove the `cat_strings_with_na` column (relied on
silent 'qux' -> NaN; redundant, cat_strings already carries nulls).
- test_category_implicit_from_pandas: use explicit None
(['a','b',None]) instead of out-of-category 'c'.
### Are these changes tested?
Yes - test-only. Verified no DeprecationWarning under pandas 3.0.3.
### Are there any user-facing changes?
No.
Closes#49255.
* GitHub Issue: #49255
Authored-by: GujaLomsadze <lomsadze.guja@gmail.com>
Signed-off-by: Rok Mihevc <rok@mihevc.org>
### Rationale for this change
Fix#50424 `Python Emscripten Crossbow` job with multiple failing points
A) failing to install Chrome - `Requested Chrome major 148, but apt repo currently publishes 150.0.7871.100`,
B) and underlying Abseil failure - `error: expected '(' before 'ABSL_INTERNAL_CONSTEXPR_SINCE_CXX17'` [logB](https://github.com/ursacomputing/crossbow/actions/runs/28963702667/job/85941369345#step:6:2995)
C) pytest 8.1.1 failing on three tests with the message - `E TypeError: importorskip() got an unexpected keyword argument 'exc_type'` [logC](https://github.com/ursacomputing/crossbow/actions/runs/28972366846/job/85971005157#step:6:6354)
The remaining intermittent issue be best dealt with in a follow up PR:
D) intermittent `loadPyodide` / Selenium timeout hang - `selenium.common.exceptions.TimeoutException: Message: script timeout` [logD](https://github.com/ursacomputing/crossbow/actions/runs/28997397611/job/86049819728#step:6:6856) and [hang log with debug](https://github.com/ursacomputing/crossbow/actions/runs/29015193964/job/86108340808#step:6:7248)
### What changes are included in this PR?
A) Changed `chrome_version: "148"` to `chrome_version: "latest"`
B) Disabled zlib for protobuf build of the host in order to avoid conda include path overriding vendor Abseil headers.
C) Added work-around for current Pyodide's bundled pytest version 8.1.1 which lacks `importorskip(..., exc_type=...)`
### Are these changes tested?
Yes, test-conda-python-emscripten job completed successfully on rerun. Some earlier runs hit the intermittent Chrome/Selenium timeout mentioned above.
### Are there any user-facing changes?
No.
* GitHub Issue: #50424
Authored-by: Tadeja Kadunc <tadeja.kadunc@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
#49511 : `Table.join_asof` outputs a non-deterministic number of chunks, so the printed table sometimes appears as one chunk and sometimes as several (rarely).
`default_memory_pool` prints `bytes_allocated=0` except when a preceding (failing) doctest allocated from the process-global pool (also rarely).
### What changes are included in this PR?
Addition of `.combine_chunks()` to `Table.join_asof` (and removal of two trailing spaces) in `table.pxi`
Change to `bytes_allocated=...` from `bytes_allocated=0` in `memory.pxi`
### Are these changes tested?
Yes, local doctest tests OK.
### Are there any user-facing changes?
No, just docstring changes.
* GitHub Issue: #49511
Authored-by: Tadeja Kadunc <tadeja.kadunc@gmail.com>
Signed-off-by: Rok Mihevc <rok@mihevc.org>
### Rationale for this change
[Pandas 2.0.0](https://pypi.org/project/pandas/2.0.0/) was released on April 3rd 2023.
The last version of Pandas 1 was [Pandas 1.5.3](https://pypi.org/project/pandas/1.5.3/) and it was released on January 19th 2023. That version only supports up to Python 3.11.
We are currently supporting up to Pandas 1.5.2 (first version supporting 3.11).
On the issue we discussed to bump our Pandas support to 2.0.0 but given 2.0.3 is the latest in the 2.0.x, was released on June 2023, only supports up to Python 3.11 and bumping to it allows use to improve even further some of the conditionals we are bumping to Pandas >= 2.0.3
### What changes are included in this PR?
Update minimal support check to Version(2.0.3) and remove all test skips, conditionals and scenarios where we are checking for Pandas < 2.0.3.
We found a couple of minor things
### Are these changes tested?
Yes via CI
### Are there any user-facing changes?
Yes, Pandas < 2 won't be supported.
* GitHub Issue: #48808
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Python 3.10 is End of Support on October 2026. We can drop support on PyArrow 26.0.0.
See: https://endoflife.date/python
### What changes are included in this PR?
Remove support for 3.10. Update minimum required Python version and update some CI jobs to use the minimum Python supported version.
Some CI jobs using default Python 3.10 (Ubuntu 20.04) have been bumped.
### Are these changes tested?
They will be tested via CI and Archery
### Are there any user-facing changes?
Yes in terms of not supporting older versions of Python. This will be a breaking change for some users but not for the API itself
* GitHub Issue: #50410
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
`test_fastparquet_cross_compatibility` is not exercised and when exercised with latest versions it fails because it's not compatible with pandas 3 and it won't be as the project is retired, see:
- https://github.com/dask/fastparquet/pull/973
### What changes are included in this PR?
Removal of fixture to mark test as `fastparquet` and remove single test that still required fastarquet.
### Are these changes tested?
Yes and no. CI is exercised as expected but `fastparquet` tests weren't exercised on CI in the first place.
### Are there any user-facing changes?
No
* GitHub Issue: #37853
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
There is currently no method to convert Arrow Table to Arrow Tensor (conversion from columnar format to a contiguous block of memory). This work is a continuation of `RecordBatch::ToTensor` work, see https://github.com/apache/arrow/issues/40058.
### What changes are included in this PR?
This PR:
- implements `Table::ToTensor` conversion
- adds bindings to Python
- adds benchmarks in C++
- removes the code in `RecordBatch::ToTensor` and uses the Table implementation (`RecordBatch::ToTensor` benchmarks checked)
### Are these changes tested?
Yes, in C++ and Python.
### Are there any user-facing changes?
No, it is a new feature.
* GitHub Issue: #40062
Lead-authored-by: AlenkaF <frim.alenka@gmail.com>
Co-authored-by: Alenka Frim <AlenkaF@users.noreply.github.com>
Co-authored-by: tadeja <tadeja@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Rok Mihevc <rok@mihevc.org>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
Casting to `binary_view` or `string_view` could leave a null variadic buffer slot when all values were inline. This could happen for casts from `binary`, `large_binary`, `string`, `large_string`, and `fixed_size_binary`.
The C Data Interface exporter reads every variadic buffer to get its size. Because of that, exporting such an array could crash, for example through PyArrow `_export_to_c`.
Validation also passed for these arrays. For all-inline view arrays, validation never needed to read an out-of-line data buffer.
### What changes are included in this PR?
This PR fixes the cast kernels so all-inline view arrays do not keep a null variadic buffer slot.
It also makes validation reject null variadic buffer slots, and makes C Data export return an error instead of crashing.
C++ and Python regression tests cover the cast, validation, and export paths.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
**This PR contains a "Critical Fix"** Exporting an all-inline view array through the C Data Interface could crash the process while using only public APIs.
* GitHub Issue: #49740
Authored-by: fenfeng9 <fenfeng9@qq.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Arrow C++ is deprecating the Feather reader/writer (#49231) so we should update the Python functions too. Feather V2 is exactly the Arrow IPC file format, making the separate `pyarrow.feather` module redundant. Users should migrate to `pyarrow.ipc`.
### What changes are included in this PR?
- Added `FutureWarning` deprecation warnings to all four public APIs in `pyarrow.feather`: `write_feather()`, `read_feather()`, `read_table()`, and `FeatherDataset`
- Extracted `_read_table_internal()` so that `read_feather()` calling `read_table()` internally does not produce double warnings
- Updated `FeatherDataset.read_table()` to use the internal function for the same reason
- Added `.. deprecated:: 24.0.0` directives to all four public API docstrings
- Added module-level `pytestmark` filter in `test_feather.py` to suppress warnings in existing tests
- Added 5 new tests verifying deprecation warnings are emitted correctly (including a no-double-warning test)
- Added `@ pytest.mark.filterwarnings` to 5 test functions in `test_dataset.py` that use feather as a utility
- Updated `feather.rst` with a deprecation notice and migration guide (including a note that `pyarrow.ipc` does not compress by default unlike `feather.write_feather`)
- Updated `formats.rst` to mark the Feather API section as deprecated
### Are these changes tested?
Yes. Five new tests verify that each deprecated function emits exactly one `FutureWarning`, and existing tests continue to pass with module/function-level warning filters.
### Are there any user-facing changes?
Yes — calling `pyarrow.feather.write_feather()`, `read_feather()`, `read_table()`, or `FeatherDataset()` now emits a `FutureWarning` directing users to `pyarrow.ipc` equivalents. Existing functionality is unchanged.
* GitHub Issue: #49232
Lead-authored-by: Piyush Kanti Chanda <piyush.chanda@databricks.com>
Co-authored-by: Isaac
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
Arrow has `atan2` and `logb` but no `hypot`. Computing a Euclidean norm currently needs `sqrt(add(multiply(x, x), multiply(y, y)))`, which can overflow at the `multiply` step even when the result is representable.
### What changes are included in this PR?
Add a `hypot` kernel backed by `std::hypot` (float32/float64, with integer/decimal inputs promoted to float64), the `Hypot()` C++ function, a `FunctionDoc` and `compute.rst` entry, and a Python expression test. `pc.hypot` is auto-exposed from the registry.
### Are these changes tested?
Yes, C++ tests for the basic cases, null/NaN/Inf handling, and an overflow case confirming it doesn't fall back to a naive `sqrt(x*x + y*y)`.
### Are there any user-facing changes?
Yes, a new `hypot` function in C++ and Python. Additive only.
* GitHub Issue: #50197
Lead-authored-by: Shrivas Shankar <shrivasshankar2019@gmail.com>
Co-authored-by: Shrivas Shankar <86219405+shrivasshankar@users.noreply.github.com>
Co-authored-by: Rok Mihevc <rok@mihevc.org>
Signed-off-by: Rok Mihevc <rok@mihevc.org>
### Rationale for this change
On the hypothesis tests we treat Warnings as errors and the job has been failing with the latest Pytest version (9.1.0). This is related to:
- https://github.com/pytest-dev/pytest/issues/13409
With the latest pytest using generators for parametrization has been deprecated.
### What changes are included in this PR?
Change the generator that was causing the failures and return a list of supported_factories instead.
### Are these changes tested?
Yes both locally and the archery test that was failing.
### Are there any user-facing changes?
No
* GitHub Issue: #50277
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Constructing a fixed-shape-tensor array from a list of individual ndarrays only
worked when each element was 1-D; ≥2-D elements failed with
`ArrowInvalid: Can only convert 1-dimensional array values`. The only workaround
was stacking the list into a single ndarray and using
`FixedShapeTensorArray.from_numpy_ndarray`.
### What changes are included in this PR?
The C++ list converter `PyListConverter::AppendNdarray` now accepts
multi-dimensional ndarray elements for **fixed-size lists** (the storage of a
fixed-shape tensor) by flattening them in C order. The fixed-size-list builder
still validates that the flattened length matches the list width, so wrong sizes
error cleanly. Variable-sized lists remain restricted to 1-D values to avoid
ambiguity. As a side benefit, plain `fixed_size_list` also accepts
multi-dimensional ndarray elements now.
### Are these changes tested?
Yes:
- `test_tensor_array_from_list_of_ndarrays` — construction from 2-D and 3-D
ndarrays, null handling, storage parity with `from_numpy_ndarray`, and the
size-mismatch error, across `int8`/`int64`/`float32`.
- `test_fixed_size_list_from_multidim_ndarray` — plain `fixed_size_list` from
multi-dim arrays, plus a check that variable-sized lists still reject 2-D.
### Are there any user-facing changes?
Yes — `pa.array([multi-dim ndarrays], type=fixed_shape_tensor(...))` (and the
same for `fixed_size_list`) now works instead of raising. Existing 1-D behavior
and variable-sized-list behavior are unchanged.
Scoped to construction only; the reverse `to_numpy` shape-preservation also
raised in the issue is intentionally left as a separate follow-up.
* GitHub Issue: #49644
Lead-authored-by: Samuel Aboderin <aboderinseun01@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Rok Mihevc <rok@mihevc.org>
### Rationale for this change
Since 24.0, pyarrow ships annotations and `py.typed` marker while the type stubs are still incomplete. This makes type checkers trust the partial stubs and report false errors in downstream code (e.g. Module has no attribute "all" for `pyarrow.compute`). Some type checkers also consume bundled `.pyi` files even without `py.typed`, so the stubs need to be withheld from wheels for now. See #49831.
### What changes are included in this PR?
This temporarily omits both `pyarrow/py.typed` and the bundled `pyarrow-stubs` / `.pyi` files from built wheels until the stubs are complete. Wheel-content validation now asserts that neither `py.typed` nor `.pyi` files are present, and wheel build scripts no longer request stub docstring injection while stubs are not installed.
### Are these changes tested?
Wheel-content validation has been updated to check the intended absence of `py.typed` and `.pyi` files.
### Are there any user-facing changes?
Type checkers no longer pick up pyarrow's incomplete stubs from wheels.
* GitHub Issue: #49831
Authored-by: Rok Mihevc <rok@mihevc.org>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
See #38584 for original PR. Will be quoted for this PR description.
### Rationale for this change
> support multi sortkey nulls first.
```
order by i nulls first, j, k nulls first;
```
> The current null sorting only supports all sortkeys, not a certain sortkey, so NullPlacement is extended to the SortKey field. Since the underlying framework is very well written, when modifying this function, you only need to pass the null_placement of each SortKey in. That’s it.
### What changes are included in this PR?
> 1.SortKey structure, NullPlacemnt transfer logic, sorting logic and Ording related, test related
> 2.Substriait related.
> 3.c_glib related.
> 4.SelectK related.
> 5.RankOptions related.
### Are these changes tested?
> yes, I changed the code inside vector_sort_test.cc and performed additional tests.
### Are there any user-facing changes?
> yes, pg database include null sorting of multiple sort keys.
**This PR includes breaking changes to public APIs.** (If there are any breaking changes to public APIs, please explain which changes are breaking. If not, you can remove this.)
I amended the original PR to be less breaking in public APIs.
Still Ordering, SortOptions, RankOptions, and RankQuantileOptions now accept a `std::optional<NullPlacement>` instead of NullPlacement, which did lead to some changes in downstream APIs and bindings. ~I also need some help with fixing the `c_glib` bindings.~
* GitHub Issue: #38558
Lead-authored-by: Alexander Taepper <alexander.taepper@gmail.com>
Co-authored-by: light-city <455954986@qq.com>
Co-authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Some of our CI jobs are failing due to a breaking change on pytest 9.1:
https://github.com/pytest-dev/pytest/pull/14293
### What changes are included in this PR?
Add `exc_type=ImportError` to the `pytest.importorskip("pyarrow.*)` calls
### Are these changes tested?
Via CI
### Are there any user-facing changes?
No
* GitHub Issue: #50176
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
PyArrow adds a `__inex_level_i__` column to the PyArrow table if the pandas dataframe has an unnamed index it wants to preserve. Currently that creates a duplicate, if such a column already exists.
### What changes are included in this PR?
Bumping the integer number in the generated column in order to not get any duplicates.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
* GitHub Issue: #46179
Authored-by: AlenkaF <frim.alenka@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Fix : https://github.com/apache/arrow/issues/50072
### Rationale for this change
The `replace_with_mask` compute function in PyArrow currently has no Python-level tests for regular use cases.
This PR adds test cases for standard usage of `replace_with_mask` to ensure the function works correctly.
### What changes are included in this PR?
Added parameterized test cases in `python/pyarrow/tests/test_compute.py`
### Are these changes tested?
Yes, Manually
### Are there any user-facing changes?
No user-facing changes.
* GitHub Issue: #50072
Authored-by: Ankit.Ahlawat@ibm.com <Ankit.Ahlawat@ibm.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
Sliced union arrays could report incorrect results from `count`.
For sliced union arrays, `span.GetValues<int8_t>(1)` and `span.GetValues<int32_t>(2)` already return offset-adjusted pointers. The logical null count path then indexed those pointers with `span.offset + i`, effectively applying the parent offset twice.
The sparse union path also checked child nullness with `i` instead of `span.offset + i`.
### What changes are included in this PR?
* Fixes logical null counting for sliced sparse and dense union arrays.
* Add C++ and Python regression tests.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
* GitHub Issue: #50113
Authored-by: fenfeng9 <fenfeng9@qq.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Sliced sparse unions could report incorrect nullness during element access.
The type id lookup used the union offset, but the child null check still used the slice-relative index.
### What changes are included in this PR?
* Updates `IsNullSparseUnion` and `ArraySpan::IsNullSparseUnion` in `data.cc` to use the parent offset when checking child nullness.
* Add C++ and Python regression tests.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
* GitHub Issue: #50105
Authored-by: fenfeng9 <fenfeng9@qq.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Fixes#50037
### What changes are included in this PR?
Add `gc.collect()` before `prior_allocation = pa.total_allocated_bytes()` in `test_table_uses_memory_pool`
and also add to `_check_to_pandas_memory_unchanged` helper with the same pattern.
Same fix as #44793
### Are these changes tested?
Not reproduced locally yet.
### Are there any user-facing changes?
No.
* GitHub Issue: #50037
Authored-by: Tadeja Kadunc <tadeja.kadunc@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
When adding support to Python 3.14 the musllinux wheels failed with newer versions of the musllinux quay.io image. In order to prepare for the next Python 3.15 release we will require newer base images and we should fix the issues we originally found.
### What changes are included in this PR?
- Update Manylinux and Musllinux quay.io images to `2026.05.31-3`
- Use mold linker instead of ld on alpine in order to solve import segfault on pyarrow musllinux amd64
- Remove Python 3.13 free-threaded wheels as Python 3.13t is not available anymore on the underlying manylinux image, see:
- https://github.com/apache/arrow/issues/50086
### Are these changes tested?
Yes via archery
### Are there any user-facing changes?
Yes, Python 3.13 free threaded was experimental, PyPA removed cp313t from the underlying manylinux docker image. We won't support wheels for 3.13 free-threaded anymore. We will support for 3.14 on-wards.
* GitHub Issue: #48028
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
`hash_any` and `hash_all` could return incorrect results for sliced nullable boolean arrays.
The validity bitmap used the slice offset, but the boolean values bitmap did not.
### What changes are included in this PR?
Apply the slice offset when reading boolean values in `hash_any` / `hash_all`.
Add C++ and Python regression tests.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
* GitHub Issue: #50043
Authored-by: fenfeng9 <fenfeng9@qq.com>
Signed-off-by: Rok Mihevc <rok@mihevc.org>
### Rationale for this change
Docs render weirdly
### What changes are included in this PR?
Fix extra blank line breaking docs block
### Are these changes tested?
I'll render docs
### Are there any user-facing changes?
No
* GitHub Issue: #20403
Authored-by: Nic Crane <thisisnic@gmail.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
Arnav Ahire