### Rationale for this change
`python/pyarrow/array.pxi` defines the class as `Decima32Array` (missing `l`).
### What changes are included in this PR?
Renaming of the class in `array.pxi` to `Decimal32Array`.
### Are these changes tested?
I haven't added a special test for this.
### Are there any user-facing changes?
No.
### Was AI used for this PR?
**PR code and description written by:**
- [X] Human
- [ ] AI
**Reviewed before submission by:**
- [X] Human
- [ ] AI
- [ ] Not reviewed
* GitHub Issue: #55285
Authored-by: AlenkaF <frim.alenka@gmail.com>
Signed-off-by: Tadeja Kadunc <tadeja.kadunc@gmail.com>
### Rationale for this change
We have many deprecated APIs in Python we should slowly remove.
### What changes are included in this PR?
Several deprecation messages are acted on.
### Are these changes tested?
By the existing test suite.
### Are there any user-facing changes?
Yes, some deprecated user-facing APIs are.removed.
### Was AI used for this PR?
In accordance to the [AI generation guidelines](https://arrow.apache.org/docs/dev/developers/overview.html#ai-generated-code), please disclose below whether and how AI was used in this PR.
**PR code and description written by:**
- [ ] Human
- [x] AI
**Reviewed before submission by:**
- [x] Human
- [x] AI
- [ ] Not reviewed
* GitHub Issue: #49153
Authored-by: Rok Mihevc <rok@mihevc.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Pandas has deprecated certain aspects of `.values` (when it looses information), which our pandas->pyarrow conversion currently relies upon. So updating that code path to avoid this usage.
### What changes are included in this PR?
Removed usage of `.values` for the deprecated cases (period, interval, datetimetz). In the custom `get_values`, we already avoided this for period and interval, added a special case for converting datetimetz to a numpy array. And ensured that other cases were we did a direct `.values` goes through this `get_values` helper.
### Are these changes tested?
Yes, I added more tests first to ensure full coverage of all cases that might run into the deprecation (e.g. the `Schema.from_pandas(df)` or `pa.array(EA)` with period/datetimetz dtypes were not currently covered)
### Are there any user-facing changes?
No
### Was AI used for this PR?
No
* GitHub Issue: #51302
Authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
In pyarrow 24.0.0, `Scalar` and `Array` gained arithmetic dunder methods (#32007) that unconditionally dispatch to `pyarrow.compute.call_function`. When the other operand is an unrecognized type, `_pack_compute_args` raises `TypeError` instead of returning `NotImplemented`. This prevents Python from falling back to the other operand's reflected methods (`__radd__`, `__rsub__`, etc.), breaking downstream libraries that relied on this protocol.
### What changes are included in this PR?
- Adds a `_compute_binary_op` helper in `scalar.pxi` that wraps `call_function` in a `try/except TypeError` and returns `NotImplemented` on failure. This follows the same pattern already used by `Scalar.__eq__` and `Array.__eq__`.
- Updates all binary arithmetic and bitwise dunders on both `Scalar` (`scalar.pxi`) and `Array` (`array.pxi`) to use this helper.
- Adds parametrized tests in `test_scalars.py` and `test_array.py` verifying that reflected operators on custom types are correctly invoked.
### Are these changes tested?
Yes. New parametrized tests (`test_dunders_return_notimplemented_for_unknown_types`) cover all 10 binary operators for both `Scalar` and `Array`. The bug was also manually reproduced against pyarrow 24.0.0 to confirm the tests exercise the right code path.
### Are there any user-facing changes?
Yes. `Scalar` and `Array` arithmetic dunders now return `NotImplemented` instead of raising `TypeError` when the other operand is not a recognized Arrow/NumPy type. This restores the pre-24.0.0 behavior where Python would fall back to the other operand's reflected method.
### AI-generated code disclosure
This PR was developed with assistance from an AI coding tool (Claude, Anthropic). All changes have been reviewed, understood, and verified.
* GitHub Issue: #49826Closes#49826
Lead-authored-by: AlenkaF <frim.alenka@gmail.com>
Co-authored-by: Alexandros Anastasiou <anastasioyaa@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
`Array.slice`, `ChunkedArray.slice`, `RecordBatch.slice`, and `Table.slice`
accept Python and NumPy integer-like values but reject Arrow integer scalars,
even though those scalars implement Python's integer-index protocol. This makes
values produced by Arrow APIs unnecessarily unusable as slice offsets and
lengths.
### What changes are included in this PR?
Normalize non-`None` offsets and lengths with `operator.index()` at the four
public Python wrapper boundaries before the existing validation and C++ calls.
Add regression coverage for all signed and unsigned Arrow integer scalar types
across the four wrappers, plus NumPy compatibility, invalid and null scalars,
negative values, offset clamping, and int64 overflow behavior.
### Are these changes tested?
Yes. Current Arrow C++ and editable PyArrow were built from source in an
isolated Python 3.13 environment. The focused new tests and existing slice
controls passed: 48 passed.
Cython translation, Python test-file compilation, Flake8, and
`git diff --check` also passed.
### Are there any user-facing changes?
Yes. The four Python `.slice()` methods now accept non-null Arrow integer
scalars for offsets and lengths. Existing Python/NumPy integer behavior and
error behavior for unsupported values are preserved.
### AI assistance
AI assisted analysis, implementation, test drafting, build work, and review.
The account holder reviewed and approved the final four-file diff. The account
holder did not personally run the commands reported above.
* GitHub Issue: #51145
Authored-by: YusefSyed <211442445+YusefSyed@users.noreply.github.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
There are already utilities to import tensor data from NumPy.
With DLPack standard, it will work across many tensor providers.
### What changes are included in this PR?
- Bind `FixedShapedTensorArray::FromTensor` in Python
- Add `ImportArrayVersionedFromDLPack` and `ImportTensorVersionedFromDLPack` in C++
- Add in Python `Array.from_dlpack` and `Tensor.from_dlpack`
Similar to the export, multidimensional tensor import require an explicit step through `Tensor`.
- No `Array::FromTensor` or even `FixedSizeListArray::FromTensor`. IMHO it does not feel as necessary in this direction but open to anyone's take one it.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes, new APIs.
* GitHub Issue: #39295
Lead-authored-by: AntoinePrv <AntoinePrv@users.noreply.github.com>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Have the same coverage as `to_numpy`.
### What changes are included in this PR?
- `FixedShapeTensorArray.__dlpack__` explicitly calling `to_tensor`
- `FixedShapeTensorArray.__dlpack_device__` (defaulted)
- `FixedShapeTensorScalar.__dlpack__` explicitly calling `to_tensor`
- `FixedShapeTensorScalar.__dlpack_device__` calling C++ device extraction on the underlying array.
If GH-51122 get merged first, I'll add the consuming side here, otherwise if this one gets merged first, I'll add it there.
### Are these changes tested?
Yes a few since this does not introduce new logic.
### Are there any user-facing changes?
Additions only.
* GitHub Issue: #38868
Authored-by: AntoinePrv <AntoinePrv@users.noreply.github.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Enable multidimensional DLPack support for Array via `to_tensor`.
### What changes are included in this PR?
- Add virtual `Array::ToTensor`
- Add `NumericArray::ToTensor` for 1D arrays
- Add `FixedSizeListArray::ToTensor` for multidimensional arrays
- Add DLPack error suggestiong tensor convertion
- Add DLPack tests with `arr.to_tensor().__dlpack__()`
Note: Nulls are explicitly supported in `to_tensor` as unspecified data. This was the current behaviour.
### Are these changes tested?
Yes
### Are there any user-facing changes?
New public Array function.
* GitHub Issue: #38868
Authored-by: AntoinePrv <AntoinePrv@users.noreply.github.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Require Numpy 2.0 to support newer features, allow newer numpy API/ABI
### What changes are included in this PR?
- Numpy 2.0+ requirement
- Pandas 2.2.2+ requirement (first wheel compatible with numpy 2.0)
- Pyodide 0.27.1 usage (numpy 2.0 api)
### Are there any user-facing changes?
Yes, in contrast to pyarrow 25, after merging this PR pyarrow 26 can't be used with Numpy 1.x or pandas 2.0-2.2.1
**This PR includes breaking changes to public APIs.**
This PR was created using GPT-5.6-Sol-xhigh, every line read & reviewed by me.
* GitHub Issue: #48473
Lead-authored-by: Adam Lippai <adam@rigo.sk>
Co-authored-by: Ádám Lippai <adam@rigo.sk>
Co-authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Pure version bump to enable implementing more features.
### What changes are included in this PR?
- Change the DLPack header file and definition.
- Use `raw_data` for `Tensor` similar to `Array` (instead of `mutable_raw_data` that is not available on immutable tensors).
- Strides may no longer be `NULL`
- Add `ExportArrayVersioned` and `ExportTensorVersioned` C APIs
- Support for `copy`
- Support for `max_version` and `copy` in `__dlpack__`
- Add `dl_device` to `__dlpack__` parameters as the spec requires but only CPU supported at the moment
### Are these changes tested?
Yes, with existing tests.
### Are there any user-facing changes?
* GitHub Issue: #41670
Authored-by: AntoinePrv <AntoinePrv@users.noreply.github.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
`pa.Array.to_pylist()` converts one element at a time through `Array::GetScalar` plus a Python `Scalar` wrapper; for list types each row additionally allocates a Python `Array` wrapper for the row's values slice and a fresh generator before recursing per element. A `sample` profile shows ~20% of runtime in CPython GC (triggered by the per-row GC-tracked allocations), ~25% in `GetScalar`, and only ~7% doing the useful work of creating the output objects — making `to_pylist` several times slower than converting via `to_pandas()` and back, and ~24x slower than `ndarray.tolist()` for plain int64. Details in #50326; this hit Apache Spark's Arrow-serialized Python UDFs (apache/spark#56940, apache/spark#56943).
### What changes are included in this PR?
Following review feedback, this adds a general scalar-free conversion mechanism instead of per-type `to_pylist` overrides:
- `Array` gains `cdef object _getitem_py(self, int64_t i)`, returning `self[i]` as a Python object. The base implementation is `GetScalar` + `Scalar.as_py`, so any type without a specialization behaves exactly as today (dates, times, timestamps, durations, decimals, dictionary, extension, unions, views, ...).
- The baseline `Array.to_pylist` becomes a single loop over `_getitem_py`. `maps_as_pydicts != None` keeps the Scalar-based path, since map→dict conversion has per-entry duplicate-key semantics.
- Specializations avoid all per-element Scalar and per-row Array-wrapper allocation:
- integers and floats (a `type_id` switch on `NumericArray`; date/time/timestamp subclasses fall through to the exact base),
- boolean,
- string/binary and large variants (`GetValue` + `PyUnicode_DecodeUTF8` / `PyBytes_FromStringAndSize`, matching `str(buf, 'utf8')` / `to_pybytes()` exactly),
- list/large_list/fixed_size_list (each row's list is built from the child's `_getitem_py` over the offset range; the wrapped child is cached on the parent array),
- map (association list of key/value tuples, matching `MapScalar.as_py`),
- struct (one dict per row; duplicate field names fall back to the Scalar path so they raise `ValueError` like `StructScalar.as_py`).
Nested types compose without any per-row wrappers. `ChunkedArray.to_pylist`, `Table.to_pylist` and `ListScalar.as_py` delegate here and speed up automatically. Follow-up candidates: string/binary views, run-end-encoded, dictionary, a fast path for date32.
Benchmarks (macOS arm64, M4 Max):
| benchmark | before | after | speedup |
|---|---|---|---|
| flat `int64` with nulls (4M) | 0.39 s | 0.028 s | 14x (~7 ns/element, on par with `ndarray.tolist`) |
| flat `string` (4M) | 0.83 s | 0.06 s | 14x |
| `list<string>` (2M rows) | 1.93 s | 0.46 s | 4.2x |
| `list<list<int32>>` (1M rows) | 2.10 s | 0.40 s | 5.2x |
| `struct<int64,string>` (1M rows) | 0.91 s | 0.07 s | 13x |
| `map<string,int64>` (1M rows) | 2.77 s | 0.74 s | 3.8x |
### Are these changes tested?
`test_to_pylist_bulk_paths` (added here) compares against the per-scalar conversion with exact element types for representative arrays including sliced views. Additionally verified with a randomized differential test against `[x.as_py() for x in arr]` with exact-type equality: all integer widths (incl. values beyond 2^62), floats (NaN/inf), boolean, string/binary (+large, multibyte), all list kinds, nested lists, struct (incl. empty struct, duplicate-field-name `ValueError`), map (incl. strict-mode duplicate-key `KeyError`), dictionary/null fallbacks, sliced/chunked arrays, and both `maps_as_pydicts` modes — no differences. `pytest test_array.py test_scalars.py test_convert_builtin.py test_table.py test_types.py`: 1295 passed.
### Are there any user-facing changes?
No behavior changes, only performance.
* GitHub Issue: #50326
This pull request and its description were written by Isaac.
Lead-authored-by: Liang-Chi Hsieh <liangchi.hsieh@databricks.com>
Co-authored-by: Isaac
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
[Pandas 2.0.0](https://pypi.org/project/pandas/2.0.0/) was released on April 3rd 2023.
The last version of Pandas 1 was [Pandas 1.5.3](https://pypi.org/project/pandas/1.5.3/) and it was released on January 19th 2023. That version only supports up to Python 3.11.
We are currently supporting up to Pandas 1.5.2 (first version supporting 3.11).
On the issue we discussed to bump our Pandas support to 2.0.0 but given 2.0.3 is the latest in the 2.0.x, was released on June 2023, only supports up to Python 3.11 and bumping to it allows use to improve even further some of the conditionals we are bumping to Pandas >= 2.0.3
### What changes are included in this PR?
Update minimal support check to Version(2.0.3) and remove all test skips, conditionals and scenarios where we are checking for Pandas < 2.0.3.
We found a couple of minor things
### Are these changes tested?
Yes via CI
### Are there any user-facing changes?
Yes, Pandas < 2 won't be supported.
* GitHub Issue: #48808
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Constructing a fixed-shape-tensor array from a list of individual ndarrays only
worked when each element was 1-D; ≥2-D elements failed with
`ArrowInvalid: Can only convert 1-dimensional array values`. The only workaround
was stacking the list into a single ndarray and using
`FixedShapeTensorArray.from_numpy_ndarray`.
### What changes are included in this PR?
The C++ list converter `PyListConverter::AppendNdarray` now accepts
multi-dimensional ndarray elements for **fixed-size lists** (the storage of a
fixed-shape tensor) by flattening them in C order. The fixed-size-list builder
still validates that the flattened length matches the list width, so wrong sizes
error cleanly. Variable-sized lists remain restricted to 1-D values to avoid
ambiguity. As a side benefit, plain `fixed_size_list` also accepts
multi-dimensional ndarray elements now.
### Are these changes tested?
Yes:
- `test_tensor_array_from_list_of_ndarrays` — construction from 2-D and 3-D
ndarrays, null handling, storage parity with `from_numpy_ndarray`, and the
size-mismatch error, across `int8`/`int64`/`float32`.
- `test_fixed_size_list_from_multidim_ndarray` — plain `fixed_size_list` from
multi-dim arrays, plus a check that variable-sized lists still reject 2-D.
### Are there any user-facing changes?
Yes — `pa.array([multi-dim ndarrays], type=fixed_shape_tensor(...))` (and the
same for `fixed_size_list`) now works instead of raising. Existing 1-D behavior
and variable-sized-list behavior are unchanged.
Scoped to construction only; the reverse `to_numpy` shape-preservation also
raised in the issue is intentionally left as a separate follow-up.
* GitHub Issue: #49644
Lead-authored-by: Samuel Aboderin <aboderinseun01@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Rok Mihevc <rok@mihevc.org>
See #38584 for original PR. Will be quoted for this PR description.
### Rationale for this change
> support multi sortkey nulls first.
```
order by i nulls first, j, k nulls first;
```
> The current null sorting only supports all sortkeys, not a certain sortkey, so NullPlacement is extended to the SortKey field. Since the underlying framework is very well written, when modifying this function, you only need to pass the null_placement of each SortKey in. That’s it.
### What changes are included in this PR?
> 1.SortKey structure, NullPlacemnt transfer logic, sorting logic and Ording related, test related
> 2.Substriait related.
> 3.c_glib related.
> 4.SelectK related.
> 5.RankOptions related.
### Are these changes tested?
> yes, I changed the code inside vector_sort_test.cc and performed additional tests.
### Are there any user-facing changes?
> yes, pg database include null sorting of multiple sort keys.
**This PR includes breaking changes to public APIs.** (If there are any breaking changes to public APIs, please explain which changes are breaking. If not, you can remove this.)
I amended the original PR to be less breaking in public APIs.
Still Ordering, SortOptions, RankOptions, and RankQuantileOptions now accept a `std::optional<NullPlacement>` instead of NullPlacement, which did lead to some changes in downstream APIs and bindings. ~I also need some help with fixing the `c_glib` bindings.~
* GitHub Issue: #38558
Lead-authored-by: Alexander Taepper <alexander.taepper@gmail.com>
Co-authored-by: light-city <455954986@qq.com>
Co-authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Docs render weirdly
### What changes are included in this PR?
Fix extra blank line breaking docs block
### Are these changes tested?
I'll render docs
### Are there any user-facing changes?
No
* GitHub Issue: #20403
Authored-by: Nic Crane <thisisnic@gmail.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
When data exceeds int32 limits, properly wraps each chunk as ExtensionArray
### What changes are included in this PR?
Modified extension type handling to support both Array and ChunkedArray storage types.
### Are these changes tested?
Yes , Manually tested the changes
### Are there any user-facing changes?
No
### This PR contains a "Critical Fix".
This change fixes a crash in list_ storage . when list data exceeds int32 limits, PyArrow automatically creates a ChunkedArray. However, ExtensionArray.from_storage() only accepts Array objects, not ChunkedArray.
* GitHub Issue: #50012
Authored-by: Ankit.Ahlawat@ibm.com <Ankit.Ahlawat@ibm.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
When converting a pandas.Categorical with tz-aware datetime categories to a PyArrow array, the timezone information was silently dropped from the dictionary array's value type. This is a silent data loss bug — no warning or error is raised, but the timezone metadata is lost.
### What changes are included in this PR?
In `python/pyarrow/array.pxi`, the Categorical conversion was using `values.categories.values(raw numpy array) `which strips timezone metadata since numpy does not support tz-aware datetimes. Changed to values.categories (pandas Index) and added from_pandas=True so PyArrow uses the pandas conversion path, which correctly preserves timezone metadata.
### Are these changes tested?
Yes. Verified manually
### Are there any user-facing changes?
Yes — this is a bug fix. Users did #49875
This PR contains a **"Critical Fix"** — timezone information was lost silently during conversion without any warning or error.
* GitHub Issue: #49875
Authored-by: Ankit.Ahlawat@ibm.com <Ankit.Ahlawat@ibm.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
The conversion from array with string type to pandas series, when array only has a `None` element, has been taking the old code path even with pandas 3.0.
### What changes are included in this PR?
Always check `dtype` in the `_array_like_to_pandas ` conversion and use pandas new default string `dtype` if available.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No, only bug fix.
* GitHub Issue: #49002
Lead-authored-by: Alenka Frim <AlenkaF@users.noreply.github.com>
Co-authored-by: AlenkaF <frim.alenka@gmail.com>
Co-authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
Please see #32007, currently, neither arrays nor scalars support Python-native arithmetic operations, such as `array + array`, it has to be done via `pyarrow.compute` API. This PR strives to fix this with custom dunder methods.
### What changes are included in this PR?
Implemented dunder methods
### Are these changes tested?
Yes
### Are there any user-facing changes?
Possibility to use Python operators directly instead of calling the `pyarrow.compute` API.
* GitHub Issue: #32007
Authored-by: Bogdan Romenskii <rmnsk@seznam.cz>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
Closes#49509
### What changes are included in this PR?
Docs formatting/typos corrected.
### Are these changes tested?
Yes, on fork branch.
### Are there any user-facing changes?
No, just corrected formatting/typos in docs.
* GitHub Issue: #49509
Authored-by: Tadeja Kadunc <tadeja.kadunc@gmail.com>
Signed-off-by: Rok Mihevc <rok@mihevc.org>
### Rationale for this change
Fix the docs. It seems this is a feature and not just happens to work as the code a little below even has an explicit comment:
https://github.com/apache/arrow/blob/0dfae701ef98aa4a26b9abbaaf3bf01130df3702/python/pyarrow/array.pxi#L1347
### What changes are included in this PR?
Document that `Array.from_buffers` takes a `list[Buffer | None]` instead of strictly `list[Buffer]`.
### Are these changes tested?
*not applicable*
### Are there any user-facing changes?
Yes, but not in an API sense :)
Authored-by: Robsdedude <dev@rouvenbauer.de>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
This PR removes the deprecated `Array.format` method as requested in issue #48102.
### What changes are included in this PR?
- Removed the deprecated `Array.format` method from the Python API
- The method will be automatically removed from the documentation (https://arrow.apache.org/docs/python/generated/pyarrow.Array.html) as it's auto-generated from the source code
### Are these changes tested?
Yes, existing tests continue to pass.
### Are there any user-facing changes?
**This PR includes breaking changes to public APIs.**
The deprecated `Array.format` method has been removed. Users who have not migrated from this deprecated method should use `Array.to_string()` instead. This change was announced in the deprecation warning since 2020.
**Migration:**
```python
# Old (deprecated)
array.format()
# New
array.to_string()
* GitHub Issue: #48102
Authored-by: Eslam Ahmed <eslamahmedd171@gmail.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
Enable `ARROW:null_count:approximate` support for `arrow::ArrayStatistics`, along with the corresponding GLib, Ruby and Python bindings.
### What changes are included in this PR?
Enable `ARROW:null_count:approximate` in C++ and bind it to `ArrayStatistics` in GLib, Ruby and Python.
### Are these changes tested?
Yes, I ran the relevant unit tests.
### Are there any user-facing changes?
Yes.
* The type of `arrow::ArrayStatistics::null_count` has been changed from `std::optional<int64_t>` to `std::optional<CountType>`
* New `garrow_array_statistics_is_null_count_exact()`/`garrow_array_statistics_get_null_count_{exact,approximate}()` functions in GLib.
* Add support for approximate value in `Arrow::ArrayStatistics#null_count` in Ruby.
* A new field `is_null_count_exact` has been added to `ArrayStatistics` in Python.
* GitHub Issue: #47103
Lead-authored-by: arash andishgar <arashandishgar1@gmail.com>
Co-authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
Implement support for the `ARROW:distinct_count:approximate` statistics attribute.
### What changes are included in this PR?
Changed the type of `arrow::ArrayStatistics::distinct_count` to support both `double` (for approximate values) and `int64_t` (for exact values).
### Are these changes tested?
Yes, I ran the corresponding unit test.
### Are there any user-facing changes?
Yes.
* C++: The type of `arrow::ArrayStatistics::distinct_count` was changed from `std::optional<int64_t>` to `std::optional<std::variant<int64_t, double>>`.
* GLib: `garrow_array_statistics_get_distinct_count()` is deprecated.
* GitHub Issue: #47101
Lead-authored-by: Arash Andishgar <arashandishgar1@gmail.com>
Co-authored-by: Sutou Kouhei <kou@clear-code.com>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Co-authored-by: Rok Mihevc <rok@mihevc.org>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
When slicing arrays with non-trivial steps we were using `numpy.arange` to generate the indices for take. As numpy is an optional dependency, implementing it via Python caused a performance penalty. Creating a pyarrow function to build our own ranges that mimics Python range or numpy arange is useful for that uses case and might also be useful for other use cases. Currently we only generate `Array[Int64]` we could potentially generate more types.
### What changes are included in this PR?
provide a `pa.arange` function that allows us to generate indices when slicing arrays.
### Are these changes tested?
Yes new tests added.
### Are there any user-facing changes?
No but a new pyarrow.arange function has been added.
* GitHub Issue: #46771
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Producing part of DLPack protocol [has been added to Arrow Arrays](https://github.com/apache/arrow/issues/33984) but is missing on Arrow Tensor class.
### What changes are included in this PR?
This PR adds support for producing DLPack struct and bindings to it in Python:
- `ExportTensor` and `ExportTensorDevice` methods on Arrow C++ Tensor class
- `__dlpack__` method on PyArrow Tensor class exposing ExportTensor method
- `__dlpack_device__` method on PyArrow Tensor class exposing ExportTensorDevice method
### Are these changes tested?
Yes
### Are there any user-facing changes?
No
* GitHub Issue: #39294
Lead-authored-by: AlenkaF <frim.alenka@gmail.com>
Co-authored-by: Alenka Frim <AlenkaF@users.noreply.github.com>
Co-authored-by: Rok Mihevc <rok@mihevc.org>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
Slicing an array in non-trivial steps raises an exception when Numpy is not installed.
#46606
### What changes are included in this PR?
I changed `np.arange(...)` to `list(range(...))` In `python/pyarrow/array.pxi`
### Are these changes tested?
Yes
### Are there any user-facing changes?
No
* GitHub Issue: #46606
Authored-by: shu-kitamura <shusei3316@yahoo.co.jp>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
#46403
### What changes are included in this PR?
A new PrettyPrinter option is added to limit elements to 100 characters by default.
### Are these changes tested?
Yes
### Are there any user-facing changes?
Yes, the default length for outputted elements when stringifying them is now different so if a user was relying on ToString of an array with large elements that result may now be changed.
* GitHub Issue: #46403
Lead-authored-by: David Sherrier <davidsherrier1997@gmail.com>
Co-authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
The `FixedShapeTensorArray.from_numpy_ndarray` method did not pass `dim_names` to the `fixed_shape_tensor` constructor, which resulted in dimension names being lost when converting from a NumPy array. This change ensures that dimension names are properly preserved when constructing a tensor array from a NumPy ndarray.
### What changes are included in this PR?
- Added an optional `dim_names` parameter to `FixedShapeTensorArray.from_numpy_ndarray`.
- If provided, the `dim_names` are now passed to the `fixed_shape_tensor` constructor.
### Are these changes tested?
- Existing tests pass, confirming no regressions to current functionality.
- Additional unit tests have been added to verify that `dim_names` are correctly handled when specified.
### Are there any user-facing changes?
- The method `FixedShapeTensorArray.from_numpy_ndarray` now accepts an optional `dim_names` argument.
- This argument is optional, and the behavior remains unchanged when it is not provided.
* GitHub Issue: #45531
Authored-by: yyossy5 <hm.hr.yossy@gmail.com>
Signed-off-by: Rok Mihevc <rok@mihevc.org>
### Rationale for this change
Update docstring to explicitly mention that FixedSizeListArray.values ignores the offset
### What changes are included in this PR?
Updated docstring to mention this
### Are these changes tested?
No
### Are there any user-facing changes?
No
* GitHub Issue: #41672
Authored-by: Nic Crane <thisisnic@gmail.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
Apache Arrow C++ can attach statistics read from Apache Parquet data to `arrow::Array`. If we have the bindings of the feature in Python, Python users can also use attached statistics.
### What changes are included in this PR?
* Add `pyarrow.ArrayStatistics`
* Add `pyarrow.Array.statistics()`.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* GitHub Issue: #45457
Lead-authored-by: Sutou Kouhei <kou@clear-code.com>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
Currently, unfortunately `MapScalar`/`Array` types are not deserialized into proper Python `dict`s, which is unfortunate since this breaks "roundtrips" from Python -> Arrow -> Python:
```
import pyarrow as pa
schema = pa.schema([pa.field('x', pa.map_(pa.string(), pa.int64()))])
data = [{'x': {'a': 1}}]
pa.RecordBatch.from_pylist(data, schema=schema).to_pylist()
# [{'x': [('a', 1)]}]
```
This is especially bad when storing TiBs of deeply nested data (think of lists in structs in maps...) that were created from Python and serialized into Arrow/Parquet, since they can't be read in again with native `pyarrow` methods without doing extremely ugly and computationally costly workarounds.
### What changes are included in this PR?
A new parameter `maps_as_pydicts` is introduced to `to_pylist`, `to_pydict`, `as_py` which will allow proper roundtrips:
```
import pyarrow as pa
schema = pa.schema([pa.field('x', pa.map_(pa.string(), pa.int64()))])
data = [{'x': {'a': 1}}]
pa.RecordBatch.from_pylist(data, schema=schema).to_pylist(maps_as_pydicts="strict")
# [{'x': {'a': 1}}]
```
### Are these changes tested?
Yes. There are tests for `to_pylist` and `to_pydict` included for `pyarrow.Table`, whilst low-level `MapScalar` and especially a nesting with `ListScalar` and `StructScalar` is tested.
Also, duplicate keys now should throw an error, which is also tested for.
### Are there any user-facing changes?
No callsites should be broken, simply a new keyword-only optional parameter is added.
* GitHub Issue: #39010
Authored-by: Jonas Dedden <university@jonas-dedden.de>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
With pandas' [PDEP-14](https://pandas.pydata.org/pdeps/0014-string-dtype.html) proposal, pandas is planning to introduce a default string dtype in pandas 3.0 (instead of the current object dtype).
This will become the default in pandas 3.0, and can be enabled with an option in the upcoming pandas 2.3 (`pd.options.future.infer_string = True`). To prepare for that, we should start using that string dtype in `to_pandas()` conversions when that option is enabled.
### What changes are included in this PR?
- If pandas >= 3.0 is used or the pandas option is enabled, ensure that `to_pandas()` calls use the default string dtype of pandas for string-like columns (string, large_string, string_view)
### Are these changes tested?
It is tested in the pandas-nightly crossbow build.
There is still one failure that is because of a bug on the pandas side (https://github.com/pandas-dev/pandas/issues/59879)
### Are there any user-facing changes?
**This PR includes breaking changes to public APIs.** Depending on the version of pandas, `to_pandas()` will change to use pandas' string dtype instead of object dtype. This is a breaking user-facing change, but essentially just following the equivalent change in default dtype on the pandas side.
* GitHub Issue: #43683
Lead-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Co-authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
### Rationale for this change
Arrow C++ and the Arrow specification now support 32-bit and 64-bit decimal types...pyarrow should too!
### What changes are included in this PR?
Added type, array, and scalar bindings.
### Are these changes tested?
Yes!
### Are there any user-facing changes?
Yes!
* GitHub Issue: #44713
Authored-by: Dewey Dunnington <dewey@fishandwhistle.net>
Signed-off-by: Dewey Dunnington <dewey@fishandwhistle.net>
### Rationale for this change
Currently `from_buffers` is not working with StringView on Python because we validate against num_buffers. This only take into account the mandatory buffers but does not take into account the variadic_spec that can be present for both string_view and binary_view
### What changes are included in this PR?
Take into account whether the type contains a variadic_spec for the non-mandatory buffers and only check lower_bound number of buffers.
### Are these changes tested?
Yes, I've added a couple of tests.
### Are there any user-facing changes?
We are exposing a new method on the Python DataType. `has_variadic_buffers` which tells us whether the number of buffers expected is only lower-bounded by num_buffers.
* GitHub Issue: #44651
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
We [added canonical JsonExtensionType](https://github.com/apache/arrow/pull/13901) and we should make it usable from Python.
### What changes are included in this PR?
Python wrapper for `JsonExtensionType` and `JsonArray` are added on Python side as well as `JsonArray` on c++ side.
### Are these changes tested?
Python tests for the extension type and array are included.
### Are there any user-facing changes?
This adds a json canonical extension type to pyarrow.
* GitHub Issue: #44066
Lead-authored-by: Rok Mihevc <rok@mihevc.org>
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Currently the docstring for Array.slice says it returns a RecordBatch. I don't see how this is possible with the existing code. My guess is that this was a copy-and-paste error back when the Array and RecordBatch slice impls were added in https://issues.apache.org/jira/browse/ARROW-547.
### What changes are included in this PR?
Just a docstring change. I copied the language from the take method so things are consistent.
### Are these changes tested?
No.
### Are there any user-facing changes?
Just docs.
Authored-by: Bryce Mecum <petridish@gmail.com>
Signed-off-by: Bryce Mecum <petridish@gmail.com>
### Rationale for this change
Being able to run pyarrow without requiring numpy.
### What changes are included in this PR?
If numpy is not present we are able to import pyarrow and run functionality.
A new CI job has been created to run some basic tests without numpy.
### Are these changes tested?
Yes via CI.
### Are there any user-facing changes?
Yes, NumPy can be removed from the user installation and pyarrow functionality still works
* GitHub Issue: #25118
Lead-authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
See #15058.
UUID datatype is common in throughout the ecosystem and Arrow as supporting it as a native type would reduce friction.
### What changes are included in this PR?
This PR implements logic for Arrow canonical extension type in C++ and a Python wrapper.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes, new extension type is added.
* Closes: #15058
Authored-by: Rok Mihevc <rok@mihevc.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
We have added bindings for the Device and MemoryManager classes (https://github.com/apache/arrow/issues/41126), and as a next step we can expose the functionality to copy a full Array or RecordBatch to a specific memory manager.
### What changes are included in this PR?
This adds a `copy_to` method on pyarrow Array and RecordBatch.
### Are these changes tested?
Yes
* GitHub Issue: #42222
Authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
### Rationale for this change
C++ and Python implementations of #43234
### What changes are included in this PR?
- Implement C++ `Bool8Type`, `Bool8Array`, `Bool8Scalar`, and tests
- Implement Python bindings to C++, as well as zero-copy numpy conversion methods
- TODO: docs waiting for rebase on #43458
### Are these changes tested?
Yes
### Are there any user-facing changes?
Bool8 extension type will be available in C++ and Python libraries
* GitHub Issue: #17682
Authored-by: Joel Lubinitsky <joellubi@gmail.com>
Signed-off-by: Felipe Oliveira Carvalho <felipekde@gmail.com>
### Rationale for this change
Add the newly ratified extension type.
### What changes are included in this PR?
The C++/Python implementation only.
### Are these changes tested?
Yes
### Are there any user-facing changes?
No.
* GitHub Issue: #43454
Lead-authored-by: David Li <li.davidm96@gmail.com>
Co-authored-by: Weston Pace <weston.pace@gmail.com>
Signed-off-by: David Li <li.davidm96@gmail.com>
### Rationale for this change
StructArray.from_array currently accepts names or fields to create the struct array. However if you already have a struct type it's more convenient to pass that in and allow the function to use it to build the StructArray instead of the user having to pull out the fields themselves.
### What changes are included in this PR?
Add a new argument to StructArray.from_array called structtype. The function will prevent both fields and structype from being passed by raising a ValueError. If structtype is not null then the existing fields argument is set from the structtype fields. This allows all of the existing code in the function to remain untouched.
### Are these changes tested?
Yes. Testing creating the structarray from fields a test is added to make sure that a struct type can be used to create the array.
### Are there any user-facing changes?
Yes, the StructArray.from_arrays function now has an extra optional argument
* GitHub Issue: #42014
Authored-by: Akshay Subramanian <173708861+shinespiked@users.noreply.github.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
PyArrow implementation for the specification additions being proposed in
https://github.com/apache/arrow/pull/40708
### What changes are included in this PR?
New `__arrow_c_device_array__` method to `pyarrow.Array` and
`pyarrow.RecordBatch`, and support in the `pyarrow.array(..)`,
`pyarrow.record_batch(..)` and `pyarrow.table(..)` functions to consume
objects that have those methods.
### Are these changes tested?
Yes (for CPU only for now, https://github.com/apache/arrow/pull/40385 is
a prerequisite to test this for CUDA)
* GitHub Issue: #38325
### Rationale for this change
The various Python reprs or the C++ `PrettyPrint` functions will currently just segfault when passing an object that has its data on a non-CPU device. In python, getting a segfault while displaying the object is very annoying, and so we should make this at least not crash.
### What changes are included in this PR?
When we detect data on a non-CPU device passed to `PrettyPrint`, we copy the necessary part (the full Arrays for Array/RecordBatch, or the full chunks that are being printed for ChunkedArray/Table) to the default CPU device, and then use the existing print utilities as is on this copied subset.
For large data, this can be potentially costly by copying a lot of data (but you can always avoid that by not printing the data), but for chunked data we will still only copy those chunks of the full dataset needed to print the object.
Longer term, we should investigate if we can actually copy sliced arrays to a different device (with actual pruning of the buffers while copying): https://github.com/apache/arrow/issues/43055
### Are these changes tested?
Yes
### Are there any user-facing changes?
No
* GitHub Issue: #41664
Lead-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Co-authored-by: Felipe Oliveira Carvalho <felipekde@gmail.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
### Rationale for this change
Common `Array` APIs should not segfault or abort on non-cpu devices.
### What changes are included in this PR?
* `device_type` and `is_cpu` methods added to the `Array` class
* Any function that segfaults, aborts, or gives incorrect results on non-cpu devices now raises an exception
### Are these changes tested?
* Unit tests added
### Are there any user-facing changes?
* `device_type` and `is_cpu` methods added to the `Array` class
* GitHub Issue: #42112
Lead-authored-by: Dane Pitkin <dpitkin@apache.org>
Co-authored-by: Dane Pitkin <dpitkin.oss@gmail.com>
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
### Rationale for this change
Follow-up on https://github.com/apache/arrow/issues/39979 which added `_export_to_c_device`/`_import_from_c_device` methods, but for now only for CPU devices.
### What changes are included in this PR?
* Ensure `pyarrow.cuda` is imported before importing data through the C Interface, to ensure the CUDA device is registered
* Add tests for exporting/importing with the device interface on CUDA
### Are these changes tested?
Yes, added tests for CUDA.
* GitHub Issue: #40384
Authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
### What changes are included in this PR?
Allow Array.filter() to take general array input.
### Are these changes tested?
Unit test added, via CI.
### Are there any user-facing changes?
No
* GitHub Issue: #42013
Authored-by: Kelvin Wu <kelvinyu1117@gmail.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
### Rationale for this change
When constructing a `MapArray` with `FromArrays` one can not supply a `null_bitmap`.
### What changes are included in this PR?
Optional `null_bitmap` argument is added to `MapArray::FromArrays`.
### Are these changes tested?
TODO (have them locally, need to clean them up and commit.
### Are there any user-facing changes?
No.
* GitHub Issue: #41684
Authored-by: AlenkaF <frim.alenka@gmail.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Alenka Frim