### Rationale for this change
Use C++ type traits for `is_nested` function to avoid redefining in Python
### What changes are included in this PR?
Expose `is_nested` function
### Are these changes tested?
Yes
### Are there any user-facing changes?
Yes
* GitHub Issue: #40282
Lead-authored-by: NNic Crane <thisisnic@gmail.com>
Co-authored-by: Nic Crane <thisisnic@gmail.com>
Co-authored-by: Hyunseok Seo <hsseo0501@gmail.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
`ObjectType` and `FileStatistics` in io/hdfs.h have been deprecated for a while and can be removed.
### What changes are included in this PR?
`ObjectType` and `FileStatistics` structs are removed and instead FileSystem API in `arrow::fs` is used. Together with this change, the hdfs connected code is moved from `cpp/src/arrow/io` to `cpp/src/arrow/filesystem` merging `FileSystem` and `HadoopFileSystem` classes from `arrow::io` into the public `HadoopFileSystem` class.
### Are these changes tested?
Existing tests should pass.
### Are there any user-facing changes?
Deprecated structs are removed and all hdfs related code is now a part of the filesystem module.
* GitHub Issue: #45747
Also closes: https://github.com/apache/arrow/issues/22457 (not sure about `io/interfaces.h`?)
Lead-authored-by: AlenkaF <frim.alenka@gmail.com>
Co-authored-by: Alenka Frim <AlenkaF@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Addresses #51349
### What changes are included in this PR?
* Adds a new `read_kms_url` member to `DecryptionConfiguration` and a parameter with the same name to `rotate_master_keys`. Both are false by default.
* Adds these to the PyArrow bindings.
### Are these changes tested?
Yes, this includes new unit tests.
### Are there any user-facing changes?
Yes, this adds a new user-facing option. Users that relied on this behaviour previously will now need to opt-in and enable the flag.
* GitHub Issue: #51349
Authored-by: Adam Reeve <adreeve@gmail.com>
Signed-off-by: Adam Reeve <adreeve@gmail.com>
**Rationale for this change**
In arrow::dataset::MakeReaderProperties, all parquet::ReaderProperties options had to be copied field-by-field from parquet_scan_options->reader_properties because ReaderProperties did not support mutating or setting MemoryPool* after construction.
**What changes are included in this PR?**
Added set_memory_pool to parquet::ReaderProperties in cpp/src/parquet/properties.h.
Added unit test TestReaderProperties.SetMemoryPool in cpp/src/parquet/properties_test.cc.
Are these changes tested?
Yes
Are there any user-facing changes?
No
* GitHub Issue: #51264
Authored-by: Diveyam Mishra <mishradiveyam@gmail.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
There are already utilities to import tensor data from NumPy.
With DLPack standard, it will work across many tensor providers.
### What changes are included in this PR?
- Bind `FixedShapedTensorArray::FromTensor` in Python
- Add `ImportArrayVersionedFromDLPack` and `ImportTensorVersionedFromDLPack` in C++
- Add in Python `Array.from_dlpack` and `Tensor.from_dlpack`
Similar to the export, multidimensional tensor import require an explicit step through `Tensor`.
- No `Array::FromTensor` or even `FixedSizeListArray::FromTensor`. IMHO it does not feel as necessary in this direction but open to anyone's take one it.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes, new APIs.
* GitHub Issue: #39295
Lead-authored-by: AntoinePrv <AntoinePrv@users.noreply.github.com>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Reconstructing a nested Schema from the Parquet Thrift metadata implies a recursive call that can blow up the stack on pathologically-nested schemas (with thousands of nesting levels or more).
By adding a limit on the schema nesting depth, we turn a stack overflow-induced crash into a regular Parquet error.
### Are these changes tested?
By additional unit tests; also privately with a proof-of-concept reproducer that induces a stack overflow exhaustion.
### Are there any user-facing changes?
In the unlikely case where a legitimate Parquet file has a deeper schema than the default schema nesting limit in this PR (100), an error will be raised when reading where it used to succeed. The user can bump the limit to circumvent the error.
**This PR contains a "Critical Fix".** It fixes a crash on a deeply nested Parquet schema that would provoke a stack overflow. It is not an exploitable vulnerability except through denial of service.
Thanks to "1K0CT" for the initial report.
* GitHub Issue: #51238
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Expose remaining csv write options to pyarrow.
### What changes are included in this PR?
Adding `eol` and `null_string` to csv.WriteOptions.
### Are these changes tested?
Yes, testing of setters included.
### Are there any user-facing changes?
* GitHub Issue: #34577
Lead-authored-by: Martin Nowak <code@dawg.eu>
Co-authored-by: AlenkaF <frim.alenka@gmail.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
Enable multidimensional DLPack support for Array via `to_tensor`.
### What changes are included in this PR?
- Add virtual `Array::ToTensor`
- Add `NumericArray::ToTensor` for 1D arrays
- Add `FixedSizeListArray::ToTensor` for multidimensional arrays
- Add DLPack error suggestiong tensor convertion
- Add DLPack tests with `arr.to_tensor().__dlpack__()`
Note: Nulls are explicitly supported in `to_tensor` as unspecified data. This was the current behaviour.
### Are these changes tested?
Yes
### Are there any user-facing changes?
New public Array function.
* GitHub Issue: #38868
Authored-by: AntoinePrv <AntoinePrv@users.noreply.github.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Pure version bump to enable implementing more features.
### What changes are included in this PR?
- Change the DLPack header file and definition.
- Use `raw_data` for `Tensor` similar to `Array` (instead of `mutable_raw_data` that is not available on immutable tensors).
- Strides may no longer be `NULL`
- Add `ExportArrayVersioned` and `ExportTensorVersioned` C APIs
- Support for `copy`
- Support for `max_version` and `copy` in `__dlpack__`
- Add `dl_device` to `__dlpack__` parameters as the spec requires but only CPU supported at the moment
### Are these changes tested?
Yes, with existing tests.
### Are there any user-facing changes?
* GitHub Issue: #41670
Authored-by: AntoinePrv <AntoinePrv@users.noreply.github.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
Closes#50801
As title, this exposes RecordBatchReaderSourceNodeOptions to Python. With it, a hash join can build its hash table once and stream the probe side through as a generator-backed RecordBatchReader, releasing each input chunk as it is consumed, instead of materializing the full probe table up front (table_source) or pinning all fragments for the plan's lifetime (dataset scan).
### Are these changes tested?
Yes
### Are there any user-facing changes?
Yeah, users can pass RecordBatchReaderSourceNodeOptions into Declaration, but its not a breaking change
* GitHub Issue: #50801
Lead-authored-by: You-Cheng Lin <106612301+owenowenisme@users.noreply.github.com>
Co-authored-by: You-Cheng Lin <mses010108@gmail.com>
Signed-off-by: Rossi Sun <zanmato1984@gmail.com>
### Rationale for this change
PyFlightServer keeps a reference towards the Python server via `OwnedRefNoGIL server_`, the Python server also keeps a reference of the C++ `PyFlightServer` creating a cycle that is never freed during the process lifetime.
### What changes are included in this PR?
Create a new `ReleasePythonServerRef` method that is called after any `server.Shutdown` (including at `__exit__`) allowing for the `OwnedRefNoGIL` to be cleared breaking the cycle. This lets normal reference counting free the previously leaked Python object.
### Are these changes tested?
Yes, the newly added tests were failing leaking the references before the fix.
Currently a test demonstrating a leak when not calling server.Shutdown is added for discussion purposes.
### Are there any user-facing changes?
No
* GitHub Issue: #50684
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Resolves [49305](https://github.com/apache/arrow/issues/49305)
`RecordBatchFileReader::CountRows` has existed in Arrow C++ (`cpp/src/arrow/ipc/reader.h`) but
was never bound in Python, so the only way to get the total number of rows of an IPC file was:
```python
num_rows = sum(reader.get_batch(i).num_rows for i in range(reader.num_record_batches))
```
### This findings are done in original opened Issue #49305
That deserializes every record batch just to read its length, which is wasteful and becomes
expensive on remote filesystems.
### What changes are included in this PR?
Adds `RecordBatchFileReader.count_rows()`:
```python
with pa.ipc.open_file(source) as reader:
reader.count_rows()
```
Three changes:
* `python/pyarrow/includes/libarrow.pxd`: declare `CResult[int64_t] CountRows()` on
`CRecordBatchFileReader`, which was the missing piece.
* `python/pyarrow/ipc.pxi`: add `count_rows()` to `_RecordBatchFileReader`, released GIL around
the call, with the same closed reader guard used by the existing `stats` property
* `python/pyarrow/tests/test_ipc.py`: tests.
On the naming: `count_rows()` follows the C++ method and is consistent with the existing
`count_rows()` on `Dataset`, `Scanner` and `Fragment`.
To be precise about the benefit, since the issue describes it as reading the count from the
metadata: the C++ implementation still walks every block, but reads only each record batch's
flatbuffer message header to pick up its length, and never touches the data buffers. So this is
a reduction in bytes read rather than in the number of reads, and the gain shows up on remote
filesystems and on files with large batches rather than in a local in memory benchmark.
This is only added to the file reader. The stream reader has no footer and cannot count rows
without consuming the stream.
### Are these changes tested?
Yes, two tests in `python/pyarrow/tests/test_ipc.py`:
* `test_file_count_rows`: count matches the sum of the written batch lengths, and counting does
not consume the reader (count, `read_all()`, count again).
* `test_file_count_rows_no_batches`: a file with a schema but no batches counts 0.
Locally `test_ipc.py` passes (72 tests) and `test_feather.py` passes (83 passed, 8 skipped,
1 xfailed), the latter because the feather reader sits on the same file reader. The docstring
example was run and produces the output shown.
### Are there any user-facing changes?
Yes, a new public method `RecordBatchFileReader.count_rows()`. No existing behaviour changes.
### AI usage disclosure
I used Claude Code to locate the unbound C++ method and the place where the declaration was
missing, and to draft the binding, the docstring and the tests. I reviewed the result, rebuilt
PyArrow locally, ran the test suites quoted above, and checked the C++ implementation of
`CountRows` myself to confirm what it actually does before describing the benefit here.
* GitHub Issue: #49305
Authored-by: Guja <127162872+GujaLomsadze@users.noreply.github.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
`pa.Array.to_pylist()` converts one element at a time through `Array::GetScalar` plus a Python `Scalar` wrapper; for list types each row additionally allocates a Python `Array` wrapper for the row's values slice and a fresh generator before recursing per element. A `sample` profile shows ~20% of runtime in CPython GC (triggered by the per-row GC-tracked allocations), ~25% in `GetScalar`, and only ~7% doing the useful work of creating the output objects — making `to_pylist` several times slower than converting via `to_pandas()` and back, and ~24x slower than `ndarray.tolist()` for plain int64. Details in #50326; this hit Apache Spark's Arrow-serialized Python UDFs (apache/spark#56940, apache/spark#56943).
### What changes are included in this PR?
Following review feedback, this adds a general scalar-free conversion mechanism instead of per-type `to_pylist` overrides:
- `Array` gains `cdef object _getitem_py(self, int64_t i)`, returning `self[i]` as a Python object. The base implementation is `GetScalar` + `Scalar.as_py`, so any type without a specialization behaves exactly as today (dates, times, timestamps, durations, decimals, dictionary, extension, unions, views, ...).
- The baseline `Array.to_pylist` becomes a single loop over `_getitem_py`. `maps_as_pydicts != None` keeps the Scalar-based path, since map→dict conversion has per-entry duplicate-key semantics.
- Specializations avoid all per-element Scalar and per-row Array-wrapper allocation:
- integers and floats (a `type_id` switch on `NumericArray`; date/time/timestamp subclasses fall through to the exact base),
- boolean,
- string/binary and large variants (`GetValue` + `PyUnicode_DecodeUTF8` / `PyBytes_FromStringAndSize`, matching `str(buf, 'utf8')` / `to_pybytes()` exactly),
- list/large_list/fixed_size_list (each row's list is built from the child's `_getitem_py` over the offset range; the wrapped child is cached on the parent array),
- map (association list of key/value tuples, matching `MapScalar.as_py`),
- struct (one dict per row; duplicate field names fall back to the Scalar path so they raise `ValueError` like `StructScalar.as_py`).
Nested types compose without any per-row wrappers. `ChunkedArray.to_pylist`, `Table.to_pylist` and `ListScalar.as_py` delegate here and speed up automatically. Follow-up candidates: string/binary views, run-end-encoded, dictionary, a fast path for date32.
Benchmarks (macOS arm64, M4 Max):
| benchmark | before | after | speedup |
|---|---|---|---|
| flat `int64` with nulls (4M) | 0.39 s | 0.028 s | 14x (~7 ns/element, on par with `ndarray.tolist`) |
| flat `string` (4M) | 0.83 s | 0.06 s | 14x |
| `list<string>` (2M rows) | 1.93 s | 0.46 s | 4.2x |
| `list<list<int32>>` (1M rows) | 2.10 s | 0.40 s | 5.2x |
| `struct<int64,string>` (1M rows) | 0.91 s | 0.07 s | 13x |
| `map<string,int64>` (1M rows) | 2.77 s | 0.74 s | 3.8x |
### Are these changes tested?
`test_to_pylist_bulk_paths` (added here) compares against the per-scalar conversion with exact element types for representative arrays including sliced views. Additionally verified with a randomized differential test against `[x.as_py() for x in arr]` with exact-type equality: all integer widths (incl. values beyond 2^62), floats (NaN/inf), boolean, string/binary (+large, multibyte), all list kinds, nested lists, struct (incl. empty struct, duplicate-field-name `ValueError`), map (incl. strict-mode duplicate-key `KeyError`), dictionary/null fallbacks, sliced/chunked arrays, and both `maps_as_pydicts` modes — no differences. `pytest test_array.py test_scalars.py test_convert_builtin.py test_table.py test_types.py`: 1295 passed.
### Are there any user-facing changes?
No behavior changes, only performance.
* GitHub Issue: #50326
This pull request and its description were written by Isaac.
Lead-authored-by: Liang-Chi Hsieh <liangchi.hsieh@databricks.com>
Co-authored-by: Isaac
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
There is currently no method to convert Arrow Table to Arrow Tensor (conversion from columnar format to a contiguous block of memory). This work is a continuation of `RecordBatch::ToTensor` work, see https://github.com/apache/arrow/issues/40058.
### What changes are included in this PR?
This PR:
- implements `Table::ToTensor` conversion
- adds bindings to Python
- adds benchmarks in C++
- removes the code in `RecordBatch::ToTensor` and uses the Table implementation (`RecordBatch::ToTensor` benchmarks checked)
### Are these changes tested?
Yes, in C++ and Python.
### Are there any user-facing changes?
No, it is a new feature.
* GitHub Issue: #40062
Lead-authored-by: AlenkaF <frim.alenka@gmail.com>
Co-authored-by: Alenka Frim <AlenkaF@users.noreply.github.com>
Co-authored-by: tadeja <tadeja@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Rok Mihevc <rok@mihevc.org>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
See #38584 for original PR. Will be quoted for this PR description.
### Rationale for this change
> support multi sortkey nulls first.
```
order by i nulls first, j, k nulls first;
```
> The current null sorting only supports all sortkeys, not a certain sortkey, so NullPlacement is extended to the SortKey field. Since the underlying framework is very well written, when modifying this function, you only need to pass the null_placement of each SortKey in. That’s it.
### What changes are included in this PR?
> 1.SortKey structure, NullPlacemnt transfer logic, sorting logic and Ording related, test related
> 2.Substriait related.
> 3.c_glib related.
> 4.SelectK related.
> 5.RankOptions related.
### Are these changes tested?
> yes, I changed the code inside vector_sort_test.cc and performed additional tests.
### Are there any user-facing changes?
> yes, pg database include null sorting of multiple sort keys.
**This PR includes breaking changes to public APIs.** (If there are any breaking changes to public APIs, please explain which changes are breaking. If not, you can remove this.)
I amended the original PR to be less breaking in public APIs.
Still Ordering, SortOptions, RankOptions, and RankQuantileOptions now accept a `std::optional<NullPlacement>` instead of NullPlacement, which did lead to some changes in downstream APIs and bindings. ~I also need some help with fixing the `c_glib` bindings.~
* GitHub Issue: #38558
Lead-authored-by: Alexander Taepper <alexander.taepper@gmail.com>
Co-authored-by: light-city <455954986@qq.com>
Co-authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
See https://github.com/apache/arrow/issues/47435.
### What changes are included in this PR?
Adds direct encryption / decryption Python API
### Are these changes tested?
Yes, see PR.
### Are there any user-facing changes?
Yes, new Python bindings.
* GitHub Issue: #47435
Authored-by: Sreesh Maheshwar <smaheshwar@palantir.com>
Signed-off-by: Adam Reeve <adreeve@gmail.com>
### Rationale for this change
`zoneinfo` is available starting with Python 3.9, so we can now assume
that it is available, and so we can switch from returning `pytz`
timezones by default to return `zoneinfo` timezones (or
`datetime.timezone` for fixed offsets).
Only keeping pytz as fallback for strings that are not supported by
`zoneinfo` but were supported by `pytz`. Later, we should maybe
deprecate that fallback.
Generally we should move away from using `pytz`, since the core
functionality of having time zones is now available in the standard
library (`zoneinfo`), and because the pytz package has several warts /
incompatibilities with stdlib datetime
(https://blog.ganssle.io/articles/2018/03/pytz-fastest-footgun.html)
### What changes are included in this PR?
Whenever we create a python timezone object, which is when converting to
pandas or when converting to a `datetime.datetime` object:
- always prefer `zoneinfo` for `datetime.datetime` objects
- prefer `zoneinfo` for pandas objects _if_ pandas >= 3, to align with
the change on the pandas side
(https://github.com/pandas-dev/pandas/issues/34916)
In either case, when preferring `zoneinfo`, we still fall back to `pytz`
for named timezones if `zoneinfo` does not recognize the zone name
(apparently pytz can have some common (older) aliases that might not
always work with zoneinfo).
This fallback is something we could deprecate and remove later on (so we
can eventually remove all usage of pytz)
### Are these changes tested?
Yes
### Are there any user-facing changes?
**This PR includes breaking changes to public APIs.**
It is a different object that we return (different class, i.e.
`zoneinfo.ZoneInfo` instead of a `pytz.tzinfo.BaseTzInfo`, both are
still subclasses of `datetime.tzinfo`), which has some differences in
the API, so for people relying on that, this is a breaking change.
For the conversion to pandas, pandas itself has made this breaking
change anyhow, so for those cases it aligns with that change of pandas.
* GitHub Issue: #15047
### Rationale for this change
Add an optional default_column_type parameter to the CSV reading API (C++ and Python) to provide a fallback type when per-column types aren’t specified, improving schema consistency and complementing the existing column_types logic.
### What changes are included in this PR?
- c++: new convert option "default_column_type" to augment logic around column_types parameter
- 3 reader tests: DefaultColumnTypePartialDefault, DefaultColumnTypeAllStringsWithHeader, DefaultColumnTypeAllStringsNoHeader). The last two tests are inspired by https://github.com/pandas-dev/pandas/pull/62242 and https://github.com/pandas-dev/pandas/issues/57666
- python: corresponding changes to make cpp change consumable from python
- python: extended test_convert_options test - include. default_column_type
- python: added new test "test_default_column_type" which tests how the field impacts schema; also test implicitly verifies leading zero preservation
- relevant documentation update for python component;
### Are these changes tested?
Yes. Existing and new tests are passing.
C++:
> [==========] Running 3 tests from 1 test suite.
> [----------] Global test environment set-up.
> [----------] 3 tests from ReaderTests
> [ RUN ] ReaderTests.DefaultColumnTypePartialDefault
> [ OK ] ReaderTests.DefaultColumnTypePartialDefault (3 ms)
> [ RUN ] ReaderTests.DefaultColumnTypeAllStringsWithHeader
> [ OK ] ReaderTests.DefaultColumnTypeAllStringsWithHeader (0 ms)
> [ RUN ] ReaderTests.DefaultColumnTypeAllStringsNoHeader
> [ OK ] ReaderTests.DefaultColumnTypeAllStringsNoHeader (0 ms)
> [----------] 3 tests from ReaderTests (4 ms total)
>
> [----------] Global test environment tear-down
> [==========] 3 tests from 1 test suite ran. (4 ms total)
> [ PASSED ] 3 tests.
All:
> [==========] 264 tests from 46 test suites ran. (452 ms total)
> [ PASSED ] 264 tests.
pyarrow:
New tests are passing.
### Are there any user-facing changes?
I believe this change is backward compatible. Parameter is optional and its default value doesn't change the existing behavior; All the existing rests are passing.
Maybe relevant: https://github.com/apache/arrow/issues/22232
Relates to https://github.com/apache/arrow/issues/47502
* GitHub Issue: #47502
* GitHub Issue: #22232
Lead-authored-by: Vlad Borovtsov <borovcov.vlad@gmail.com>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
`ColumnChunkMetaData.to_dict()` method omits `bloom_filter_offset` and `bloom_filter_length` even when a bloom filter is written to the Parquet file. Users cannot programmatically verify bloom filter presence via the Python metadata API without resorting to file size comparison.
### What changes are included in this PR?
1. `python/pyarrow/includes/libparquet.pxd`: Declare `bloom_filter_offset()` and `bloom_filter_length()` (both optional[int64_t]) on `CColumnChunkMetaData`. This is to expose the existing C++ methods to Cython.
2. `python/pyarrow/_parquet.pyx`: Add `bloom_filter_offset` and `bloom_filter_length` properties to `ColumnChunkMetaData` (returns int when set, None otherwise). Add both fields to `to_dict()` and `__repr__`.
3. `python/pyarrow/tests/parquet/test_metadata.py`: Add `test_bloom_filter_offset_in_metadata` verifying that columns with a bloom filter expose non-None integer values and that `to_dict()` contains the keys, while columns without a bloom filter return None.
### Are these changes tested?
Yes. `test_bloom_filter_offset_in_metadata` in test_metadata.py covers:
- Column with bloom filter: bloom_filter_offset and bloom_filter_length are non-None integers
- Column without bloom filter: both return None
- Both keys present in to_dict() output
<img width="863" height="215" alt="image" src="https://github.com/user-attachments/assets/d465d6bd-55d1-4c5f-9f11-6a60b3bf1cbe" />
Here is closer look on the logic output:
<img width="464" height="424" alt="image" src="https://github.com/user-attachments/assets/6e4810f2-c1c0-41ea-b559-00f99d42e2c4" />
output:
```python
col_a bloom_filter_offset: 10699
col_a bloom_filter_length: 1040
col_b bloom_filter_offset: None
col_b bloom_filter_length: None
col_a to_dict(): {'file_offset': 0, 'file_path': '', 'physical_type': 'BYTE_ARRAY', 'num_values': 1000, 'path_in_schema': 'a', 'is_stats_set': True, 'statistics': {'has_min_max': True, 'min': 'id_0', 'max': 'id_999', 'null_count': 0, 'distinct_count': None, 'num_values': 1000, 'physical_type': 'BYTE_ARRAY'}, 'geo_statistics': None, 'compression': 'SNAPPY', 'encodings': ('PLAIN', 'RLE', 'RLE_DICTIONARY'), 'has_dictionary_page': True, 'dictionary_page_offset': 4, 'data_page_offset': 4035, 'total_compressed_size': 5336, 'total_uncompressed_size': 11208, 'bloom_filter_offset': 10699, 'bloom_filter_length': 1040}
```
* GitHub Issue: #49927
Lead-authored-by: Haziq Hakimi Mazlisham <95128744+haziqishere@users.noreply.github.com>
Co-authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
The goal of the MR is to support the low level python raw fds (`os.open()`)
### What changes are in this PR?
As an alternative to the `str` path, the `pa.OSFile` accepts the `int` parameter as well.
### Are these changes tested?
Yes, new end-to-end tests were added
### Are there any user-facing changes?
Yes, the API is extended with a new parameter type
This MR was created by Codex Cloud.
* GitHub Issue: #49751
Authored-by: Ádám Lippai <adam@rigo.sk>
Signed-off-by: Antoine Pitrou <antoine@python.org>
Fixes#49376
### Rationale for this change
Adds ability to enable the writing of Parquet Bloom filters via pyarrow.
### What changes are included in this PR?
Adds `bloom_filter_options` to `parquet.write_table`.
### Are these changes tested?
Yes, new tests are added.
### Are there any user-facing changes?
Adds an option (defaults to `None`) to `parquet.write_table`.
* GitHub Issue: #49376
Authored-by: seidl <seidl2@llnl.gov>
Signed-off-by: mwish <maplewish117@gmail.com>
### Rationale for this change
Currently we are not building wheels with OpenTelemetry.
### What changes are included in this PR?
- Adding a new feature on our `vcpkg.json` to install required opentelemetry-cpp
- Enabling the opentelemetry feature when downloading dependencies via vcpkg for our wheels.
- Setting `ARROW_WITH_OPENTELEMETRY=ON` on the wheel build scripts
- Minor fix on checking exposed symbols (opentelemetry symbols are exposed)
- Minor improvement to `python/examples/flight/middleware.py` to be able to easily enable opentelemetry middleware tracing for testing.
- Fix on `server_tracing_middleware.cc` to retrieve tracer from `arrow::internal::tracing::GetTracer();` instead of `otel::trace::Provider::GetTracerProvider()->GetTracer("arrow");`
- Expose missing symbols to enable Windows builds
- Fix Windows compatibility adding windows compat header.
- Add `is_opentelemetry_enabled` utility function in pyarrow to validate whether OpenTelemetry was enabled or not when building Arrow C++. Expose this information on `pa.show_info()`.
### Are these changes tested?
Yes, wheels are being successfully built and tested locally with a built wheel that traces are being generated when properly configured.
Added test to validate traces appear on stdout and I've also validated Acero traces are being correctly generated by running the test suite and seeing the opentelemetry traces on Jaeger UI.
### Are there any user-facing changes?
Yes, PyArrow wheel users will be able to enable OpenTelemetry traces.
* GitHub Issue: #49382
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Fix https://github.com/apache/arrow/issues/49078
### What changes are included in this PR?
- Implement getters on the C++ side of `AzureOptions`, for the values that are currently stored only on the python side.
- This required adding some more member variables
- I decided to add `ClearCredentials` , so that it can't get into strange states by configuring one credential type then another. IMO configuring the credentials during initialisation on the `AzureOptions` would be neater but I don't want to make this PR too big.
- Update the C++ side `AzureOptions::Equals`
- Remove python side `self` attributes and instead depend on getters from C++ side.
### Are these changes tested?
- Updated tests on the C++ side for the updated `Equals` and newly added getter methods.
- Added a fixture `pickle_with_and_without_subtree_filesystem`, which can be used in place of the `pickle_module`. This adds test combinations with and without wrapping the filesystem in a `SubTreeFilesystem` before pickling it.
### Are there any user-facing changes?
Only that pickling `SubTreeFileSystem(base_path, AzureFileSystem(...))` now works properly.
* GitHub Issue: #49078
Authored-by: Thomas Newton <thomas.w.newton@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
`FileReader::ReadRowGroup(s)` previously returned `Status` and required callers to pass an `out` parameter.
### What changes are included in this PR?
Introduce `Result<std::shared_ptr<Table>>` returning APIs to allow clearer error propagation:
- Add new Result-returning `ReadRowGroup()` / `ReadRowGroups()` methods
- Deprecate the old Status/out-parameter overloads
- Update C++ callers and R/Python/GLib bindings to use the new API
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
Status versions of FileReader::ReadRowGroup(s) have been deprecated.
```cpp
virtual ::arrow::Status ReadRowGroup(int i, const std::vector<int>& column_indices,
std::shared_ptr<::arrow::Table>* out);
virtual ::arrow::Status ReadRowGroup(int i, std::shared_ptr<::arrow::Table>* out);
virtual ::arrow::Status ReadRowGroups(const std::vector<int>& row_groups,
const std::vector<int>& column_indices,
std::shared_ptr<::arrow::Table>* out);
virtual ::arrow::Status ReadRowGroups(const std::vector<int>& row_groups,
std::shared_ptr<::arrow::Table>* out);
```
* GitHub Issue: #48949
Lead-authored-by: fenfeng9 <fenfeng9@qq.com>
Co-authored-by: fenfeng9 <36840213+fenfeng9@users.noreply.github.com>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Co-authored-by: Gang Wu <ustcwg@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
`FileReader::ReadTable` previously returned `Status` and required callers to pass an `out` parameter.
### What changes are included in this PR?
Introduce `Result<std::shared_ptr<Table>>` returning API to allow clearer error propagation:
- Add new Result-returning `ReadTable()` methods
- Deprecate the old Status versions
- Migrate all callers to use the new API
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
Status version FileReader::ReadTable has been deprecated.
```cpp
virtual ::arrow::Status ReadTable(std::shared_ptr<::arrow::Table>* out);
virtual ::arrow::Status ReadTable(const std::vector<int>& column_indices,
std::shared_ptr<::arrow::Table>* out);
```
* GitHub Issue: #48394
Lead-authored-by: fenfeng9 <fenfeng9@qq.com>
Co-authored-by: fenfeng9 <36840213+fenfeng9@users.noreply.github.com>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
To close or discuss #48167
`inverse_permutation` and `scatter` functions got implemented via #44393, PR #44394.
### What changes are included in this PR?
Python tests for `scatter`, `inverse_permutation` kernels and bindings for `InversePermutationOptions` and `ScatterOptions`.
### Are these changes tested?
Yes, tests added in test_compute.py.
### Are there any user-facing changes?
Bindings for `InversePermutationOptions` and `ScatterOptions` are added.
#### This PR includes breaking changes to public APIs.
Options `InversePermutationOptions` changed from accepting parameter
`std::shared_ptr<DataType> output_type = NULLPTR`
to
`std::optional<std::shared_ptr<DataType>> output_type = std::nullopt`
* GitHub Issue: #48167
Lead-authored-by: Tadeja Kadunc <tadeja.kadunc@gmail.com>
Co-authored-by: Rossi Sun <zanmato1984@gmail.com>
Co-authored-by: tadeja <tadeja@users.noreply.github.com>
Signed-off-by: Rossi Sun <zanmato1984@gmail.com>
### Rationale for this change
See #48096, exposes `parquet.WriterProperties max_rows_per_page` argument to Python's API.
### What changes are included in this PR?
Added the argument
### Are these changes tested?
Yes, since the metadata doesn't have any info about the number of pages, a naive end-to-end test was used to ensure the implementation correctness.
### Are there any user-facing changes?
The ability to set the `max_rows_per_page` directly from PyArrow.
* GitHub Issue: #48096
Authored-by: Bogdan Romenskii <rmnsk@seznam.cz>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Enables external key material and rotation for individual parquet files in PyArrow. This change does not address any parquet dataset encryption functionality.
### What changes are included in this PR?
This PR enables external key material for parquet encryption from PyArrow:
Optional parquet_file_path and FileSystem paramters to CryptoFactory - mirroring the interface for CryptoFactory in C++
1. Exposes the rotate_master_keys method of CryptoFactory
2. Adds Cython classes for FileKeyMaterialStore, FileSystemKeyMaterialStore, and KeyMaterial - but does not expose these from PyArrow encryption. I included these changes only so that a unit test may verify an external store without leaking the implementation details for the store into the test.
### Are these changes tested?
Yes - I've modified an existing test (previously marked pytest.xfail) to do a basic read write test and verify creation of the external key material store and added a test for CryptoFactory.rotate_master_keys.
### Are there any user-facing changes?
1. Users may optionally supply a parquet file path and FileSystem to CryptoFactory methods that provide en/decryption_properties. Doing so in conjunction with setting EncryptionConfiguration.internal_key_material=False enables external key material from pyarrow.
2. PyArrow CryptoFactory now has a rotate_master_keys method exposing key rotation functionality from C++ CryptoFactory.
* GitHub Issue: #31869
Lead-authored-by: Patrick Parsons <114018+pfparsons@users.noreply.github.com>
Co-authored-by: Adam Reeve <adreeve@gmail.com>
Signed-off-by: Adam Reeve <adreeve@gmail.com>
### Rationale for this change
Enable `ARROW:null_count:approximate` support for `arrow::ArrayStatistics`, along with the corresponding GLib, Ruby and Python bindings.
### What changes are included in this PR?
Enable `ARROW:null_count:approximate` in C++ and bind it to `ArrayStatistics` in GLib, Ruby and Python.
### Are these changes tested?
Yes, I ran the relevant unit tests.
### Are there any user-facing changes?
Yes.
* The type of `arrow::ArrayStatistics::null_count` has been changed from `std::optional<int64_t>` to `std::optional<CountType>`
* New `garrow_array_statistics_is_null_count_exact()`/`garrow_array_statistics_get_null_count_{exact,approximate}()` functions in GLib.
* Add support for approximate value in `Arrow::ArrayStatistics#null_count` in Ruby.
* A new field `is_null_count_exact` has been added to `ArrayStatistics` in Python.
* GitHub Issue: #47103
Lead-authored-by: arash andishgar <arashandishgar1@gmail.com>
Co-authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
See GH-47861. With this change, the extension array variation takes ~192MB of memory instead of 7GB.
From what I can tell, this is because the `PandasOptions` struct is copied around frequently (for example it seems like there is an `ExtensionWriter` for each extension column and each `ExtensionWriter` has a copy of `PandasOptions` which has a set of all extension columns). I haven't fully traced the PandasOptions structure, but it seems to get copied and modified in some codepaths so I have decided to put the column sets into a `std::shared_ptr` rather than pass around a `shared_ptr<PandasOptions>`.
### What changes are included in this PR?
The `PandasOptions` column sets have been swapped from `std::unordered_set<std::string>` to `std::shared_ptr<const std::unordered_set<std::string>>` and usages have been updated.
### Are these changes tested?
Yes, no regression in the pytests. Also tested memory usage by hand.
### Are there any user-facing changes?
All changes are internal to the pyarrow C++ binding code. There are no changes to the exposed Python API.
* GitHub Issue: #47861
Lead-authored-by: Will Gulian <williamgulian@gmail.com>
Co-authored-by: Will Gulian <git@will.gl>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Please see #47441 and #41476.
The `ArrowWriterProperties.write_time_adjusted_to_utc` flag is available in C++, yet isn't accessible from Python. This PR introduces the said flag to Python API as well.
### What changes are included in this PR?
Exposure of `use_time_adjusted_to_utc` boolean argument in Python's API.
### Are these changes tested?
Yes, roundtrip parquet tests for all combinations of time types and their respective time units.
### Are there any user-facing changes?
The users will be able to adjust the said flag directly from Python API.
* GitHub Issue: #47441
Lead-authored-by: Bogdan Romenskii <rmnsk@seznam.cz>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Expose CSV writer option quoting_header for pyarrow. Addresses https://github.com/apache/arrow/issues/47575
### What changes are included in this PR?
Cython changes for parsing quoting_header option in a manner similar to quoting_style
### Are these changes tested?
Yes, added a unit test under test_csv.py
### Are there any user-facing changes?
Add QuotingStyle quoting_header option in WriteOptions for pyarrow
* GitHub Issue: #47575
Authored-by: Ayush Bansal <abansal1008@gmail.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
Please see #47358
### What changes are included in this PR?
Added `__repr__` dunder methods to `ipc.IpcWriteOptions`, `ipc.IpcReadOptions` and `flight.FlightCallOptions`
### Are these changes tested?
Yes
### Are there any user-facing changes?
Nicer representations of the classes mentioned above
* GitHub Issue: #47358
Lead-authored-by: Bogdan Romenskii <rmnsk@seznam.cz>
Co-authored-by: Bogdan Romenskii <77816165+rmnskb@users.noreply.github.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
Passing encryption keys to Parquet via strings is insecure. Users are encouraged to use the `SecureString` based methods introduced in #46017 instead. To enforce this migration, deprecated methods are removed.
### What changes are included in this PR?
Remove deprecated string-based methods.
### Are these changes tested?
Existing Parquet encryption tests.
### Are there any user-facing changes?
C++ user code that passes encryption keys to Parquet is affected and has to be adjusted. Python user code is not affected.
**This PR includes breaking changes to public APIs.**
* GitHub Issue: #47338
Authored-by: Enrico Minack <github@enrico.minack.dev>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Now that we have implemented dictionary replacement / dictionary deltas on DoGet/DoExchange, exposing ReadStatistics can help us easily understand / debug / diagnose much better what are the different messages being sent.
### What changes are included in this PR?
Add API to expose `arrow::ipc::ReadStats stats()` on `MetadataRecordBatchReader`. As we are mainly using `RecordBatchStreamReader` everywhere now, the stats are already populated so we just have to expose them.
Add also Python bindings.
### Are these changes tested?
Yes via several Python tests we are validating the stats are retrieved and the results are the expected ones.
### Are there any user-facing changes?
No breaking changes, new API to retrieve Statistics from the readers.
* GitHub Issue: #47422
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Implement support for the `ARROW:distinct_count:approximate` statistics attribute.
### What changes are included in this PR?
Changed the type of `arrow::ArrayStatistics::distinct_count` to support both `double` (for approximate values) and `int64_t` (for exact values).
### Are these changes tested?
Yes, I ran the corresponding unit test.
### Are there any user-facing changes?
Yes.
* C++: The type of `arrow::ArrayStatistics::distinct_count` was changed from `std::optional<int64_t>` to `std::optional<std::variant<int64_t, double>>`.
* GLib: `garrow_array_statistics_get_distinct_count()` is deprecated.
* GitHub Issue: #47101
Lead-authored-by: Arash Andishgar <arashandishgar1@gmail.com>
Co-authored-by: Sutou Kouhei <kou@clear-code.com>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Co-authored-by: Rok Mihevc <rok@mihevc.org>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
Please see Github Issue #47123
### What changes are included in this PR?
Added public Type Enums that mimic the original private variable groups used for internal type checking.
### Are these changes tested?
Yes. Partly for now.
### Are there any user-facing changes?
No, just additional features were added: they will now able to access the underlying types directly via the Type Enums.
* GitHub Issue: #47123
Lead-authored-by: Bogdan Romenskii <rmnsk@seznam.cz>
Co-authored-by: Alenka Frim <AlenkaF@users.noreply.github.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
As mentioned in #46728, if Arrow C++ was built debug, and PyArrow wasn't, test_gdb.py runs tests that fail.
### What changes are included in this PR?
The CMAKE_BUILD_TYPE environment variable is propagated from build into PyArrow, where it's checked to skip unit tests.
### Are these changes tested?
Yes. I have built PyArrow in release, debug, and relwithdebinfo and observed the new behavior. Because CMakeLists.txt was changed, I built PyArrow twice via setup.py and pip install, and checked the new function.
### Are there any user-facing changes?
Devs may skip unit tests that would fail. PyArrow now has build_info() with information about the build type.
* GitHub Issue: #46728
Lead-authored-by: Eric Dinse <293818+dinse@users.noreply.github.com>
Co-authored-by: Antoine Pitrou <antoine@python.org>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
See https://github.com/apache/arrow/issues/46629.
### What changes are included in this PR?
This PR updates the `DatasetFactory.inspect` method so that it accepts new `promote_options` and `fragments` parameters. Since we parse string into a `MergeOptions` struct in three different places, this PR defines the helper function `_parse_field_merge_options`.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
This adds optional parameters to a public method. It changes the default behavior from checking one fragment to checking all fragments (the old documentation said it inspected "all data fragments" even though it didn't).
* GitHub Issue: #46629
Lead-authored-by: Hadrian Reppas <hadrian@hudson-trading.com>
Co-authored-by: Hadrian Reppas <hadrian@uchicago.edu>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### What does this PR do?
Adds a new compute kernel `utf8_zero_fill` to Arrow Compute module. This kernel zero-pads UTF-8 strings while preserving leading signs.
### Why is it needed?
Python's `str.zfill()` behavior is useful and expected in many data cleaning scenarios. Arrow lacked a direct equivalent.
### Implementation details
- Introduces `Utf8ZeroFillTransform` with dedicated `ZeroFillOptions`
- Registers `utf8_zero_fill` as a compute kernel
- Adds Python bindings and a clean alias: `utf8_zfill = utf8_zero_fill`
- Includes sign-aware padding logic
### Are these changes tested?
- C++ unit tests: `TestStringKernels, Utf8ZeroFill`
- Python tests: `test_utf8_zfill()`
Ran both tests
* GitHub Issue: #46683
Authored-by: iabhi4 <iamonecool@gmail.com>
Signed-off-by: Antoine Pitrou <pitrou@free.fr>
### Rationale for this change
`ClientSecretCredential` is another method to authenticate to Azure resources using Service Principle (SP). We work on AzureML, where there are two main types of compute ComputeInstance (personal VM) and ComputeCluster. In context of AzureML you can work interactively on ComputeInstance, where pyarrow defaults to`DefaultAzureCredential` when reading/writing data over `abfss://` protocol. However when running AzureML "jobs" which is run non-interactively execution, the context (network) is different and `DefaultAzureCredential` doesn't work. An alternative method for non-interactive job is to use Service Principle i.e `ClientSecretCredential`. This way we can create `AzureFileSystem` object and give it pyarrow `dataset`
### What changes are included in this PR?
Adding ConfigureClientSecretCredential to AzureFilesystem, which is already implemented at the C++, but hasn't been propagated to python library. This pull request just propagates C++ method to pyarrow
### Are these changes tested?
Yes, test have been included in the relevant file
### Are there any user-facing changes?
Docs of `AzureFileSystem` have been updated in the code. It would be amazing for those changes to surface on the website, which I'm happy to try to help with if needed
* GitHub Issue: #46833
Lead-authored-by: Kirill Tsyganov <kirill.tsyganov@outlook.com.au>
Co-authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Currently, when using the pyarrow.fs.S3FileSystem, it is not immediately obvious how provide an alternative TLS certificate authority when working with (for example) a self-hosted minio using a self-signed certificate. It is possible to do this via environment variables, as per #40754; however, the user may not be able to easily specify these depending on their python execution environment.
### What changes are included in this PR?
Adds an additional parameter to S3FileSystem to specify the file path to a custom certificate.
### Are these changes tested?
Yes. A unit-test for S3FileSystem pickling is included; and I have manually verified that I can access files from a self-hosted minio with TLS enabled.
### Are there any user-facing changes?
An additional parameter `tls_ca_file_path` is available on the S3FileSystem constructor.
* GitHub Issue: #40754
Authored-by: bw513 <192128226+bw513@users.noreply.github.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
When slicing arrays with non-trivial steps we were using `numpy.arange` to generate the indices for take. As numpy is an optional dependency, implementing it via Python caused a performance penalty. Creating a pyarrow function to build our own ranges that mimics Python range or numpy arange is useful for that uses case and might also be useful for other use cases. Currently we only generate `Array[Int64]` we could potentially generate more types.
### What changes are included in this PR?
provide a `pa.arange` function that allows us to generate indices when slicing arrays.
### Are these changes tested?
Yes new tests added.
### Are there any user-facing changes?
No but a new pyarrow.arange function has been added.
* GitHub Issue: #46771
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Producing part of DLPack protocol [has been added to Arrow Arrays](https://github.com/apache/arrow/issues/33984) but is missing on Arrow Tensor class.
### What changes are included in this PR?
This PR adds support for producing DLPack struct and bindings to it in Python:
- `ExportTensor` and `ExportTensorDevice` methods on Arrow C++ Tensor class
- `__dlpack__` method on PyArrow Tensor class exposing ExportTensor method
- `__dlpack_device__` method on PyArrow Tensor class exposing ExportTensorDevice method
### Are these changes tested?
Yes
### Are there any user-facing changes?
No
* GitHub Issue: #39294
Lead-authored-by: AlenkaF <frim.alenka@gmail.com>
Co-authored-by: Alenka Frim <AlenkaF@users.noreply.github.com>
Co-authored-by: Rok Mihevc <rok@mihevc.org>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
Arrow is quite a heavy dependency and some users don't need all the tools we provide bundled with libarrow. Moving Arrow Compute to its own shared library allows users installations that better suit their needs having smaller memory footprint if necessary.
It might also help some users adding new kernels into an existing Arrow without recompiling it.
### What changes are included in this PR?
- Move all the Arrow Compute kernel functions to a new `ArrowCompute` shared library (`libarrow_compute.so`).
- Create a new API to Initialize arrow compute registering the existing Kernels into the `FunctionRegistry`
- Update Python/R/CGLib bindings to automatically register the Compute kernels transparently to the user.
- Update Linux Packaging to provide the new arrow-compute library.
- Update documentation with new requirements to call `arrow::compute::Initialize()`
### Are these changes tested?
Yes on all CI jobs.
### Are there any user-facing changes?
Yes. The Arrow compute functions will be provided as a different library. Any user using Arrow Compute from C++ directly will require a call to `arrow::compute::Initialize()` in order for the functions and kernels to be registered
**This PR includes breaking changes to public APIs.**
* GitHub Issue: #25025
Lead-authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Co-authored-by: Bryce Mecum <petridish@gmail.com>
Co-authored-by: Sutou Kouhei <kou@clear-code.com>
Co-authored-by: Rossi Sun <zanmato1984@gmail.com>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
#46403
### What changes are included in this PR?
A new PrettyPrinter option is added to limit elements to 100 characters by default.
### Are these changes tested?
Yes
### Are there any user-facing changes?
Yes, the default length for outputted elements when stringifying them is now different so if a user was relying on ToString of an array with large elements that result may now be changed.
* GitHub Issue: #46403
Lead-authored-by: David Sherrier <davidsherrier1997@gmail.com>
Co-authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
When reading a Parquet LIST logical type (or a repeated field without a logical type), Parquet C++ automatically reads it as a Arrow List array.
However, this can in some cases run into the 32-bit offsets limit. We'd like to be able to choose to read as LargeList instead, even if there is no serialized Arrow schema in the Parquet file.
### What changes are included in this PR?
* Add a Parquet read option `list_type` to select which Arrow type to read LIST / repeated Parquet columns into
* Fix an index truncation bug when writing a huge single chunk of data to Parquet
### Are these changes tested?
Yes, the functionality is tested. However, I wasn't able to write a unit test that wouldn't consume a horrendous amount of time or memory writing/reading a list with offsets larger than 2**32.
### Are there any user-facing changes?
No, only an API improvement.
* GitHub Issue: #46676
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
When we added Float16 we did not update pyarrow to be able to convert from Python objects to Arrow. Float16 required numpy and it crashed if numpy was not present.
### What changes are included in this PR?
Allow to not require numpy to generate float16 scalars and arrays on pyarrow and do not fail if numpy is not present.
### Are these changes tested?
Yes, new tests have been added
### Are there any user-facing changes?
No changes for old functionality. Users will be allowed to use float16 without requiring to use np.float16 and directly from Python objects
* GitHub Issue: #46611
Lead-authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
