### Rationale for this change
`pa.show_info()` prints its Compression Codecs section from a hardcoded list:
```python
codecs = ["brotli", "bz2", "gzip", "lz4_frame", "lz4", "snappy", "zstd"]
```
That list has drifted from `_ensure_compression()` in `io.pxi`, which is what
actually decides which codec names work. Two problems follow:
- `lz4` and `lz4_frame` both map to `CCompressionType_LZ4_FRAME`, so the same
codec is reported twice under two names.
- `lz4_raw` maps to `CCompressionType_LZ4` and is a separate codec, but it is
not listed at all, even though `pa.Codec.is_available('lz4_raw')` is True.
So seven lines are printed for six codecs, and the codec that is missing is a
real one.
Before:
```
Compression Codecs:
brotli : Enabled
bz2 : Enabled
gzip : Enabled
lz4_frame : Enabled
lz4 : Enabled
snappy : Enabled
zstd : Enabled
```
After:
```
Compression Codecs:
brotli : Enabled
bz2 : Enabled
gzip : Enabled
lz4 : Enabled
lz4_raw : Enabled
snappy : Enabled
zstd : Enabled
```
### What changes are included in this PR?
One list in `python/pyarrow/__init__.py`: drop the duplicate `lz4_frame` and add
the missing `lz4_raw`. `lz4` is kept as the name for the frame codec since that
is the one the docstrings lead with (`'lz4' (or 'lz4_frame')`).
On deriving this programmatically rather than hardcoding it — agreed, that is
the better fix. The canonical mapping lives in `_ensure_compression()` in
`io.pxi`, and neither `Codec` nor the C++ `compression.h` currently exposes the
set of supported codecs, so it needs a new accessor on the Cython side for
`show_info()` to read. I am happy to do that here, with the caveat that I cannot
compile the Cython extension in my environment and would be relying on CI to
verify it. If you would prefer, I can keep this change minimal and open a
follow-up issue for the programmatic version. Let me know which you would rather
have.
### Are these changes tested?
Not by a new test. `show_info()` has no existing test coverage and its output is
a diagnostic print, so I did not want to pin the exact text without knowing
whether you would want that — happy to add one if you do.
Verified by applying the same change to an installed pyarrow and reading the
output back, and by walking every name `_ensure_compression()` accepts to
confirm which of them map onto the same C++ enum value:
```
bz2 -> BZ2, gzip -> GZIP, brotli -> BROTLI,
lz4 -> LZ4_FRAME, lz4_frame -> LZ4_FRAME, # same codec, two names
lz4_raw -> LZ4, # distinct, was not listed
snappy -> SNAPPY, zstd -> ZSTD
```
### Are there any user-facing changes?
Only the output of `pa.show_info()`, which now lists each supported codec once
and includes `lz4_raw`. No API change.
### Was AI used for this PR?
In accordance to the [AI generation guidelines](https://arrow.apache.org/docs/dev/developers/overview.html#ai-generated-code), please disclose below whether and how AI was used in this PR.
**PR code and description written by:**
- [x] Human
- [ ] AI
**Reviewed before submission by:**
- [x] Human
- [ ] AI
- [ ] Not reviewed
* GitHub Issue: #52024
Authored-by: nishad shabbir <nishadshabbir28@gmail.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
`ObjectType` and `FileStatistics` in io/hdfs.h have been deprecated for a while and can be removed.
### What changes are included in this PR?
`ObjectType` and `FileStatistics` structs are removed and instead FileSystem API in `arrow::fs` is used. Together with this change, the hdfs connected code is moved from `cpp/src/arrow/io` to `cpp/src/arrow/filesystem` merging `FileSystem` and `HadoopFileSystem` classes from `arrow::io` into the public `HadoopFileSystem` class.
### Are these changes tested?
Existing tests should pass.
### Are there any user-facing changes?
Deprecated structs are removed and all hdfs related code is now a part of the filesystem module.
* GitHub Issue: #45747
Also closes: https://github.com/apache/arrow/issues/22457 (not sure about `io/interfaces.h`?)
Lead-authored-by: AlenkaF <frim.alenka@gmail.com>
Co-authored-by: Alenka Frim <AlenkaF@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
With `scikit-build-core` our Cython tests fail to find PyArrow C++ headers due to get_include returning a non-existent directory.
### What changes are included in this PR?
Editable builds location of libraries and headers are located at ``
Previous to this change:
```
$ python -c "import pyarrow as pa; print(pa.get_include())"
/home/raulcd/code/arrow/python/pyarrow/include
$ ls /home/raulcd/code/arrow/python/pyarrow/include
ls: cannot access '/home/raulcd/code/arrow/python/pyarrow/include': No such file or directory
```
with this change:
```
$ python -c "import pyarrow as pa; print(pa.get_include())"
/home/raulcd/code/pyarrow-dev/lib/python3.13/site-packages/pyarrow/include
$ ls /home/raulcd/code/pyarrow-dev/lib/python3.13/site-packages/pyarrow/include
arrow
```
### Are these changes tested?
Yes, locally for editable installs and on CI for non-editable.
### Are there any user-facing changes?
The only change is that headers, generated Cython cpp files and built shared libraries are installed on the virtualenv.
* GitHub Issue: #49473
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Currently we are not building wheels with OpenTelemetry.
### What changes are included in this PR?
- Adding a new feature on our `vcpkg.json` to install required opentelemetry-cpp
- Enabling the opentelemetry feature when downloading dependencies via vcpkg for our wheels.
- Setting `ARROW_WITH_OPENTELEMETRY=ON` on the wheel build scripts
- Minor fix on checking exposed symbols (opentelemetry symbols are exposed)
- Minor improvement to `python/examples/flight/middleware.py` to be able to easily enable opentelemetry middleware tracing for testing.
- Fix on `server_tracing_middleware.cc` to retrieve tracer from `arrow::internal::tracing::GetTracer();` instead of `otel::trace::Provider::GetTracerProvider()->GetTracer("arrow");`
- Expose missing symbols to enable Windows builds
- Fix Windows compatibility adding windows compat header.
- Add `is_opentelemetry_enabled` utility function in pyarrow to validate whether OpenTelemetry was enabled or not when building Arrow C++. Expose this information on `pa.show_info()`.
### Are these changes tested?
Yes, wheels are being successfully built and tested locally with a built wheel that traces are being generated when properly configured.
Added test to validate traces appear on stdout and I've also validated Acero traces are being correctly generated by running the test suite and seeing the opentelemetry traces on Jaeger UI.
### Are there any user-facing changes?
Yes, PyArrow wheel users will be able to enable OpenTelemetry traces.
* GitHub Issue: #49382
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### What changes are included in this PR?
Remove unused imports from `python/pyarrow/__init__.py`
### Are these changes tested?
Existing tests
### Are there any user-facing changes?
No
* GitHub Issue: #48667
Authored-by: Ruifeng Zheng <ruifengz@apache.org>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
As mentioned in #46728, if Arrow C++ was built debug, and PyArrow wasn't, test_gdb.py runs tests that fail.
### What changes are included in this PR?
The CMAKE_BUILD_TYPE environment variable is propagated from build into PyArrow, where it's checked to skip unit tests.
### Are these changes tested?
Yes. I have built PyArrow in release, debug, and relwithdebinfo and observed the new behavior. Because CMakeLists.txt was changed, I built PyArrow twice via setup.py and pip install, and checked the new function.
### Are there any user-facing changes?
Devs may skip unit tests that would fail. PyArrow now has build_info() with information about the build type.
* GitHub Issue: #46728
Lead-authored-by: Eric Dinse <293818+dinse@users.noreply.github.com>
Co-authored-by: Antoine Pitrou <antoine@python.org>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
When slicing arrays with non-trivial steps we were using `numpy.arange` to generate the indices for take. As numpy is an optional dependency, implementing it via Python caused a performance penalty. Creating a pyarrow function to build our own ranges that mimics Python range or numpy arange is useful for that uses case and might also be useful for other use cases. Currently we only generate `Array[Int64]` we could potentially generate more types.
### What changes are included in this PR?
provide a `pa.arange` function that allows us to generate indices when slicing arrays.
### Are these changes tested?
Yes new tests added.
### Are there any user-facing changes?
No but a new pyarrow.arange function has been added.
* GitHub Issue: #46771
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
`PyExtensionType` has been deprecated in https://github.com/apache/arrow/pull/38608 and can now be removed.
### What changes are included in this PR?
`PyExtensionType` functionality is removed from PyArrow code.
### Are these changes tested?
Existing tests should pass.
### Are there any user-facing changes?
Deprecated classes are removed.
* GitHub Issue: #46198
Authored-by: AlenkaF <frim.alenka@gmail.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
To run `test_cython` on Windows, we need a copy of msvcp140.dll in the library search path.
### What changes are included in this PR?
- Updated `pa.get_library_dirs()` behavior to add the `pyarrow.libs` folder to the library search path
### Are these changes tested?
Not yet.
### Are there any user-facing changes?
Only a bugfix.
Closes#45530
* GitHub Issue: #45530
Authored-by: Bryce Mecum <petridish@gmail.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
We've accumulated multiple workarounds for various Cython issues over the years.
Now that we require at least Cython 3.0, we can remove some of those.
### Are these changes tested?
Yes, by existing CI builds and tests.
### Are there any user-facing changes?
No.
* GitHub Issue: #45433
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Arrow C++ and the Arrow specification now support 32-bit and 64-bit decimal types...pyarrow should too!
### What changes are included in this PR?
Added type, array, and scalar bindings.
### Are these changes tested?
Yes!
### Are there any user-facing changes?
Yes!
* GitHub Issue: #44713
Authored-by: Dewey Dunnington <dewey@fishandwhistle.net>
Signed-off-by: Dewey Dunnington <dewey@fishandwhistle.net>
### Rationale for this change
We [added canonical JsonExtensionType](https://github.com/apache/arrow/pull/13901) and we should make it usable from Python.
### What changes are included in this PR?
Python wrapper for `JsonExtensionType` and `JsonArray` are added on Python side as well as `JsonArray` on c++ side.
### Are these changes tested?
Python tests for the extension type and array are included.
### Are there any user-facing changes?
This adds a json canonical extension type to pyarrow.
* GitHub Issue: #44066
Lead-authored-by: Rok Mihevc <rok@mihevc.org>
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Allows to concatenate recordbatches in Python
### What changes are included in this PR?
Adds `concat_batches` function and tests
### Are these changes tested?
yes
### Are there any user-facing changes?
A new public function has been added
* GitHub Issue: #44125
---------
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
### Rationale for this change
See #15058.
UUID datatype is common in throughout the ecosystem and Arrow as supporting it as a native type would reduce friction.
### What changes are included in this PR?
This PR implements logic for Arrow canonical extension type in C++ and a Python wrapper.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes, new extension type is added.
* Closes: #15058
Authored-by: Rok Mihevc <rok@mihevc.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
C++ and Python implementations of #43234
### What changes are included in this PR?
- Implement C++ `Bool8Type`, `Bool8Array`, `Bool8Scalar`, and tests
- Implement Python bindings to C++, as well as zero-copy numpy conversion methods
- TODO: docs waiting for rebase on #43458
### Are these changes tested?
Yes
### Are there any user-facing changes?
Bool8 extension type will be available in C++ and Python libraries
* GitHub Issue: #17682
Authored-by: Joel Lubinitsky <joellubi@gmail.com>
Signed-off-by: Felipe Oliveira Carvalho <felipekde@gmail.com>
### Rationale for this change
Add the newly ratified extension type.
### What changes are included in this PR?
The C++/Python implementation only.
### Are these changes tested?
Yes
### Are there any user-facing changes?
No.
* GitHub Issue: #43454
Lead-authored-by: David Li <li.davidm96@gmail.com>
Co-authored-by: Weston Pace <weston.pace@gmail.com>
Signed-off-by: David Li <li.davidm96@gmail.com>
### Rationale for this change
Add bindings for the C++ `arrow::Device` and `arrow::MemoryManager` classes.
### What changes are included in this PR?
Basic bindings by adding the `pyarrow.Device` and `pyarrow.MemoryManager` classes, and just tested for CPU.
What is not included here are additional methods on the `MemoryManager` class (eg to allocate or copy buffers), and this is also not yet tested for CUDA. Planning to do this as follow-ups, and first doing those basic bindings should enable further enhancements to be done in parallel.
### Are these changes tested?
Yes, for the CPU device only.
* GitHub Issue: #41126
Authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
### Rationale for this change
We want to use the new `AzureFileSystem` in `pyarrow`.
### What changes are included in this PR?
- Add minimal python bindings for `AzureFileSystem`. This includes just enough to run the python tests against azurite plus default credential auth to enable real use of this once this PR merges.
- Adding additional configuration options and remaining authentication options can be done as a follow up.
- I tried to copy the existing pybinds for GCS and S3
- Explicitly set `ARROW_AZURE=OFF` rather than relying on defaults. The defaults are different for builds vs tests so this was causing tests to be enabled while Azure was disabled during the build.
### Are these changes tested?
Enabled the the python filesystem tests for the new filesystem. I had to skip azure in a couple of the tests though because they are not yet working on the C++ side. I created Github issues to resolve these https://github.com/apache/arrow/issues/40025 and https://github.com/apache/arrow/issues/40026 and added TODO comments where relevant, that reference these Github issues.
### Are there any user-facing changes?
`pyarrow` users can now use the native `AzureFileSystem` to get much better reliability and performance compared to `adlfs` based options.
* Closes: #39968
* GitHub Issue: #39968
Lead-authored-by: Thomas Newton <thomas.w.newton@gmail.com>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
### Rationale for this change
Add bindings to the ListView and LargeListView array formats.
### What changes are included in this PR?
* Add initial implementation for ListView and LargeListView
* Add basic unit tests
### Are these changes tested?
* Basic unit tests only (follow up PRs will be needed to implement full functionality)
### Are there any user-facing changes?
Yes, documentation is updated in this PR to include the new PyArrow objects.
* Closes: #39812
Lead-authored-by: Dane Pitkin <dane@voltrondata.com>
Co-authored-by: Dane Pitkin <48041712+danepitkin@users.noreply.github.com>
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
### Rationale for this change
First step for https://github.com/apache/arrow/issues/39633: exposing the Array, DataType and Scalar classes for BinaryView and StringView, such that those can already be represented in pyarrow.
(I exposed a variant of StringBuilder as well, just for now to be able to create test data)
* Closes: #39651
Authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
### Rationale for this change
Resolves: https://github.com/apache/arrow/issues/36441
### What changes are included in this PR?
- Add python bindings for `CacheOptions` from the C++ side.
- Allow setting `cache_options` on `ParquetFragmentScanOptions` from the python side.
- Adjust some of the comments on `CacheOptions`
### Are these changes tested?
Yes. I added python side tests for these newly available configs similar to other configs. I have not added an integration test that ensures setting the configs on the python side leads to correctly using them on the C++ side.
### Are there any user-facing changes?
Yes. The are new configs available on the python side but the defaults are unchanged. I've added/updated docstrings where relevant.
* Closes: #36441
Lead-authored-by: Thomas Newton <thomas.w.newton@gmail.com>
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
### Rationale for this change
Add a function to change the path where timezone db should be found as a small wrapper around the setting of a C++ option `GlobalOptions`.
### What changes are included in this PR?
New function `configure_tzdb`.
### Are these changes tested?
### Are there any user-facing changes?
No.
* Closes: #35600
Lead-authored-by: AlenkaF <frim.alenka@gmail.com>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Co-authored-by: Alenka Frim <AlenkaF@users.noreply.github.com>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
### Rationale for this change
These concrete types appear to have been overlooked.
### What changes are included in this PR?
n/a
### Are these changes tested?
n/a
### Are there any user-facing changes?
Yes - I didn't rebuild docs, do I need to do that?
* Closes: #35413
Authored-by: Spencer Nelson <spencer@b612foundation.org>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
### Rationale for this change
In the C++ the fixed shape tensor canonical extension type is implementated https://github.com/apache/arrow/pull/8510 so we can add bindings to the extension type in Python.
### What changes are included in this PR?
Binding for fixed shape tensor canonical extension type.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
* Closes: #34882
Lead-authored-by: Alenka Frim <frim.alenka@gmail.com>
Co-authored-by: Alenka Frim <AlenkaF@users.noreply.github.com>
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Co-authored-by: Rok Mihevc <rok@mihevc.org>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
### Rationale for this change
Follow-up on https://github.com/apache/arrow/pull/34570 (exposing the new RunEndEncoded array and type in pyarrow) to also add the scalar class.
### Are there any user-facing changes?
The example from the issue now works:
```
In [15]: run_ends = [3, 5, 10, 19]
...: values = [1, 2, 1, 3]
...: ree_array = pa.RunEndEncodedArray.from_arrays(run_ends, values)
In [16]: ree_array[0]
Out[16]: <pyarrow.RunEndEncodedScalar: 1>
```
* Closes: #34686
Authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
### Rationale for this change
We can't maintain Plasma.
### What changes are included in this PR?
These changes remove all Plasma related codes.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* Closes: #33243
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
pyarrow.get_libraries() returns ['arrow', 'arrow_python'] but it should be ['arrow_python', 'arrow'] because libarrow_python.so depends on libarrow.so.
* Closes: #14943
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
A first set of easy removals of deprecated functions / classes in the top-level namespace and the `pyarrow.compat` module.
Authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
Even though the GDB extension should theoretically work with a release build, it's difficult to test for, as we can't lookup any variables from the GDB prompt.
Closes#12253 from pitrou/ARROW-15442-gdb-tests-release-build
Lead-authored-by: Antoine Pitrou <antoine@python.org>
Co-authored-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
- Refactored ObjectWriter helpers from arrow_to_pandas, so they can be
used for plain python types as well (generalized the lowest level so
it can work on both PyObject** and an adapter for PyList.
- Add DateOffset to static pandas imports
- Tried to start laying out code in a way to use C++ for Array.to_pylist
(feel free to comment).
Support importing from timeinterval, relativedelta and DateOffset types
(this is actually mostly duck types, the one complication is that
relativedelta has a property weeks that is automatically calculated, so
some type checking is necessary).
Open questions:
- Should we be more strict on duck typing imports? I chose generalism
over performance here (rechecking non-present attributes, etc)?
- Is the new arrow_to_python.h desirable (I think this can be easily
extended for other types)?
- My python is rusty and Python C-API even more so, please don't assume
I know exactly what I'm doing :)
Closes#11302 from emkornfield/interval_python
Lead-authored-by: Micah Kornfield <micahk@google.com>
Co-authored-by: emkornfield <emkornfield@gmail.com>
Co-authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
This adds functions to change the pool size at runtime, but doesn't adjust the default size of 8.
Closes#10316 from lidavidm/arrow-12760
Authored-by: David Li <li.davidm96@gmail.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
Second batch of changes related to making pyarrow build more modular. `hdfs-io` is no longer included in `pyarrow.lib` but has been separated to its own module.
This PR is based on https://github.com/apache/arrow/pull/10131Closes#10159 from amol-/ARROW-12506-2
Authored-by: Alessandro Molina <amol@turbogears.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
In this model, a StopSource is instantiated by the consumer, which passes a corresponding StopToken to producer API(s).
Closes#9528 from pitrou/ARROW-8732-cancel-v2
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
Also add Python factory functions `pa.sparse_union` and `pa.dense_union`.
Closes#9560 from pitrou/ARROW-11665-py-decimal-doc
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
AVX512 may be supported by the CPU but not by the Windows kernel (which needs to save and restore 512-bit registers on context switch).
Also, add a `GetRuntimeInfo()` API and the corresponding Python function `pa.runtime_info()` to query the current SIMD level.
Closes#9398 from pitrou/ARROW-11427-win-zmm-detect
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Neal Richardson <neal.p.richardson@gmail.com>
Error out cleanly if an individual string is larger than 2**31 bytes when creating a binary or string array.
Closes#8653 from pitrou/ARROW-10560-py-crash-huge-string
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
nishad shabbir