### Rationale for this change
Require Numpy 2.0 to support newer features, allow newer numpy API/ABI
### What changes are included in this PR?
- Numpy 2.0+ requirement
- Pandas 2.2.2+ requirement (first wheel compatible with numpy 2.0)
- Pyodide 0.27.1 usage (numpy 2.0 api)
### Are there any user-facing changes?
Yes, in contrast to pyarrow 25, after merging this PR pyarrow 26 can't be used with Numpy 1.x or pandas 2.0-2.2.1
**This PR includes breaking changes to public APIs.**
This PR was created using GPT-5.6-Sol-xhigh, every line read & reviewed by me.
* GitHub Issue: #48473
Lead-authored-by: Adam Lippai <adam@rigo.sk>
Co-authored-by: Ádám Lippai <adam@rigo.sk>
Co-authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Currently we are not building wheels with OpenTelemetry.
### What changes are included in this PR?
- Adding a new feature on our `vcpkg.json` to install required opentelemetry-cpp
- Enabling the opentelemetry feature when downloading dependencies via vcpkg for our wheels.
- Setting `ARROW_WITH_OPENTELEMETRY=ON` on the wheel build scripts
- Minor fix on checking exposed symbols (opentelemetry symbols are exposed)
- Minor improvement to `python/examples/flight/middleware.py` to be able to easily enable opentelemetry middleware tracing for testing.
- Fix on `server_tracing_middleware.cc` to retrieve tracer from `arrow::internal::tracing::GetTracer();` instead of `otel::trace::Provider::GetTracerProvider()->GetTracer("arrow");`
- Expose missing symbols to enable Windows builds
- Fix Windows compatibility adding windows compat header.
- Add `is_opentelemetry_enabled` utility function in pyarrow to validate whether OpenTelemetry was enabled or not when building Arrow C++. Expose this information on `pa.show_info()`.
### Are these changes tested?
Yes, wheels are being successfully built and tested locally with a built wheel that traces are being generated when properly configured.
Added test to validate traces appear on stdout and I've also validated Acero traces are being correctly generated by running the test suite and seeing the opentelemetry traces on Jaeger UI.
### Are there any user-facing changes?
Yes, PyArrow wheel users will be able to enable OpenTelemetry traces.
* GitHub Issue: #49382
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Please see Github Issue #47123
### What changes are included in this PR?
Added public Type Enums that mimic the original private variable groups used for internal type checking.
### Are these changes tested?
Yes. Partly for now.
### Are there any user-facing changes?
No, just additional features were added: they will now able to access the underlying types directly via the Type Enums.
* GitHub Issue: #47123
Lead-authored-by: Bogdan Romenskii <rmnsk@seznam.cz>
Co-authored-by: Alenka Frim <AlenkaF@users.noreply.github.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
Arrow C++ and the Arrow specification now support 32-bit and 64-bit decimal types...pyarrow should too!
### What changes are included in this PR?
Added type, array, and scalar bindings.
### Are these changes tested?
Yes!
### Are there any user-facing changes?
Yes!
* GitHub Issue: #44713
Authored-by: Dewey Dunnington <dewey@fishandwhistle.net>
Signed-off-by: Dewey Dunnington <dewey@fishandwhistle.net>
### Rationale for this change
Being able to run pyarrow without requiring numpy.
### What changes are included in this PR?
If numpy is not present we are able to import pyarrow and run functionality.
A new CI job has been created to run some basic tests without numpy.
### Are these changes tested?
Yes via CI.
### Are there any user-facing changes?
Yes, NumPy can be removed from the user installation and pyarrow functionality still works
* GitHub Issue: #25118
Lead-authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Follow-up on https://github.com/apache/arrow/issues/39979 which added `_export_to_c_device`/`_import_from_c_device` methods, but for now only for CPU devices.
### What changes are included in this PR?
* Ensure `pyarrow.cuda` is imported before importing data through the C Interface, to ensure the CUDA device is registered
* Add tests for exporting/importing with the device interface on CUDA
### Are these changes tested?
Yes, added tests for CUDA.
* GitHub Issue: #40384
Authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
### Rationale for this change
Add bindings for the C++ `arrow::Device` and `arrow::MemoryManager` classes.
### What changes are included in this PR?
Basic bindings by adding the `pyarrow.Device` and `pyarrow.MemoryManager` classes, and just tested for CPU.
What is not included here are additional methods on the `MemoryManager` class (eg to allocate or copy buffers), and this is also not yet tested for CUDA. Planning to do this as follow-ups, and first doing those basic bindings should enable further enhancements to be done in parallel.
### Are these changes tested?
Yes, for the CPU device only.
* GitHub Issue: #41126
Authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
### Rationale for this change
Add bindings to the ListView and LargeListView array formats.
### What changes are included in this PR?
* Add initial implementation for ListView and LargeListView
* Add basic unit tests
### Are these changes tested?
* Basic unit tests only (follow up PRs will be needed to implement full functionality)
### Are there any user-facing changes?
Yes, documentation is updated in this PR to include the new PyArrow objects.
* Closes: #39812
Lead-authored-by: Dane Pitkin <dane@voltrondata.com>
Co-authored-by: Dane Pitkin <48041712+danepitkin@users.noreply.github.com>
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
### Rationale for this change
First step for https://github.com/apache/arrow/issues/39633: exposing the Array, DataType and Scalar classes for BinaryView and StringView, such that those can already be represented in pyarrow.
(I exposed a variant of StringBuilder as well, just for now to be able to create test data)
* Closes: #39651
Authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
### Rationale for this change
DLPack is selected for Array API protocol so it is important to have it implemented for Arrow/PyArrow Arrays also. This is possible for primitive type arrays (int, uint and float) with no validity buffer. Device support is not in scope of this PR (CPU only).
### What changes are included in this PR?
- `ExportArray` and `ExportDevice` methods on Arrow C++ Arrays
- `__dlpack__` method on the base PyArrow Array class exposing `ExportArray` method
- `__dlpack_device__` method on the base PyArrow Array class exposing `ExportDevice` method
### Are these changes tested?
Yes, tests are added to `dlpack_test.cc` and `test_array.py`.
### Are there any user-facing changes?
No.
* Closes: #33984
Lead-authored-by: AlenkaF <frim.alenka@gmail.com>
Co-authored-by: Alenka Frim <AlenkaF@users.noreply.github.com>
Co-authored-by: Antoine Pitrou <antoine@python.org>
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Finalizing https://github.com/apache/arrow/issues/33976 now the basic functionality for working with Acero has been added and this is used internally: publicly exposing the low-level building blocks.
### What changes are included in this PR? (user facing)
A new `pyarrow.acero` module exposing the Declaration and ExecNodeOptions subclasses (from `pyarrow._acero` cython submodule).
### Are these changes tested?
The functionality was already tested. The existing tests are updated to use the public import.
* Issue: #33976
Authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
This PR moves all the `arrow/python` definitions to its own `libarrow_python.pxd` file as requested on ARROW-16062
Closes#12873 from raulcd/ARROW-16062
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
- Refactored ObjectWriter helpers from arrow_to_pandas, so they can be
used for plain python types as well (generalized the lowest level so
it can work on both PyObject** and an adapter for PyList.
- Add DateOffset to static pandas imports
- Tried to start laying out code in a way to use C++ for Array.to_pylist
(feel free to comment).
Support importing from timeinterval, relativedelta and DateOffset types
(this is actually mostly duck types, the one complication is that
relativedelta has a property weeks that is automatically calculated, so
some type checking is necessary).
Open questions:
- Should we be more strict on duck typing imports? I chose generalism
over performance here (rechecking non-present attributes, etc)?
- Is the new arrow_to_python.h desirable (I think this can be easily
extended for other types)?
- My python is rusty and Python C-API even more so, please don't assume
I know exactly what I'm doing :)
Closes#11302 from emkornfield/interval_python
Lead-authored-by: Micah Kornfield <micahk@google.com>
Co-authored-by: emkornfield <emkornfield@gmail.com>
Co-authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
Address all docstrings to make sure they pass `archery numpydoc --allow-rule PR01`
Closes#11245 from amol-/ARROW-13637
Authored-by: Alessandro Molina <amol@turbogears.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
This adds functions to change the pool size at runtime, but doesn't adjust the default size of 8.
Closes#10316 from lidavidm/arrow-12760
Authored-by: David Li <li.davidm96@gmail.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
Second batch of changes related to making pyarrow build more modular. `hdfs-io` is no longer included in `pyarrow.lib` but has been separated to its own module.
This PR is based on https://github.com/apache/arrow/pull/10131Closes#10159 from amol-/ARROW-12506-2
Authored-by: Alessandro Molina <amol@turbogears.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
This provides sufficient coverage to support round trip between C++ and Java. There are still some gaps in python. Based on review, I will open JIRAs to track missing functionality (i.e. parquet support in C++). Marking as draft until i can triage CI failures but early feedback is welcome.
Open questions I have:
[C++]
* Should we retain logic in decimal() factory function to adjust type on scale/precision or take an explicit argument or keep it as an alias for decimal128?
[Java]
* Naming: Would Decimal256 be better then BigDecimal?
Closes#8475 from emkornfield/decimal256
Lead-authored-by: Mingyu Zhong <69326943+MingyuZhong@users.noreply.github.com>
Co-authored-by: Micah Kornfield <micahk@google.com>
Co-authored-by: Micah Kornfield <emkornfield@gmail.com>
Co-authored-by: emkornfield <emkornfield@gmail.com>
Co-authored-by: Ezra <eumen@google.com>
Signed-off-by: Micah Kornfield <emkornfield@gmail.com>
The AWS SDK on manylinux packages uses a custom-compiled libcurl that is configured for the certificates of the build system. However, there's no standard location for CA certificates on Linux, so we must instead configure the filesystem layer at runtime with Python's own TLS configuration values (which are assumed to be correct, otherwise Python itself couldn't connect to TLS hosts).
Closes#7580 from pitrou/ARROW-9261-manylinux-s3-certificates
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
This is some routine scrubbing that deals with some remnants of the prior dual py2/3 codebase.
Closes#7421 from wesm/ARROW-9030
Authored-by: Wes McKinney <wesm@apache.org>
Signed-off-by: Wes McKinney <wesm@apache.org>
Also splits `UnionType -> SparseUnionType and DenseUnionType` and similar for `UnionArray`, `UnionScalar`.
`SparseUnionArray` no longer includes the unused offsets buffer
Closes#7378 from bkietz/8866-Split-TypeUNION-into-Type
Lead-authored-by: Benjamin Kietzman <bengilgit@gmail.com>
Co-authored-by: Wes McKinney <wesm@apache.org>
Signed-off-by: Wes McKinney <wesm@apache.org>
This provides the `CAST(data AS target_type)` SQL idiom. The target_type is provided via CastOptions (FWIW I believe this is the most correct approach for handling the target_type). As a result we no longer need to maintain separate binding boilerplate in Python for Array vs. ChunkedArray
Closes#7258 from wesm/ARROW-8918
Authored-by: Wes McKinney <wesm+git@apache.org>
Signed-off-by: Wes McKinney <wesm+git@apache.org>
This patch is a major reworking of our development strategy for implementing array-valued functions and applying them in a query processing setting.
The design was partly inspired by my previous work designing Ibis (https://github.com/ibis-project/ibis -- the "expr" subsystem and the way that operators validate input types and resolve output types). Using only function names and input types, you can determine the output types of each function and resolve the "execute" function that performs a unit of work processing a batch of data. This will allow us to build deferred column expressions and then (eventually) do parallel execution.
There are a ton of details, but one nice thing is that there is now a single API entry point for invoking any function by its name:
```c++
Result<Datum> CallFunction(ExecContext* ctx, const std::string& func_name,
const std::vector<Datum>& args,
const FunctionOptions* options = NULLPTR);
```
What occurs when you do this:
* A `Function` instance is looked up in the global `FunctionRegistry`
* Given the descriptors of `args` (their types and shapes -- array or scalar), the Function searches for `Kernel` that is able to process those types and shapes. A kernel might be able to do `array[T0], array[T1]` or only `scalar[T0], scalar[T1]`, for example. This permits kernel specialization to treat different type and shape combinations
* The kernel is executed iteratively against `args` based on what `args` contains -- if there are ChunkedArrays, they will be split into contiguous pieces. Kernels never see ChunkedArray, only Array or Scalar
* The Executor implementation is able to split contiguous Array inputs into smaller chunks, which is important for parallel execution. See `ExecContext::set_exec_chunksize`
To summarize: the REGISTRY contains FUNCTIONS. A FUNCTION contains KERNELS. A KERNEL is a specific implementation of a function that services a particular type combination.
An additional effort in this patch is to radically simplify the process of creating kernels that are based on a scalar function. To do this, there is a growing collection of template-based kernel generation classes in compute/kernels/codegen_internal.h that will surely be the topic of much debate. I want to make it a lot easier for people to add new kernels.
There are some other incidental changes in the PR, such as changing the convenience APIs like `Cast` to return `Result`. I'm afraid we may have to live with the API breakage unless someone else wants to add backward compatibility code for the old APIs.
I have to apologize for making such a large PR. I've been working long hours on this for nearly a month and the process of porting all of our existing functionality and making the unit tests pass caused much iteration in the "framework" part of the code, such that it would have been a huge time drain to review incomplete iterations of the framework that had not been proven to capture the functionality that previously existed in the project.
Given the size of this PR and that fact that it completely blocks any work into src/arrow/compute, I don't think we should let this sit unmerged for more than 4 or 5 days, tops. I'm committed to responding to all of your questions and working to address your feedback about the design and improving the documentation and code comments. I tried to leave copious comments to explain my thought process in various places. Feel free to make any and all comments in this PR or in whatever form you like. I don't think that merging should be blocked on stylistic issues.
Closes#7240 from wesm/ARROW-8792-kernels-revamp
Lead-authored-by: Wes McKinney <wesm+git@apache.org>
Co-authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Wes McKinney <wesm+git@apache.org>
Depends on #6420.
Reduces the number of docstring violations from 1335 to 793 (fixes 542).
This is going to require more patches, but we need to start somewhere.
Closes#6444 from kszucs/docstrings
Authored-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
https://issues.apache.org/jira/browse/ARROW-7261
Conversion from python lists works, but conversion to python (to numpy/pandas) not yet.
Closes#5906 from jorisvandenbossche/ARROW-7261-fixed-size-list and squashes the following commits:
5046c85fc <Joris Van den Bossche> fixup rebase
feb648834 <Joris Van den Bossche> fix C++ docstring
7b4f57102 <Joris Van den Bossche> other updates
d01b45d6a <Joris Van den Bossche> fix list_size of 0 + add tests
da2a2b3ea <Joris Van den Bossche> Use Result instead of Status
8b3161937 <Joris Van den Bossche> known null_count of 0
cb68db4d5 <Joris Van den Bossche> use int32 for list_size
88a5cbdd9 <Joris Van den Bossche> add proper python -> arrow conversion
a502fa729 <Joris Van den Bossche> edits for feedback
8d94b0f7a <Joris Van den Bossche> ARROW-7261: Add Python support for Fixed Size List type
Authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
https://issues.apache.org/jira/browse/ARROW-5855
- [x] Basic wrappers for DurationArray/Type/Value
- [x] Numpy / pandas conversion
- [x] Python conversion
Closes#5566 from jorisvandenbossche/ARROW-5855-duration-python and squashes the following commits:
cd9f12aa6 <Joris Van den Bossche> Type -> ArrowType
7458000cf <Joris Van den Bossche> templated TimestampConverter
56bd1b3b3 <Joris Van den Bossche> update for feedback
f4d6b4b64 <Joris Van den Bossche> handle python 2 compat
6bfafbe14 <Joris Van den Bossche> clean-up
779e724f0 <Joris Van den Bossche> implement python conversion
03d399223 <Joris Van den Bossche> ARROW-5855: Support for Duration (timedelta) type
Authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
Closes#4979 from pitrou/ARROW-6084-py-large-list and squashes the following commits:
4266ea2c6 <Antoine Pitrou> ARROW-6084: Support LargeList
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Wes McKinney <wesm+git@apache.org>
Also fix a bug in Take / Filter for large binary types.
Closes#4927 from pitrou/ARROW-6000-py-large-binary and squashes the following commits:
0bb1b7f89 <Antoine Pitrou> Fix Take on LargeBinary / LargeString data
9672ca4c3 <Antoine Pitrou> ARROW-6000: Add support for LargeString and LargeBinary types
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
Author: Antoine Pitrou <antoine@python.org>
Closes#4267 from pitrou/ARROW-5260-from-components-other-process and squashes the following commits:
bc56aa668 <Antoine Pitrou> ARROW-5260: Fix crash when deserializating from components in another process
Starting with Python 3.7, built-in dict is insertion-ordered.
No need to return a slightly heavier and costlier OrderedDict.
Author: Antoine Pitrou <antoine@python.org>
Closes#4190 from pitrou/ARROW-5179 and squashes the following commits:
799727ba9 <Antoine Pitrou> ARROW-5179: Return plain dicts, not OrderedDict, on Python 3.7+
Author: Antoine Pitrou <antoine@python.org>
Closes#4164 from pitrou/ARROW-5178-table-from-pydict and squashes the following commits:
7b8fad5a3 <Antoine Pitrou> ARROW-5178: Add Table.from_pydict()
Warning: hold your nose for this one =) I think this can be made cleaner, but I just wanted to triage all the problems and make sure we don't introduce a hard dependency on pandas again
Also resolves ARROW-4794: make pandas an optional dependency
Author: Wes McKinney <wesm+git@apache.org>
Closes#3893 from wesm/ARROW-4637 and squashes the following commits:
3c353b6b <Wes McKinney> do not override orc global mark in test_orc.py
59fdf8ab <Wes McKinney> Tweak pyarrow._orc import to see if it fixes MSVC
747c8937 <Wes McKinney> Address Python 2.7 unicode interaction issue with Cython
4d94b2e1 <Wes McKinney> Add benchmark from ARROW-4629
f7bf7741 <Wes McKinney> Cythonize pandas API shim for better performance
4a2549aa <Wes McKinney> Remove TF testing from travis_script_manylinux1.sh
5061c6e1 <Wes McKinney> Do not require pandas to run unit tests
385cfe55 <Wes McKinney> Finish pandas API shim; do not eagerly import pandas, add to CI
804587a0 <Wes McKinney> add import test script
bb0240e2 <Wes McKinney> Begin to refactor to make references to pandas more centralized and lazy
Author: Antoine Pitrou <antoine@python.org>
Closes#3224 from pitrou/ARROW-3620-document-cpu-count and squashes the following commits:
15fda9ba <Antoine Pitrou> ARROW-3620: Document pa.cpu_count() in Sphinx API docs
There are two areas where `nthreads` cannot be replaced immediately by the global thread pool:
1. when converting Pandas data to Arrow table or record batch, since it uses a Python `ThreadPoolExecutor` from pure Python code (see `dataframe_to_arrays` in `pandas_compat.py`)
2. when reading or writing Parquet data, since `parquet-cpp` relies on parallelization facilities in the stable version of Arrow (see https://github.com/apache/parquet-cpp/pull/467)
Elsewhere, we add a `use_threads` boolean argument and deprecate `nthreads`.
Author: Antoine Pitrou <antoine@python.org>
Closes#2078 from pitrou/ARROW-2568 and squashes the following commits:
91187bf6 <Antoine Pitrou> Move use_threads flag into PandasOptions
a7aeed0e <Antoine Pitrou> Factor out secession predicate
f601d4e9 <Antoine Pitrou> ThreadPool::State pointer is const
4567a2c3 <Antoine Pitrou> Add a two-argument variant of ParallelFor() that uses the global CPU thread pool
d0a527ab <Antoine Pitrou> Restore single-thread path in WriteTableToBlocks()
2171e6e2 <Antoine Pitrou> On Windows, avoid shutting down the global thread pool at process exit
934e5a11 <Antoine Pitrou> Add & operator between Statuses
96397076 <Antoine Pitrou> Factor out deprecation logic
6b685406 <Antoine Pitrou> Fix MSVC warning
6b6f64a1 <Antoine Pitrou> Make ThreadPool capacity an int, not a size_t
61669755 <Antoine Pitrou> Rename CPUThreadPool() to GetCpuThreadPool()
d4eb8d47 <Antoine Pitrou> Export ThreadPool and CPUThreadPool()
6985afe6 <Antoine Pitrou> Lint
fcca62f5 <Antoine Pitrou> Emit FutureWarning (which is visible by default) rather than DeprecationWarning
172fba37 <Antoine Pitrou> Use C++ API, rather than multiprocessing, in pyarrow.{cpu_count,set_cpu_count}
50310b88 <Antoine Pitrou> Add API function to get desired ThreadPool capacity
3417325c <Antoine Pitrou> ARROW-2568: WIP
* Partial implementation of ARROW-1964
* Only implements StringBuilder not the other builders, notably the DictionaryBuilder mentioned in issue 1964
Author: Donal Simmie <donal.simmie@gmail.com>
Closes#1930 from dsimmie/ARROW-1964-Partial and squashes the following commits:
e4e54ec <Donal Simmie> Fixed review comments from #1930be2bd39 <Donal Simmie> Exposed StringBuilder to Cython - partial implementation of ARROW-1964
Also add Decimal fodder for other benchmarks. @cpcloud
Author: Antoine Pitrou <antoine@python.org>
Closes#1798 from pitrou/ARROW-2357-benchmarks-pandas-object-is-null and squashes the following commits:
b1c6460 <Antoine Pitrou> ARROW-2357: Add microbenchmark for PandasObjectIsNull()
