### Rationale for this change
See GH-47861. With this change, the extension array variation takes ~192MB of memory instead of 7GB.
From what I can tell, this is because the `PandasOptions` struct is copied around frequently (for example it seems like there is an `ExtensionWriter` for each extension column and each `ExtensionWriter` has a copy of `PandasOptions` which has a set of all extension columns). I haven't fully traced the PandasOptions structure, but it seems to get copied and modified in some codepaths so I have decided to put the column sets into a `std::shared_ptr` rather than pass around a `shared_ptr<PandasOptions>`.
### What changes are included in this PR?
The `PandasOptions` column sets have been swapped from `std::unordered_set<std::string>` to `std::shared_ptr<const std::unordered_set<std::string>>` and usages have been updated.
### Are these changes tested?
Yes, no regression in the pytests. Also tested memory usage by hand.
### Are there any user-facing changes?
All changes are internal to the pyarrow C++ binding code. There are no changes to the exposed Python API.
* GitHub Issue: #47861
Lead-authored-by: Will Gulian <williamgulian@gmail.com>
Co-authored-by: Will Gulian <git@will.gl>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
As mentioned in #46728, if Arrow C++ was built debug, and PyArrow wasn't, test_gdb.py runs tests that fail.
### What changes are included in this PR?
The CMAKE_BUILD_TYPE environment variable is propagated from build into PyArrow, where it's checked to skip unit tests.
### Are these changes tested?
Yes. I have built PyArrow in release, debug, and relwithdebinfo and observed the new behavior. Because CMakeLists.txt was changed, I built PyArrow twice via setup.py and pip install, and checked the new function.
### Are there any user-facing changes?
Devs may skip unit tests that would fail. PyArrow now has build_info() with information about the build type.
* GitHub Issue: #46728
Lead-authored-by: Eric Dinse <293818+dinse@users.noreply.github.com>
Co-authored-by: Antoine Pitrou <antoine@python.org>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
When slicing arrays with non-trivial steps we were using `numpy.arange` to generate the indices for take. As numpy is an optional dependency, implementing it via Python caused a performance penalty. Creating a pyarrow function to build our own ranges that mimics Python range or numpy arange is useful for that uses case and might also be useful for other use cases. Currently we only generate `Array[Int64]` we could potentially generate more types.
### What changes are included in this PR?
provide a `pa.arange` function that allows us to generate indices when slicing arrays.
### Are these changes tested?
Yes new tests added.
### Are there any user-facing changes?
No but a new pyarrow.arange function has been added.
* GitHub Issue: #46771
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
When we added Float16 we did not update pyarrow to be able to convert from Python objects to Arrow. Float16 required numpy and it crashed if numpy was not present.
### What changes are included in this PR?
Allow to not require numpy to generate float16 scalars and arrays on pyarrow and do not fail if numpy is not present.
### Are these changes tested?
Yes, new tests have been added
### Are there any user-facing changes?
No changes for old functionality. Users will be allowed to use float16 without requiring to use np.float16 and directly from Python objects
* GitHub Issue: #46611
Lead-authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
We deprecated serialization functionality in C++ code in https://github.com/apache/arrow/pull/44064. We can now remove it.
### What changes are included in this PR?
Serialization functionality is removed from PyArrow C++ code.
### Are these changes tested?
Tests should pass.
### Are there any user-facing changes?
Serialization functionality is removed from PyArrow C++ code.
* GitHub Issue: #43587
Authored-by: AlenkaF <frim.alenka@gmail.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### What changes are included in this PR?
Make `arrow.flight.Timestamp` nanoseconds precision and OS-independent.
Use the `CTimePoint` for `FlightEndpoint.expiration_time` to have OS-independent nanoseconds precision.
https://github.com/apache/arrow/blob/093655c60783321c786a8e69632f185d37520f4d/python/pyarrow/includes/libarrow_flight.pxd#L133-L139
This reverts workaround for `expiration_time` from #43537.
### Are these changes tested?
Existing unit tests are adjusted to the new precision.
### Are there any user-facing changes?
**This PR includes breaking changes to public APIs.**
The values of `FlightEndpoint.expiration_time` now have nanoseconds precision on any operating system.
* GitHub Issue: #44679
Authored-by: Enrico Minack <github@enrico.minack.dev>
Signed-off-by: David Li <li.davidm96@gmail.com>
### Rationale for this change
The C++ classes `FlightInfo` and `FlightEndpoint` have attributes that are not available via the Python API.
### What changes are included in this PR?
Make the following attributes available in Python:
- `FlightInfo.ordered`
- `FlightInfo.app_metadata`
- `FlightEndpoint.expiration_time`
- `FlightEndpoint.app_metadata`
Also makes existing attributes optional in constructor:
- `FlightInfo.total_records`
- `FlightInfo.total_bytes`
### Are these changes tested?
Existing tests that test existing attributes are extended.
### Are there any user-facing changes?
Yes, changes are backward compatible.
* GitHub Issue: #36954
Lead-authored-by: Enrico Minack <github@enrico.minack.dev>
Co-authored-by: Adam Reeve <adreeve@gmail.com>
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Co-authored-by: David Li <li.davidm96@gmail.com>
Signed-off-by: David Li <li.davidm96@gmail.com>
### Rationale for this change
Being able to run pyarrow without requiring numpy.
### What changes are included in this PR?
If numpy is not present we are able to import pyarrow and run functionality.
A new CI job has been created to run some basic tests without numpy.
### Are these changes tested?
Yes via CI.
### Are there any user-facing changes?
Yes, NumPy can be removed from the user installation and pyarrow functionality still works
* GitHub Issue: #25118
Lead-authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
The `requested_schema` portion of the `__arrow_c_stream__()` protocol methods errored in all cases if passed an unequal schema. There was a note about figuring out how to check the cast before doing it and a comment in https://github.com/apache/arrow/issues/40066 about how it should be done lazily. This PR (hopefully) solves both!
### What changes are included in this PR?
- Added `arrow::py::CastingRecordBatchReader`, which wraps a `arrow::RecordBatchReader`, casting each batch as it is pulled.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes: the current approach adds `RecordBatchReader.cast()` as the way to access the casting reader.
* Closes: #40066
* GitHub Issue: #40066
Lead-authored-by: Dewey Dunnington <dewey@fishandwhistle.net>
Co-authored-by: Dewey Dunnington <dewey@voltrondata.com>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
### Rationale for this change
`PyArray_DescrFromScalar` returns a new reference, so we should be careful to decref it when we don't use it anymore.
### Are these changes tested?
No.
### Are there any user-facing changes?
No.
* Closes: #39599
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
### Rationale for this change
### What changes are included in this PR?
Spelling fixes to python/
### Are these changes tested?
### Are there any user-facing changes?
* Closes: #38944
Authored-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
### Rationale for this change
A deadlock can happen in a C++ destructor in the following case:
* the C++ destructor is called from Python, holding the GIL
* the C++ destructor waits for a threaded task to finish
* the threaded task has invoked some Python code which is waiting to acquire the GIL
### What changes are included in this PR?
To reliably present such a deadlock, introduce `std::shared_ptr` and `std::unique_ptr` wrappers that release the GIL when deallocating the embedded pointer.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
* Closes: #38676
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Demonstrate how to deal with async Flight from Python.
### What changes are included in this PR?
Add bindings for GetFlightInfo.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes, new APIs.
* Closes: #37093
Lead-authored-by: David Li <li.davidm96@gmail.com>
Co-authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
Do not coerce temporal types to nanosecond when pandas >= 2.0 is imported, since pandas now supports s/ms/us time units.
This PR adds support for the following Arrow -> Pandas conversions, which previously all defaulted to `datetime64[ns]` or `datetime64[ns, <TZ>]`:
```
date32 -> datetime64[ms]
date64 -> datetime64[ms]
datetime64[s] -> datetime64[s]
datetime64[ms] -> datetime64[ms]
datetime64[us] -> datetime64[us]
datetime64[s, <TZ>] -> datetime64[s, <TZ>]
datetime64[ms, <TZ>] -> datetime64[ms, <TZ>]
datetime64[us, <TZ>] -> datetime64[us, <TZ>]
```
### Rationale for this change
Pandas 2.0 introduces proper support for temporal types.
### Are these changes tested?
Yes. Pytests added and updated.
### Are there any user-facing changes?
Yes, arrow-to-pandas default conversion behavior will change when users have pandas >= 2.0, but a legacy option is exposed to provide backwards compatibility.
* Closes: #33321
Lead-authored-by: Dane Pitkin <dane@voltrondata.com>
Co-authored-by: Dane Pitkin <48041712+danepitkin@users.noreply.github.com>
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
### Rationale for this change
Explained in issue #34729
### What changes are included in this PR?
- Add support for list of maps when converting Arrow to Pandas. There doesn't seem to be a strong reason to omit this. Previously it was a hard error as unsupported, due to a bool check.
- Refactor Arrow Map -> Pandas to support two paths: (1) list of tuples, or (2) pydicts
- Add another option in PandasOptions to enable (2), above
- Bugfix in nested pydicts -> Arrow maps.
- Unit tests
### Are these changes tested?
Unit tests are added in `test_pandas.py`
### Are there any user-facing changes?
- An additional option flag in PandasOptions
- Enable list of maps to Pandas, which was previously disabled
* Closes: #34729
Authored-by: Mike Lui <mikelui@meta.com>
Signed-off-by: Will Jones <willjones127@gmail.com>
This PR moves all the `arrow/python` definitions to its own `libarrow_python.pxd` file as requested on ARROW-16062
Closes#12873 from raulcd/ARROW-16062
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
