mirror of
https://github.com/apache/arrow.git
synced 2026-10-28 16:47:19 +08:00
bretttullyandJoris Van den Bossche
8548c2261b
### Rationale for this change This is a long standing [pandas ticket](https://github.com/pandas-dev/pandas/issues/53011) with some fairly horrible workarounds, where complex arrow types do not serialise well to pandas as the pandas metadata string is not parseable. However, `types_mapper` always had highest priority as it overrode what was set before. ### What changes are included in this PR? By switching the logical ordering, it means that we don't need to call `_pandas_api.pandas_dtype(dtype)` when using the pyarrow backend, thus resolving the issue of complex `dtype` with `list` or `struct`. It will likely still fail if the numpy backend is used, but at least this gives a working solution rather than an inability to load files at all. ### Are these changes tested? Existing tests should stay unchanged and a new test for the complex type has been added ### Are there any user-facing changes? **This PR contains a "Critical Fix".** This makes `pd.read_parquet(..., dtype_backend="pyarrow")` work with complex data types where the metadata added by pyarrow during `pd.to_parquet` is not serialisable and currently throwing an exception. This issue currently prevents the use of pyarrow as the default backend for pandas. * GitHub Issue: #39914 Lead-authored-by: bretttully <brett.tully@nearmap.com> Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com> Co-authored-by: Brett Tully <brett.tully@nearmap.com> Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Python library for Apache Arrow
This library provides a Python API for functionality provided by the Arrow C++ libraries, along with tools for Arrow integration and interoperability with pandas, NumPy, and other software in the Python ecosystem.
Installing
Across platforms, you can install a recent version of pyarrow with the conda package manager:
conda install pyarrow -c conda-forge
On Linux, macOS, and Windows, you can also install binary wheels from PyPI with pip:
pip install pyarrow
If you encounter any issues importing the pip wheels on Windows, you may need to install the Visual C++ Redistributable for Visual Studio 2015.
Development
See Python Development in the documentation subproject.
Building the documentation
See documentation build instructions in the documentation subproject.