mirror of
https://github.com/apache/arrow.git
synced 2026-10-28 16:47:19 +08:00
Sylvain LesageandTadeja Kadunc
68610e4c2b
### Rationale for this change The doc page https://arrow.apache.org/docs/python/generated/pyarrow.parquet.write_table.html does not show `use_content_defined_chunking` as an argument of `write_table`: <img width="1571" height="821" alt="Screenshot From 2025-09-04 14-45-32" src="https://github.com/user-attachments/assets/a9fb4d7f-9b4d-4ce6-92c8-97b781831f33" /> Note that it's documented below in the list of parameters: <img width="1800" height="1062" alt="Screenshot From 2025-09-04 14-46-20" src="https://github.com/user-attachments/assets/d9d6c289-97c3-476b-820d-eeb30f8f791f" /> ### What changes are included in this PR? Change the API signatures of `pyarrow.parquet.ParquetWriter` and `pyarrow.parquet.write_table`: `use_content_defined_chunking` parameter is explicitly added, with a default value (`False`), as mentioned by the comment: https://github.com/apache/arrow/blob/512dcd77dcf3910c41a06d2a1620c729d92d6748/python/pyarrow/parquet/core.py#L1953-L1954 Reorder the parameters in docstring `_parquet_writer_arg_docs` to match the method signature. Add a test. Also users won't get this error anymore `TypeError: unexpected parquet write option: use_content_defined_chunking` when running with released wheel (since 21.0.0) : ```shell python -c "import pyarrow.dataset as ds; ds.ParquetFileFormat().make_write_options(use_content_defined_chunking=True)" ``` or when running : ```python import pyarrow as pa, pyarrow.parquet as pq pq.write_to_dataset(pa.table({'a':[1, 2, 3]}), 'cdc_dataset', use_content_defined_chunking=True) ``` ### Are these changes tested? Yes ### Are there any user-facing changes? Yes: 1. `use_content_defined_chunking` parameter is explicitly added to the API signature and documentation of `pyarrow.parquet.ParquetWriter` and `pyarrow.parquet.write_table` 2. `use_content_defined_chunking` is now accepted in `pyarrow.dataset.ParquetFileFormat.make_write_options` and `pyarrow.parquet.write_to_dataset` instead of raising `TypeError` * GitHub Issue: #47499 Lead-authored-by: Sylvain Lesage <severo@rednegra.net> Co-authored-by: Tadeja Kadunc <tadeja.kadunc@gmail.com> Signed-off-by: Tadeja Kadunc <tadeja.kadunc@gmail.com>
Python library for Apache Arrow
This library provides a Python API for functionality provided by the Arrow C++ libraries, along with tools for Arrow integration and interoperability with pandas, NumPy, and other software in the Python ecosystem.
Installing
Across platforms, you can install a recent version of pyarrow with the conda package manager:
conda install pyarrow -c conda-forge
On Linux, macOS, and Windows, you can also install binary wheels from PyPI with pip:
pip install pyarrow
If you encounter any issues importing the pip wheels on Windows, you may need to install the latest Visual C++ Redistributable for Visual Studio.
Development
See Python Development in the documentation subproject.
Building the documentation
See documentation build instructions in the documentation subproject.