mirror of
https://github.com/apache/arrow.git
synced 2026-10-28 16:47:19 +08:00
Enrico Minack
70f5c269ae
### Rationale for this change Fixes #51361. ### What changes are included in this PR? Method `FileReaderImpl::ReadColumn` should derive `records_to_read` from the RowGroup rather than ColumnChunk's `num_values`. Deriving the right ColumnChunk index in `FileReaderImpl::DecodeRowGroups` is not trivial for nested schemas. This simplifies `FileReaderImpl::ReadColumn` and fixes #51361. This was silently masked for full-schema reads and for columns with identical num_values(), but surfaces as a hard failure when an earlier, unselected column requires decryption: reading only a trailing plaintext column of a partially column-key-encrypted, plaintext-footer Parquet file threw "Cannot decrypt ColumnMetadata" even though the requested column was never encrypted. This never corrupts data on unencrypted files: ReadColumn's wrong index is only ever used to look up ColumnChunk(i)->num_values(), a count fed into the *already-correct* reader as an upper bound on how many records to decode. Every row contributes at least one definition/repetition-level entry, so num_values() for any column is always >= that row group's true row count, and every column in a row group shares the same row count. ### Are these changes tested? Yes, in the context of reading a plaintext column of a partially encrypted Parquet file. This cannot be tested with non-encrypted files. ### Are there any user-facing changes? No. ### Was AI used for this PR? In accordance to the [AI generation guidelines](https://arrow.apache.org/docs/dev/developers/overview.html#ai-generated-code), please disclose below whether and how AI was used in this PR. **PR code and description written by:** - [X] Human - [X] AI **Reviewed before submission by:** - [X] Human - [ ] AI - [ ] Not reviewed * GitHub Issue: #51361 Lead-authored-by: Enrico Minack <enrico.minack@insightsoftmax.com> Co-authored-by: Enrico Minack <github@enrico.minack.dev> Signed-off-by: Adam Reeve <adreeve@gmail.com>
Python library for Apache Arrow
This library provides a Python API for functionality provided by the Arrow C++ libraries, along with tools for Arrow integration and interoperability with pandas, NumPy, and other software in the Python ecosystem.
Installing
Across platforms, you can install a recent version of pyarrow with the conda package manager:
conda install pyarrow -c conda-forge
On Linux, macOS, and Windows, you can also install binary wheels from PyPI with pip:
pip install pyarrow
If you encounter any issues importing the pip wheels on Windows, you may need to install the latest Visual C++ Redistributable for Visual Studio.
Development
See Python Development in the documentation subproject.
Building the documentation
See documentation build instructions in the documentation subproject.