### Rationale for this change
See https://github.com/apache/parquet-format/issues/600 for rationale.
### What changes are included in this PR?
This PR adds support for using `TimestampType` to annotate `FIXED_LEN_BYTE_ARRAY(12)` values. It also adds functionality to convert FLBA(12) values to Arrow INT64 timestamps with the following flags/logic to handle overflow:
1. The conversion is guarded behind the `convert_flba_timestamps` property (default true). If false, conversion fails regardless of value.
2. If the FLBA(12) value overflows max int64 or underflows min int64, the `flba_timestamp_clamp_on_overflow` property (default false) is consulted. If false, conversion fails. If true, the value is clamped to min/max int64.
### Are these changes tested?
Yes, via unit tests and an e2e test that reads the file added in parquet-testing (https://github.com/apache/parquet-testing/pull/123).
### Are there any user-facing changes?
No
* GitHub Issue: #50915
Authored-by: Divjot Arora <div.arora@databricks.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Reading bloom filters from encrypted Parquet files previously raised an exception. This change implements encrypted bloom filter deserialization by decrypting the Thrift header (module id 8) and bitset (module id 9) separately, and adds the necessary validation and tests.
### What changes are included in this PR?
- Wire metadata decryptor creation into the bloom filter reader
- Add BlockSplitBloomFilter::DeserializeEncrypted(...) for encrypted bloom filters
- Remove the fuzzer workaround that swallowed encrypted bloom filter exceptions
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* GitHub Issue: #48334
Authored-by: fenfeng9 <fenfeng9@qq.com>
Signed-off-by: Gang Wu <ustcwg@gmail.com>
### Rationale for this change
The `is_{min/max}_value_exact` fields exist on the thrift definition and some implementations are already using them and truncating min and max values. This PR aims to expose those values and to default to true when writing files on C++ as no truncation is happening at the moment. If min/max statistics are generated we can set `is_{min/max}_value_exact` to true.
Truncation for string and binary min/max is out of scope for this PR, we can do this on a following one.
### What changes are included in this PR?
- The fields have been added to EncodedStatistics and Statistics along with the Thrift integration.
- Tests and validation with new parquet-testing file generated where there fields are present (https://github.com/apache/parquet-testing/pull/88)
- Tests with existing files without the fields.
- Update existing tests to validate the new fields.
- Add new fields to `ParquetFilePrinter`
### Are these changes tested?
Yes on CI.
### Are there any user-facing changes?
Yes, the new fields will be available for the users on the API when reading Parquet files.
* GitHub Issue: #46905
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
If new logical types are added to Parquet, how should they be handled? (Or is it easy enough to just support them as soon as they are added to the spec?)
### What changes are included in this PR?
When setting the option `convert_unknown_logical_types=True` (in Python or in the ArrowReaderProperties), unknown logical types are returned simply as their storage type. We could also make the default behaviour without an option (like DuckDB/nanoparquet/BigQuery) or make this the default behaviour *with* an option to revert to the old behaviour (error).
### Are these changes tested?
The C++ type conversion behaviour is tested; however, a higher level test would require merging https://github.com/apache/parquet-testing/pull/72 (after which I can update the submodule here and add a test in C++ and in Python).
### Are there any user-facing changes?
Yes, and documentation was updated.
* GitHub Issue: #41764
Example:
```python
import urllib
import shutil
import duckdb
from pyarrow import parquet
# https://github.com/apache/parquet-testing/pull/72
url = "https://github.com/apache/parquet-testing/raw/6e8e78107f9c3a4db4dabb75e0b96a9c67d511eb/bad_data/ARROW-GH-41764-unknown-logical-type.parquet"
with urllib.request.urlopen(url) as src, open("unknown_type.parquet", "wb") as dst:
shutil.copyfileobj(src, dst)
parquet.read_table("unknown_type.parquet", convert_unknown_logical_types=True)
#> pyarrow.Table
#> column with known type: string
#> column with unknown type: binary
#> ----
#> column with known type: [["known string 1","known string 2","known string 3"]]
#> column with unknown type: [[756E6B6E6F776E20737472696E672031,756E6B6E6F776E20737472696E672032,756E6B6E6F776E20737472696E672033]]
file = parquet.ParquetFile("unknown_type.parquet", convert_unknown_logical_types=True)
file.read()
#> pyarrow.Table
#> column with known type: string
#> column with unknown type: binary
#> ----
#> column with known type: [["known string 1","known string 2","known string 3"]]
#> column with unknown type: [[756E6B6E6F776E20737472696E672031,756E6B6E6F776E20737472696E672032,756E6B6E6F776E20737472696E672033]]
# DuckDB does this by default
duckdb.read_parquet(url)
#> ┌────────────────────────┬──────────────────────────┐
#> │ column with known type │ column with unknown type │
#> │ varchar │ blob │
#> ├────────────────────────┼──────────────────────────┤
#> │ known string 1 │ unknown string 1 │
#> │ known string 2 │ unknown string 2 │
#> │ known string 3 │ unknown string 3 │
#> └────────────────────────┴──────────────────────────┘
# nanoparquet does this by default
! Rscript -e "str(nanoparquet::read_parquet('unknown_type.parquet'))"
#> Classes ‘tbl’ and 'data.frame': 3 obs. of 2 variables:
#> $ column with known type : chr "known string 1" "known string 2" "known string 3"
#> $ column with unknown type:List of 3
#> ..$ : raw 75 6e 6b 6e ...
#> ..$ : raw 75 6e 6b 6e ...
#> ..$ : raw 75 6e 6b 6e ...
```
Lead-authored-by: Dewey Dunnington <dewey@wherobots.com>
Co-authored-by: Dewey Dunnington <dewey@fishandwhistle.net>
Co-authored-by: Dewey Dunnington <dewey@voltrondata.com>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Signed-off-by: Dewey Dunnington <dewey@wherobots.com>
### Rationale for this change
See #45185. Invalid repetition levels would previously only cause a fatal error in debug builds.
### What changes are included in this PR?
Converts an existing `ARROW_DCHECK_EQ` of the repetition level with a check that will raise an exception in release builds too.
### Are these changes tested?
Yes, using a new example file (https://github.com/apache/parquet-testing/pull/67)
### Are there any user-facing changes?
Yes, reading columns with invalid repetition levels as Arrow arrays will now raise an exception.
* GitHub Issue: #45185
Lead-authored-by: Adam Reeve <adreeve@gmail.com>
Co-authored-by: mwish <maplewish117@gmail.com>
Signed-off-by: mwish <maplewish117@gmail.com>
### Rationale for this change
This makes the test data readable by other Parquet implementations that validate the repetition levels.
### What changes are included in this PR?
* Corrects the generation of encryption test files so that the int64 list columns correctly start lists with repetition level 0.
* Updates the parquet-testing submodule to use the corrected files.
### Are these changes tested?
Yes, covered by existing tests.
### Are there any user-facing changes?
No
* GitHub Issue: #45073
Authored-by: Adam Reeve <adreeve@gmail.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
The current C++ parquet implementation interprets following parquet schema as `array<struct<array:array<int>>>, which is wrong:
```
optional group a (LIST) {
repeated group array (LIST) {
repeated int32 array;
}
}
```
### What changes are included in this PR?
According to the parquet spec, the above schema should be inferred as `array<array<int>>`.
### Are these changes tested?
Yes, a test case has been added to verify the fix.
### Are there any user-facing changes?
No.
* GitHub Issue: #43994
Authored-by: Gang Wu <ustcwg@gmail.com>
Signed-off-by: Gang Wu <ustcwg@gmail.com>
### Rationale for this change
Introducing more bad_data for testing
### What changes are included in this PR?
* Upgrade parquet-testing
* Introduce more bad_data
* Update fuzz generation
### Are these changes tested?
They're tests :-)
### Are there any user-facing changes?
no
* GitHub Issue: #43703
Authored-by: mwish <maplewish117@gmail.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### What changes are included in this PR?
Implement the format addition described in https://issues.apache.org/jira/browse/PARQUET-2414 .
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes (additional types supported for Parquet encoding).
* GitHub Issue: #39978
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
In https://github.com/apache/parquet-testing/pull/45 , an integration file for BYTE_STREAM_SPLIT was added to the parquet-testing repo.
### What changes are included in this PR?
Add a test reading that file and ensuring the decoded values are as expected.
### Are these changes tested?
By definition.
### Are there any user-facing changes?
No.
* Closes: #39560
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### What changes are included in this PR?
Adding support in GZipCodec to decompress concatenated gzip members
### Are these changes tested?
test is attached
### Are there any user-facing changes?
no
* Closes: #38271
Lead-authored-by: amassalha <amassalha@speedata.io>
Co-authored-by: Atheel Massalha <amassalha@speedata.io>
Signed-off-by: mwish <maplewish117@gmail.com>
### Rationale for this change
[ RUN ] TestArrowReaderAdHoc.ReadFloat16Files
/arrow/cpp/src/parquet/arrow/arrow_reader_writer_test.cc:4129: Failure
Failed
'reader->ReadTable(&table)' failed with NotImplemented: Support for codec 'snappy' not built
Google Test trace:
/arrow/cpp/src/parquet/arrow/arrow_reader_writer_test.cc:4123: path = /arrow/cpp/submodules/parquet-testing/data/float16_nonzeros_and_nans.parquet
[ FAILED ] TestArrowReaderAdHoc.ReadFloat16Files (1 ms)
[----------] 6 tests from TestArrowReaderAdHoc (11 ms total)
### What changes are included in this PR?
Use uncompressed files from parquet-testing.
### Are these changes tested?
Make sure the test does not fail.
### Are there any user-facing changes?
No
* Closes: #38823
Authored-by: Gang Wu <ustcwg@gmail.com>
Signed-off-by: Gang Wu <ustcwg@gmail.com>
### Rationale for this change
Validates compatibility between implementations when reading `Float16` columns.
### What changes are included in this PR?
- Bumps `parquet-testing` commit to latest to use the recently-added files
- Adds reader tests for C++ and Go in the same vein as https://github.com/apache/arrow-rs/pull/5003
### Are these changes tested?
Yes
### Are there any user-facing changes?
No
* Closes: #38751
Authored-by: benibus <bpharks@gmx.com>
Signed-off-by: Matt Topol <zotthewizard@gmail.com>
### Rationale for this change
Encrypted parquet files have been updated to apache/parquet-testing repo. We can use the new files to verify encrypted page index.
### What changes are included in this PR?
Minor refactor read_configurations_test.cc to check page index from all parquet files.
### Are these changes tested?
This is indeed a unit test enhancement.
### Are there any user-facing changes?
NO.
* Closes: #38346
Authored-by: Gang Wu <ustcwg@gmail.com>
Signed-off-by: mwish <maplewish117@gmail.com>
### Rationale for this change
Support counting CRC for dictionary page
### What changes are included in this PR?
* [x] Support counting crc for dictionary page when crc enabled
* [x] Support verify crc for dictionary page when checking crc enabled
* [x] Generate CRC dictionary file using parquet-mr, and do cross-verification
### Are these changes tested?
Yes
### Are there any user-facing changes?
Yes, dictionary page could use crc after that
* Closes: #34147
Authored-by: mwish <maplewish117@gmail.com>
Signed-off-by: Will Jones <willjones127@gmail.com>
This PR is a replacement for #10491 which appears to be abandoned. I've fixed the issues pointed out in review comments as well as a few more things I noticed. I've also made one significant change to the API; rather than `CryptoFactory::RotateMasterKeys` operating on a whole directory, it rotates keys for a single file, as this is much more flexible and allows users to decide what files to rotate keys for, and there didn't appear to be a good reason for this to work at the whole directory level. This does mean the API diverges slightly from the parquet-mr API though.
### Rationale for this change
Use of external key material allows rotating master encryption keys without having to rewrite Parquet file data. See https://docs.google.com/document/d/1bEu903840yb95k9q2X-BlsYKuXoygE4VnMDl9xz_zhk/edit?usp=sharing for more details.
### What changes are included in this PR?
Adds support for writing and reading external key material for Parquet files from C++, as well as a new `CryptoFactory::RotateMasterKeys` function that allows re-encrypting key encryption keys or data encryption keys with latest versions of master keys.
### Are these changes tested?
Yes, unit tests are included. I've added an additional test that reads a file generated with parquet-mr (used by Spark) from the parquet-testing repository. This requires merging the PR at https://github.com/apache/parquet-testing/pull/36 and updating the parquet-testing submodule before the new test will pass.
### Are there any user-facing changes?
Yes, the existing `internal_key_material` option in `parquet::encryption::EncryptionConfiguration` will now work and use external key material. This requires using two new parameters (`file_path` and `file_system`) in `CryptoFactory::GetFileEncryptionProperties` and `CryptoFactory::GetFileDecryptionProperties`, which are needed so that we know where to write/read the external key material. Note that this means external key material won't work from Python until the new parameters are exposed in Python too.
This changes the `CryptoFactory` ABI but the API is still source compatible. Is this okay or should new overloads be added that take the new parameters?
The `CryptoFactory::RotateMasterKeys` function is also a new public facing API.
* Closes: #25986
Lead-authored-by: Adam Reeve <adreeve@gmail.com>
Co-authored-by: Revital Sur <eres@il.ibm.com>
Co-authored-by: Maya Anderson <mayaa@il.ibm.com>
Signed-off-by: Will Jones <willjones127@gmail.com>
This PR fixes the issue of handling NaNs in the Parquet predicate push-down.
While computing the valid bounds for a column, if the max or min of the column is null, the range should ignore that.
* Closes: #28074
Authored-by: Sanjiban Sengupta <sanjiban.sg@gmail.com>
Signed-off-by: Weston Pace <weston.pace@gmail.com>
The original Parquet Bloom Filter spec, added in 2018 (PARQUET-41), was based on the Murmur3 hash function. The spec was later changed to use the XXH64 hash function from xxHash (PARQUET-1609); however, Parquet C++ wasn't updated for this and kept implementing the original spec.
This PR switches the bloom filter implementation to the current version of the Bloom Filter spec (as of https://github.com/apache/parquet-format/commit/3fb10e00c2204bf1c6cc91e094c59e84cefcee33). Conformance is tested using a dedicated test file in the parquet-testing repository (`bloom_filter.xxhash.bin`).
Lead-authored-by: mwish <maplewish117@gmail.com>
Co-authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
Basically this patch has defined the interface of `ColumnIndex` and `OffsetIndex`. Implementation classes are also provided to deserialize byte stream, wrap thrift message and provide access to their attributes.
BTW, the naming style throughout the code base looks confusing to me. I have tried to follow what I have understood from the parquet sub-directory. Please correct me if anything is incorrect.
Lead-authored-by: Gang Wu <ustcwg@gmail.com>
Co-authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
Currently, parquet-cpp does not support columns encoded with RLE. Although the users of RLE are quite sparse with uses of one of the 3 types [Repetition and definition levels, dictionary indices and boolean values in data pages], [Parquet-encodings](https://parquet.apache.org/docs/file-format/data-pages/encodings/). Some implementations do encode this directly on boolean columns (Athena on AWS). Even though there is encoding and decoding support for repetition and definition levels, there is no support for boolean column with RLE.
This PR integrates the column scanning to support columns with RLE. The first 4 bytes of the data length are size of the encoded data, which is parsed first and then passes to decoder.
Added two tests with RLE boolean encoded parquet file to validate that values can be parsed individually and in a batch.
Lead-authored-by: Nishanth Thimmegowda <nishanth.thimmegowda@snowflake.com>
Co-authored-by: sfc-gh-nthimmegowda <nishanth.thimmegowda@snowflake.com>
Co-authored-by: Ubuntu <ubuntu@ip-172-29-51-62.us-west-2.compute.internal>
Co-authored-by: Ubuntu <ubuntu@ip-172-29-51-79.us-west-2.compute.internal>
Co-authored-by: Ubuntu <ubuntu@ip-172-29-51-6.us-west-2.compute.internal>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
I've realized there were some changes on the parquet-testing submodule. This PR updates it to the latest version.
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
Looks like we have DeltaLengthByteArrayDecoder implemented. Enabling it in this commit to support DELTA_LENGTH_BYTE_ARRAY decoding
Authored-by: Muthunagappan Muthuraman <m.muthuraman@snowflake.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
1. Add "Advance" method in arrow::BitUtil::BitReader.
2. Basic implement of DeltaLengthByteArrayDecoder and DeltaByteArrayDecoder.
2. Test case of DeltaByteArrayDecoder, which relies on DeltaLengthByteArrayDecoder.
TODO:
1. Read corrupted files written with bug([PARQUET-246](https://issues.apache.org/jira/browse/PARQUET-246)).
2. Add a test case of DeltaLengthByteArrayDecoder, if possible.
Closes#10978 from shanhuuang/PARQUET-492
Lead-authored-by: shanhuuang <shanhuuang@gmail.com>
Co-authored-by: Shan Huang <85105837+shanhuuang@users.noreply.github.com>
Co-authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
Only works for int32/int64 when delta_bit_width_<=32.
1. Update the implement of DeltaBitPackDecoder
2. Revise the code of putting/getting ZigZag int and the unit test
3. Add the implement of putting/getting ZigZag int64 and unit test
Closes#10627 from shanhuuang/ARROW-13206
Lead-authored-by: shanhuuang <shanhuuang@gmail.com>
Co-authored-by: pincheng.hs <pincheng.hs@alibaba-inc.com>
Co-authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
The compatibility code was incomplete as it assumed a single "frame" per data page.
It turns out that Hadoop tries to compress data in blocks of 128 kiB.
Closes#9244 from pitrou/ARROW-9177-lz4-hadoop-frames
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
This patch makes arrow's Lz4Codec compatible with the Hadoop Lz4Codec by prepending 8 bytes (two `uint32_t`s representing expected decompressed size in bytes and compressed size in bytes) to arrow's lz4 decompression/compression.
Closes#7789 from patrickpai/PARQUET-1878
Lead-authored-by: Patrick Pai <patrick.m.pai@gmail.com>
Co-authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Wes McKinney <wesm+git@apache.org>
Closes#5502 from alippai/patch-2 and squashes the following commits:
136087fcf <Adam Lippai> ARROW-6687: Add regression tests for the new datafusion .parquet support
Authored-by: Adam Lippai <adam@rigo.sk>
Signed-off-by: Paddy Horan <paddyhoran@hotmail.com>
…adable
pyarrow needs to handle dictionary page offset = 0 as a special case to be compatible with java parquet reader.
Author: shyam <shyam@dremio.com>
Closes#4359 from shyambits2004/5322 and squashes the following commits:
f47762ab0 <shyam> Parquet files with dictionary page offset as 0 is not readable
Fixes the issue with min-max statistics built by parquet-mr 1.10+:
https://issues.apache.org/jira/browse/PARQUET-1494
Author: Ildar Musin <zildermann@gmail.com>
Closes#3441 from zilder/fix_parquet_1494 and squashes the following commits:
c5ba5fc07 <Ildar Musin> Test case for min-max statistics on binary column built with parquet-mr
029f73047 <Ildar Musin> PARQUET-1494: Recognize statistics built with UNSIGNED sort order by parquet-mr 1.10.0 onwards
Author: Wes McKinney <wesm+git@apache.org>
Closes#3242 from wesm/PARQUET-1481 and squashes the following commits:
b074227ba <Wes McKinney> Add test case with example corrupt data file
59400a2f1 <Wes McKinney> Throw exception when encountering bad Thrift metadata in RecordReader
This imports parquet-rs source code into Apache Arrow Rust implementation. I include most of the source code except a few things such as `fuzz` and benchmarks. Thinking about adding them later.
The module hierarchy now looks like:
- arrow: all the arrow code
- parquet: all the parquet code (in future, parquet-arrow integration will live here)
- util: common util libraries shared between arrow and parquet (I'll try to move the utils from parquet to here in future).
Author: Chao Sun <sunchao@uber.com>
Author: Chao Sun <sunchao@apache.org>
Closes#3050 from sunchao/import-parquet and squashes the following commits:
2ce98bd2a <Chao Sun> Update git submodule
2d296f8f7 <Chao Sun> ARROW-4028: Merge parquet-rs codebase
This change enables support for DECIMALs backed by BYTE_ARRAYs on disk. It does this by creating a TransferFunctor routine that transforms a ByteArrayType to an ::arrow::Decimal128Type.
The routine does this by:
1. Creating an arrow::BinaryArray from the RecordReader's builder
2. Allocating a buffer for the arrow::Decimal128Array
3. Converting the big-endian bytes in each BinaryArray entry to two integers
representing the high and low bits of each decimal value.
Author: Ted Haining <thaining@xcalar.com>
Closes#2646 from thaining/parquet-1160-byte-array-decimals and squashes the following commits:
0bad382e <Ted Haining> Updated parquet-testing to SHA that includes necessary test files.
30f3a278 <Ted Haining> This change enables support for DECIMALs backed by BYTE_ARRAYs on disk. It does this by creating a TransferFunctor routine that transforms a ByteArrayType to an ::arrow::Decimal128Type.
system. Add unit test label granularity options, ability to add component group
targets like 'make parquet' that build libraries and tests
Change-Id: I250fd10da3a9505952115b3cec18cd7cb5589bdb
Divjot Arora