### Rationale for this change
The usage of disparate integer types (`int64_t`, `uint32_t`) makes our checks and computations fragile, especially with C++ adding its own integer promotion rules across arithmetic operations.
We have had at least one report (courtesy of Ada Logics and Claude) where a carefully crafted Parquet file can read from an invalid pointer due to arithmetic overflow in the 32-bit domain.
### What changes are included in this PR?
Use `int64_t` throughout most internal APIs and code paths when reading Parquet metadata. Other types such as `uint32_t` should only be used where necessary when interacting with third-party libraries such as Thrift C++.
### Are these changes tested?
By existing tests, and manually using said hand-crafted Parquet file.
### Are there any user-facing changes?
Some APIs taking a `uint32_t*` inout-parameter are deprecated, alternatives taking a `int64_t` value are available.
* GitHub Issue: #50971
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Running parquet-scan without arguments currently triggers a confusing IOError instead of showing how to use the tool.
### What changes are included in this PR?
Updated the argument validation from argc < 1 (always false) to argc < 2.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
Old behavior:
$ parquet-scan
Parquet error: IOError: Failed to open local file ''. Detail: [errno 2] No such file or directory
New behavior:
$ parquet-scan
Usage: parquet-scan [--batch-size=] [--columns=...]
* GitHub Issue: #49539
Authored-by: Dominique Belhachemi <db@domibel.de>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
This continues adding functionality to the Meson configuration
### What changes are included in this PR?
This adds the parquet directory to the Meson configuration
### Are these changes tested?
Yes
### Are there any user-facing changes?
No
* GitHub Issue: #46410
Authored-by: Will Ayd <william.ayd@icloud.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
I want to use this in the C data interface statistics documents: https://github.com/apache/arrow/pull/43553
### What changes are included in this PR?
Add an executable that reads an Apache Parquet file and dumps statistics read as `arrow::ArrayStatistics`.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
* GitHub Issue: #44815
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
* We don't need to link to `libarrow` explicitly because `parquet_shared`/`parquet_static` does it automatically
* `--help` shows `--debug` but the implementation accepts `--json` not `--debug`
### What changes are included in this PR?
* Remove the redundant `libarrow` link
* Accept `--debug` not `--json`
### Are these changes tested?
Manually.
### Are there any user-facing changes?
Yes.
* GitHub Issue: #43393
Authored-by: Alkis Evlogimenos <alkis.evlogimenos@databricks.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
This binary will make it a lot easier for customers to share their parquet metadata with the community so that we can build a repository of footers that can be used for advancing the state of metadata in parquet.
### What changes are included in this PR?
Usage from the file binary itself:
```
Usage: parquet-dump-footer
-h|--help Print help and exit
--no-scrub Do not scrub potentially confidential metadata
--debug Output text represenation of footer for inspection
--in <uri> Input file (required): must be an URI or an absolute local path
--out <path> Output file (optional, default stdout)
Dump the footer of a Parquet file to stdout or to a file, optionally with
potentially confidential metadata scrubbed.
```
### Are these changes tested?
Manually on existing parquet files.
### Are there any user-facing changes?
No.
* GitHub Issue: #42102
Lead-authored-by: Alkis Evlogimenos <alkis.evlogimenos@databricks.com>
Co-authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
This "leaner" implementation reduces the generated code size of this C++ file from 2307768 bytes to 1192608 bytes in gcc 10.3.0. The benchmarks are also faster (on my avx2 laptop):
before
```
-----------------------------------------------------------------------------------------------
Benchmark Time CPU Iterations UserCounters...
-----------------------------------------------------------------------------------------------
GreaterArrayArrayInt64/32768/10000 32.1 us 32.1 us 21533 items_per_second=1020.16M/s null_percent=0.01 size=32.768k
GreaterArrayArrayInt64/32768/100 32.1 us 32.1 us 21603 items_per_second=1019.27M/s null_percent=1 size=32.768k
GreaterArrayArrayInt64/32768/10 32.1 us 32.1 us 21479 items_per_second=1020.82M/s null_percent=10 size=32.768k
GreaterArrayArrayInt64/32768/2 32.0 us 32.0 us 21468 items_per_second=1023.12M/s null_percent=50 size=32.768k
GreaterArrayArrayInt64/32768/1 32.3 us 32.3 us 21720 items_per_second=1013.44M/s null_percent=100 size=32.768k
GreaterArrayArrayInt64/32768/0 31.6 us 31.6 us 21828 items_per_second=1036.94M/s null_percent=0 size=32.768k
GreaterArrayScalarInt64/32768/10000 20.8 us 20.8 us 33461 items_per_second=1.57238G/s null_percent=0.01 size=32.768k
GreaterArrayScalarInt64/32768/100 20.9 us 20.9 us 33625 items_per_second=1.56611G/s null_percent=1 size=32.768k
GreaterArrayScalarInt64/32768/10 20.8 us 20.8 us 33553 items_per_second=1.57338G/s null_percent=10 size=32.768k
GreaterArrayScalarInt64/32768/2 20.9 us 20.9 us 33348 items_per_second=1.5687G/s null_percent=50 size=32.768k
GreaterArrayScalarInt64/32768/1 20.9 us 20.9 us 33419 items_per_second=1.56879G/s null_percent=100 size=32.768k
GreaterArrayScalarInt64/32768/0 20.5 us 20.5 us 34116 items_per_second=1.59837G/s null_percent=0 size=32.768k
```
after
```
-----------------------------------------------------------------------------------------------
Benchmark Time CPU Iterations UserCounters...
-----------------------------------------------------------------------------------------------
GreaterArrayArrayInt64/32768/10000 18.1 us 18.1 us 38751 items_per_second=1.81199G/s null_percent=0.01 size=32.768k
GreaterArrayArrayInt64/32768/100 17.5 us 17.5 us 39374 items_per_second=1.86821G/s null_percent=1 size=32.768k
GreaterArrayArrayInt64/32768/10 19.0 us 19.0 us 33941 items_per_second=1.72066G/s null_percent=10 size=32.768k
GreaterArrayArrayInt64/32768/2 18.0 us 18.0 us 39589 items_per_second=1.81817G/s null_percent=50 size=32.768k
GreaterArrayArrayInt64/32768/1 18.1 us 18.1 us 39061 items_per_second=1.80719G/s null_percent=100 size=32.768k
GreaterArrayArrayInt64/32768/0 17.5 us 17.5 us 39813 items_per_second=1.87031G/s null_percent=0 size=32.768k
GreaterArrayScalarInt64/32768/10000 16.3 us 16.3 us 42281 items_per_second=2.01525G/s null_percent=0.01 size=32.768k
GreaterArrayScalarInt64/32768/100 16.5 us 16.5 us 42266 items_per_second=1.98195G/s null_percent=1 size=32.768k
GreaterArrayScalarInt64/32768/10 16.5 us 16.5 us 41872 items_per_second=1.98615G/s null_percent=10 size=32.768k
GreaterArrayScalarInt64/32768/2 16.3 us 16.3 us 42130 items_per_second=2.00447G/s null_percent=50 size=32.768k
GreaterArrayScalarInt64/32768/1 16.2 us 16.2 us 42391 items_per_second=2.02296G/s null_percent=100 size=32.768k
GreaterArrayScalarInt64/32768/0 15.9 us 15.9 us 43498 items_per_second=2.0614G/s null_percent=0 size=32.768k
```
Authored-by: Wes McKinney <wesm@apache.org>
Signed-off-by: Wes McKinney <wesm@apache.org>
Parent issue: ARROW-16755. Also resolves ARROW-16819
ArraySpan has no shared pointers at all and is much cheaper to pass around, copy, and basically eliminates the current significant overhead associated with ExecBatch ExecBatchIterator.
This PR isn't going to show meaningful performance gains in function or expression evaluation -- that will require implementing a more streamlined expression evaluator that is based on ArraySpan.
This is only an intermediate patch to try to limit the scope of work as much as possible and facilitate follow up PRs. I have a long list of things I would like to do pretty much right away in follow up patches
Some notes:
* The ArraySpan retains pointers to the buffers that were used to populate it because in many places in the existing scalar kernels, we have to "go back" to a `shared_ptr<ArrayData>`
* There are multiple places where having only `const DataType*` or `const Scalar*` would disallow the use of APIs that require either `shared_ptr<DataType>` or `shared_ptr<Scalar>`, so I added `std::enable_shared_from_this` on these classes. I don't know whether this increases the initialization cost of `make_shared<T>` if anyone knows, but I hope that in the future we can remove `std::enable_shared_from_this`. It would be better to have `Scalar::Copy` and `DataType::Copy` methods so this isn't necessary, but rather than try to hack this in this PR, I left this for follow on work
* A few kernels have been refactored to always write into preallocated memory
(IsIn, IndexIn, IsNull, IsValid)
* Some internal APIs were best refactored to use ArraySpan, such as
ArrayBuilder::AppendArraySlice, stuff in arrow/util/int_util.h
In the interest of getting this merged sooner rather than later, rather than trying to make everything perfect here let's try to fix any glaring / serious issues that you see otherwise leave many improvements for follow up patches, otherwise any work in the scalar kernels codebase will be blocked.
Authored-by: Wes McKinney <wesm@apache.org>
Signed-off-by: Wes McKinney <wesm@apache.org>
Closes#5158 from kou/cpp-stop-needless-static-linking and squashes the following commits:
4b4fe0991 <Sutou Kouhei> ARROW-6309: Stop needless static linking
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
- Add a top-level script that reformats all known CMake files in the source tree
- Add a lint step in Travis that checks CMake files are properly formatted
Requires [cmake_format](https://github.com/cheshirekow/cmake_format) and Python 3.
Author: Antoine Pitrou <antoine@python.org>
Closes#3606 from pitrou/ARROW-4363-cmake-format and squashes the following commits:
762df291 <Antoine Pitrou> Add mention in cpp/README.md
9073af98 <Antoine Pitrou> ARROW-4363: Add CMake format checks
Author: Renat Valiullin <rip.nsk@gmail.com>
Closes#3545 from rip-nsk/ARROW-4456 and squashes the following commits:
939a4fb3e <Renat Valiullin> Add parquet tools to ci/cpp-msvc-build-main
03422aa99 <Renat Valiullin> remove dependency on getopt in parquet tools
Apparently CMake really does not want you to do `make $TARGET install` out of the box; I searched various threads about this and there's no great solutions. For expert users, this provides that option of installing only targets that have been built, while others will be ignored.
Because the possibility of users shooting themselves in the foot is so high with this, it doesn't make sense to enable by default. In the hands of an expert though, this can significantly reduce build times and make it possible to build libraries and unit tests for only a part of the project, then only install those libraries.
This will install all header files regardless of what libraries are built; I didn't see any easy way to work that out since you have to have knowledge of what headers are used by what library.
Resolves ARROW-3994
Author: Wes McKinney <wesm+git@apache.org>
Closes#3172 from wesm/ARROW-3184 and squashes the following commits:
583a916e0 <Wes McKinney> plasma_store_server requires static libraries
3c2a21ea1 <Wes McKinney> Add plasma_store_server to 'plasma' target
85fda6419 <Wes McKinney> Build plasma again in Python build for now
1b3ac57dc <Wes McKinney> Fix multiline comment in CMake
d3ce84c4e <Wes McKinney> More option reorg
b6630605c <Wes McKinney> Reorganize CMake options a bit more logically. Add more explicit warning about ARROW_OPTIONAL_INSTALL
262058b2f <Wes McKinney> Do not build Gandiva JNI bindings by default
918fdb371 <Wes McKinney> Fix ARROW_TEST_INCLUDE_LABELS option to actually work
578bc58f5 <Wes McKinney> Use GLOB instead of GLOB_RECURSE
daaafa214 <Wes McKinney> Misc fixes
a84643d6e <Wes McKinney> Fix header install option
f899bdd99 <Wes McKinney> Work around ARROW-4026 via environment variable for now
001a3ad57 <Wes McKinney> Pass in ARROW_TEST_INCLUDE_LABELS via environment variable
a1df9ab3d <Wes McKinney> Clarify documentation
2eca8a740 <Wes McKinney> Enable modular builds and install with ARROW_OPTIONAL_INSTALL option. Remove ARROW_GANDIVA_BUILD_TESTS. Add helper function for installing header files. Build fewer targets using these options in some Travis CI jobs
Author: Hatem Helal <Hatem.Helal@mathworks.co.uk>
Author: Hatem Helal <hatem.helal@gmail.com>
Closes#3148 from hatemhelal/parquet-1473-wip and squashes the following commits:
8983ae629 <Hatem Helal> remove default case from switch to make any omission a compile-time error
a87c3ba99 <Hatem Helal> Add helper function parquet::ParquetVersionToString and use it in parquet-reader tool
The other CLI tools use dashes. Since this is a helper utility I think it is OK to just change this
Author: Wes McKinney <wesm+git@apache.org>
Closes#2994 from wesm/PARQUET-1271 and squashes the following commits:
cc9ae18fd <Wes McKinney> Rename parquet_reader to parquet-reader for consistency
Author: Deepak Majeti <deepak.majeti@hpe.com>
Closes#2715 from majetideepak/PARQUET-1426 and squashes the following commits:
a94387d77 <Deepak Majeti> PARQUET-1426: Improve parquet-dump-schema usability
This is a minor change useful for debugging.
Now parquet_reader tool has --print-key-value-metadata which when present, dump of key values of file metadata is done.
Created https://issues.apache.org/jira/browse/PARQUET-1256
Author: Jacek Pliszka <Jacek.Pliszka@gmail.com>
Closes#450 from JacekPliszka/master and squashes the following commits:
0d9a108 [Jacek Pliszka] Added --print-key-value-metadata option to parquet_reader tool
Change-Id: Ie6ad3037bcfe59128ad39d2408c8a2656666fbc1
"parquet_reader" and friends should be installed along with the Parquet libraries.
Author: Antoine Pitrou <antoine@python.org>
Closes#455 from pitrou/PARQUET-1270-install-tools and squashes the following commits:
44c486c [Antoine Pitrou] PARQUET-1270: Install executable tools
Change-Id: I96fa2c9262716eeceddbf2d844e3805095431693
These warnings will catch a number of things that have bitten us in the past, like missing virtual destructors. This brings Parquet's compiler warnings up to the same quality as Arrow's
Author: Wes McKinney <wes.mckinney@twosigma.com>
Author: Wes McKinney <wesmckinn@gmail.com>
Closes#425 from wesm/PARQUET-1177 and squashes the following commits:
3769a8c [Wes McKinney] Add -Wno-missing-noreturn
5b6cd80 [Wes McKinney] Compile with /bigobj in MSVC
cc5bca0 [Wes McKinney] Add noreturn to static methods in ParquetException
e3ffb71 [Wes McKinney] Fix -Wconversion warnings in decode_benchmark.cc
758a216 [Wes McKinney] Fix warnings on macOS Clang
3aef3b4 [Wes McKinney] Do not pass -Werror via PARQUET_CXXFLAGS
5a98e81 [Wes McKinney] Fix usage of PrimitiveArray::raw_values
c848855 [Wes McKinney] Fix compiler warnings with gcc 4.9
ca9a374 [Wes McKinney] Add SetupCxxFlags.cmake from Apache Arrow. Add PARQUET_BUILD_WARNING_LEVEL flag. Fix Clang compiler warnings
Change-Id: I428d1d90bc4eb3dab8b56a538d1eb58656664b74
See ARROW-1377
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closes#385 from wesm/PARQUET-1083 and squashes the following commits:
359cd09 [Wes McKinney] Factor main logic in parquet-scan.cc into a library function, so that library users can use for performance testing
Change-Id: Ia50d136c380c4d42d6c62577e02a9533df6fa6fe
The main change is horizontal alignment. We should also do a clang-tidy pass sometime to do some further scrubbing
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closes#375 from wesm/PARQUET-1068 and squashes the following commits:
b81145d [Wes McKinney] Modify .clang-format to use straight Google format with 90-character line width
Change-Id: If8345d1d2a03d785ed41a5848de2c40e4bf53b5b
Author: Max Risuhin <risuhin.max@gmail.com>
Closes#313 from MaxRis/PARQUET-679 and squashes the following commits:
c5c4a8c [Max Risuhin] PARQUET-679: Local Windows build and Appveyor support
Change-Id: I6ca8937537e035bce8b9c83fb811ea6d58c6d3f0
Made minor formatting changes to DebugPrint
No support to print values. Only the metadata is JSON formatted in this patch.
Author: Deepak Majeti <deepak.majeti@hpe.com>
Closes#310 from majetideepak/PARQUET-958 and squashes the following commits:
4d9cbbd [Deepak Majeti] change DebugPrint to take filename
3c78bc0 [Deepak Majeti] use raw string
97f016a [Deepak Majeti] add test and clang format
ec12ddb [Deepak Majeti] add JSONPrint
9c697e2 [Deepak Majeti] fix CMake flag for benchmarks
Change-Id: Iaf4f4ba609ea5d26a061c9905e5770580ea6b0ed
Author: Deepak Majeti <deepak.majeti@hpe.com>
Closes#307 from majetideepak/PARQUET-508 and squashes the following commits:
de1364a [Deepak Majeti] add to api
1dc6260 [Deepak Majeti] PARQUET-508: Add ParquetFilePrinter
Change-Id: Ic581d6c47fbe2a055622e9345918f6b64ebaf08e
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closes#259 from wesm/PARQUET-892 and squashes the following commits:
1e932c4 [Wes McKinney] Specify public link targets for parquet_static so that transitive dependencies are linked in executables
Change-Id: Ib0bea0cd272919dffa82aa1c12d7658ea15d9cb5
I bumped the thirdparty git hash to Arrow trunk. Pardon the diff noise from clang format. Also resolves PARQUET-729.
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closes#167 from wesm/PARQUET-728 and squashes the following commits:
222e72c [Wes McKinney] Write multiple columns in file-serialize-test. Fix PARQUET-729
ef56871 [Wes McKinney] Incorporate upstream Arrow API changes, clang format with LLVM 3.8.0
Change-Id: I673b1538c93623ae50ebf70cd8d9c7048781c982
I wrote a sample file and the metadata seems to be correct.
@xhochy I fixed some missing metadata like `dictionary_page_offset`. You might want to check if this fixes the Drill problem.
Author: Deepak Majeti <deepak.majeti@hpe.com>
Closes#156 from majetideepak/PARQUET-711 and squashes the following commits:
25f5a7e [Deepak Majeti] fix schema and descr. Resolves PARQUET-705 and PARQUET-707
8b4784d [Deepak Majeti] Review comments to add methods back
fdbc761 [Deepak Majeti] fix clang error and comments
c6cb071 [Deepak Majeti] convert DCHECKS to Exceptions in metadata
ada3ac2 [Deepak Majeti] clang format
d9c9131 [Deepak Majeti] Use metadata builders in parquet writer
Change-Id: Iabe80b1cbe3fd8f1de6239187058b6402b160975
Added a ReadBatchValues() API to the Column class.
Added a parquet-scan tool
Separated examples into benchmarks/tools
added clang tidy and clang format to benchmarks and tools
Author: Deepak Majeti <deepak.majeti@hpe.com>
Closes#144 from majetideepak/parquetscan and squashes the following commits:
cc7f183 [Deepak Majeti] Removed GetRemainingInPage API
44da480 [Deepak Majeti] add scan all in public api
20829b8 [Deepak Majeti] clang-format
da62354 [Deepak Majeti] ScanAllValues API
e385f61 [Deepak Majeti] put clang-* in the root directory
9ff785c [Deepak Majeti] use c++ random
d854bde [Deepak Majeti] parquet scan tool
Change-Id: I1e5d1e42aa5a3e8dfbe6b556dd0081bb0ed7f4d8
Antoine Pitrou