This also add two new docker configurations (`cpp-static-only` and `cpp-shared-only`) to test for these issues.
This should also fix https://issues.apache.org/jira/browse/ARROW-4848
Author: Uwe L. Korn <uwelk@xhochy.com>
Closes#3928 from xhochy/ARROW-4903 and squashes the following commits:
a8070e35f <Uwe L. Korn> clang-format
20b779525 <Uwe L. Korn> cmake-format
d16900d2b <Uwe L. Korn> ARROW-4903: Fix static/shared-only builds
This patch ended up being a bit more of a bloodbath than I planned: please accept my apologies.
Associated changes in this patch:
* Split up builder.h/builder.cc into a new arrow/array directory. Public arrow/builder.h API preserved. I think this code is going to keep growing more specialized components, so I think we should get out ahead of it by having a subdirectory to contain files related to implementation details
* Implement ChunkedBinaryBuilder, ChunkedStringBuilder classes, add tests and benchmarks
* Deprecate parquet::arrow methods returning Array
* Allow implicit construction of Datum from its variant types (makes for a lot nicer syntax)
As far as what code to review, focus efforts on
* src/parquet/arrow
* src/arrow/array/builder_binary.h/cc, array-binary-test.cc, builder-benchmark
* src/arrow/compute changes
* Python changes
I'm going to tackle ARROW-2970 which should not be complicated after this patch; I will submit that as a PR after this is reviews and merged.
Author: Wes McKinney <wesm+git@apache.org>
Closes#3171 from wesm/ARROW-3762 and squashes the following commits:
822451280 <Wes McKinney> Fix int conversion warning on Windows
695ffc9df <Wes McKinney> Remove unimplemented and unused ChunkedBinaryBuilder ctor
5a525115c <Wes McKinney> Use strnlen to compute string length. Inline BinaryBuilder::AppendNextOffset
b90eb4b71 <Wes McKinney> Restore sstream include to pretty_print.cc
3669201be <Wes McKinney> Fix deprecated API use
5fdbbb261 <Wes McKinney> Rename columnar/ directory to array/
8ffaec1ef <Wes McKinney> Address preliminary code comments. Check in missing files
81e787c69 <Wes McKinney> Fix up Python bindings, unit test
2efae064c <Wes McKinney> Finish scaffolding. Get fully compiling again and original parquet-arrow test suite passing
3d075e4aa <Wes McKinney> Additional refactoring to make things chunked. Allow implicit construction of arrow::compute::Datum
922811278 <Wes McKinney> More refactoring
716322377 <Wes McKinney> Split up builder.h, builder.cc into smaller headers, compilation units. add failing test case for ARROW-3762. Add ChunkedBinaryBuilder, make BinaryBuilder Append methods inline
- Implement the `ADD_EXAMPLE` cmake function with new ctest label
`example`, also covered by the `runexample` target. This can be
toggled via the `ARROW_BUILD_EXAMPLES` option which is ON by default.
- Implement fully working `row-wise-conversion-example.cc` and add it to
the default build.
- Update documentation to embed (manually) the newly created example.
Author: François Saint-Jacques <fsaintjacques@gmail.com>
Closes#3078 from fsaintjacques/ARROW-3470-out-of-date-example and squashes the following commits:
fab63f6f <François Saint-Jacques> ARROW-3470: Fix status macro
1eba067d <François Saint-Jacques> ARROW-3470: Fix row-wise example
When compiling example, reader-writer.cc, it shows following compilation errors:
no member named 'cout' in namespace 'std'
PARQUET_THROW_NOT_OK(arrow::PrettyPrint(*array, 4, &std::cout));
in multiple places.
Fix:
Add the #include<iostream>
Test:
make passed
Author: Wes McKinney <wesm+git@apache.org>
Author: wangmiao1981 <wm624@hotmail.com>
Closes#3049 from wangmiao1981/example and squashes the following commits:
ecca664a2 <Wes McKinney> Build all examples in examples/parquet when PARQUET_BUILD_EXAMPLES=ON
64d51b214 <wangmiao1981> add include file
I split the changes into multiple commits to ease the review.
Used the example program to test the new API.
I will add unit tests once we converge on the API after review.
Thanks to @anatolishein for collaborating with the API design.
Author: Deepak Majeti <deepak.majeti@hpe.com>
Closes#484 from majetideepak/PARQUET-1372 and squashes the following commits:
143ed51 [Deepak Majeti] improve comments
c10fe08 [Deepak Majeti] Add test
d12b10b [Deepak Majeti] Review comments
cb99b3f [Deepak Majeti] fix compiler warnings
e179a4c [Deepak Majeti] add example header
710bbe0 [Deepak Majeti] clang format
9e03004 [Deepak Majeti] reorg examples
410a3af [Deepak Majeti] remove flush_on_close
e148817 [Deepak Majeti] add BufferedPageWriter
26a52c1 [Deepak Majeti] clang format
20049c0 [Deepak Majeti] modify examples
9db26a2 [Deepak Majeti] Combine RowGroupWriter2 with RowGroupWriter
cb7d69c [Deepak Majeti] fix compiler errors
21642b3 [Deepak Majeti] clang format
530b835 [Deepak Majeti] example for RowGroupWriter2
0fc1f5c [Deepak Majeti] Extend Column Writer to flush pages on Close
f2f420d [Deepak Majeti] RowGroupWriter2, implementation that writes all columns at once
Change-Id: I749cbde733780c9a6499df6738b2236124b8a9f7
Depends on https://github.com/apache/parquet-cpp/pull/434
Author: Uwe L. Korn <uwelk@xhochy.com>
Author: Korn, Uwe <Uwe.Korn@blue-yonder.com>
Closes#436 from xhochy/PARQUET-1196 and squashes the following commits:
a938da7 [Uwe L. Korn] Check Status for PrettyPrint
15d62f3 [Uwe L. Korn] PARQUET-1196: Example parquet_arrow project
1280fd5 [Korn, Uwe] PARQUET-1200: Support reading a single Arrow column from a Parquet file
Change-Id: I907f2276b319491f6e02117f4a21ab2383006a99
These warnings will catch a number of things that have bitten us in the past, like missing virtual destructors. This brings Parquet's compiler warnings up to the same quality as Arrow's
Author: Wes McKinney <wes.mckinney@twosigma.com>
Author: Wes McKinney <wesmckinn@gmail.com>
Closes#425 from wesm/PARQUET-1177 and squashes the following commits:
3769a8c [Wes McKinney] Add -Wno-missing-noreturn
5b6cd80 [Wes McKinney] Compile with /bigobj in MSVC
cc5bca0 [Wes McKinney] Add noreturn to static methods in ParquetException
e3ffb71 [Wes McKinney] Fix -Wconversion warnings in decode_benchmark.cc
758a216 [Wes McKinney] Fix warnings on macOS Clang
3aef3b4 [Wes McKinney] Do not pass -Werror via PARQUET_CXXFLAGS
5a98e81 [Wes McKinney] Fix usage of PrimitiveArray::raw_values
c848855 [Wes McKinney] Fix compiler warnings with gcc 4.9
ca9a374 [Wes McKinney] Add SetupCxxFlags.cmake from Apache Arrow. Add PARQUET_BUILD_WARNING_LEVEL flag. Fix Clang compiler warnings
Change-Id: I428d1d90bc4eb3dab8b56a538d1eb58656664b74
The main change is horizontal alignment. We should also do a clang-tidy pass sometime to do some further scrubbing
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closes#375 from wesm/PARQUET-1068 and squashes the following commits:
b81145d [Wes McKinney] Modify .clang-format to use straight Google format with 90-character line width
Change-Id: If8345d1d2a03d785ed41a5848de2c40e4bf53b5b
Author: Phillip Cloud <cpcloud@gmail.com>
Closes#369 from cpcloud/PARQUET-1053 and squashes the following commits:
e0598b4 [Phillip Cloud] PARQUET-1053: Fix unused result warnings due to unchecked Statuses
Change-Id: I91d267f56685c0e1267b6069c5bff52f2a1eca15
Extern template visibility contributes to be a mystery to me, but this fixes the regression from PARQUET-991.
cc @saatvikshah1994
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closes#352 from wesm/PARQUET-1029 and squashes the following commits:
3f52852 [Wes McKinney] Some extern template symbols not being exported in gcc
Change-Id: Icd71f67290c25de8f9fc38153b48a70e20bfa438
This supersedes #309 and incorporates the `std::shared_ptr<const KeyValueMetadata>` pattern so less copying is needed in Parquet for metadata inbound from Arrow (and vice versa).
close#309
Author: Wes McKinney <wes.mckinney@twosigma.com>
Author: Phillip Cloud <cpcloud@gmail.com>
Closes#314 from wesm/PARQUET-595 and squashes the following commits:
c0199c5 [Wes McKinney] Remove some more std::string includes
3d3be4e [Wes McKinney] Remove string include
b2ed09e [Wes McKinney] Add backwards compatible schema APIs
116575a [Wes McKinney] Use std::shared_ptr<const KeyValueMetadata> from upstream Arrow
5116eaa [Phillip Cloud] Add support for reading/writing Schema-level Arrow metadata
Change-Id: Ib46a73ac77cc952b032f0f93ee3297808b9f959e
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closes#259 from wesm/PARQUET-892 and squashes the following commits:
1e932c4 [Wes McKinney] Specify public link targets for parquet_static so that transitive dependencies are linked in executables
Change-Id: Ib0bea0cd272919dffa82aa1c12d7658ea15d9cb5
Also implements PARQUET-808: opening file with existing metadata object.
This allows a user to create a reader only for the purposes of obtaining the metadata.
Do you all think it's worth having a convenience method for reading the metadata out of a file?
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closes#213 from wesm/PARQUET-807 and squashes the following commits:
c1b5c7c [Wes McKinney] Use ReadMetaData function in test
d382cca [Wes McKinney] Add note about ARROW-455
05ecd37 [Wes McKinney] Implement/test opening with provided metadata. Do not close Arrow output files automatically
d790bb5 [Wes McKinney] Tweak
0dd4184 [Wes McKinney] Add ReadMetaData convenience method
97527ba [Wes McKinney] Change FileMetaData in ParquetFileReader to a shared_ptr so that ownership can be transferred away
Change-Id: Ic76c48671b41816d3dc1ebb736eb5a06c907c95d
This refactoring is a bit of a bloodbath, but I've attempted to preserve as much API backwards compatibility as possible.
Several points
* Arrow does not use exceptions, so will need to be very careful about making sure that no Status goes unchecked. I've tried to get most of them, but might have missed some
* parquet-cpp still exposes an abstract file read and write API as before, but this makes it easy to pass in an Arrow file handle (e.g. HDFS, OS files, memory maps, etc.)
* Custom memory allocators will need to subclass `arrow::MemoryPool` instead. If this becomes onerous for some reason, we can try to find alternatives, but basically it's the exact same class as `parquet::MemoryAllocator`
Does not require any upstream changes in Arrow.
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closes#210 from wesm/arrow-consolidation and squashes the following commits:
ef81084 [Wes McKinney] Configurable Arrow linkage. Slight .travis.yml cleaning
50b44f0 [Wes McKinney] Make some const refs
8438f86 [Wes McKinney] Revert ParquetFileReader::Open to use std::unique_ptr<RandomAccessFile>
671d981 [Wes McKinney] Actually tee output to console
ca8df13 [Wes McKinney] Do not hide test output from travis logs
f516115 [Wes McKinney] Add public link libs to dependencies to avoid race conditions with external projects
414c75f [Wes McKinney] README cleanups
be1acb5 [Wes McKinney] Move thirdparty ep's / setup to separate cmake module
46342ea [Wes McKinney] Remove unneeded ParquetAllocator interface, cleaning
b546f08 [Wes McKinney] Use MemoryAllocator alias within parquet core
8c1226d [Wes McKinney] Add Arrow to list of third party deps. Needs to be added to thirdparty
f9d8a2a [Wes McKinney] Check some unchecked Statuses
0d04820 [Wes McKinney] Fix benchmark builds. Do not fail in benchmarks if gtest.h is included due to <tr1/tuple> issue
ee312af [Wes McKinney] cpplint
6a05cd9 [Wes McKinney] Update installed header files
8d962f1 [Wes McKinney] Build and unit tests pass again
c82e2b4 [Wes McKinney] More refactoring
6ec5b71 [Wes McKinney] Re-expose original abstract IO interfaces, add Arrow subclasses that wrap inptu
c320c95 [Wes McKinney] clang-format
f10080c [Wes McKinney] Fix missed include
6ade22f [Wes McKinney] First cut refactoring, not fully compiling yet
Change-Id: Ibc4238396c035e185346c2562a0c6dc029c254ff
Uwe L. Korn