HTTPS is more secure.
I will leave this patch up in case anyone else wants to look for HTTP URLs to change to HTTPS. It's a bit hard to find them because of the `http://` in the Apache license header. Maybe we should change our Apache license headers to use https?
Author: Wes McKinney <wesm+git@apache.org>
Closes#4463 from wesm/use-https and squashes the following commits:
6ee7a4ef4 <Wes McKinney> Update some URLs from http to https
Comparison only supported for the left argument to be an array and the right argument a scalar. This extends support for comparing two arrays, but also supporting the case where the left argument is a scalar and the right an array.
Author: François Saint-Jacques <fsaintjacques@gmail.com>
Closes#4398 from fsaintjacques/ARROW-4990-compare-array-array and squashes the following commits:
864c67966 <François Saint-Jacques> Conform style guide
1402a98ee <François Saint-Jacques> Address review
b498eddc9 <François Saint-Jacques> mvcc again
c427313e8 <François Saint-Jacques> Make mvcc happy
56e467725 <François Saint-Jacques> autoformat
f6f5274fb <François Saint-Jacques> Supports comparison of Arrays
8643411d5 <François Saint-Jacques> Add binary operation support to PropagateNulls
53703acdd <François Saint-Jacques> Add length() to Datum.
Author: Antoine Pitrou <antoine@python.org>
Closes#4468 from pitrou/ARROW-5507-plasma-cuda-compile-error and squashes the following commits:
66387d290 <Antoine Pitrou> ARROW-5507: Fix compile error
This is for consistency and also for readability ("Integer" or "Number" do not explicitly stand out as *type* classes or instances).
Author: Antoine Pitrou <antoine@python.org>
Closes#4470 from pitrou/ARROW-5334-type-classes-type-suffix and squashes the following commits:
e55a372d9 <Antoine Pitrou> ARROW-5334: Ensure all type classes end with "Type"
Author: shengjun.li <shengjun.li@zilliz.com>
Closes#4277 from shengjun1985/gpu and squashes the following commits:
7207ed769 <shengjun.li> ARROW-5285: implement to release GpuProcessHandle
To be honest I'm not sure how these semicolons aren't causing an issue currently. Either way, removing them seems like a noop and improves clarity in my opinion.
Author: Brian Hulette <bhulette@google.com>
Closes#4422 from TheNeuralBit/semicolon and squashes the following commits:
af8c0a1e8 <Brian Hulette> remove semicolons after macro definitions
Author: Sutou Kouhei <kou@clear-code.com>
Closes#4447 from kou/cpp-rapidjson-version-check and squashes the following commits:
3271d8769 <Sutou Kouhei> Check required RapidJSON version
This enables ASAN/UBSAN in the clang builds. Pushing this for PR early to see if there are any comments on approach or additional checks to remove (the first two fixes where related to passing null values to memcpy).
Note the UBSAN behavior should already be configured to exclude unaligned pointer errors (I'll file a separate JIRA for fixing those).
Remaining for this PR:
* [X] Try to fix as many UBSAN/ASAN errors as possible (going to time box this to a week), and then blacklist the rest with a follow-up JIRA
CC @wesm @pitrou in case you have any early feedback
Author: Micah Kornfield <emkornfield@gmail.com>
Closes#4347 from emkornfield/use_asan and squashes the following commits:
bba5824c4 <Micah Kornfield> fix io/memory.cc asan
19ee908d7 <Micah Kornfield> handle signed and unsigned enum loading in parquet to support windows
dc300d4cb <Micah Kornfield> add back asan
05467be2f <Micah Kornfield> ubsan stuff
LLVM 7.1.0 was released to fix https://llvm.org/PR39427 . It needs
ABI breakage. So 7.1.0 isn't 7.0.2. No other big changes exist between 7.1.0
and 7.0.1. So we can support both LLVM 7.1 and 7.0.
LLVM 7.1.0 is tested in building deb packages. Changed under
dev/tasks/linux-packages/ are for running test in building deb
packages.
We can't run tests for deb packages of Ubuntu 16.04. Because Ubuntu 16.04's
packages are old. So dev/tasks/linux-packages/debian.ubuntu-xenial/ is created.
It's based on dev/tasks/linux-packages/debian/ before we change.
Author: Sutou Kouhei <kou@clear-code.com>
Closes#4453 from kou/cpp-llvm-7.1 and squashes the following commits:
6dbb9f3be <Sutou Kouhei> Add support for LLVM 7.1
We currently do not write EOS at the end of a Message stream inside the File format. As a result, the file cannot be parsed sequentially. This change prepares for other implementations or future reference features that parse a File sequentially... i.e. without access to seek().
Author: John Muehlhausen <jmuehlhausen@kershnertrading.com>
Closes#4372 from jgm-ktg/master and squashes the following commits:
2ce546cc3 <John Muehlhausen> remove JS/Java for separate JIRA issues
350177829 <John Muehlhausen> ARROW-5395: Python test case
9e83053d0 <John Muehlhausen> incorporate first suggestions for writing EOS in File
901dfeab0 <John Muehlhausen> ARROW-5395: Utilize stream EOS in File format
This yields approximately 15% net improvement in build time when building with msbuild and `/MP` on a quad-core desktop. Parallel test execution times on Linux/macOS should be unaffected -- I took care to only combine executables where it made sense and where execution time was already very fast (100s of milliseconds or less)
Author: Wes McKinney <wesm+git@apache.org>
Closes#4442 from wesm/ARROW-4504 and squashes the following commits:
35b81f63a <Wes McKinney> Revert changes to io/CMakeLists.txt
19beb21b7 <Wes McKinney> Merge unit test objects to get down to 78 unit tests total
Previously, this case would cause ArrowException: Unknown error, but ArrowInvalid makes more sense and is more likely to be dealt with by callers of pyarrow.array.
Author: Robin Kåveland Hansen <Robin.Kaveland.Hansen@posten.no>
Closes#4440 from kaaveland/master and squashes the following commits:
b95aa0253 <Robin Kåveland Hansen> ARROW-5430: Raise ArrowInvalid for pyints larger than int64
Seeking is required when calling an implicitly-positioned operation such as Read() after ReadAt().
Author: Antoine Pitrou <antoine@python.org>
Closes#4417 from pitrou/ARROW-2835-read-at-seek and squashes the following commits:
d74c60eea <Antoine Pitrou> ARROW-2835: Make file position undefined after ReadAt()
Also make Python Flight support compatible with PYARROW_BUNDLE_ARROW_CPP.
Author: Antoine Pitrou <antoine@python.org>
Closes#4415 from pitrou/ARROW-5441-find-arrow-flight and squashes the following commits:
c40fdcf91 <Antoine Pitrou> ARROW-5441: Implement FindArrowFlight.cmake
The array/ subfolder is a more natural place to organize algorithms and other code relating to manipulating columnar array objects
Author: Wes McKinney <wesm+git@apache.org>
Closes#4445 from wesm/ARROW-5289 and squashes the following commits:
002dfc987 <Wes McKinney> Move arrow/util/concatenate* to arrow/array
This will help prevent Doxygen-related build failures
Author: Wes McKinney <wesm+git@apache.org>
Closes#4441 from wesm/ARROW-4159 and squashes the following commits:
a019c1484 <Wes McKinney> Build with -Wdocumentation on clang
Author: Renat Valiullin <rip.nsk@gmail.com>
Closes#4403 from rip-nsk/ARROW-5433 and squashes the following commits:
2daadd94c <Renat Valiullin> replace column name by column path and better type information
Author: Antoine Pitrou <antoine@python.org>
Closes#4410 from pitrou/ARROW-3294-flight-windows and squashes the following commits:
bd4979b33 <Antoine Pitrou> ARROW-3294: Support Flight on Windows
This is a long overdue unification of platform code that wasn't possible until after the monorepo merge that occurred last year. This should also permit us to take a more consistent approach with regards to asynchronous IO.
A backwards compatibility layer is provided for the now deprecated `parquet::RandomAccessSource` and `parquet::OutputStream` classes.
Some incidental changes were required to get things to work:
* ARROW-5428: Adding a "read extent" option to BufferedInputStream to limit the extent of bytes read from the underlying raw stream
* `arrow::io::InputStream::Peek` needed to have its API changed to return Status, because of the next point
* `arrow::io::BufferedOutputStream::Peek` will expand the buffer if a Peek is requested that is larger than the buffer. The idea is that it should be possible to "look ahead" in the stream without altering the stream position. This is needed as part of finding the next data header (which can be large or small depending on statistics size, etc.) in a Parquet stream
* Added a `[]` operator to `Buffer` to facilitate testing
* Some continued "flattening" of the "parquet/util" directory to be simpler
Some outstanding questions:
* The Parquet reader and writer classes assumed exclusive ownership of the file handles, and they are closed when the Parquet file is closed. Arrow files are shared, and so calling `Close` is not appropriate. I've attempted to preserve this logic by having Close called in the destructors of the wrapper classes in `parquet/deprecated_io.h`
An issue I ran into
* Changes in https://github.com/apache/arrow/commit/d82ac407fab1d4b28669b8f7a940f88d39dfd874 introduced a unit test with meaningful trailing whitespace, which my editor strips away. I've commented out the offending test and will have to open a JIRA about fixing
Author: Wes McKinney <wesm+git@apache.org>
Closes#4404 from wesm/parquet-use-arrow-io and squashes the following commits:
f010a8ec5 <Wes McKinney> Add missing PARQUET_EXPORT macros
50f7b921d <Wes McKinney> Add missing PARQUET_EXPORT
3b27ac262 <Wes McKinney> Follow changes in c_glib, fix Doxygen warning
7c1ae55c3 <Wes McKinney> ReadableFile::Peek now returns NotImplemented
cc7789e8f <Wes McKinney> Fix unit tests
b6e173922 <Wes McKinney> Allow unbounded peeks in BufferedInputStream
cd2a3cd70 <Wes McKinney> Add unit tests for legacy Parquet input/output wrappers
e03f07d65 <Wes McKinney> remove outdated comment
4c40bf2e1 <Wes McKinney> Adapt Python bindings
769974a6e <Wes McKinney> Tests passing again
1886de841 <Wes McKinney> column_writer more similar to before
7efc1aca6 <Wes McKinney> Fix one bug
30f1f4d62 <Wes McKinney> Get things compiling again, but tests are broken
4efb4e707 <Wes McKinney> Implement expanding-peek logic, change signature of InputStream::Peak to be able to return Status
db1877e8c <Wes McKinney> More progress toward compilation, port over parquet::BufferedInputStream unit tests
b05a71213 <Wes McKinney> More refactoring
66be1af04 <Wes McKinney> Port more code, add basic wrapper implementation for legacy IO interfaces
59143ddec <Wes McKinney> Start a bit of refactoring/consolidation in prep for using Arrow IO interfaces
I think we need to put cmake-format back into the doghouse until we observe a period of stability in the formatting algorithm
Author: Wes McKinney <wesm+git@apache.org>
Closes#4423 from wesm/ARROW-5453 and squashes the following commits:
40c4f8a43 <Wes McKinney> Update to cmake-format=0.5.2 and pin again
Without this change, failed assertions do not cause failed unit tests on Windows as Antoine had discovered today.
The conda-forge test package seems unusable in shared library form so I'm forcing BUNDLED builds on MSVC (if not otherwise specified) and removing the hacks in the docs and Appveyor to pass `-DGTest_SOURCE=BUNDLED`
After this change the static CRT Windows tests do not work anymore (because gtest.dll has its own copy of the static CRT, which leads to conflicts), so we will have to explore testing the static CRT build as follow up work. See ARROW-5426
Author: Wes McKinney <wesm+git@apache.org>
Closes#4380 from wesm/gtest-windows-dll and squashes the following commits:
515f75b0 <Wes McKinney> Only set GTest_SOURCE to BUNDLED on MSVC if not passed explicitly
8678632d <Wes McKinney> Revert more unwanted cmake changes
c88d125d <Wes McKinney> Revert unwanted cmake-format changes
ac73ae37 <Wes McKinney> Disable static CRT build
c83c7d63 <Wes McKinney> Use gtest shared lib
Author: Antoine Pitrou <antoine@python.org>
Closes#4379 from pitrou/ARROW-5378-local-filesystem and squashes the following commits:
c36e3dc8b <Antoine Pitrou> ARROW-5378: Local filesystem implementation
The goal of this change is to mark benchmarks candidate for automated regression checks. Some benchmarks were refactored for various reasons:
- Code deduplication where needed
- Uniform input size for benchmarks in the same suite, usually to minimize the effect of cache hierarchy.
- Favor external repetitions over manual repetitions and mintime when possible.
- Add cmake ARROW_BUILD_BENCHMARKS_REFERENCE to toggle reference benchmarks.
- Remove default benchmark filter of `^Regression`.
- Remove `BM_` prefix from benchmark names
- Fixes and addons to `archery` to support the `--pdb` option for debugging and add support for benchmarks reported as `items_per_seconds`.
- Adds `archery benchmark list` sub-command to list suites and benchmarks
Author: François Saint-Jacques <fsaintjacques@gmail.com>
Closes#4285 from fsaintjacques/ARROW-5269-regression-candidates and squashes the following commits:
83780304d <François Saint-Jacques> Address comments
4a570ab90 <François Saint-Jacques> Reformat
a9ecffd98 <François Saint-Jacques> address comments
d8e779aab <François Saint-Jacques> Reformat
8334c68f7 <François Saint-Jacques> Remove BM_ benchmark prefix
5cebe2c23 <François Saint-Jacques> Normalize benchmarks
908f05a5b <François Saint-Jacques> Change default repetitions to 10 instead of 20
ffe723f05 <François Saint-Jacques> Add `benchmark list` sub-command.
e0876474b <François Saint-Jacques> Refactor JSON benchmarks
a7b4f5f50 <François Saint-Jacques> Improve archery benchmark support
bea76ed0e <François Saint-Jacques> Refactor csv benchmarks
f207974c0 <François Saint-Jacques> Refactor CompareFilter benchmarks
b7911df3d <François Saint-Jacques> Refactor Bitmap benchmarks
1b5135ebd <François Saint-Jacques> Refactor Builder benchmarks
Author: Antoine Pitrou <antoine@python.org>
Closes#4396 from pitrou/ARROW-5419-csv-empty-null-strings and squashes the following commits:
cb086d1c <Antoine Pitrou> ARROW-5419: Allow recognizing empty strings as null strings in CSV files
Author: shengjun.li <shengjun.li@zilliz.com>
Closes#4272 from shengjun1985/master and squashes the following commits:
126a60e35 <shengjun.li> Add a comment.
f7113c2cc <shengjun.li> ARROW-5283: erase object id in client when abort object
Author: Antoine Pitrou <antoine@python.org>
Closes#4392 from pitrou/ARROW-5413-csv-skip-bom and squashes the following commits:
12f8d1d3 <Antoine Pitrou> ARROW-5413: Skip UTF8 BOM in CSV files
This PR updates the parquet-cpp implementation to use parameterized logical type annotations as requested in [PARQUET-1411](https://issues.apache.org/jira/browse/PARQUET-1411). The primary contributions are:
1. Amend the parquet.thrift file consistent with the Parquet format repository to allow Thrift to recognize and serialize a new LogicalType
2. Introduce and integrate a LogicalAnnotation class (and subclasses) in the parquet-cpp library to handle functionality related to this new attribute
3. Expand the public API to include construction functions for GroupNode and PrimitiveNode schema classes that accept the new LogicalAnnotation parameter
4. Add basic unit and integration tests for the new code.
Some (hopefully time saving) notes for reviewers:
- The center of gravity for the PR is the LogicalAnnotation class, so it might be best to start by having a look at it's interface in types.h
- LogicalType would have been a more natural name for this class, but unfortunately that was already in use (in the public API) for the concept Thrift calls "converted type". Here's a refresher chart of some relevant concepts and their names in the code:
| Concept | Thrift realization | Parquet-CPP realization |
|---------|--------------------|-------------------------|
| Physical storage type | enum Type | enum parquet::Type::type |
| Converted type | enum ConvertedType; struct parquet::format::ConvertedType | enum parquet::LogicalType::type; struct parquet::schema::DecimalMetadata|
| Logical annotation | union LogicalType; struct parquet::format::LogicalType | class parquet::LogicalAnnotation (enum parquet::LogicalAnnotation::Type) |
- After this change, the LogicalAnnotation member (stored in Node.logical_annotation_) is the primary controller of logical typing in the library. To simplify backward compatibility tasks, the existing converted type/decimal metadata pair is retained and populated at construction with compatible values (where possible).
- While new Make() functions are introduced to accomodate schema node construction with the new annotations, of course the existing corresponding public API functions accepting converted types are retained. The maintainers might wish to consider deprecating those functions.
- As required by the parquet format specification, after this change, both the converted and logical types are serialized; the new logical type (if present) takes precedence on deserialization, but the legacy converted type is still recognized in the absence of a logical type.
>The existing converted type INTERVAL has no corresponding logical type in the parquet.thrift definition, so it can only be serialized as a converted type. As with any other element in which only converted type information is present, upon deserialization an equivalent logical annotation is instantiated and it is treated internally as any other type.
- There are several ancillary functions, such as printing and equivalence checking, in which the converted types are still used; in most cases these fallbacks and double-checks are probably redundant. Note, however, that in the JSON output *both* converted and logical types are intentionally emitted, on the assumption that consumers might be relying on the current format as part of a quasi-public API. The maintainers may wish to consider deprecating the converted type fields there as well.
I'm far from an expert in the Parquet data format or it's implementing libraries, so I may have misunderstood or omitted things; all corrections or recommendations are obviously welcome.
Author: TP Boudreau <tpboudreau@gmail.com>
Closes#4185 from tpboudreau/PARQUET-1411 and squashes the following commits:
00ef84d2b <TP Boudreau> Widen access to implementation classes
fdfbf87fc <TP Boudreau> Include logical annotation in method ColumnDescriptor::ToString()
a5e2386b2 <TP Boudreau> Add and integrate logical annotations for schema nodes
This is an RFC based on the discussion ongoing. If this seems like the accepted approach I'll add a Python unit test
Author: Wes McKinney <wesm+git@apache.org>
Closes#4386 from wesm/parquet-metadata-set-file-path and squashes the following commits:
e77cf7a8a <Wes McKinney> Add method to set file path in a parquet::FileMetaData instance
It looks like some formatting has changed going from 4 to 5. For the most part it looks more readable, but I'm not sure.
Author: Micah Kornfield <emkornfield@gmail.com>
Closes#4388 from emkornfield/unpin_cmake and squashes the following commits:
e2d48fd2f <Micah Kornfield> run cmake format and turn off autosort
c965fc994 <Micah Kornfield> ARROW-5245: Unpin cmake format
Author: Wes McKinney <wesm+git@apache.org>
Closes#4240 from wesm/PARQUET-1243 and squashes the following commits:
dd61458b5 <Wes McKinney> clang-format
75ee6ef01 <Wes McKinney> Print expected size of standard file footer, use consistent style for constants in parquet/file_reader.cc
3b3f47523 <Wes McKinney> Raise a better exception when reading a length-0 Parquet file
Otherwise libraries built with C++11 enabled can have ABI conflicts when built against applications built with C++ >= 17
Author: Benjamin Kietzman <bengilgit@gmail.com>
Closes#4378 from bkietz/5404-nonstdstring-view-conflicts-with-stdstri and squashes the following commits:
4ff84802b <Benjamin Kietzman> force usage of nonstd::sv_lite::string_view instead of std::string_view
I haven't changed the client/server construction interfaces to really take advantage of this. I would rather follow up with creating builders instead, as proposed, as that would be cleaner than special-casing the logic in the current constructors.
Author: David Li <li.davidm96@gmail.com>
Closes#4047 from lihalite/flight-uris and squashes the following commits:
870f6ebb8 <David Li> Add more builder options for Java Flight servers
5c127631b <David Li> Make Python Flight bindings more complete
675acc9bc <David Li> Introduce builder for C++ Flight servers
474419667 <David Li> Use builder for Flight server in Java
460442cea <David Li> Use URIs for Flight locations
…adable
pyarrow needs to handle dictionary page offset = 0 as a special case to be compatible with java parquet reader.
Author: shyam <shyam@dremio.com>
Closes#4359 from shyambits2004/5322 and squashes the following commits:
f47762ab0 <shyam> Parquet files with dictionary page offset as 0 is not readable
Author: Micah Kornfield <emkornfield@gmail.com>
Closes#4335 from emkornfield/fix_compression_tests and squashes the following commits:
86ddd9963 <Micah Kornfield> revert cmopressed size
e37f8ad55 <Micah Kornfield> Change back to using resize
9b51c5809 <Micah Kornfield> ARROW-5362: Fix compression test memory usage
This class which is a syntactic sugar layer on `arrow::ResizableBuffer` was only being used in one place, for dictionary decoding. I don't think we need to maintain this, and it also features possible instantiated template symbol issues
Author: Wes McKinney <wesm+git@apache.org>
Closes#4354 from wesm/PARQUET-1583 and squashes the following commits:
24535175 <Wes McKinney> lint
29af0799 <Wes McKinney> Remove superfluous parquet::Vector class
Author: Micah Kornfield <emkornfield@gmail.com>
Author: Micah Kornfield <micahk@google.com>
Closes#4338 from emkornfield/add_debug_string and squashes the following commits:
25980a37a <Micah Kornfield> Add unit test
9568a6744 <Micah Kornfield> Add ToString method to ColumnDescripotor
Because we can't run autogen.sh from cmake.
We can use jemalloc installed by pacman but we can't use it. Because
we always use bundled jemalloc for now.
Author: Kouhei Sutou <kou@clear-code.com>
Closes#4341 from kou/cpp-mingw-disable-jemalloc-by-default and squashes the following commits:
e107efbd <Kouhei Sutou> Disable jemalloc by default with MinGW
Author: Antoine Pitrou <antoine@python.org>
Closes#4267 from pitrou/ARROW-5260-from-components-other-process and squashes the following commits:
bc56aa668 <Antoine Pitrou> ARROW-5260: Fix crash when deserializating from components in another process
Wes McKinney