mirrors/arrow
0
0 Fork 0
mirror of https://github.com/apache/arrow.git synced 2026-10-28 16:47:19 +08:00
Code Activity

Some docs reorg, begin rewriting cpp/README.md into docs/source/developers/cpp.rst

Browse Source
This commit is contained in:
Wes McKinney committed 2019-03-17 14:52:20 -05:00
1 parent 9d73e0a544
commit 84dd680a24
15 files changed
+871 -781
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

README.md
+3 -47
View File
@@ -49,23 +49,9 @@ Major components of the project include:
Arrow is an [Apache Software Foundation](https://www.apache.org) project. Learn more at
[arrow.apache.org](https://arrow.apache.org).
## What's in the Arrow libraries?
## How to Contribute
The reference Arrow libraries contain a number of distinct software components:
- Columnar vector and table-like containers (similar to data frames) supporting
flat or nested types
- Fast, language agnostic metadata messaging layer (using Google's Flatbuffers
library)
- Reference-counted off-heap buffer memory management, for zero-copy memory
sharing and handling memory-mapped files
- Low-overhead IO interfaces to files on disk, HDFS (C++ only)
- Self-describing binary wire formats (streaming and batch/file-like) for
remote procedure calls (RPC) and
interprocess communication (IPC)
- Integration tests for verifying binary compatibility between the
implementations (e.g. sending data from Java to C++)
- Conversions to and from other in-memory data structures
Please read our latest [project contribution guide][5].
## Getting involved
@@ -79,38 +65,8 @@ integrations in other projects, we'd be happy to have you involved:
- [Learn the format][2]
- Contribute code to one of the reference implementations
## How to Contribute
We prefer to receive contributions in the form of GitHub pull requests. Please
send pull requests against the [github.com/apache/arrow][4] repository.
If you are looking for some ideas on what to contribute, check out the [JIRA
issues][3] for the Apache Arrow project. Comment on the issue and/or contact
[dev@arrow.apache.org](http://mail-archives.apache.org/mod_mbox/arrow-dev/)
with your questions and ideas.
If you’d like to report a bug but don’t have time to fix it, you can still post
it on JIRA, or email the mailing list
[dev@arrow.apache.org](http://mail-archives.apache.org/mod_mbox/arrow-dev/)
To contribute a patch:
1. Break your work into small, single-purpose patches if possible. It’s much
harder to merge in a large change with a lot of disjoint features.
2. Create a JIRA for your patch on the [Arrow Project
JIRA](https://issues.apache.org/jira/browse/ARROW).
3. Submit the patch as a GitHub pull request against the master branch. For a
tutorial, see the GitHub guides on forking a repo and sending a pull
request. Prefix your pull request name with the JIRA name (ex:
https://github.com/apache/arrow/pull/240).
4. Make sure that your code passes the unit tests. You can find instructions
how to run the unit tests for each Arrow component in its respective README
file.
5. Add new unit tests for your code.
Thank you in advance for your contributions!
[1]: mailto:dev-subscribe@arrow.apache.org
[2]: https://github.com/apache/arrow/tree/master/format
[3]: https://issues.apache.org/jira/browse/ARROW
[4]: https://github.com/apache/arrow
[5]: https://github.com/apache/arrow/blob/master/docs/source/developers/contributing.rst
ci/conda_env_cpp.yml
+1 -1
View File
@@ -38,6 +38,6 @@ python
rapidjson
re2
snappy
thrift-cpp
thrift-cpp=0.12.0
zlib
zstd
cpp/README.md
+7 -543
View File
@@ -17,554 +17,18 @@
under the License.
-->
# Apache Arrow C++ codebase
# Apache Arrow C++
This directory contains the code and build system for the Arrow C++ libraries,
as well as for the C++ libraries for Apache Parquet.
## System setup
## Installation
Arrow uses CMake as a build configuration system. Currently, it supports
in-source and out-of-source builds with the latter one being preferred.
See http://arrow.apache.org/install/ for the latest instructions how to install
pre-compiled binary versions of the library.
Building Arrow requires:
## Source Builds and Development
* A C++11-enabled compiler. On Linux, gcc 4.8 and higher should be sufficient.
* CMake 3.2 or higher
* Boost
* Bison/flex (for building Apache Thrift from source only,
a parquet dependency.)
Please refer to our latest [C++ Development Documentation][1].
Testing arrow with ctest requires:
* python
On Ubuntu/Debian you can install the requirements with:
```shell
sudo apt-get install \
autoconf \
build-essential \
cmake \
libboost-dev \
libboost-filesystem-dev \
libboost-regex-dev \
libboost-system-dev \
python \
bison \
flex
```
On Alpine Linux:
```shell
apk add autoconf \
bash \
boost-dev \
cmake \
g++ \
gcc \
make
```
On macOS, you can use [Homebrew][1]:
```shell
git clone https://github.com/apache/arrow.git
cd arrow
brew update && brew bundle --file=c_glib/Brewfile
```
If you are developing on Windows, see the [Windows developer guide][2].
## Building Arrow
Simple release build:
git clone https://github.com/apache/arrow.git
cd arrow/cpp
mkdir release
cd release
cmake -DARROW_BUILD_TESTS=ON ..
make unittest
Simple debug build:
git clone https://github.com/apache/arrow.git
cd arrow/cpp
mkdir debug
cd debug
cmake -DCMAKE_BUILD_TYPE=Debug -DARROW_BUILD_TESTS=ON ..
make unittest
If you do not need to build the test suite, you can omit the
`ARROW_BUILD_TESTS` option (the default is not to build the unit tests).
Detailed unit test logs will be placed in the build directory under
`build/test-logs`.
On some Linux distributions, running the test suite might require setting an
explicit locale. If you see any locale-related errors, try setting the
environment variable (which requires the `locales` package or equivalent):
```
export LC_ALL="en_US.UTF-8"
```
## Modular Build Targets
Since there are several major parts of the C++ project, we have provided
modular CMake targets for building each library component, group of unit tests
and benchmarks, and their dependencies:
* `make arrow` for Arrow core libraries
* `make parquet` for Parquet libraries
* `make gandiva` for Gandiva (LLVM expression compiler) libraries
* `make plasma` for Plasma libraries, server
To build the unit tests or benchmarks, add `-tests` or `-benchmarks` to the
target name. So `make arrow-tests` will build the Arrow core unit tests. Using
the `-all` target, e.g. `parquet-all`, will build everything.
If you wish to only build and install one or more project subcomponents, we
have provided the CMake option `ARROW_OPTIONAL_INSTALL` to only install targets
that have been built. For example, if you only wish to build the Parquet
libraries, its tests, and its dependencies, you can run:
```
cmake .. -DARROW_PARQUET=ON -DARROW_OPTIONAL_INSTALL=ON -DARROW_BUILD_TESTS=ON
make parquet
make install
```
If you omit an explicit target when invoking `make`, all targets will be built.
## Parquet Development Notes
To build the C++ libraries for Apache Parquet, add the flag
`-DARROW_PARQUET=ON` when invoking CMake. The Parquet libraries and unit tests
can be built with the `parquet` make target:
```shell
make parquet
```
Running `ctest -L unittest` will run all built C++ unit tests, while `ctest -L
parquet` will run only the Parquet unit tests. The unit tests depend on an
environment variable `PARQUET_TEST_DATA` that depends on a git submodule to the
repository https://github.com/apache/parquet-testing:
```shell
git submodule update --init
export PARQUET_TEST_DATA=$ARROW_ROOT/cpp/submodules/parquet-testing/data
```
Here `$ARROW_ROOT` is the absolute path to the Arrow codebase.
### Statically linking to Arrow on Windows
The Arrow headers on Windows static library builds (enabled by the CMake
option `ARROW_BUILD_STATIC`) use the preprocessor macro `ARROW_STATIC` to
suppress dllimport/dllexport marking of symbols. Projects that statically link
against Arrow on Windows additionally need this definition. The Unix builds do
not use the macro.
### Building/Running benchmarks
Follow the directions for simple build except run cmake
with the `--ARROW_BUILD_BENCHMARKS` parameter set correctly:
cmake -DARROW_BUILD_TESTS=ON -DARROW_BUILD_BENCHMARKS=ON ..
and instead of make unittest run either `make; ctest` to run both unit tests
and benchmarks or `make benchmark` to run only the benchmark tests.
Benchmark logs will be placed in the build directory under `build/benchmark-logs`.
### Testing with LLVM AddressSanitizer
To use AddressSanitizer (ASAN) to find bad memory accesses or leaks with LLVM,
pass `-DARROW_USE_ASAN=ON` when building. You must use clang to compile with
ASAN, and `ARROW_USE_ASAN` is mutually-exclusive with the valgrind option
`ARROW_TEST_MEMCHECK`.
### Building/Running fuzzers
Fuzzers can help finding unhandled exceptions and problems with untrusted input
that may lead to crashes, security issues and undefined behavior. They do this
by generating random input data and observing the behavior of the executed
code. To build the fuzzer code, LLVM is required (GCC-based compilers won't
work). You can build them using the following code:
cmake -DARROW_FUZZING=ON -DARROW_USE_ASAN=ON ..
`ARROW_FUZZING` will enable building of fuzzer executables as well as enable the
addition of coverage helpers via `ARROW_USE_COVERAGE`, so that the fuzzer can observe
the program execution.
It is also wise to enable some sanitizers like `ARROW_USE_ASAN` (see above), which
activates the address sanitizer. This way, we ensure that bad memory operations
provoked by the fuzzer will be found early. You may also enable other sanitizers as
well. Just keep in mind that some of them do not work together and some may result
in very long execution times, which will slow down the fuzzing procedure.
Now you can start one of the fuzzer, e.g.:
./debug/debug/ipc-fuzzing-test
This will try to find a malformed input that crashes the payload and will show the
stack trace as well as the input data. After a problem was found this way, it should
be reported and fixed. Usually, the fuzzing process cannot be continued until the
fix is applied, since the fuzzer usually converts to the problem again.
If you build fuzzers with ASAN, you need to set the `ASAN_SYMBOLIZER_PATH`
environment variable to the absolute path of `llvm-symbolizer`, which is a tool
that ships with LLVM.
```shell
export ASAN_SYMBOLIZER_PATH=$(type -p llvm-symbolizer)
```
Note that some fuzzer builds currently reject paths with a version qualifier
(like `llvm-sanitizer-5.0`). To overcome this, set an appropriate symlink
(here, when using LLVM 5.0):
```shell
ln -sf /usr/bin/llvm-sanitizer-5.0 /usr/bin/llvm-sanitizer
```
There are some problems that may occur during the compilation process:
- libfuzzer was not distributed with your LLVM: `ld: file not found: .../libLLVMFuzzer.a`
- your LLVM is too old: `clang: error: unsupported argument 'fuzzer' to option 'fsanitize='`
### Third-party dependencies and configuration
Arrow depends on a number of third-party libraries. We support these in a few
ways:
* Building dependencies from source by downloading archives from the internet
* Building dependencies from source using from local archives (to allow offline
builds)
* Building with locally-installed libraries
See [thirdparty/README.md][5] for details about these options and how to
configure your build toolchain.
### Building Python integration library (optional)
The optional `arrow_python` shared library can be built by passing
`-DARROW_PYTHON=on` to CMake. This must be installed or in your library load
path to be able to build pyarrow, the Arrow Python bindings.
The Python library must be built against the same Python version for which you
are building pyarrow, e.g. Python 2.7 or Python 3.6. NumPy must also be
installed.
### Building CUDA extension library (optional)
The optional `arrow_cuda` shared library can be built by passing
`-DARROW_CUDA=on`. This requires a CUDA installation to build, and to use many
of the functions you must have a functioning CUDA-compatible GPU.
The CUDA toolchain used to build the library can be customized by using the
`$CUDA_HOME` environment variable.
This library is still in Alpha stages, and subject to API changes without
deprecation warnings.
### Building Apache ORC integration (optional)
The optional arrow reader for the Apache ORC format (found in the
`arrow::adapters::orc` namespace) can be built by passing `-DARROW_ORC=on`.
This is currently not supported on windows. Note that this functionality is
still in Alpha stages, and subject to API changes without deprecation warnings.
### Building and developing Gandiva (optional)
The Gandiva library supports compiling and evaluating expressions on arrow
data. It uses LLVM for doing just-in-time compilation of the expressions.
In addition to the arrow dependencies, gandiva requires :
* On linux, gcc 4.9 or higher C++11-enabled compiler.
* LLVM
On Ubuntu/Debian you can install these requirements with:
```shell
sudo apt-add-repository -y "deb http://llvm.org/apt/trusty/ llvm-toolchain-trusty-7.0 main"
sudo apt-get update -qq
sudo apt-get install llvm-7.0-dev
```
On macOS, you can use [Homebrew][1]:
```shell
brew install llvm@7
```
The optional `gandiva` libraries and tests can be built by passing
`-DARROW_GANDIVA=on`.
```shell
cmake .. -DARROW_GANDIVA=ON -DARROW_BUILD_TESTS=ON
make
ctest -L gandiva
```
This library is still in Alpha stages, and subject to API changes without
deprecation warnings.
### Building and developing Flight (optional)
In addition to the Arrow dependencies, Flight requires:
* gRPC (>= 1.14, roughly)
* Protobuf (>= 3.6, earlier versions may work)
* c-ares (used by gRPC)
By default, Arrow will try to download and build these dependencies
when building Flight.
The optional `flight` libraries and tests can be built by passing
`-DARROW_FLIGHT=ON`.
```shell
cmake .. -DARROW_FLIGHT=ON -DARROW_BUILD_TESTS=ON
make
```
You can also use existing installations of the extra dependencies.
When building, set the environment variables `GRPC_HOME` and/or
`PROTOBUF_HOME` and/or `CARES_HOME`.
You may try using system libraries for gRPC and Protobuf, but these
are likely to be too old.
On Ubuntu/Debian, you can try:
```shell
sudo apt-get install libgrpc-dev libgrpc++-dev protobuf-compiler-grpc libc-ares-dev
```
Note that the version of gRPC in Ubuntu 18.10 is too old; you will
have to install gRPC from source. (Ubuntu 19.04/Debian Sid may work.)
On macOS, you can try [Homebrew][1]:
```shell
brew install grpc
```
You can also install gRPC from source. In this case, you must install
gRPC to generate the necessary files for CMake to find gRPC:
```shell
cmake -DgRPC_INSTALL=ON -DgRPC_BUILD_TESTS=OFF -DgRPC_PROTOBUF_PROVIDER=package -DgRPC_ZLIB_PROVIDER=package -DgRPC_CARES_PROVIDER=package -DgRPC_SSL_PROVIDER=package
```
You can then specify `-DgRPC_DIR` to `cmake`.
### API documentation
To generate the (html) API documentation, run the following command in the apidoc
directory:
doxygen Doxyfile
This requires [Doxygen](http://www.doxygen.org) to be installed.
## Development
This project follows [Google's C++ Style Guide][3] with minor exceptions:
* We relax the line length restriction to 90 characters.
* We use the NULLPTR macro defined in `src/arrow/util/macros.h` to
support building C++/CLI (ARROW-1134)
* We use doxygen style comments ("///") instead of line comments ("//")
in header files.
### Memory Pools
We provide a default memory pool with `arrow::default_memory_pool()`. As a
matter of convenience, some of the array builder classes have constructors
which use the default pool without explicitly passing it. You can disable these
constructors in your application (so that you are accounting properly for all
memory allocations) by defining `ARROW_NO_DEFAULT_MEMORY_POOL`.
### Header files
We use the `.h` extension for C++ header files. Any header file name not
containing `internal` is considered to be a public header, and will be
automatically installed by the build.
### Error Handling and Exceptions
For error handling, we use `arrow::Status` values instead of throwing C++
exceptions. Since the Arrow C++ libraries are intended to be useful as a
component in larger C++ projects, using `Status` objects can help with good
code hygiene by making explicit when a function is expected to be able to fail.
For expressing invariants and "cannot fail" errors, we use DCHECK macros
defined in `arrow/util/logging.h`. These checks are disabled in release builds
and are intended to catch internal development errors, particularly when
refactoring. These macros are not to be included in any public header files.
Since we do not use exceptions, we avoid doing expensive work in object
constructors. Objects that are expensive to construct may often have private
constructors, with public static factory methods that return `Status`.
There are a number of object constructors, like `arrow::Schema` and
`arrow::RecordBatch` where larger STL container objects like `std::vector` may
be created. While it is possible for `std::bad_alloc` to be thrown in these
constructors, the circumstances where they would are somewhat esoteric, and it
is likely that an application would have encountered other more serious
problems prior to having `std::bad_alloc` thrown in a constructor.
### Extra debugging help
If you use the CMake option `-DARROW_EXTRA_ERROR_CONTEXT=ON` it will compile
the libraries with extra debugging information on error checks inside the
`RETURN_NOT_OK` macro. In unit tests with `ASSERT_OK`, this will yield error
outputs like:
```
../src/arrow/ipc/ipc-read-write-test.cc:609: Failure
Failed
NotImplemented: ../src/arrow/ipc/ipc-read-write-test.cc:574 code: writer->WriteRecordBatch(batch)
../src/arrow/ipc/writer.cc:778 code: CheckStarted()
../src/arrow/ipc/writer.cc:755 code: schema_writer.Write(&dictionaries_)
../src/arrow/ipc/writer.cc:730 code: WriteSchema()
../src/arrow/ipc/writer.cc:697 code: WriteSchemaMessage(schema_, dictionary_memo_, &schema_fb)
../src/arrow/ipc/metadata-internal.cc:651 code: SchemaToFlatbuffer(fbb, schema, dictionary_memo, &fb_schema)
../src/arrow/ipc/metadata-internal.cc:598 code: FieldToFlatbuffer(fbb, *schema.field(i), dictionary_memo, &offset)
../src/arrow/ipc/metadata-internal.cc:508 code: TypeToFlatbuffer(fbb, *field.type(), &children, &layout, &type_enum, dictionary_memo, &type_offset)
Unable to convert type: decimal(19, 4)
```
### Deprecations and API Changes
We use the compiler definition `ARROW_NO_DEPRECATED_API` to disable APIs that
have been deprecated. It is a good practice to compile third party applications
with this flag to proactively catch and account for API changes.
### Keeping includes clean with include-what-you-use
We have provided a `build-support/iwyu/iwyu.sh` convenience script for invoking
Google's [include-what-you-use][4] tool, also known as IWYU. This includes
various suppressions for more informative output. After building IWYU
(following instructions in the README), you can run it on all files by running:
```shell
CC="clang-4.0" CXX="clang++-4.0" cmake -DCMAKE_EXPORT_COMPILE_COMMANDS=ON ..
../build-support/iwyu/iwyu.sh all
```
This presumes that `include-what-you-use` and `iwyu_tool.py` are in your
`$PATH`. If you compiled IWYU using a different version of clang, then
substitute the version number above accordingly.
We have provided a Docker-based IWYU to make it easier to run these
checks. This can be run using the docker-compose setup in the `dev/` directory
```shell
# If you have not built the base image already
docker build -t arrow_integration_xenial_base -f dev/docker_common/Dockerfile.xenial.base .
dev/run_docker_compose.sh iwyu
```
### Linting
We require that you follow a certain coding style in the C++ code base.
You can check your code abides by that coding style by running:
make lint
You can also fix any formatting errors automatically:
make format
These commands require `clang-format-6.0` (and not any other version).
You may find the required packages at http://releases.llvm.org/download.html
or use the Debian/Ubuntu APT repositories on https://apt.llvm.org/. On macOS
with [Homebrew][1] you can get it via `brew install llvm@6`.
Depending on how you installed clang-format, the build system may not be able
to find it. You can provide an explicit path to your LLVM installation (or the
root path for the clang tools) with the environment variable
`$CLANG_TOOLS_PATH` or by passing `-DClangTools_PATH=$PATH_TO_CLANG_TOOLS` when
invoking CMake.
Additionally, all CMake files should go through an automatic formatter.
You'll need Python 3 and [cmake_format](https://github.com/cheshirekow/cmake_format)
installed. Then in the top-level directory run the `run-cmake-format.py`
script.
## Checking for ABI and API stability
To build ABI compliance reports, you need to install the two tools
`abi-dumper` and `abi-compliance-checker`.
Build Arrow C++ in Debug mode, alternatively you could use `-Og` which also
builds with the necessary symbols but includes a bit of code optimization.
Once the build has finished, you can generate ABI reports using:
```
abi-dumper -lver 9 debug/libarrow.so -o ABI-9.dump
```
The above version number is freely selectable. As we want to compare versions,
you should now `git checkout` the version you want to compare it to and re-run
the above command using a different version number. Once both reports are
generated, you can build a comparision report using
```
abi-compliance-checker -l libarrow -d1 ABI-PY-9.dump -d2 ABI-PY-10.dump
```
The report is then generated in `compat_reports/libarrow` as a HTML.
## Continuous Integration
Pull requests are run through travis-ci for continuous integration. You can avoid
build failures by running the following checks before submitting your pull request:
make unittest
make lint
# The next command may change your code. It is recommended you commit
# before running it.
make format # requires clang-format is installed
We run our CI builds with more compiler warnings enabled for the Clang
compiler. Please run CMake with
`-DBUILD_WARNING_LEVEL=CHECKIN`
to avoid failures due to compiler warnings.
Note that the clang-tidy target may take a while to run. You might consider
running clang-tidy separately on the files you have added/changed before
invoking the make target to reduce iteration time. Also, it might generate warnings
that aren't valid. To avoid these you can add a line comment `// NOLINT`. If
NOLINT doesn't suppress the warnings, you add the file in question to
the .clang-tidy-ignore file. This will allow `make check-clang-tidy` to pass in
travis-CI (but still surface the potential warnings in `make clang-tidy`). Ideally,
both of these options would be used rarely. Current known uses-cases when they are required:
* Parameterized tests in google test.
## CMake version requirements
We support CMake 3.2 and higher. Some features require a newer version of CMake:
* Building the benchmarks requires 3.6 or higher
* Building zstd from source requires 3.7 or higher
* Building Gandiva JNI bindings requires 3.11 or higher
[1]: https://brew.sh/
[2]: https://github.com/apache/arrow/blob/master/cpp/apidoc/Windows.md
[3]: https://google.github.io/styleguide/cppguide.html
[4]: https://github.com/include-what-you-use/include-what-you-use
[5]: https://github.com/apache/arrow/blob/master/cpp/thirdparty/README.md
[1]: https://github.com/apache/arrow/blob/master/docs/source/developers/cpp.rst
cpp/thirdparty/README.md
Vendored
+1 -88
View File
@@ -19,91 +19,4 @@
# Arrow C++ Thirdparty Dependencies
The version numbers for our third-party dependencies are listed in
`thirdparty/versions.txt`. This is used by the CMake build system as well as
the dependency downloader script (see below), which can be used to set up
offline builds.
## Configuring your own build toolchain
To set up your own specific build toolchain, here are the relevant environment
variables
* brotli: `BROTLI_HOME`, can be disabled with `-DARROW_WITH_BROTLI=off`
* Boost: `BOOST_ROOT`
* double-conversion: `DOUBLE_CONVERSION_HOME`
* Googletest: `GTEST_HOME` (only required to build the unit tests)
* gflags: `GFLAGS_HOME` (only required to build the unit tests)
* glog: `GLOG_HOME` (only required if `ARROW_USE_GLOG=ON`)
* Google Benchmark: `GBENCHMARK_HOME` (only required if building benchmarks)
* Flatbuffers: `FLATBUFFERS_HOME` (only required for -DARROW_IPC=on, which is
the default)
* Hadoop: `HADOOP_HOME` (only required for the HDFS I/O extensions)
* jemalloc: `JEMALLOC_HOME`
* lz4: `LZ4_HOME`, can be disabled with `-DARROW_WITH_LZ4=off`
* Apache ORC: `ORC_HOME`
* protobuf: `PROTOBUF_HOME`
* rapidjson: `RAPIDJSON_HOME`
* re2: `RE2_HOME` (only required to build Gandiva currently)
* snappy: `SNAPPY_HOME`, can be disabled with `-DARROW_WITH_SNAPPY=off`
* thrift: `THRIFT_HOME`
* zlib: `ZLIB_HOME`, can be disabled with `-DARROW_WITH_ZLIB=off`
* zstd: `ZSTD_HOME`, can be disabled with `-DARROW_WITH_ZSTD=off`
If you have all of your toolchain libraries installed at the same prefix, you
can use the environment variable `$ARROW_BUILD_TOOLCHAIN` to automatically set
all of these variables. Note that `ARROW_BUILD_TOOLCHAIN` will not set
`BOOST_ROOT`, so if you have custom Boost installation, you must set this
environment variable separately.
## Configuring for offline builds
If you do not use the above variables to direct the Arrow build system to
preinstalled dependencies, they will be built automatically by the build
system. The source archive for each dependency will be downloaded via the
internet, which can cause issues in environments with limited access to the
internet.
To enable offline builds, you can download the source artifacts yourself and
use environment variables of the form `ARROW_$LIBRARY_URL` to direct the build
system to read from a local file rather than accessing the internet.
To make this easier for you, we have prepared a script
`thirdparty/download_dependencies.sh` which will download the correct version
of each dependency to a directory of your choosing. It will print a list of
bash-style environment variable statements at the end to use for your build
script:
```shell
# Download tarballs into `$HOME/arrow-thirdparty-deps`
$ ./thirdparty/download_dependencies $HOME/arrow-thirdparty
# Environment variables for offline Arrow build
export ARROW_BOOST_URL=$HOME/arrow-thirdparty/boost-1.67.0.tar.gz
export ARROW_BROTLI_URL=$HOME/arrow-thirdparty/brotli-v0.6.0.tar.gz
export ARROW_DOUBLE_CONVERSION_URL=$HOME/arrow-thirdparty/double-conversion-v3.1.1.tar.gz
export ARROW_FLATBUFFERS_URL=$HOME/arrow-thirdparty/flatbuffers-02a7807dd8d26f5668ffbbec0360dc107bbfabd5.tar.gz
export ARROW_GBENCHMARK_URL=$HOME/arrow-thirdparty/gbenchmark-v1.4.1.tar.gz
export ARROW_GFLAGS_URL=$HOME/arrow-thirdparty/gflags-v2.2.0.tar.gz
export ARROW_GLOG_URL=$HOME/arrow-thirdparty/glog-v0.3.5.tar.gz
export ARROW_GRPC_URL=$HOME/arrow-thirdparty/grpc-v1.14.1.tar.gz
export ARROW_GTEST_URL=$HOME/arrow-thirdparty/gtest-1.8.0.tar.gz
export ARROW_LZ4_URL=$HOME/arrow-thirdparty/lz4-v1.7.5.tar.gz
export ARROW_ORC_URL=$HOME/arrow-thirdparty/orc-1.5.4.tar.gz
export ARROW_PROTOBUF_URL=$HOME/arrow-thirdparty/protobuf-v3.6.1.tar.gz
export ARROW_RAPIDJSON_URL=$HOME/arrow-thirdparty/rapidjson-v1.1.0.tar.gz
export ARROW_RE2_URL=$HOME/arrow-thirdparty/re2-2018-10-01.tar.gz
export ARROW_SNAPPY_URL=$HOME/arrow-thirdparty/snappy-1.1.3.tar.gz
export ARROW_THRIFT_URL=$HOME/arrow-thirdparty/thrift-0.11.0.tar.gz
export ARROW_ZLIB_URL=$HOME/arrow-thirdparty/zlib-1.2.8.tar.gz
export ARROW_ZSTD_URL=$HOME/arrow-thirdparty/zstd-v1.3.7.tar.gz
```
This can be automated by using inline source/eval:
```shell
$ source <(./thirdparty/download_dependencies $HOME/arrow-thirdparty-deps)
```
You can then invoke CMake to create the build directory and it will use the
declared environment variable pointing to downloaded archives instead of
downloading them (one for each build dir!).
See the "Dependency Management" section in the C++ Developer Documentation.
docs/README.md
+1 -1
View File
@@ -26,5 +26,5 @@ Instructions for building the documentation site are found in
[docs/source/building.rst][1]. The build depends on the API
documentation for some of the project subcomponents.
[1]: https://github.com/apache/arrow/blob/master/docs/source/building.rst
[1]: https://github.com/apache/arrow/blob/master/docs/source/developers/documentation.rst
[2]: https://github.com/apache/arrow/tree/master/docs/source/format
docs/source/developers/contributing.rst
+88
View File
@@ -0,0 +1,88 @@
.. Licensed to the Apache Software Foundation (ASF) under one
.. or more contributor license agreements. See the NOTICE file
.. distributed with this work for additional information
.. regarding copyright ownership. The ASF licenses this file
.. to you under the Apache License, Version 2.0 (the
.. "License"); you may not use this file except in compliance
.. with the License. You may obtain a copy of the License at
.. http://www.apache.org/licenses/LICENSE-2.0
.. Unless required by applicable law or agreed to in writing,
.. software distributed under the License is distributed on an
.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
.. KIND, either express or implied. See the License for the
.. specific language governing permissions and limitations
.. under the License.
.. _contributing:
***********************
Contribution Guidelines
***********************
There are many ways to contribute to Apache Arrow:
* Contributing code (we call them "patches")
* Writing documentation (another form of code, in a way)
* Participating in discussions on JIRA or the mailing list
* Helping users of the libraries
* Reporting bugs and asking questions
Mailing Lists and Issue Tracker
===============================
Projects in The Apache Software Foundation ("the ASF") use public, archived
mailing lists to create a public record of each project's development
activities and decision making process. As such, all contributors generally
must be subscribed to the dev@arrow.apache.org mailing list to participate in
the community.
Note that you must be subscribed to the mailing list in order to post to it. To
subscribe, send a blank email to dev-subscribe@arrow.apache.org.
We use the `ASF JIRA <https://issues.apache.org/jira>`_ to manage our
development "todo" list and to maintain changelogs for releases. You must
create an account and be added as a "Contributor" to Apache Arrow to be able to
assign yourself issues. Any project maintainer will be able to help you with
this one-time setup.
GitHub issues
-------------
We support GitHub issues as a lightweight way to ask questions and engage with
the Arrow developer community. We use JIRA for maintaining a queue of
development work and as the public record for work on the project. So, feel
free to open GitHub issues, but bugs and feature requests will eventually need
to end up in JIRA, either before or after completing a pull request. Don't be
surprised if you are immediately asked by a project maintainer to open a JIRA
issue.
How to contribute patches
=========================
We prefer to receive contributions in the form of GitHub pull requests. Please
send pull requests against the `github.com/apache/arrow
<https://github.com/apache/arrow>`_ repository following the procedure below.
If you are looking for some ideas on what to contribute, check out the JIRA
issues for the Apache Arrow project. Comment on the issue and/or contact
dev@arrow.apache.org with your questions and ideas.
If you’d like to report a bug but don’t have time to fix it, you can still post
it on JIRA, or email the mailing list dev@arrow.apache.org.
To contribute a patch:
* Break your work into small, single-purpose patches if possible. It’s much
harder to merge in a large change with a lot of disjoint features.
* Create a JIRA for your patch on the Arrow Project JIRA.
* Submit the patch as a GitHub pull request against the master branch. For a
tutorial, see the GitHub guides on forking a repo and sending a pull
request. Prefix your pull request name with the JIRA name (ex:
https://github.com/apache/arrow/pull/240).
* Make sure that your code passes the unit tests. You can find instructions how
to run the unit tests for each Arrow component in its respective README file.
* Add new unit tests for your code.
Thank you in advance for your contributions!
docs/source/developers/cpp.rst
+674
View File
@@ -0,0 +1,674 @@
.. Licensed to the Apache Software Foundation (ASF) under one
.. or more contributor license agreements. See the NOTICE file
.. distributed with this work for additional information
.. regarding copyright ownership. The ASF licenses this file
.. to you under the Apache License, Version 2.0 (the
.. "License"); you may not use this file except in compliance
.. with the License. You may obtain a copy of the License at
.. http://www.apache.org/licenses/LICENSE-2.0
.. Unless required by applicable law or agreed to in writing,
.. software distributed under the License is distributed on an
.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
.. KIND, either express or implied. See the License for the
.. specific language governing permissions and limitations
.. under the License.
.. _cpp-development:
***************
C++ Development
***************
System setup
============
Arrow uses CMake as a build configuration system. Currently, it supports
in-source and out-of-source builds with the latter one being preferred.
Building Arrow requires:
* A C++11-enabled compiler. On Linux, gcc 4.8 and higher should be sufficient.
* CMake 3.2 or higher
* Boost
* ``bison`` and ``flex`` (for building Apache Thrift from source only, a
Parquet dependency.)
Running the unit tests using ``ctest`` requires:
* python
On Ubuntu/Debian you can install the requirements with:
.. code-block:: shell
sudo apt-get install \
autoconf \
build-essential \
cmake \
libboost-dev \
libboost-filesystem-dev \
libboost-regex-dev \
libboost-system-dev \
python \
bison \
flex
On Alpine Linux:
.. code-block:: shell
apk add autoconf \
bash \
boost-dev \
cmake \
g++ \
gcc \
make
On macOS, you can use [Homebrew][1]:
.. code-block:: shell
git clone https://github.com/apache/arrow.git
cd arrow
brew update && brew bundle --file=c_glib/Brewfile
If you are developing on Windows, see the [Windows developer guide][2].
Building
========
Minimal release build:
.. code-block:: shell
git clone https://github.com/apache/arrow.git
cd arrow/cpp
mkdir release
cd release
cmake -DARROW_BUILD_TESTS=ON ..
make unittest
Minimal debug build:
.. code-block:: shell
git clone https://github.com/apache/arrow.git
cd arrow/cpp
mkdir debug
cd debug
cmake -DCMAKE_BUILD_TYPE=Debug -DARROW_BUILD_TESTS=ON ..
make unittest
If you do not need to build the test suite, you can omit the
``ARROW_BUILD_TESTS`` option (the default is not to build the unit tests).
Known issues
~~~~~~~~~~~~
On some Linux distributions, running the test suite might require setting an
explicit locale. If you see any locale-related errors, try setting the
environment variable (which requires the `locales` package or equivalent):
.. code-block:: shell
export LC_ALL="en_US.UTF-8"
Optional Components
===================
Linting and Continuous Integration
==================================
Our continuous integration builds in Travis CI and Appveyor run the unit test
suites on a variety of platforms and configuration, including using
``valgrind`` to check for memory leaks or bad memory accesses. In addition, the
codebase is subjected to a number of code style and code cleanliness checks.
In order to have a clean build, your modified git branch must pass the
following checks:
* C++ builds without compiler warnings with ``-DBUILD_WARNING_LEVEL=CHECKIN``
* C++ unit test suite with valgrind enabled, use ``-DARROW_TEST_MEMCHECK=ON``
when invoking CMake
* Passes cpplint checks, checked with ``make lint``
* Conforms to ``clang-format`` style, checked with ``make check-format``
* Passes C++/CLI header file checks, invoked with
``cpp/build-support/lint_cpp_cli.py cpp/src``
* CMake files pass style checks, can be fixed by running
``run-cmake-format.py`` from the root of the repository. This requires Python 3 and
[cmake_format](https://github.com/cheshirekow/cmake_format) (note: this currently
does not work on Windows)
In order to account for variations in the behavior of ``clang-format`` between
major versions of LLVM, we pin the version of ``clang-format`` used. Currently
this is ``clang-format-7`` from LLVM 7.0.x. If LLVM 7 is not available in your
system package manager, you may find the required packages at
http://releases.llvm.org/download.html or use the Debian/Ubuntu APT
repositories on https://apt.llvm.org/. On macOS with [Homebrew][1] you can get
it via ``brew install llvm@7``.
Depending on how you installed clang-format, the build system may not be able
to find it. You can provide an explicit path to your LLVM installation (or the
root path for the clang tools) with the environment variable
`$CLANG_TOOLS_PATH` or by passing ``-DClangTools_PATH=$PATH_TO_CLANG_TOOLS`` when
invoking CMake.
To make linting more reproducible for everyone, we provide a ``docker-compose``
target that is executable from the root of the repository:
.. code-block:: shell
docker-compose run lint
See :ref:`integration` for more information about the project's
``docker-compose`` configuration.
Benchmarking
============
Follow the directions for simple build except run cmake with the
``ARROW_BUILD_BENCHMARKS`` parameter set to ``ON``:
cmake -DARROW_BUILD_TESTS=ON -DARROW_BUILD_BENCHMARKS=ON ..
and instead of make unittest run either ``make; ctest`` to run both unit tests
and benchmarks or ``make benchmark`` to run only the benchmarks. Benchmark logs
will be placed in the build directory under ``build/benchmark-logs``.
You can also invoke a single benchmark executable directly:
.. code-block:: shell
./release/arrow-builder-benchmark
The build system uses ``CMAKE_BUILD_TYPE=release`` by default which enables
compiler optimizations. It is also recommended to disable CPU throttling or
such hardware features as "Turbo Boost" to obtain more consistent and
comparable. benchmark results
Modular Build Targets
=====================
Since there are several major parts of the C++ project, we have provided
modular CMake targets for building each library component, group of unit tests
and benchmarks, and their dependencies:
* ``make arrow`` for Arrow core libraries
* ``make parquet`` for Parquet libraries
* ``make gandiva`` for Gandiva (LLVM expression compiler) libraries
* ``make plasma`` for Plasma libraries, server
To build the unit tests or benchmarks, add ``-tests`` or ``-benchmarks`` to the
target name. So ``make arrow-tests`` will build the Arrow core unit
tests. Using the ``-all`` target, e.g. ``parquet-all``, will build everything.
If you wish to only build and install one or more project subcomponents, we
have provided the CMake option ``ARROW_OPTIONAL_INSTALL`` to only install
targets that have been built. For example, if you only wish to build the
Parquet libraries, its tests, and its dependencies, you can run:
.. code-block:: shell
cmake .. -DARROW_PARQUET=ON \
-DARROW_OPTIONAL_INSTALL=ON \
-DARROW_BUILD_TESTS=ON
make parquet
make install
If you omit an explicit target when invoking ``make``, all targets will be
built.
Apache Parquet Development
==========================
To build the C++ libraries for Apache Parquet, add the flag
``-DARROW_PARQUET=ON`` when invoking CMake. The Parquet libraries and unit tests
can be built with the ``parquet`` make target:
```shell
make parquet
```
Running ``ctest -L unittest`` will run all built C++ unit tests, while ``ctest -L
parquet`` will run only the Parquet unit tests. The unit tests depend on an
environment variable ``PARQUET_TEST_DATA`` that depends on a git submodule to the
repository https://github.com/apache/parquet-testing:
```shell
git submodule update --init
export PARQUET_TEST_DATA=$ARROW_ROOT/cpp/submodules/parquet-testing/data
```
Here ``$ARROW_ROOT`` is the absolute path to the Arrow codebase.
### Statically linking to Arrow on Windows
The Arrow headers on Windows static library builds (enabled by the CMake
option ``ARROW_BUILD_STATIC``) use the preprocessor macro ``ARROW_STATIC`` to
suppress dllimport/dllexport marking of symbols. Projects that statically link
against Arrow on Windows additionally need this definition. The Unix builds do
not use the macro.
### Testing with LLVM AddressSanitizer
To use AddressSanitizer (ASAN) to find bad memory accesses or leaks with LLVM,
pass ``-DARROW_USE_ASAN=ON`` when building. You must use clang to compile with
ASAN, and ``ARROW_USE_ASAN`` is mutually-exclusive with the valgrind option
``ARROW_TEST_MEMCHECK``.
### Building/Running fuzzers
Fuzzers can help finding unhandled exceptions and problems with untrusted input
that may lead to crashes, security issues and undefined behavior. They do this
by generating random input data and observing the behavior of the executed
code. To build the fuzzer code, LLVM is required (GCC-based compilers won't
work). You can build them using the following code:
cmake -DARROW_FUZZING=ON -DARROW_USE_ASAN=ON ..
``ARROW_FUZZING`` will enable building of fuzzer executables as well as enable the
addition of coverage helpers via ``ARROW_USE_COVERAGE``, so that the fuzzer can observe
the program execution.
It is also wise to enable some sanitizers like ``ARROW_USE_ASAN`` (see above), which
activates the address sanitizer. This way, we ensure that bad memory operations
provoked by the fuzzer will be found early. You may also enable other sanitizers as
well. Just keep in mind that some of them do not work together and some may result
in very long execution times, which will slow down the fuzzing procedure.
Now you can start one of the fuzzer, e.g.:
./debug/debug/ipc-fuzzing-test
This will try to find a malformed input that crashes the payload and will show the
stack trace as well as the input data. After a problem was found this way, it should
be reported and fixed. Usually, the fuzzing process cannot be continued until the
fix is applied, since the fuzzer usually converts to the problem again.
If you build fuzzers with ASAN, you need to set the ``ASAN_SYMBOLIZER_PATH``
environment variable to the absolute path of ``llvm-symbolizer``, which is a tool
that ships with LLVM.
```shell
export ASAN_SYMBOLIZER_PATH=$(type -p llvm-symbolizer)
```
Note that some fuzzer builds currently reject paths with a version qualifier
(like ``llvm-sanitizer-5.0``). To overcome this, set an appropriate symlink
(here, when using LLVM 5.0):
```shell
ln -sf /usr/bin/llvm-sanitizer-5.0 /usr/bin/llvm-sanitizer
```
There are some problems that may occur during the compilation process:
- libfuzzer was not distributed with your LLVM: ``ld: file not found: .../libLLVMFuzzer.a``
- your LLVM is too old: ``clang: error: unsupported argument 'fuzzer' to option 'fsanitize='``
### Building Python integration library (optional)
The optional ``arrow_python`` shared library can be built by passing
``-DARROW_PYTHON=on`` to CMake. This must be installed or in your library load
path to be able to build pyarrow, the Arrow Python bindings.
The Python library must be built against the same Python version for which you
are building pyarrow, e.g. Python 2.7 or Python 3.6. NumPy must also be
installed.
### Building CUDA extension library (optional)
The optional ``arrow_cuda`` shared library can be built by passing
``-DARROW_CUDA=on``. This requires a CUDA installation to build, and to use many
of the functions you must have a functioning CUDA-compatible GPU.
The CUDA toolchain used to build the library can be customized by using the
``$CUDA_HOME`` environment variable.
This library is still in Alpha stages, and subject to API changes without
deprecation warnings.
### Building Apache ORC integration (optional)
The optional arrow reader for the Apache ORC format (found in the
``arrow::adapters::orc`` namespace) can be built by passing ``-DARROW_ORC=on``.
This is currently not supported on windows. Note that this functionality is
still in Alpha stages, and subject to API changes without deprecation warnings.
### Building and developing Gandiva (optional)
The Gandiva library supports compiling and evaluating expressions on arrow
data. It uses LLVM for doing just-in-time compilation of the expressions.
In addition to the arrow dependencies, gandiva requires :
* On linux, gcc 4.9 or higher C++11-enabled compiler.
* LLVM
On Ubuntu/Debian you can install these requirements with:
```shell
sudo apt-add-repository -y "deb http://llvm.org/apt/trusty/ llvm-toolchain-trusty-7.0 main"
sudo apt-get update -qq
sudo apt-get install llvm-7.0-dev
```
On macOS, you can use [Homebrew][1]:
```shell
brew install llvm@7
```
The optional ``gandiva`` libraries and tests can be built by passing
``-DARROW_GANDIVA=on``.
```shell
cmake .. -DARROW_GANDIVA=ON -DARROW_BUILD_TESTS=ON
make
ctest -L gandiva
```
This library is still in Alpha stages, and subject to API changes without
deprecation warnings.
### Building and developing Flight (optional)
In addition to the Arrow dependencies, Flight requires:
* gRPC (>= 1.14, roughly)
* Protobuf (>= 3.6, earlier versions may work)
* c-ares (used by gRPC)
By default, Arrow will try to download and build these dependencies
when building Flight.
The optional ``flight`` libraries and tests can be built by passing
``-DARROW_FLIGHT=ON``.
```shell
cmake .. -DARROW_FLIGHT=ON -DARROW_BUILD_TESTS=ON
make
```
You can also use existing installations of the extra dependencies.
When building, set the environment variables ``GRPC_HOME`` and/or
``PROTOBUF_HOME`` and/or ``CARES_HOME``.
You may try using system libraries for gRPC and Protobuf, but these
are likely to be too old.
On Ubuntu/Debian, you can try:
```shell
sudo apt-get install libgrpc-dev libgrpc++-dev protobuf-compiler-grpc libc-ares-dev
```
Note that the version of gRPC in Ubuntu 18.10 is too old; you will
have to install gRPC from source. (Ubuntu 19.04/Debian Sid may work.)
On macOS, you can try [Homebrew][1]:
```shell
brew install grpc
```
You can also install gRPC from source. In this case, you must install
gRPC to generate the necessary files for CMake to find gRPC:
```shell
cmake -DgRPC_INSTALL=ON -DgRPC_BUILD_TESTS=OFF -DgRPC_PROTOBUF_PROVIDER=package -DgRPC_ZLIB_PROVIDER=package -DgRPC_CARES_PROVIDER=package -DgRPC_SSL_PROVIDER=package
```
You can then specify ``-DgRPC_DIR`` to ``cmake``.
API documentation with Doxygen
==============================
To generate the API documentation with Doxygen, run the following command from
the ``cpp/apidoc`` directory:
.. code-block:: shell
doxygen Doxyfile
This requires `Doxygen <https://www.doxygen.org>`_ to be installed.
## Development
This project follows [Google's C++ Style Guide][3] with minor exceptions:
* We relax the line length restriction to 90 characters.
* We use the NULLPTR macro defined in ``src/arrow/util/macros.h`` to
support building C++/CLI (ARROW-1134)
* We use doxygen style comments ("///") instead of line comments ("//")
in header files.
### Memory Pools
We provide a default memory pool with ``arrow::default_memory_pool()``. As a
matter of convenience, some of the array builder classes have constructors
which use the default pool without explicitly passing it. You can disable these
constructors in your application (so that you are accounting properly for all
memory allocations) by defining ``ARROW_NO_DEFAULT_MEMORY_POOL``.
### Header files
We use the ``.h`` extension for C++ header files. Any header file name not
containing ``internal`` is considered to be a public header, and will be
automatically installed by the build.
### Error Handling and Exceptions
For error handling, we use ``arrow::Status`` values instead of throwing C++
exceptions. Since the Arrow C++ libraries are intended to be useful as a
component in larger C++ projects, using ``Status`` objects can help with good
code hygiene by making explicit when a function is expected to be able to fail.
For expressing invariants and "cannot fail" errors, we use DCHECK macros
defined in ``arrow/util/logging.h``. These checks are disabled in release builds
and are intended to catch internal development errors, particularly when
refactoring. These macros are not to be included in any public header files.
Since we do not use exceptions, we avoid doing expensive work in object
constructors. Objects that are expensive to construct may often have private
constructors, with public static factory methods that return ``Status``.
There are a number of object constructors, like ``arrow::Schema`` and
``arrow::RecordBatch`` where larger STL container objects like ``std::vector`` may
be created. While it is possible for ``std::bad_alloc`` to be thrown in these
constructors, the circumstances where they would are somewhat esoteric, and it
is likely that an application would have encountered other more serious
problems prior to having ``std::bad_alloc`` thrown in a constructor.
### Extra debugging help
If you use the CMake option ``-DARROW_EXTRA_ERROR_CONTEXT=ON`` it will compile
the libraries with extra debugging information on error checks inside the
``RETURN_NOT_OK`` macro. In unit tests with ``ASSERT_OK``, this will yield error
outputs like:
```
../src/arrow/ipc/ipc-read-write-test.cc:609: Failure
Failed
NotImplemented: ../src/arrow/ipc/ipc-read-write-test.cc:574 code: writer->WriteRecordBatch(batch)
../src/arrow/ipc/writer.cc:778 code: CheckStarted()
../src/arrow/ipc/writer.cc:755 code: schema_writer.Write(&dictionaries_)
../src/arrow/ipc/writer.cc:730 code: WriteSchema()
../src/arrow/ipc/writer.cc:697 code: WriteSchemaMessage(schema_, dictionary_memo_, &schema_fb)
../src/arrow/ipc/metadata-internal.cc:651 code: SchemaToFlatbuffer(fbb, schema, dictionary_memo, &fb_schema)
../src/arrow/ipc/metadata-internal.cc:598 code: FieldToFlatbuffer(fbb, *schema.field(i), dictionary_memo, &offset)
../src/arrow/ipc/metadata-internal.cc:508 code: TypeToFlatbuffer(fbb, *field.type(), &children, &layout, &type_enum, dictionary_memo, &type_offset)
Unable to convert type: decimal(19, 4)
```
### Deprecations and API Changes
We use the compiler definition ``ARROW_NO_DEPRECATED_API`` to disable APIs that
have been deprecated. It is a good practice to compile third party applications
with this flag to proactively catch and account for API changes.
### Keeping includes clean with include-what-you-use
We have provided a ``build-support/iwyu/iwyu.sh`` convenience script for invoking
Google's [include-what-you-use][4] tool, also known as IWYU. This includes
various suppressions for more informative output. After building IWYU
(following instructions in the README), you can run it on all files by running:
```shell
CC="clang-4.0" CXX="clang++-4.0" cmake -DCMAKE_EXPORT_COMPILE_COMMANDS=ON ..
../build-support/iwyu/iwyu.sh all
```
This presumes that ``include-what-you-use`` and ``iwyu_tool.py`` are in your
``$PATH``. If you compiled IWYU using a different version of clang, then
substitute the version number above accordingly.
We have provided a Docker-based IWYU to make it easier to run these
checks. This can be run using the docker-compose setup in the ``dev/`` directory
```shell
# If you have not built the base image already
docker build -t arrow_integration_xenial_base -f dev/docker_common/Dockerfile.xenial.base .
dev/run_docker_compose.sh iwyu
```
## Checking for ABI and API stability
To build ABI compliance reports, you need to install the two tools
``abi-dumper`` and ``abi-compliance-checker``.
Build Arrow C++ in Debug mode, alternatively you could use ``-Og`` which also
builds with the necessary symbols but includes a bit of code optimization.
Once the build has finished, you can generate ABI reports using:
```
abi-dumper -lver 9 debug/libarrow.so -o ABI-9.dump
```
The above version number is freely selectable. As we want to compare versions,
you should now ``git checkout`` the version you want to compare it to and re-run
the above command using a different version number. Once both reports are
generated, you can build a comparision report using
```
abi-compliance-checker -l libarrow -d1 ABI-PY-9.dump -d2 ABI-PY-10.dump
```
The report is then generated in ``compat_reports/libarrow`` as a HTML.
## CMake version requirements
We support CMake 3.2 and higher. Some features require a newer version of CMake:
* Building the benchmarks requires 3.6 or higher
* Building zstd from source requires 3.7 or higher
* Building Gandiva JNI bindings requires 3.11 or higher
Dependency Management
=====================
The build system supports a number of third-party dependencies
* ``BOOST``: for cross-platform support
* ``BROTLI``: for data compression
* ``double-conversion``: for text-to-numeric conversions
* ``Snappy``: for data compression
* ``gflags``: for command line utilities (formerly Googleflags)
* ``glog``: for logging
* ``Thrift``: Apache Thrift, for data serialization
* ``Protobuf``: Google Protocol Buffers, for data serialization
* ``GTEST``: Googletest, for testing
* ``benchmark``: Google benchmark, for testing
* ``RapidJSON``: for data serialization
* ``Flatbuffers``: for data serialization
* ``ZLIB``: for data compression
* ``BZip2``: for data compression
* ``LZ4``: for data compression
* ``ZSTD``: for data compression
* ``RE2``: for regular expressions
* ``gRPC``: for remote procedure calls
* ``c-ares``: a dependency of gRPC
* ``LLVM``: a dependency of Gandiva
The CMake option ``ARROW_DEPENDENCY_SOURCE`` is a global option that instructs
the build system how to resolve each dependency. There are a few options:
* Building the dependency automatically from source
* Finding the dependency in system paths using CMake's built-in
``find_package`` function, or using ``pkg-config`` for packages that do not
have this feature
* Finding the dependency in a non-standard location, like a conda environment
or Homebrew.
Setting the Boost path
~~~~~~~~~~~~~~~~~~~~~~
Vendored / Bundled Packages
~~~~~~~~~~~~~~~~~~~~~~~~~~~
When using the ``BUNDLED`` method to build a dependency from source, the
version number from ``cpp/thirdparty/versions.txt`` is used. There is also a
dependency source downloader script (see below), which can be used to set up
offline builds.
Offline Builds
~~~~~~~~~~~~~~
If you do not use the above variables to direct the Arrow build system to
preinstalled dependencies, they will be built automatically by the build
system. The source archive for each dependency will be downloaded via the
internet, which can cause issues in environments with limited access to the
internet.
To enable offline builds, you can download the source artifacts yourself and
use environment variables of the form `ARROW_$LIBRARY_URL` to direct the build
system to read from a local file rather than accessing the internet.
To make this easier for you, we have prepared a script
`thirdparty/download_dependencies.sh` which will download the correct version
of each dependency to a directory of your choosing. It will print a list of
bash-style environment variable statements at the end to use for your build
script:
```shell
# Download tarballs into `$HOME/arrow-thirdparty-deps`
$ ./thirdparty/download_dependencies $HOME/arrow-thirdparty
# Environment variables for offline Arrow build
export ARROW_BOOST_URL=$HOME/arrow-thirdparty/boost-1.67.0.tar.gz
export ARROW_BROTLI_URL=$HOME/arrow-thirdparty/brotli-v0.6.0.tar.gz
export ARROW_DOUBLE_CONVERSION_URL=$HOME/arrow-thirdparty/double-conversion-v3.1.1.tar.gz
export ARROW_FLATBUFFERS_URL=$HOME/arrow-thirdparty/flatbuffers-02a7807dd8d26f5668ffbbec0360dc107bbfabd5.tar.gz
export ARROW_GBENCHMARK_URL=$HOME/arrow-thirdparty/gbenchmark-v1.4.1.tar.gz
export ARROW_GFLAGS_URL=$HOME/arrow-thirdparty/gflags-v2.2.0.tar.gz
export ARROW_GLOG_URL=$HOME/arrow-thirdparty/glog-v0.3.5.tar.gz
export ARROW_GRPC_URL=$HOME/arrow-thirdparty/grpc-v1.14.1.tar.gz
export ARROW_GTEST_URL=$HOME/arrow-thirdparty/gtest-1.8.0.tar.gz
export ARROW_LZ4_URL=$HOME/arrow-thirdparty/lz4-v1.7.5.tar.gz
export ARROW_ORC_URL=$HOME/arrow-thirdparty/orc-1.5.4.tar.gz
export ARROW_PROTOBUF_URL=$HOME/arrow-thirdparty/protobuf-v3.6.1.tar.gz
export ARROW_RAPIDJSON_URL=$HOME/arrow-thirdparty/rapidjson-v1.1.0.tar.gz
export ARROW_RE2_URL=$HOME/arrow-thirdparty/re2-2018-10-01.tar.gz
export ARROW_SNAPPY_URL=$HOME/arrow-thirdparty/snappy-1.1.3.tar.gz
export ARROW_THRIFT_URL=$HOME/arrow-thirdparty/thrift-0.11.0.tar.gz
export ARROW_ZLIB_URL=$HOME/arrow-thirdparty/zlib-1.2.8.tar.gz
export ARROW_ZSTD_URL=$HOME/arrow-thirdparty/zstd-v1.3.7.tar.gz
```
This can be automated by using inline source/eval:
```shell
$ source <(./thirdparty/download_dependencies $HOME/arrow-thirdparty-deps)
```
You can then invoke CMake to create the build directory and it will use the
declared environment variable pointing to downloaded archives instead of
downloading them (one for each build dir!).
[1]: https://brew.sh/
[2]: https://github.com/apache/arrow/blob/master/cpp/apidoc/Windows.md
[3]: https://google.github.io/styleguide/cppguide.html
[4]: https://github.com/include-what-you-use/include-what-you-use
[5]: https://github.com/apache/arrow/blob/master/cpp/thirdparty/README.md
docs/source/developers/documentation.rst
+1 -1
View File
@@ -65,7 +65,7 @@ These two steps are mandatory and must be executed in order.
This step requires the the pyarrow library is installed
in your python environment. One way to accomplish
this is to follow the build instructions at :ref:`development`
this is to follow the build instructions at :ref:`python-development`
and then run `python setup.py install` in arrow/python
(it is best to do this in a dedicated conda/virtual environment).
docs/source/developers/index.rst
+3 -3
View File
@@ -15,11 +15,11 @@
.. specific language governing permissions and limitations
.. under the License.
Developing Apache Arrow
=======================
.. toctree::
:maxdepth: 2
contributing
cpp
python
integration
documentation
docs/source/developers/integration.rst
+2
View File
@@ -15,6 +15,8 @@
.. specific language governing permissions and limitations
.. under the License.
.. _integration:
Integration Testing
===================
docs/source/python/development.rst → docs/source/developers/python.rst
+64 -84
View File
@@ -16,11 +16,11 @@
.. under the License.
.. currentmodule:: pyarrow
.. _development:
.. _python-development:
***********
Development
***********
******************
Python Development
******************
Developing on Linux and MacOS
=============================
@@ -31,25 +31,20 @@ System Requirements
On macOS, any modern XCode (6.4 or higher; the current version is 8.3.1) is
sufficient.
On Linux, for this guide, we recommend using gcc 4.8 or 4.9, or clang 3.7 or
On Linux, for this guide, we require a minimum of gcc 4.8, or clang 3.7 or
higher. You can check your version by running
.. code-block:: shell
$ gcc --version
On Ubuntu 16.04 and higher, you can obtain gcc 4.9 with:
If the system compiler is older than gcc 4.8, it can be set to a newer version
using the ``$CC`` and ``$CXX`` environment variables:
.. code-block:: shell
$ sudo apt-get install g++-4.9
Finally, set gcc 4.9 as the active compiler using:
.. code-block:: shell
export CC=gcc-4.9
export CXX=g++-4.9
export CC=gcc-4.8
export CXX=g++-4.8
Environment Setup and Build
---------------------------
@@ -74,7 +69,7 @@ Using Conda
~~~~~~~~~~~
Let's create a conda environment with all the C++ build and Python dependencies
from conda-forge:
from conda-forge, targeting development for Python 3.7:
On Linux and OSX:
@@ -84,7 +79,12 @@ On Linux and OSX:
--file arrow/ci/conda_env_unix.yml \
--file arrow/ci/conda_env_cpp.yml \
--file arrow/ci/conda_env_python.yml \
python=3.6
compilers \
python=3.7
As of January 2019, the `compilers` package is needed on many Linux distributions to use packages from conda-forge.
With this out of the way, you can now activate the conda environment
conda activate pyarrow-dev
@@ -95,11 +95,7 @@ about our build toolchain:
.. code-block:: shell
export ARROW_BUILD_TYPE=release
export ARROW_BUILD_TOOLCHAIN=$CONDA_PREFIX
export ARROW_HOME=$CONDA_PREFIX
export PARQUET_HOME=$CONDA_PREFIX
export BOOST_HOME=$CONDA_PREFIX
Using pip
~~~~~~~~~
@@ -161,10 +157,7 @@ about our build toolchain:
.. code-block:: shell
export ARROW_BUILD_TYPE=release
export ARROW_HOME=$(pwd)/dist
export PARQUET_HOME=$(pwd)/dist
export LD_LIBRARY_PATH=$(pwd)/dist/lib:$LD_LIBRARY_PATH
Build and test
@@ -177,26 +170,34 @@ Now build and install the Arrow C++ libraries:
mkdir arrow/cpp/build
pushd arrow/cpp/build
cmake -DCMAKE_BUILD_TYPE=$ARROW_BUILD_TYPE \
-DCMAKE_INSTALL_PREFIX=$ARROW_HOME \
cmake -DCMAKE_INSTALL_PREFIX=$ARROW_HOME \
-DCMAKE_INSTALL_LIBDIR=lib \
-DARROW_PARQUET=on \
-DARROW_PYTHON=on \
-DARROW_PLASMA=on \
-DARROW_BUILD_TESTS=OFF \
-DARROW_FLIGHT=ON \
-DARROW_GANDIVA=ON \
-DARROW_ORC=ON \
-DARROW_PARQUET=ON \
-DARROW_PYTHON=ON \
-DARROW_PLASMA=ON \
-DARROW_BUILD_TESTS=ON \
..
make -j4
make install
popd
If you don't want to build and install the Plasma in-memory object store,
you can omit the ``-DARROW_PLASMA=on`` flag.
Also, if multiple versions of Python are installed in your environment,
you may have to pass additional parameters to cmake so that
it can find the right executable, headers and libraries.
For example, specifying `-DPYTHON_EXECUTABLE=$VIRTUAL_ENV/bin/python`
(assuming that you're in virtualenv) enables cmake to choose
the python executable which you are using.
Many of these components are optional, and can be switched off by setting them
to ``OFF``:
* ``ARROW_FLIGHT``: RPC framework
* ``ARROW_GANDIVA``: LLVM-based expression compiler
* ``ARROW_ORC``: Support for Apache ORC file format
* ``ARROW_PARQUET``: Support for Apache Parquet file format
* ``ARROW_PLASMA``: Shared memory object store
If multiple versions of Python are installed in your environment, you may have
to pass additional parameters to cmake so that it can find the right
executable, headers and libraries. For example, specifying
`-DPYTHON_EXECUTABLE=$VIRTUAL_ENV/bin/python` (assuming that you're in
virtualenv) enables cmake to choose the python executable which you are using.
.. note::
@@ -210,11 +211,15 @@ Now, build pyarrow:
.. code-block:: shell
pushd arrow/python
python setup.py build_ext --build-type=$ARROW_BUILD_TYPE \
--with-parquet --with-plasma --inplace
export PYARROW_WITH_FLIGHT=1
export PYARROW_WITH_GANDIVA=1
export PYARROW_WITH_ORC=1
export PYARROW_WITH_PARQUET=1
python setup.py build_ext --build-type=$ARROW_BUILD_TYPE --inplace
popd
If you did not build with plasma, you can omit ``--with-plasma``.
If you did not build one of the optional components, set the corresponding
``PYARROW_WITH_$COMPONENT`` environment variable to 0.
You should be able to run the unit tests with:
@@ -242,56 +247,25 @@ libraries), one can set ``--bundle-arrow-cpp``:
pip install wheel # if not installed
python setup.py build_ext --build-type=$ARROW_BUILD_TYPE \
--with-parquet --with-plasma --bundle-arrow-cpp bdist_wheel
--bundle-arrow-cpp bdist_wheel
Again, if you did not build with plasma, you should omit ``--with-plasma``.
Building with CUDA support
~~~~~~~~~~~~~~~~~~~~~~~~~~
Building with optional ORC integration
--------------------------------------
To build Arrow with support for the `Apache ORC file format <https://orc.apache.org/>`_,
we recommend the following:
#. Install the ORC C++ libraries and tools using ``conda``:
.. code-block:: shell
conda install -c conda-forge orc
#. Set ``ORC_HOME`` and ``PROTOBUF_HOME`` to the location of the installed
Orc and protobuf C++ libraries, respectively (otherwise Arrow will try
to download source versions of those libraries and recompile them):
.. code-block:: shell
export ORC_HOME=$CONDA_PREFIX
export PROTOBUF_HOME=$CONDA_PREFIX
#. Add ``-DARROW_ORC=on`` to the CMake flags.
#. Add ``--with-orc`` to the ``setup.py`` flags.
Known issues
------------
If using packages provided by conda-forge (see "Using Conda" above)
together with a reasonably recent compiler, you may get "undefined symbol"
errors when importing pyarrow. In that case you'll need to force the C++
ABI version to the older version used by conda-forge binaries:
The :mod:`pyarrow.cuda` module offers support for using Arrow platform
components with Nvidia's CUDA-enabled GPU devices. To build with this support,
pass ``-DARROW_CUDA=ON`` when building the C++ libraries, and set the following
environment variable when building pyarrow:
.. code-block:: shell
export CXXFLAGS="-D_GLIBCXX_USE_CXX11_ABI=0"
export PYARROW_CXXFLAGS=$CXXFLAGS
Be sure to add ``-DCMAKE_CXX_FLAGS=$CXXFLAGS`` to the cmake invocations
when rebuilding.
export PYARROW_WITH_CUDA=1
Developing on Windows
=====================
First, we bootstrap a conda environment similar to the `C++ build instructions
<https://github.com/apache/arrow/blob/master/cpp/apidoc/Windows.md>`_. This
includes all the dependencies for Arrow and the Apache Parquet C++ libraries.
First, we bootstrap a conda environment similar to above, but skipping some of
the Linux/macOS-only packages:
First, starting from fresh clones of Apache Arrow:
@@ -313,20 +287,18 @@ Now, we build and install Arrow C++ libraries
mkdir cpp\build
cd cpp\build
set ARROW_BUILD_TOOLCHAIN=%CONDA_PREFIX%\Library
set ARROW_HOME=C:\thirdparty
cmake -G "Visual Studio 14 2015 Win64" ^
-DCMAKE_INSTALL_PREFIX=%ARROW_HOME% ^
-DCMAKE_BUILD_TYPE=Release ^
-DARROW_BUILD_TESTS=on ^
-DARROW_CXXFLAGS="/WX /MP" ^
-DARROW_GANDIVA=on ^
-DARROW_PARQUET=on ^
-DARROW_PYTHON=on ..
cmake --build . --target INSTALL --config Release
cd ..\..
If building with LLVM, also add `-DARROW_GANDIVA=ON`.
After that, we must put the install directory's bin path in our ``%PATH%``:
.. code-block:: shell
@@ -358,6 +330,14 @@ Getting ``python-test.exe`` to run is a bit tricky because your
Now ``python-test.exe`` or simply ``ctest`` (to run all tests) should work.
Windows Caveats
---------------
Some components are not supported yet on Windows:
* Flight RPC
* Plasma
Building the Documentation
==========================
docs/source/index.rst
+22 -8
View File
@@ -18,15 +18,25 @@
Apache Arrow
============
Apache Arrow is a cross-language development platform for in-memory data. It
specifies a standardized language-independent columnar memory format for flat
and hierarchical data, organized for efficient analytic operations on modern
hardware. It also provides computational libraries and zero-copy streaming
messaging and interprocess communication.
Apache Arrow is a development platform for in-memory analytics. It contains a
set of technologies that enable big data systems to process and move data
fast. It specifies a standardized language-independent columnar memory format
for flat and hierarchical data, organized for efficient analytic operations on
modern hardware.
The project is developing a multi-language collection of libraries for solving
systems problems related to in-memory analytical data processing. This includes
such topics as:
* Zero-copy shared memory and RPC-based data movement
* Reading and writing file formats (like CSV, Apache ORC, and Apache Parquet)
* In-memory analytics and query processing
.. _toc.columnar:
.. toctree::
:maxdepth: 1
:caption: Memory Format
:caption: Arrow Columnar Format
format/README
format/Guidelines
@@ -34,15 +44,19 @@ messaging and interprocess communication.
format/Metadata
format/IPC
.. _toc.usage:
.. toctree::
:maxdepth: 2
:caption: Languages
:caption: Library Usage
cpp/index
python/index
.. _toc.development:
.. toctree::
:maxdepth: 2
:caption: Developers
:caption: Development and Contributing
developers/index
docs/source/python/index.rst
-1
View File
@@ -47,6 +47,5 @@ files into Arrow structures.
cuda
extending
api
development
getting_involved
benchmarks
docs/source/python/install.rst
+1 -1
View File
@@ -63,4 +63,4 @@ need to install the `Visual C++ Redistributable for Visual Studio 2015
Installing from source
----------------------
See :ref:`development`.
See :ref:`python-development`.
docs/source/python/parquet.rst
+3 -3
View File
@@ -37,7 +37,7 @@ which includes a native, multithreaded C++ adapter to and from in-memory Arrow
data. PyArrow includes Python bindings to this code, which thus enables reading
and writing Parquet files with pandas as well.
Obtaining PyArrow with Parquet Support
Obtaining pyarrow with Parquet Support
--------------------------------------
If you installed ``pyarrow`` with pip or conda, it should be built with Parquet
@@ -49,8 +49,8 @@ support bundled:
If you are building ``pyarrow`` from source, you must use
``-DARROW_PARQUET=ON`` when compiling the C++ libraries and enable the Parquet
extensions when building ``pyarrow``. See the :ref:`Development <development>`
page for more details.
extensions when building ``pyarrow``. See the :ref:`Python Development
<python-development>` page for more details.
Reading and Writing Single Files
--------------------------------