mirrors/arrow
0
0 Fork 0
mirror of https://github.com/apache/arrow.git synced 2026-10-29 00:56:31 +08:00
Code Activity

Finish initial documentation revamp for 0.13, stopping here

Browse Source
This commit is contained in:
Wes McKinney committed 2019-03-17 14:52:20 -05:00
1 parent 84dd680a24
commit 8d1f33e199
7 files changed
+670 -716
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

cpp/apidoc/Windows.md
-291
View File
@@ -1,291 +0,0 @@
<!---
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
# Developing Arrow C++ on Windows
## System setup, conda, and conda-forge
Since some of the Arrow developers work in the Python ecosystem, we are
investing time in maintaining the thirdparty build dependencies for Arrow and
related C++ libraries using the conda package manager. Others are free to add
other development instructions for Windows here.
### conda and package toolchain
[Miniconda][1] is a minimal Python distribution including the conda package
manager. To get started, download and install a 64-bit distribution.
We recommend using packages from [conda-forge][2].
Launch cmd.exe and run following commands:
```shell
conda config --add channels conda-forge
```
Now, you can bootstrap a build environment (call from the root directory of the
Arrow codebase):
```shell
conda create -n arrow-dev --file=ci\conda_env_cpp.yml
```
> **Note:** Make sure to get the `conda-forge` build of `gflags` as the
> naming of the library differs from that in the `defaults` channel.
Activate just created conda environment with pre-installed packages from
previous step:
```shell
activate arrow-dev
```
We are using the [cmake][4] tool to support Windows builds.
To allow cmake to pick up 3rd party dependencies, you should set
`ARROW_BUILD_TOOLCHAIN` environment variable to contain `Library` folder
path of new created on previous step `arrow-dev` conda environment.
To set `ARROW_BUILD_TOOLCHAIN` environment variable visible only for current terminal
session you can run following. `%CONDA_PREFIX` is set by conda to the current environment
root by the `activate` script.
```shell
set ARROW_BUILD_TOOLCHAIN=%CONDA_PREFIX%\Library
```
To validate value of `ARROW_BUILD_TOOLCHAIN` environment variable you can run following terminal command:
```shell
echo %ARROW_BUILD_TOOLCHAIN%
```
As alternative to `ARROW_BUILD_TOOLCHAIN`, it's possible to configure path
to each 3rd party dependency separately by setting appropriate environment
variable:
`FLATBUFFERS_HOME` variable with path to `flatbuffers` installation
`RAPIDJSON_HOME` variable with path to `rapidjson` installation
`GFLAGS_HOME` variable with path to `gflags` installation
`SNAPPY_HOME` variable with path to `snappy` installation
`ZLIB_HOME` variable with path to `zlib` installation
`BROTLI_HOME` variable with path to `brotli` installation
`LZ4_HOME` variable with path to `lz4` installation
`ZSTD_HOME` variable with path to `zstd` installation
### Customize static libraries names lookup of 3rd party dependencies
If you decided to use pre-built 3rd party dependencies libs, it's possible to
configure Arrow's cmake build script to search for customized names of 3rd
party static libs.
`brotli`. Set `BROTLI_HOME` environment variable. Pass
`-DBROTLI_MSVC_STATIC_LIB_SUFFIX=%BROTLI_SUFFIX%` to link with
brotli*%BROTLI_SUFFIX%.lib. For brotli versions <= 0.6.0 installed from
conda-forge this must be set to `_static`, otherwise the default of `-static`
is used.
`snappy`. Set `SNAPPY_HOME` environment variable. Pass
`-DSNAPPY_MSVC_STATIC_LIB_SUFFIX=%SNAPPY_SUFFIX%` to link with
snappy%SNAPPY_SUFFIX%.lib.
`lz4`. Set `LZ4_HOME` environment variable. Pass
`-LZ4_MSVC_STATIC_LIB_SUFFIX=%LZ4_SUFFIX%` to link with
lz4%LZ4_SUFFIX%.lib.
`zstd`. Set `ZSTD_HOME` environment variable. Pass
`-ZSTD_MSVC_STATIC_LIB_SUFFIX=%ZSTD_SUFFIX%` to link with
zstd%ZSTD_SUFFIX%.lib.
### Visual Studio
Microsoft provides the free Visual Studio Community edition. When doing
development, you must launch the developer command prompt using:
#### Visual Studio 2015
```
"C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\vcvarsall.bat" amd64
```
#### Visual Studio 2017
```
"C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\Tools\VsDevCmd.bat" -arch=amd64
```
It's easiest to configure a console emulator like [cmder][3] to automatically
launch this when starting a new development console.
## Building with Ninja and clcache
We recommend the [Ninja](https://ninja-build.org/) build system for better
build parallelization, and the optional
[clcache](https://github.com/frerich/clcache/) compiler cache which keeps
track of past compilations to avoid running them over and over again
(in a way similar to the Unix-specific "ccache").
Activate your conda build environment to first install those utilities:
```shell
activate arrow-dev
conda install -c conda-forge ninja
pip install git+https://github.com/frerich/clcache.git
```
Change working directory in cmd.exe to the root directory of Arrow and
do an out of source build by generating Ninja files:
```shell
cd cpp
mkdir build
cd build
cmake -G "Ninja" -DCMAKE_BUILD_TYPE=Release ..
cmake --build . --config Release
```
## Building with NMake
Activate your conda build environment:
```shell
activate arrow-dev
```
Change working directory in cmd.exe to the root directory of Arrow and
do an out of source build using `nmake`:
```shell
cd cpp
mkdir build
cd build
cmake -G "NMake Makefiles" -DCMAKE_BUILD_TYPE=Release ..
cmake --build . --config Release
nmake
```
When using conda, only release builds are currently supported.
## Building using Visual Studio (MSVC) Solution Files
Activate your conda build environment:
```shell
activate arrow-dev
```
Change working directory in cmd.exe to the root directory of Arrow and
do an out of source build by generating a MSVC solution:
```shell
cd cpp
mkdir build
cd build
cmake -G "Visual Studio 14 2015 Win64" -DCMAKE_BUILD_TYPE=Release ..
cmake --build . --config Release
```
## Debug build
To build Debug version of Arrow you should have pre-installed a Debug version
of boost libs.
It's recommended to configure cmake build with the following variables for
Debug build:
`-DARROW_BOOST_USE_SHARED=OFF` - enables static linking with boost debug libs and
simplifies run-time loading of 3rd parties. (Recommended)
`-DBOOST_ROOT` - sets the root directory of boost libs. (Optional)
`-DBOOST_LIBRARYDIR` - sets the directory with boost lib files. (Optional)
Command line to build Arrow in Debug might look as following:
```shell
cd cpp
mkdir build
cd build
cmake -G "Visual Studio 14 2015 Win64" ^
-DARROW_BOOST_USE_SHARED=OFF ^
-DCMAKE_BUILD_TYPE=Debug ^
-DBOOST_ROOT=C:/local/boost_1_63_0 ^
-DBOOST_LIBRARYDIR=C:/local/boost_1_63_0/lib64-msvc-14.0 ^
..
cmake --build . --config Debug
```
To get the latest build instructions, you can reference [ci/appveyor-built.bat][5], which is used by automated Appveyor builds.
## Replicating Appveyor Builds
For people more familiar with linux development but need to replicate a failing appveyor build, here are some rough notes from
replicating the Static_Crt_Builds (make unittest will probably still fail but many unit tests can be made with there individual
make targets).
1. Microsoft offers trial VMs for [Windows with Microsoft Visual Studio][6]. Download and install a version.
2. Run the VM and install CMake and Miniconda or Anaconda (these instructions assume Anaconda).
3. Download boost from [6] and install it (run from command prompt opened by "Developer Command Prompt for MSVC 2017"):
```shell
cd $EXTRACT_BOOST_DIRECTORY
.\bootstrap.bat
@rem This is for static libraries needed for static_crt_build in appvyor
.\b2 link=static -with-filesystem -with-regex -with-system install
@rem this should put libraries and headers in c:\Boost
```
4. Activate ananaconda/miniconda:
```
@rem this might differ for miniconda
C:\Users\User\Anaconda3\Scripts\activate
```
5. Clone and change directories to the arrow source code (you might need to install git).
6. Setup environment variables:
```shell
@rem Change the build type based on which appveyor job you want.
SET JOB=Static_Crt_Build
SET GENERATOR=Ninja
SET APPVEYOR_BUILD_WORKER_IMAGE=Visual Studio 2017
SET USE_CLCACHE=false
SET ARROW_BUILD_GANDIVA=OFF
SET ARROW_LLVM_VERSION=7.0.*
SET PYTHON=3.6
SET ARCH=64
SET ARROW_BUILD_TOOLCHAIN=%CONDA_PREFIX%\Library
SET PATH=C:\Users\User\Anaconda3;C:\Users\User\Anaconda3\Scripts;C:\Users\User\Anaconda3\Library\bin;%PATH%
SET BOOST_LIBRARYDIR=C:\Boost\lib
SET BOOST_ROOT=C:\Boost
```
7. Run appveyor scripts:
```shell
.\ci\appveyor-install.bat
@rem this might fail but at this point most unit tests should be buildable by there individual targets
@rem see next line for example.
.\ci\appveyor-build.bat
cmake --build . --config Release --target arrow-compute-hash-test
```
[1]: https://conda.io/miniconda.html
[2]: https://conda-forge.github.io/
[3]: http://cmder.net/
[4]: https://cmake.org/
[5]: https://github.com/apache/arrow/blob/master/ci/appveyor-cpp-build.bat
[6]: https://developer.microsoft.com/en-us/windows/downloads/virtual-machines
cpp/thirdparty/README.md
Vendored
+4 -1
View File
@@ -19,4 +19,7 @@
# Arrow C++ Thirdparty Dependencies
See the "Dependency Management" section in the C++ Developer Documentation.
See the "Build Dependency Management" section in the [C++ Developer
Documentation][1].
[1]: https://github.com/apache/arrow/blob/master/docs/source/developers/cpp.rst
docs/source/developers/cpp.rst
+584 -370
View File
File diff suppressed because it is too large. Load diff
docs/source/developers/python.rst
+74 -9
View File
@@ -22,7 +22,77 @@
Python Development
******************
Developing on Linux and MacOS
This page provides general Python development guidelines and source build
instructions for all platforms.
Coding Style
============
We follow a similar PEP8-like coding style to the `pandas project
<https://github.com/pandas-dev/pandas>`_.
The code must pass ``flake8`` (available from pip or conda) or it will fail the
build. Check for style errors before submitting your pull request with:
.. code-block:: shell
flake8 .
flake8 --config=.flake8.cython .
Unit Testing
============
We are using `pytest <https://docs.pytest.org/en/latest/>`_ to develop our unit
test suite. After building the project (see below) you can run its unit tests
like so:
.. code-block:: shell
pytest pyarrow
Package requirements to run the unit tests are found in
``requirements-test.txt`` and can be installed if needed with ``pip -r
requirements-test.txt``.
The project has a number of custom command line options for its test
suite. Some tests are disabled by default, for example. To see all the options,
run
.. code-block:: shell
pytest pyarrow --help
and look for the "custom options" section.
Test Groups
-----------
We have many tests that are grouped together using pytest marks. Some of these
are disabled by default. To enable a test group, pass ``--$GROUP_NAME``,
e.g. ``--parquet``. To disable a test group, prepend ``disable``, so
``--disable-parquet`` for example. To run **only** the unit tests for a
particular group, prepend ``only-`` instead, for example ``--only-parquet``.
The test groups currently include:
* ``gandiva``: tests for Gandiva expression compiler (uses LLVM)
* ``hdfs``: tests that use libhdfs or libhdfs3 to access the Hadoop filesystem
* ``hypothesis``: tests that use the ``hypothesis`` module for generating
random test cases. Note that ``--hypothesis`` doesn't work due to a quirk
with pytest, so you have to pass ``--enable-hypothesis``
* ``large_memory``: Test requiring a large amount of system RAM
* ``orc``: Apache ORC tests
* ``parquet``: Apache Parquet tests
* ``plasma``: Plasma Object Store tests
* ``s3``: Tests for Amazon S3
* ``tensorflow``: Tests that involve TensorFlow
Benchmarking
------------
For running the benchmarks, see :ref:`python-benchmarks`.
Building on Linux and MacOS
=============================
System Requirements
@@ -88,7 +158,7 @@ With this out of the way, you can now activate the conda environment
conda activate pyarrow-dev
For Windows, see the `Developing on Windows`_ section below.
For Windows, see the `Building on Windows`_ section below.
We need to set some environment variables to let Arrow's build system know
about our build toolchain:
@@ -261,8 +331,8 @@ environment variable when building pyarrow:
export PYARROW_WITH_CUDA=1
Developing on Windows
=====================
Building on Windows
===================
First, we bootstrap a conda environment similar to above, but skipping some of
the Linux/macOS-only packages:
@@ -337,8 +407,3 @@ Some components are not supported yet on Windows:
* Flight RPC
* Plasma
Building the Documentation
==========================
See :ref:`building-docs` for instructions to build the HTML documentation.
docs/source/index.rst
+1 -1
View File
@@ -48,7 +48,7 @@ such topics as:
.. toctree::
:maxdepth: 2
:caption: Library Usage
:caption: Arrow Libraries
cpp/index
python/index
docs/source/python/benchmarks.rst
+2
View File
@@ -15,6 +15,8 @@
.. specific language governing permissions and limitations
.. under the License.
.. _python-benchmarks:
Benchmarks
==========
python/README.md
+5 -44
View File
@@ -1,4 +1,4 @@
<!---
y<!---
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
@@ -40,52 +40,13 @@ pip install pyarrow
## Development
### Coding Style
We follow a similar PEP8-like coding style to the [pandas project][3].
The code must pass `flake8` (available from pip or conda) or it will fail the
build. Check for style errors before submitting your pull request with:
```
flake8 .
flake8 --config=.flake8.cython .
```
### Building from Source
See the [Development][2] page in the documentation.
### Running the unit tests
We are using [pytest][4] to develop our unit test suite. After building the
project using `setup.py build_ext --inplace`, you can run its unit tests like
so:
```bash
pytest pyarrow
```
The project has a number of custom command line options for its test
suite. Some tests are disabled by default, for example. To see all the options,
run
```bash
pytest pyarrow --help
```
and look for the "custom options" section.
For running the benchmarks, see the [Sphinx documentation][5].
See [Python Development][2] in the documentation subproject.
### Building the documentation
```bash
pip install -r ../docs/requirements.txt
python setup.py build_sphinx -s ../docs/source
```
See [documentation build instructions][1] in the documentation subproject.
[2]: https://github.com/apache/arrow/blob/master/docs/source/python/development.rst
[1]: https://github.com/apache/arrow/blob/master/docs/source/developers/documentation.rst
[2]: https://github.com/apache/arrow/blob/master/docs/source/developers/python.rst
[3]: https://github.com/pandas-dev/pandas
[4]: https://docs.pytest.org/en/latest/
[5]: https://arrow.apache.org/docs/latest/python/benchmarks.html