arrow/docs/source/developers/python/building.rst

.. Licensed to the Apache Software Foundation (ASF) under one
.. or more contributor license agreements.  See the NOTICE file
.. distributed with this work for additional information
.. regarding copyright ownership.  The ASF licenses this file
.. to you under the Apache License, Version 2.0 (the
.. "License"); you may not use this file except in compliance
.. with the License.  You may obtain a copy of the License at

..   http://www.apache.org/licenses/LICENSE-2.0

.. Unless required by applicable law or agreed to in writing,
.. software distributed under the License is distributed on an
.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
.. KIND, either express or implied.  See the License for the
.. specific language governing permissions and limitations
.. under the License.

.. currentmodule:: pyarrow
.. highlight:: console
.. _build_pyarrow:

================
Building PyArrow
================


This page provides source build instructions for PyArrow for all platforms.

System Requirements
===================

.. tab-set::
   :sync-group: language

   .. tab-item:: Linux and macOS
      :sync: linux-macos

      On macOS, any modern XCode or Xcode Command Line Tools (``xcode-select --install``)
      are sufficient.

      On Linux, for this guide, we require a minimum of gcc or clang 9.
      You can check your version by running

      .. code-block::

         $ gcc --version

      If the system compiler is older than gcc 9, it can be set to a newer version
      using the ``$CC`` and ``$CXX`` environment variables:

      .. code-block::

         $ export CC=gcc-9
         $ export CXX=g++-9


   .. tab-item:: Windows
      :sync: wins

      Building on Windows requires one of the following compilers to be
      installed:

      - `Build Tools for Visual Studio 2022 <https://aka.ms/vs/17/release/vs_BuildTools.exe>`_ or
      - Visual Studio 2022

      During the setup of Build Tools, ensure at least one Windows SDK
      is selected.

Environment setup
=================

First, start from a fresh clone of Apache Arrow:

.. code-block::

   $ git clone https://github.com/apache/arrow.git

There are two supported ways to set up the build environment for PyArrow: using
**Conda** to manage the dependencies or using **pip** with manual dependency
management.

Both methods are shown bellow for Linux and macOS. For Windows, only the
Conda-based setup is currently documented, skipping some of the
Linux/macOS-only packages.

Note that in case you are not using conda on a Windows platform, Arrow C++
libraries need to be bundled with ``pyarrow``. For extra information see the
Windows tab under the :ref:`pyarrow_build_section` section.

.. tab-set::
   :sync-group: language

   .. tab-item:: Linux and macOS using conda
      :sync: linux-macos

      Pull in the test data and setup the environment variables:

      .. code-block::

         $ pushd arrow
         $ git submodule update --init
         $ export PARQUET_TEST_DATA="${PWD}/cpp/submodules/parquet-testing/data"
         $ export ARROW_TEST_DATA="${PWD}/testing/data"
         $ popd

      The `conda <https://conda.io/>`_ package manager allows installing build-time
      dependencies for Arrow C++ and PyArrow as pre-built binaries, which can make
      Arrow development easier and faster.

      Let's create a conda environment with all the C++ build and Python dependencies
      from conda-forge, targeting development for Python 3.13:

      .. code-block::

         $ conda create -y -n pyarrow-dev -c conda-forge \
               --file arrow/ci/conda_env_unix.txt \
               --file arrow/ci/conda_env_cpp.txt \
               --file arrow/ci/conda_env_python.txt \
               --file arrow/ci/conda_env_gandiva.txt \
               compilers \
               python=3.13 \
               pandas

      As of January 2019, the ``compilers`` package is needed on many Linux
      distributions to use packages from conda-forge.

      With this out of the way, you can now activate the conda environment

      .. code-block::

         $ conda activate pyarrow-dev


      We need to set some environment variables to let Arrow's build system know
      about our build toolchain:

      .. code-block::

         $ export ARROW_HOME=$CONDA_PREFIX


   .. tab-item:: Linux and macOS using pip

      .. warning::

         If you installed Python using the Anaconda distribution or `Miniconda
         <https://conda.io/miniconda.html>`_, you cannot currently use a
         pip-based virtual environment. Please follow the conda-based development
         instructions instead.

      Pull in the test data and setup the environment variables:

      .. code-block::

         $ pushd arrow
         $ git submodule update --init
         $ export PARQUET_TEST_DATA="${PWD}/cpp/submodules/parquet-testing/data"
         $ export ARROW_TEST_DATA="${PWD}/testing/data"
         $ popd

      **Using system and bundled dependencies**

      If not using conda, you must arrange for your system to provide the required
      build tools and dependencies.  Note that if some dependencies are absent,
      the Arrow C++ build chain may still be able to download and compile them
      on the fly, but this will take a longer time than with pre-installed binaries.

      On macOS, use Homebrew to install all dependencies required for
      building Arrow C++:

      .. code-block::

         $ brew update && brew bundle --file=arrow/cpp/Brewfile

      See :ref:`here <cpp-build-dependency-management>` for a list of dependencies you
      may need.

      On Debian/Ubuntu, you need the following minimal set of dependencies:

      .. code-block::

         $ sudo apt-get install build-essential ninja-build cmake python3-dev

      Now, let's create a Python virtual environment with all Python dependencies
      in the same folder as the repositories, and a target installation folder:

      .. code-block::

         $ python3 -m venv pyarrow-dev
         $ source ./pyarrow-dev/bin/activate
         $ pip install -r arrow/python/requirements-build.txt

         $ # This is the folder where we will install the Arrow libraries during
         $ # development
         $ mkdir dist

      If your CMake version is too old on Linux, you could get a newer one via
      ``pip install cmake``.

      We need to set some environment variables to let Arrow's build system know
      about our build toolchain:

      .. code-block::

         $ export ARROW_HOME=$(pwd)/dist
         $ export LD_LIBRARY_PATH=$(pwd)/dist/lib:$LD_LIBRARY_PATH
         $ export CMAKE_PREFIX_PATH=$ARROW_HOME:$CMAKE_PREFIX_PATH

   .. tab-item:: Windows
      :sync: wins

      Let's create a conda environment with all the C++ build and Python dependencies
      from conda-forge, targeting development for Python 3.13:

      .. code-block::

         $ conda create -y -n pyarrow-dev -c conda-forge ^
               --file arrow\ci\conda_env_cpp.txt ^
               --file arrow\ci\conda_env_python.txt ^
               --file arrow\ci\conda_env_gandiva.txt ^
               python=3.13
         $ conda activate pyarrow-dev

      Now, we can build and install Arrow C++ libraries.

      We set the path of the installation directory of the Arrow C++
      libraries as ``ARROW_HOME``. When using a conda environment,
      Arrow C++ is installed in the environment directory, which path
      is saved in the `CONDA_PREFIX <https://docs.conda.io/projects/conda-build/en/latest/user-guide/environment-variables.html#environment-variables-that-affect-the-build-process>`_
      environment variable.

      .. code-block::

         $ set ARROW_HOME=%CONDA_PREFIX%\Library

Build
=====

First we need to configure, build and install the Arrow C++ libraries.
Then we can build PyArrow.

Build C++
---------

.. tab-set::
   :sync-group: language

   .. tab-item:: Linux and macOS
      :sync: linux-macos

      Now build the Arrow C++ libraries and install them into the directory we
      created above (stored in ``$ARROW_HOME``):

      .. code-block::

         $ cmake -S arrow/cpp -B arrow/cpp/build \
               -DCMAKE_INSTALL_PREFIX=$ARROW_HOME \
               --preset ninja-release-python
         $ cmake --build arrow/cpp/build --target install

      **About presets**

      ``ninja-release-python`` is not the only preset available - if you would like a
      build with more features like CUDA, Flight and Gandiva support you may opt for
      the ``ninja-release-python-maximal`` preset. If you wanted less features, (i.e.
      removing ORC and dataset support) you could opt for
      ``ninja-release-python-minimal``. Changing the word ``release`` to ``debug``
      with any of the aforementioned presets will generate a debug build of Arrow.

      **Individual components**

      The presets are provided as a convenience, but you may instead opt to
      specify the individual components:

      .. code-block::

         $ cmake -S arrow/cpp -B arrow/cpp/build \
               -DCMAKE_INSTALL_PREFIX=$ARROW_HOME \
               -DCMAKE_BUILD_TYPE=Debug \
               -DARROW_BUILD_TESTS=ON \
               -DARROW_COMPUTE=ON \
               -DARROW_CSV=ON \
               -DARROW_DATASET=ON \
               -DARROW_FILESYSTEM=ON \
               -DARROW_HDFS=ON \
               -DARROW_JSON=ON \
               -DARROW_PARQUET=ON \
               -DARROW_WITH_BROTLI=ON \
               -DARROW_WITH_BZ2=ON \
               -DARROW_WITH_LZ4=ON \
               -DARROW_WITH_SNAPPY=ON \
               -DARROW_WITH_ZLIB=ON \
               -DARROW_WITH_ZSTD=ON \
               -DPARQUET_REQUIRE_ENCRYPTION=ON
         $ cmake --build arrow/cpp/build --target install -j4

      If multiple versions of Python are installed in your environment, you may have
      to pass additional parameters to CMake so that it can find the right
      executable, headers and libraries.  For example, specifying
      ``-DPython3_EXECUTABLE=<path/to/bin/python>`` lets CMake choose the
      Python executable which you are using.

      .. note::

         On Linux systems with support for building on multiple architectures,
         ``make`` may install libraries in the ``lib64`` directory by default. For
         this reason we recommend passing ``-DCMAKE_INSTALL_LIBDIR=lib`` because the
         Python build scripts assume the library directory is ``lib``

      .. note::

         If you have conda installed but are not using it to manage dependencies,
         and you have trouble building the C++ library, you may need to set
         ``-DARROW_DEPENDENCY_SOURCE=AUTO`` or some other value (described
         :ref:`here <cpp-build-dependency-management>`)
         to explicitly tell CMake not to use conda.

   .. tab-item:: Windows
      :sync: wins

      There are presets provided as a convenience for building C++ (see Linux and macOS
      tab). Here we will instead specify the individual components:

      .. code-block::

         $ mkdir arrow\cpp\build
         $ pushd arrow\cpp\build
         $ cmake -G "Ninja" ^
               -DCMAKE_INSTALL_PREFIX=%ARROW_HOME% ^
               -DCMAKE_UNITY_BUILD=ON ^
               -DARROW_COMPUTE=ON ^
               -DARROW_CSV=ON ^
               -DARROW_CXXFLAGS="/WX /MP" ^
               -DARROW_DATASET=ON ^
               -DARROW_FILESYSTEM=ON ^
               -DARROW_HDFS=ON ^
               -DARROW_JSON=ON ^
               -DARROW_PARQUET=ON ^
               -DARROW_WITH_LZ4=ON ^
               -DARROW_WITH_SNAPPY=ON ^
               -DARROW_WITH_ZLIB=ON ^
               -DARROW_WITH_ZSTD=ON ^
               ..
         $ cmake --build . --target install --config Release
         $ popd

Optional build components
^^^^^^^^^^^^^^^^^^^^^^^^^

There are several optional components that can be enabled or disabled by setting
specific flags to ``ON`` or ``OFF``, respectively. See the list of
:ref:`python-dev-components` below.

You may choose between different kinds of C++ build types:

* ``-DCMAKE_BUILD_TYPE=Release`` (the default) produces a build with optimizations
  enabled and debugging information disabled;
* ``-DCMAKE_BUILD_TYPE=Debug`` produces a build with optimizations
  disabled and debugging information enabled;
* ``-DCMAKE_BUILD_TYPE=RelWithDebInfo`` produces a build with both optimizations
  and debugging information enabled.

.. seealso::
   :ref:`Building Arrow C++ <cpp-building-building>`.

   For any other C++ build challenges, see :ref:`cpp-development`.

In case you may need to rebuild the C++ part due to errors in the process it is
advisable to delete the build folder, see :ref:`stale_artifacts`.
If the build has passed successfully and you need to rebuild due to latest pull
from git main, then this step is not needed.

.. _pyarrow_build_section:

Build PyArrow
-------------

If you did build one of the optional components in C++, the equivalent components
will be enabled by default for building pyarrow. This default can be overridden
by setting the corresponding ``PYARROW_WITH_$COMPONENT`` environment variable
to 0 or 1, see :ref:`python-dev-components` below.

To build PyArrow run:

.. tab-set::
   :sync-group: language

   .. tab-item:: Linux and macOS
      :sync: linux-macos

      .. code-block::

         $ pushd arrow/python
         $ pip install --no-build-isolation --editable . -vv
         $ popd

   .. tab-item:: Windows
      :sync: wins

      .. code-block::

         $ pushd arrow\python
         $ pip install --no-build-isolation --editable . -vv
         $ popd

      .. note::

         If you are using Conda with Python 3.9 or earlier, you must
         set ``CONDA_DLL_SEARCH_MODIFICATION_ENABLE=1``.

      .. note::

         With the above instructions the Arrow C++ libraries are not bundled with
         the Python extension. This is recommended for development as it allows the
         C++ libraries to be re-built separately.

         If you are using the conda package manager then conda will ensure the Arrow C++
         libraries are found. **In case you are NOT using conda** then you have to:

         * add the path of installed DLL libraries to ``PATH`` every time before
           importing ``pyarrow``, or
         * bundle the Arrow C++ libraries with ``pyarrow``.

         **Bundle Arrow C++ and PyArrow**

         If you want to bundle the Arrow C++ libraries with ``pyarrow``, set the
         ``PYARROW_BUNDLE_ARROW_CPP`` environment variable before building ``pyarrow``:

         .. code-block::

            $ set PYARROW_BUNDLE_ARROW_CPP=ON
            $ pip install --no-build-isolation --editable . -vv

         Note that bundled Arrow C++ libraries will not be automatically
         updated when rebuilding Arrow C++.

This creates an *editable install*, meaning changes to the Python source code
will be reflected immediately without needing to reinstall the package.
The ``--no-build-isolation`` flag ensures that the build uses your current
environment's dependencies instead of creating an isolated one.

To set the number of threads used to compile PyArrow's C++/Cython components,
set the ``CMAKE_BUILD_PARALLEL_LEVEL`` environment variable.

If you build PyArrow but then make changes to the Arrow C++ or PyArrow code,
you can end up with stale build artifacts. This can lead to
unexpected behavior or errors. To avoid this, you can clean the build
artifacts before rebuilding. See :ref:`python-dev-env-variables`.

By default, PyArrow will be built in release mode even if Arrow C++ has been
built in debug mode. To create a debug build of PyArrow, run
``pip install --no-build-isolation -vv -C cmake.build-type=Debug .``.
A ``relwithdebinfo`` build can be created similarly.

Self-Contained Wheel
^^^^^^^^^^^^^^^^^^^^

If you're preparing a PyArrow wheel for distribution (e.g., for PyPI), you'll
need to build a self-contained wheel (including the Arrow and Parquet C++
libraries). This ensures that all necessary native libraries are bundled inside
the wheel, so users can install it without needing to have Arrow or Parquet
installed separately on their system.

To do this, set the ``PYARROW_BUNDLE_ARROW_CPP`` environment variable before building ``pyarrow``:

.. code-block::

   $ export PYARROW_BUNDLE_ARROW_CPP=ON
   $ pip install build wheel  # if not installed
   $ python -m build --sdist --wheel . --no-isolation

This option is typically only needed for releases or distribution scenarios,
not for local development.

.. _stale_artifacts:

Deleting stale build artifacts
------------------------------

When there have been changes to the structure of the Arrow C++ library or PyArrow,
a thorough cleaning is recommended as a first attempt to fixing build errors.

.. note::

   It is not necessarily intuitive from the error itself that the problem is due to stale artifacts.
   Example of a build error from stale artifacts is
   ``Unknown CMake command "arrow_keep_backward_compatibility"``.

To delete stale Arrow C++ build artifacts:

.. code-block::

   $ rm -rf arrow/cpp/build

To delete stale PyArrow build artifacts:

.. code-block::

   $ git clean -Xfd python

If using a Conda environment, there are some build artifacts that get installed in
``$ARROW_HOME`` (aka ``$CONDA_PREFIX``). For example, ``$ARROW_HOME/lib/cmake/Arrow*``,
``$ARROW_HOME/include/arrow``, ``$ARROW_HOME/lib/libarrow*``, etc.

These files can be manually deleted. If unsure which files to erase, one approach
is to recreate the Conda environment.

Either delete the current one, and start fresh:

.. code-block::

   $ conda deactivate
   $ conda remove -n pyarrow-dev

Or, less destructively, create a different environment with a different name.

Docker examples
---------------

If you are having difficulty building the Python library from source, take a
look at the `python/examples/minimal_build <https://github.com/apache/arrow/tree/main/python/examples/minimal_build>`_
directory which illustrates a complete build and test from source both with
the conda- and pip-based build methods.

Test
====

Now you are ready to install test dependencies and run :ref:`python-unit-testing`, as
described in development section.

.. _python-dev-env-variables:

Relevant environment variables and build options
================================================

List of relevant environment variables that can be used to build
PyArrow are:

.. list-table::
   :widths: 20 20 20
   :header-rows: 1

   * - PyArrow environment variable
     - Description
     - Default value
   * - ``CMAKE_BUILD_PARALLEL_LEVEL``
     - Number of processes used to compile PyArrow’s C++/Cython components
     - ``''``
   * - ``CMAKE_GENERATOR``
     - Example: ``'Visual Studio 17 2022 Win64'``
     - ``''``
   * - ``PYARROW_CXXFLAGS``
     - Extra C++ compiler flags
     - ``''``
   * - ``PYARROW_GENERATE_COVERAGE``
     - Setting ``Xlinetrace`` flag to true for the Cython compiler
     - ``false``
   * - ``PYARROW_BUNDLE_ARROW_CPP``
     - Bundle the Arrow C++ libraries
     - ``0`` (``OFF``)
   * - ``PYARROW_BUNDLE_CYTHON_CPP``
     - Bundle the C++ files generated by Cython
     - ``0`` (``OFF``)

To set the build type (e.g. ``Debug``, ``Release``, ``RelWithDebInfo``), pass
``-C cmake.build-type=Debug`` to ``pip install`` or to ``python -m build``.

For extra CMake arguments you can use the ``-C cmake.args=``
argument when building PyArrow. For example, to build a version of PyArrow
with ``ARROW_SIMD_LEVEL=NONE``, you can run
``pip install --no-build-isolation -vv -C cmake.args="-DARROW_SIMD_LEVEL=NONE" .``.

On PyArrow 24.0.0 we migrated our Python build backend from setuptools to
scikit-build-core, which is a CMake-based build system. Previous versions used
``PYARROW_BUILD_TYPE`` and ``PYARROW_CMAKE_OPTIONS`` environment variables
to customize the CMake invocation. This is no longer supported.
Instead, use the ``-C cmake.build-type=<build_type>`` and ``-C cmake.args=-D<OPTION>=<VALUE>`` option as described above.

To enable verbose output from the build tool, pass
``-C build.verbose=true`` to ``pip install`` or to ``python -m build``.

.. _python-dev-components:

Relevant components
===================

The components being disabled or enabled when building PyArrow is by default
based on how Arrow C++ is build (i.e. it follows the ``ARROW_$COMPONENT`` flags).
However, the ``PYARROW_WITH_$COMPONENT`` environment variables can still be used
to override this when building PyArrow (e.g. to disable components, or to enforce
certain components to be built):

.. list-table::
   :widths: 30 30
   :header-rows: 1

   * - Arrow flags/options
     - Corresponding environment variables for PyArrow
   * - ``ARROW_GCS``
     - ``PYARROW_WITH_GCS``
   * - ``ARROW_S3``
     - ``PYARROW_WITH_S3``
   * - ``ARROW_AZURE``
     - ``PYARROW_WITH_AZURE``
   * - ``ARROW_HDFS``
     - ``PYARROW_WITH_HDFS``
   * - ``ARROW_CUDA``
     - ``PYARROW_WITH_CUDA``
   * - ``ARROW_SUBSTRAIT``
     - ``PYARROW_WITH_SUBSTRAIT``
   * - ``ARROW_FLIGHT``
     - ``PYARROW_WITH_FLIGHT``
   * - ``ARROW_ACERO``
     - ``PYARROW_WITH_ACERO``
   * - ``ARROW_DATASET``
     - ``PYARROW_WITH_DATASET``
   * - ``ARROW_PARQUET``
     - ``PYARROW_WITH_PARQUET``
   * - ``PARQUET_REQUIRE_ENCRYPTION``
     - ``PYARROW_WITH_PARQUET_ENCRYPTION``
   * - ``ARROW_ORC``
     - ``PYARROW_WITH_ORC``
   * - ``ARROW_GANDIVA``
     - ``PYARROW_WITH_GANDIVA`` (deprecated since version 24.0.0)

Installing Nightly Packages
===========================

.. warning::
    These packages are not official releases. Use them at your own risk.

PyArrow has nightly wheels for testing purposes hosted at
`scientific-python-nightly-wheels
<https://anaconda.org/scientific-python-nightly-wheels/pyarrow>`_.

These may be suitable for downstream libraries in their continuous integration
setup to maintain compatibility with the upcoming PyArrow features,
deprecations, and/or feature removals.

To install the most recent nightly version of PyArrow, run:

.. code-block:: bash

    pip install \
      -i https://pypi.anaconda.org/scientific-python-nightly-wheels/simple \
      pyarrow