### Rationale for this change
Require Numpy 2.0 to support newer features, allow newer numpy API/ABI
### What changes are included in this PR?
- Numpy 2.0+ requirement
- Pandas 2.2.2+ requirement (first wheel compatible with numpy 2.0)
- Pyodide 0.27.1 usage (numpy 2.0 api)
### Are there any user-facing changes?
Yes, in contrast to pyarrow 25, after merging this PR pyarrow 26 can't be used with Numpy 1.x or pandas 2.0-2.2.1
**This PR includes breaking changes to public APIs.**
This PR was created using GPT-5.6-Sol-xhigh, every line read & reviewed by me.
* GitHub Issue: #48473
Lead-authored-by: Adam Lippai <adam@rigo.sk>
Co-authored-by: Ádám Lippai <adam@rigo.sk>
Co-authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Add [Python 3.15](https://docs.python.org/3.15/whatsnew/3.15.html) to the pyarrow nightly builds, so that issues can be detected early in the development cycles of both cpython & pyarrow.
Note:
- ~~CFFI and~~ Pandas doesn't have nightly 3.15 wheels, and are accordingly excluded.
- Numpy shipped 3.15rc1 in 2.5.2 ~~doesn't ship a 3.15 wheel, but is needed to build. It is building from source.~~
- [3.15 Schedule](https://peps.python.org/pep-0790/) This PR targets 3.15.0rc1. rc2 is expected 2026-09-01 and final 2026-10-01.
### What changes are included in this PR?
- Adding cp315/cp315t to the build matrix
- Excluding ~~cffi/~~ pandas for 3.15, as the wheels aren't yet available.
### 2026-08-13 edit
- Bump libcst to 1.9.0
- Update manylinux image to 2026.08.08-1 (for 3.15rc1)
- Add -rc to PYTHON_IMAGE_TAG
- Disable pandas tests if Pandas isn't installed:
### Are these changes tested?
- Yes, tested on a local Linux build using the `ninja-release-python` preset (not maximal)
### Are there any user-facing changes?
No functional changes, other than allowing the user to use Python 3.15.
* GitHub Issue: #48172
### Progress Notes (2026-08-15)
Each platform uses different sources for Python builds, each with different release schedules and paths:
- [manylinux](https://github.com/pypa/manylinux) is prebuilt with latest Python builds in `/opt/python`, so using the latest manylinux build is sufficient. 3.15 wasn't available until beta 1.
- musllinux: Uses Alpine system python built from https://github.com/docker-library/python/pull/1123. Might need to bump ALPINE_LINUX version in [arrow/dev/tasks/python-wheels/github.linux.yml](https://github.com/apache/arrow/blob/e611f48081ec927e18e5bcb72fbd071c15c76b08/dev/tasks/python-wheels/github.linux.yml#L35).
- musllinux free-threaded: uses [astral's builds](https://github.com/astral-sh/python-build-standalone). Edit `python-free-threaded-wheel-musllinux-test-imports.dockerfile` + `-test-unittests.dockerfile`: set the python_patch_version, and set build_date in `compose.yaml`.
- macos: Downloads official build from python.org, see `ci/scripts/install_python.sh`. For early builds, needed to disable `--only-binary` since at least one dependency needed to be built (no 3.15 wheels available yet)... only macos uses --only-binary
- Windows: installs builds using `choco`, see `python-wheel-windows-test-vs2022.dockerfile` and `python-free-threaded-wheel-windows-test-vs2022.dockerfile`
For 3.15.0rc1 builds, some care was needed because certain paths needed 3.15.0 and others needed the full 3.15.0rc1.
Lead-authored-by: Paul Timmins <paul@iqmo.com>
Co-authored-by: paul@iqmo.com <paul@iqmo.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
`scikit-build-core` 1.0 added `[tool.scikit-build.sdist.force-include]` in order to force the inclusion of files out of the source tree.
This allows us to remove our workaround with a custom build-backend to solve adding our license files from the main Arrow repository.
### What changes are included in this PR?
Add configuration to force-include `LICENSE.txt` and `NOTICE.txt` and remove custom build-backend.
### Are these changes tested?
Yes, CI and local
### Are there any user-facing changes?
No
* GitHub Issue: #50493
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Python 3.10 is End of Support on October 2026. We can drop support on PyArrow 26.0.0.
See: https://endoflife.date/python
### What changes are included in this PR?
Remove support for 3.10. Update minimum required Python version and update some CI jobs to use the minimum Python supported version.
Some CI jobs using default Python 3.10 (Ubuntu 20.04) have been bumped.
### Are these changes tested?
They will be tested via CI and Archery
### Are there any user-facing changes?
Yes in terms of not supporting older versions of Python. This will be a breaking change for some users but not for the API itself
* GitHub Issue: #50410
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Since 24.0, pyarrow ships annotations and `py.typed` marker while the type stubs are still incomplete. This makes type checkers trust the partial stubs and report false errors in downstream code (e.g. Module has no attribute "all" for `pyarrow.compute`). Some type checkers also consume bundled `.pyi` files even without `py.typed`, so the stubs need to be withheld from wheels for now. See #49831.
### What changes are included in this PR?
This temporarily omits both `pyarrow/py.typed` and the bundled `pyarrow-stubs` / `.pyi` files from built wheels until the stubs are complete. Wheel-content validation now asserts that neither `py.typed` nor `.pyi` files are present, and wheel build scripts no longer request stub docstring injection while stubs are not installed.
### Are these changes tested?
Wheel-content validation has been updated to check the intended absence of `py.typed` and `.pyi` files.
### Are there any user-facing changes?
Type checkers no longer pick up pyarrow's incomplete stubs from wheels.
* GitHub Issue: #49831
Authored-by: Rok Mihevc <rok@mihevc.org>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Move our PyArrow build backend from setuptools and a custom setup.py to scikit-build-core which is just build backend for CMake related projects.
### What changes are included in this PR?
Move from setuptools to scikit-build-core and remove PyArrow setup.py. Update some of the build requirements and minor fixes.
A custom build backend has been also been created in order to wrap scikit-build-core in order to fix problems on License files for monorepos.
pyproject.toml metadata validation expects license files to exist before exercising the build backend that's why we create symlinks. Our thin build backend will just make those symlinks hard-links in order for license and notice files to contain the contents and be added as part of the sdist.
Remove flags that are not used anymore (were only part of setup.py) and documented and validated how the same flags have to be used now.
### Are these changes tested?
Yes all Python CI tests, wheels and sdist are successful.
### Are there any user-facing changes?
Yes, users building PyArrow will now require the new build dependencies to exercise the build and depending on the flags used they might require to use the new documented way of using those flags.
* GitHub Issue: #36411
Lead-authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Co-authored-by: Rok Mihevc <rok@mihevc.org>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
This is the first in series of PRs adding type annotations to pyarrow and resolving #32609.
### What changes are included in this PR?
This PR establishes infrastructure for type checking:
- Adds CI workflow for running mypy, pyright, and ty type checkers on linux, macos and windows
- Configures type checkers to validate stub files (excluding source files for now)
- Adds PEP 561 `py.typed` marker to enable type checking
- Updates wheel build scripts to include stub files in distributions
- Creates initial minimal stub directory structure
- Updates developer documentation with type checking workflow
### Are these changes tested?
No. This is mostly a CI change.
### Are there any user-facing changes?
This does not add any actual annotations (only `py.typed` marker) so user should not be affected.
* GitHub Issue: #32609
* GitHub Issue: #49102
Lead-authored-by: Rok Mihevc <rok@mihevc.org>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Co-authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Rok Mihevc <rok@mihevc.org>
### Rationale for this change
When building locally, I get many errors along the lines of
```
Please ensure the files specified are contained by the root
of the Python package (normally marked by `pyproject.toml`).
By 2026-Mar-20, you need to update your project and remove deprecated calls
or your builds will no longer be supported.
See https://packaging.python.org/en/latest/specifications/glob-patterns/ for details.
```
<img width="958" height="755" alt="terminal demo" src="https://github.com/user-attachments/assets/67f0e261-c4d2-403c-b004-688dfaaccda6" />
### What changes are included in this PR?
- Make the licence [SPDX compliant](https://spdx.org/licenses)
- Remove the licence classifier
- Move the licence files from `setup.cfg` to `pyproject.toml`
- Fix the [disallowed glob patterns](https://packaging.python.org/en/latest/specifications/glob-patterns) via symlinks
- Bumped the minimum version of `setuptools` due to macOS CI failures (don't know why this happened, caching maybe?)
I appreciate the symlink change might prove controversial. See discussions in https://github.com/apache/arrow/issues/45867, fixes https://github.com/apache/arrow/issues/45867.
### Are these changes tested?
When I rebuild locally, I get no errors any more.
### Are there any user-facing changes?
Yes. The minimum required version of `setuptools` is now `77`. However, this is available on `>=3.9` so won't affect anyone really.
* GitHub Issue: #45867
Authored-by: Patrick J. Roddy <patrickjamesroddy@gmail.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Python 3.9 is End of Support on 31st October 2025. We can drop support for pyarrow 22.0.0.
See: https://endoflife.date/python
### What changes are included in this PR?
Remove support for 3.9. Update minimum required Python version and update some CI jobs to use the minimum Python supported version
### Are these changes tested?
They will be tested via CI and Archery
### Are there any user-facing changes?
Yes in terms of not supporting older versions of Python. This will be a breaking change for some users but not for the API itself
* GitHub Issue: #47443
Lead-authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Upgrading Cython to version 3.1
### What changes are included in this PR?
Replace all mentions of earlier versions with 3.1
### Are these changes tested?
Nah but I'll run CI
### Are there any user-facing changes?
Only if they're using Cython < 3.1
* GitHub Issue: #47370
Authored-by: Nic Crane <thisisnic@gmail.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
Modern packaging requires non-user facing dependencies to be placed in a `dependency-groups` as per [PEP 735](https://peps.python.org/pep-0735.)
### What changes are included in this PR?
Changes the `project.optional-dependencies` to `dependency-groups`.
### Are these changes tested?
Not locally yet. I've tried searching for use of the word `test`. Obviously this comes up a lot in the code base, so testing in CI here.
### Are there any user-facing changes?
Not really, no. The only difference is people would have to do `pip install --group test pyarrow` rather than `pip install pyarrow[test]`.
* GitHub Issue: #47137
Authored-by: Patrick J. Roddy <patrickjamesroddy@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
We want to migrate to pre-commit from `archery lint`.
### What changes are included in this PR?
Use pre-commit for numpydoc but this doesn't support Cython files.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
* GitHub Issue: #46546
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
We do not require to support Cython 2 anymore.
### What changes are included in this PR?
Bump required version of Cython
### Are these changes tested?
Via CI
### Are there any user-facing changes?
Yes, pyarrow cannot be compiled with Cython2 anymore.
* GitHub Issue: #45237
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
I noticed a few typos in the documentation and a single typo in the pyproject file. This change corrects these typos.
### What changes are included in this PR?
The following spelling corrections have been applied:
fo -> to
scenerios -> scenarios
wich -> which
suiteable -> suitable
avaiable -> available (this is in the pyproject.toml file regarding a NumPy comment for older Python versions)
### Are these changes tested?
No executable code is modified with these changes.
### Are there any user-facing changes?
Only the documentation is being updated.
Authored-by: Tyler White <50381805+IndexSeek@users.noreply.github.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
## WIP
Creating the PR to validate if / what are the CI failures
### Rationale for this change
Once the issue has been merged:
- https://github.com/apache/arrow/issues/25118
We do not require numpy as a pyarrow dependency.
### What changes are included in this PR?
Remove numpy as a required dependency
### Are these changes tested?
Via CI
### Are there any user-facing changes?
Yes, pyarrow won't install numpy as a required dependency.
* GitHub Issue: #43846
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Python 3.8 is End of Support on 31st October 2023. We can drop support for pyarrow 18.0.0.
See: https://endoflife.date/python
### What changes are included in this PR?
Remove support for 3.8. Update minimum required Python version and update some CI jobs to use the minimum Python supported version
### Are these changes tested?
They will be tested via Archery
### Are there any user-facing changes?
Yes in terms of not supporting older versions of Python. This will be a breaking change for some users but not for the API itself
* GitHub Issue: #43518
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
### Rationale for this change
Currently we include everything when building wheels, see:
```
$ pip install pyarrow
Collecting pyarrow
Downloading pyarrow-17.0.0-cp310-cp310-manylinux_2_28_x86_64.whl (39.9 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 39.9/39.9 MB 33.8 MB/s eta 0:00:00
Collecting numpy>=1.16.6
Using cached numpy-2.0.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (19.3 MB)
Installing collected packages: numpy, pyarrow
Successfully installed numpy-2.0.0 pyarrow-17.0.0
(test-env) $ ls test-env/lib/python3.10/site-packages/
benchmarks/ distutils-precedence.pth numpy-2.0.0.dist-info/ pip-22.0.2.dist-info/ pyarrow-17.0.0.dist-info/ setuptools-59.6.0.dist-info/
cmake_modules/ examples/ numpy.libs/ pkg_resources/ scripts/
_distutils_hack/ numpy/ pip/ pyarrow/ setuptools/
```
### What changes are included in this PR?
Use `include` as seen here: https://setuptools.pypa.io/en/latest/userguide/package_discovery.html#finding-simple-packages
### Are these changes tested?
Will check via the build wheel on CI
### Are there any user-facing changes?
No and yes :)
We will remove unnecessary files
* GitHub Issue: #43299
Lead-authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Co-authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
### Rationale for this change
Like #43519 mentionies, now that the first `rc` is out, it's probably time to add CI coverage for Python 3.13 (and also start building wheels).
### What changes are included in this PR?
I'm fairly new to the build/CI processes of the project, but I tried to follow the same template as #37901. I'll follow up afterwards with adding CI coverage for the free-threaded build as well.
* GitHub Issue: #43519
Lead-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
### Rationale for this change
Newer (possibly unreleased) version of the Python build tools check that the classifiers are valid and the build failed due to this typo.
### What changes are included in this PR?
Fix the spelling of a word
### Are these changes tested?
Build will fail without these changes, has no run-time effect.
### Are there any user-facing changes?
no
Authored-by: Thomas A Caswell <tcaswell@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
Ensure we can build pyarrow against numpy 2.0 nightly (update pyproject.toml to allow this), and test this by building our nightly wheels with numpy nightly. This also ensures that other projects that use our nightly wheels to test together with numpy nightly can do that (numpy 2.0 changes the ABI, so to run with numpy 2.0, your package needs to be built with numpy 2.x; currently pyarrow installed with our nightly wheel will fail to import when also numpy nightly is installed).
See the parent issue https://github.com/apache/arrow/issues/39532 for details, and https://numpy.org/devdocs/dev/depending_on_numpy.html#numpy-2-0-specific-advice for a direct link to the NumPy guidelines on updating build dependencies for NumPy 2.0.
* Closes: #39555
Lead-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Co-authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
### Rationale for this change
Enable Cython 3.
### What changes are included in this PR?
* Unpin Cython version in builds.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes, pyarrow will build with cython 3 by default.
* Closes: #37742
Authored-by: Dane Pitkin <dane@voltrondata.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
### What changes are included in this PR?
Pin the version of `setuptools_scm` to `<8.0.0`.
* Closes: #37803
Authored-by: AlenkaF <frim.alenka@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
Building with cython 0.29.11 fails but works with latest 0.29.34.
* Closes: #34933
Authored-by: Jacob Wujciak-Jens <jacob@wujciak.de>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Our `setup.py` script now uses the `find_namespace_packages` function which is not provided by old setuptools versions.
Fixes source release verification script on Ubuntu 18.04.
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Antoine Pitrou <pitrou@free.fr>
`oldest-supported-numpy` is a meta-package which defines the numpy version pins across platforms and python versions, this should make the python package maintenance easier and more future-proof.
cc @jorisvandenbossche
Closes#11606 from kszucs/oldest-numpy
Authored-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
Adds support and testing for Python 3.9. I am looking for review as this change may have touched too many things, but I'm also looking to get the CI to test all the different environments.
H/T: @kou, the documentation and #5685 for helping me get this off the ground.
Closes#8386 from terencehonles/arrow-10224
Authored-by: Terence D. Honles <terence@honles.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
Closes#6048 from xhochy/ARROW-7407 and squashes the following commits:
54465249e <Uwe L. Korn> ARROW-7407: Declare NumPy a PEP517 build dependency
Authored-by: Uwe L. Korn <uwe.korn@quantco.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
Author: Korn, Uwe <Uwe.Korn@blue-yonder.com>
Closes#3590 from xhochy/ARROW-4024 and squashes the following commits:
3fa306ec7 <Korn, Uwe> ARROW-4024: Raise minimal Cython version to 0.29
Raúl Cumplido