### Rationale for this change
https://github.com/apache/arrow/pull/48191/ added 3.15.0rc1 to build matrix. At the time, Pandas was not yet shipping wheels for 3.15.0 and was disabled.
Now that [Pandas has 3.15.0rc1 wheels](https://github.com/pandas-dev/pandas/issues/66740), this PR makes a small change to add the nightly index and target the correct Pandas dev0 suffix.
### What changes are included in this PR?
- Reenables Pandas tests
- Adds nightly find-links and version for 3.15 via marker in the requirements-test.txt and requirements-wheel-test.txt files
### Are these changes tested?
Tested against manylinux cp314, cp315 and cp315t.
### Are there any user-facing changes?
No
* GitHub Issue: #50943
Authored-by: paul@iqmo.com <paul@iqmo.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Add [Python 3.15](https://docs.python.org/3.15/whatsnew/3.15.html) to the pyarrow nightly builds, so that issues can be detected early in the development cycles of both cpython & pyarrow.
Note:
- ~~CFFI and~~ Pandas doesn't have nightly 3.15 wheels, and are accordingly excluded.
- Numpy shipped 3.15rc1 in 2.5.2 ~~doesn't ship a 3.15 wheel, but is needed to build. It is building from source.~~
- [3.15 Schedule](https://peps.python.org/pep-0790/) This PR targets 3.15.0rc1. rc2 is expected 2026-09-01 and final 2026-10-01.
### What changes are included in this PR?
- Adding cp315/cp315t to the build matrix
- Excluding ~~cffi/~~ pandas for 3.15, as the wheels aren't yet available.
### 2026-08-13 edit
- Bump libcst to 1.9.0
- Update manylinux image to 2026.08.08-1 (for 3.15rc1)
- Add -rc to PYTHON_IMAGE_TAG
- Disable pandas tests if Pandas isn't installed:
### Are these changes tested?
- Yes, tested on a local Linux build using the `ninja-release-python` preset (not maximal)
### Are there any user-facing changes?
No functional changes, other than allowing the user to use Python 3.15.
* GitHub Issue: #48172
### Progress Notes (2026-08-15)
Each platform uses different sources for Python builds, each with different release schedules and paths:
- [manylinux](https://github.com/pypa/manylinux) is prebuilt with latest Python builds in `/opt/python`, so using the latest manylinux build is sufficient. 3.15 wasn't available until beta 1.
- musllinux: Uses Alpine system python built from https://github.com/docker-library/python/pull/1123. Might need to bump ALPINE_LINUX version in [arrow/dev/tasks/python-wheels/github.linux.yml](https://github.com/apache/arrow/blob/e611f48081ec927e18e5bcb72fbd071c15c76b08/dev/tasks/python-wheels/github.linux.yml#L35).
- musllinux free-threaded: uses [astral's builds](https://github.com/astral-sh/python-build-standalone). Edit `python-free-threaded-wheel-musllinux-test-imports.dockerfile` + `-test-unittests.dockerfile`: set the python_patch_version, and set build_date in `compose.yaml`.
- macos: Downloads official build from python.org, see `ci/scripts/install_python.sh`. For early builds, needed to disable `--only-binary` since at least one dependency needed to be built (no 3.15 wheels available yet)... only macos uses --only-binary
- Windows: installs builds using `choco`, see `python-wheel-windows-test-vs2022.dockerfile` and `python-free-threaded-wheel-windows-test-vs2022.dockerfile`
For 3.15.0rc1 builds, some care was needed because certain paths needed 3.15.0 and others needed the full 3.15.0rc1.
Lead-authored-by: Paul Timmins <paul@iqmo.com>
Co-authored-by: paul@iqmo.com <paul@iqmo.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
This PR is a follow up to #48971 and should address #44655. Previously, the C++ AzureFileSystem couldn't be compiled on Windows due to a incomplete type definition. Now that this has been resolved, we should be able to enable building on the Windows Platform for PyArrow as well.
### What changes are included in this PR?
This PR sets the flags to enable building AzureFileSystem for PyArrow wheels on Windows. This will bring functionality in line with the Linux and macOS wheel builds.
### Are these changes tested?
These changes enable existing testing that's used for other C++ Filesystems in PyArrow. Tests will validate the class is importable from the built wheels which seems to be the existing precedent. I can add other tests if needed, but will likely need some direction.
### Are there any user-facing changes?
This PR will enable use of AzureFileSystem on Windows. Currently, trying to use or import the AzureFileSystem from PyArrow raises an ArrowNotImplementedError.
```python
>>> from pyarrow.fs import AzureFileSystem
ImportError: The pyarrow installation is not built with support for 'AzureFileSystem'
```
```python
>>> import pyarrow.fs as fs
>>> fs.FileSystem.from_uri('abfss://container@ account.dfs.core.windows.net/path')
pyarrow.lib.ArrowNotImplementedError: Got Azure Blob File System URI but Arrow compiled without Azure Blob File System support
```
After these changes, support should be available.
* GitHub Issue: #44655
Lead-authored-by: Nate Prewitt <nate.prewitt@gmail.com>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Co-authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
Pandas Windows-free threaded wheels are available again.
### What changes are included in this PR?
Reverts the change to temporarily disable and skip testing pandas for Windows free-threaded wheels.
### Are these changes tested?
Via archery
### Are there any user-facing changes?
No
* GitHub Issue: #46054
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Jacob Wujciak-Jens <jacob@wujciak.de>
### Rationale for this change
Pandas is currently not providing nightly wheels for free-threaded wheels. It has been failing for a while on their nightly channels and built wheels have been removed. Building from source fails on our free-threaded job. In order to provide our pyarrow Windows free-threaded wheels for 20.0.0 we temporarily disable installing pandas and testing pandas on free-threaded.
### What changes are included in this PR?
Temporarily disable pandas from being installed and tested on the Windows free-threaded wheel.
### Are these changes tested?
Via archery
### Are there any user-facing changes?
No
* GitHub Issue: #46041
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Jacob Wujciak-Jens <jacob@wujciak.de>
### Rationale for this change
We have two Windows issues and this PR is addressing both:
1. PyArrow's `download_tzdata_on_windows` can fail due to TLS issues in certain CI environments.
2. The Python wheel test infrastructure needs a tzinfo database for ORC and the automation fetching that started failing because the URL was made invalid upstream.
These two issues are being solved in one PR simply because they appeared together during the 19.0.1 release process but they're separate.
### What changes are included in this PR?
1. Makes `download_tzdata_on_windows` more robust to TLS errors by attempting to use `requests` if it's available and falling back to urllib otherwise.
2. Switches our Windows wheel test infrastructure to grab a tzinfo database from the tzdata package on PyPi instead of from a mirror URL. This should be much more stable for us over time.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
* GitHub Issue: #45295
Lead-authored-by: Bryce Mecum <petridish@gmail.com>
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Bryce Mecum <petridish@gmail.com>
### Rationale for this change
There's no blockers anymore for building Windows wheels for the free-threaded build, so this PR adds the necessary configuration to do that.
### What changes are included in this PR?
- Add jobs to build free-threaded wheels on Windows
- Move VCPKG-related stuff to a reusable base job
### Are these changes tested?
No tests required.
### Are there any user-facing changes?
No.
* GitHub Issue: #44421
Lead-authored-by: Antoine Pitrou <antoine@python.org>
Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
We need to ship the C++ standard library with our Windows wheels, as it is not guaranteed that a recent enough version is present on the system. However, some other Python libraries may require an even more recent version than the one we ship. This may incur crashes when PyArrow is imported before such other Python library, as the older version of the C++ standard library would be used by both.
### What changes are included in this PR?
Use a [fixed-up version](https://github.com/adang1345/delvewheel/pull/59) of delvewheel that allows us to name-mangle an individual DLL, and name-mangle `msvcp140.dll` to ensure that other Python libraries do not reuse the version we ship.
### Are these changes tested?
By regular wheel build tests.
* Closes: #44855
* GitHub Issue: #33981
* GitHub Issue: #44855
Lead-authored-by: Antoine Pitrou <antoine@python.org>
Co-authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
We currently build Windows wheels with S3 enabled, but we disable it for testing.
### What changes are included in this PR?
Ensure Minio is installed and re-enable S3 for testing.
### Are these changes tested?
Yes, by construction (on CI).
### Are there any user-facing changes?
No.
* GitHub Issue: #44095
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Currently we include everything when building wheels, see:
```
$ pip install pyarrow
Collecting pyarrow
Downloading pyarrow-17.0.0-cp310-cp310-manylinux_2_28_x86_64.whl (39.9 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 39.9/39.9 MB 33.8 MB/s eta 0:00:00
Collecting numpy>=1.16.6
Using cached numpy-2.0.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (19.3 MB)
Installing collected packages: numpy, pyarrow
Successfully installed numpy-2.0.0 pyarrow-17.0.0
(test-env) $ ls test-env/lib/python3.10/site-packages/
benchmarks/ distutils-precedence.pth numpy-2.0.0.dist-info/ pip-22.0.2.dist-info/ pyarrow-17.0.0.dist-info/ setuptools-59.6.0.dist-info/
cmake_modules/ examples/ numpy.libs/ pkg_resources/ scripts/
_distutils_hack/ numpy/ pip/ pyarrow/ setuptools/
```
### What changes are included in this PR?
Use `include` as seen here: https://setuptools.pypa.io/en/latest/userguide/package_discovery.html#finding-simple-packages
### Are these changes tested?
Will check via the build wheel on CI
### Are there any user-facing changes?
No and yes :)
We will remove unnecessary files
* GitHub Issue: #43299
Lead-authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Co-authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
### Rationale for this change
Installing the GCS testbench using the same Python that's being used to test PyArrow is fragile: some testbench versions may not be compatible, or there could be conflicts among the dependencies of the respective libraries.
### What changes are included in this PR?
Use `pipx` to install the GCS testbench in a separate, controlled environment, using an appropriate Python version.
### Are these changes tested?
Yes, by CI.
### Are there any user-facing changes?
No.
* GitHub Issue: #38183
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
Starting with https://github.com/apache/arrow/pull/41580, the pyarrow tests now also rely on a file in the parquet-testing submodule. And the path to that directory is controlled by `PARQUET_TEST_DATA`, which appears to be set wrongly in the wheel test scripts, causing all wheel builds to fail at the moment.
* GitHub Issue: #43785
Authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
### Rationale for this change
The pyarrow orc reader always crashes when it tries to create an internal orc reader. This is caused by failing to read tz database on the local host. This also disables windows wheel build when ARROW_ORC is turned on.
### What changes are included in this PR?
Download IANA timezone database on the test host and explicitly setting TZDIR to make the CIs happy.
### Are these changes tested?
Make sure all python wheel windows CIs pass.
### Are there any user-facing changes?
No.
* GitHub Issue: #36026
Authored-by: Gang Wu <ustcwg@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
If I build pyarrow with everything and then I remove some of the Arrow CPP .so in order to have a minimal build I can't import pyarrow because it requires libarrow and libparquet. This is relevant in order to have a minimal build for Conda. Please see the related issue for more information.
### What changes are included in this PR?
Move libarrow parquet encryption for pyarrow to its own shared object.
### Are these changes tested?
I will run extensive CI with extra python archery tests.
### Are there any user-facing changes?
No, and yes :) There will be a new .so on pyarrow but shouldn't be relevant in my opinion.
* Closes: #39006
Lead-authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Co-authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
GCS could be enabled on Windows wheels.
### What changes are included in this PR?
Enabling GCS on Windows wheel
### Are these changes tested?
Crossbow jobs for wheels run tests for GCS now.
I have tested locally that I can install the built wheel and I can import `GcsFileSystem`:
```
Python 3.9.12 (main, Apr 4 2022, 05:22:27) [MSC v.1916 64 bit (AMD64)] :: Anaconda, Inc. on win32
Warning:
This Python interpreter is in a conda environment, but the environment has
not been activated. Libraries may fail to load. To activate this environment
please see https://conda.io/activation
Type "help", "copyright", "credits" or "license" for more information.
>>> from pyarrow.fs import GcsFileSystem
>>> fs = GcsFileSystem(access_token='abc',target_service_account='service_account@ apache',credential_token_expiration=datetime.now(),default_bucket_location='us-west2',scheme='https', endpoint_override='localhost:8999')
>>> fs.default_bucket_location
'us-west2'
>>> fs.create_dir('hello')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "pyarrow\_fs.pyx", line 593, in pyarrow._fs.FileSystem.create_dir
File "pyarrow\error.pxi", line 113, in pyarrow.lib.check_status
PermissionError: [Errno 13] google::cloud::Status(UNAUTHENTICATED: Permanent error GetBucketMetadata: Could not create a OAuth2 access token to authenticate the request. The request was not sent, as such an access token is required to complete the request successfully. Learn more about Google Cloud authentication at https://cloud.google.com/docs/authentication. The underlying error message was: Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.). Detail: [errno 13] Permission denied
```
### Are there any user-facing changes?
No but Windows wheels should contain `ARROW_GCS`
* Closes: #35193
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Jacob Wujciak-Jens <jacob@wujciak.de>
### Rationale for this change
See the linked issue
### What changes are included in this PR?
C++:
* remove all compute/exec/* from libarrow
* rename compute/exec -> acero and make libarrow_acero
* add new ARROW_ACERO option, required if ARROW_DATASET is on
* libarrow_dataset now depends on libarrow_acero
c_glib: add the new libarrow_acero dependency - we disallow building glib without it
python: added PYARROW_BUILD_ACERO, set to on if DATASETS are built
### Are these changes tested?
All the standard tests do work properly.
I manually compiled C++ with:
* no ARROW_ACERO
* ARROW_ACERO and no ARROW_DATASET
* ARROW_ACERO and ARROW_DATASET and no ARROW_SUBSTRAIT
I manually compiled python without ACERO & DATASET and with ACERO and without DATASET
### Are there any user-facing changes?
If users include compute/exec files directly then they'll have to update their code.
* Closes: #15280
Lead-authored-by: Davide Pasetto <dpasetto69@gmail.com>
Co-authored-by: Li Jin <ice.xelloss@gmail.com>
Co-authored-by: Weston Pace <weston.pace@gmail.com>
Signed-off-by: Li Jin <ice.xelloss@gmail.com>
### Rationale for this change
We can't maintain Plasma.
### What changes are included in this PR?
These changes remove all Plasma related codes.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* Closes: #33243
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
With no explicit exit code `exit /b` returns 0. This made the the step never fail.
The failing test is only for demonstration and will of course be removed prior to merging this PR.
I also checked that the same issue isn't happening anywhere else.
Closes#12844 from assignUser/ARROW-15951-make-ci-fail
Authored-by: Jacob Wujciak-Jens <jacob@wujciak.de>
Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
Also resolves:
- ARROW-11213: [Packaging][Python] Dockerize wheel building on windows
- ARROW-11215: [CI] Use named volumes by default for caching in docker-compose
- ARROW-11231: [Python][Packaging] Enable mimalloc in wheels
Main features:
- dockerize windows builds
- use vcpkg as the dependency source where we can explicitly pin a working version without worrying about the drag of build environment over time
Potential follow-up:
- use vcpkg on macos as well
cc @kou @xhochy
## Manylinux testing
Should be straightforward (I'm going to shared pre-built images for quicker testing).
## Windows testing
Only windows host is able to run windows containers. I'm virtualizing windows on macOS (should work on linux as well) using virtualbox:
```bash
git clone https://github.com/StefanScherer/windows-docker-machine
cd windows-docker-machine
# grant more resources in the vagrantfile under the virtualbox section (I use 8 cores and 16GiB of ram)
vagrant up --provider virtualbox 2019-box
docker context ls
docker context use 2019-box
docker image ls
```
Now docker should use the windows docker daemon.
```bash
pip install -e dev/archery[docker]
# the volumes defined in the docker compose file should be commented out and pass arrow's source explicitly
# this is required because compose is being executed on a unix host and the directory is mounted through a virtual machine
PYTHON=3.6 archery docker run --no-pull --using-docker-cli -v C:$(pwd):C:/arrow python-wheel-windows-vs2017 cmd.exe
# then within the container execute a build or do other interactive things
arrow\ci\scripts\python_wheel_windown_build.bat
```
Closes#9096 from kszucs/wheel-vcpkg
Lead-authored-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
Co-authored-by: Sutou Kouhei <kou@clear-code.com>
Co-authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
paultiq