24 Commits
Author SHA1 Message Date
paultiq 9f0341314c GH-52522: [CI][Python] Build with released CPython 3.15.0 (#53264)
### Rationale for this change

Python 3.15.0 released 2026-10-09 (https://www.python.org/downloads/release/python-3150/).

This is a follow-up to https://github.com/apache/arrow/pull/48191, bumping the release pins and versions from 3.15.0rc2 to 3.15.0 (released).

### What changes are included in this PR?
- Change version reference from 3.15.0rc2 to 3.15.0
- Update manylinux and musllinux quay.io build dates
- Update build date for python-build-standalone
- Bump PYTHON_WHEEL_WINDOWS_IMAGE_REVISION and PYTHON_WHEEL_WINDOWS_TEST_IMAGE_REVISION

### Are these changes tested?
No

### Are there any user-facing changes?
No

### Was AI used for this PR?

**PR code and description written by:**

- [X] Human
- [ ] AI

**Reviewed before submission by:**

- [X] Human
- [X] AI
- [ ] Not reviewed

* GitHub Issue: #52522

Authored-by: paul@iqmo.com <paul@iqmo.com>
Signed-off-by: Raúl Cumplido <raulcd@apache.org>
2026-10-16 13:41:48 +01:00
paultiq a91ae52ea7 GH-51208: [CI][Python] Update builds to CPython 3.15.0rc2 (cp315) (#51209)
### Rationale for this change

https://github.com/apache/arrow/pull/48191 added 3.15 to CI, but pins to rc1 in a few places.

rc2 is now available, this is a minimal update from rc1 to rc2. 

### What changes are included in this PR?

Moves pin from rc1 to rc2 in the dockerfiles and install_python.sh

### Are these changes tested?

Tested the manylinux build

### Are there any user-facing changes?

No
* GitHub Issue: #51208

Authored-by: paul@iqmo.com <paul@iqmo.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
2026-09-10 09:07:03 +02:00
paultiqandpaul@iqmo.com ba5ca205d1 GH-48172: [Python] Add cp315 to build (#48191)
### Rationale for this change
Add [Python 3.15](https://docs.python.org/3.15/whatsnew/3.15.html) to the pyarrow nightly builds, so that issues can be detected early in the development cycles of both cpython & pyarrow. 

Note: 
- ~~CFFI and~~ Pandas doesn't have nightly 3.15 wheels, and are accordingly excluded. 
- Numpy shipped 3.15rc1 in 2.5.2 ~~doesn't ship a 3.15 wheel, but is needed to build. It is building from source.~~
- [3.15 Schedule](https://peps.python.org/pep-0790/) This PR targets 3.15.0rc1. rc2 is expected 2026-09-01 and final 2026-10-01. 

### What changes are included in this PR?
- Adding cp315/cp315t to the build matrix
- Excluding ~~cffi/~~ pandas for 3.15, as the wheels aren't yet available.

### 2026-08-13 edit
- Bump libcst to 1.9.0
- Update manylinux image to 2026.08.08-1 (for 3.15rc1)
- Add -rc to PYTHON_IMAGE_TAG
- Disable pandas tests if Pandas isn't installed: 

### Are these changes tested?
- Yes, tested on a local Linux build using the `ninja-release-python` preset (not maximal)

### Are there any user-facing changes?
No functional changes, other than allowing the user to use Python 3.15. 
* GitHub Issue: #48172

### Progress Notes (2026-08-15)
Each platform uses different sources for Python builds, each with different release schedules and paths:

- [manylinux](https://github.com/pypa/manylinux) is prebuilt with latest Python builds in `/opt/python`, so using the latest manylinux build is sufficient. 3.15 wasn't available until beta 1. 
- musllinux: Uses Alpine system python built from https://github.com/docker-library/python/pull/1123. Might need to bump ALPINE_LINUX version in [arrow/dev/tasks/python-wheels/github.linux.yml](https://github.com/apache/arrow/blob/e611f48081ec927e18e5bcb72fbd071c15c76b08/dev/tasks/python-wheels/github.linux.yml#L35). 
- musllinux free-threaded: uses [astral's builds](https://github.com/astral-sh/python-build-standalone). Edit `python-free-threaded-wheel-musllinux-test-imports.dockerfile` + `-test-unittests.dockerfile`: set the python_patch_version, and set build_date in `compose.yaml`. 
- macos: Downloads official build from python.org, see `ci/scripts/install_python.sh`. For early builds, needed to disable `--only-binary` since at least one dependency needed to be built (no 3.15 wheels available yet)... only macos uses --only-binary
- Windows: installs builds using `choco`, see `python-wheel-windows-test-vs2022.dockerfile` and `python-free-threaded-wheel-windows-test-vs2022.dockerfile`

For 3.15.0rc1 builds, some care was needed because certain paths needed 3.15.0 and others needed the full 3.15.0rc1. 

Lead-authored-by: Paul Timmins <paul@iqmo.com>
Co-authored-by: paul@iqmo.com <paul@iqmo.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
2026-08-21 14:18:33 +02:00
Raúl Cumplido 934fa870a7 GH-50410: [Python][Packaging] Drop Python 3.10 support (#50411)
### Rationale for this change

Python 3.10 is End of Support on October 2026. We can drop support on PyArrow 26.0.0.
See: https://endoflife.date/python

### What changes are included in this PR?

Remove support for 3.10. Update minimum required Python version and update some CI jobs to use the minimum Python supported version.
Some CI jobs using default Python 3.10 (Ubuntu 20.04) have been bumped.

### Are these changes tested?

They will be tested via CI and Archery

### Are there any user-facing changes?

Yes in terms of not supporting older versions of Python. This will be a breaking change for some users but not for the API itself

* GitHub Issue: #50410

Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
2026-07-09 12:33:32 +02:00
Raúl Cumplido b35883ea37 GH-48028: [Python][Packaging] Update quay.io base manylinux and musllinux images for Python wheels and remove cp313t (#50082)
### Rationale for this change

When adding support to Python 3.14 the musllinux wheels failed with newer versions of the musllinux quay.io image. In order to prepare for the next Python 3.15 release we will require newer base images and we should fix the issues we originally found.

### What changes are included in this PR?

- Update Manylinux and Musllinux quay.io images to `2026.05.31-3`
- Use mold linker instead of ld on alpine in order to solve import segfault on pyarrow musllinux amd64
- Remove Python 3.13 free-threaded wheels as Python 3.13t is not available anymore on the underlying manylinux image, see:
-  https://github.com/apache/arrow/issues/50086

### Are these changes tested?

Yes via archery

### Are there any user-facing changes?

Yes, Python 3.13 free threaded was experimental, PyPA removed cp313t from the underlying manylinux docker image. We won't support wheels for 3.13 free-threaded anymore. We will support for 3.14 on-wards.

* GitHub Issue: #48028

Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
2026-06-06 05:42:26 +09:00
Alenka Frim e90bacded2 GH-48280: [CI] PYTHON_PATCH_VERSION docker warnings (#48282)
### Rationale for this change

Warnings in the docker compose step, see https://github.com/ursacomputing/crossbow/actions/runs/19167940422/job/54793047049#step:7:55

### What changes are included in this PR?

Instead of adding `PYTHON_PATCH_VERSION` to `.env` I have moved the use of the patch version to the free-threaded musllinux jobs as `PYTHON_PATCH_VERSION` was only needed there.

I have also updated all Python patch versions from `3.13.7` to `3.13.9`.

### Are these changes tested?

Yes, CI and extended python wheels builds should pass.

### Are there any user-facing changes?

No.

* GitHub Issue: #48280

Authored-by: AlenkaF <frim.alenka@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
2025-12-04 07:07:17 +09:00
Alenka Frim f268c43ccd GH-47438: [Python][Packaging] Set up wheel building for Python 3.14 (#47616)
### Rationale for this change

Python 3.14 is currently in a prerelease status and is expected to have a final release in October this year (https://peps.python.org/pep-0745/).

We should ensure we are fully ready to support Python 3.14 for the PyArrow 22 release.

### What changes are included in this PR?

This PR  updates wheels for Python 3.14.

### Are these changes tested?

Tested in the CI and with extended builds.

### Are there any user-facing changes?

No, but users will be able to use PyArrow with Python 3.14.

* GitHub Issue: #47438

---

Todo:

- Update the image revision name in `.env`
- Add 3.14 conda build ([arrow/dev/tasks/tasks.yml](https://github.com/apache/arrow/blob/d803afcc43f5d132506318fd9e162d33b2c3d4cd/dev/tasks/tasks.yml#L809)) when https://github.com/conda-forge/pyarrow-feedstock/pull/156 is merged 

Follow-ups:

- https://github.com/apache/arrow/issues/47437

Authored-by: AlenkaF <frim.alenka@gmail.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
2025-10-14 06:17:08 +02:00
Raúl CumplidoandSutou Kouhei aedb752762 GH-47443: [Python][Packaging] Drop Python 3.9 support (#47478)
### Rationale for this change

Python 3.9 is End of Support on 31st October 2025. We can drop support for pyarrow 22.0.0.
See: https://endoflife.date/python

### What changes are included in this PR?

Remove support for 3.9. Update minimum required Python version and update some CI jobs to use the minimum Python supported version

### Are these changes tested?

They will be tested via CI and Archery

### Are there any user-facing changes?

Yes in terms of not supporting older versions of Python. This will be a breaking change for some users but not for the API itself
* GitHub Issue: #47443

Lead-authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
2025-09-09 09:44:55 +02:00
Hiroyuki Sato ddfa762963 GH-46496: [CI][Dev] Fix shellcheck SC2086 errors in ci/scripts directory (#46497)
### Rationale for this change

We are trying to implement shellcheck on all sh files in #44748.

### What changes are included in this PR?

SC2086 check requires quoting like `${url}` -> `"${url}"`.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

No.

* GitHub Issue: #46496

Authored-by: Hiroyuki Sato <hiroysato@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
2025-05-21 10:07:58 +09:00
Joris Van den Bossche ec3d2839d1 GH-43519: [Python][CI] Update Python 3.13 rc to final 3.13.2 (#44375)
### Rationale for this change

The final Python 3.13.0 is out now, so we can update those versions

* GitHub Issue: #43519

Authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
2025-02-17 16:05:37 +01:00
Lysandros NikolaouandAntoine Pitrou 1c1f7f359d GH-43964: [Python] Build macOS and manylinux wheels for free-threading (#43965)
### Rationale for this change

Building free-threaded wheels is necessary to support the free-threaded build. We probably want to upload these wheels as nightlies somewhere as well, so that downstream users can test the free-threading-related changes.

### What changes are included in this PR?

- Add necessary configuration to build 3.13 free-threading wheels on `manylinux`.
- Do necessary changes to build free-threaded wheels on macOS as well.

### Are these changes tested?

I tested the `manylinux` wheel builds. macOS is still untested, since it's not dockerized.

### Are there any user-facing changes?

No.

Related to #43536.
* GitHub Issue: #43964

Lead-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
Co-authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
2024-09-18 11:07:13 +02:00
Raúl Cumplido aa6ab95a12 GH-43518: [Python][Packaging][CI] Drop Python 3.8 support (#43970)
### Rationale for this change

Python 3.8 is End of Support on 31st October 2023. We can drop support for pyarrow 18.0.0.
See: https://endoflife.date/python

### What changes are included in this PR?

Remove support for 3.8. Update minimum required Python version and update some CI jobs to use the minimum Python supported version

### Are these changes tested?

They will be tested via Archery

### Are there any user-facing changes?

Yes in terms of not supporting older versions of Python. This will be a breaking change for some users but not for the API itself

* GitHub Issue: #43518

Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
2024-09-16 15:38:38 +02:00
Lysandros NikolaouandJoris Van den Bossche 3e9384bbf4 GH-43519: [Python] Set up wheel building for Python 3.13 (#43539)
### Rationale for this change

Like #43519 mentionies, now that the first `rc` is out, it's probably time to add CI coverage for Python 3.13 (and also start building wheels).

### What changes are included in this PR?

I'm fairly new to the build/CI processes of the project, but I tried to follow the same template as #37901. I'll follow up afterwards with adding CI coverage for the free-threaded build as well.
* GitHub Issue: #43519

Lead-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
2024-08-22 15:27:40 +02:00
Josh Soref e3c8187c61 GH-38925: [CI] Fix spelling (#38926)
### Rationale for this change

### What changes are included in this PR?

Spelling fixes to ci/

### Are these changes tested?

### Are there any user-facing changes?

* Closes: #38925

Authored-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
2023-12-10 21:22:08 +09:00
Raúl Cumplido 7695935a2e GH-37880: [CI][Python][Packaging] Add support for Python 3.12 (#37901)
### Rationale for this change

Python 3.12 will be released in the next couple of weeks. We should add the wheels for pyarrow on our 14.0.0 release.

### What changes are included in this PR?

This PR adds jobs to build pyarrow wheels for Python 3.12.

### Are these changes tested?

They will be tested via archery tasks

### Are there any user-facing changes?

No but users will be able to use pyarrow with Python 3.12

* Closes: #37880

Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
2023-10-11 16:33:18 +02:00
Raúl CumplidoandSutou Kouhei 8d021c401f GH-34788: [Python][Packaging][CI] Drop Python 3.7 support (#36061)
### Rationale for this change

Python 3.7 is End of Support on 27th June 2023. We can drop support for pyarrow 13.0.0.
See: https://endoflife.date/python

### What changes are included in this PR?

Remove support for 3.7. Update minimum required Python version and update some CI jobs to use the minimum Python supported version

### Are these changes tested?

They will be tested via Archery

### Are there any user-facing changes?

Yes in terms of not supporting older versions of Python. This will be a breaking change for some users but not for the API itself
* Closes: #34788

Lead-authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
2023-06-22 05:36:24 +09:00
Raúl Cumplido e21d5aab76 ARROW-17487: [Python][Packaging][CI] Add support for Python 3.11 (#14499)
This PR adds jobs to build pyarrow wheels for Python 3.11.

Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
2022-11-05 05:32:09 +09:00
Antoine PitrouandKrisztián Szűcs 875af62cf2 ARROW-15890: [CI][Python] Use venv instead of virtualenv
Closes #12592 from pitrou/python-venv

Lead-authored-by: Antoine Pitrou <antoine@python.org>
Co-authored-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
2022-03-09 18:48:12 +01:00
Antoine Pitrou 0f4d662ae5 ARROW-12706: [Python] Drop Python 3.6 support
Closes #12026 from pitrou/ARROW-12706-drop-py36

Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
2021-12-29 13:40:52 +01:00
Krisztián SzűcsandJiayu Liu 721aa019aa ARROW-14410: [Python][Packaging] Use numpy 1.21.3 to build python 3.10 wheels for macOS and windows
Closes #11499 from kszucs/ARROW-14410

Lead-authored-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
Co-authored-by: Jiayu Liu <jiayu.liu@airbnb.com>
Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
2021-10-21 19:27:04 +02:00
Krisztián Szűcs 75d07a59ee ARROW-14377: [Packaging][Python] Python 3.9 installation fails in macOS wheel build
Closes #11462 from kszucs/python-install-script

Authored-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
2021-10-19 15:33:55 +02:00
Krisztián SzűcsandAntoine Pitrou 15f4e56c8f ARROW-14217: [Python][CI] Add support for python 3.10
Closes #11316 from kszucs/py310

Lead-authored-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
Co-authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
2021-10-18 12:22:22 +02:00
Krisztián Szűcs 5b56cb57ec ARROW-13461: [Python][Packaging] Build M1 wheels for python 3.8
Closes #10812 from kszucs/apple-silicon-py38

Authored-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
2021-07-27 17:56:18 +02:00
Krisztián SzűcsandUrsa ac050c6c58 ARROW-10658: [Python][Packaging] Wheel builds for Apple Silicon
The following compiles the 3.9 wheel on apple silicon though I'm getting a segfault on import despite that all binaries have the right arch and linkage/bundling seems correct. I wasn't able to produce a coredump yet.

Closes #10659 from kszucs/m1

Lead-authored-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
Co-authored-by: Ursa <ursa@ursa.org>
Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
2021-07-20 00:10:43 +02:00