### Rationale for this change
This PR moves crossbow builds to short-lived S3 credentials. It does this via GitHub's OIDC provider and an AWS ROLE_ARN with a trust relationship configured. This configuration is detailed here:
- [Use IAM roles to connect GitHub Actions to actions in AWS](https://aws.amazon.com/blogs/security/use-iam-roles-to-connect-github-actions-to-actions-in-aws/)
The benefits of short-lived / temporary credentials discussed further here: [AWS Well-architected Framework Best Practices SEC02-BP02 Use temporary credentials](https://docs.aws.amazon.com/wellarchitected/latest/framework/sec_identities_unique.html)
sccache supports a variety of S3 creds: [S3 credentials](https://github.com/mozilla/sccache/blob/main/docs/S3.md#credentials)
### What changes are included in this PR?
Use [AssumeRoleWithWebIdentity](https://github.com/mozilla/sccache/blob/main/docs/S3.md#credentials) for sccache S3 credentials.
By assuming the AWS_ROLE_ARN, the sccache session relies on credentials that expire\*, issued only to workflows in the trusted repository, and scoped to the sccache bucket.
Specific tasks:
- create a role in IAM with the bucket and object level permissions,
- trust the github oidc-provider,
- define AWS_ROLE_ARN and set id-token permission
- grant sts:AssumeRoleWithWebIdentity for the specific repositories or organization.
- pass AWS_SESSION_TOKEN
- add a `aws-actions/configure-aws-credentials@ v6` step to the workflow.
Separately, someone will need to add the role and trust to AWS:
- Procedure is in https://aws.amazon.com/blogs/security/use-iam-roles-to-connect-github-actions-to-actions-in-aws/
- Set a "Maximum session duration" to maximum runtime of any job... maybe 4 hours? 8 hours?
- Permissions: Role needs BucketLevel and ObjectLevels permissions to the bucket.
\* I can share an example if needed
### Are these changes tested?
Yes, locally and on a fork.
### Are there any user-facing changes?
No
* GitHub Issue: #50899
Authored-by: paul@iqmo.com <paul@iqmo.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
Add [Python 3.15](https://docs.python.org/3.15/whatsnew/3.15.html) to the pyarrow nightly builds, so that issues can be detected early in the development cycles of both cpython & pyarrow.
Note:
- ~~CFFI and~~ Pandas doesn't have nightly 3.15 wheels, and are accordingly excluded.
- Numpy shipped 3.15rc1 in 2.5.2 ~~doesn't ship a 3.15 wheel, but is needed to build. It is building from source.~~
- [3.15 Schedule](https://peps.python.org/pep-0790/) This PR targets 3.15.0rc1. rc2 is expected 2026-09-01 and final 2026-10-01.
### What changes are included in this PR?
- Adding cp315/cp315t to the build matrix
- Excluding ~~cffi/~~ pandas for 3.15, as the wheels aren't yet available.
### 2026-08-13 edit
- Bump libcst to 1.9.0
- Update manylinux image to 2026.08.08-1 (for 3.15rc1)
- Add -rc to PYTHON_IMAGE_TAG
- Disable pandas tests if Pandas isn't installed:
### Are these changes tested?
- Yes, tested on a local Linux build using the `ninja-release-python` preset (not maximal)
### Are there any user-facing changes?
No functional changes, other than allowing the user to use Python 3.15.
* GitHub Issue: #48172
### Progress Notes (2026-08-15)
Each platform uses different sources for Python builds, each with different release schedules and paths:
- [manylinux](https://github.com/pypa/manylinux) is prebuilt with latest Python builds in `/opt/python`, so using the latest manylinux build is sufficient. 3.15 wasn't available until beta 1.
- musllinux: Uses Alpine system python built from https://github.com/docker-library/python/pull/1123. Might need to bump ALPINE_LINUX version in [arrow/dev/tasks/python-wheels/github.linux.yml](https://github.com/apache/arrow/blob/e611f48081ec927e18e5bcb72fbd071c15c76b08/dev/tasks/python-wheels/github.linux.yml#L35).
- musllinux free-threaded: uses [astral's builds](https://github.com/astral-sh/python-build-standalone). Edit `python-free-threaded-wheel-musllinux-test-imports.dockerfile` + `-test-unittests.dockerfile`: set the python_patch_version, and set build_date in `compose.yaml`.
- macos: Downloads official build from python.org, see `ci/scripts/install_python.sh`. For early builds, needed to disable `--only-binary` since at least one dependency needed to be built (no 3.15 wheels available yet)... only macos uses --only-binary
- Windows: installs builds using `choco`, see `python-wheel-windows-test-vs2022.dockerfile` and `python-free-threaded-wheel-windows-test-vs2022.dockerfile`
For 3.15.0rc1 builds, some care was needed because certain paths needed 3.15.0 and others needed the full 3.15.0rc1.
Lead-authored-by: Paul Timmins <paul@iqmo.com>
Co-authored-by: paul@iqmo.com <paul@iqmo.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Python 3.10 is End of Support on October 2026. We can drop support on PyArrow 26.0.0.
See: https://endoflife.date/python
### What changes are included in this PR?
Remove support for 3.10. Update minimum required Python version and update some CI jobs to use the minimum Python supported version.
Some CI jobs using default Python 3.10 (Ubuntu 20.04) have been bumped.
### Are these changes tested?
They will be tested via CI and Archery
### Are there any user-facing changes?
Yes in terms of not supporting older versions of Python. This will be a breaking change for some users but not for the API itself
* GitHub Issue: #50410
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Our Archery docker setup had support for running foreign-arch containers (for example a ARM64 container on a x86-64 native host, using emulation), but it failed working at some point.
### Are these changes tested?
By existing CI runs, as well as locally (by running a couple ARM64 containers on a x86-64 native host).
### Are there any user-facing changes?
No.
* GitHub Issue: #32994
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
The docker base docker image which installs all the vcpkg dependencies to build Windows wheels was failing to commit the layer for gRPC due to size.
### What changes are included in this PR?
Two separate fixes that independently fix the issue:
- Patch vcpkg for Windows to remove unconditional `/Z7`
- Use D:/ drive instead of C:/ as it has bigger size for Docker storage
### Are these changes tested?
Yes via archery
### Are there any user-facing changes?
No
* GitHub Issue: #49676
Lead-authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Co-authored-by: Bryce Mecum <petridish@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Move our PyArrow build backend from setuptools and a custom setup.py to scikit-build-core which is just build backend for CMake related projects.
### What changes are included in this PR?
Move from setuptools to scikit-build-core and remove PyArrow setup.py. Update some of the build requirements and minor fixes.
A custom build backend has been also been created in order to wrap scikit-build-core in order to fix problems on License files for monorepos.
pyproject.toml metadata validation expects license files to exist before exercising the build backend that's why we create symlinks. Our thin build backend will just make those symlinks hard-links in order for license and notice files to contain the contents and be added as part of the sdist.
Remove flags that are not used anymore (were only part of setup.py) and documented and validated how the same flags have to be used now.
### Are these changes tested?
Yes all Python CI tests, wheels and sdist are successful.
### Are there any user-facing changes?
Yes, users building PyArrow will now require the new build dependencies to exercise the build and depending on the flags used they might require to use the new documented way of using those flags.
* GitHub Issue: #36411
Lead-authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Co-authored-by: Rok Mihevc <rok@mihevc.org>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Currently we are not building wheels with OpenTelemetry.
### What changes are included in this PR?
- Adding a new feature on our `vcpkg.json` to install required opentelemetry-cpp
- Enabling the opentelemetry feature when downloading dependencies via vcpkg for our wheels.
- Setting `ARROW_WITH_OPENTELEMETRY=ON` on the wheel build scripts
- Minor fix on checking exposed symbols (opentelemetry symbols are exposed)
- Minor improvement to `python/examples/flight/middleware.py` to be able to easily enable opentelemetry middleware tracing for testing.
- Fix on `server_tracing_middleware.cc` to retrieve tracer from `arrow::internal::tracing::GetTracer();` instead of `otel::trace::Provider::GetTracerProvider()->GetTracer("arrow");`
- Expose missing symbols to enable Windows builds
- Fix Windows compatibility adding windows compat header.
- Add `is_opentelemetry_enabled` utility function in pyarrow to validate whether OpenTelemetry was enabled or not when building Arrow C++. Expose this information on `pa.show_info()`.
### Are these changes tested?
Yes, wheels are being successfully built and tested locally with a built wheel that traces are being generated when properly configured.
Added test to validate traces appear on stdout and I've also validated Acero traces are being correctly generated by running the test suite and seeing the opentelemetry traces on Jaeger UI.
### Are there any user-facing changes?
Yes, PyArrow wheel users will be able to enable OpenTelemetry traces.
* GitHub Issue: #49382
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
The `wheel-manylinux-2-28-cp312-cp312-amd64` performs the `almalinux-8` wheels verification and it fails with out of space: `At least 746MB more space needed on the / filesystem.`
### What changes are included in this PR?
Use our existing `github_free_space` macro to clean space on the runner.
### Are these changes tested?
Yes, via archery. I've validated the failure happened on the PR without the change and was solved with the clean up.
### Are there any user-facing changes?
No
* GitHub Issue: #48807
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
Warnings in the docker compose step, see https://github.com/ursacomputing/crossbow/actions/runs/19167940422/job/54793047049#step:7:55
### What changes are included in this PR?
Instead of adding `PYTHON_PATCH_VERSION` to `.env` I have moved the use of the patch version to the free-threaded musllinux jobs as `PYTHON_PATCH_VERSION` was only needed there.
I have also updated all Python patch versions from `3.13.7` to `3.13.9`.
### Are these changes tested?
Yes, CI and extended python wheels builds should pass.
### Are there any user-facing changes?
No.
* GitHub Issue: #48280
Authored-by: AlenkaF <frim.alenka@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
Experimentation in https://github.com/apache/arrow/pull/47637 suggested that [enabling inter-procedural optimization (IPO)](https://cmake.org/cmake/help/latest/variable/CMAKE_INTERPROCEDURAL_OPTIMIZATION.html) could not only improve performance in some workloads, but also reduce the code size of Arrow libraries.
There's around 2% size reduction on the tested examples:
|wheel|size|
|---------|--------|
|last successful nightly `wheel-manylinux-2-28-cp312-cp312-arm64` | 44605918 bytes|
|with change `wheel-manylinux-2-28-cp312-cp312-arm64` | 43807946 bytes|
|last successful nightly `wheel-windows-cp311-cp311-amd64` | 28093080 bytes|
|with change `wheel-windows-cp311-cp311-amd64` | 27391407 bytes|
### What changes are included in this PR?
Add `-DCMAKE_INTERPROCEDURAL_OPTIMIZATION=ON` to Arrow C++ and wheels for both manylinux and windows.
### Are these changes tested?
Yes via CI
### Are there any user-facing changes?
No
* GitHub Issue: #47643
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Our vcpkg version used for the wheels is a really old version. Some wheels like the macOS ones have started failing due to really old dependencies using unsupported cake versions.
### What changes are included in this PR?
- Update the vcpkg version to the latest released
- Update minio version to solve failure due to newer aws-sdk
- Remove unnecessary vcpkg patches
- Add patch to ORC vcpkg to solve issue with exception not being propagated on std::run_once on musllinux.
- Avoid adding some flight sources on Windows unity builds that cause windows headers clashing.
### Are these changes tested?
Yes, via archery on CI.
### Are there any user-facing changes?
No
* GitHub Issue: #43416
* GitHub Issue: #47614
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Python 3.9 is End of Support on 31st October 2025. We can drop support for pyarrow 22.0.0.
See: https://endoflife.date/python
### What changes are included in this PR?
Remove support for 3.9. Update minimum required Python version and update some CI jobs to use the minimum Python supported version
### Are these changes tested?
They will be tested via CI and Archery
### Are there any user-facing changes?
Yes in terms of not supporting older versions of Python. This will be a breaking change for some users but not for the API itself
* GitHub Issue: #47443
Lead-authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
We started uploading nightly wheels to the scientific python channel on anaconda:
https://anaconda.org/scientific-python-nightly-wheels/pyarrow
Which are the documented nightlies:
https://arrow.apache.org/docs/dev/developers/python.html#installing-nightly-packages
We updated the documentation and have been deprecating with a long release cycle.
Uploading wheels to gemfury sometimes fail due to trying to upload the same version of the wheels.
### What changes are included in this PR?
Stop uploading nightly wheels to gemfury.
### Are these changes tested?
No
### Are there any user-facing changes?
Yes, nightly wheels will not be available on gemfury anymore.
* GitHub Issue: #43904
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
We can cover all supported platforms by manylinux_2_28. Amazon Linux 2 needs manylinux2014 and it's still maintained (it'll reach EOL on 2026-06-30) but we dropped support for Amazon Linux 2. See also: #36282
### What changes are included in this PR?
Remove manylinux2014 related codes.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* GitHub Issue: #46959
Lead-authored-by: Sutou Kouhei <kou@clear-code.com>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
Please check #18036.
### What changes are included in this PR?
Almost everything needed for building and testing Python wheels for musllinux.
The side-effect of these changes are the exclusion of 2 unit tests when running on musl based systems.
The skipped unit tests are:
- test_uwsgi_integration
- test_print_stats
### Are these changes tested?
I was able to successfully generate a musllinux wheel by running the following:
```
docker-compose build python-wheel-musllinux-1-2
docker-compose run python-wheel-musllinux-1-2
```
I was also able to run the following tests with no errors:
```
python-wheel-musllinux-test-imports
python-free-threaded-wheel-musllinux-test-imports
python-wheel-musllinux-test-unittests
python-free-threaded-wheel-musllinux-test-unittests
```
### Are there any user-facing changes?
Yes.
* GitHub Issue: #18036
Authored-by: Nuno Veloso <nveloso21@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
See https://github.com/apache/arrow/issues/45156. Refactors our previous Windows Docker CI images from an unofficial image to the official Microsoft one based on Windows Server 2022 and adds VS2022 BuildTools.
### What changes are included in this PR?
- New Windows base Dockerfiles
- Updated Dockerfiles which build on top of that base
- Updated Docker Compose services to use new images
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
* GitHub Issue: #45156
Lead-authored-by: Bryce Mecum <petridish@gmail.com>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Signed-off-by: Bryce Mecum <petridish@gmail.com>
### Rationale for this change
The final Python 3.13.0 is out now, so we can update those versions
* GitHub Issue: #43519
Authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
We want to upgrade our CMake version to 3.25 as discussed on the ML:
https://lists.apache.org/thread/h8jp16ktrj11fmjmjhlg6xvkvv9wzvjk
### What changes are included in this PR?
- Bump minimal CMake version to 3.25
- Manually install CMake on distributions where CMake < 3.25 was installed via package repositories
- Minor fixes to CI in order to have passing builds everywhere
### Are these changes tested?
Yes, via CI.
### Are there any user-facing changes?
Yes, the minimum CMake version to be used to build Arrow is bumped to 3.25.
**This PR includes breaking changes to build systems.**
* GitHub Issue: #44950
Lead-authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Co-authored-by: Bryce Mecum <petridish@gmail.com>
Co-authored-by: Sutou Kouhei <kou@clear-code.com>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Co-authored-by: Neal Richardson <neal.p.richardson@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
Ubuntu 20.04 will reach EOL on 2025-05.
### What changes are included in this PR?
Remove a wheel test that uses Ubuntu 20.04.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
* GitHub Issue: #45482
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
The upload wasn't failing with the empty, non existent dir.
### What changes are included in this PR?
- Fix the path
- Prevent the upload from failing silently when no wheels are found
### Are these changes tested?
crossbow
### Are there any user-facing changes?
no
* GitHub Issue: #45155
Lead-authored-by: Jacob Wujciak-Jens <jacob@wujciak.de>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Signed-off-by: Jacob Wujciak-Jens <jacob@wujciak.de>
### Rationale for this change
There's no blockers anymore for building Windows wheels for the free-threaded build, so this PR adds the necessary configuration to do that.
### What changes are included in this PR?
- Add jobs to build free-threaded wheels on Windows
- Move VCPKG-related stuff to a reusable base job
### Are these changes tested?
No tests required.
### Are there any user-facing changes?
No.
* GitHub Issue: #44421
Lead-authored-by: Antoine Pitrou <antoine@python.org>
Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
We need to ship the C++ standard library with our Windows wheels, as it is not guaranteed that a recent enough version is present on the system. However, some other Python libraries may require an even more recent version than the one we ship. This may incur crashes when PyArrow is imported before such other Python library, as the older version of the C++ standard library would be used by both.
### What changes are included in this PR?
Use a [fixed-up version](https://github.com/adang1345/delvewheel/pull/59) of delvewheel that allows us to name-mangle an individual DLL, and name-mangle `msvcp140.dll` to ensure that other Python libraries do not reuse the version we ship.
### Are these changes tested?
By regular wheel build tests.
* Closes: #44855
* GitHub Issue: #33981
* GitHub Issue: #44855
Lead-authored-by: Antoine Pitrou <antoine@python.org>
Co-authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
We can use GitHub Actions' standard features not Jinja2 features to generate test image names.
### What changes are included in this PR?
Use `GITHUB_ENV` instead of Jinja2.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
* GitHub Issue: https://github.com/apache/arrow/issues/45076
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
We already test some Python wheels on Ubuntu 20.04 or 22.04, depending on the system Python version. Also do so with Ubuntu 24.04.
### Are these changes tested?
Yes, by definition.
### Are there any user-facing changes?
No.
* GitHub Issue: #45047
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
We are currently downloading and building ORC every time for our wheels. We should not require to bundle ORC ourselves and we could just use vcpkg.
### What changes are included in this PR?
Stop using `ORC_SOURCE=BUNDLED` and enable the vcpkg orc feature.
### Are these changes tested?
Yes, on the wheels.
### Are there any user-facing changes?
No
* GitHub Issue: #45041
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
We're using only Docker level cache for vcpkg used for wheels. If we have any vcpkg related changes, all vcpkg ports are rebuilt. It's time consuming.
### What changes are included in this PR?
Enable NuGet + GitHub Packages based cache. It's port level cache. So we don't need to rebuild all ports when we have any vcpkg related changes.
See also: https://learn.microsoft.com/en-us/vcpkg/consume/binary-caching-github-packages
NuGet + GitHub Packages based cache isn't enabled with manylinux2014 + aarch64. Because EPEL for CentOS 7 + aarch64 provides old Mono. (FYI: EPEL for CentOS 7 + x86_64 provides newer Mono.) We can't use old Mono to run NuGet on Linux.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
* GitHub Issue: #43951
Lead-authored-by: Sutou Kouhei <kou@clear-code.com>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Co-authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
This is a follow-on to https://github.com/apache/arrow/pull/43566 which was merged earlier today. See https://github.com/apache/arrow/issues/43570 for more detail on the entire set of changes.
### What changes are included in this PR?
- Replace instances where we call or document calling `docker-compose` with `docker compose`.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
This PR includes the require user-facing changes to document things.
* GitHub Issue: #43570
Lead-authored-by: Bryce Mecum <petridish@gmail.com>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
Building free-threaded wheels is necessary to support the free-threaded build. We probably want to upload these wheels as nightlies somewhere as well, so that downstream users can test the free-threading-related changes.
### What changes are included in this PR?
- Add necessary configuration to build 3.13 free-threading wheels on `manylinux`.
- Do necessary changes to build free-threaded wheels on macOS as well.
### Are these changes tested?
I tested the `manylinux` wheel builds. macOS is still untested, since it's not dockerized.
### Are there any user-facing changes?
No.
Related to #43536.
* GitHub Issue: #43964
Lead-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
Co-authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Python 3.8 is End of Support on 31st October 2023. We can drop support for pyarrow 18.0.0.
See: https://endoflife.date/python
### What changes are included in this PR?
Remove support for 3.8. Update minimum required Python version and update some CI jobs to use the minimum Python supported version
### Are these changes tested?
They will be tested via Archery
### Are there any user-facing changes?
Yes in terms of not supporting older versions of Python. This will be a breaking change for some users but not for the API itself
* GitHub Issue: #43518
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
### Rationale for this change
S3 support is enabled when building PyArrow for macOS (both on PR builds and in Crossbow wheel builds), but minio wan't installed before testing, therefore S3 support in PyArrow was not tested at all.
### What changes are included in this PR?
Ensure Minio is installed before running PyArrow tests on macOS builds.
### Are these changes tested?
Yes, by construction.
### Are there any user-facing changes?
No.
* GitHub Issue: #44111
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
v2 and v3 are deprecated and can fail CI builds, bump to v4.
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
In https://github.com/apache/arrow/pull/43936 I noticed that core dumps were not written out for crashing C++ tests. One problem is that, by default, Ubuntu hosts pipe core dumps to `apport`, but it is not available inside containers. Another is that the `ulimit` must be set in the host, not in the container.
In addition, this PR restores automatic traceback generation when running C++ tests, on Linux and macOS jobs.
### Are these changes tested?
Manually by introducing a spurious segfault and running Docker containers.
### Are there any user-facing changes?
No.
Lead-authored-by: Antoine Pitrou <antoine@python.org>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
As discussed on the main issue is interesting for discoverability to have the wheels uploaded to the nightly channel.
### What changes are included in this PR?
Added macro to upload wheel to scientific python channel
### Are these changes tested?
Via archery
### Are there any user-facing changes?
No but nightly wheels will be available on scientific python channel
* GitHub Issue: #40216
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Jacob Wujciak-Jens <jacob@wujciak.de>
We can now use the GCS testbench even if we are testing a Python version that does not support it.
* GitHub Issue: #43883
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
The Docker client version on the ARM64 self-hosted runners is now recent enough, so we don't need to use `docker-compose` there anymore.
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Like #43519 mentionies, now that the first `rc` is out, it's probably time to add CI coverage for Python 3.13 (and also start building wheels).
### What changes are included in this PR?
I'm fairly new to the build/CI processes of the project, but I tried to follow the same template as #37901. I'll follow up afterwards with adding CI coverage for the free-threaded build as well.
* GitHub Issue: #43519
Lead-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
### Rationale for this change
`.env` is a shell script. We can use `VCPKG=... # comment` form in a shell script but our handmade `VCPKG` value in `.env` detection doesn't care about the case.
### What changes are included in this PR?
Use shell instead of handmade detection.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
* GitHub Issue: #43618
Lead-authored-by: Sutou Kouhei <kou@clear-code.com>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
The `docker-compose` utility is legacy, while the Docker CLI client now includes a `compose` subcommand that attempts to be compatible with docker-compose YAML files.
### What changes are included in this PR?
1. Call `docker compose` by default when `archery docker` is invoked, not `docker-compose`
2. Add an option to switch back to `docker-compose`, in case of Docker CLI client is too old and doesn't support some options (such as `--file`).
### Are these changes tested?
Yes, on CI.
### Are there any user-facing changes?
No.
* GitHub Issue: #43608
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
The newer version of LLVM on AlmaLinux 8 fails on the pyarrow.gandiva tests
### What changes are included in this PR?
Temporarily remove Gandiva on Python checks for AlmaLinux 8.
### Are these changes tested?
Via CI
### Are there any user-facing changes?
No
* GitHub Issue: #43059
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
vcpkg doesn't work with CMake 3.29.1.
See also: https://github.com/microsoft/vcpkg/issues/37968
### What changes are included in this PR?
Use CMake 3.29.0 temporary.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
* GitHub Issue: #41124
Lead-authored-by: Sutou Kouhei <kou@clear-code.com>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Co-authored-by: Jacob Wujciak-Jens <jacob@wujciak.de>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
We can't use multiple top-level `env:` in workflow. GH-40949 introduced a top-level `env:` by `macros.github_header()`. It broke workflows that already have top-level `env:`.
### What changes are included in this PR?
Omit top-level `env:` key and reuse the top-level `env:` key generated by `macros.github_header()` in workflows.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
* GitHub Issue: #41088
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
The `docker-compose` utility is progressively being removed from GHA-provided runners:
https://github.com/actions/runner-images/issues/9557
### What changes are included in this PR?
Use `docker` client CLI directly instead of `docker-compose` where possible.
### Are these changes tested?
Yes, this should fix the sporadic CI failures because of the above migration.
### Are there any user-facing changes?
No, except additional optional env var `ARCHERY_DEBUG`.
* GitHub Issue: #40954
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
This PR aims to upgrade to a new major version of Apache ORC: https://orc.apache.org/news/2024/03/08/ORC-2.0.0/
### What changes are included in this PR?
This PR upgrades ORC dependency from 1.9.2 to 2.0.0.
### Are these changes tested?
Pass the CIs.
### Are there any user-facing changes?
No.
* GitHub Issue: #40507
Lead-authored-by: Antoine Pitrou <antoine@python.org>
Co-authored-by: Gang Wu <ustcwg@gmail.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
If we use GitHub hosted M1 macOS runner instead of self-hosted M1 macOS runner, we don't need to maintain self-hosted M1 macOS runner.
### What changes are included in this PR?
Use GitHub hosted M1 macOS runner except `r-binary-packages`'s `r-packages` job. It uses macOS 11 but GitHub hosted M1 macOS runner is 13 or 14.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
* GitHub Issue: #40077
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
### What changes are included in this PR?
Enable Azure in linux and mac os wheel builds. Tried to copy GCS
Don't enable Azure for windows builds because windows builds where all failing. Failures were a combination of cmake version or `Could not find a package configuration file provided by "wil"`. I think it makes sense to come back to windows builds in another PR.
### Are these changes tested?
There is no new functionality to test.
### Are there any user-facing changes?
No
* Closes: #39352
Authored-by: Thomas Newton <thomas.w.newton@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
paultiq