37 Commits
Author SHA1 Message Date
paultiq 120b0b760d GH-50899: [CI][C++] Use OIDC instead of ACCESS_KEY and SECRET_KEY for sccache's s3 creds (#51028)
### Rationale for this change
This PR moves crossbow builds to short-lived S3 credentials. It does this via GitHub's OIDC provider and an AWS ROLE_ARN with a trust relationship configured. This configuration is detailed here: 
- [Use IAM roles to connect GitHub Actions to actions in AWS](https://aws.amazon.com/blogs/security/use-iam-roles-to-connect-github-actions-to-actions-in-aws/)

The benefits of short-lived / temporary credentials discussed further here: [AWS Well-architected Framework Best Practices SEC02-BP02 Use temporary credentials](https://docs.aws.amazon.com/wellarchitected/latest/framework/sec_identities_unique.html)

sccache supports a variety of S3 creds: [S3 credentials](https://github.com/mozilla/sccache/blob/main/docs/S3.md#credentials)

### What changes are included in this PR?

Use [AssumeRoleWithWebIdentity](https://github.com/mozilla/sccache/blob/main/docs/S3.md#credentials) for sccache S3 credentials.

By assuming the AWS_ROLE_ARN, the sccache session relies on credentials that expire\*, issued only to workflows in the trusted repository, and scoped to the sccache bucket.

Specific tasks: 
- create a role in IAM with the bucket and object level permissions, 
- trust the github oidc-provider, 
- define AWS_ROLE_ARN and set id-token permission
- grant sts:AssumeRoleWithWebIdentity for the specific repositories or organization. 
- pass AWS_SESSION_TOKEN
- add a `aws-actions/configure-aws-credentials@ v6` step to the workflow. 

Separately, someone will need to add the role and trust to AWS:
- Procedure is in https://aws.amazon.com/blogs/security/use-iam-roles-to-connect-github-actions-to-actions-in-aws/ 
- Set a "Maximum session duration" to maximum runtime of any job... maybe 4 hours? 8 hours? 
- Permissions: Role needs BucketLevel and ObjectLevels permissions to the bucket. 

\* I can share an example if needed

### Are these changes tested?
Yes, locally and on a fork. 

### Are there any user-facing changes?
No
* GitHub Issue: #50899

Authored-by: paul@iqmo.com <paul@iqmo.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
2026-09-06 16:04:43 +09:00
Antoine Pitrou bb765833bf GH-50991: [CI] Enable sccache just like ccache on Docker-based builds (#50992)
### Rationale for this change

Enable sccache in more Docker-based builds on Crossbow CI, so as to make jobs faster and consume less resources.

### Are these changes tested?

Yes, by existing Crossbow CI runs.

Example results:

| Crossbow build | before | after |
| - | - | - |
| test-conda-python-3.11-dask-latest | [21 min](https://github.com/ursacomputing/crossbow/actions/runs/32845958261/job/97795563834#logs) | [12 min](https://github.com/ursacomputing/crossbow/actions/runs/32869077595/job/97929158178#logs) |
| test-debian-c-glib | [25 min](https://github.com/ursacomputing/crossbow/actions/runs/32845947896/job/97795530901#logs) | [8 min](https://github.com/ursacomputing/crossbow/actions/runs/32884848700/job/97925726050#logs) |
| test-fedora-python | [29 min](https://github.com/ursacomputing/crossbow/actions/runs/32845950296/job/97795538646#logs) | [13 min](https://github.com/ursacomputing/crossbow/actions/runs/32869078556/job/97881532720#logs) |
| test-ubuntu-ruby | [24 min](https://github.com/ursacomputing/crossbow/actions/runs/32845958839/job/97795565180#logs) | [8 min](https://github.com/ursacomputing/crossbow/actions/runs/32882612209/job/97915489163#logs) |
| wheel-manylinux-2-28-cp314-cp314-arm64 | [20 min](https://github.com/ursacomputing/crossbow/actions/runs/32869224975/job/97871946857#logs) | [14 min](https://github.com/ursacomputing/crossbow/actions/runs/32876686036/job/97896176720#logs) |

### Are there any user-facing changes?

No.

* GitHub Issue: #50991

Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
2026-08-26 11:20:55 +02:00
paultiqandpaul@iqmo.com ba5ca205d1 GH-48172: [Python] Add cp315 to build (#48191)
### Rationale for this change
Add [Python 3.15](https://docs.python.org/3.15/whatsnew/3.15.html) to the pyarrow nightly builds, so that issues can be detected early in the development cycles of both cpython & pyarrow. 

Note: 
- ~~CFFI and~~ Pandas doesn't have nightly 3.15 wheels, and are accordingly excluded. 
- Numpy shipped 3.15rc1 in 2.5.2 ~~doesn't ship a 3.15 wheel, but is needed to build. It is building from source.~~
- [3.15 Schedule](https://peps.python.org/pep-0790/) This PR targets 3.15.0rc1. rc2 is expected 2026-09-01 and final 2026-10-01. 

### What changes are included in this PR?
- Adding cp315/cp315t to the build matrix
- Excluding ~~cffi/~~ pandas for 3.15, as the wheels aren't yet available.

### 2026-08-13 edit
- Bump libcst to 1.9.0
- Update manylinux image to 2026.08.08-1 (for 3.15rc1)
- Add -rc to PYTHON_IMAGE_TAG
- Disable pandas tests if Pandas isn't installed: 

### Are these changes tested?
- Yes, tested on a local Linux build using the `ninja-release-python` preset (not maximal)

### Are there any user-facing changes?
No functional changes, other than allowing the user to use Python 3.15. 
* GitHub Issue: #48172

### Progress Notes (2026-08-15)
Each platform uses different sources for Python builds, each with different release schedules and paths:

- [manylinux](https://github.com/pypa/manylinux) is prebuilt with latest Python builds in `/opt/python`, so using the latest manylinux build is sufficient. 3.15 wasn't available until beta 1. 
- musllinux: Uses Alpine system python built from https://github.com/docker-library/python/pull/1123. Might need to bump ALPINE_LINUX version in [arrow/dev/tasks/python-wheels/github.linux.yml](https://github.com/apache/arrow/blob/e611f48081ec927e18e5bcb72fbd071c15c76b08/dev/tasks/python-wheels/github.linux.yml#L35). 
- musllinux free-threaded: uses [astral's builds](https://github.com/astral-sh/python-build-standalone). Edit `python-free-threaded-wheel-musllinux-test-imports.dockerfile` + `-test-unittests.dockerfile`: set the python_patch_version, and set build_date in `compose.yaml`. 
- macos: Downloads official build from python.org, see `ci/scripts/install_python.sh`. For early builds, needed to disable `--only-binary` since at least one dependency needed to be built (no 3.15 wheels available yet)... only macos uses --only-binary
- Windows: installs builds using `choco`, see `python-wheel-windows-test-vs2022.dockerfile` and `python-free-threaded-wheel-windows-test-vs2022.dockerfile`

For 3.15.0rc1 builds, some care was needed because certain paths needed 3.15.0 and others needed the full 3.15.0rc1. 

Lead-authored-by: Paul Timmins <paul@iqmo.com>
Co-authored-by: paul@iqmo.com <paul@iqmo.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
2026-08-21 14:18:33 +02:00
Raúl Cumplido 934fa870a7 GH-50410: [Python][Packaging] Drop Python 3.10 support (#50411)
### Rationale for this change

Python 3.10 is End of Support on October 2026. We can drop support on PyArrow 26.0.0.
See: https://endoflife.date/python

### What changes are included in this PR?

Remove support for 3.10. Update minimum required Python version and update some CI jobs to use the minimum Python supported version.
Some CI jobs using default Python 3.10 (Ubuntu 20.04) have been bumped.

### Are these changes tested?

They will be tested via CI and Archery

### Are there any user-facing changes?

Yes in terms of not supporting older versions of Python. This will be a breaking change for some users but not for the API itself

* GitHub Issue: #50410

Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
2026-07-09 12:33:32 +02:00
Antoine Pitrou 1c15d136b4 GH-32994: [Dev][Archery] Fix multi-arch Docker configuration (#50125)
### Rationale for this change

Our Archery docker setup had support for running foreign-arch containers (for example a ARM64 container on a x86-64 native host, using emulation), but it failed working at some point.

### Are these changes tested?

By existing CI runs, as well as locally (by running a couple ARM64 containers on a x86-64 native host).

### Are there any user-facing changes?

No.

* GitHub Issue: #32994

Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
2026-06-10 14:37:45 +02:00
Raúl Cumplido 2009cc9196 GH-48807: [CI] Clean up space on GitHub runner to fix manylinux wheel failure (#48790)
### Rationale for this change

The `wheel-manylinux-2-28-cp312-cp312-amd64` performs the `almalinux-8` wheels verification and it fails with out of space: `At least 746MB more space needed on the / filesystem.`

### What changes are included in this PR?

Use our existing `github_free_space` macro to clean space on the runner.

### Are these changes tested?

Yes, via archery. I've validated the failure happened on the PR without the change and was solved with the clean up.

### Are there any user-facing changes?

No

* GitHub Issue: #48807

Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
2026-01-10 07:14:10 +09:00
Alenka Frim e90bacded2 GH-48280: [CI] PYTHON_PATCH_VERSION docker warnings (#48282)
### Rationale for this change

Warnings in the docker compose step, see https://github.com/ursacomputing/crossbow/actions/runs/19167940422/job/54793047049#step:7:55

### What changes are included in this PR?

Instead of adding `PYTHON_PATCH_VERSION` to `.env` I have moved the use of the patch version to the free-threaded musllinux jobs as `PYTHON_PATCH_VERSION` was only needed there.

I have also updated all Python patch versions from `3.13.7` to `3.13.9`.

### Are these changes tested?

Yes, CI and extended python wheels builds should pass.

### Are there any user-facing changes?

No.

* GitHub Issue: #48280

Authored-by: AlenkaF <frim.alenka@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
2025-12-04 07:07:17 +09:00
Alenka Frim f268c43ccd GH-47438: [Python][Packaging] Set up wheel building for Python 3.14 (#47616)
### Rationale for this change

Python 3.14 is currently in a prerelease status and is expected to have a final release in October this year (https://peps.python.org/pep-0745/).

We should ensure we are fully ready to support Python 3.14 for the PyArrow 22 release.

### What changes are included in this PR?

This PR  updates wheels for Python 3.14.

### Are these changes tested?

Tested in the CI and with extended builds.

### Are there any user-facing changes?

No, but users will be able to use PyArrow with Python 3.14.

* GitHub Issue: #47438

---

Todo:

- Update the image revision name in `.env`
- Add 3.14 conda build ([arrow/dev/tasks/tasks.yml](https://github.com/apache/arrow/blob/d803afcc43f5d132506318fd9e162d33b2c3d4cd/dev/tasks/tasks.yml#L809)) when https://github.com/conda-forge/pyarrow-feedstock/pull/156 is merged 

Follow-ups:

- https://github.com/apache/arrow/issues/47437

Authored-by: AlenkaF <frim.alenka@gmail.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
2025-10-14 06:17:08 +02:00
Raúl Cumplido 771e14b9dd GH-47643: [Python][Packaging] Enable CMAKE_INTERPROCEDURAL_OPTIMIZATION for wheels (#47733)
### Rationale for this change

Experimentation in https://github.com/apache/arrow/pull/47637 suggested that [enabling inter-procedural optimization (IPO)](https://cmake.org/cmake/help/latest/variable/CMAKE_INTERPROCEDURAL_OPTIMIZATION.html) could not only improve performance in some workloads, but also reduce the code size of Arrow libraries.

There's around 2% size reduction on the tested examples:

|wheel|size|
|---------|--------|
|last successful nightly `wheel-manylinux-2-28-cp312-cp312-arm64` | 44605918 bytes|
|with change `wheel-manylinux-2-28-cp312-cp312-arm64` | 43807946 bytes|
|last successful nightly `wheel-windows-cp311-cp311-amd64` | 28093080 bytes|
|with change `wheel-windows-cp311-cp311-amd64` | 27391407 bytes|

### What changes are included in this PR?

Add `-DCMAKE_INTERPROCEDURAL_OPTIMIZATION=ON`  to Arrow C++ and wheels for both manylinux and windows. 

### Are these changes tested?

Yes via CI

### Are there any user-facing changes?

No

* GitHub Issue: #47643

Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
2025-10-13 09:14:41 +02:00
Raúl Cumplido cde3f6a1e2 GH-47614: [CI] Upgrade vcpkg on our CI (#47627)
### Rationale for this change

Our vcpkg version used for the wheels is a really old version. Some wheels like the macOS ones have started failing due to really old dependencies using unsupported cake versions.

### What changes are included in this PR?

- Update the vcpkg version to the latest released
- Update minio version to solve failure due to newer aws-sdk
- Remove unnecessary vcpkg patches
- Add patch to ORC vcpkg to solve issue with exception not being propagated on std::run_once on musllinux.
- Avoid adding some flight sources on Windows unity builds that cause windows headers clashing.

### Are these changes tested?

Yes, via archery on CI.

### Are there any user-facing changes?

No

* GitHub Issue: #43416
* GitHub Issue: #47614

Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
2025-10-03 09:45:17 +02:00
Raúl CumplidoandSutou Kouhei aedb752762 GH-47443: [Python][Packaging] Drop Python 3.9 support (#47478)
### Rationale for this change

Python 3.9 is End of Support on 31st October 2025. We can drop support for pyarrow 22.0.0.
See: https://endoflife.date/python

### What changes are included in this PR?

Remove support for 3.9. Update minimum required Python version and update some CI jobs to use the minimum Python supported version

### Are these changes tested?

They will be tested via CI and Archery

### Are there any user-facing changes?

Yes in terms of not supporting older versions of Python. This will be a breaking change for some users but not for the API itself
* GitHub Issue: #47443

Lead-authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
2025-09-09 09:44:55 +02:00
Raúl Cumplido d80e6fff05 GH-43904: [CI][Python] Stop uploading nightly wheels to gemfury (#47470)
### Rationale for this change

We started uploading nightly wheels to the scientific python channel on anaconda:
https://anaconda.org/scientific-python-nightly-wheels/pyarrow
Which are the documented nightlies:
https://arrow.apache.org/docs/dev/developers/python.html#installing-nightly-packages
We updated the documentation and have been deprecating with a long release cycle.

Uploading wheels to gemfury sometimes fail due to trying to upload the same version of the wheels.

### What changes are included in this PR?

Stop uploading nightly wheels to gemfury.

### Are these changes tested?

No

### Are there any user-facing changes?

Yes, nightly wheels will not be available on gemfury anymore.

* GitHub Issue: #43904

Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
2025-09-03 09:59:04 +02:00
f3fc297560 GH-46959: [Python][Packaging] Drop support for manylinux2014 (#46965)
### Rationale for this change

We can cover all supported platforms by manylinux_2_28. Amazon Linux 2 needs manylinux2014 and it's still maintained (it'll reach EOL on 2026-06-30) but we dropped support for Amazon Linux 2. See also: #36282

### What changes are included in this PR?

Remove manylinux2014 related codes.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

Yes.
* GitHub Issue: #46959

Lead-authored-by: Sutou Kouhei <kou@clear-code.com>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
2025-07-03 05:29:14 +09:00
Nuno Veloso 6fee049c81 GH-18036: [Packaging] Build Python wheel for musllinux (#45470)
### Rationale for this change
Please check #18036.

### What changes are included in this PR?

Almost everything needed for building and testing Python wheels for musllinux.
The side-effect of these changes are the exclusion of 2 unit tests when running on musl based systems.
The skipped unit tests are:
- test_uwsgi_integration
- test_print_stats

### Are these changes tested?

I was able to successfully generate a musllinux wheel by running the following:
 
```
docker-compose build python-wheel-musllinux-1-2
docker-compose run python-wheel-musllinux-1-2
```

I was also able to run the following tests with no errors:

```
python-wheel-musllinux-test-imports
python-free-threaded-wheel-musllinux-test-imports
python-wheel-musllinux-test-unittests
python-free-threaded-wheel-musllinux-test-unittests
```

### Are there any user-facing changes?

Yes.
* GitHub Issue: #18036

Authored-by: Nuno Veloso <nveloso21@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
2025-03-27 10:22:20 +09:00
Joris Van den Bossche ec3d2839d1 GH-43519: [Python][CI] Update Python 3.13 rc to final 3.13.2 (#44375)
### Rationale for this change

The final Python 3.13.0 is out now, so we can update those versions

* GitHub Issue: #43519

Authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
2025-02-17 16:05:37 +01:00
Sutou Kouhei c8a7defecb GH-45482: [CI][Python] Don't use Ubuntu 20.04 for wheel test (#45483)
### Rationale for this change

Ubuntu 20.04 will reach EOL on 2025-05.

### What changes are included in this PR?

Remove a wheel test that uses Ubuntu 20.04.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

No.
* GitHub Issue: #45482

Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
2025-02-11 07:10:51 +09:00
Sutou Kouhei ba79a481c6 GH-45307: [CI] Use GitHub hosted arm runner (#45308)
### Rationale for this change

GitHub hosted arm runner is available:
https://github.blog/changelog/2025-01-16-linux-arm64-hosted-runners-now-available-for-free-in-public-repositories-public-preview/

### What changes are included in this PR?

Use `ubuntu-24.04-arm` instead of self-hosted runner.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

No.
* GitHub Issue: #45307

Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
2025-01-21 11:16:44 +01:00
Sutou Kouhei 02a165922e GH-45076: [CI][Packaging][Python] Simplify dev/tasks/python-wheels/github.linux.yml (#45077)
### Rationale for this change

We can use GitHub Actions' standard features not Jinja2 features to generate test image names.

### What changes are included in this PR?

Use `GITHUB_ENV` instead of Jinja2.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

No.

* GitHub Issue: https://github.com/apache/arrow/issues/45076

Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
2024-12-20 12:16:51 +01:00
Antoine Pitrou 2533a9e143 GH-45047: [CI][Python][Packaging] Test 3.12 wheels on Ubuntu 24.04 (#45042)
### Rationale for this change

We already test some Python wheels on Ubuntu 20.04 or 22.04, depending on the system Python version. Also do so with Ubuntu 24.04.

### Are these changes tested?

Yes, by definition.

### Are there any user-facing changes?

No.
* GitHub Issue: #45047

Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
2024-12-18 06:44:01 +09:00
Antoine Pitrou 8f336c4478 GH-45039: [CI][Packaging][Python] Fix Docker push step for free-threaded wheel builds (#45040)
* GitHub Issue: #45039

Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
2024-12-17 18:22:17 +01:00
df40f7ad46 GH-43951: [CI][Python] Use GitHub Packages for vcpkg cache (#44644)
### Rationale for this change

We're using only Docker level cache for vcpkg used for wheels. If we have any vcpkg related changes, all vcpkg ports are rebuilt. It's time consuming.

### What changes are included in this PR?

Enable NuGet + GitHub Packages based cache. It's port level cache. So we don't need to rebuild all ports when we have any vcpkg related changes.

See also: https://learn.microsoft.com/en-us/vcpkg/consume/binary-caching-github-packages

NuGet + GitHub Packages based cache isn't enabled with manylinux2014 + aarch64. Because EPEL for CentOS 7 + aarch64 provides old Mono. (FYI: EPEL for CentOS 7 + x86_64 provides newer Mono.) We can't use old Mono to run NuGet on Linux.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

No.
* GitHub Issue: #43951

Lead-authored-by: Sutou Kouhei <kou@clear-code.com>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Co-authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
2024-11-15 18:41:50 +09:00
Bryce MecumandSutou Kouhei 736d7067a0 GH-43570: [CI][Dev][Docs] Update references to "docker-compose" with "docker compose" (#43575)
### Rationale for this change

This is a follow-on to https://github.com/apache/arrow/pull/43566 which was merged earlier today. See https://github.com/apache/arrow/issues/43570 for more detail on the entire set of changes.

### What changes are included in this PR?

- Replace instances where we call or document calling `docker-compose` with `docker compose`.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

This PR includes the require user-facing changes to document things.
* GitHub Issue: #43570

Lead-authored-by: Bryce Mecum <petridish@gmail.com>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
2024-11-15 18:14:09 +09:00
Lysandros NikolaouandAntoine Pitrou 1c1f7f359d GH-43964: [Python] Build macOS and manylinux wheels for free-threading (#43965)
### Rationale for this change

Building free-threaded wheels is necessary to support the free-threaded build. We probably want to upload these wheels as nightlies somewhere as well, so that downstream users can test the free-threading-related changes.

### What changes are included in this PR?

- Add necessary configuration to build 3.13 free-threading wheels on `manylinux`.
- Do necessary changes to build free-threaded wheels on macOS as well.

### Are these changes tested?

I tested the `manylinux` wheel builds. macOS is still untested, since it's not dockerized.

### Are there any user-facing changes?

No.

Related to #43536.
* GitHub Issue: #43964

Lead-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
Co-authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
2024-09-18 11:07:13 +02:00
Raúl Cumplido aa6ab95a12 GH-43518: [Python][Packaging][CI] Drop Python 3.8 support (#43970)
### Rationale for this change

Python 3.8 is End of Support on 31st October 2023. We can drop support for pyarrow 18.0.0.
See: https://endoflife.date/python

### What changes are included in this PR?

Remove support for 3.8. Update minimum required Python version and update some CI jobs to use the minimum Python supported version

### Are these changes tested?

They will be tested via Archery

### Are there any user-facing changes?

Yes in terms of not supporting older versions of Python. This will be a breaking change for some users but not for the API itself

* GitHub Issue: #43518

Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
2024-09-16 15:38:38 +02:00
Antoine Pitrou 15158bd20d MINOR: [CI] Bump actions/{download,upload}-artifact version (#44086)
v2 and v3 are deprecated and can fail CI builds, bump to v4.

Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
2024-09-12 17:43:46 +02:00
e710b6eb37 MINOR: [CI][C++] Enable core dumps and stack traces in Linux/macOS jobs (#43937)
### Rationale for this change

In https://github.com/apache/arrow/pull/43936 I noticed that core dumps were not written out for crashing C++ tests. One problem is that, by default, Ubuntu hosts pipe core dumps to `apport`, but it is not available inside containers. Another is that the `ulimit` must be set in the host, not in the container.

In addition, this PR restores automatic traceback generation when running C++ tests, on Linux and macOS jobs.

### Are these changes tested?

Manually by introducing a spurious segfault and running Docker containers.

### Are there any user-facing changes?

No.

Lead-authored-by: Antoine Pitrou <antoine@python.org>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
2024-09-11 09:19:59 +09:00
Raúl Cumplido 3a6135b66d GH-40216: [CI][Packaging][Python] Upload pyarrow nightly wheels to scientific python channel on Anaconda (#43862)
### Rationale for this change

As discussed on the main issue is interesting for discoverability to have the wheels uploaded to the nightly channel.

### What changes are included in this PR?

Added macro to upload wheel to scientific python channel

### Are these changes tested?

Via archery

### Are there any user-facing changes?

No but nightly wheels will be available on scientific python channel
* GitHub Issue: #40216

Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Jacob Wujciak-Jens <jacob@wujciak.de>
2024-09-02 23:37:03 +01:00
Antoine Pitrou ce1e724d7e MINOR: [CI] Use docker compose on self-hosted ARM builds (#43844)
### Rationale for this change

The Docker client version on the ARM64 self-hosted runners is now recent enough, so we don't need to use `docker-compose` there anymore.

Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
2024-08-27 19:17:55 +02:00
Lysandros NikolaouandJoris Van den Bossche 3e9384bbf4 GH-43519: [Python] Set up wheel building for Python 3.13 (#43539)
### Rationale for this change

Like #43519 mentionies, now that the first `rc` is out, it's probably time to add CI coverage for Python 3.13 (and also start building wheels).

### What changes are included in this PR?

I'm fairly new to the build/CI processes of the project, but I tried to follow the same template as #37901. I'll follow up afterwards with adding CI coverage for the free-threaded build as well.
* GitHub Issue: #43519

Lead-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
2024-08-22 15:27:40 +02:00
Antoine Pitrou e54ad41ee5 GH-43608: [CI][Archery] Prefer docker compose over docker-compose (#43586)
### Rationale for this change

The `docker-compose` utility is legacy, while the Docker CLI client now includes a `compose` subcommand that attempts to be compatible with docker-compose YAML files.

### What changes are included in this PR?

1. Call `docker compose` by default when `archery docker` is invoked, not `docker-compose`
2. Add an option to switch back to `docker-compose`, in case of Docker CLI client is too old and doesn't support some options (such as `--file`).

### Are these changes tested?

Yes, on CI.

### Are there any user-facing changes?

No.
* GitHub Issue: #43608

Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
2024-08-08 14:56:52 +09:00
Raúl Cumplido 76de7c2774 GH-43059: [CI][Gandiva] Disable Python Gandiva tests on AlmaLinux 8 (#43093)
### Rationale for this change

The newer version of LLVM on AlmaLinux 8 fails on the pyarrow.gandiva tests

### What changes are included in this PR?

Temporarily remove Gandiva on Python checks for AlmaLinux 8.

### Are these changes tested?

Via CI
### Are there any user-facing changes?

No
* GitHub Issue: #43059

Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
2024-07-02 10:04:28 +09:00
Antoine Pitrou be3b78902f GH-40954: [CI] Fix use of obsolete docker-compose command on Github Actions (#40949)
### Rationale for this change

The `docker-compose` utility is progressively being removed from GHA-provided runners:
https://github.com/actions/runner-images/issues/9557

### What changes are included in this PR?

Use `docker` client CLI directly instead of `docker-compose` where possible.

### Are these changes tested?

Yes, this should fix the sporadic CI failures because of the above migration.

### Are there any user-facing changes?

No, except additional optional env var `ARCHERY_DEBUG`.
* GitHub Issue: #40954

Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
2024-04-03 10:33:31 +02:00
Simon PerkinsandSutou Kouhei 42d42b1194 GH-33804: [Python] Add support for manylinux_2_28 wheel (#34818)
Closes #33804 

### Rationale for this change

At some point, it would be useful to support the new C++ ABI `_GLIBCXX_USE_CXX11_ABI=1` in pyarrow wheels, especially when moving to C++17:

- https://github.com/apache/arrow/issues/32415

I wanted to create a pyarrow wheel that supported the above ABI and adapted the existing CENTOS 7 manylinux2014 Dockerfile/wheel to produce a AlmaLinux 8 manylinux_2_28 Dockerfile/wheel.

Publishing wheels with a new ABI needs [careful consideration](https://pypackaging-native.github.io/key-issues/native-dependencies/cpp_deps/) so I think this is low priority, but I thought I'd provide this manylinux_2_28 implementation in case it was useful for current/future adoption. 

### What changes are included in this PR?

A manylinux_2_28 Dockerfile, adopted from the existing manylinux2014 Dockerfile

### Are these changes tested?

Manually tested at present

### Are there any user-facing changes?

Yes, there's a major ABI change, as pyarrow will be compiled with `_GLIBCXX_USE_CXX11_ABI=1`
* Closes: #33804

Supercedes:
* https://github.com/apache/arrow/pull/33805
* Closes: #33804

Lead-authored-by: Simon Perkins <simon.perkins@gmail.com>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
2023-05-04 06:38:17 +09:00
eitsupi 34addbb5e1 GH-34812: [Packaging][Python] Use self-hosted arm64 Linux runner instead of Travis CI for Linux arm64 wheels (#34835)
### Rationale for this change

Change one of the jobs that uses Travis CI to run on GitHub Actions self-hosted runner, reducing the number of Travis CI jobs.

### What changes are included in this PR?

Changes to use self-hosted arm64 linux runner for Crossbow jobs building wheels.

### Are these changes tested?

Yes

### Are there any user-facing changes?

No
* Closes: #34812

Authored-by: SHIMA Tatsuya <ts1s1andn@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
2023-04-03 16:01:58 +09:00
Krisztián SzűcsandSutou Kouhei f7b143959e ARROW-10349: [Python] Build and publish aarch64 wheels
This should work in theory, I'm building the image to see how well does vcpkg support arm packages.

```bash
ARCH=arm64v8 archery docker build python-wheel-manylinux-2014
```

Closes #9285 from kszucs/manylinux-arm64

Lead-authored-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
Co-authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
2021-03-15 11:52:48 +01:00
Sutou Kouhei 6a2c135a93 ARROW-11275: [Packaging][wheel][Linux] Fix paths for Gemfury
Closes #9220 from kou/packaging-wheel-linux-fix-path

Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
2021-01-16 22:27:21 +01:00
ddd9c6dd4e ARROW-11212: [Packaging][Python] Use vcpkg as dependency source for manylinux and windows wheels
Also resolves:
- ARROW-11213: [Packaging][Python] Dockerize wheel building on windows
- ARROW-11215: [CI] Use named volumes by default for caching in docker-compose
- ARROW-11231: [Python][Packaging] Enable mimalloc in wheels

Main features:
- dockerize windows builds
- use vcpkg as the dependency source where we can explicitly pin a working version without worrying about the drag of build environment over time

Potential follow-up:
- use vcpkg on macos as well

cc @kou @xhochy

## Manylinux testing

Should be straightforward (I'm going to shared pre-built images for quicker testing).

## Windows testing

Only windows host is able to run windows containers. I'm virtualizing windows on macOS (should work on linux as well) using virtualbox:

```bash
git clone https://github.com/StefanScherer/windows-docker-machine
cd windows-docker-machine
# grant more resources in the vagrantfile under the virtualbox section (I use 8 cores and 16GiB of ram)
vagrant up --provider virtualbox 2019-box
docker context ls
docker context use 2019-box
docker image ls
```

Now docker should use the windows docker daemon.

```bash
pip install -e dev/archery[docker]
# the volumes defined in the docker compose file should be commented out and pass arrow's source explicitly
# this is required because compose is being executed on a unix host and the directory is mounted through a virtual machine
PYTHON=3.6 archery docker run --no-pull --using-docker-cli -v C:$(pwd):C:/arrow python-wheel-windows-vs2017 cmd.exe
# then within the container execute a build or do other interactive things
arrow\ci\scripts\python_wheel_windown_build.bat
```

Closes #9096 from kszucs/wheel-vcpkg

Lead-authored-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
Co-authored-by: Sutou Kouhei <kou@clear-code.com>
Co-authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
2021-01-15 08:47:17 +09:00