### Rationale for this change
The docker base docker image which installs all the vcpkg dependencies to build Windows wheels was failing to commit the layer for gRPC due to size.
### What changes are included in this PR?
Two separate fixes that independently fix the issue:
- Patch vcpkg for Windows to remove unconditional `/Z7`
- Use D:/ drive instead of C:/ as it has bigger size for Docker storage
### Are these changes tested?
Yes via archery
### Are there any user-facing changes?
No
* GitHub Issue: #49676
Lead-authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Co-authored-by: Bryce Mecum <petridish@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
We started uploading nightly wheels to the scientific python channel on anaconda:
https://anaconda.org/scientific-python-nightly-wheels/pyarrow
Which are the documented nightlies:
https://arrow.apache.org/docs/dev/developers/python.html#installing-nightly-packages
We updated the documentation and have been deprecating with a long release cycle.
Uploading wheels to gemfury sometimes fail due to trying to upload the same version of the wheels.
### What changes are included in this PR?
Stop uploading nightly wheels to gemfury.
### Are these changes tested?
No
### Are there any user-facing changes?
Yes, nightly wheels will not be available on gemfury anymore.
* GitHub Issue: #43904
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
See https://github.com/apache/arrow/issues/45156. Refactors our previous Windows Docker CI images from an unofficial image to the official Microsoft one based on Windows Server 2022 and adds VS2022 BuildTools.
### What changes are included in this PR?
- New Windows base Dockerfiles
- Updated Dockerfiles which build on top of that base
- Updated Docker Compose services to use new images
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
* GitHub Issue: #45156
Lead-authored-by: Bryce Mecum <petridish@gmail.com>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Signed-off-by: Bryce Mecum <petridish@gmail.com>
### Rationale for this change
The upload wasn't failing with the empty, non existent dir.
### What changes are included in this PR?
- Fix the path
- Prevent the upload from failing silently when no wheels are found
### Are these changes tested?
crossbow
### Are there any user-facing changes?
no
* GitHub Issue: #45155
Lead-authored-by: Jacob Wujciak-Jens <jacob@wujciak.de>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Signed-off-by: Jacob Wujciak-Jens <jacob@wujciak.de>
### Rationale for this change
There's no blockers anymore for building Windows wheels for the free-threaded build, so this PR adds the necessary configuration to do that.
### What changes are included in this PR?
- Add jobs to build free-threaded wheels on Windows
- Move VCPKG-related stuff to a reusable base job
### Are these changes tested?
No tests required.
### Are there any user-facing changes?
No.
* GitHub Issue: #44421
Lead-authored-by: Antoine Pitrou <antoine@python.org>
Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
We need to ship the C++ standard library with our Windows wheels, as it is not guaranteed that a recent enough version is present on the system. However, some other Python libraries may require an even more recent version than the one we ship. This may incur crashes when PyArrow is imported before such other Python library, as the older version of the C++ standard library would be used by both.
### What changes are included in this PR?
Use a [fixed-up version](https://github.com/adang1345/delvewheel/pull/59) of delvewheel that allows us to name-mangle an individual DLL, and name-mangle `msvcp140.dll` to ensure that other Python libraries do not reuse the version we ship.
### Are these changes tested?
By regular wheel build tests.
* Closes: #44855
* GitHub Issue: #33981
* GitHub Issue: #44855
Lead-authored-by: Antoine Pitrou <antoine@python.org>
Co-authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
We're using only Docker level cache for vcpkg used for wheels. If we have any vcpkg related changes, all vcpkg ports are rebuilt. It's time consuming.
### What changes are included in this PR?
Enable NuGet + GitHub Packages based cache. It's port level cache. So we don't need to rebuild all ports when we have any vcpkg related changes.
See also: https://learn.microsoft.com/en-us/vcpkg/consume/binary-caching-github-packages
NuGet + GitHub Packages based cache isn't enabled with manylinux2014 + aarch64. Because EPEL for CentOS 7 + aarch64 provides old Mono. (FYI: EPEL for CentOS 7 + x86_64 provides newer Mono.) We can't use old Mono to run NuGet on Linux.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
* GitHub Issue: #43951
Lead-authored-by: Sutou Kouhei <kou@clear-code.com>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Co-authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
v2 and v3 are deprecated and can fail CI builds, bump to v4.
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
As discussed on the main issue is interesting for discoverability to have the wheels uploaded to the nightly channel.
### What changes are included in this PR?
Added macro to upload wheel to scientific python channel
### Are these changes tested?
Via archery
### Are there any user-facing changes?
No but nightly wheels will be available on scientific python channel
* GitHub Issue: #40216
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Jacob Wujciak-Jens <jacob@wujciak.de>
### Rationale for this change
The `docker-compose` utility is legacy, while the Docker CLI client now includes a `compose` subcommand that attempts to be compatible with docker-compose YAML files.
### What changes are included in this PR?
1. Call `docker compose` by default when `archery docker` is invoked, not `docker-compose`
2. Add an option to switch back to `docker-compose`, in case of Docker CLI client is too old and doesn't support some options (such as `--file`).
### Are these changes tested?
Yes, on CI.
### Are there any user-facing changes?
No.
* GitHub Issue: #43608
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
Old version of vcpkg is blocking https://github.com/apache/arrow/issues/39352
### What changes are included in this PR?
- Upgrade vcpkg
- Fix ports patches
- Upgrade visual studio used in windows wheel builds. VS2019 is now required for the vcpkg `abseil` build.
- Move `VCPKG_FORCE_SYSTEM_BINARIES` to be set before vcpkg install to fix vcpkg install on linux ARM.
- Fix for LLVM 17 which requires that an executable exports "llvm_orc_registerEHFrameSectionWrapper()" and "llvm_orc_unregisterEHFrameSectionWrapper()". This effects the java builds that depend on llvm from vcpkg for gandiva.
- Update image used for python wheel builds on windows to 2024-02-05
### Are these changes tested?
Does not change any behaviour so should be covered by existing tests
### Are there any user-facing changes?
There shouldn't be
* Closes: #39621
Lead-authored-by: Thomas Newton <thomas.w.newton@gmail.com>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Co-authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
It's useful for debugging on local.
### What changes are included in this PR?
Add `actions/upload-artifact` step like other jobs for manylinux and macOS.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
* Closes: #36450
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
# Overview
This pull request:
1. Removes hard-coded dependencies on "master" as the default branch name in the crossbow infrastructure and CI template files.
# Implementation
1. Removed comment/text references to "master" branch, including URLs to other repositories.
2. Modified `core.py` to add a new `default_branch` property and a new method `is_default_branch`, for checking whether on the default branch, to the `Target` class.
3. Modified CI template files to use the new `is_default_branch` function to check whether on the default branch.
# Testing
1. Using [lafiona/crossbow](https://github.com/lafiona/crossbow) as a queue repository for qualification.
2. Ran modified template jobs. All failures appeared to be unrelated to the changes.
3. The branch names for all relevant qualification jobs are prefixed with `build-34-*`.
4. Example of a passing job: [https://github.com/lafiona/crossbow/actions/runs/2920227769](https://github.com/lafiona/crossbow/actions/runs/2920227769)
5. Example of a failing job: [https://github.com/lafiona/crossbow/runs/7998190113](https://github.com/lafiona/crossbow/runs/7998190113) - in this example, the *"Push Docker Image"* workflow step is not included, since we are not on the default branch. The failure appears to be related to issues fetching R package resources and not related to the default branch checking logic. There were a variety of other kinds of failures, but none of them appear related to the default branch checking logic.
# Future Directions
1. Remove "master" from `default_branch` name property of `Target` class.
2. Remove all remaining uses of "master" terminology in crossbow.
3. [ARROW-17512](https://issues.apache.org/jira/browse/ARROW-17512): Address minor issues with crossbow documentation.
# Notes
1. Thank you to @lafiona for her help with this pull request!
2. Due to unexpected technical issues, we opened this pull request as a follow up to https://github.com/apache/arrow/pull/13750. Please see https://github.com/apache/arrow/pull/13750 for more discussion regarding qualification efforts.
Lead-authored-by: Kevin Gurney <kgurney@mathworks.com>
Co-authored-by: Fiona La <fionala7@gmail.com>
Signed-off-by: Alessandro Molina <amol@turbogears.org>
This also updates pinned vcpkg to use xsimd 8.1.0.
This also implements auto python-wheel-windows-vs2017 image update mechanism.
We have a problem of "docker build" on Windows. "docker build" doesn't reuse pulled
image as caches. "docker build" always rebuilds an image. This implements manual
reuse mechanism like the following:
if ! docker pull; then
docker build # build only when built images don't exist
fi
docker run
But this doesn't work when ci/docker/python-wheel-windows-vs2017.dockerfile is updated
but pinned vcpkg revision isn't changed. In the case, "docker build" isn't run because
"docker pull" is succeeded.
To work this mechanism, this introduces "PYTHON_WHEEL_WINDOWS_IMAGE_REVISION".
We must bump it manually when we update ci/docker/python-wheel-windows-vs2017.dockerfile.
"PYTHON_WHEEL_WINDOWS_IMAGE_REVISION" is used in tag name. So "docker pull"
is failed with new "PYTHON_WHEEL_WINDOWS_IMAGE_REVISION" and then "docker build"
is used.
Lead-authored-by: Sutou Kouhei <kou@clear-code.com>
Co-authored-by: Yibo Cai <yibo.cai@arm.com>
Co-authored-by: Wes McKinney <wesm@apache.org>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
Main highlights:
- prepared for unittesting
- introduced jinja macros to reduce code duplication for crossbow related snippets, like arrow checkout or release uploading
- the tasks.yml is jinja templated now
Closes#9913 from kszucs/crossbow-to-archery
Authored-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
Also resolves:
- ARROW-11213: [Packaging][Python] Dockerize wheel building on windows
- ARROW-11215: [CI] Use named volumes by default for caching in docker-compose
- ARROW-11231: [Python][Packaging] Enable mimalloc in wheels
Main features:
- dockerize windows builds
- use vcpkg as the dependency source where we can explicitly pin a working version without worrying about the drag of build environment over time
Potential follow-up:
- use vcpkg on macos as well
cc @kou @xhochy
## Manylinux testing
Should be straightforward (I'm going to shared pre-built images for quicker testing).
## Windows testing
Only windows host is able to run windows containers. I'm virtualizing windows on macOS (should work on linux as well) using virtualbox:
```bash
git clone https://github.com/StefanScherer/windows-docker-machine
cd windows-docker-machine
# grant more resources in the vagrantfile under the virtualbox section (I use 8 cores and 16GiB of ram)
vagrant up --provider virtualbox 2019-box
docker context ls
docker context use 2019-box
docker image ls
```
Now docker should use the windows docker daemon.
```bash
pip install -e dev/archery[docker]
# the volumes defined in the docker compose file should be commented out and pass arrow's source explicitly
# this is required because compose is being executed on a unix host and the directory is mounted through a virtual machine
PYTHON=3.6 archery docker run --no-pull --using-docker-cli -v C:$(pwd):C:/arrow python-wheel-windows-vs2017 cmd.exe
# then within the container execute a build or do other interactive things
arrow\ci\scripts\python_wheel_windown_build.bat
```
Closes#9096 from kszucs/wheel-vcpkg
Lead-authored-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
Co-authored-by: Sutou Kouhei <kou@clear-code.com>
Co-authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
