### Rationale for this change
Trying to reduce the size of libarrow.so and remove google-cloud-cpp SDK on some builds. Allow for users to plug and play based on requirements and divide our functionality into cleaner modules.
### What changes are included in this PR?
Unconditionally build GCS and the google-cloud-cpp SDK into a different module libarrow_gcs.so outside of libarrow.so.
Update bindings to link against the new libarrow_gcs.so library.
Update the Linux Package jobs to have the new module into a different package.
### Are these changes tested?
Yes via CI
### Are there any user-facing changes?
Yes, users will need to either link against libarrow_s3.so or register using LoadFileSystemFactories
### Was AI used for this PR?
In accordance to the [AI generation guidelines](https://arrow.apache.org/docs/dev/developers/overview.html#ai-generated-code), please disclose below whether and how AI was used in this PR.
**PR code and description written by:**
- [x] Human
- [x] AI
**Reviewed before submission by:**
- [x] Human
- [ ] AI
- [ ] Not reviewed
* GitHub Issue: #54902
Authored-by: Raúl Cumplido <raulcd@apache.org>
Signed-off-by: Raúl Cumplido <raulcd@apache.org>
### Rationale for this change
`debuild` doesn't add OpenTelemetry dependencies to `libarrow.so.XXX` automatically because OpenTelemetry package on Debian doesn't have SO version.
### What changes are included in this PR?
Add `opentelemetry-cpp` dependency explicitly.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
### Was AI used for this PR?
In accordance to the [AI generation guidelines](https://arrow.apache.org/docs/dev/developers/overview.html#ai-generated-code), please disclose below whether and how AI was used in this PR.
**PR code and description written by:**
- [x] Human
- [ ] AI
**Reviewed before submission by:**
- [x] Human
- [x] AI
- [ ] Not reviewed
* GitHub Issue: #55112
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Raúl Cumplido <raulcd@apache.org>
### Rationale for this change
Linux packaging jobs for Deb packages are currently failing due to wrong libarrow_s32600 still being referred to.
### What changes are included in this PR?
Update regex to cover numbers on libarrowxxx package names.
Update the wrong entries at control.in.
### Are these changes tested?
I have tested the regex locally with:
```
$ sed -i.bak -E -e "s/(lib(arrow|gandiva|parquet)[-a-z0-9]*)2700/\\12800/g" dev/tasks/linux-packages/apache-arrow/debian/control.in
```
And validated the diff contains libarrow s3 updated:
```diff
@@ -122,13 +122,13 @@ Pre-Depends: ${misc:Pre-Depends}
Depends:
${misc:Depends},
${shlibs:Depends},
- libarrow-acero2700 (= ${binary:Version}),
- libparquet2700 (= ${binary:Version})
+ libarrow-acero2800 (= ${binary:Version}),
+ libparquet2800 (= ${binary:Version})
Description: Apache Arrow is a data processing library for analysis
.
This package provides C++ library files for Dataset module.
-Package: libarrow-s3-2700
+Package: libarrow-s3-2800
Section: libs
Architecture: any
Multi-Arch: same
```
### Are there any user-facing changes?
No
### Was AI used for this PR?
In accordance to the [AI generation guidelines](https://arrow.apache.org/docs/dev/developers/overview.html#ai-generated-code), please disclose below whether and how AI was used in this PR.
**PR code and description written by:**
- [x] Human
- [ ] AI
**Reviewed before submission by:**
- [x] Human
- [ ] AI
- [ ] Not reviewed
* GitHub Issue: #54595
Authored-by: Raúl Cumplido <raulcd@apache.org>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
`curl --version` uses the `${MAJOR}.${MINOR}.${PATCH}-rc${RC}` format for RC curl and Debian GNU/Linux sid uses RC curl.
### What changes are included in this PR?
Accept the `-rc${RC}` part and ignore it.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
### Was AI used for this PR?
In accordance to the [AI generation guidelines](https://arrow.apache.org/docs/dev/developers/overview.html#ai-generated-code), please disclose below whether and how AI was used in this PR.
**PR code and description written by:**
- [x] Human
- [ ] AI
**Reviewed before submission by:**
- [x] Human
- [ ] AI
- [ ] Not reviewed
* GitHub Issue: #51817
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Fix#51751
homebrew-core's formulae got letter-folders in 2023 but our macro from commit 5ad5ddc kept copying to top-level `Formula/` hence homebrew-cpp job tested homebrew-core's formulae instead of both apache-arrow.rb and apache-arrow-glib.rb in `dev/tasks/homebrew-formulae/`
### What changes are included in this PR?
Copy to `Formula/a/` and
change to `formula_opt_prefix("llvm")` per audit check
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
### Was AI used for this PR?
**PR code and description written by:**
- [X] Human
- [ ] AI
**Reviewed before submission by:**
- [X] Human
- [x] AI
- [ ] Not reviewed
* GitHub Issue: #51751
Authored-by: Tadeja Kadunc <tadeja.kadunc@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
The vendored uriparser in `cpp/src/arrow/vendored/uriparser/` is 0.9.3, which is old and misses upstream security fixes. Instead of refreshing the vendored copy, this makes uriparser an external dependency like our other third-party dependencies.
### What changes are included in this PR?
- Add `uriparser` to `ARROW_THIRDPARTY_DEPENDENCIES`
- Add `build_uriparser()` that builds uriparser 1.0.2 with `FetchContent` for `uriparser_SOURCE=BUNDLED` (static; docs, tests, tools and `wchar_t` support are disabled). The built library is bundled into `arrow_bundled_dependencies`.
- Require uriparser 0.9.6 or later for `uriparser_SOURCE=SYSTEM`. All uriparser APIs used by Arrow are available in 0.9.6, so supported platforms such as Ubuntu 22.04 can use their distribution package.
- Add `cmake_modules/FinduriparserAlt.cmake`. It tries uriparser's CMake package configuration first and falls back to pkg-config (`liburiparser.pc`), because Debian and Ubuntu's `liburiparser-dev` provides only the pkg-config file.
- Add `uriparser::uriparser` to `ARROW_STATIC_INSTALL_INTERFACE_LIBS` for `uriparser_SOURCE=SYSTEM`, so that static library users link uriparser.
- meson: Use the system uriparser 0.9.6 or later if available. Otherwise, build uriparser 1.0.2 via `subprojects/uriparser.wrap` with the CMake subproject support.
- Remove `cpp/src/arrow/vendored/uriparser/` and its section in `LICENSE.txt`.
- Packaging: Install uriparser in our Docker images, conda environment, `Brewfile`, vcpkg manifests, MSYS2 setup and Linux packages build environments.
- Packaging: Pass `-Duriparser_SOURCE` in `ci/scripts/cpp_build.sh`.
- Packaging: Add uriparser to the dependency list in `docs/source/developers/cpp/building.rst`.
***NOTICE*** : The floor is 0.9.6 rather than 1.0.2, so supported platforms such as Ubuntu 22.04 can use their distro package and CI does not have to build uriparser from source.
### Are these changes tested?
All existing tests pass.
### Are there any user-facing changes?
Now Arrow requires uriparser or builds it . No user facing changes as such because no API changes .
**This PR contains a "Critical Fix".** It has security fixes that has gone in uriparser.
**AI Usage**:
- Used for testing <-> building loop.
* GitHub Issue: #51007
Authored-by: Narendra <naskastark11@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Fix#51329
Additional downstream static-linking test, after #50195 moved S3 into the separate ArrowS3 library.
### What changes are included in this PR?
Test downstream static linking of `ArrowS3::arrow_s3_static` via minimal_build example and `cpp_test.sh`,
Fix dependencies for static Azure and GCS consumers: append `libcurl` for bundled GCS, append `libcurl` and `LibXml2` for bundled Azure(NOT WIN32),
append links for system Azure SDK
and pkg-config --static arrow to include `libcurl` and `libxml-2.0` for bundled GCS/Azure
### Are these changes tested?
Verified by CI
### Are there any user-facing changes?
Downstream consumers now have required static dependencies for Azure/GCS
* GitHub Issue: #51329
Authored-by: Tadeja Kadunc <tadeja.kadunc@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
The Windows ARM job was failing on CMake configure. The real problem is that the Windows-11-arm64 GitHub runner updated VS Code from 2022 to 2026 here: https://github.com/actions/runner-images/issues/14592
The workflow was hardcoding `vcvarsall.bat` path and wasn't really setting those up.
### What changes are included in this PR?
Use vswhere to find VSCode installation path instead of manually setting it up.
I investigating using: https://github.com/ilammy/msvc-dev-cmd which is allowed by infra and used on nanoarrow and adbc but it feels unmaintained at the moment (No new releases in 3 years).
This has been changed everywhere where we were manually setting this path.
### Are these changes tested?
Yes via CI
### Are there any user-facing changes?
No
### Was AI used for this PR?
In accordance to the [AI generation guidelines](https://arrow.apache.org/docs/dev/developers/overview.html#ai-generated-code), please disclose below whether and how AI was used in this PR.
**PR code and description written by:**
- [x] Human
- [x] AI
AI used to analyze and first implementation.
Re-implemented on a second bash step from me as it's much cleaner and clearer.
**Reviewed before submission by:**
- [x] Human
- [ ] AI
- [ ] Not reviewed
* GitHub Issue: #51494
Lead-authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
Due to homebrew dropping support for building bottles for macOS Intel several CI jobs started failing. Some other arm64 related jobs also failed because old bottles are also not built. There's a ML discussion about dropping support for macOS Intel, see for more details:
- https://lists.apache.org/thread/l3vmknbjhwf60crrfp3hwb7p5m4qjw7z
### What changes are included in this PR?
Bump macOS arm runners to macos-26 and macOS Intel runners (that don't require homebrew bottles) to macos-26-intel.
Drop the macOS Intel source verification jobs that require Homebrew bottles in order to run.
### Are these changes tested?
Yes on CI. The only untested ones are the bumps on binary verification.
### Are there any user-facing changes?
No
### Was AI used for this PR?
In accordance to the [AI generation guidelines](https://arrow.apache.org/docs/dev/developers/overview.html#ai-generated-code), please disclose below whether and how AI was used in this PR.
**PR code and description written by:**
- [x] Human
- [ ] AI
**Reviewed before submission by:**
- [x] Human
- [ ] AI
- [ ] Not reviewed
* GitHub Issue: #51402
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Several dependabot bumps were raised but were missing the `tasks/github.yml` file and our CI is currently failing for Windows matlab builds due to caching.
### What changes are included in this PR?
Bump both the matlab actions on the workflow for this repo and the one used on archery.
Do not use the built folder for computing the hash for they hash key.
### Are these changes tested?
Via CI and archery.
### Are there any user-facing changes?
No
### Was AI used for this PR?
In accordance to the [AI generation guidelines](https://arrow.apache.org/docs/dev/developers/overview.html#ai-generated-code), please disclose below whether and how AI was used in this PR.
**PR code and description written by:**
- [x] Human
- [ ] AI
**Reviewed before submission by:**
- [x] Human
- [ ] AI
- [ ] Not reviewed
AI was used to analyze the CI failures.
* GitHub Issue: #51335
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Trying to reduce the size of `libarrow.so` and remove AWS SDK on some builds. Allow for users to plug and play based on requirements and divide our functionality into cleaner modules.
### What changes are included in this PR?
Unconditionally build S3 and the AWS SDK into a different module `libarrow_s3.so` outside of `libarrow.so`.
Update bindings to link against the new `libarrow_s3.so` library.
Update the Linux Package jobs to have the new module into a different package.
### Are these changes tested?
Yes via CI
### Are there any user-facing changes?
Yes, users will need to either link against `libarrow_s3.so` or register using `LoadFileSystemFactories`
* GitHub Issue: #50194
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
When the issue was originally opened, the [test-conda-python-3.10-hdfs-2.9.2](https://github.com/ursacomputing/crossbow/actions/runs/22944679479/job/66593987920) failed sporadically on a PR when upgrading some dependencies due to time out rebuilding the docker image.
A big chunk of time was spent on the job just downloading HDFS 2.9.2.
HDFS 2.9.2 was released on 2018. We test 2.9.2 and 3.2.1, released on 2019.
### What changes are included in this PR?
Bump to newer versions, latests are 3.5.0 (2026) and from the v2 series, 2.10.2 (2022).
Bump default JDK to 17. The only other place where we currently use the JDK version on env was for Spark where we already are manually using `JDK=17` on tasks.yml so no real change for any other job.
### Are these changes tested?
Yes via the archery jobs
### Are there any user-facing changes?
No
* GitHub Issue: #49524
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
The `test-ubuntu-24.04-cpp-bundled-offline` Crossbow job usually takes one hour to complete:
https://github.com/ursacomputing/crossbow/actions/runs/34175056969/job/101902703505
This is because, despite sccache being configured, the S3-stored compilation cache is unavailable because the network is shut off during compilation. And the job compiles all dependencies from scratch.
### What changes are included in this PR?
Move the CI job to the C++ Extra workflow, so as to take advantage of ccache-based compilation caching using `apache/infrastructure-actions/stash`.
### Are these changes tested?
By definition, yes.
### Are there any user-facing changes?
No.
* GitHub Issue: #51254
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
`reprotest` step is flaky. It seems that it's killed by OOM (out-of-memory).
### What changes are included in this PR?
It seems that `diffoscope` is heavy. Disable it.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
* GitHub Issue: #51217
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
In the current release workflow, we update `CHANGELOG.md` only in the maintenance branches. So `CHANGELOG.md` in main misses entries.
We can update `CHANGELOG.md` in our release workflow.
### What changes are included in this PR?
* Run `archery release changelog add` in `post-10-bump-versions.sh`
* Add missing entries to `CHANGELOG.md`
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* GitHub Issue: #51098
Lead-authored-by: Sutou Kouhei <kou@clear-code.com>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
CI clang version outdated, false positive failures against CRAN
### What changes are included in this PR?
Update clang manually
### Are these changes tested?
Will run CI
### Are there any user-facing changes?
No
* GitHub Issue: #50298
Authored-by: Nic Crane <thisisnic@gmail.com>
Signed-off-by: Nic Crane <thisisnic@gmail.com>
### Rationale for this change
This PR moves crossbow builds to short-lived S3 credentials. It does this via GitHub's OIDC provider and an AWS ROLE_ARN with a trust relationship configured. This configuration is detailed here:
- [Use IAM roles to connect GitHub Actions to actions in AWS](https://aws.amazon.com/blogs/security/use-iam-roles-to-connect-github-actions-to-actions-in-aws/)
The benefits of short-lived / temporary credentials discussed further here: [AWS Well-architected Framework Best Practices SEC02-BP02 Use temporary credentials](https://docs.aws.amazon.com/wellarchitected/latest/framework/sec_identities_unique.html)
sccache supports a variety of S3 creds: [S3 credentials](https://github.com/mozilla/sccache/blob/main/docs/S3.md#credentials)
### What changes are included in this PR?
Use [AssumeRoleWithWebIdentity](https://github.com/mozilla/sccache/blob/main/docs/S3.md#credentials) for sccache S3 credentials.
By assuming the AWS_ROLE_ARN, the sccache session relies on credentials that expire\*, issued only to workflows in the trusted repository, and scoped to the sccache bucket.
Specific tasks:
- create a role in IAM with the bucket and object level permissions,
- trust the github oidc-provider,
- define AWS_ROLE_ARN and set id-token permission
- grant sts:AssumeRoleWithWebIdentity for the specific repositories or organization.
- pass AWS_SESSION_TOKEN
- add a `aws-actions/configure-aws-credentials@ v6` step to the workflow.
Separately, someone will need to add the role and trust to AWS:
- Procedure is in https://aws.amazon.com/blogs/security/use-iam-roles-to-connect-github-actions-to-actions-in-aws/
- Set a "Maximum session duration" to maximum runtime of any job... maybe 4 hours? 8 hours?
- Permissions: Role needs BucketLevel and ObjectLevels permissions to the bucket.
\* I can share an example if needed
### Are these changes tested?
Yes, locally and on a fork.
### Are there any user-facing changes?
No
* GitHub Issue: #50899
Authored-by: paul@iqmo.com <paul@iqmo.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
The test-r-linux-as-cran nightly job has been failing since 2026-08-25 because `r/jarl.toml` (added in #50851) ends up in the package tarball and R CMD check flags it as a non-standard top-level file.
### What changes are included in this PR?
Adds `jarl.toml` to `r/.Rbuildignore`, and removes the now-unused `r/.lintr` config along with its `.Rbuildignore` and RAT exclude entries.
### Are these changes tested?
Checked locally that the `.Rbuildignore` pattern matches `jarl.toml` without excluding anything else. The `test-r-linux-as-cran` crossbow job will confirm the NOTE is gone.
### Are there any user-facing changes?
No.
* GitHub Issue: #51152
Authored-by: Nic Crane <thisisnic@gmail.com>
Signed-off-by: Nic Crane <thisisnic@gmail.com>
### Rationale for this change
Add a Claude Code skill to guide folks through the CRAN release process, and add `.claude/` to the RAT excludes.
### What changes are included in this PR?
- New skill at `.claude/skills/r-cran-release/SKILL.md` with step-by-step CRAN release instructions
- Add `.claude/*` to `dev/release/rat_exclude_files.txt`
- Remove the "Prepare and Check Package That Will Be Released to CRAN" section from `r/PACKAGING.md` (covered by later steps)
### Are these changes tested?
No code changes to test.
### Are there any user-facing changes?
No.
* GitHub Issue: #50492
Authored-by: Nic Crane <thisisnic@gmail.com>
Signed-off-by: Nic Crane <thisisnic@gmail.com>
### Rationale for this change
Just as the C++ test suite gained coverage for extension-wrapped unions in #50927 (GH-50623), the cross-implementation integration tests should exercise this case so that every tested implementation handles an extension type whose storage is a union.
### What changes are included in this PR?
A new archery integration datagen case, `extension_union`, with two columns:
- `sparse_union_ext`: a sparse union (`int32`/`utf8`) wrapped in an extension type
- `dense_union_ext`: a dense union (`int16`/`binary`) wrapped in an extension type
`ExtensionField` already delegates its storage type/children/generation to the wrapped field, so no new datagen machinery is needed.
### Are these changes tested?
The generator produces valid integration JSON locally (schema carries the `ARROW:extension:*` metadata on the union-typed fields, children preserved, batch sizes `[0, 7]`). The cross-implementation integration matrix in CI is the real test here — I've intentionally added the case with no per-implementation skips so the matrix can show which implementations still need a temporary `.skip_tester(...)`; I'll add those (with tracking links) based on the results.
### Are there any user-facing changes?
No. This only adds integration-test data.
Closes#50993.
* GitHub Issue: #50993
Lead-authored-by: Alb3e3 <74142887+Alb3e3@users.noreply.github.com>
Co-authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Require Numpy 2.0 to support newer features, allow newer numpy API/ABI
### What changes are included in this PR?
- Numpy 2.0+ requirement
- Pandas 2.2.2+ requirement (first wheel compatible with numpy 2.0)
- Pyodide 0.27.1 usage (numpy 2.0 api)
### Are there any user-facing changes?
Yes, in contrast to pyarrow 25, after merging this PR pyarrow 26 can't be used with Numpy 1.x or pandas 2.0-2.2.1
**This PR includes breaking changes to public APIs.**
This PR was created using GPT-5.6-Sol-xhigh, every line read & reviewed by me.
* GitHub Issue: #48473
Lead-authored-by: Adam Lippai <adam@rigo.sk>
Co-authored-by: Ádám Lippai <adam@rigo.sk>
Co-authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Add [Python 3.15](https://docs.python.org/3.15/whatsnew/3.15.html) to the pyarrow nightly builds, so that issues can be detected early in the development cycles of both cpython & pyarrow.
Note:
- ~~CFFI and~~ Pandas doesn't have nightly 3.15 wheels, and are accordingly excluded.
- Numpy shipped 3.15rc1 in 2.5.2 ~~doesn't ship a 3.15 wheel, but is needed to build. It is building from source.~~
- [3.15 Schedule](https://peps.python.org/pep-0790/) This PR targets 3.15.0rc1. rc2 is expected 2026-09-01 and final 2026-10-01.
### What changes are included in this PR?
- Adding cp315/cp315t to the build matrix
- Excluding ~~cffi/~~ pandas for 3.15, as the wheels aren't yet available.
### 2026-08-13 edit
- Bump libcst to 1.9.0
- Update manylinux image to 2026.08.08-1 (for 3.15rc1)
- Add -rc to PYTHON_IMAGE_TAG
- Disable pandas tests if Pandas isn't installed:
### Are these changes tested?
- Yes, tested on a local Linux build using the `ninja-release-python` preset (not maximal)
### Are there any user-facing changes?
No functional changes, other than allowing the user to use Python 3.15.
* GitHub Issue: #48172
### Progress Notes (2026-08-15)
Each platform uses different sources for Python builds, each with different release schedules and paths:
- [manylinux](https://github.com/pypa/manylinux) is prebuilt with latest Python builds in `/opt/python`, so using the latest manylinux build is sufficient. 3.15 wasn't available until beta 1.
- musllinux: Uses Alpine system python built from https://github.com/docker-library/python/pull/1123. Might need to bump ALPINE_LINUX version in [arrow/dev/tasks/python-wheels/github.linux.yml](https://github.com/apache/arrow/blob/e611f48081ec927e18e5bcb72fbd071c15c76b08/dev/tasks/python-wheels/github.linux.yml#L35).
- musllinux free-threaded: uses [astral's builds](https://github.com/astral-sh/python-build-standalone). Edit `python-free-threaded-wheel-musllinux-test-imports.dockerfile` + `-test-unittests.dockerfile`: set the python_patch_version, and set build_date in `compose.yaml`.
- macos: Downloads official build from python.org, see `ci/scripts/install_python.sh`. For early builds, needed to disable `--only-binary` since at least one dependency needed to be built (no 3.15 wheels available yet)... only macos uses --only-binary
- Windows: installs builds using `choco`, see `python-wheel-windows-test-vs2022.dockerfile` and `python-free-threaded-wheel-windows-test-vs2022.dockerfile`
For 3.15.0rc1 builds, some care was needed because certain paths needed 3.15.0 and others needed the full 3.15.0rc1.
Lead-authored-by: Paul Timmins <paul@iqmo.com>
Co-authored-by: paul@iqmo.com <paul@iqmo.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
When working with patch releases currently the maintenance branch used is not correct and the scripts for 02-source and 03-binary fail to create the PR or find it.
### What changes are included in this PR?
Use the correct maintenance branch format taking into account the differences between major releases and patch releases.
Updated release documentation to make it clearer that only the `maint-X.Y.x` branch is used for patch releases.
### Are these changes tested?
Yes, as I had to cut RC1 for the 25.0.1 release I've been able to validate that they work as expected now.
### Are there any user-facing changes?
No
* GitHub Issue: #50811
Lead-authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Co-authored-by: tadeja <tadeja@users.noreply.github.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
We updated from bookworm to trixie our binary signing image. When trying to sign the packages it failed with the following error:
```
Downloading: centos - 100.0% [63/63] 00:01:26 00:00:00 0/s
error: Unable to open sqlite database /var/lib/rpm/rpmdb.sqlite: unable to open database file
error: cannot open Packages index using sqlite - Operation not permitted (1)
error: cannot open Packages database in /var/lib/rpm
error: can't create transaction lock on /var/lib/rpm/.rpm.lock (No such file or directory)
error: /tmp/apache-arrow-binary20260703-49-htq2me.asc: key 1 import failed.
```
### What changes are included in this PR?
Use `sudo rpm` when signing binaries on the container.
### Are these changes tested?
Yes, I've used that to upload and sign binaries for 25.0.1
### Are there any user-facing changes?
No
* GitHub Issue: #50358
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
GH-50697
I have tested the RPM ODBC installer on AlmaLinux 9. The ODBC driver works as expected on AlmaLinux 9. The only issue I found was that the `connection-options.md` guide was not included in the package installation. This PR adds `connection-options.md` to the RPM ODBC package so users can reference the connection options documentation after installation.
Summary of testing results:
1. Clean install - driver works as expected and registered successfully, arrow installer installs `isql` and `unixODBC` during installation.
2. Uninstall - driver is uninstalled and unregistered automatically, arrow installer flags and uninstalls `isql` and `unixODBC` as unused dependency during installation.
If user wants to avoid this behavior, they can add `--noautoremove `flag to make `dnf` keep `isql` and `unixODBC` during ODBC uninstallation.
3. Upgrade - driver can be upgraded to new version.
4. Data retrieval works as expected, no dependency issues found.
### What changes are included in this PR?
- Add `connection-options.md` to the RPM ODBC package installation.
- Example location after install: `/usr/share/doc/arrow2600-flight-sql-odbc-libs/connection-options.md`
- Fix Windows ODBC installer to include docs `LICENSE.txt` and `NOTICE.txt`
- Add ODBC installer description to ODBC `README.md`
### Are these changes tested?
Yes
### Are there any user-facing changes?
Yes
* GitHub Issue: #50697
Lead-authored-by: Alina (Xi) Li <Alina.Li@improving.com>
Co-authored-by: Alina (Xi) Li <96995091+alinaliBQ@users.noreply.github.com>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Co-authored-by: justing-bq <justin.gossett@improving.com>
Signed-off-by: David Li <li.davidm96@gmail.com>
### Rationale for this change
Fix#50758 - `llvm-toolchain-20` package was [removed from Debian experimental on Jul 14](https://tracker.debian.org/news/1774937/removed-12018-1-from-experimental/), so the nightly `test-debian-experimental-cpp-gcc-15` fails with `E: Unable to locate package clang-20` / `llvm-20-dev`.
### What changes are included in this PR?
Update LLVM 20 to 22 on Debian experimental in `dev/tasks/tasks.yml`
### Are these changes tested?
Yes, built locally and verified via crossbow.
### Are there any user-facing changes?
No.
* GitHub Issue: #50758
Authored-by: Tadeja Kadunc <tadeja.kadunc@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
We need this when we break our Yum repositories.
### What changes are included in this PR?
We can recover Yum repositories too by `dev/release/binary-recover.sh`.
### Are these changes tested?
Yes. I recovered our Yum repositories with this.
### Are there any user-facing changes?
No.
* GitHub Issue: #50674
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### What changes are included?
This PR introduces a reusable `JsonWriter` wrapper around simdjson's JSON builder API and migrates the integration JSON writer to use it instead of RapidJSON.
Specifically, this PR:
- Adds a reusable `JsonWriter` abstraction in `arrow/json`.
- Migrates the integration JSON writer implementation to `JsonWriter`.
- Updates the integration tests to use `JsonWriter`.
- Adds unit tests for `JsonWriter`.
This is part of the incremental migration from RapidJSON to simdjson.
### Performance
I compared the serialization performance of `IntegrationJsonWriter::WriteRecordBatch()` + `Finish()` using a temporary benchmark with a 1M-row `RecordBatch`. The benchmark was run 10 times on both `main` and this branch.
| Branch | Average Time |
|--------|-------------:|
| `main` | 2157.4 ms |
| This PR | 2153.6 ms |
No measurable performance regression was observed.
### Are these changes tested?
Yes.
I added unit tests for `JsonWriter` and verified that both JSON and integration tests pass locally:
- `arrow-json-test`
- `arrow-json-integration-test`
### Are there any user-facing changes?
No.
* GitHub Issue: #50567
Authored-by: Aaditya Srinivasan <aadityasri03@gmail.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Crossbow currently computes development versions through internal `setuptools_scm` APIs. Those APIs are unstable and require Archery to pin an otherwise unnecessary runtime dependency.
### What changes are included in this PR?
- Compute the development version from public `git describe` output.
- Preserve the existing behavior for released, development, release-candidate, and dirty tags.
- Remove `setuptools_scm` from the Crossbow extra.
- Add unit coverage plus a real temporary Git repository test.
### Are these changes tested?
- `python -m pytest -q dev/archery/archery/crossbow/tests/test_core.py` (31 passed)
- `flake8 --max-line-length=88` on the changed Python files
- `autopep8 --diff --max-line-length=88` on the changed Python files
- Verified Archery imports when `setuptools_scm` is not installed
### Are there any user-facing changes?
No. This removes an internal dependency while preserving Crossbow version behavior.
Closes#40163
* GitHub Issue: #40163
Authored-by: Liang Hu <lh3057@columbia.edu>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
This is the sub issue #44748.
Fix only ShellCheck SC2086 errors in the dev directory.
* SC2086: Double quote to prevent globbing and word splitting.
```
shellcheck dev/release/post-01-tag.sh
In dev/release/post-01-tag.sh line 34:
git tag -a ${release_tag} ${release_candidate_tag} -m "[Release] Apache Arrow Release ${version}"
^------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
^----------------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
git tag -a "${release_tag}" "${release_candidate_tag}" -m "[Release] Apache Arrow Release ${version}"
In dev/release/post-01-tag.sh line 35:
git push upstream ${release_tag}
^------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
git push upstream "${release_tag}"
For more information:
https://www.shellcheck.net/wiki/SC2086 -- Double quote to prevent globbing ...
```
### What changes are included in this PR?
* SC2086: Quote variables.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
* GitHub Issue: #50670
Authored-by: Hiroyuki Sato <hiroysato@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
This PR is the first in a series of PRs for GH-35460 to migrate Arrow's JSON implementation from RapidJSON to simdjson.
As the foundation for the migration, this PR introduces simdjson into Arrow's build system, updates the bundled simdjson version to v4.6.4, and migrates the self-contained `arrow::json::internal::ObjectParser`.
### What changes are included in this PR?
This PR:
- Introduces `simdjson` as a third-party dependency.
- Adds CMake support for discovering and building `simdjson`.
- Updates the bundled simdjson dependency to **v4.6.4**.
- Adds system package and vcpkg integration required for simdjson across supported build environments.
- Links `simdjson` into the Arrow JSON library.
- Migrates `arrow::json::internal::ObjectParser` from RapidJSON to the simdjson DOM API.
- Adds dedicated unit tests covering:
- String retrieval
- Boolean retrieval
- String map extraction
- Invalid JSON
- Missing keys
- Incorrect types
- Non-object root documents
- Empty objects
This establishes the build infrastructure needed for the remaining JSON migration, which will be completed incrementally in follow-up PRs.
### Performance
On the ObjectParser benchmark, the simdjson implementation achieved approximately **6.3×** higher throughput than the previous RapidJSON implementation (about **84%** lower execution time).
### Are these changes tested?
Yes.
- Added dedicated unit tests for `ObjectParser`.
- Verified the ObjectParser JSON test suite locally.
- Updated the build system and validated the dependency integration across supported CI configurations (currently 41 CI checks passing; remaining failures are limited to two Windows-specific workflows under investigation).
* GitHub Issue: #50566
Authored-by: Aaditya Srinivasan <aadityasri03@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
It's convenient when `dnf install arrow26-flight-sql-odbc-libs` registers our driver automatically.
### What changes are included in this PR?
* Generate `arrow-flight-sql-odbc-template.ini` for `odbcinst -i -d -f`
* Run `odbcinst -i -d -f arrow-flight-sql-odbc-template.ini` automatically on install
* Run `odbcinst -u -d -n "Apache Arrow Flight SQL ODBC Driver"` automatically on uninstall
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* GitHub Issue: #50518
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
Standard release task
### What changes are included in this PR?
Add previous version to compatibility matrix
### Are these changes tested?
CI
### Are there any user-facing changes?
No
Authored-by: Nic Crane <thisisnic@gmail.com>
Signed-off-by: Nic Crane <thisisnic@gmail.com>
### Rationale for this change
#48886 replaced github3 with pygithub in archery. Unlike github3, pygithub's `Auth.Token` requires a non-None string, so `_github_login()` crashes when no token is provided. The `r_nightly.yml` workflow doesn't pass `CROSSBOW_GITHUB_TOKEN` to archery, so it has been failing since June 19 (#50532). No R nightly binaries have been uploaded for a month.
### What changes are included in this PR?
Fall back to unauthenticated `Github()` when no token is provided, restoring the pre-migration behavior. The crossbow repo is public so auth isn't required for downloading artifacts.
### Are these changes tested?
Tested locally by running `archery crossbow download-artifacts` without a token set.
### Are there any user-facing changes?
No.
* GitHub Issue: #50532
Authored-by: Nic Crane <thisisnic@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
This PR adds support for Azure. The Arrow R package already has support for AWS and GCS, and the Arrow C++ library has had support for Azure for a couple years now. Support for Azure is already available in pyarrow.
This would close#32123.
### What changes are included in this PR?
- A new class `AzureFileSystem` that's analogous to `S3FileSystem`/`GcsFileSystem`, along with a helper function `az_container()` that's analogous to `s3_bucket()`/`gcs_bucket()`.
- Updates to `src/filesystem.cpp` to interact with the machinery in `arrow/filesystem/azurefs.h`.
- Updates to the configuration and build scripts to support building with support for Azure.
- Updates to the vignettes on cloud storage, installation, and developer setup.
### Are these changes tested?
Yes. See `tests/testthat/test-azure.R`.
### Are there any user-facing changes?
Yes. There is a new function `az_container()`, serving the analogous role to `s3_bucket()`/`gcs_bucket()`, along with an R6 class `AzureFileSystem`, again serving the same role as `S3FileSystem` and `GcsFileSystem`. There is also a function `arrow_with_azure()` to indicate if Arrow was built with support for Azure.
* GitHub Issue: #32123
Lead-authored-by: Collin Brown <collinbrown95@gmail.com>
Co-authored-by: Steve Martin <marberts@protonmail.com>
Co-authored-by: Steve Martin <62676717+marberts@users.noreply.github.com>
Co-authored-by: Collin Brown <8021046+Collinbrown95@users.noreply.github.com>
Co-authored-by: Jonathan Keane <jkeane@gmail.com>
Co-authored-by: Collin Brown <collin.j.brown95@gmail.com>
Co-authored-by: Collin Brown <cbrown631@gatech.edu>
Signed-off-by: Jonathan Keane <jkeane@gmail.com>
### Rationale for this change
[Pandas 2.0.0](https://pypi.org/project/pandas/2.0.0/) was released on April 3rd 2023.
The last version of Pandas 1 was [Pandas 1.5.3](https://pypi.org/project/pandas/1.5.3/) and it was released on January 19th 2023. That version only supports up to Python 3.11.
We are currently supporting up to Pandas 1.5.2 (first version supporting 3.11).
On the issue we discussed to bump our Pandas support to 2.0.0 but given 2.0.3 is the latest in the 2.0.x, was released on June 2023, only supports up to Python 3.11 and bumping to it allows use to improve even further some of the conditionals we are bumping to Pandas >= 2.0.3
### What changes are included in this PR?
Update minimal support check to Version(2.0.3) and remove all test skips, conditionals and scenarios where we are checking for Pandas < 2.0.3.
We found a couple of minor things
### Are these changes tested?
Yes via CI
### Are there any user-facing changes?
Yes, Pandas < 2 won't be supported.
* GitHub Issue: #48808
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Python 3.10 is End of Support on October 2026. We can drop support on PyArrow 26.0.0.
See: https://endoflife.date/python
### What changes are included in this PR?
Remove support for 3.10. Update minimum required Python version and update some CI jobs to use the minimum Python supported version.
Some CI jobs using default Python 3.10 (Ubuntu 20.04) have been bumped.
### Are these changes tested?
They will be tested via CI and Archery
### Are there any user-facing changes?
Yes in terms of not supporting older versions of Python. This will be a breaking change for some users but not for the API itself
* GitHub Issue: #50410
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
The Windows Wheels verification job is currently failing due to a change of API on the conda create call.
### What changes are included in this PR?
Remove the deprecated `-f` flag.
### Are these changes tested?
Yes, I have pushed the branch to the Apache Arrow repository instead of my fork to be able to manually trigger the workflow for the RC via workflow dispatch in order to validate the changes.
### Are there any user-facing changes?
No
* GitHub Issue: #50383
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
If we provide RPM for Flight SQL ODBC driver, users can install it easily.
### What changes are included in this PR?
* Add `arrow${MAJOR_VERSION}-flight-sql-odbc-libs`
* Add `arrow-flight-sql-odbc-devel`
### Are these changes tested?
No.
We need to generate configuration files but it's out-of-scope of this PR. We need to work on it as a separated task.
### Are there any user-facing changes?
Yes.
* GitHub Issue: #47877
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
Raúl Cumplido