### Rationale for this change
Trying to reduce the size of libarrow.so and remove google-cloud-cpp SDK on some builds. Allow for users to plug and play based on requirements and divide our functionality into cleaner modules.
### What changes are included in this PR?
Unconditionally build GCS and the google-cloud-cpp SDK into a different module libarrow_gcs.so outside of libarrow.so.
Update bindings to link against the new libarrow_gcs.so library.
Update the Linux Package jobs to have the new module into a different package.
### Are these changes tested?
Yes via CI
### Are there any user-facing changes?
Yes, users will need to either link against libarrow_s3.so or register using LoadFileSystemFactories
### Was AI used for this PR?
In accordance to the [AI generation guidelines](https://arrow.apache.org/docs/dev/developers/overview.html#ai-generated-code), please disclose below whether and how AI was used in this PR.
**PR code and description written by:**
- [x] Human
- [x] AI
**Reviewed before submission by:**
- [x] Human
- [ ] AI
- [ ] Not reviewed
* GitHub Issue: #54902
Authored-by: Raúl Cumplido <raulcd@apache.org>
Signed-off-by: Raúl Cumplido <raulcd@apache.org>
### Rationale for this change
We often get private security reports for some issues which don't threaten confidentiality or integrity.
For these kinds of issues, we would rather have them reported through the usual public means, as it follows our open development principles more closely, and lessens the required paperwork (no CVE, etc.).
### What changes are included in this PR?
Add a paragraph in the C++ security model about what should be reported as a security vulnerability vs. a regular bug.
### Are these changes tested?
N/A.
### Are there any user-facing changes?
No.
### Was AI used for this PR?
In accordance to the [AI generation guidelines](https://arrow.apache.org/docs/dev/developers/overview.html#ai-generated-code), please disclose below whether and how AI was used in this PR.
**PR code and description written by:**
- [x] Human
- [ ] AI
**Reviewed before submission by:**
- [x] Human
- [ ] AI
- [ ] Not reviewed
* GitHub Issue: #49760
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
We have many deprecated APIs in Python we should slowly remove.
### What changes are included in this PR?
Several deprecation messages are acted on.
### Are these changes tested?
By the existing test suite.
### Are there any user-facing changes?
Yes, some deprecated user-facing APIs are.removed.
### Was AI used for this PR?
In accordance to the [AI generation guidelines](https://arrow.apache.org/docs/dev/developers/overview.html#ai-generated-code), please disclose below whether and how AI was used in this PR.
**PR code and description written by:**
- [ ] Human
- [x] AI
**Reviewed before submission by:**
- [x] Human
- [x] AI
- [ ] Not reviewed
* GitHub Issue: #49153
Authored-by: Rok Mihevc <rok@mihevc.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
Co-authored-by: Dhirhan Kanesalingam <[dhirhan17@ gmail.com](mailto:dhirhan17@ gmail.com)>
With help from : @ emkornfield, @ wgtmac
### Rationale for this change
[Adaptive Lossless Floating-Point (ALP)](https://docs.google.com/document/d/1PlyUSfqCqPVwNt8XA-CfRqsbc0NKRG0Kk1FigEm3JOg/edit?tab=t.0#heading=h.5xf60mx6q7xk)
is designed for floating-point data that commonly represents decimal values.
For these workloads, ALP can provide better compression and faster decoding
than general-purpose compression or existing Parquet encodings.
This PR adds ALP support for Parquet `FLOAT` and `DOUBLE` columns in the Arrow
C++ implementation.
### Specification
- [Original ALP specification](https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit)
- [Parquet `AlpEncoding.md`](https://github.com/apache/parquet-format/blob/main/AlpEncoding.md)
(merged through apache/parquet-format#539; currently in Preview)
### What changes are included in this PR?
This PR adds:
- The core ALP compression and decompression implementation.
- Sampling logic for selecting encoding parameters.
- Page metadata serialization, validation, and decoding.
- Parquet encoders and decoders for `FLOAT` and `DOUBLE`.
- Incremental decoding of ALP pages one vector at a time.
- CMake and Meson build integration.
- C++ documentation describing how to use the encoding.
ALP is opt-in on the write path. It is used only when the writer explicitly
selects `Encoding::ALP` and dictionary encoding is disabled. Arrow does not
currently select ALP automatically based on the input data.
### Are these changes tested?
Yes. Test coverage includes:
- Interoperability with the ALP conformance file in `parquet-testing`.
- `FLOAT` and `DOUBLE` round trips.
- Decimal-like, constant, random, and exception-heavy inputs.
- Empty and all-null pages.
- Configurable vector sizes.
- Truncated or malformed page metadata.
- Invalid offsets, element counts, bit widths, and exception positions.
- Batched and incremental decoding.
### Are there any user-facing changes?
Yes. Arrow C++ can read Parquet pages encoded with ALP. Writers can explicitly
select ALP for supported floating-point columns.
ALP requires reader support and is not selected automatically, so existing
writer behavior remains unchanged unless users opt in.
* GitHub Issue: #48701
Lead-authored-by: Prateek Gaur <prateek.gaur@snowflake.com>
Co-authored-by: Gang Wu <ustcwg@gmail.com>
Signed-off-by: Gang Wu <ustcwg@gmail.com>
### Rationale for this change
The Union layout documentation does not explain that the type IDs stored in the types buffer can differ from the child array indices, or how the optional `typeIds` metadata maps children to physical type IDs.
### What changes are included in this PR?
- Document the relationship between Union child array indices, physical type IDs, and the optional `typeIds` metadata.
- Clarify that the types buffer stores the type ID for each slot.
Fixes#22342.
### Are these changes tested?
`git diff --check` passes and the RST change was reviewed against `Schema.fbs` and the existing Union layout documentation. A local Sphinx/RST checker is not installed in this environment.
### Are there any user-facing changes?
No.
* GitHub Issue: #22342
Lead-authored-by: erichanwang <ericwang559@gmail.com>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Arrow is currently missing remainder and modulo kernels.
### What changes are included in this PR?
Add the kernels `remainder`, `remainder_checked`, `modulo`, and `modulo_checked`, following the terminology in the [divmod proposal](https://lists.apache.org/thread/h3t6nz1ys2k2hnbrjvwyoxkf70cps8sh):
* `remainder` uses truncated (C/C++) semantics — the result has the sign of the dividend, e.g. `remainder(-7, 3) == -1`.
* `modulo` uses floored (Python/R) semantics — the result has the sign of the divisor, e.g. `modulo(-7, 3) == 2`.
Both are supported for integer, floating-point and decimal inputs, and are exposed as `compute::Remainder` / `compute::Modulo` in `api_scalar.h`.
Decimal arguments are promoted to a common scale like `add`, but the result type is resolved by a dedicated resolver rather than reusing `ResolveDecimalAdditionOrSubtractionOutput`: a remainder is always smaller in magnitude than the divisor, so no extra digit is needed for a carry, and the result is `precision = max(p1, p2)`, `scale = s1`. This keeps maximum-precision inputs (`decimal128(38, 0)`, `decimal256(76, 0)`) from overflowing the decimal precision range.
Left as follow-ups: combined divmod #27909, floor division #39386, and duration support.
### Are these changes tested?
Yes. New tests cover truncated vs. floored semantics across signed/unsigned integer, floating-point and decimal types, division by zero, `INT_MIN % -1` overflow, and decimal result-type resolution (same type in/out, mixed precisions and scales, Decimal128/Decimal256 promotion, decimal/integer promotion, and maximum-precision decimals).
### Are there any user-facing changes?
Yes — four new compute functions (`remainder`, `remainder_checked`, `modulo`, `modulo_checked`), the corresponding `compute::Remainder` / `compute::Modulo` C++ APIs, and a new `BasicDecimal256 operator%`. These are additions only; no existing API is changed or removed.
* GitHub Issue: #46901
* xref: https://github.com/apache/arrow/pull/46902,
* xref: https://github.com/apache/arrow/pull/28497
Lead-authored-by: Fangchen Li <fangchen.li@outlook.com>
Co-authored-by: Rok Mihevc <54589+rok@users.noreply.github.com>
Co-authored-by: tadeja <864005+tadeja@users.noreply.github.com>
Signed-off-by: Rok Mihevc <rok@mihevc.org>
### Rationale for this change
NumPy 2.0 added `StringDType`, a variable-width UTF-8 string dtype. `pa.array` currently rejects it with `Unsupported numpy type 2056`. Now that pyarrow requires NumPy 2.0 the conversion can use the public `NpyString_*` C API.
This supersedes #50951. The first commit is @ alippai's first commit from that PR, unchanged. His later commits added batching and new bulk builder APIs in Arrow C++; those can come later in their own PR with benchmarks, as discussed there. Nothing outside the NumPy conversion code changes here.
### What changes are included in this PR?
- `NumPyDtypeToArrow` maps `StringDType` to `string`, so it is also the inferred type for `pa.array`, `pa.infer_type`, `pa.from_numpy_dtype` and lists of such arrays. `large_string` and `string_view` can be requested with `type`. Any other type raises a `TypeError` naming the requested type.
- The conversion holds the dtype's allocator lock for the whole array and never holds the GIL while taking it.
- A null entry in a StringDType array becomes an Arrow null unless the dtype's `na_object` is a `str`. NumPy reports such entries as that string from `__getitem__`, in ufuncs and comparisons, and in casts, and there is no way to tell them apart from a regular entry holding the same string, so the conversion writes the string too. NaN-like sentinels such as `np.nan` and `pd.NA`, `None`, and arbitrary objects all become arrow nulls.
- A short section in `docs/source/python/numpy.rst` describing the support and the semantics for converting missing data.
If left unfixed, https://github.com/apache/arrow/issues/51156 also becomes possible to trigger from Python by creating a StringDType array with a very large entry.
### Are these changes tested?
Yes. Tests cover the three target types, strings crossing NumPy's short, medium and long storage thresholds, non-ASCII input, strided input, `mask`, `None`, NaN and string sentinels, the rejected target type, and list inference. I also ran a threaded test that converts an array while other threads hit NumPy's null-comparison error path on the same array to check the lock ordering.
### Are there any user-facing changes?
Yes. `pa.array` and the inference functions accept `StringDType` arrays.
AI disclosure: I used an AI model to work on the follow-up commits on top of @ alippai's first commit and for code review.
* GitHub Issue: #42018
* GitHub Issue: #51652
Lead-authored-by: Nathan Goldbaum <nathan.goldbaum@gmail.com>
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Co-authored-by: Adam Lippai <adam@rigo.sk>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
### Rationale for this change
The vendored uriparser in `cpp/src/arrow/vendored/uriparser/` is 0.9.3, which is old and misses upstream security fixes. Instead of refreshing the vendored copy, this makes uriparser an external dependency like our other third-party dependencies.
### What changes are included in this PR?
- Add `uriparser` to `ARROW_THIRDPARTY_DEPENDENCIES`
- Add `build_uriparser()` that builds uriparser 1.0.2 with `FetchContent` for `uriparser_SOURCE=BUNDLED` (static; docs, tests, tools and `wchar_t` support are disabled). The built library is bundled into `arrow_bundled_dependencies`.
- Require uriparser 0.9.6 or later for `uriparser_SOURCE=SYSTEM`. All uriparser APIs used by Arrow are available in 0.9.6, so supported platforms such as Ubuntu 22.04 can use their distribution package.
- Add `cmake_modules/FinduriparserAlt.cmake`. It tries uriparser's CMake package configuration first and falls back to pkg-config (`liburiparser.pc`), because Debian and Ubuntu's `liburiparser-dev` provides only the pkg-config file.
- Add `uriparser::uriparser` to `ARROW_STATIC_INSTALL_INTERFACE_LIBS` for `uriparser_SOURCE=SYSTEM`, so that static library users link uriparser.
- meson: Use the system uriparser 0.9.6 or later if available. Otherwise, build uriparser 1.0.2 via `subprojects/uriparser.wrap` with the CMake subproject support.
- Remove `cpp/src/arrow/vendored/uriparser/` and its section in `LICENSE.txt`.
- Packaging: Install uriparser in our Docker images, conda environment, `Brewfile`, vcpkg manifests, MSYS2 setup and Linux packages build environments.
- Packaging: Pass `-Duriparser_SOURCE` in `ci/scripts/cpp_build.sh`.
- Packaging: Add uriparser to the dependency list in `docs/source/developers/cpp/building.rst`.
***NOTICE*** : The floor is 0.9.6 rather than 1.0.2, so supported platforms such as Ubuntu 22.04 can use their distro package and CI does not have to build uriparser from source.
### Are these changes tested?
All existing tests pass.
### Are there any user-facing changes?
Now Arrow requires uriparser or builds it . No user facing changes as such because no API changes .
**This PR contains a "Critical Fix".** It has security fixes that has gone in uriparser.
**AI Usage**:
- Used for testing <-> building loop.
* GitHub Issue: #51007
Authored-by: Narendra <naskastark11@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
We only list the official Arrow implementations in our docs though there are a number of third-party open source implementations existing.
### What changes are included in this PR?
- Third-party open source implementations are added to the docs,
- `nanoarrow` is moved under the _Official Implementations_ list.
### Are these changes tested?
Via docs preview.
### Are there any user-facing changes?
No.
### Was AI used for this PR?
In accordance to the [AI generation guidelines](https://arrow.apache.org/docs/dev/developers/overview.html#ai-generated-code), please disclose below whether and how AI was used in this PR.
**PR code and description written by:**
- [x] Human
- [ ] AI
**Reviewed before submission by:**
- [x] Human
- [ ] AI
- [ ] Not reviewed
* GitHub Issue: #47472
Authored-by: AlenkaF <frim.alenka@gmail.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
Summary:
- Document that explicit dataset schemas for partitioned datasets must include partition fields used in filters or projections.
- Add a doctest-backed example showing a Hive partition field used with `count_rows`.
Validation:
- `/tmp/arrow-doccheck-30800/bin/python -m pytest --doctest-glob="*.rst" docs/source/python/dataset.rst -q` — passed, `1 passed`
- `git diff --check HEAD~1 HEAD` — passed
* GitHub Issue: #30800
Authored-by: kalyanamdewri <kalyanampriyam@gmail.com>
Signed-off-by: Nic Crane <thisisnic@gmail.com>
### Rationale for this change
`starts_with`, `ends_with` and `match_substring` have no Substrait mapping, so any expression using them fails to serialize:
```python
import pyarrow as pa
import pyarrow.compute as pc
from pyarrow.substrait import serialize_expressions
schema = pa.schema([pa.field("cat", pa.string())])
serialize_expressions([pc.starts_with(pc.field("cat"), "al")], ["f"], schema)
# ArrowNotImplementedError: No conversion function exists to convert the
# Arrow function starts_with to a Substrait call
```
Comparisons, `isin` and arithmetic serialize fine, so this is a per-function gap. It matters for engines that ingest a PyArrow filter through Substrait, where an unmappable function becomes a hard failure rather than a fallback.
See #50988.
### What changes are included in this PR?
Map the three kernels onto `starts_with`, `ends_with` and `contains` from Substrait's `functions_string.yaml`, in both directions.
The signatures do not line up. Substrait passes the pattern as a second argument, while the Arrow kernels are unary and carry it in `MatchSubstringOptions`. So:
* encoding lifts `MatchSubstringOptions::pattern` out into a literal argument
* decoding requires that argument to be a non-null string literal, and returns `NotImplemented` otherwise
Substrait's `case_sensitivity` option maps onto `ignore_case`. `CASE_INSENSITIVE_ASCII` has no Arrow equivalent and returns `NotImplemented`.
Both caveats are documented in `docs/source/cpp/acero/substrait.rst`.
### Are these changes tested?
Yes.
`Substrait.StringMatchExpressionSerialization` round-trips all three functions with and without `ignore_case`. `Substrait.StringMatchExpressionDeserialization` deserializes hand-written Substrait JSON to cover the default (no option) case and the two rejection paths. `test_serializing_string_match_expressions` round-trips each function through `pyarrow.substrait`.
### Are there any user-facing changes?
Yes. `pc.starts_with`, `pc.ends_with` and `pc.match_substring` can now be serialized to Substrait and consumed back, and Acero can consume plans using Substrait's `starts_with`, `ends_with` and `contains`. No existing behaviour changes: these previously raised.
* GitHub Issue: #50988
Authored-by: Jonas Dedden <mail@jonas-dedden.de>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
Addresses #51349
### What changes are included in this PR?
* Adds a new `read_kms_url` member to `DecryptionConfiguration` and a parameter with the same name to `rotate_master_keys`. Both are false by default.
* Adds these to the PyArrow bindings.
### Are these changes tested?
Yes, this includes new unit tests.
### Are there any user-facing changes?
Yes, this adds a new user-facing option. Users that relied on this behaviour previously will now need to opt-in and enable the flag.
* GitHub Issue: #51349
Authored-by: Adam Reeve <adreeve@gmail.com>
Signed-off-by: Adam Reeve <adreeve@gmail.com>
### Rationale for this change
The existing Arrow spec is misleading and allows certain Arrow types to
be typed value Variant fields. The Parquet Shredding spec specifically
rejects them - Unsigned integers, Null.
Decimal mapping needs clarification about precision.
### What changes are included in this PR?
Change the Primitive Type Mappings table for Parquet Variant extension
type.
Rendered version -
https://github.com/sdf-jkl/arrow/blob/gh50622-variant-typed-value-mappings/docs/source/format/CanonicalExtensions.rst#primitive-type-mappings
### Are these changes tested?
N/A
### Are there any user-facing changes?
Changes to the Arrow docs
Closes#50622
* GitHub Issue: #50622
### Rationale for this change
Anyone could assign themselves an issue by commenting "take", including issues already assigned to someone else. Assignment should be handled by committers and collaborators instead.
### What changes are included in this PR?
Removes the `issue_assign` job from the comment bot workflow and updates the contributor docs to say that a committer or collaborator will assign issues on request.
### Are these changes tested?
No. Docs were checked with sphinx-lint via pre-commit.
### Are there any user-facing changes?
No. Contributors can no longer self-assign issues by commenting "take".
* GitHub Issue: #51232
Lead-authored-by: Nic Crane <thisisnic@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Nic Crane <thisisnic@gmail.com>
### Rationale for this change
Trying to reduce the size of `libarrow.so` and remove AWS SDK on some builds. Allow for users to plug and play based on requirements and divide our functionality into cleaner modules.
### What changes are included in this PR?
Unconditionally build S3 and the AWS SDK into a different module `libarrow_s3.so` outside of `libarrow.so`.
Update bindings to link against the new `libarrow_s3.so` library.
Update the Linux Package jobs to have the new module into a different package.
### Are these changes tested?
Yes via CI
### Are there any user-facing changes?
Yes, users will need to either link against `libarrow_s3.so` or register using `LoadFileSystemFactories`
* GitHub Issue: #50194
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Exempt contributors with repository collaborator access from the concurrent pull request limit.
The workflow comments and documentation are updated accordingly.
### Are these changes tested?
Not yet.
### Are there any user-facing changes?
No.
Closes: #51331
* GitHub Issue: #51331
Lead-authored-by: Aaditya Srinivasan <aadityasri03@gmail.com>
Co-authored-by: Aaditya Srinivasan <156181482+Reranko05@users.noreply.github.com>
Co-authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
Too many low-quality or drive-by AI-generated PRs from non-committers blocking up review capacity
### What changes are included in this PR?
Prevent non-committers opening more than 3 concurrent PRs. This may be a little annoying for those who are engaging with effort, but PRs can be reopened later and so is an overall net positive.
Committers can reopen closed PRs on behalf of non-committers without this triggering.
### Are these changes tested?
I'll test on my fork
### Are there any user-facing changes?
No
### AI Usage
Generated this, but asked questions about security etc
* GitHub Issue: #51199
Lead-authored-by: Nic Crane <thisisnic@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Nic Crane <thisisnic@gmail.com>
### Rationale for this change
The Python memory/IO docs list `pyarrow.HdfsFile` as a `NativeFile` subclass, but that class no longer exists (legacy HDFS was deprecated in 2.0 and later removed). The `:class:` reference has no target, and HDFS access now uses `pyarrow.fs.HadoopFileSystem`, which returns regular `NativeFile` objects. It's the only unresolved pyarrow cross-reference on the page.
### What changes are included in this PR?
Removes the stale `pyarrow.HdfsFile` bullet in docs/source/python/memory.rst. One line, docs only.
### Are these changes tested?
Confirmed `pyarrow.HdfsFile` isn't in pyarrow, the ref appears only once, and all remaining `:class:` refs on the page resolve after removal.
### Are there any user-facing changes?
Documentation only.
Authored-by: Ritika shrestha <87307821+ritsth@users.noreply.github.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
There are already utilities to import tensor data from NumPy.
With DLPack standard, it will work across many tensor providers.
### What changes are included in this PR?
- Bind `FixedShapedTensorArray::FromTensor` in Python
- Add `ImportArrayVersionedFromDLPack` and `ImportTensorVersionedFromDLPack` in C++
- Add in Python `Array.from_dlpack` and `Tensor.from_dlpack`
Similar to the export, multidimensional tensor import require an explicit step through `Tensor`.
- No `Array::FromTensor` or even `FixedSizeListArray::FromTensor`. IMHO it does not feel as necessary in this direction but open to anyone's take one it.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes, new APIs.
* GitHub Issue: #39295
Lead-authored-by: AntoinePrv <AntoinePrv@users.noreply.github.com>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Deprecate Tensor and SparseTensor messages in the IPC specification.
Deprecate C++ APIs for reading and writing Tensor and SparseTensor IPC messages.
Deprecating IPC support for Tensor and SparseTensor was decided in https://lists.apache.org/thread/w7nvhjyks3p3smw5kq4ylv0nkwxq2s2q
### Are these changes tested?
By existing CI tests.
### Are there any user-facing changes?
No.
* GitHub Issue: #51100
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
While setting up the C++ build on macOS, I encountered an IOError during testing because ARROW_TEST_DATA and PARQUET_TEST_DATA were not set. The current documentation lacks a clear warning that these exports are mandatory for a successful test pass.
### What changes are included in this PR?
I added a .. note:: block to building.rst clarifying the need to initialize submodules and export absolute paths for test data folders.
### Are these changes tested?
Yes, I built the documentation locally using Sphinx (make html) to verify the formatting renders correctly.
### Are there any user-facing changes?
Yes, this updates the developer-facing documentation for C++ building.
Authored-by: Meher <mehergupta06@gmail.com>
Signed-off-by: Nic Crane <thisisnic@gmail.com>
### Rationale for this change
Example code did not compile because it used dot (`.`) instead of arrow (`->`) in C++ code.
### What changes are included in this PR?
One C++ example code in a documentation to have dot (`.`) replaced by arrow (`->`).
### Are these changes tested?
Manually, when I've tried to copy-and-paste example, and it did not compile.
### Are there any user-facing changes?
One external example will change.
Authored-by: alexeyroytman <33158117+alexeyroytman@users.noreply.github.com>
Signed-off-by: Nic Crane <thisisnic@gmail.com>
### Rationale for this change
Three links in the docs return 404. Each one is the reference target for advice the page is giving, so following it leaves the reader with nothing:
- the checkstyle config a Java contributor is told to follow,
- the conda install guide in the Crossbow setup instructions,
- the ROCm reference in the C device data interface spec.
### What changes are included in this PR?
Three URL strings.
- `developers/java/development.rst`: `apache/arrow/blob/main/java/dev/checkstyle/checkstyle.xml` becomes `apache/arrow-java/blob/main/dev/checkstyle/checkstyle.xml`. The file moved with the Java split and the `java/` path segment went with it; it is 10,851 bytes on `arrow-java@ main`. This was the last remaining `apache/arrow/.../java/...` link under `docs/source`.
- `developers/continuous_integration/crossbow.rst`: `conda.io/docs/user-guide/install/index.html` becomes `docs.conda.io/projects/conda/en/latest/user-guide/install/index.html`.
- `format/CDeviceDataInterface.rst`: the `_ROCm:` target becomes `amd.com/en/products/software/rocm.html`, since AMD retired `en/graphics/servers-solutions-rocm`.
### Are these changes tested?
Not by the test suite; nothing outside `docs/source` changed and no code is touched. Each old URL was requested and returns 404, each new one returns 200, and the two GitHub paths were checked through the contents API rather than the HTML pages, because github.com rate-limits a bulk sweep and a 429 is indistinguishable from a 404. I did not build the docs locally.
### Are there any user-facing changes?
Documentation only. Three links that were broken now resolve.
Disclosure: prepared with AI assistance (Claude, via Claude Code), per the AI-generated Code Guidance in the contributing overview. I reviewed the diff; it introduces no third-party material.
* GitHub Issue: #51148
Authored-by: Roshan Ramani <roshanramani.dev@gmail.com>
Signed-off-by: Nic Crane <thisisnic@gmail.com>
### Rationale for this change
The current parquet section is one-page and covering lots of topics. As per https://github.com/apache/arrow/issues/46601 the parquet section is most visited and would therefore need a bit of maintenance/content update.
### What changes are included in this PR?
- Parquet documentation is split in 4 different subpages (index.rst, parquet.rst, parquet_datasets.rst and parquet_encryption.rst).
- Some additional content is added
### Are these changes tested?
Via the docs preview
### Are there any user-facing changes?
No
* GitHub Issue: #47686
Lead-authored-by: AlenkaF <frim.alenka@gmail.com>
Co-authored-by: Alenka Frim <AlenkaF@users.noreply.github.com>
Co-authored-by: Adam Reeve <adreeve@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
Require Numpy 2.0 to support newer features, allow newer numpy API/ABI
### What changes are included in this PR?
- Numpy 2.0+ requirement
- Pandas 2.2.2+ requirement (first wheel compatible with numpy 2.0)
- Pyodide 0.27.1 usage (numpy 2.0 api)
### Are there any user-facing changes?
Yes, in contrast to pyarrow 25, after merging this PR pyarrow 26 can't be used with Numpy 1.x or pandas 2.0-2.2.1
**This PR includes breaking changes to public APIs.**
This PR was created using GPT-5.6-Sol-xhigh, every line read & reviewed by me.
* GitHub Issue: #48473
Lead-authored-by: Adam Lippai <adam@rigo.sk>
Co-authored-by: Ádám Lippai <adam@rigo.sk>
Co-authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Fix#46646 by replacing the `lintr` pre-commit hook (which requires system R/Rscript to bootstrap, causing hard failures when R isn't installed, end even if installed still gives errors when using conda (https://github.com/apache/arrow/issues/47997)) with jarl (https://jarl.etiennebacher.com/, a Rust-native R linter distributed as a standalone binary or as a python package).
### What changes are included in this PR?
- `.pre-commit-config.yaml`: swapped the local `lintr` hook for `etiennebacher/jarl-pre-commit` (jarl-check, rev 0.5.0)
- `r/jarl.toml`: new config file, excludes the same generated files that `r/.lintr` and `r/air.toml` already exclude
- NOTE: for now I did not remove `r/.lintr` in case people would use that locally or through their IDE, so it might be useful to keep. But for our CI it could be removed.
- `.github/workflows/dev.yml`: dropped r-base from the CI apt install and the now-unused `~/.local/share/renv/cache` from the pre-commit cache path, since no pre-commit hook needs R anymore
- `r/vignettes/developers/workflow.Rmd` and `docs/source/developers/guide/step_by_step/styling.rst`: updated references from lintr to jarl
At the moment, running `jarl check` gives 42 pre-existing errors (13 auto-fixable), so CI will be failing at the moment.
Those should be fixed (or temporarily suppressed) in a next commit. EDIT: for now I added those to the configuration as ignored rules, so CI would be green. Those can be fixed as a follow-up issue.
Also note the differences between lintr and jarl listed at https://github.com/apache/arrow/issues/46646#issuecomment-5253845795. Especially no cyclocomp (complexity) or object_length equivalent exists in jarl, so we have to see if we are fine with dropping those checks.
Code changes are largely generated by Claude, I did review everything, but I am no R developer ..
### Are these changes tested?
pre-commit runs on CI
### Are there any user-facing changes?
No
* GitHub Issue: #47937
* GitHub Issue: #46646
Authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Nic Crane <thisisnic@gmail.com>
Summary:
- Document that quoted numeric CSV values are still subject to type inference.
- Add a doctest-backed example showing how `ConvertOptions.column_types` preserves leading zeros.
- AI assistance was used for issue triage and drafting this documentation example.
Validation:
- `/tmp/arrow-doccheck-46853/bin/python -m pytest --doctest-glob="*.rst" docs/source/python/csv.rst -q` — passed, `1 passed`
- `git diff --check HEAD~1 HEAD` — passed
* GitHub Issue: #46853
Authored-by: kalyanamdewri <kalyanampriyam@gmail.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
Closes#50801
As title, this exposes RecordBatchReaderSourceNodeOptions to Python. With it, a hash join can build its hash table once and stream the probe side through as a generator-backed RecordBatchReader, releasing each input chunk as it is consumed, instead of materializing the full probe table up front (table_source) or pinning all fragments for the plan's lifetime (dataset scan).
### Are these changes tested?
Yes
### Are there any user-facing changes?
Yeah, users can pass RecordBatchReaderSourceNodeOptions into Declaration, but its not a breaking change
* GitHub Issue: #50801
Lead-authored-by: You-Cheng Lin <106612301+owenowenisme@users.noreply.github.com>
Co-authored-by: You-Cheng Lin <mses010108@gmail.com>
Signed-off-by: Rossi Sun <zanmato1984@gmail.com>
### Rationale for this change
When working with patch releases currently the maintenance branch used is not correct and the scripts for 02-source and 03-binary fail to create the PR or find it.
### What changes are included in this PR?
Use the correct maintenance branch format taking into account the differences between major releases and patch releases.
Updated release documentation to make it clearer that only the `maint-X.Y.x` branch is used for patch releases.
### Are these changes tested?
Yes, as I had to cut RC1 for the 25.0.1 release I've been able to validate that they work as expected now.
### Are there any user-facing changes?
No
* GitHub Issue: #50811
Lead-authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Co-authored-by: tadeja <tadeja@users.noreply.github.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Follow-up to https://github.com/apache/arrow/issues/49232#issuecomment-5105290467 / #49590
### What changes are included in this PR?
Remove warnings from `write_feather()`, `read_feather()`, `read_table()`, and `FeatherDataset`.
Warn **only** on writing with `version=1` / reading Feather V1 file.
Update docs to V1-only and keep IPC migration guide.
Additionally corrected notes like `deprecated as of 24.0.0` to `25.0.0` instead.
`DeprecationWarning` in place of current `FutureWarning`.
### Are these changes tested?
Yes, by CI.
### Are there any user-facing changes?
Yes!
Feather V2 APIs no longer emit deprecation warnings.
Reading/writing the legacy Feather V1 format emits `DeprecationWarning` in place of `FutureWarning`.
* GitHub Issue: #50808
Lead-authored-by: Tadeja Kadunc <tadeja.kadunc@gmail.com>
Co-authored-by: tadeja <tadeja@users.noreply.github.com>
Co-authored-by: Rok Mihevc <rok@mihevc.org>
Signed-off-by: Rok Mihevc <rok@mihevc.org>
### Rationale for this change
We need this when we break our Yum repositories.
### What changes are included in this PR?
We can recover Yum repositories too by `dev/release/binary-recover.sh`.
### Are these changes tested?
Yes. I recovered our Yum repositories with this.
### Are there any user-facing changes?
No.
* GitHub Issue: #50674
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
[Pandas 2.0.0](https://pypi.org/project/pandas/2.0.0/) was released on April 3rd 2023.
The last version of Pandas 1 was [Pandas 1.5.3](https://pypi.org/project/pandas/1.5.3/) and it was released on January 19th 2023. That version only supports up to Python 3.11.
We are currently supporting up to Pandas 1.5.2 (first version supporting 3.11).
On the issue we discussed to bump our Pandas support to 2.0.0 but given 2.0.3 is the latest in the 2.0.x, was released on June 2023, only supports up to Python 3.11 and bumping to it allows use to improve even further some of the conditionals we are bumping to Pandas >= 2.0.3
### What changes are included in this PR?
Update minimal support check to Version(2.0.3) and remove all test skips, conditionals and scenarios where we are checking for Pandas < 2.0.3.
We found a couple of minor things
### Are these changes tested?
Yes via CI
### Are there any user-facing changes?
Yes, Pandas < 2 won't be supported.
* GitHub Issue: #48808
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Python 3.10 is End of Support on October 2026. We can drop support on PyArrow 26.0.0.
See: https://endoflife.date/python
### What changes are included in this PR?
Remove support for 3.10. Update minimum required Python version and update some CI jobs to use the minimum Python supported version.
Some CI jobs using default Python 3.10 (Ubuntu 20.04) have been bumped.
### Are these changes tested?
They will be tested via CI and Archery
### Are there any user-facing changes?
Yes in terms of not supporting older versions of Python. This will be a breaking change for some users but not for the API itself
* GitHub Issue: #50410
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Add guidance about handling or avoiding memory bombs in file formats that involve optional compression.
### Are these changes tested?
N/A.
### Are there any user-facing changes?
No.
* GitHub Issue: #50313
Lead-authored-by: Antoine Pitrou <pitrou@free.fr>
Co-authored-by: Antoine Pitrou <antoine@python.org>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: Dewey Dunnington <dewey@dunnington.ca>
Signed-off-by: Antoine Pitrou <antoine@python.org>
In file `docs/source/cpp/memory.rst`:
* Removes a duplicate line
* Minor grammar fix
### Rationale for this change
I believe the statement
> The associated class :class:`arrow::MemoryManager` specifies how to allocate on a given device.
was repeated twice and there was some grammatical mistake in the same paragraph
* GitHub Issue: #50087
Authored-by: OmBiradar <ombiradar04@gmail.com>
Signed-off-by: Bryce Mecum <petridish@gmail.com>
### Rationale for this change
Arrow C++ is deprecating the Feather reader/writer (#49231) so we should update the Python functions too. Feather V2 is exactly the Arrow IPC file format, making the separate `pyarrow.feather` module redundant. Users should migrate to `pyarrow.ipc`.
### What changes are included in this PR?
- Added `FutureWarning` deprecation warnings to all four public APIs in `pyarrow.feather`: `write_feather()`, `read_feather()`, `read_table()`, and `FeatherDataset`
- Extracted `_read_table_internal()` so that `read_feather()` calling `read_table()` internally does not produce double warnings
- Updated `FeatherDataset.read_table()` to use the internal function for the same reason
- Added `.. deprecated:: 24.0.0` directives to all four public API docstrings
- Added module-level `pytestmark` filter in `test_feather.py` to suppress warnings in existing tests
- Added 5 new tests verifying deprecation warnings are emitted correctly (including a no-double-warning test)
- Added `@ pytest.mark.filterwarnings` to 5 test functions in `test_dataset.py` that use feather as a utility
- Updated `feather.rst` with a deprecation notice and migration guide (including a note that `pyarrow.ipc` does not compress by default unlike `feather.write_feather`)
- Updated `formats.rst` to mark the Feather API section as deprecated
### Are these changes tested?
Yes. Five new tests verify that each deprecated function emits exactly one `FutureWarning`, and existing tests continue to pass with module/function-level warning filters.
### Are there any user-facing changes?
Yes — calling `pyarrow.feather.write_feather()`, `read_feather()`, `read_table()`, or `FeatherDataset()` now emits a `FutureWarning` directing users to `pyarrow.ipc` equivalents. Existing functionality is unchanged.
* GitHub Issue: #49232
Lead-authored-by: Piyush Kanti Chanda <piyush.chanda@databricks.com>
Co-authored-by: Isaac
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
Arrow has `atan2` and `logb` but no `hypot`. Computing a Euclidean norm currently needs `sqrt(add(multiply(x, x), multiply(y, y)))`, which can overflow at the `multiply` step even when the result is representable.
### What changes are included in this PR?
Add a `hypot` kernel backed by `std::hypot` (float32/float64, with integer/decimal inputs promoted to float64), the `Hypot()` C++ function, a `FunctionDoc` and `compute.rst` entry, and a Python expression test. `pc.hypot` is auto-exposed from the registry.
### Are these changes tested?
Yes, C++ tests for the basic cases, null/NaN/Inf handling, and an overflow case confirming it doesn't fall back to a naive `sqrt(x*x + y*y)`.
### Are there any user-facing changes?
Yes, a new `hypot` function in C++ and Python. Additive only.
* GitHub Issue: #50197
Lead-authored-by: Shrivas Shankar <shrivasshankar2019@gmail.com>
Co-authored-by: Shrivas Shankar <86219405+shrivasshankar@users.noreply.github.com>
Co-authored-by: Rok Mihevc <rok@mihevc.org>
Signed-off-by: Rok Mihevc <rok@mihevc.org>
### What changes are included in this PR?
Updates documentation and integration tests to reflect improvements in
the dotnet implementation.
### Are these changes tested?
N/A
### Are there any user-facing changes?
N/A
* GitHub Issue: #50285
---------
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
### Rationale for this change
Some implementations are known to produce null variadic buffers in BinaryView arrays, and expose them on the C Data Interface.
Since Arrow C++, and perhaps other implementations, make the wrong assumption that imported variadic buffers cannot be null, clarify the docs.
### Are these changes tested?
N/A.
### Are there any user-facing changes?
No.
* GitHub Issue: #50172
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Allow us to write an arrow list view array into a parquet file and read list view type from parquet.
### What changes are included in this PR?
We will sort the list views based on the offsets before inserting the array in Parquet writer, and generate a size array in Parquet reader.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
* GitHub Issue: #38849
Authored-by: Zehua Zou <zehuazou2000@gmail.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
Raúl Cumplido