### Rationale for this change
Trying to reduce the size of libarrow.so and remove google-cloud-cpp SDK on some builds. Allow for users to plug and play based on requirements and divide our functionality into cleaner modules.
### What changes are included in this PR?
Unconditionally build GCS and the google-cloud-cpp SDK into a different module libarrow_gcs.so outside of libarrow.so.
Update bindings to link against the new libarrow_gcs.so library.
Update the Linux Package jobs to have the new module into a different package.
### Are these changes tested?
Yes via CI
### Are there any user-facing changes?
Yes, users will need to either link against libarrow_s3.so or register using LoadFileSystemFactories
### Was AI used for this PR?
In accordance to the [AI generation guidelines](https://arrow.apache.org/docs/dev/developers/overview.html#ai-generated-code), please disclose below whether and how AI was used in this PR.
**PR code and description written by:**
- [x] Human
- [x] AI
**Reviewed before submission by:**
- [x] Human
- [ ] AI
- [ ] Not reviewed
* GitHub Issue: #54902
Authored-by: Raúl Cumplido <raulcd@apache.org>
Signed-off-by: Raúl Cumplido <raulcd@apache.org>
### Rationale for this change
We often get private security reports for some issues which don't threaten confidentiality or integrity.
For these kinds of issues, we would rather have them reported through the usual public means, as it follows our open development principles more closely, and lessens the required paperwork (no CVE, etc.).
### What changes are included in this PR?
Add a paragraph in the C++ security model about what should be reported as a security vulnerability vs. a regular bug.
### Are these changes tested?
N/A.
### Are there any user-facing changes?
No.
### Was AI used for this PR?
In accordance to the [AI generation guidelines](https://arrow.apache.org/docs/dev/developers/overview.html#ai-generated-code), please disclose below whether and how AI was used in this PR.
**PR code and description written by:**
- [x] Human
- [ ] AI
**Reviewed before submission by:**
- [x] Human
- [ ] AI
- [ ] Not reviewed
* GitHub Issue: #49760
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
Co-authored-by: Dhirhan Kanesalingam <[dhirhan17@ gmail.com](mailto:dhirhan17@ gmail.com)>
With help from : @ emkornfield, @ wgtmac
### Rationale for this change
[Adaptive Lossless Floating-Point (ALP)](https://docs.google.com/document/d/1PlyUSfqCqPVwNt8XA-CfRqsbc0NKRG0Kk1FigEm3JOg/edit?tab=t.0#heading=h.5xf60mx6q7xk)
is designed for floating-point data that commonly represents decimal values.
For these workloads, ALP can provide better compression and faster decoding
than general-purpose compression or existing Parquet encodings.
This PR adds ALP support for Parquet `FLOAT` and `DOUBLE` columns in the Arrow
C++ implementation.
### Specification
- [Original ALP specification](https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit)
- [Parquet `AlpEncoding.md`](https://github.com/apache/parquet-format/blob/main/AlpEncoding.md)
(merged through apache/parquet-format#539; currently in Preview)
### What changes are included in this PR?
This PR adds:
- The core ALP compression and decompression implementation.
- Sampling logic for selecting encoding parameters.
- Page metadata serialization, validation, and decoding.
- Parquet encoders and decoders for `FLOAT` and `DOUBLE`.
- Incremental decoding of ALP pages one vector at a time.
- CMake and Meson build integration.
- C++ documentation describing how to use the encoding.
ALP is opt-in on the write path. It is used only when the writer explicitly
selects `Encoding::ALP` and dictionary encoding is disabled. Arrow does not
currently select ALP automatically based on the input data.
### Are these changes tested?
Yes. Test coverage includes:
- Interoperability with the ALP conformance file in `parquet-testing`.
- `FLOAT` and `DOUBLE` round trips.
- Decimal-like, constant, random, and exception-heavy inputs.
- Empty and all-null pages.
- Configurable vector sizes.
- Truncated or malformed page metadata.
- Invalid offsets, element counts, bit widths, and exception positions.
- Batched and incremental decoding.
### Are there any user-facing changes?
Yes. Arrow C++ can read Parquet pages encoded with ALP. Writers can explicitly
select ALP for supported floating-point columns.
ALP requires reader support and is not selected automatically, so existing
writer behavior remains unchanged unless users opt in.
* GitHub Issue: #48701
Lead-authored-by: Prateek Gaur <prateek.gaur@snowflake.com>
Co-authored-by: Gang Wu <ustcwg@gmail.com>
Signed-off-by: Gang Wu <ustcwg@gmail.com>
### Rationale for this change
Arrow is currently missing remainder and modulo kernels.
### What changes are included in this PR?
Add the kernels `remainder`, `remainder_checked`, `modulo`, and `modulo_checked`, following the terminology in the [divmod proposal](https://lists.apache.org/thread/h3t6nz1ys2k2hnbrjvwyoxkf70cps8sh):
* `remainder` uses truncated (C/C++) semantics — the result has the sign of the dividend, e.g. `remainder(-7, 3) == -1`.
* `modulo` uses floored (Python/R) semantics — the result has the sign of the divisor, e.g. `modulo(-7, 3) == 2`.
Both are supported for integer, floating-point and decimal inputs, and are exposed as `compute::Remainder` / `compute::Modulo` in `api_scalar.h`.
Decimal arguments are promoted to a common scale like `add`, but the result type is resolved by a dedicated resolver rather than reusing `ResolveDecimalAdditionOrSubtractionOutput`: a remainder is always smaller in magnitude than the divisor, so no extra digit is needed for a carry, and the result is `precision = max(p1, p2)`, `scale = s1`. This keeps maximum-precision inputs (`decimal128(38, 0)`, `decimal256(76, 0)`) from overflowing the decimal precision range.
Left as follow-ups: combined divmod #27909, floor division #39386, and duration support.
### Are these changes tested?
Yes. New tests cover truncated vs. floored semantics across signed/unsigned integer, floating-point and decimal types, division by zero, `INT_MIN % -1` overflow, and decimal result-type resolution (same type in/out, mixed precisions and scales, Decimal128/Decimal256 promotion, decimal/integer promotion, and maximum-precision decimals).
### Are there any user-facing changes?
Yes — four new compute functions (`remainder`, `remainder_checked`, `modulo`, `modulo_checked`), the corresponding `compute::Remainder` / `compute::Modulo` C++ APIs, and a new `BasicDecimal256 operator%`. These are additions only; no existing API is changed or removed.
* GitHub Issue: #46901
* xref: https://github.com/apache/arrow/pull/46902,
* xref: https://github.com/apache/arrow/pull/28497
Lead-authored-by: Fangchen Li <fangchen.li@outlook.com>
Co-authored-by: Rok Mihevc <54589+rok@users.noreply.github.com>
Co-authored-by: tadeja <864005+tadeja@users.noreply.github.com>
Signed-off-by: Rok Mihevc <rok@mihevc.org>
### Rationale for this change
`starts_with`, `ends_with` and `match_substring` have no Substrait mapping, so any expression using them fails to serialize:
```python
import pyarrow as pa
import pyarrow.compute as pc
from pyarrow.substrait import serialize_expressions
schema = pa.schema([pa.field("cat", pa.string())])
serialize_expressions([pc.starts_with(pc.field("cat"), "al")], ["f"], schema)
# ArrowNotImplementedError: No conversion function exists to convert the
# Arrow function starts_with to a Substrait call
```
Comparisons, `isin` and arithmetic serialize fine, so this is a per-function gap. It matters for engines that ingest a PyArrow filter through Substrait, where an unmappable function becomes a hard failure rather than a fallback.
See #50988.
### What changes are included in this PR?
Map the three kernels onto `starts_with`, `ends_with` and `contains` from Substrait's `functions_string.yaml`, in both directions.
The signatures do not line up. Substrait passes the pattern as a second argument, while the Arrow kernels are unary and carry it in `MatchSubstringOptions`. So:
* encoding lifts `MatchSubstringOptions::pattern` out into a literal argument
* decoding requires that argument to be a non-null string literal, and returns `NotImplemented` otherwise
Substrait's `case_sensitivity` option maps onto `ignore_case`. `CASE_INSENSITIVE_ASCII` has no Arrow equivalent and returns `NotImplemented`.
Both caveats are documented in `docs/source/cpp/acero/substrait.rst`.
### Are these changes tested?
Yes.
`Substrait.StringMatchExpressionSerialization` round-trips all three functions with and without `ignore_case`. `Substrait.StringMatchExpressionDeserialization` deserializes hand-written Substrait JSON to cover the default (no option) case and the two rejection paths. `test_serializing_string_match_expressions` round-trips each function through `pyarrow.substrait`.
### Are there any user-facing changes?
Yes. `pc.starts_with`, `pc.ends_with` and `pc.match_substring` can now be serialized to Substrait and consumed back, and Acero can consume plans using Substrait's `starts_with`, `ends_with` and `contains`. No existing behaviour changes: these previously raised.
* GitHub Issue: #50988
Authored-by: Jonas Dedden <mail@jonas-dedden.de>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
Trying to reduce the size of `libarrow.so` and remove AWS SDK on some builds. Allow for users to plug and play based on requirements and divide our functionality into cleaner modules.
### What changes are included in this PR?
Unconditionally build S3 and the AWS SDK into a different module `libarrow_s3.so` outside of `libarrow.so`.
Update bindings to link against the new `libarrow_s3.so` library.
Update the Linux Package jobs to have the new module into a different package.
### Are these changes tested?
Yes via CI
### Are there any user-facing changes?
Yes, users will need to either link against `libarrow_s3.so` or register using `LoadFileSystemFactories`
* GitHub Issue: #50194
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Example code did not compile because it used dot (`.`) instead of arrow (`->`) in C++ code.
### What changes are included in this PR?
One C++ example code in a documentation to have dot (`.`) replaced by arrow (`->`).
### Are these changes tested?
Manually, when I've tried to copy-and-paste example, and it did not compile.
### Are there any user-facing changes?
One external example will change.
Authored-by: alexeyroytman <33158117+alexeyroytman@users.noreply.github.com>
Signed-off-by: Nic Crane <thisisnic@gmail.com>
### Rationale for this change
Add guidance about handling or avoiding memory bombs in file formats that involve optional compression.
### Are these changes tested?
N/A.
### Are there any user-facing changes?
No.
* GitHub Issue: #50313
Lead-authored-by: Antoine Pitrou <pitrou@free.fr>
Co-authored-by: Antoine Pitrou <antoine@python.org>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: Dewey Dunnington <dewey@dunnington.ca>
Signed-off-by: Antoine Pitrou <antoine@python.org>
In file `docs/source/cpp/memory.rst`:
* Removes a duplicate line
* Minor grammar fix
### Rationale for this change
I believe the statement
> The associated class :class:`arrow::MemoryManager` specifies how to allocate on a given device.
was repeated twice and there was some grammatical mistake in the same paragraph
* GitHub Issue: #50087
Authored-by: OmBiradar <ombiradar04@gmail.com>
Signed-off-by: Bryce Mecum <petridish@gmail.com>
### Rationale for this change
Arrow has `atan2` and `logb` but no `hypot`. Computing a Euclidean norm currently needs `sqrt(add(multiply(x, x), multiply(y, y)))`, which can overflow at the `multiply` step even when the result is representable.
### What changes are included in this PR?
Add a `hypot` kernel backed by `std::hypot` (float32/float64, with integer/decimal inputs promoted to float64), the `Hypot()` C++ function, a `FunctionDoc` and `compute.rst` entry, and a Python expression test. `pc.hypot` is auto-exposed from the registry.
### Are these changes tested?
Yes, C++ tests for the basic cases, null/NaN/Inf handling, and an overflow case confirming it doesn't fall back to a naive `sqrt(x*x + y*y)`.
### Are there any user-facing changes?
Yes, a new `hypot` function in C++ and Python. Additive only.
* GitHub Issue: #50197
Lead-authored-by: Shrivas Shankar <shrivasshankar2019@gmail.com>
Co-authored-by: Shrivas Shankar <86219405+shrivasshankar@users.noreply.github.com>
Co-authored-by: Rok Mihevc <rok@mihevc.org>
Signed-off-by: Rok Mihevc <rok@mihevc.org>
### Rationale for this change
Allow us to write an arrow list view array into a parquet file and read list view type from parquet.
### What changes are included in this PR?
We will sort the list views based on the offsets before inserting the array in Parquet writer, and generate a size array in Parquet reader.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
* GitHub Issue: #38849
Authored-by: Zehua Zou <zehuazou2000@gmail.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
The documentation example for TupleRangeFromTable was missing the cast_options and ctx parameters, so it didn't match the actual function signature
### What changes are included in this PR?
Add them
### Are these changes tested?
No
### Are there any user-facing changes?
Better docs
* GitHub Issue: #17081
Authored-by: Nic Crane <thisisnic@gmail.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Now that we have a general security model for the Arrow specs, add a security model for Arrow C++ specifically, meant to describe appropriate API usage.
### Are these changes tested?
N/A.
### Are there any user-facing changes?
No.
* GitHub Issue: #49274
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Closes#49509
### What changes are included in this PR?
Docs formatting/typos corrected.
### Are these changes tested?
Yes, on fork branch.
### Are there any user-facing changes?
No, just corrected formatting/typos in docs.
* GitHub Issue: #49509
Authored-by: Tadeja Kadunc <tadeja.kadunc@gmail.com>
Signed-off-by: Rok Mihevc <rok@mihevc.org>
### Rationale for this change
Switch to std::chrono for MSVC to be able to use the system-provided timezone automatically on Windows.
### What changes are included in this PR?
This adds `chrono_internal.h` that uses C++20 std::chrono timezone/calendar APIs on compilers with support (MSVC only for now) and falls back to vendored `date.h` otherwise.
### Are these changes tested?
Partially tested locally and partially to be tested on CI.
### Are there any user-facing changes?
Yes, Windows users will no longer need to install the IANA tzdb (see [instructions here](https://arrow.apache.org/docs/cpp/build_system.html#runtime-dependencies) and [here](https://arrow.apache.org/docs/python/install.html#tzdata-on-windows)). We possibly have tzdb download set up in CI too and should update it appropriately.
* GitHub Issue: #48593
Authored-by: Rok Mihevc <rok@mihevc.org>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
The atfork handlers we register in Arrow C++ are generally useful if the Arrow APIs are meant to be used from the child process, but they also have the unfortunate effect of executing non-async-signal-safe code in the child process even if Arrow is not be used there. That is [not allowed by POSIX](https://pubs.opengroup.org/onlinepubs/9699919799/functions/pthread_atfork.html) if the parent process is multi-threaded.
There are situations where fork() is only called just before exec(), and therefore it is not necessary to run any atfork handler.
### What changes are included in this PR?
1. Add a `GetEnvVarInteger` utility function to automate parsing of a numeric environment variable
2. Remove hard-coded size limitations for environment variable values on Windows
3. Add basic unit tests for our APIs for getting and setting environment variables
4. Add an environment variable `ARROW_REGISTER_ATFORK` to disable the registration of atfork handlers at runtime
### Are these changes tested?
The new environment variable cannot be easily tested automatically, so I've checked it manually.
### Are there any user-facing changes?
No, only a new feature.
* GitHub Issue: #49146
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Writing support for Parquet bloom filter has been added so the doc is out of date.
### What changes are included in this PR?
Update the doc to reflect the status that bloom filter is fully supported but not integrated with data read api.
### Are these changes tested?
N/A
### Are there any user-facing changes?
No
* GitHub Issue: #34785
Authored-by: Gang Wu <ustcwg@gmail.com>
Signed-off-by: Gang Wu <ustcwg@gmail.com>
**Description:**
This pull request corrects minor typos in the documentation files:
- In `docs/source/cpp/compute.rst`, the word "invokable" is now spelled correctly.
- In `docs/source/developers/cpp/acero.rst`, the word "initially" is now spelled correctly.
These changes improve the clarity and professionalism of the documentation without affecting any code functionality.
Authored-by: kilavvy <140459108+kilavvy@users.noreply.github.com>
Signed-off-by: Nic Crane <thisisnic@gmail.com>
### Rationale for this change
We decided to migrate Arrow C++ to C++20 in this discussion: https://lists.apache.org/thread/48zlj0dn2y0f53y2k37qsr90y781wfnj
### What changes are included in this PR?
1. Build configuration updates (CMake files etc.) to build with C++20 instead of C++17
2. C++-level fixes to ensure compilation succeeds:
* Workarounds for deprecation of atomic access on `std::shared_ptr<T>` (the replacement `std::atomic<std::shared_ptr<T>>` is unfortunately not supported in all standard library implementations)
* Workaround for [Abseil ABI issues](https://github.com/abseil/abseil-cpp/issues/1624)
* Trivial update to `arrow/util/string.h` to call a C++20 API (to validate that C++20 is actually enabled)
3. CI configuration updates to get enough C++20 support on the various platforms:
* Bump default clang and LLVM versions to 18
* Use AlmaLinux 10 for release verification (AlmaLinux 8 is still tested on other CI builds)
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes, Arrow C++ will now require a C++20-compliant compiler.
* GitHub Issue: #45885
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Columns can b encrypted with individual keys. For this, the column name have to be set in `EncryptionConfiguration::column_keys`. This poses the following challenges for columns with nested fields like `MapType`, `ListType`, and `StructType`. Encrypting a column of such type requires providing an encryption key for all nested (leaf) fields. Ideally, the column name should be sufficient (as it is for any other data type) to encrypt all nested fields.
### What changes are included in this PR?
The column name can be used to encrypt all nested fields of `StrutType`, `MapType`, and `ListType` columns with the same encryption key. The current column naming scheme can still be used for backward compatibility.
### Are these changes tested?
Tested in C++ and Python.
### Are there any user-facing changes?
Column encryption can be configured with less code and more intuitive naming.
Documentation and examples updated.
Fixes#41246.
* GitHub Issue: #41246
Authored-by: Enrico Minack <github@enrico.minack.dev>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
The docs should be more precise in describing when and how Arrow exports traces.
It should also generally discourage the use of `ARROW_WITH_OPENTELEMETRY` in executables as it modifies global OTel SDK resources.
### What changes are included in this PR?
More precise language in the docs.
### Are these changes tested?
No.
### Are there any user-facing changes?
Yes, docs.
* GitHub Issue: #48046
Authored-by: Johan Peltenburg <johan.peltenburg+code@gmail.com>
Signed-off-by: Bryce Mecum <petridish@gmail.com>
### Rationale for this change
As described in #44345, `Decimal32`/`Decimal64` have been implemented but Parquet has poor support. This change allows to write `Decimal32`/`Decimal64` into Parquet file the same way as `Decimal128`/`Decimal256` and to read `Decimal32`/`Decimal64` from an existing Parquet file.
### What changes are included in this PR?
1. Support writing `Decimal32`/`Decimal64` as `INT32`/`INT64`/`BYTE_ARRAY`/`FIXED_LEN_BYTE_ARRAY` into Parquet file.
2. Support reading Parquet column with logical type Decimal. Either reading type from metadata or infering Arrow Decimal type is supported.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes. A flag named `smallest_decimal_enabled_` is added in `ArrowReaderProperties`. To maintain backward compatibility, only when the flag is `true`, Arrow will infer Decimal with small precision to `Decimal32`/`Decimal64` instead of `Decimal128`.
* GitHub Issue: #44345
Authored-by: Zehua Zou <41586196+HuaHuaY@users.noreply.github.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Parameterizing the comparison of `metadata` and `arrow::Schema` via `arrow::EqualOptions`.
### What changes are included in this PR?
* Added two attributes to `arrow::EqualOptions` for parameterizing the comparison of `metadata` and `schema`.
### Are these changes tested?
Yes, I ran the relevant unit tests.
### Are there any user-facing changes?
* Added `use_schema()` to `arrow::EqualOptions`.
* Added `use_metadata()` to `arrow::EqualOptions`.
* Added a new overload for `arrow::RecordBatch::Equals` that accepts only another `arrow::RecordBatch` and an `arrow::EqualOptions` instance.
* GitHub Issue: #46835
Lead-authored-by: Arash Andishgar <arashandishgar1@gmail.com>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
The table in the "Other available packages" section of build_system.rst was refactored for improved readability.
### What changes are included in this PR?
Refactored the table in the "Other available packages" section of build_system.rst
### Are these changes tested?
Verified the preview of docs
### Are there any user-facing changes?
No.
* GitHub Issue: #47157
Lead-authored-by: Vidhya Ravikumar <vidhyarbiji@gmail.com>
Co-authored-by: vidhya.ravikumar <vidhya.ravikumar@ibm.com>
Co-authored-by: Bryce Mecum <petridish@gmail.com>
Signed-off-by: Bryce Mecum <petridish@gmail.com>
### Rationale for this change
We want the ThreadPool default capacity to follow the CPU affinity set by the user, if any.
For example:
```console
$ python -c "import pyarrow as pa; print(pa.cpu_count())"
24
$ taskset -c 5,6,7 python -c "import pyarrow as pa; print(pa.cpu_count())"
3
```
### What changes are included in this PR?
- Implement and expose CPU affinity detection as a utility function in `arrow/io_util.h`; on non-Linux platform, it returns `Status::NotImplemented`
- Use CPU affinity count, if available, to choose the default ThreadPool capacity
(note: based on original changes by Zihan Qi in PR #46034)
### Are these changes tested?
By unit tests on CI, and by hand locally.
### Are there any user-facing changes?
ThreadPool capacity now follows CPU affinity settings on Linux.
* GitHub Issue: #45860
Lead-authored-by: AntoinePrv <AntoinePrv@users.noreply.github.com>
Co-authored-by: Zihan Qi <zihan.qi@tum.de>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
We decided to move Skyhook to its own repository here: https://lists.apache.org/thread/f5ndxf0o6v4wbzgoy6nsrj4xkn34f380
We've created https://github.com/apache/arrow-skyhook and extract Skyhook related commits to the repository.
### What changes are included in this PR?
Remove Skyhook from the main repository
### Are these changes tested?
CI is validated.
### Are there any user-facing changes?
Yes, Skyhook is removed from the main repository.
* GitHub Issue: #47225
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
We were missing documentation for some of the newer CMake packages and targets we've split out. This adds documentation for those (acero, compute).
### What changes are included in this PR?
- Updates the table in build_system.rst to include ArrowAcero and ArrowCompute
### Are these changes tested?
No, will render in CI though for others to see.
### Are there any user-facing changes?
No.
* GitHub Issue: #47153
Authored-by: Bryce Mecum <petridish@gmail.com>
Signed-off-by: Bryce Mecum <petridish@gmail.com>
### Rationale for this change
As documented in #35166, when Arrow performs a sum, product, or mean of an array of type `decimalX(P, S)`, it returns a scalar of type `decimalX(P, S)`. This is true even if the aggregate does not fit in the specified precision. For instance, a sum of two `decimal128(1, 0)`'s such as `1 + 9` is a `decimal128(2, 0)`. But (in Python):
```python
import pyarrow as pa
from decimal import Decimal
arr = pa.array([Decimal("1"), Decimal("9")], type=pa.decimal128(1, 0))
assert arr.sum().type == pa.decimal128(1, 0)
```
This is recognized in the rules for _binary_ addition and multiplication of decimals (see footnote 1 in [this section](https://arrow.apache.org/docs/cpp/compute.html#arithmetic-functions)), but this does not apply to array aggregates.
In #35166 I did a bit of research following a question from @ westonpace , and it seems that there's no standard approach to this across DBMS's, but a common solution is to set the precision of the result of a sum to the maximum possible precision of the underlying type. That is, a sum of `decimal128(1, 0)`'s becomes a `decimal128(38, 0)`.
However, products and means differ further. For instance, in both instances, duckdb converts a decimal to a double, which makes sense as the precision of the product of an array of decimals would likely be huge, e.g., an array of size N with precision 2 decimals would have precision at least 2^N.
This PR implements the minimum possible change: replace the return types of the sum aggregate of `decimal128(P, S)` to `decimal128(38, S)`, `decimal256(P, S)` to `decimal256(76, S)`, `decimal32(P, S)` to `decimal32(9, S)`, and `decimal64(P, S)` to `decimal64(18, S)`.
### What changes are included in this PR?
- [x] Update C++ kernels to support the change
- [x] Update docs to reflect the change
- [x] Fix tests in the languages that depend on the C++ engine
- [x] Determine if there are other languages which do _not_ depend on the C++ engine which should also be updated
### Are these changes tested?
They are tested in the following implementations:
- [x] Python
- [x] C++
### Are there any user-facing changes?
Yes. This changes the return type of a scalar aggregate of decimals.
**This PR includes breaking changes to public APIs.**
Specifically, the return type of a scalar aggregate of decimals changes. This is unlikely to break downstream applications as the underlying data has not changed, but if an application relies on the (incorrect!) type information for some reason, it would break.
* GitHub Issue: #35166
Lead-authored-by: Kevin H Wilson <khwilson@gmail.com>
Co-authored-by: Rossi Sun <zanmato1984@gmail.com>
Co-authored-by: Kevin Wilson <khwilson@gmail.com>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Signed-off-by: Rossi Sun <zanmato1984@gmail.com>
### What does this PR do?
Adds a new compute kernel `utf8_zero_fill` to Arrow Compute module. This kernel zero-pads UTF-8 strings while preserving leading signs.
### Why is it needed?
Python's `str.zfill()` behavior is useful and expected in many data cleaning scenarios. Arrow lacked a direct equivalent.
### Implementation details
- Introduces `Utf8ZeroFillTransform` with dedicated `ZeroFillOptions`
- Registers `utf8_zero_fill` as a compute kernel
- Adds Python bindings and a clean alias: `utf8_zfill = utf8_zero_fill`
- Includes sign-aware padding logic
### Are these changes tested?
- C++ unit tests: `TestStringKernels, Utf8ZeroFill`
- Python tests: `test_utf8_zfill()`
Ran both tests
* GitHub Issue: #46683
Authored-by: iabhi4 <iamonecool@gmail.com>
Signed-off-by: Antoine Pitrou <pitrou@free.fr>
### Rationale for this change
Half-float is currently supported by none of the arithmetic compute functions.
### What changes are included in this PR?
Add half-float support for a small subset of arithmetic compute functions: `is_nan`, `is_inf`, `is_finite`, `negate`, `negate_checked`, `sign`.
Also improve the kernel generation infrastructure so as to make further half-float kernels easier to implement.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
* GitHub Issue: #45083
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Arrow is quite a heavy dependency and some users don't need all the tools we provide bundled with libarrow. Moving Arrow Compute to its own shared library allows users installations that better suit their needs having smaller memory footprint if necessary.
It might also help some users adding new kernels into an existing Arrow without recompiling it.
### What changes are included in this PR?
- Move all the Arrow Compute kernel functions to a new `ArrowCompute` shared library (`libarrow_compute.so`).
- Create a new API to Initialize arrow compute registering the existing Kernels into the `FunctionRegistry`
- Update Python/R/CGLib bindings to automatically register the Compute kernels transparently to the user.
- Update Linux Packaging to provide the new arrow-compute library.
- Update documentation with new requirements to call `arrow::compute::Initialize()`
### Are these changes tested?
Yes on all CI jobs.
### Are there any user-facing changes?
Yes. The Arrow compute functions will be provided as a different library. Any user using Arrow Compute from C++ directly will require a call to `arrow::compute::Initialize()` in order for the functions and kernels to be registered
**This PR includes breaking changes to public APIs.**
* GitHub Issue: #25025
Lead-authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Co-authored-by: Bryce Mecum <petridish@gmail.com>
Co-authored-by: Sutou Kouhei <kou@clear-code.com>
Co-authored-by: Rossi Sun <zanmato1984@gmail.com>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
We now support more types but the documentation suggested that some weren't supported.
### What changes are included in this PR?
Documentation was updated to reflect the status of supported types.
### Are these changes tested?
No code changes!
### Are there any user-facing changes?
No
* GitHub Issue: #46599
Lead-authored-by: Dewey Dunnington <dewey@wherobots.com>
Co-authored-by: Dewey Dunnington <dewey@fishandwhistle.net>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Signed-off-by: Dewey Dunnington <dewey@wherobots.com>
### Rationale for this change
The docs don't build cleanly for me locally and some warnings and errors that are consequential are being ignored. Some of these issues ended up being as severe as entirely missing tables.
### What changes are included in this PR?
- Fixes for each issue I ran into. Each change is a commit with whatever extra detail I needed included in the commit body.
- This also includes an auto-update of our Doxyfile which silences a number of deprecation warnings. I took a look at the diff and it seemed pretty safe and the docs look good to me locally. We can do a preview in CI first though.
### Are these changes tested?
Yes, each change was tested locally.
### Are there any user-facing changes?
No.
* GitHub Issue: #46520
Authored-by: Bryce Mecum <petridish@gmail.com>
Signed-off-by: Bryce Mecum <petridish@gmail.com>
### Rationale for this change
Parquet has almost no support for LargeBinary and BinaryView data:
* on writing, those types are not supported at all
* on reading, data is decoded as regular Binary data with automatic chunking; if the stored Arrow schema points to a LargeBinary field, the data is later cast to that type
### What changes are included in this PR?
* Refactor the BYTE_ARRAY column decoders to allow decoding directly into a LargeBinaryBuilder or a BinaryViewBuilder
* Add a `binary_type` setting to `ArrowReaderProperties` to change the type that BYTE_ARRAY columns are decoded to by default
* Support reading Parquet GEOMETRY types with a LargeBinary or BinaryView storage
* Add benchmarks for reading and writing BinaryView data from/to Parquet
* Add the corresponding Python bindings
### Are these changes tested?
Yes.
### Are there any user-facing changes?
New APIs and improved functionality.
* GitHub Issue: #43041
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
The [Array](https://arrow.apache.org/docs/cpp/api/array.html) section of the API Doc
does not explain `FlatArray`, `PrimitiveArray`, and `RunEndEncodedArray` .
### What changes are included in this PR?
Add `FlatArray`, `PrimitiveArray`, and `RunEndEncodedArray` API docs
### Are these changes tested?
No. (Document only)
### Are there any user-facing changes?
No.
* GitHub Issue: #46537
Authored-by: Hiroyuki Sato <hiroysato@gmail.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### What changes are included in this PR?
1. Move current Acero Developer's Guild under user doc to where under developer docs;
2. Make it the main doc enclosing elaborating docs like Swiss table.
### Are these changes tested?
No need.
### Are there any user-facing changes?
None.
* GitHub Issue: #46475
Lead-authored-by: Rossi Sun <zanmato1984@gmail.com>
Co-authored-by: Bryce Mecum <petridish@gmail.com>
Signed-off-by: Rossi Sun <zanmato1984@gmail.com>
### Rationale for this change
Data retrieved via IPC is expected to provide memory-aligned arrays, but data retrieved via C++ Flight client is mis-aligned. Datafusion (Rust), which requires data type-specific alignment, cannot handle such data: #43552.
https://arrow.apache.org/docs/format/Columnar.html#buffer-alignment-and-padding
### What changes are included in this PR?
This adds option `arrow::ipc::IpcReadOptions.ensure_alignment` of type `arrow::ipc::Alignment` to configure how RecordBatch array buffers decoded by IPC are realigned. It supports no realignment (default), data type-specific alignment and 64-byte alignment.
Implementation mirrors that of [`align_buffers` in arrow-rs](https://github.com/apache/arrow-rs/blob/3293a8c2f9062fca93bee2210d540a1d25155bf5/arrow-data/src/data.rs#L698-L711) (https://github.com/apache/arrow-rs/pull/4681).
### Are these changes tested?
Configuration flag tested in unit test. Integration test with Flight server.
Manually end-to-end tested that memory alignment fixes issue with reproduction code provided in #43552.
### Are there any user-facing changes?
Adds option `IpcReadOptions.ensure_alignment` and enum type `Alignment`.
* GitHub Issue: #32276
Lead-authored-by: Enrico Minack <github@enrico.minack.dev>
Co-authored-by: David Li <li.davidm96@gmail.com>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Signed-off-by: David Li <li.davidm96@gmail.com>
### Rationale for this change
These functions are generally useful and stable so it would be a good idea to clearly include them in the public API. I'm starting here with just the basic ones in order to make the PR small. See https://github.com/apache/arrow/issues/45908 for more information.
### What changes are included in this PR?
- Moves `ArrayFromJSON`, `ChunkedArrayFromJSON`, `DictArrayFromJSON`, `ScalarFromJSON`, `DictScalarFromJSON` from `arrow::ipc::internal` namespace to `arrow::json` so it's clearer they're part of the public API and that they're more useful than just for IPC
- Renames each of the above from `{Array,...}FromJSON` to `{Array,...}FromJSONString` to avoid confusion between these helpers and the main JSON(L) reader
- Renames `arrow/util/json_simple.{h,cc}` to `arrow/json/from_string.{h,cc}` both because of the namespace jump but also because the filename is more clear.
- Adds User Guide and adds a listing in the API docs for moved functions
### Are these changes tested?
Yes.
### Are there any user-facing changes?
This expands the scope of our public API but does not break any existing public APIs though it's possible users
* GitHub Issue: #45908
Lead-authored-by: Bryce Mecum <petridish@gmail.com>
Co-authored-by: Benjamin Kietzman <bengilgit@gmail.com>
Co-authored-by: Enrico Minack <github@enrico.minack.dev>
Signed-off-by: Bryce Mecum <petridish@gmail.com>
### What changes are included in this PR?
Add cpp developer documentation for row table, making it under the compute category.
### Are these changes tested?
No need.
### Are there any user-facing changes?
None.
* GitHub Issue: #46209
Lead-authored-by: Rossi Sun <zanmato1984@gmail.com>
Co-authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Co-authored-by: Bryce Mecum <petridish@gmail.com>
Signed-off-by: Rossi Sun <zanmato1984@gmail.com>
### Rationale for this change
Improving docs by removing typos
### What changes are included in this PR?
Fix typos in rst files
### Are these changes tested?
Documentation-only changes
### Are there any user-facing changes?
Documentation-only changes
Authored-by: omahs <73983677+omahs@users.noreply.github.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
Currently, the Arrow C++ CSV converter does not support parsing strings into `duration` types. This limits CSV ingestion capabilities when handling datasets with time-based intervals represented as numeric strings (e.g., `1000`, `2000000`). This PR adds support for parsing such strings into Arrow's `DurationType`.
Note: Human-readable duration formats such as `1s`, `2m`, or `3h` are not supported in this PR. Support for those formats may be considered in a future enhancement.
### What changes are included in this PR?
- Added `DurationValueDecoder` using `StringConverter<DurationType>`
- Registered support in both standard and dictionary converters
- Added unit tests covering:
- Basic parsing across all time units (s, ms, µs, ns)
- Null and custom null values
- Whitespace handling and error cases
### Are these changes tested?
Yes, conversion logic is fully covered by new tests in `converter_test.cc`.
### Are there any user-facing changes?
Yes, users can now convert duration strings in CSV files to Arrow `duration` arrays by specifying the appropriate schema type.
* GitHub Issue: #40278
Lead-authored-by: Zihan Qi <zihan.qi@tum.de>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
There are no Extension Array/Type documentation in the [API Reference page](https://arrow.apache.org/docs/cpp/api.html).
### What changes are included in this PR?
Add Extension Array / Type API documents.
### Are these changes tested?
No. (Add document only)
### Are there any user-facing changes?
No.
* GitHub Issue: #46318
Authored-by: Hiroyuki Sato <hiroysato@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
When reading a parquet dataset where the physical schema has inconsistent column order for top level columns Arrow can still read the table. However it cannot handle similar inconsistency in the order of struct fields and raises errors like
```
Traceback (most recent call last):
File "/home/tomnewton/arrow/cpp/src/arrow/compute/example.py", line 30, in <module>
table_read = pq.read_table(
File "/home/tomnewton/.local/lib/python3.8/site-packages/pyarrow/parquet/core.py", line 1843, in read_table
return dataset.read(columns=columns, use_threads=use_threads,
File "/home/tomnewton/.local/lib/python3.8/site-packages/pyarrow/parquet/core.py", line 1485, in read
table = self._dataset.to_table(
File "pyarrow/_dataset.pyx", line 562, in pyarrow._dataset.Dataset.to_table
File "pyarrow/_dataset.pyx", line 3804, in pyarrow._dataset.Scanner.to_table
File "pyarrow/error.pxi", line 155, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
pyarrow.lib.ArrowTypeError: struct fields don't match or are in the wrong order: Input fields: struct<sub_column0: int32, sub_column1: int32> output fields: struct<sub_column1: int32, sub_column0: int32>
```
This issue is quite closely related to https://github.com/apache/arrow/issues/44555
### What changes are included in this PR?
Change the implementation of `CastStruct::Exec` to be primarily based on the column names rather than the column order. Each input field can still only be used once and if there are many input fields with the same name they will be used in the order of the input fields.
Alternatives I considered:
Implement this behaviour in the same place as the equivalent logic for top level columns at https://github.com/apache/arrow/blob/6252e9ceeb0f8544c14f79d895a37ac198131f88/cpp/src/arrow/compute/expression.cc#L669. This would effect parquet scans without modifying cast behaviour.
I decided against this because I want this behaviour to work recursively e.g. if there are nested structs or structs inside arrays of maps, etc.
Have a config option to switch between field name and field order based matching. This would make things more explicit but there would be 2 code paths to maintain instead of one.
IMO the logic I've implemented where each input can only be used once and column order is maintained for duplicate names achieves what I want without breaking any usecases that rely on column order and without too much complexity. So I decided a config option was not necessary.
### Are these changes tested?
Yes. A few new assertions were added but mostly it was a case of adjusting the expected behaviour on existing tests.
### Are there any user-facing changes?
Yes. Casts that require changing the struct field order will now succeed without error.
* GitHub Issue: #45028
Authored-by: Thomas Newton <thomas.w.newton@gmail.com>
Signed-off-by: David Li <li.davidm96@gmail.com>
### Rationale for this change
Moving S3 to a separate module will allow separate build the module for use by builds of arrow which did not include S3 funcionality.
### What changes are included in this PR?
- the factory for S3 filesystems is added to the registry
- an internal function is added which unregisters functions for testing purposes
### Are these changes tested?
A test for just the modularization is added which unregisters built in factories. This way we're definitely testing loading new factories from module libraries.
### Are there any user-facing changes?
No
* GitHub Issue: #40343
Lead-authored-by: Benjamin Kietzman <bengilgit@gmail.com>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Signed-off-by: Benjamin Kietzman <bengilgit@gmail.com>
### Rationale for this change
Allow the `pivot_wider` and `hash_pivot_wider` functions to accept an integral pivot key column, in addition to binary-like.
Since the `key_names` option is a vector of strings, they are cast to the appropriate pivot key type for matching.
### Are these changes tested?
Yes, by new unit tests.
### Are there any user-facing changes?
No.
* GitHub Issue: #45732
Lead-authored-by: Antoine Pitrou <antoine@python.org>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Signed-off-by: Antoine Pitrou <pitrou@free.fr>
### Rationale for this change
Encrypting columns with nested fields with a column key is not trivial since only leaf fields are allowed in the column key map. Documentation emphasizes this fact and provides examples.
### What changes are included in this PR?
This amends the documentation on encryption for C++ and Python.
### Are these changes tested?
Only documentation.
### Are there any user-facing changes?
Only documentation.
* GitHub Issue: #41246
Lead-authored-by: Enrico Minack <github@enrico.minack.dev>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
Raúl Cumplido