### Rationale for this change
This is the sub issue #44748.
* SC2046: Quote this to prevent word splitting
* SC2086: Double quote to prevent globbing and word splitting
* SC2223: This default assignment may cause DoS due to globbing. Quote it
```
In build_arrow.sh line 23:
mkdir -p $ARROW_BUILD_DIR
^--------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
mkdir -p "$ARROW_BUILD_DIR"
In build_arrow.sh line 24:
pushd $ARROW_BUILD_DIR
^--------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
pushd "$ARROW_BUILD_DIR"
In build_arrow.sh line 30:
$ARROW_CMAKE_OPTIONS
^------------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
"$ARROW_CMAKE_OPTIONS"
In build_arrow.sh line 32:
make -j$NPROC
^----^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
make -j"$NPROC"
In build_example.sh line 21:
mkdir -p $EXAMPLE_BUILD_DIR
^----------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
mkdir -p "$EXAMPLE_BUILD_DIR"
In build_example.sh line 22:
pushd $EXAMPLE_BUILD_DIR
^----------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
pushd "$EXAMPLE_BUILD_DIR"
In run_static.sh line 21:
: ${ARROW_DIR:=/arrow}
^------------------^ SC2223 (info): This default assignment may cause DoS due to globbing. Quote it.
In run_static.sh line 22:
: ${EXAMPLE_DIR:=/io}
^-----------------^ SC2223 (info): This default assignment may cause DoS due to globbing. Quote it.
In run_static.sh line 23:
: ${ARROW_BUILD_DIR:=/build/arrow}
^-- SC2223 (info): This default assignment may cause DoS due to globbing. Quote it.
In run_static.sh line 24:
: ${EXAMPLE_BUILD_DIR:=/build/example}
^-- SC2223 (info): This default assignment may cause DoS due to globbing. Quote it.
In run_static.sh line 26:
: ${ARROW_DEPENDENCY_SOURCE:=BUNDLED}
^-- SC2223 (info): This default assignment may cause DoS due to globbing. Quote it.
In run_static.sh line 34:
mkdir -p $ARROW_BUILD_DIR
^--------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
mkdir -p "$ARROW_BUILD_DIR"
In run_static.sh line 35:
pushd $ARROW_BUILD_DIR
^--------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
pushd "$ARROW_BUILD_DIR"
In run_static.sh line 39:
cmake $ARROW_DIR/cpp \
^--------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
cmake "$ARROW_DIR"/cpp \
In run_static.sh line 43:
-DARROW_DEPENDENCY_SOURCE=${ARROW_DEPENDENCY_SOURCE} \
^------------------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
-DARROW_DEPENDENCY_SOURCE="${ARROW_DEPENDENCY_SOURCE}" \
In run_static.sh line 46:
$ARROW_CMAKE_OPTIONS
^------------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
"$ARROW_CMAKE_OPTIONS"
In run_static.sh line 48:
make -j$NPROC
^----^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
make -j"$NPROC"
In run_static.sh line 60:
rm -rf $EXAMPLE_BUILD_DIR
^----------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
rm -rf "$EXAMPLE_BUILD_DIR"
In run_static.sh line 61:
mkdir -p $EXAMPLE_BUILD_DIR
^----------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
mkdir -p "$EXAMPLE_BUILD_DIR"
In run_static.sh line 62:
pushd $EXAMPLE_BUILD_DIR
^----------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
pushd "$EXAMPLE_BUILD_DIR"
In run_static.sh line 64:
cmake $EXAMPLE_DIR -DARROW_LINK_SHARED=OFF
^----------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
cmake "$EXAMPLE_DIR" -DARROW_LINK_SHARED=OFF
In run_static.sh line 76:
pushd $EXAMPLE_DIR
^----------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
pushd "$EXAMPLE_DIR"
In run_static.sh line 78:
$EXAMPLE_BUILD_DIR/arrow-example
^----------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
"$EXAMPLE_BUILD_DIR"/arrow-example
In run_static.sh line 87:
rm -rf $EXAMPLE_BUILD_DIR
^----------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
rm -rf "$EXAMPLE_BUILD_DIR"
In run_static.sh line 88:
mkdir -p $EXAMPLE_BUILD_DIR
^----------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
mkdir -p "$EXAMPLE_BUILD_DIR"
In run_static.sh line 90:
-o $EXAMPLE_BUILD_DIR/arrow-example \
^----------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
-o "$EXAMPLE_BUILD_DIR"/arrow-example \
In run_static.sh line 91:
$EXAMPLE_DIR/example.cc \
^----------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
"$EXAMPLE_DIR"/example.cc \
In run_static.sh line 92:
$(PKG_CONFIG_PATH=$ARROW_BUILD_DIR/lib/pkgconfig \
^-- SC2046 (warning): Quote this to prevent word splitting.
In run_static.sh line 104:
pushd $EXAMPLE_DIR
^----------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
pushd "$EXAMPLE_DIR"
In run_static.sh line 106:
$EXAMPLE_BUILD_DIR/arrow-example
^----------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
"$EXAMPLE_BUILD_DIR"/arrow-example
For more information:
https://www.shellcheck.net/wiki/SC2046 -- Quote this to prevent word splitt...
https://www.shellcheck.net/wiki/SC2086 -- Double quote to prevent globbing ...
https://www.shellcheck.net/wiki/SC2223 -- This default assignment may cause...
```
### What changes are included in this PR?
* SC2046: Quote command substitutions
* SC2086: Quote variables
* SC2223: Quote default assignments
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
### Was AI used for this PR?
In accordance to the [AI generation guidelines](https://arrow.apache.org/docs/dev/developers/overview.html#ai-generated-code), please disclose below whether and how AI was used in this PR.
**PR code and description written by:**
- [x] Human
- [ ] AI
**Reviewed before submission by:**
- [x] Human
- [ ] AI
- [ ] Not reviewed
* GitHub Issue: #51670
Authored-by: Hiroyuki Sato <hiroysato@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
Changes around UriParser, including #51007, caused example-cpp-minimal-build-static-system-dependency to fail to build.
### What changes are included in this PR?
This change adds liburiparser-dev to the system dependency image.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
### Was AI used for this PR?
In accordance to the [AI generation guidelines](https://arrow.apache.org/docs/dev/developers/overview.html#ai-generated-code), please disclose below whether and how AI was used in this PR.
**PR code and description written by:**
- [x] Human
- [ ] AI
**Reviewed before submission by:**
- [x] Human
- [ ] AI
- [ ] Not reviewed
* GitHub Issue: #51682
Authored-by: Hiroyuki Sato <hiroysato@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Fix#51329
Additional downstream static-linking test, after #50195 moved S3 into the separate ArrowS3 library.
### What changes are included in this PR?
Test downstream static linking of `ArrowS3::arrow_s3_static` via minimal_build example and `cpp_test.sh`,
Fix dependencies for static Azure and GCS consumers: append `libcurl` for bundled GCS, append `libcurl` and `LibXml2` for bundled Azure(NOT WIN32),
append links for system Azure SDK
and pkg-config --static arrow to include `libcurl` and `libxml-2.0` for bundled GCS/Azure
### Are these changes tested?
Verified by CI
### Are there any user-facing changes?
Downstream consumers now have required static dependencies for Azure/GCS
* GitHub Issue: #51329
Authored-by: Tadeja Kadunc <tadeja.kadunc@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
This is the sub issue #44748.
```
In build_arrow.sh line 23:
mkdir -p $ARROW_BUILD_DIR
^--------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
mkdir -p "$ARROW_BUILD_DIR"
In build_arrow.sh line 24:
pushd $ARROW_BUILD_DIR
^--------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
pushd "$ARROW_BUILD_DIR"
In build_arrow.sh line 33:
$ARROW_CMAKE_OPTIONS
^------------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
"$ARROW_CMAKE_OPTIONS"
In build_arrow.sh line 35:
make -j$NPROC
^----^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
make -j"$NPROC"
In build_example.sh line 21:
mkdir -p $EXAMPLE_BUILD_DIR
^----------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
mkdir -p "$EXAMPLE_BUILD_DIR"
In build_example.sh line 22:
pushd $EXAMPLE_BUILD_DIR
^----------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
pushd "$EXAMPLE_BUILD_DIR"
In build_example.sh line 24:
cmake /io $EXAMPLE_CMAKE_OPTIONS
^--------------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
cmake /io "$EXAMPLE_CMAKE_OPTIONS"
For more information:
https://www.shellcheck.net/wiki/SC2086 -- Double quote to prevent globbing ...
```
### What changes are included in this PR?
* SC2086: Quote variables
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
### Was AI used for this PR?
**PR code and description written by:**
- [x] Human
- [ ] AI
**Reviewed before submission by:**
- [x] Human
- [ ] AI
- [ ] Not reviewed
* GitHub Issue: #51485
Authored-by: Hiroyuki Sato <hiroysato@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
`debian-forky-arm64` and `debian-forky-amd64` Packaging Build jobs failing with `Error: Unable to locate package libboost-system-dev`.
### What changes are included in this PR?
Remove `libboost-system-dev` from `debian/control.in`
Remove apt-install line ` libboost-system-dev \` from apt Dockerfiles (debian-bookworm, debian-forky, debian-trixie, ubuntu-jammy, ubuntu-noble, ubuntu-resolute), C++ CI Dockerfiles (debian-13-cpp, debian-experimental-cpp, ubuntu-22.04-cpp, ubuntu-24.04-cpp) and from `cpp/examples/minimal_build/system_dependency.dockerfile`.
Remove unused `[provide]` `boost_system` reference from `cpp/subprojects/boost.wrap`.
### Are these changes tested?
By Packaging CI, jobs should pass.
### Are there any user-facing changes?
No.
* GitHub Issue: #50065
Authored-by: Tadeja Kadunc <tadeja.kadunc@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Arrow requires CMake 3.25 but was still using deprecated `COMPILE_FLAGS` property. Recommanded to use `COMPILE_OPTIONS` (introduced in CMake 3.11).
### What changes are included in this PR?
Replaced `COMPILE_FLAGS` with `COMPILE_OPTIONS` across `CMakeLists.txt` files, converted space separated strings to semicolon-separated lists, and removed obsolete TODO comments.
### Are these changes tested?
Yes, through CI build and existing tests.
### Are there any user-facing changes?
No.
* GitHub Issue: #48980
Authored-by: Hyukjin Kwon <gurwls223@apache.org>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
`FileReader::ReadTable` previously returned `Status` and required callers to pass an `out` parameter.
### What changes are included in this PR?
Introduce `Result<std::shared_ptr<Table>>` returning API to allow clearer error propagation:
- Add new Result-returning `ReadTable()` methods
- Deprecate the old Status versions
- Migrate all callers to use the new API
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
Status version FileReader::ReadTable has been deprecated.
```cpp
virtual ::arrow::Status ReadTable(std::shared_ptr<::arrow::Table>* out);
virtual ::arrow::Status ReadTable(const std::vector<int>& column_indices,
std::shared_ptr<::arrow::Table>* out);
```
* GitHub Issue: #48394
Lead-authored-by: fenfeng9 <fenfeng9@qq.com>
Co-authored-by: fenfeng9 <36840213+fenfeng9@users.noreply.github.com>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
This PR fixes several typos in `arrow/cpp/*`
### What changes are included in this PR?
This PR proposes to fix a typo in `arrow/cpp/*`
### Are these changes tested?
Manually tested via my browser.
### Are there any user-facing changes?
No, dev-only.
Authored-by: Hyukjin Kwon <gurwls223@apache.org>
Signed-off-by: David Li <li.davidm96@gmail.com>
### Rationale for this change
We decided to migrate Arrow C++ to C++20 in this discussion: https://lists.apache.org/thread/48zlj0dn2y0f53y2k37qsr90y781wfnj
### What changes are included in this PR?
1. Build configuration updates (CMake files etc.) to build with C++20 instead of C++17
2. C++-level fixes to ensure compilation succeeds:
* Workarounds for deprecation of atomic access on `std::shared_ptr<T>` (the replacement `std::atomic<std::shared_ptr<T>>` is unfortunately not supported in all standard library implementations)
* Workaround for [Abseil ABI issues](https://github.com/abseil/abseil-cpp/issues/1624)
* Trivial update to `arrow/util/string.h` to call a C++20 API (to validate that C++20 is actually enabled)
3. CI configuration updates to get enough C++20 support on the various platforms:
* Bump default clang and LLVM versions to 18
* Use AlmaLinux 10 for release verification (AlmaLinux 8 is still tested on other CI builds)
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes, Arrow C++ will now require a C++20-compliant compiler.
* GitHub Issue: #45885
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Columns can b encrypted with individual keys. For this, the column name have to be set in `EncryptionConfiguration::column_keys`. This poses the following challenges for columns with nested fields like `MapType`, `ListType`, and `StructType`. Encrypting a column of such type requires providing an encryption key for all nested (leaf) fields. Ideally, the column name should be sufficient (as it is for any other data type) to encrypt all nested fields.
### What changes are included in this PR?
The column name can be used to encrypt all nested fields of `StrutType`, `MapType`, and `ListType` columns with the same encryption key. The current column naming scheme can still be used for backward compatibility.
### Are these changes tested?
Tested in C++ and Python.
### Are there any user-facing changes?
Column encryption can be configured with less code and more intuitive naming.
Documentation and examples updated.
Fixes#41246.
* GitHub Issue: #41246
Authored-by: Enrico Minack <github@enrico.minack.dev>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
While working on PR #48336 I quickly got irritated by the clumsiness of the Parquet encryption configuration API. This issue attempts to reduce the verbosity of `ColumnEncryptionProperties` construction.
### What changes are included in this PR?
1. Remove unused `column_path` member in `ColumnEncryptionProperties`
2. Add convenience `ColumnEncryptionProperties` factory functions to avoid going through the clumsy `Builder` API
### Are these changes tested?
Yes
### Are there any user-facing changes?
Two deprecated constructors and a removed accessor for an useless property.
**This PR includes breaking changes to public APIs.** (If there are any breaking changes to public APIs, please explain which changes are breaking. If not, you can remove this.)
* GitHub Issue: #48337
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
A C++ linting error was introduced with #40705.
### What changes are included in this PR?
Linter fixed issue in `cpp/examples/arrow/parquet_read_write.cc`
Authored-by: Rok Mihevc <rok@mihevc.org>
Signed-off-by: Rok Mihevc <rok@mihevc.org>
### Rationale for this change
Is there a reason we're using a low value here. All other examples use `128*1024` or `64*1024`.
I was stumped by this as I used it without really reading about the parameter and spent a day figuring out why my parquet writes were so slow.
### What changes are included in this PR?
Increase `chunk_size` to `64*1024`
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
Authored-by: Shadab Zafar <dufferzafar0@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
Docker Compose recommends `compose.yaml`:
https://docs.docker.com/compose/intro/compose-application-model/#the-compose-file
> The default path for a Compose file is compose.yaml (preferred) or compose.yml that is placed in the working directory. Compose also supports docker-compose.yaml and docker-compose.yml for backwards compatibility of earlier versions. If both files exist, Compose prefers the canonical compose.yaml.
### What changes are included in this PR?
- Update several `docker-compose.yml` file names to `compose.yaml`
- Update doc and comment references
- Update archery references to the file
- Update workflows and CODEOWNERS to take new file into account.
### Are these changes tested?
Yes via CI and some archery jobs.
### Are there any user-facing changes?
No
* GitHub Issue: #47806
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
This continues adding functionality to the Meson configuration
### What changes are included in this PR?
This adds the parquet directory to the Meson configuration
### Are these changes tested?
Yes
### Are there any user-facing changes?
No
* GitHub Issue: #46410
Authored-by: Will Ayd <william.ayd@icloud.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
fix example running error
```
****** Scan Example ******
Error occurred: No function registered with name: make_struct
```
### What changes are included in this PR?
call arrow::compute::initialize ahead
### Are these changes tested?
yes
### Are there any user-facing changes?
no
* GitHub Issue: #47243
Lead-authored-by: egolearner <lijiliang@outlook.com>
Co-authored-by: egolearner <45122959+egolearner@users.noreply.github.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
We decided to move Skyhook to its own repository here: https://lists.apache.org/thread/f5ndxf0o6v4wbzgoy6nsrj4xkn34f380
We've created https://github.com/apache/arrow-skyhook and extract Skyhook related commits to the repository.
### What changes are included in this PR?
Remove Skyhook from the main repository
### Are these changes tested?
CI is validated.
### Are there any user-facing changes?
Yes, Skyhook is removed from the main repository.
* GitHub Issue: #47225
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
Static building for JNI build on macOS is failing in apache/arrow-java. We should avoid this in apache/arrow.
See also: https://github.com/apache/arrow-java/pull/799
### What changes are included in this PR?
* Add a CI job for JNI build on macOS
* Fix build problems for bundled AWS SDK for C++
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* GitHub Issue: #47222
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
Cryptographic keys must be kept private. Using the new `arrow::util::SecureString` is vital for storing secrets securely.
### What changes are included in this PR?
Uses the `arrow::util::SecureString` introduced in #46626 for cryptographic keys throughout Parquet encryption.
### Are these changes tested?
Unit tests.
### Are there any user-facing changes?
APIs that hand over secrets to Arrow require the secret to be encapsulated in a `SecureString`.
**This PR includes breaking changes to public APIs.**
TODO:
- provide instructions for migration
Supersedes #12890.
* GitHub Issue: #31603
Lead-authored-by: Enrico Minack <github@enrico.minack.dev>
Co-authored-by: Antoine Pitrou <antoine@python.org>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Current system requirements is outdated and was not updated to match the CMake update on:
- https://github.com/apache/arrow/pull/44989
While fixing the above it was noted that there were still other pending references on some of our codebase (docs + minor CMake)
### What changes are included in this PR?
Update:
- R system requirements to CMake >= 3.25
- Build docs to refer to 3.25
- Update a couple of CMake references
### Are these changes tested?
CI and Archery has been exercised, no new tests are required.
### Are there any user-facing changes?
No
* GitHub Issue: #46831
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### What changes are included in this PR?
* Pass movable objects by value to functions that keep a copy of them.
* Use emplace() instead of insert() when the arguments can be moved safely.
* Remove unnecessary const qualifier on member variables that makes the class non-copyable and non-movable.
* Remove unnecessary constructors and destructors that can be generated automatically by compilers.
### Are these changes tested?
Yes, by existing tests.
### Are there any user-facing changes?
No.
* GitHub Issue: #37891
Authored-by: Eddie Chang <kalcifer7319@gmail.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Arrow is quite a heavy dependency and some users don't need all the tools we provide bundled with libarrow. Moving Arrow Compute to its own shared library allows users installations that better suit their needs having smaller memory footprint if necessary.
It might also help some users adding new kernels into an existing Arrow without recompiling it.
### What changes are included in this PR?
- Move all the Arrow Compute kernel functions to a new `ArrowCompute` shared library (`libarrow_compute.so`).
- Create a new API to Initialize arrow compute registering the existing Kernels into the `FunctionRegistry`
- Update Python/R/CGLib bindings to automatically register the Compute kernels transparently to the user.
- Update Linux Packaging to provide the new arrow-compute library.
- Update documentation with new requirements to call `arrow::compute::Initialize()`
### Are these changes tested?
Yes on all CI jobs.
### Are there any user-facing changes?
Yes. The Arrow compute functions will be provided as a different library. Any user using Arrow Compute from C++ directly will require a call to `arrow::compute::Initialize()` in order for the functions and kernels to be registered
**This PR includes breaking changes to public APIs.**
* GitHub Issue: #25025
Lead-authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Co-authored-by: Bryce Mecum <petridish@gmail.com>
Co-authored-by: Sutou Kouhei <kou@clear-code.com>
Co-authored-by: Rossi Sun <zanmato1984@gmail.com>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
https://github.com/apache/arrow/issues/45908 brought these helpers into the public API but didn't consider changes to their API. This PR makes all the helpers use the standard Result-pattern to make them more ergonomic. We can do this now without a breaking change because this and https://github.com/apache/arrow/issues/45908 will be part of Arrow 21.
### What changes are included in this PR?
- Refactored all FromJSONString helpers to use the Result pattern (instead of using outparams)
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
* GitHub Issue: #46439
Lead-authored-by: Bryce Mecum <petridish@gmail.com>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
These functions are generally useful and stable so it would be a good idea to clearly include them in the public API. I'm starting here with just the basic ones in order to make the PR small. See https://github.com/apache/arrow/issues/45908 for more information.
### What changes are included in this PR?
- Moves `ArrayFromJSON`, `ChunkedArrayFromJSON`, `DictArrayFromJSON`, `ScalarFromJSON`, `DictScalarFromJSON` from `arrow::ipc::internal` namespace to `arrow::json` so it's clearer they're part of the public API and that they're more useful than just for IPC
- Renames each of the above from `{Array,...}FromJSON` to `{Array,...}FromJSONString` to avoid confusion between these helpers and the main JSON(L) reader
- Renames `arrow/util/json_simple.{h,cc}` to `arrow/json/from_string.{h,cc}` both because of the namespace jump but also because the filename is more clear.
- Adds User Guide and adds a listing in the API docs for moved functions
### Are these changes tested?
Yes.
### Are there any user-facing changes?
This expands the scope of our public API but does not break any existing public APIs though it's possible users
* GitHub Issue: #45908
Lead-authored-by: Bryce Mecum <petridish@gmail.com>
Co-authored-by: Benjamin Kietzman <bengilgit@gmail.com>
Co-authored-by: Enrico Minack <github@enrico.minack.dev>
Signed-off-by: Bryce Mecum <petridish@gmail.com>
### Rationale for this change
`GetRecordBatchReader` in reader.h and `NewRowGroup` with `chunk_size` in writer.h have been deprecated in 19.0.0.
### What changes are included in this PR?
Deprecated functions are removed.
### Are these changes tested?
Existing tests should pass.
### Are there any user-facing changes?
No, deprecated functionality is removed.
* GitHub Issue: #46132
Authored-by: AlenkaF <frim.alenka@gmail.com>
Signed-off-by: Krisztian Szucs <szucs.krisztian@gmail.com>
### Rationale for this change
`arrow/util/logging.h` exports `DCHECK()` family macros as internal macros. They are conflicted with macros provided by glog when glog headers are included after `arrow/util/logging.h`.
### What changes are included in this PR?
* Move `DCHECK()` family macros to `arrow/util/logging_internal.h`
* Use `arrow/util/logging_internal.h` not `.../logging.h` in `*.cc`
* Use `ARROW_DCHECK*()` macros instead of `DCHECK*()` macros in headers including `*_internal.h`
* Use `ARROW_DCHECK*()` macros instead of `DCHECK*()` macros in examples
* Use `ARROW_DCHECK*()` macros instead of `DCHECK*()` macros in `python/`
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* GitHub Issue: #46011
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
OpenSSL encryption / decryption is wrapped by AesEncryptor / AesDencryptor, which is used by multiple threads of a single scanner or by multiple concurrent scanners when scanning a dataset. Some thread may call `WipeOut` while other threads still use the instance.
### What changes are included in this PR?
- Remove the `WipeOut` methods and related datastructures entirely.
- Each call into `CtrEncrypt` / `CtrDecrypt` and `GcmEncrypt` / `GcmDecrypt` uses its own `EVP_CIPHER_CTX` instance, making this thread-safe.
After fixing this `"AesDecryptor was wiped out"` issue, two other segmentation faults surfaced: GH-44988. This has also been addressed here as it can only be exposed after fixing the wipe-out issue.
Fixes GH-43057.
Fixes GH-44852.
Fixes GH-44988.
### Are these changes tested?
A unit test that scans a dataset concurrently reproduced the initial issue in 30% of the test runs.
### Are there any user-facing changes?
No.
* GitHub Issue: #43057
Lead-authored-by: Enrico Minack <github@enrico.minack.dev>
Co-authored-by: Antoine Pitrou <antoine@python.org>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Signed-off-by: Antoine Pitrou <pitrou@free.fr>
### Rationale for this change
Encrypting columns with nested fields with a column key is not trivial since only leaf fields are allowed in the column key map. Documentation emphasizes this fact and provides examples.
### What changes are included in this PR?
This amends the documentation on encryption for C++ and Python.
### Are these changes tested?
Only documentation.
### Are there any user-facing changes?
Only documentation.
* GitHub Issue: #41246
Lead-authored-by: Enrico Minack <github@enrico.minack.dev>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
We want to upgrade our CMake version to 3.25 as discussed on the ML:
https://lists.apache.org/thread/h8jp16ktrj11fmjmjhlg6xvkvv9wzvjk
### What changes are included in this PR?
- Bump minimal CMake version to 3.25
- Manually install CMake on distributions where CMake < 3.25 was installed via package repositories
- Minor fixes to CI in order to have passing builds everywhere
### Are these changes tested?
Yes, via CI.
### Are there any user-facing changes?
Yes, the minimum CMake version to be used to build Arrow is bumped to 3.25.
**This PR includes breaking changes to build systems.**
* GitHub Issue: #44950
Lead-authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Co-authored-by: Bryce Mecum <petridish@gmail.com>
Co-authored-by: Sutou Kouhei <kou@clear-code.com>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Co-authored-by: Neal Richardson <neal.p.richardson@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
I think the intention was to disable the default allocator which used to be jemalloc. However since #44951 we have switched the default allocator from jemalloc to mimalloc. We should update the examples.
### What changes are included in this PR?
Change disabling jemalloc to disabling mimalloc for examples.
### Are these changes tested?
Manually tested.
### Are there any user-facing changes?
None.
* GitHub Issue: #45219
Authored-by: Rossi Sun <zanmato1984@gmail.com>
Signed-off-by: Rossi Sun <zanmato1984@gmail.com>
In the comments the includes of `(arrow/)parquet/arrow/reader.h` were broken. Removing the prefix `arrow` makes this include in line with the includes used at the top of the file.
* GitHub Issue: #45027
Authored-by: Nis Meinert <nis.meinert@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
We're migrating `arrow::Status` + output variable API to `arrow::Result` API.
### What changes are included in this PR?
* Add `arrow::Result<std::unique_ptr<FileReader>> parquet::arrow::OpenFile()`
* Deprecate `arrow::Status parquet::arrow::OpenFile()`
* Use the added `arrow::Result` version in our code base
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* GitHub Issue: #44784
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
This is a follow-on to https://github.com/apache/arrow/pull/43566 which was merged earlier today. See https://github.com/apache/arrow/issues/43570 for more detail on the entire set of changes.
### What changes are included in this PR?
- Replace instances where we call or document calling `docker-compose` with `docker compose`.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
This PR includes the require user-facing changes to document things.
* GitHub Issue: #43570
Lead-authored-by: Bryce Mecum <petridish@gmail.com>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
The current logic for detecting whether the `CMAKE_BUILD_TYPE` is set is incorrect. That variable is never fully undefined; by default, in cases where it is unset is actually set to the empty string. Therefore, the condition that must be checked is not whether the variable is defined, but whether it tests to a truthy value (i.e. is a non-empty string).
I consider this a minor change so I have not opened an associated issue.
### What changes are included in this PR?
This PR changes `if(NOT DEFINED CMAKE_BUILD_TYPE)` to `if(NOT CMAKE_BUILD_TYPE)`.
### Are these changes tested?
Since this fixes a particular CMake build scenario I am not sure if a test is merited, or where one would be added.
### Are there any user-facing changes?
No.
Authored-by: Vyas Ramasubramani <vyasr@nvidia.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
Moving LocalFileSystem into the registry is a good first usage and will help us hammer out which aspects of built in file systems should remain public.
### What changes are included in this PR?
A factory for LocalFileSystem is added to the registry. `FileSystem::MakeUri` ( https://github.com/apache/arrow/issues/18316 ) is added to enable roundtripping filesystems through URIs. `file://` URIs now support a use_mmap query parameter, and `local://` URIs are also supported as an alias.
<h6 id="reduce">Reducing the set of bound classes</h6>
Some unnecessary bindings to the LocalFileSystem concrete class are removed. This demonstrates that with a registered factory pattern, it is no longer necessary to keep a class hierarchy public for binding. Eventually (if desired), we can move concrete subclasses of FileSystem entirely out of public headers.
### Are these changes tested?
Yes, all existing tests for file:// URIs continue to pass
### Are there any user-facing changes?
For consistency, local:// URIs will now be considered equivalent to corresponding file:// URIs
* GitHub Issue: #40342
Authored-by: Benjamin Kietzman <bengilgit@gmail.com>
Signed-off-by: Benjamin Kietzman <bengilgit@gmail.com>
### Rationale for this change
Each filesystem implementation carries unique and potentially heavy dependencies, so it'd be useful to build them separately. Furthermore, one typically doesn't need all of them at the same time and building separate modules would allow them to be dynamically loaded as necessary. Finally, defining this interface allows custom filesystem implementations to be supported seamlessly.
### What changes are included in this PR?
An initial sketch of a registry, with documentation as if the registry were complete to illustrate intended usage.
### Are these changes tested?
A toy module is added and a single unit test too.
### Are there any user-facing changes?
Users would be able to add their own filesystem implementations to the registry
* Closes: #38309
Authored-by: Benjamin Kietzman <bengilgit@gmail.com>
Signed-off-by: Benjamin Kietzman <bengilgit@gmail.com>
### Rationale for this change
If we can build most of `cpp/src/arrow/**/*.cc` before all bundled libraries are built, we can reduce build time.
### What changes are included in this PR?
* Remove the `toolchain` internal CMake target
* Remove `ARROW_SHARED_LINK_LIBS`
* Remove `ARROW_STATIC_LINK_LIBS`
* Move the following variables to `cpp/src/arrow/CMakeLists.txt`
* `ARROW_SHARED_PRIVATE_LINK_LIBS`
* `ARROW_SHARED_INSTALL_INTERFACE_LIBS`
* `ARROW_STATIC_INSTALL_INTERFACE_LIBS`
* `ARROW_TEST_LINK_TOOLCHAIN`
* `ARROW_TEST_SHARED_LINK_LIBS`
* `ARROW_TEST_STATIC_LINK_LIBS`
* `ARROW_SYSTEM_LINK_LIBS`
* Add internal `OBJECT` libraries that have minimal dependencies
* Remove unused `cpp/src/arrow/util/benchmark_main.cc`
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
* Closes: #39823
* GitHub Issue: #39823
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
As proposed in #36246 , by splitting function option structs from `function.h`, we can reduce the including of `function.h`. So that the total build time could be reduced.
The total parser time could be reduced from 722.3s to 709.7s. And the `function.h` along with its transitive inclusion of `kernel.h` don't show up in expensive headers any more.
The detailed analysis result before and after this PR are attached:
[analyze-before.txt](https://github.com/apache/arrow/files/13756923/analyze-before.txt)
[analyze-after.txt](https://github.com/apache/arrow/files/13756924/analyze-after.txt)
Disclaimer (quote from https://github.com/apache/arrow/issues/36246#issuecomment-1866974963):
> Note that the time diff is not absolute. The ClangBuildAnalyzer result differs from time to time. I guess it depends on the idle-ness of the building machine when doing the experiment. But the time reduction is almost certain, though sometimes more sometimes less. And the inclusion times of the questioning headers are reduced for sure, as shown in the attachments in my other comment.
### What changes are included in this PR?
Move function option structs into own `compute/options.h`, and change including `function.h` to including `options.h` wherever fits.
### Are these changes tested?
Build is testing.
### Are there any user-facing changes?
There could be potential build failures for user code (quote from https://github.com/apache/arrow/issues/36246#issuecomment-1866980969):
> The header function.h remains in compute/api.h, with and without this PR. The proposed PR removes function.h from api_xxx.h (then includes options.h instead), as proposed in the initial description of this issue. This results in compile failures for user code which includes only compute/api_xxx.h but not compute/api.h, and meanwhile uses CallFunction which is declared in function.h.
But I think it's OK as described in https://github.com/apache/arrow/issues/36246#issuecomment-1867018578.
* Closes: #39357
Authored-by: zanmato <zanmato1984@gmail.com>
Signed-off-by: Felipe Oliveira Carvalho <felipekde@gmail.com>
### Rationale for this change
### What changes are included in this PR?
### Are these changes tested?
### Are there any user-facing changes?
* Closes: #38977
Authored-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
Trying to put `EstimatedBufferedValueBytes` from `TypedColumnWriter` to `ColumnWriter`.
### What changes are included in this PR?
put `EstimatedBufferedValueBytes` from `TypedColumnWriter` to `ColumnWriter`.
### Are these changes tested?
No, just interface change
### Are there any user-facing changes?
Yes, interface changed
* Closes: #38887
Authored-by: mwish <maplewish117@gmail.com>
Signed-off-by: mwish <maplewish117@gmail.com>
### Rationale for this change
Datasets and Substrait are heavy subcomponents (as they have non-trivial dependencies), so should not be enabled unwillingly in all CI builds.
### What changes are included in this PR?
Disable Dataset and Substrait in `ci/scripts/cpp_build.sh`, enable them explicitly in relevant builds.
Disable more subcomponents in CI builds that don't need them, especially "minimal" builds.
### Are these changes tested?
Yes, by existing CI jobs.
### Are there any user-facing changes?
No.
* Closes: #37500
Lead-authored-by: Antoine Pitrou <antoine@python.org>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Co-authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
Previously, when input an non-owned string, `arrow::io::BufferReader` would zero-copy it. It would cause lifetime problem. This patch add `FromString` to help build from `std::string`.
### What changes are included in this PR?
* Add a ctor for FromString(std::string), and deprecate non-owning ctors
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Some APIs are being deprecated. Users can use the new interface.
* Closes: #37212
Authored-by: mwish <maplewish117@gmail.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Fix the job and rename so it's picked up as part of the nightlies
### What changes are included in this PR?
Rename + fixes to the job.
### Are these changes tested?
Via CI
### Are there any user-facing changes?
No
* Closes: #37308
Lead-authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
We need to support Ubuntu 20.04 (CMake 3.16) or later, CentOS 7 (CMake 3.17) or later, and Debian GNU/Linux bullseye (CMake 3.18) or later.
So we can require CMake 3.16 or later.
### What changes are included in this PR?
Require CMake 3.16 or later.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* Closes: #34921
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
Support a custom io-executor in Acero QueryOptions.
### What changes are included in this PR?
Allow user to define a customed io-executor.
### Are these changes tested?
Currently not
### Are there any user-facing changes?
User can specify an io-executor
* Closes: #35125
Authored-by: mwish <maplewish117@gmail.com>
Signed-off-by: Weston Pace <weston.pace@gmail.com>
### Rationale for this change
The documentation for Acero was incomplete. This PR refactors the existing documentation and adds several entirely new sections to form a complete design document for Acero.
### What changes are included in this PR?
Some existing documentation is cleaned up. Acero documentation is moved into its own folder and broken into several pages.
### Are these changes tested?
The documentation is built as part of the CI but I wouldn't say it is fully tested.
### Are there any user-facing changes?
There are not code changes (other than the removal of two legacy methods) but there are many user-facing documentation changes.
* Closes: #32335
Lead-authored-by: Weston Pace <weston.pace@gmail.com>
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Co-authored-by: Benjamin Kietzman <bengilgit@gmail.com>
Signed-off-by: Weston Pace <weston.pace@gmail.com>
### Rationale for this change
See the linked issue
### What changes are included in this PR?
C++:
* remove all compute/exec/* from libarrow
* rename compute/exec -> acero and make libarrow_acero
* add new ARROW_ACERO option, required if ARROW_DATASET is on
* libarrow_dataset now depends on libarrow_acero
c_glib: add the new libarrow_acero dependency - we disallow building glib without it
python: added PYARROW_BUILD_ACERO, set to on if DATASETS are built
### Are these changes tested?
All the standard tests do work properly.
I manually compiled C++ with:
* no ARROW_ACERO
* ARROW_ACERO and no ARROW_DATASET
* ARROW_ACERO and ARROW_DATASET and no ARROW_SUBSTRAIT
I manually compiled python without ACERO & DATASET and with ACERO and without DATASET
### Are there any user-facing changes?
If users include compute/exec files directly then they'll have to update their code.
* Closes: #15280
Lead-authored-by: Davide Pasetto <dpasetto69@gmail.com>
Co-authored-by: Li Jin <ice.xelloss@gmail.com>
Co-authored-by: Weston Pace <weston.pace@gmail.com>
Signed-off-by: Li Jin <ice.xelloss@gmail.com>
### Rationale for this change
We can't maintain Plasma.
### What changes are included in this PR?
These changes remove all Plasma related codes.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* Closes: #33243
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
Hiroyuki Sato