### Rationale for this change
This is the sub issue #44748.
* SC2046: Quote this to prevent word splitting
* SC2086: Double quote to prevent globbing and word splitting
* SC2223: This default assignment may cause DoS due to globbing. Quote it
```
In build_arrow.sh line 23:
mkdir -p $ARROW_BUILD_DIR
^--------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
mkdir -p "$ARROW_BUILD_DIR"
In build_arrow.sh line 24:
pushd $ARROW_BUILD_DIR
^--------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
pushd "$ARROW_BUILD_DIR"
In build_arrow.sh line 30:
$ARROW_CMAKE_OPTIONS
^------------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
"$ARROW_CMAKE_OPTIONS"
In build_arrow.sh line 32:
make -j$NPROC
^----^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
make -j"$NPROC"
In build_example.sh line 21:
mkdir -p $EXAMPLE_BUILD_DIR
^----------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
mkdir -p "$EXAMPLE_BUILD_DIR"
In build_example.sh line 22:
pushd $EXAMPLE_BUILD_DIR
^----------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
pushd "$EXAMPLE_BUILD_DIR"
In run_static.sh line 21:
: ${ARROW_DIR:=/arrow}
^------------------^ SC2223 (info): This default assignment may cause DoS due to globbing. Quote it.
In run_static.sh line 22:
: ${EXAMPLE_DIR:=/io}
^-----------------^ SC2223 (info): This default assignment may cause DoS due to globbing. Quote it.
In run_static.sh line 23:
: ${ARROW_BUILD_DIR:=/build/arrow}
^-- SC2223 (info): This default assignment may cause DoS due to globbing. Quote it.
In run_static.sh line 24:
: ${EXAMPLE_BUILD_DIR:=/build/example}
^-- SC2223 (info): This default assignment may cause DoS due to globbing. Quote it.
In run_static.sh line 26:
: ${ARROW_DEPENDENCY_SOURCE:=BUNDLED}
^-- SC2223 (info): This default assignment may cause DoS due to globbing. Quote it.
In run_static.sh line 34:
mkdir -p $ARROW_BUILD_DIR
^--------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
mkdir -p "$ARROW_BUILD_DIR"
In run_static.sh line 35:
pushd $ARROW_BUILD_DIR
^--------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
pushd "$ARROW_BUILD_DIR"
In run_static.sh line 39:
cmake $ARROW_DIR/cpp \
^--------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
cmake "$ARROW_DIR"/cpp \
In run_static.sh line 43:
-DARROW_DEPENDENCY_SOURCE=${ARROW_DEPENDENCY_SOURCE} \
^------------------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
-DARROW_DEPENDENCY_SOURCE="${ARROW_DEPENDENCY_SOURCE}" \
In run_static.sh line 46:
$ARROW_CMAKE_OPTIONS
^------------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
"$ARROW_CMAKE_OPTIONS"
In run_static.sh line 48:
make -j$NPROC
^----^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
make -j"$NPROC"
In run_static.sh line 60:
rm -rf $EXAMPLE_BUILD_DIR
^----------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
rm -rf "$EXAMPLE_BUILD_DIR"
In run_static.sh line 61:
mkdir -p $EXAMPLE_BUILD_DIR
^----------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
mkdir -p "$EXAMPLE_BUILD_DIR"
In run_static.sh line 62:
pushd $EXAMPLE_BUILD_DIR
^----------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
pushd "$EXAMPLE_BUILD_DIR"
In run_static.sh line 64:
cmake $EXAMPLE_DIR -DARROW_LINK_SHARED=OFF
^----------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
cmake "$EXAMPLE_DIR" -DARROW_LINK_SHARED=OFF
In run_static.sh line 76:
pushd $EXAMPLE_DIR
^----------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
pushd "$EXAMPLE_DIR"
In run_static.sh line 78:
$EXAMPLE_BUILD_DIR/arrow-example
^----------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
"$EXAMPLE_BUILD_DIR"/arrow-example
In run_static.sh line 87:
rm -rf $EXAMPLE_BUILD_DIR
^----------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
rm -rf "$EXAMPLE_BUILD_DIR"
In run_static.sh line 88:
mkdir -p $EXAMPLE_BUILD_DIR
^----------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
mkdir -p "$EXAMPLE_BUILD_DIR"
In run_static.sh line 90:
-o $EXAMPLE_BUILD_DIR/arrow-example \
^----------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
-o "$EXAMPLE_BUILD_DIR"/arrow-example \
In run_static.sh line 91:
$EXAMPLE_DIR/example.cc \
^----------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
"$EXAMPLE_DIR"/example.cc \
In run_static.sh line 92:
$(PKG_CONFIG_PATH=$ARROW_BUILD_DIR/lib/pkgconfig \
^-- SC2046 (warning): Quote this to prevent word splitting.
In run_static.sh line 104:
pushd $EXAMPLE_DIR
^----------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
pushd "$EXAMPLE_DIR"
In run_static.sh line 106:
$EXAMPLE_BUILD_DIR/arrow-example
^----------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
"$EXAMPLE_BUILD_DIR"/arrow-example
For more information:
https://www.shellcheck.net/wiki/SC2046 -- Quote this to prevent word splitt...
https://www.shellcheck.net/wiki/SC2086 -- Double quote to prevent globbing ...
https://www.shellcheck.net/wiki/SC2223 -- This default assignment may cause...
```
### What changes are included in this PR?
* SC2046: Quote command substitutions
* SC2086: Quote variables
* SC2223: Quote default assignments
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
### Was AI used for this PR?
In accordance to the [AI generation guidelines](https://arrow.apache.org/docs/dev/developers/overview.html#ai-generated-code), please disclose below whether and how AI was used in this PR.
**PR code and description written by:**
- [x] Human
- [ ] AI
**Reviewed before submission by:**
- [x] Human
- [ ] AI
- [ ] Not reviewed
* GitHub Issue: #51670
Authored-by: Hiroyuki Sato <hiroysato@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
Changes around UriParser, including #51007, caused example-cpp-minimal-build-static-system-dependency to fail to build.
### What changes are included in this PR?
This change adds liburiparser-dev to the system dependency image.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
### Was AI used for this PR?
In accordance to the [AI generation guidelines](https://arrow.apache.org/docs/dev/developers/overview.html#ai-generated-code), please disclose below whether and how AI was used in this PR.
**PR code and description written by:**
- [x] Human
- [ ] AI
**Reviewed before submission by:**
- [x] Human
- [ ] AI
- [ ] Not reviewed
* GitHub Issue: #51682
Authored-by: Hiroyuki Sato <hiroysato@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Fix#51329
Additional downstream static-linking test, after #50195 moved S3 into the separate ArrowS3 library.
### What changes are included in this PR?
Test downstream static linking of `ArrowS3::arrow_s3_static` via minimal_build example and `cpp_test.sh`,
Fix dependencies for static Azure and GCS consumers: append `libcurl` for bundled GCS, append `libcurl` and `LibXml2` for bundled Azure(NOT WIN32),
append links for system Azure SDK
and pkg-config --static arrow to include `libcurl` and `libxml-2.0` for bundled GCS/Azure
### Are these changes tested?
Verified by CI
### Are there any user-facing changes?
Downstream consumers now have required static dependencies for Azure/GCS
* GitHub Issue: #51329
Authored-by: Tadeja Kadunc <tadeja.kadunc@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
`debian-forky-arm64` and `debian-forky-amd64` Packaging Build jobs failing with `Error: Unable to locate package libboost-system-dev`.
### What changes are included in this PR?
Remove `libboost-system-dev` from `debian/control.in`
Remove apt-install line ` libboost-system-dev \` from apt Dockerfiles (debian-bookworm, debian-forky, debian-trixie, ubuntu-jammy, ubuntu-noble, ubuntu-resolute), C++ CI Dockerfiles (debian-13-cpp, debian-experimental-cpp, ubuntu-22.04-cpp, ubuntu-24.04-cpp) and from `cpp/examples/minimal_build/system_dependency.dockerfile`.
Remove unused `[provide]` `boost_system` reference from `cpp/subprojects/boost.wrap`.
### Are these changes tested?
By Packaging CI, jobs should pass.
### Are there any user-facing changes?
No.
* GitHub Issue: #50065
Authored-by: Tadeja Kadunc <tadeja.kadunc@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
We decided to migrate Arrow C++ to C++20 in this discussion: https://lists.apache.org/thread/48zlj0dn2y0f53y2k37qsr90y781wfnj
### What changes are included in this PR?
1. Build configuration updates (CMake files etc.) to build with C++20 instead of C++17
2. C++-level fixes to ensure compilation succeeds:
* Workarounds for deprecation of atomic access on `std::shared_ptr<T>` (the replacement `std::atomic<std::shared_ptr<T>>` is unfortunately not supported in all standard library implementations)
* Workaround for [Abseil ABI issues](https://github.com/abseil/abseil-cpp/issues/1624)
* Trivial update to `arrow/util/string.h` to call a C++20 API (to validate that C++20 is actually enabled)
3. CI configuration updates to get enough C++20 support on the various platforms:
* Bump default clang and LLVM versions to 18
* Use AlmaLinux 10 for release verification (AlmaLinux 8 is still tested on other CI builds)
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes, Arrow C++ will now require a C++20-compliant compiler.
* GitHub Issue: #45885
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Docker Compose recommends `compose.yaml`:
https://docs.docker.com/compose/intro/compose-application-model/#the-compose-file
> The default path for a Compose file is compose.yaml (preferred) or compose.yml that is placed in the working directory. Compose also supports docker-compose.yaml and docker-compose.yml for backwards compatibility of earlier versions. If both files exist, Compose prefers the canonical compose.yaml.
### What changes are included in this PR?
- Update several `docker-compose.yml` file names to `compose.yaml`
- Update doc and comment references
- Update archery references to the file
- Update workflows and CODEOWNERS to take new file into account.
### Are these changes tested?
Yes via CI and some archery jobs.
### Are there any user-facing changes?
No
* GitHub Issue: #47806
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
Static building for JNI build on macOS is failing in apache/arrow-java. We should avoid this in apache/arrow.
See also: https://github.com/apache/arrow-java/pull/799
### What changes are included in this PR?
* Add a CI job for JNI build on macOS
* Fix build problems for bundled AWS SDK for C++
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* GitHub Issue: #47222
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
Current system requirements is outdated and was not updated to match the CMake update on:
- https://github.com/apache/arrow/pull/44989
While fixing the above it was noted that there were still other pending references on some of our codebase (docs + minor CMake)
### What changes are included in this PR?
Update:
- R system requirements to CMake >= 3.25
- Build docs to refer to 3.25
- Update a couple of CMake references
### Are these changes tested?
CI and Archery has been exercised, no new tests are required.
### Are there any user-facing changes?
No
* GitHub Issue: #46831
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
We want to upgrade our CMake version to 3.25 as discussed on the ML:
https://lists.apache.org/thread/h8jp16ktrj11fmjmjhlg6xvkvv9wzvjk
### What changes are included in this PR?
- Bump minimal CMake version to 3.25
- Manually install CMake on distributions where CMake < 3.25 was installed via package repositories
- Minor fixes to CI in order to have passing builds everywhere
### Are these changes tested?
Yes, via CI.
### Are there any user-facing changes?
Yes, the minimum CMake version to be used to build Arrow is bumped to 3.25.
**This PR includes breaking changes to build systems.**
* GitHub Issue: #44950
Lead-authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Co-authored-by: Bryce Mecum <petridish@gmail.com>
Co-authored-by: Sutou Kouhei <kou@clear-code.com>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Co-authored-by: Neal Richardson <neal.p.richardson@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
I think the intention was to disable the default allocator which used to be jemalloc. However since #44951 we have switched the default allocator from jemalloc to mimalloc. We should update the examples.
### What changes are included in this PR?
Change disabling jemalloc to disabling mimalloc for examples.
### Are these changes tested?
Manually tested.
### Are there any user-facing changes?
None.
* GitHub Issue: #45219
Authored-by: Rossi Sun <zanmato1984@gmail.com>
Signed-off-by: Rossi Sun <zanmato1984@gmail.com>
### Rationale for this change
This is a follow-on to https://github.com/apache/arrow/pull/43566 which was merged earlier today. See https://github.com/apache/arrow/issues/43570 for more detail on the entire set of changes.
### What changes are included in this PR?
- Replace instances where we call or document calling `docker-compose` with `docker compose`.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
This PR includes the require user-facing changes to document things.
* GitHub Issue: #43570
Lead-authored-by: Bryce Mecum <petridish@gmail.com>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
The current logic for detecting whether the `CMAKE_BUILD_TYPE` is set is incorrect. That variable is never fully undefined; by default, in cases where it is unset is actually set to the empty string. Therefore, the condition that must be checked is not whether the variable is defined, but whether it tests to a truthy value (i.e. is a non-empty string).
I consider this a minor change so I have not opened an associated issue.
### What changes are included in this PR?
This PR changes `if(NOT DEFINED CMAKE_BUILD_TYPE)` to `if(NOT CMAKE_BUILD_TYPE)`.
### Are these changes tested?
Since this fixes a particular CMake build scenario I am not sure if a test is merited, or where one would be added.
### Are there any user-facing changes?
No.
Authored-by: Vyas Ramasubramani <vyasr@nvidia.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
Datasets and Substrait are heavy subcomponents (as they have non-trivial dependencies), so should not be enabled unwillingly in all CI builds.
### What changes are included in this PR?
Disable Dataset and Substrait in `ci/scripts/cpp_build.sh`, enable them explicitly in relevant builds.
Disable more subcomponents in CI builds that don't need them, especially "minimal" builds.
### Are these changes tested?
Yes, by existing CI jobs.
### Are there any user-facing changes?
No.
* Closes: #37500
Lead-authored-by: Antoine Pitrou <antoine@python.org>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Co-authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
We need to support Ubuntu 20.04 (CMake 3.16) or later, CentOS 7 (CMake 3.17) or later, and Debian GNU/Linux bullseye (CMake 3.18) or later.
So we can require CMake 3.16 or later.
### What changes are included in this PR?
Require CMake 3.16 or later.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* Closes: #34921
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
We can't maintain Plasma.
### What changes are included in this PR?
These changes remove all Plasma related codes.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* Closes: #33243
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
This PR switches our build system to require C++17 instead of C++11.
Because the conda packaging jobs are out of sync with the conda-forge files, the Windows conda packaging jobs are broken with this change. The related task (sync conda packaging files with conda-forge) is tracked in ARROW-17635.
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
ARROW-9171 and ARROW-17231 are also fixed.
Our CMake packages are broken. For example, `find_package(Parquet)`
doesn't work without specifying `CMAKE_MODULE_PATH`.
`find_package(${PACKAGE})` searches
`${PREFIX}/${PACKAGE}/${PACKAGE}Config.cmake` or
`${PREFIX}/${PACKAGE}/Find${PACKAGE}.cmake`. But our .cmake files are
always installed `${PREFIX}/arrow/`. So `find_package(Parquet)` can't find
`${PREFIX}/arrow/FindParquet.cmake` because "`/arrow/`" isn't
"`/${PACKAGE}`".
This change fixes this by installing `${PACKAGE}Config.cmake` to
`${PREFIX}/${PACKAGE}/` instead of `${PREFIX}/arrow/`.
This also removes all `Find${PACKAGE}.cmake`. We only provides
`${PACKAGE}Config.cmake`. Our `Find${PACKAGE}.cmake` can find `${PACKAGE}`
by CMake, pkg-config or manual .so/.h search. But we don't need to
support pkg-config nor manual .so/.h search. We can use
`${PACKAGE}Config.cmake` to support CMake package search. So this
removes all `Find${PACKAGE}.cmake`.
This also introduces namespace to our CMake targets. For example,
`arrow_shared` is exported as `Arrow::arrow_shared` and
`parquet_static` is exported as `Parquet::parquet_static`. But no
namespace targets such as `arrow_shared` and `parquet_static` are
still also exported for keeping backward compatibility. But this
requires CMake 3.18 or later for users because `add_library(ALIAS)`
for non-global `IMPORTED` library is available since CMake 3.18.
(`Plasma::plasma-store-server` target is also added for
`plasma-store-server` executable.)
FYI: We can resolve this problem by using `COMPONENTS` feature of
`find_package()`. For example, `find_package(Arrow COMPONENTS
Parquet)` is used instead of `find_package(Parquet)`. With
`COMPONENTS`, `${PACKAGE}` is always "Arrow". So we can still install
our .cmake files to `${PREFIX}/arrow/`. But this approach breaks
backward compatibility. So I choose `${PREFIX}/${PACKAGE}/*.cmake`
approach.
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
test-build-vcpkg-win
https://github.com/ursacomputing/crossbow/runs/6892291430?check_suite_focus=true
Could not find a configuration file for package "xsimd" that is compatible
with requested version "8.1.0".
The following configuration files were considered but not accepted:
D:/a/crossbow/crossbow/arrow/cpp/vcpkg_installed/x64-windows/share/xsimd/xsimdConfig.cmake, version: 7.4.9
Call Stack (most recent call first):
cmake_modules/ThirdpartyToolchain.cmake:253 (find_package)
cmake_modules/ThirdpartyToolchain.cmake:2209 (resolve_dependency)
CMakeLists.txt:567 (include)
example-cpp-minimal-build-static-system-dependency
https://github.com/ursacomputing/crossbow/runs/6892331642?check_suite_focus=true
CMake Error at cmake_modules/ThirdpartyToolchain.cmake:253 (find_package):
By not providing "Findxsimd.cmake" in CMAKE_MODULE_PATH this project has
asked CMake to find a package configuration file provided by "xsimd", but
CMake did not find one.
Could not find a package configuration file provided by "xsimd" (requested
version 8.1.0) with any of the following names:
xsimdConfig.cmake
xsimd-config.cmake
Add the installation prefix of "xsimd" to CMAKE_PREFIX_PATH or set
"xsimd_DIR" to a directory containing one of the above files. If "xsimd"
provides a separate development package or SDK, be sure it has been
installed.
Call Stack (most recent call first):
cmake_modules/ThirdpartyToolchain.cmake:2209 (resolve_dependency)
CMakeLists.txt:567 (include)
test-r-depsource-system
https://github.com/ursacomputing/crossbow/runs/6892324900?check_suite_focus=true
CMake Error at cmake_modules/ThirdpartyToolchain.cmake:253 (find_package):
By not providing "Findxsimd.cmake" in CMAKE_MODULE_PATH this project has
asked CMake to find a package configuration file provided by "xsimd", but
CMake did not find one.
Could not find a package configuration file provided by "xsimd" (requested
version 8.1.0) with any of the following names:
xsimdConfig.cmake
xsimd-config.cmake
Add the installation prefix of "xsimd" to CMAKE_PREFIX_PATH or set
"xsimd_DIR" to a directory containing one of the above files. If "xsimd"
provides a separate development package or SDK, be sure it has been
installed.
Call Stack (most recent call first):
cmake_modules/ThirdpartyToolchain.cmake:2209 (resolve_dependency)
CMakeLists.txt:567 (include)
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
Bump the vendored `google-cloud-cpp` version so that `CMAKE_CXX_STANDARD` doesn't get ignored.
Also cleanup `CMAKE_CXX_STANDARD` handling in a few other places.
Closes#12257 from pitrou/ARROW-15451-gcs-cxx-standard
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
This demonstrates how to register a normal gRPC service on a Flight server, allowing you to use all of gRPC's features and ecosystem while still serving Flight, all on the same port. This can be helpful for when DoAction is too limited/inconvenient an extension point and you are already using gRPC.
Closes#11524 from lidavidm/arrow-14440
Authored-by: David Li <li.davidm96@gmail.com>
Signed-off-by: Yibo Cai <yibo.cai@arm.com>
This PR replaces most of the use of hardcoded paths
to things `#!/bin` and `#!/usr/bin`, so most of the tools
in `dev` and a bunch of the C++ development tools are usable
on systems that don't have bash at this location.
Closes#10953 from cpcloud/arrow-13648
Authored-by: Phillip Cloud <417981+cpcloud@users.noreply.github.com>
Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
The `io::IOContext` class allows passing various settings such as the MemoryPool used for allocation and the Executor for async methods.
Closes#9474 from pitrou/ARROW-10420-io-context
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Benjamin Kietzman <bengilgit@gmail.com>
Because we require RE2 with ARROW_COMPUTE by default.
See also: ARROW-10541 and #8756Closes#8786 from kou/cpp-example-static
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
Also rename `New(File|Stream)Writer` to `Make(File|Stream)Writer`
Closes#8084 from pitrou/ARROW-9874-make-writer
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
This adds support for using the `arrow_static` CMake target with system static libraries.
This adds the following CMake options:
* `ARROW_DEPENDENCY_USE_SHARED` (`ON` by default): The default value of `ARROW_*_USE_SHARED` except `ARROW_THRIFT_USE_SHARED` on Windows.
* `ARROW_BZ2_USE_SHARED`: Use shared library of bzip2.
* `ARROW_LZ4_USE_SHARED`: Use shared library of LZ4.
* `ARROW_THRIFT_USE_SHARED` (`OFF` by default on Windows): Use shared library of libthrift. This is `OFF` by
default on Windows because it seems that Thrift's DLL support isn't well for not.
* `ARROW_UTF8PROC_USE_SHARED`: Use shared library of utf8proc.
* `ARROW_SNAPPY_USE_SHARED`: Use shared library of Snappy.
* `ARROW_ZSTD_USE_SHARED`: Use shared library of ZStandard.
Note that users must specify `ARROW_*_USE_SHARED=OFF` explicitly when they want to use static library. This is a backward
incompatible change for some cases. For example, users can use shared library or static library without specifying any options
before this change. Because old `FindLz4.cmake` find both shared library and static library.
Note that upstream `SnappyConfig.cmake` and `zstdConfig.cmake` are never used with this change. We always use our
`Find*.cmake` for them to process `ARROW_*_USE_SHARED`.
Note that the `ZSTD_SOURCE` CMake option is renamed to `zstd_SOURCE`.
Some dependencies don't support static linking with `ARROW_*_USE_SHARED=OFF` yes. For example, gRPC may ignore it.
If `gRPCConfig.cmake` is installed, our `FindgRPCAlt.cmake` isn't used. It means that `ARROW_GRPC_USE_SHARED=OFF`
is ignored. We can work on this as further tasks.
RE2 is only required by Gandiva but it's searched by `find_package(Arrow)` not `find_package(Gandiva)`. It should be fixed
by a follow-up task.
Closes#7842 from tobim/ARROW-9412-unbundled-static-dependencies
Lead-authored-by: Tobias Mayer <tobim@fastmail.fm>
Co-authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
This PR is a renewed attempt to address the brokenness of our static libraries and their exported CMake targets.
To summarize what's wrong: if any BUNDLED library source is used, or if the jemalloc or mimalloc allocators are used, then the `libarrow.a` static library that we've been installing is unusable. If someone tries to link with it, they get a linker error looking for the missing dependencies. Additionally, the CMake target names exported for `arrow_static` includes the list of all of these bundled libraries, so even attempting to use the `arrow_static` target does not work since the third party CMake build cannot find the missing dependencies.
As one downstream problem caused by this, we've been unable thus far to ship our non-system allocators in packages (R in particular) that do static linking.
Since it is neither safe nor reasonable to expect third party projects to attempt to provide substitute static libraries for the missing dependencies, I have implemented the following:
* Using an MIT-licensed CMake solution, I splice together all of the bundled static libraries to create and install `libarrow_bundled_dependencies.a` (and `arrow_bundled_dependencies.lib` with MSVC). I have verified that this bundle is viable and can be used to create statically linked executables
* Bundled dependencies are no longer exported in ArrowTargets-*.cmake. I spent a bunch of time trying to figure out how to get CMake to automatically pull in the "dependency bundle" when using `arrow_static`, but I couldn't figure out how to do it.
* Added a `ARROW_LINK_SHARED` option to cpp/examples/minimal_build/CMakeLists.txt to show how to create statically linked executables using arrow_static and the dependency bundle.
* Added a "run_static.sh" script to enable a bundled statically-linked build to be tested with Docker (perhaps we should add this to the CI job?)
* Added "run_static.bat" to exhibit a statically linked build with MSVC on Windows
Finally, I verified that I can now successfully create statically linked builds now on Linux, macOS, and Windows.
[1]: https://gist.github.com/cristianadam/ef920342939a89fae3e8a85ca9459b49Closes#7696 from wesm/dependency-bundling
Lead-authored-by: Wes McKinney <wesm@apache.org>
Co-authored-by: Neal Richardson <neal.p.richardson@gmail.com>
Co-authored-by: Wes McKinney <wesm+git@apache.org>
Signed-off-by: Wes McKinney <wesm@apache.org>
* Default for ARROW_COMPUTE/DATASET/FILESYSTEM/JSON set to OFF
* Add ARROW_CSV option, set to OFF by default (I could be swayed about this, but I think it is good to not create the perception that users are forced to build a module they don't need)
* Disable unit tests that don't build if ARROW_COMPUTE=OFF
The minimal Docker build now outputs
https://gist.github.com/wesm/ff70f46f5bc256d6b1d7b979aab8e5d0
The minimal build time on my 8-core laptop is reduced from 72 seconds to 25 seconds. Here is the log for the default build (with jemalloc turned off) without this patch
https://gist.github.com/wesm/fb5300a1989420158b0148e1c49ab6d6
Note that the default for ARROW_JEMALLOC is left to ON. I agree based on other discussion that having it off by default might cause developers doing performance testing to draw the wrong conclusions due to worse performance from the system allocator.
TODO
- [ ] Check conda
- [ ] Check Homebrew
- [ ] Check wheels
- [ ] Check Linux packages
Closes#5890 from wesm/more-minimal-default-build and squashes the following commits:
3fde8c4fc <Wes McKinney> Do not pass ARROW_HDFS twice in ci/PKGBUILD
b0de54927 <Wes McKinney> Fix PKGBUILD, add documentation to developers/cpp.rst, fix manylinux wheel READMEs
99640e1a9 <Wes McKinney> Many modules are implied by ARROW_PYTHON=ON. Requires HDFS when building Python
93b8bca59 <Wes McKinney> Explicitly build optional modules in MSVC Appveyor script
439af405a <Wes McKinney> Enable various modules when ARROW_PYTHON=ON
e3058a895 <Wes McKinney> ARROW_COMPUTE required for arrow_python and parquet
48c16f01e <Wes McKinney> Build uriparser unconditionally as not worth the trouble to conditionally disable
cd381aaf7 <Wes McKinney> Provide for blanket env variable for minimal build script
9cbeabfb9 <Wes McKinney> More minimal default options, add ARROW_CSV CMake option
Authored-by: Wes McKinney <wesm+git@apache.org>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
With these changes, the minimal build now only requires double-conversion_ep. We still have to disable a number of components manually to obtain the minimal build, so there is more work to do
Closes#5611 from wesm/ARROW-6782 and squashes the following commits:
65b5faec6 <Wes McKinney> Boost also required for Gandiva
6f67a7eac <Wes McKinney> Do not require Boost for minimal C++ build
Authored-by: Wes McKinney <wesm+git@apache.org>
Signed-off-by: Wes McKinney <wesm+git@apache.org>
From the look of the diff, I would say that configuring a comprehensive build looks too complicated to me (it's only relevant for developers and packagers, though, so more a maintainability nuisance). Perhaps ARROW-6585 can help with this
Also fixes a unit test that only fails when all compressors are disabled. I needed to add a `Codec::IsAvailable` function to determine whether a codec's support has been built, which seems useful anyway.
Closes#5607 from wesm/ARROW-6631 and squashes the following commits:
19b83ac09 <Wes McKinney> Update docker_integration.sh to not manually disable compression
dbb4b8d70 <Wes McKinney> Review comments
bfb08d63e <Sutou Kouhei> Disable Zstandard on Ubuntu 16.04
01b75e9c3 <Sutou Kouhei> Sort alphabetically
b454874bf <Wes McKinney> Add function to determine if support for codec is available
12100b2a9 <Wes McKinney> Do not build any compression libraries by default in C++ build
Lead-authored-by: Wes McKinney <wesm+git@apache.org>
Co-authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Wes McKinney <wesm+git@apache.org>
Closes#5515 from pitrou/ARROW-6610-cmake-filesystem-flag and squashes the following commits:
b74741dde <Antoine Pitrou> ARROW-6610: Add cmake option to disable filesystem layer
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
The intent of this Dockerfile is to help with testing stripped down C++ core builds per the mailing list discussion. This was tested with the patch for ARROW-6590. Hopefully based on the outcome of the discussion we can make the build script build.sh shorter by having more barebones default options
Closes#5431 from wesm/cpp-minimal-dockerfile and squashes the following commits:
2288862eb <Wes McKinney> Remove unneeded autoconf from cpp developer docs -- included in build-essential
40e2fd98c <Wes McKinney> Remove more apt packages
07053b4b1 <Wes McKinney> Remove some unneeded apt packages
8fd0c3ed5 <Wes McKinney> Use local Arrow for minimal build
7b9ab6166 <Wes McKinney> Shallower clone depth
22d402c23 <Wes McKinney> Move instructions to build.sh
685ea1693 <Wes McKinney> remove file
64324090c <Wes McKinney> Add minimal C++ build Dockerfile to dev/cpp_minimal
Authored-by: Wes McKinney <wesm+git@apache.org>
Signed-off-by: Wes McKinney <wesm+git@apache.org>
Hiroyuki Sato