### Rationale for this change
This is the sub issue #44748.
* SC2034: `test_dir` appears unused. Verify it or export it.
* SC1091: Not following: (error message here)
* SC2086: Double quote to prevent globbing and word splitting.
```
shellcheck ci/scripts/python_test.sh
In ci/scripts/python_test.sh line 23:
test_dir=${1}/python/build/dist
^------^ SC2034 (warning): test_dir appears unused. Verify use (or export if used externally).
In ci/scripts/python_test.sh line 26:
. "${ARROW_PYTHON_VENV}/bin/activate"
^-- SC1091 (info): Not following: ./bin/activate: openBinaryFile: does not exist (No such file or directory)
In ci/scripts/python_test.sh line 45:
: ${PYARROW_TEST_ACERO:=${ARROW_ACERO:-ON}}
^-- SC2223 (info): This default assignment may cause DoS due to globbing. Quote it.
In ci/scripts/python_test.sh line 46:
: ${PYARROW_TEST_AZURE:=${ARROW_AZURE:-ON}}
^-- SC2223 (info): This default assignment may cause DoS due to globbing. Quote it.
In ci/scripts/python_test.sh line 47:
: ${PYARROW_TEST_CUDA:=${ARROW_CUDA:-ON}}
^-- SC2223 (info): This default assignment may cause DoS due to globbing. Quote it.
In ci/scripts/python_test.sh line 48:
: ${PYARROW_TEST_DATASET:=${ARROW_DATASET:-ON}}
^-- SC2223 (info): This default assignment may cause DoS due to globbing. Quote it.
In ci/scripts/python_test.sh line 49:
: ${PYARROW_TEST_FLIGHT:=${ARROW_FLIGHT:-ON}}
^-- SC2223 (info): This default assignment may cause DoS due to globbing. Quote it.
In ci/scripts/python_test.sh line 50:
: ${PYARROW_TEST_GANDIVA:=${ARROW_GANDIVA:-ON}}
^-- SC2223 (info): This default assignment may cause DoS due to globbing. Quote it.
In ci/scripts/python_test.sh line 51:
: ${PYARROW_TEST_GCS:=${ARROW_GCS:-ON}}
^-- SC2223 (info): This default assignment may cause DoS due to globbing. Quote it.
In ci/scripts/python_test.sh line 52:
: ${PYARROW_TEST_HDFS:=${ARROW_HDFS:-ON}}
^-- SC2223 (info): This default assignment may cause DoS due to globbing. Quote it.
In ci/scripts/python_test.sh line 53:
: ${PYARROW_TEST_ORC:=${ARROW_ORC:-ON}}
^-- SC2223 (info): This default assignment may cause DoS due to globbing. Quote it.
In ci/scripts/python_test.sh line 54:
: ${PYARROW_TEST_PARQUET:=${ARROW_PARQUET:-ON}}
^-- SC2223 (info): This default assignment may cause DoS due to globbing. Quote it.
In ci/scripts/python_test.sh line 55:
: ${PYARROW_TEST_PARQUET_ENCRYPTION:=${PARQUET_REQUIRE_ENCRYPTION:-ON}}
^-- SC2223 (info): This default assignment may cause DoS due to globbing. Quote it.
In ci/scripts/python_test.sh line 56:
: ${PYARROW_TEST_S3:=${ARROW_S3:-ON}}
^-- SC2223 (info): This default assignment may cause DoS due to globbing. Quote it.
In ci/scripts/python_test.sh line 72:
pytest -r s ${PYTEST_ARGS} --pyargs pyarrow
^------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
pytest -r s "${PYTEST_ARGS}" --pyargs pyarrow
In ci/scripts/python_test.sh line 76:
pytest ${PYTEST_RST_ARGS} ${arrow_dir}/docs/source/python
^----------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
^----------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
pytest "${PYTEST_RST_ARGS}" "${arrow_dir}"/docs/source/python
For more information:
https://www.shellcheck.net/wiki/SC2034 -- test_dir appears unused. Verify u...
https://www.shellcheck.net/wiki/SC1091 -- Not following: ./bin/activate: op...
https://www.shellcheck.net/wiki/SC2086 -- Double quote to prevent globbing ...
```
### What changes are included in this PR?
* SC2034: Add a ShellCheck directive to disable the unused variable check.
* SC1091: Add a ShellCheck directive to disable source file checking.
* SC2086: Quote variables.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
* GitHub Issue: #50535
Lead-authored-by: Hiroyuki Sato <hiroysato@gmail.com>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
In many places in the Python User Guide the code exampels are written with IPython directive (elsewhere code-block is used). IPython directives are converted to IPython format (`In` and `Out` during the doc build). This can lead to slower builds.
### What changes are included in this PR?
IPython directives are converted to runnable code-block (with `>>>` and `...`) and pytest doctest support for `.rst` files is added to the `conda-python-docs` CI job. This means the code in the Python User Guide is tested separately to the building of the documentation.
### Are these changes tested?
Yes, with the CI.
### Are there any user-facing changes?
Changes to the Python User Guide examples will have to be tested with `pytest --doctest-glob='*.rst' docs/source/python/file.rst`
* GitHub Issue: #28859
Lead-authored-by: AlenkaF <frim.alenka@gmail.com>
Co-authored-by: Alenka Frim <AlenkaF@users.noreply.github.com>
Co-authored-by: tadeja <tadeja@users.noreply.github.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
Debian 11 will reach EOL on 2024-07.
### What changes are included in this PR?
* Use Debian 12.
* Enable build options that can't be enabled with Debian 11.
* `ARROW_AZURE`
* `ARROW_FLIGHT_SQL`
* `ARROW_WITH_OPENTELEMETRY`
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
* GitHub Issue: #40454
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
We want python side tests of `AzureFileSystem` to run in CI.
### What changes are included in this PR?
- Add missing `export` to enable Azure pyarrow tests
- Enable azure in sdist tests.
- Enable Azure on macos python builds
- Enable azure in conda builds and install dependencies (Azure C++ SDK and azurite)
- Enable retries on C++ tests to mitigate https://github.com/apache/arrow/issues/40121
Probably all of this should have been included in https://github.com/apache/arrow/pull/39971
### Are these changes tested?
There is no new functionality to test
### Are there any user-facing changes?
No
* Closes: #40079
* GitHub Issue: #40079
Authored-by: Thomas Newton <thomas.w.newton@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
### What changes are included in this PR?
Enable Azure in linux and mac os wheel builds. Tried to copy GCS
Don't enable Azure for windows builds because windows builds where all failing. Failures were a combination of cmake version or `Could not find a package configuration file provided by "wil"`. I think it makes sense to come back to windows builds in another PR.
### Are these changes tested?
There is no new functionality to test.
### Are there any user-facing changes?
No
* Closes: #39352
Authored-by: Thomas Newton <thomas.w.newton@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
Disable debug memory pool for ASAN and Valgrind so that they can detect more subtle memory issues regarding to buffer tail bytes.
### What changes are included in this PR?
1. Add a `none` option to debug memory pool env var to make other things slightly easier.
2. Change `*_test.sh` scripts to conditionally set debug memory pool env var.
3. Top-level docker compose change to pass none to debug memory pool env var for ASAN and Valgrind.
### Are these changes tested?
The CI should cover it well.
### Are there any user-facing changes?
No.
* Closes: #39973
Authored-by: Ruoxi Sun <zanmato1984@gmail.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
If I build pyarrow with everything and then I remove some of the Arrow CPP .so in order to have a minimal build I can't import pyarrow because it requires libarrow and libparquet. This is relevant in order to have a minimal build for Conda. Please see the related issue for more information.
### What changes are included in this PR?
Move libarrow parquet encryption for pyarrow to its own shared object.
### Are these changes tested?
I will run extensive CI with extra python archery tests.
### Are there any user-facing changes?
No, and yes :) There will be a new .so on pyarrow but shouldn't be relevant in my opinion.
* Closes: #39006
Lead-authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Co-authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
See the linked issue
### What changes are included in this PR?
C++:
* remove all compute/exec/* from libarrow
* rename compute/exec -> acero and make libarrow_acero
* add new ARROW_ACERO option, required if ARROW_DATASET is on
* libarrow_dataset now depends on libarrow_acero
c_glib: add the new libarrow_acero dependency - we disallow building glib without it
python: added PYARROW_BUILD_ACERO, set to on if DATASETS are built
### Are these changes tested?
All the standard tests do work properly.
I manually compiled C++ with:
* no ARROW_ACERO
* ARROW_ACERO and no ARROW_DATASET
* ARROW_ACERO and ARROW_DATASET and no ARROW_SUBSTRAIT
I manually compiled python without ACERO & DATASET and with ACERO and without DATASET
### Are there any user-facing changes?
If users include compute/exec files directly then they'll have to update their code.
* Closes: #15280
Lead-authored-by: Davide Pasetto <dpasetto69@gmail.com>
Co-authored-by: Li Jin <ice.xelloss@gmail.com>
Co-authored-by: Weston Pace <weston.pace@gmail.com>
Signed-off-by: Li Jin <ice.xelloss@gmail.com>
This PR moves `src/arrow/python` directory into `pyarrow` and arranges PyArrow to build it. The build on the Python side is made in two steps:
1. `_run_cmake_pyarrow_cpp()` where the C++ part of the pyarrow is build first (the part that was moved in the refactoring)
2. `_run_cmake()` where pyarrow is built as before
No changes are needed in the build process from the user side to successfully build pyarrow after this refactoring. The test for PyArrow CPP will however be moved into Cython and can currently be run with:
```shell
>>> pushd python/build/dist/temp
>>> ctest
```
Lead-authored-by: Alenka Frim <frim.alenka@gmail.com>
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
Checked locally on a Ubuntu 20.04 host with:
```
archery docker run ubuntu-cuda-python
```
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
Incorporate GCS file system into python and other bug fixes.
Bugs/Other changes:
- Add GCS bindings mostly based on AWS bindings in Python and associated unit tests
- Tell was incorrect, it double counted when the stream was constructed with an offset.
- Missed setting the define in config.cmake which means `FileSystemFromUri was never tested and didn't compile this is now fixed`
- Refine logic for GetFileInfo with a single path to recognize prefixes followed by a slash as a directory. This allows datasets to work as expected with a toy dataset generated on local-filesystem and copied to the cloud (I believe this is typical of how other systems write to GCS as well.
- Switch convention for creating directories to always end in "/" and make use of this as another indicator. From testing with a sample iceberg table it appears this is the convention used for hive-partitioning, so I assume this is common practice for other Hive related writers (i.e. what we want to support).
- Fix bug introduced in https://github.com/apache/arrow/commit/a5e45cecb24229433b825dac64e0ffd10d400e8c which caused failures when a deletion occurred on a bucket (not an object in the bucket).
- Ensure output streams are closed on destruction (this is consistent with S3)
Lead-authored-by: Micah Kornfield <micahk@google.com>
Co-authored-by: emkornfield <emkornfield@gmail.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
Debug memory checks can be enabled by setting the environment variable ARROW_DEBUG_MEMORY_POOL
to one of the recognized values "abort", "trap" or "warn" (regardless of whether Arrow
was compiled in debug or release mode).
The only implemented check adds a suffix past the allocation and checks that it isn't clobbered on deallocation.
Adding a prefix would be more expensive because of alignment constraints.
Also add a fix by Will Jones for ARROW-14047 (Parquet Arrow reader sets null values in buffer overflow), since otherwise CI would fail.
Closes#12330 from pitrou/ARROW-15550-debug-memory-checks
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
Tests for optional components could be silently skipped if by accident the component would fail importing despite being built.
Closes#12301 from pitrou/ARROW-15504-py-test-optional-components
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
Still need to port the R changes from #10470
Tested locally using:
```
PYARROW_TEST_SLOW=ON PYARROW_TEST_LARGE_MEMORY=ON ./run_test.sh -sv pyarrow/tests/
```
Closes#10556 from kszucs/fff
Authored-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
Closes#6484 from pitrou/ARROW-7930-jpype-integration and squashes the following commits:
686ec6537 <Antoine Pitrou> ARROW-7930: Test jpype integration
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
Python's "development mode" enables some runtime checks and warnings
that help developers spot errors in their code (such as resource leaks).
Closes#6473 from pitrou/ARROW-7915-py-dev-mode and squashes the following commits:
1c2b91dcd <Antoine Pitrou> Enable Python dev mode also on AppVeyor
e4363961d <Antoine Pitrou> ARROW-7915: Enable development mode in tests
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
- ccache configuration is centralised in the docker-compose.yml
- stricter build directory paths
- java jni tests were not running, since the vendoring of flatbuffers there was a hidden linking error
Closes#5982 from kszucs/explicit_use_ccache and squashes the following commits:
90fcd0c05 <Krisztián Szűcs> Sort env definitions alphabetically
30a64dbb5 <Krisztián Szűcs> Lint CMake
d76282bde <Krisztián Szűcs> enable plasma java client; fix flatbuffers link error because it is bundled now
b6d47db8c <Krisztián Szűcs> Fix PLASMA_STORE binary path
c4648afa4 <Krisztián Szűcs> Explicit ccache; centralized config in docker-compose
Authored-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
Also improve some builds to disable unnecessary components.
Closes#5809 from pitrou/ARROW-7133-gh-actions-all-branches and squashes the following commits:
463bf5fc8 <Antoine Pitrou> WIP
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
## Projecting ideas from ursabot
### Parametric docker images
The images are better parameterized now, meaning that we can build more variant of the same service. Couple of examples:
```console
UBUNTU=16.04 docker-compose build ubuntu-cpp
ARCH=arm64v8 UBUNTU=18.04 docker-compose build ubuntu-cpp
PYTHON=3.6 docker-compose build conda-python
ARCH=arm32v7 PYTHON=3.6 PANDAS=0.25 docker-compose build conda-python-pandas
```
Each variant has it's own docker image following a string naming schema:
`{org}/{arch}-{platform}-{platform-version}[[-{variant}-{variant-version}]..]:latest`
### Use *_build.sh and *_test.sh for each job
The docker images provide the environment, and each language backend usually should implement two scripts, a `build.sh` and a `test.sh`. This way dependent build like the docker python, r or c glib are able to reuse the build script of the ancestor without running its tests.
With small enough scripts, if the environment is properly set up even the non-docker builds should be reproducible locally. GitHub Actions support bash scripts across all three platforms, so we can reuse the same `*_build.sh` and `*_test.sh` scripts to execute the builds either in docker, on the CI or locally.
## Using GitHub Actions for running the builds
Regardless of the CI we're going to choose, the isolation constraint of different platforms requires some sort of virtualisation. Currently linux (and windows, but I have not tried it yet) has lightweight containerisation, so we should keep the linux builds isolated in docker containers. The rest of the platforms (windows and macOS) should be executed on the CI system.
GitHub Actions support all three major platforms, linux, windows and macOS. I've added cross platform builds for a couple of languages, like Rust, and Go, the rest are work in progress.
### Workflow
A workflow should define all builds of a language, mostly because the path filters can be defined on workflow level. For example the python builds should be triggered if either a cpp/** or a python/** file changes which can be covered in the same workflow file.
## Feature parity with the current builds
Reaching feature parity with all of the builds below is not a goal for this PR, the difficult ones should at least have a tracking JIRA ticket.
### Travis-CI
- [x] **Lint, Release tests**:
- `Lint / C++, Python, R, Rust, Docker, RAT`
- `Dev / Source Release`
- [x] **C++ unit tests w/ conda-forge toolchain, coverage**: without coverage
- `C++ / AMD64 Conda C++`
- [x] **Python 3.6 unit tests, conda-forge toolchain, coverage**: without coverage
- `Python / AMD64 Conda Python 3.6`
- [x] **[OS X] C++ w/ Xcode 9.3**:
- `C++ / AMD64 MacOS 10.14 C++`: with Xcode 10.3
- [x] **[OS X] Python w/ Xcode 9.3**:
- `Python / AMD64 MacOS 10.14 Python 3`: with Xcode 10.3
- [x] **Java OpenJDK8 and OpenJDK11**:
- `Java / AMD64 Debian Java JDK 8 Maven 3.5.2`
- `Java / AMD64 Debian Java JDK 11 Maven 3.6.2`
- [x] **Protocol / Flight Integration Tests**:
- `Dev / Protocol Test`
- [x] **NodeJS**: without running lint and coverage
- `NodeJS / AMD64 Debian NodeJS 11`
- [x] **C++ & GLib & Ruby w/ gcc 5.4**:
- `C++ / AMD64 Debian 10 C++`: with GCC 8.3
- `C++ / AMD64 Ubuntu 16.04 C++`: with GCC 5.4
- `C++ / AMD64 Ubuntu 18.04 C++`: with GCC 7.4
- `C GLib / AMD64 Ubuntu 18.04 C GLib`
- `Ruby / AMD64 Ubuntu 18.04 Ruby`
- [x] **[OS X] C++ & GLib & Ruby w/ XCode 10.2 & Homebrew**
- `C++ / AMD64 MacOS 10.14 C++`: with Xcode 10.3
- `C GLib / AMD64 MacOS 10.14 C Glib`: with Xcode 10.3
- `Ruby / AMD64 MacOS 10.14 Ruby`: with Xcode 10.3
- [x] **Go**: without coverage
- `Go / AMD64 Debian Go 1.12`
- [x] **R (with and without libarrow)**:
- `R / AMD64 Conda R 3.6`: with libarrow
- `R / AMD64 Ubuntu 18.04 R 3.6` with libarrow
### Appveyor
- ~JOB=Build, GENERATOR=Ninja, CONFIGURATION=Release, APPVEYOR_BUILD_WORKER_IMAGE=Visual Studio 2017~
- ~JOB=Toolchain, GENERATOR=Ninja, CONFIGURATION=Release, ARROW_S3=ON, ARROW_BUILD_FLIGHT=ON, ARROW_BUILD_GANDIVA=ON~
- ~JOB=Build_Debug, GENERATOR=Ninja, CONFIGURATION=Debug~
- ~JOB=MinGW32, MINGW_ARCH=i686, MINGW_PACKAGE_PREFIX=mingw-w64-i686, MINGW_PREFIX=c:\msys64\mingw32, MSYSTEM=MINGW32, USE_CLCACHE=false~
- ~JOB=MinGW64, MINGW_ARCH=x86_64, MINGW_PACKAGE_PREFIX=mingw-w64-x86_64, MINGW_PREFIX=c:\msys64\mingw64, MSYSTEM=MINGW64, USE_CLCACHE=false~
- [x] **JOB=Rust, TARGET=x86_64-pc-windows-msvc, USE_CLCACHE=false**:
- `Rust / AMD64 Windows 2019 Rust nightly-2019-09-25`
- [x] **JOB=C#, APPVEYOR_BUILD_WORKER_IMAGE=Visual Studio 2017, USE_CLCACHE=false**
- `C# / AMD64 Windows 2019 C# 2.2.103`
- [x] **JOB=Go, MINGW_PACKAGE_PREFIX=mingw-w64-x86_64 ...**:
- `Go / AMD64 Windows 2019 Go 1.12`
- ~JOB=R with libarrow, USE_CLCACHE=false, TEST_R_WITH_ARROW=TRUE, RWINLIB_LOCAL=%APPVEYOR_BUILD_FOLDER%\libarrow.zip~
### Github Actions
- [x] **Windows MSVC C++ / Build (Visual Studio 16 2019)**:
- `C++ / AMD64 Windows 2019 C++`: without tests
- [x] **Windows MSVC C++ / Build (Visual Studio 15 2017)**:
- `C++ / AMD64 Windows 2016 C++`: without tests
- [x] **Linux docker-compose / Test (C++ w/ clang-7 & system packages)**: all have llvm for gandiva but the compiler is set to gcc
- `C++ / AMD64 Debian 10 C++`: with GCC 8.3
- `C++ / AMD64 Ubuntu 16.04 C++`: with GCC 5.4
- `C++ / AMD64 Ubuntu 18.04 C++`: with GCC 7.4
- [x] **Linux docker-compose / Test (Rust)**: without rustfmt
- `Rust / AMD64 Debian Rust nightly-2019-09-25`
- [x] **Linux docker-compose / Test (Lint, Release tests)**:
- `Lint / C++, Python, R, Rust, Docker, RAT`
- `Dev / Source Release`
### Nightly Crossbow tests
The packaging builds are out of the scope if this PR, but the nightly **dockerized test** task are in.
Nightly tests:
- [x] docker-r
- [x] docker-r-conda
- [x] docker-r-sanitizer
- [x] docker-rust
- [x] docker-cpp
- [x] docker-cpp-cmake32
- [x] docker-cpp-release
- [x] docker-cpp-static-only
- [x] docker-c_glib
- [x] docker-go
- [x] docker-python-2.7
- [x] docker-python-3.6
- [x] docker-python-3.7
- [x] docker-python-2.7-nopandas
- [x] docker-python-3.6-nopandas
- [x] docker-java
- [x] docker-js
- [x] docker-docs
- [x] docker-lint
- [x] docker-iwyu: included in the lint
- [x] docker-clang-format: included in the lint
- [x] docker-pandas-master
- [x] docker-dask-integration
- [x] docker-hdfs-integration
- [x] docker-spark-integration
- [x] docker-turbodbc-integration
# TODOs left:
- [x] Fix the Apidoc generation for c_glib
- [x] Fix the JNI test for Gandiva and ORC
- [x] Test that crossbow tests are passing
- ~Optionally restore the travis configuration to incrementally decommission old builds~
## Follow-up JIRAs:
- [Archery] Consider porting the docker tool of ursabot to archery
- [Archery] Consider to use archery with or instead of the pre-commit hooks
- [Archery] Create a wrapper script in archery for docker compose in order to run the containers with the host's user and group
- [C++] GCC 5.4.0 has a compile errors, reproduce with UBUNTU=16.04 docker-compose run ubuntu-cpp
- [C++][CI] Test the ported fuzzit integration image
- [C++][CI] Turn off unnecessary features in the integration tests (spark/turbodbc/dask/hdfs)
- [C++][CI] Revisit ASAN UBSAN settings in every C++ based image
- [CI] Consider re-adding the removed debian testing image is removed
- [Go][CI] Pre-install the go dependencies in the dockerfile using go get
- [JS][CI] Pre-install the JS dependencies in the dockerfile
- [Rust][CI] Pre-install the rust dependencies in the dockerfile
- [Java][CI] Pre-install the java dependencies in the dockerfile
- [Ruby][CI] Pre-install the ruby dependencies in the dockerfile and remove it from the test script
- [C#][CI] Pre-install the C# dependencies in the dockerfile
- [R][CI] Fix the r-sanitizer build https://issues.apache.org/jira/browse/ARROW-6957
- [GLIB][MacOS] Fail to execute lua examples (fails to load 'lgi.corelgilua51' despite that lgi is installed)
- [C++][CMake] Automatically set ARROW_GANDIVA_PC_CXX_FLAGS for conda and OSX sdk (see cpp_build.sh)
- [C++][CI] Hiveserver2 instegration test fails to connect to impala container
- [CI][Spark] Support specific Spark version in the integration tet including latest
- [JS][CI] Move nodejs linting from js_build.sh to archery
- [Python][CI] create a docker image for python ASV benchmarks and fix the script
- [CI] Find a short but related prefix for the env vars used for the docker-compose file to prevent collisions
- [C#] the docker container fails to run because of the ubuntu host versions, see https://github.com/dotnet/core/issues/3509
- [C++][Windows] Enable more features on the windows GHA build
- [Doc] document docker-compose usage in the developer sphinx guide
- [CI][C++] Add .ccache to the docker-compose mounts
- [Archery][CI] Refactor the ci/scripts to a sourceable bash functions or to archery directly
- [C++][CI] Use scripts/util_coredump.sh to show automatic backtraces
- [C++] Fix the hanging C++ tests in Windows 2019
- [CI] Ask INFRA to set up the DOCKERHUB_* secrets for GitHub actions
- [C++][CI] Running Gandiva tests fails on Fedora:
Reproduce with: `docker-compose run -e ARROW_GANDIVA=ON fedora-cpp`
```
Running gandiva-internals-test, redirecting output into /build/cpp/build/test-logs/gandiva-internals-test.txt (attempt 1/1)
1364
: CommandLine Error: Option 'x86-experimental-vector-widening-legalization' registered more than once!
1365
LLVM ERROR: inconsistency in registered CommandLine options
1366
/build/cpp/src/gandiva
```
- [JS][CI] NodeJS build fails on Github Actions Windows node
```
> NODE_NO_WARNINGS=1 gulp build
# 'NODE_NO_WARNINGS' is not recognized as an internal or external command,
# operable program or batch file.
# npm ERR! code ELIFECYCLE
# npm ERR! errno 1
# npm ERR! apache-arrow@1.0.0-SNAPSHOT build: `NODE_NO_WARNINGS=1 gulp build`
# npm ERR! Exit status 1
# npm ERR!
# npm ERR! Failed at the apache-arrow@1.0.0-SNAPSHOT build script.
# npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
```
Closes#5589 from kszucs/docker-refactor and squashes the following commits:
5105d12e6 <Krisztián Szűcs> Rename pull-request folder to dev_cron
e9e9a7eec <Krisztián Szűcs> Use underscores for naming the workflow files
a92c99d03 <Krisztián Szűcs> Disable hanging C++ tests on windows
f158c89b5 <Krisztián Szűcs> Attempt to push from apache/arrow master; Don't push from crossbow tasks
0e1d470a1 <Krisztián Szűcs> Turn off ORC on macOS C++ test due to link error
258db5cff <Krisztián Szűcs> Only push docker images from apache/arrow repository
acdfcf086 <Krisztián Szűcs> Remove ORC from the brewfile
5102b85b1 <Krisztián Szűcs> Fix nodeJS workflow
032d6a388 <Krisztián Szűcs> Turn off 2 python builds
7f15b97a8 <Krisztián Szűcs> Filter branches
48b8d128a <Krisztián Szűcs> Fix workflows
36ad9d297 <Krisztián Szűcs> Disable builds
0f603af0c <Krisztián Szűcs> master only and cron workflows
28cc2d78d <Krisztián Szűcs> Rename Java JNI workflow
bcd8af7b7 <Krisztián Szűcs> Port the remaining travis utility scripts
ed5688154 <Krisztián Szűcs> Usage comments; recommend installing pandas from the docs because of its removal from conda_env_python
3c8c023ce <Krisztián Szűcs> Use Arch in volumes; some comments; remove conda version 'latest' from the images
771b023a8 <Krisztián Szűcs> Cleanup files; separate JNI builds
97ff8a122 <Krisztián Szűcs> Push docker images only from master
dc00b4297 <Krisztián Szűcs> Enable path filters
e0e2e1f46 <Krisztián Szűcs> Fix pandas master build
3814e0828 <Krisztián Szűcs> Fix manylinux volumes
c18edda70 <Krisztián Szűcs> Add CentOS version to the manylinux image names
c8b9dd6b1 <Krisztián Szűcs> Missing --pyargs argument for the python test command
33e646981 <Krisztián Szűcs> Turn off gandiva and flight for the HDFS test
b9c547889 <Krisztián Szűcs> Refactor docker-compose file and use it with github actions.
Authored-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
