mirror of
https://github.com/apache/arrow.git
synced 2026-10-29 00:56:31 +08:00
main
27
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Antoine Pitrou
|
503cee9aee |
GH-51138: [Python] Fix ccache efficiency (#51139)
### Rationale for this change Our current ccache support for PyArrow currently has two issues: 1. The custom configuration is set using environment variables, but those are not passed to the build processes according to the [CMake doc](https://cmake.org/cmake/help/latest/command/set.html#set-environment-variable) (I have verified this by enabling ccache logging to see the configuration values for each build command) 2. Cython-generated C++ sources are written in the temporary build directory (because of build isolation), and therefore their path changes everytime. Consequently, ccache would fail reusing previous compilation results. ### What changes are included in this PR? 1. Pass custom ccache configuration using command-line arguments, not environment variables. 2. Set the ccache configuration variable `base_dir` to the temporary build directory so that ccache strips away the build directory and hits previously cached results obtained from a different build directory. ### Are these changes tested? Yes, locally I confirmed that ccache now efficiently reuses compilation outputs for Cython-generated C++ sources. Before: ``` real 0m50,313s user 4m22,991s sys 0m24,805s ``` After: ``` real 0m18,698s user 1m14,727s sys 0m7,518s ``` ### Are there any user-facing changes? No. * GitHub Issue: #51138 Authored-by: Antoine Pitrou <antoine@python.org> Signed-off-by: Antoine Pitrou <antoine@python.org> |
||
|
|
f1b21f1945 |
GH-36411: [Python] Use scikit-build-core as build backend for PyArrow and get rid of setup.py (#49259)
### Rationale for this change Move our PyArrow build backend from setuptools and a custom setup.py to scikit-build-core which is just build backend for CMake related projects. ### What changes are included in this PR? Move from setuptools to scikit-build-core and remove PyArrow setup.py. Update some of the build requirements and minor fixes. A custom build backend has been also been created in order to wrap scikit-build-core in order to fix problems on License files for monorepos. pyproject.toml metadata validation expects license files to exist before exercising the build backend that's why we create symlinks. Our thin build backend will just make those symlinks hard-links in order for license and notice files to contain the contents and be added as part of the sdist. Remove flags that are not used anymore (were only part of setup.py) and documented and validated how the same flags have to be used now. ### Are these changes tested? Yes all Python CI tests, wheels and sdist are successful. ### Are there any user-facing changes? Yes, users building PyArrow will now require the new build dependencies to exercise the build and depending on the flags used they might require to use the new documented way of using those flags. * GitHub Issue: #36411 Lead-authored-by: Raúl Cumplido <raulcumplido@gmail.com> Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com> Co-authored-by: Rok Mihevc <rok@mihevc.org> Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com> |
||
|
|
bb81a6eeee |
GH-49102: [CI] Add type checking infrastructure and CI workflow for type annotations (#48618)
### Rationale for this change This is the first in series of PRs adding type annotations to pyarrow and resolving #32609. ### What changes are included in this PR? This PR establishes infrastructure for type checking: - Adds CI workflow for running mypy, pyright, and ty type checkers on linux, macos and windows - Configures type checkers to validate stub files (excluding source files for now) - Adds PEP 561 `py.typed` marker to enable type checking - Updates wheel build scripts to include stub files in distributions - Creates initial minimal stub directory structure - Updates developer documentation with type checking workflow ### Are these changes tested? No. This is mostly a CI change. ### Are there any user-facing changes? This does not add any actual annotations (only `py.typed` marker) so user should not be affected. * GitHub Issue: #32609 * GitHub Issue: #49102 Lead-authored-by: Rok Mihevc <rok@mihevc.org> Co-authored-by: Sutou Kouhei <kou@cozmixng.org> Co-authored-by: Raúl Cumplido <raulcumplido@gmail.com> Signed-off-by: Rok Mihevc <rok@mihevc.org> |
||
|
Antoine Pitrou
|
939dd95ac7 |
GH-49029: [Doc] Run sphinx-build in parallel (#49026)
### Rationale for this change `sphinx-build` allows for parallel operation, but it builds serially by default and that can be very slow on our docs given the amount of documents (many of them auto-generated from API docs). ### Are these changes tested? By existing CI jobs. ### Are there any user-facing changes? No. * GitHub Issue: #49029 Authored-by: Antoine Pitrou <antoine@python.org> Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com> |
||
|
Hiroyuki Sato
|
0ae71f81d5 |
GH-47306: [CI][Dev] Fix shellcheck errors in the ci/scripts/python_build.sh (#47307)
### Rationale for this change This is the sub issue #44748. * SC1091: Not following: ./bin/activate: openBinaryFile: does not exist * SC2034: foo appears unused * SC2086: Double quote to prevent globbing and word splitting * SC2223: This default assignment may cause DoS due to globbing. Quote it. * SC2236: Use `-n` instead of `! -z` ``` shellcheck ci/scripts/python_build.sh In ci/scripts/python_build.sh line 28: : ${BUILD_DOCS_PYTHON:=OFF} ^-----------------------^ SC2223 (info): This default assignment may cause DoS due to globbing. Quote it. In ci/scripts/python_build.sh line 31: git config --global --add safe.directory ${arrow_dir} ^----------^ SC2086 (info): Double quote to prevent globbing and word splitting. Did you mean: git config --global --add safe.directory "${arrow_dir}" In ci/scripts/python_build.sh line 35: . "${ARROW_PYTHON_VENV}/bin/activate" ^-- SC1091 (info): Not following: ./bin/activate: openBinaryFile: does not exist (No such file or directory) In ci/scripts/python_build.sh line 53: if [ ! -z "${CONDA_PREFIX}" ]; then ^-- SC2236 (style): Use -n instead of ! -z. In ci/scripts/python_build.sh line 77: : ${CMAKE_PREFIX_PATH:=${ARROW_HOME}} ^-- SC2223 (info): This default assignment may cause DoS due to globbing. Quote it. In ci/scripts/python_build.sh line 85: rm -rf ${python_build_dir} ^-----------------^ SC2086 (info): Double quote to prevent globbing and word splitting. Did you mean: rm -rf "${python_build_dir}" In ci/scripts/python_build.sh line 86: cp -aL ${source_dir} ${python_build_dir} ^-----------^ SC2086 (info): Double quote to prevent globbing and word splitting. ^-----------------^ SC2086 (info): Double quote to prevent globbing and word splitting. Did you mean: cp -aL "${source_dir}" "${python_build_dir}" In ci/scripts/python_build.sh line 87: pushd ${python_build_dir} ^-----------------^ SC2086 (info): Double quote to prevent globbing and word splitting. Did you mean: pushd "${python_build_dir}" In ci/scripts/python_build.sh line 101: rm -rf ${python_build_dir}/docs/source ^-----------------^ SC2086 (info): Double quote to prevent globbing and word splitting. Did you mean: rm -rf "${python_build_dir}"/docs/source In ci/scripts/python_build.sh line 102: mkdir -p ${python_build_dir}/docs ^-----------------^ SC2086 (info): Double quote to prevent globbing and word splitting. Did you mean: mkdir -p "${python_build_dir}"/docs In ci/scripts/python_build.sh line 103: cp -a ${arrow_dir}/docs/source ${python_build_dir}/docs/ ^----------^ SC2086 (info): Double quote to prevent globbing and word splitting. ^-----------------^ SC2086 (info): Double quote to prevent globbing and word splitting. Did you mean: cp -a "${arrow_dir}"/docs/source "${python_build_dir}"/docs/ In ci/scripts/python_build.sh line 104: rm -rf ${python_build_dir}/format ^-----------------^ SC2086 (info): Double quote to prevent globbing and word splitting. Did you mean: rm -rf "${python_build_dir}"/format In ci/scripts/python_build.sh line 105: cp -a ${arrow_dir}/format ${python_build_dir}/ ^----------^ SC2086 (info): Double quote to prevent globbing and word splitting. ^-----------------^ SC2086 (info): Double quote to prevent globbing and word splitting. Did you mean: cp -a "${arrow_dir}"/format "${python_build_dir}"/ In ci/scripts/python_build.sh line 106: rm -rf ${python_build_dir}/cpp/examples ^-----------------^ SC2086 (info): Double quote to prevent globbing and word splitting. Did you mean: rm -rf "${python_build_dir}"/cpp/examples In ci/scripts/python_build.sh line 107: mkdir -p ${python_build_dir}/cpp ^-----------------^ SC2086 (info): Double quote to prevent globbing and word splitting. Did you mean: mkdir -p "${python_build_dir}"/cpp In ci/scripts/python_build.sh line 108: cp -a ${arrow_dir}/cpp/examples ${python_build_dir}/cpp/ ^----------^ SC2086 (info): Double quote to prevent globbing and word splitting. ^-----------------^ SC2086 (info): Double quote to prevent globbing and word splitting. Did you mean: cp -a "${arrow_dir}"/cpp/examples "${python_build_dir}"/cpp/ In ci/scripts/python_build.sh line 109: rm -rf ${python_build_dir}/ci ^-----------------^ SC2086 (info): Double quote to prevent globbing and word splitting. Did you mean: rm -rf "${python_build_dir}"/ci In ci/scripts/python_build.sh line 110: cp -a ${arrow_dir}/ci/ ${python_build_dir}/ ^----------^ SC2086 (info): Double quote to prevent globbing and word splitting. ^-----------------^ SC2086 (info): Double quote to prevent globbing and word splitting. Did you mean: cp -a "${arrow_dir}"/ci/ "${python_build_dir}"/ In ci/scripts/python_build.sh line 111: ncpus=$(python -c "import os; print(os.cpu_count())") ^---^ SC2034 (warning): ncpus appears unused. Verify use (or export if used externally). In ci/scripts/python_build.sh line 113: pushd ${build_dir} ^----------^ SC2086 (info): Double quote to prevent globbing and word splitting. Did you mean: pushd "${build_dir}" In ci/scripts/python_build.sh line 116: ${python_build_dir}/docs/source \ ^-----------------^ SC2086 (info): Double quote to prevent globbing and word splitting. Did you mean: "${python_build_dir}"/docs/source \ In ci/scripts/python_build.sh line 117: ${build_dir}/docs ^----------^ SC2086 (info): Double quote to prevent globbing and word splitting. Did you mean: "${build_dir}"/docs For more information: https://www.shellcheck.net/wiki/SC2034 -- ncpus appears unused. Verify use ... https://www.shellcheck.net/wiki/SC1091 -- Not following: ./bin/activate: op... https://www.shellcheck.net/wiki/SC2086 -- Double quote to prevent globbing ... ``` ### What changes are included in this PR? * SC1091: Skip file check * SC2034: remove the variable * SC2086: Quote variables * SC2223: Quote variables * SC2236: Use `-n` instead of `! -z` ### Are these changes tested? Yes. ### Are there any user-facing changes? No. * GitHub Issue: #47306 Authored-by: Hiroyuki Sato <hiroysato@gmail.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com> |
||
|
Sutou Kouhei
|
46e7816093 |
GH-41430: [Docs] Use sphinxcontrib-mermaid instead of generating images from .mmd (#41455)
### Rationale for this change This is for easy to maintain. ### What changes are included in this PR? * Install sphinxcontrib-mermaid * Install Chromium to generate SVG from .mmd * Use Debian instead of Ubuntu for building docs because Ubuntu provides Chromium only via snap * Use a normal user not root to build documents because Mermaid require additional `--no-sandbox` argument when we use root ### Are these changes tested? Yes. ### Are there any user-facing changes? No. * GitHub Issue: #41430 Authored-by: Sutou Kouhei <kou@clear-code.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com> |
||
|
Hyunseok Seo
|
80e1d7a85b |
GH-41314: [CI][Python] Add a job on ARM64 macOS (#41313)
### Rationale for this change We need the GitHub-hosted ARM64 macOS to do more specific testing for the Python test. ### What changes are included in this PR? Add a job on ARM64 macOS (version: 14) for python ### Are these changes tested? Yes. ### Are there any user-facing changes? No. * GitHub Issue: #41314 Authored-by: Hyunseok Seo <hsseo0501@gmail.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com> |
||
|
Sutou Kouhei
|
93816475f7 |
GH-40454: [CI][Debian] Update Debian to 12 from 11 (#40455)
### Rationale for this change Debian 11 will reach EOL on 2024-07. ### What changes are included in this PR? * Use Debian 12. * Enable build options that can't be enabled with Debian 11. * `ARROW_AZURE` * `ARROW_FLIGHT_SQL` * `ARROW_WITH_OPENTELEMETRY` ### Are these changes tested? Yes. ### Are there any user-facing changes? No. * GitHub Issue: #40454 Authored-by: Sutou Kouhei <kou@clear-code.com> Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com> |
||
|
Thomas Newton
|
a0dec7f393 |
GH-39352: [FS][Azure] Enable azure in builds (#39971)
### Rationale for this change ### What changes are included in this PR? Enable Azure in linux and mac os wheel builds. Tried to copy GCS Don't enable Azure for windows builds because windows builds where all failing. Failures were a combination of cmake version or `Could not find a package configuration file provided by "wil"`. I think it makes sense to come back to windows builds in another PR. ### Are these changes tested? There is no new functionality to test. ### Are there any user-facing changes? No * Closes: #39352 Authored-by: Thomas Newton <thomas.w.newton@gmail.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com> |
||
|
|
4b7d699ab9 |
GH-37500: [CI][C++] Disable Dataset and Substrait by default (#37501)
### Rationale for this change Datasets and Substrait are heavy subcomponents (as they have non-trivial dependencies), so should not be enabled unwillingly in all CI builds. ### What changes are included in this PR? Disable Dataset and Substrait in `ci/scripts/cpp_build.sh`, enable them explicitly in relevant builds. Disable more subcomponents in CI builds that don't need them, especially "minimal" builds. ### Are these changes tested? Yes, by existing CI jobs. ### Are there any user-facing changes? No. * Closes: #37500 Lead-authored-by: Antoine Pitrou <antoine@python.org> Co-authored-by: Sutou Kouhei <kou@cozmixng.org> Co-authored-by: Sutou Kouhei <kou@clear-code.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com> |
||
|
|
f137f29f53 |
GH-15280: [C++][Python][GLib] add libarrow_acero containing everything previously in compute/exec (#34711)
### Rationale for this change See the linked issue ### What changes are included in this PR? C++: * remove all compute/exec/* from libarrow * rename compute/exec -> acero and make libarrow_acero * add new ARROW_ACERO option, required if ARROW_DATASET is on * libarrow_dataset now depends on libarrow_acero c_glib: add the new libarrow_acero dependency - we disallow building glib without it python: added PYARROW_BUILD_ACERO, set to on if DATASETS are built ### Are these changes tested? All the standard tests do work properly. I manually compiled C++ with: * no ARROW_ACERO * ARROW_ACERO and no ARROW_DATASET * ARROW_ACERO and ARROW_DATASET and no ARROW_SUBSTRAIT I manually compiled python without ACERO & DATASET and with ACERO and without DATASET ### Are there any user-facing changes? If users include compute/exec files directly then they'll have to update their code. * Closes: #15280 Lead-authored-by: Davide Pasetto <dpasetto69@gmail.com> Co-authored-by: Li Jin <ice.xelloss@gmail.com> Co-authored-by: Weston Pace <weston.pace@gmail.com> Signed-off-by: Li Jin <ice.xelloss@gmail.com> |
||
|
Sutou Kouhei
|
2c9340dd17 |
GH-33243: [Plasma] Remove (#34718)
### Rationale for this change We can't maintain Plasma. ### What changes are included in this PR? These changes remove all Plasma related codes. ### Are these changes tested? Yes. ### Are there any user-facing changes? Yes. * Closes: #33243 Authored-by: Sutou Kouhei <kou@clear-code.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com> |
||
|
emkornfieldandemkornfield
|
7b5912deea |
ARROW-14892: [Python][C++] GCS Bindings (#12763)
Incorporate GCS file system into python and other bug fixes. Bugs/Other changes: - Add GCS bindings mostly based on AWS bindings in Python and associated unit tests - Tell was incorrect, it double counted when the stream was constructed with an offset. - Missed setting the define in config.cmake which means `FileSystemFromUri was never tested and didn't compile this is now fixed` - Refine logic for GetFileInfo with a single path to recognize prefixes followed by a slash as a directory. This allows datasets to work as expected with a toy dataset generated on local-filesystem and copied to the cloud (I believe this is typical of how other systems write to GCS as well. - Switch convention for creating directories to always end in "/" and make use of this as another indicator. From testing with a sample iceberg table it appears this is the convention used for hive-partitioning, so I assume this is common practice for other Hive related writers (i.e. what we want to support). - Fix bug introduced in https://github.com/apache/arrow/commit/a5e45cecb24229433b825dac64e0ffd10d400e8c which caused failures when a deletion occurred on a bucket (not an object in the bucket). - Ensure output streams are closed on destruction (this is consistent with S3) Lead-authored-by: Micah Kornfield <micahk@google.com> Co-authored-by: emkornfield <emkornfield@gmail.com> Signed-off-by: Antoine Pitrou <antoine@python.org> |
||
|
Vibhatha Abeykoon
|
c544a8bb02 |
ARROW-15779: [Python] Create python bindings for Substrait consumer
The PR includes the initial integration of Substrait to Python - [x] Adding a util API for consuming Substrait - [x] Adding a C++ example for using Substrait with Util API - [x] Adding Python Bindings for Substrait using the Util API - [x] Adding CMake changes to integrate `engine` module (experimental) : comments and suggestions are much appreciated to improve this component - [x] Adding a Python example to consume a Substrait plan (experimental) Closes #12672 from vibhatha/arrow-15779 Authored-by: Vibhatha Abeykoon <vibhatha@gmail.com> Signed-off-by: Weston Pace <weston.pace@gmail.com> |
||
|
Raúl Cumplido
|
c16bbe18e3 |
ARROW-16219: [CI] Fix git config to prevent SCM tools failure
This PR fixes the CI failures due to the latest git release fixing CVE-2022-24765. I have been able to see the build passing the scm step with the change here: https://app.travis-ci.com/github/raulcd/arrow/builds/249688925 The above build fails due to some tests failing but not related with installation anymore. And this was the failure before the change: https://app.travis-ci.com/github/raulcd/arrow/builds/249683847 I also have been able to reproduce the issue on the `verify-conda-rc` locally. The failure: ``` $ docker-compose run conda-verify-rc .... Preparing transaction: done Verifying transaction: done Executing transaction: done /arrow/python /arrow / Traceback (most recent call last): File "/arrow/python/setup.py", line 607, in <module> setup( File "/tmp/arrow-HEAD.YUVPq/mambaforge/envs/conda-source/lib/python3.10/site-packages/setuptools/__init__.py", line 87, in setup return distutils.core.setup(**attrs) File "/tmp/arrow-HEAD.YUVPq/mambaforge/envs/conda-source/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 109, in setup _setup_distribution = dist = klass(attrs) File "/tmp/arrow-HEAD.YUVPq/mambaforge/envs/conda-source/lib/python3.10/site-packages/setuptools/dist.py", line 462, in __init__ _Distribution.__init__( File "/tmp/arrow-HEAD.YUVPq/mambaforge/envs/conda-source/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 293, in __init__ self.finalize_options() File "/tmp/arrow-HEAD.YUVPq/mambaforge/envs/conda-source/lib/python3.10/site-packages/setuptools/dist.py", line 886, in finalize_options ep(self) File "/tmp/arrow-HEAD.YUVPq/mambaforge/envs/conda-source/lib/python3.10/site-packages/setuptools/dist.py", line 907, in _finalize_setup_keywords ep.load()(self, ep.name, value) File "/tmp/arrow-HEAD.YUVPq/mambaforge/envs/conda-source/lib/python3.10/site-packages/setuptools_scm/integration.py", line 75, in version_keyword _assign_version(dist, config) File "/tmp/arrow-HEAD.YUVPq/mambaforge/envs/conda-source/lib/python3.10/site-packages/setuptools_scm/integration.py", line 51, in _assign_version _version_missing(config) File "/tmp/arrow-HEAD.YUVPq/mambaforge/envs/conda-source/lib/python3.10/site-packages/setuptools_scm/__init__.py", line 106, in _version_missing raise LookupError( LookupError: setuptools-scm was unable to detect version for /arrow. Make sure you're either building from a fully intact git repository or PyPI tarballs. Most other sources (such as GitHub's tarballs, a git checkout without the .git folder) don't contain the necessary metadata and will not work. For example, if you're using pip, instead of https://github.com/user/proj/archive/master.zip use git+https://github.com/user/proj.git#egg=proj Failed to verify release candidate. See /tmp/arrow-HEAD.YUVPq for details. ``` Waiting to validate the verify-rc fix at the moment, will update once the local build finishes Closes #12945 from raulcd/ARROW-16219 Authored-by: Raúl Cumplido <raulcumplido@gmail.com> Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com> |
||
|
Antoine Pitrou
|
a6e51a039f |
MINOR: [CI][Python] Fix default values for optional component testing
Fix the CI failures on Crossbow jobs "test-debian-11-python-3" and "test-fedora-33-python-3". See e.g. https://dev.azure.com/ursacomputing/crossbow/_build/results?buildId=20528&view=logs&j=0da5d1d9-276d-5173-c4c4-9d4d4ed14fdb&t=d9b15392-e4ce-5e4c-0c8c-b69645229181 Closes #12549 from pitrou/minor-python Authored-by: Antoine Pitrou <antoine@python.org> Signed-off-by: Antoine Pitrou <antoine@python.org> |
||
|
|
4e1ba57254 |
ARROW-9947: [Python] High-level Python API for Parquet encryption of files.
Exposes in PyArrow the high-level C++ API for Parquet encryption that was added in https://github.com/apache/arrow/pull/8023. Design document: https://docs.google.com/document/d/1i1M5f5azLEmASj9XQZ_aQLl5Fr5F0CvnyPPVu1xaD9U A test is added for writing and reading encrypted parquet files using a simple in-memory KMS for testing, that is not to be used as an example of a KMS client. In addition, there is an example KMS client using Vault KMS. This PR handles only the file-level encryption and decryption. Dataset is handled in separate PRs. The investigation of the multithreading model of PME is separated into a separate issue, independent of this one. Closes #10450 from andersonm-ibm/encryption Lead-authored-by: Maya Anderson <mayaa@il.ibm.com> Co-authored-by: andersonm-ibm <63074550+andersonm-ibm@users.noreply.github.com> Co-authored-by: roee88 <roee88@gmail.com> Co-authored-by: Antoine Pitrou <antoine@python.org> Signed-off-by: Antoine Pitrou <antoine@python.org> |
||
|
Antoine Pitrou
|
5c1b5c1f6b |
ARROW-15243: [CI][Python] Make PyArrow installation more robust in CI
On Debian/Ubuntu, the system-provided setuptools is patched to install into the system-specific `dist-packages` directory. However, the newer setuptools from upstream would install PyArrow in the wrong location (`site-packages`). Therefore, use "pip install" instead of "python setup.py build" to get the right behaviour. Closes #12069 from pitrou/ARROW-15243-py-install Authored-by: Antoine Pitrou <antoine@python.org> Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com> |
||
|
Alessandro MolinaandKrisztián Szűcs
|
511d6d5a26 |
ARROW-14937: [Doc] Make sure the docs directory is mounted as a volume
Closes #11822 from amol-/ARROW-14937 Lead-authored-by: Alessandro Molina <amol@turbogears.org> Co-authored-by: Krisztián Szűcs <szucs.krisztian@gmail.com> Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com> |
||
|
Krisztián SzűcsandSutou Kouhei
|
be9a22b9b7 |
ARROW-14505: [CI][Docs] Exercise documentation builds more frequently
Closes #11567 from kszucs/build-docs-on-main-branch Lead-authored-by: Krisztián Szűcs <szucs.krisztian@gmail.com> Co-authored-by: Sutou Kouhei <kou@cozmixng.org> Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com> |
||
|
Zhuo PengandAntoine Pitrou
|
d0126e713c |
ARROW-7362: [Python][C++] Added ListArray.Flatten() that properly flattens a ListArray
Currently ListArray.flatten() simply returns the child array. If a ListArray is a slice of another ListArray, they will share the same child array, however the expected behavior (I think) of flatten() should be returning an Array that's a concatenation of all the sub-lists in the ListArray, so the slicing offset should be taken into account. For example: ```python a = pa.array([[1], [2], [3]]) assert a.flatten().equals(pa.array([1,2,3])) # expected: a.slice(1).flatten().equals(pa.array([2, 3])) ``` Closes #6006 from brills/flatten and squashes the following commits: |
||
|
Krisztián Szűcs
|
9cb49f3d34 |
ARROW-6341: [Python] Implement low-level bindings for Dataset
Closes #5237 from kszucs/ARROW-6341 and squashes the following commits: |
||
|
Krisztián Szűcs
|
54cb9c0d32 |
ARROW-7310: [Python] Expose HDFS implementation for pyarrow.fs
Closes #5990 from kszucs/ARROW-7310 and squashes the following commits: |
||
|
Krisztián Szűcs
|
7102d7eeef |
ARROW-7346: [CI] Explicit usage of ccache across the builds
- ccache configuration is centralised in the docker-compose.yml - stricter build directory paths - java jni tests were not running, since the vendoring of flatbuffers there was a hidden linking error Closes #5982 from kszucs/explicit_use_ccache and squashes the following commits: |
||
|
Sutou Kouhei
|
40c000f695 |
ARROW-7334: [CI][Python] Use Python 3 on macOS
Currently, Python 2 is used. Closes #5979 from kou/ci-python-macos-use-python3 and squashes the following commits: |
||
|
Antoine PitrouandKrisztián Szűcs
|
7adcb7eafd |
ARROW-6954: [Python] [CI] Add Python 3.8 to CI matrix
Also use a separate conda environment rather than installing everything in the base conda environment, as conda can complain about conflicts. Closes #5877 from pitrou/ARROW-6954-py38-ci and squashes the following commits: |
||
|
Krisztián Szűcs
|
7bc2b0f357 |
ARROW-7101: [CI] Refactor docker-compose setup and use it with GitHub Actions
## Projecting ideas from ursabot
### Parametric docker images
The images are better parameterized now, meaning that we can build more variant of the same service. Couple of examples:
```console
UBUNTU=16.04 docker-compose build ubuntu-cpp
ARCH=arm64v8 UBUNTU=18.04 docker-compose build ubuntu-cpp
PYTHON=3.6 docker-compose build conda-python
ARCH=arm32v7 PYTHON=3.6 PANDAS=0.25 docker-compose build conda-python-pandas
```
Each variant has it's own docker image following a string naming schema:
`{org}/{arch}-{platform}-{platform-version}[[-{variant}-{variant-version}]..]:latest`
### Use *_build.sh and *_test.sh for each job
The docker images provide the environment, and each language backend usually should implement two scripts, a `build.sh` and a `test.sh`. This way dependent build like the docker python, r or c glib are able to reuse the build script of the ancestor without running its tests.
With small enough scripts, if the environment is properly set up even the non-docker builds should be reproducible locally. GitHub Actions support bash scripts across all three platforms, so we can reuse the same `*_build.sh` and `*_test.sh` scripts to execute the builds either in docker, on the CI or locally.
## Using GitHub Actions for running the builds
Regardless of the CI we're going to choose, the isolation constraint of different platforms requires some sort of virtualisation. Currently linux (and windows, but I have not tried it yet) has lightweight containerisation, so we should keep the linux builds isolated in docker containers. The rest of the platforms (windows and macOS) should be executed on the CI system.
GitHub Actions support all three major platforms, linux, windows and macOS. I've added cross platform builds for a couple of languages, like Rust, and Go, the rest are work in progress.
### Workflow
A workflow should define all builds of a language, mostly because the path filters can be defined on workflow level. For example the python builds should be triggered if either a cpp/** or a python/** file changes which can be covered in the same workflow file.
## Feature parity with the current builds
Reaching feature parity with all of the builds below is not a goal for this PR, the difficult ones should at least have a tracking JIRA ticket.
### Travis-CI
- [x] **Lint, Release tests**:
- `Lint / C++, Python, R, Rust, Docker, RAT`
- `Dev / Source Release`
- [x] **C++ unit tests w/ conda-forge toolchain, coverage**: without coverage
- `C++ / AMD64 Conda C++`
- [x] **Python 3.6 unit tests, conda-forge toolchain, coverage**: without coverage
- `Python / AMD64 Conda Python 3.6`
- [x] **[OS X] C++ w/ Xcode 9.3**:
- `C++ / AMD64 MacOS 10.14 C++`: with Xcode 10.3
- [x] **[OS X] Python w/ Xcode 9.3**:
- `Python / AMD64 MacOS 10.14 Python 3`: with Xcode 10.3
- [x] **Java OpenJDK8 and OpenJDK11**:
- `Java / AMD64 Debian Java JDK 8 Maven 3.5.2`
- `Java / AMD64 Debian Java JDK 11 Maven 3.6.2`
- [x] **Protocol / Flight Integration Tests**:
- `Dev / Protocol Test`
- [x] **NodeJS**: without running lint and coverage
- `NodeJS / AMD64 Debian NodeJS 11`
- [x] **C++ & GLib & Ruby w/ gcc 5.4**:
- `C++ / AMD64 Debian 10 C++`: with GCC 8.3
- `C++ / AMD64 Ubuntu 16.04 C++`: with GCC 5.4
- `C++ / AMD64 Ubuntu 18.04 C++`: with GCC 7.4
- `C GLib / AMD64 Ubuntu 18.04 C GLib`
- `Ruby / AMD64 Ubuntu 18.04 Ruby`
- [x] **[OS X] C++ & GLib & Ruby w/ XCode 10.2 & Homebrew**
- `C++ / AMD64 MacOS 10.14 C++`: with Xcode 10.3
- `C GLib / AMD64 MacOS 10.14 C Glib`: with Xcode 10.3
- `Ruby / AMD64 MacOS 10.14 Ruby`: with Xcode 10.3
- [x] **Go**: without coverage
- `Go / AMD64 Debian Go 1.12`
- [x] **R (with and without libarrow)**:
- `R / AMD64 Conda R 3.6`: with libarrow
- `R / AMD64 Ubuntu 18.04 R 3.6` with libarrow
### Appveyor
- ~JOB=Build, GENERATOR=Ninja, CONFIGURATION=Release, APPVEYOR_BUILD_WORKER_IMAGE=Visual Studio 2017~
- ~JOB=Toolchain, GENERATOR=Ninja, CONFIGURATION=Release, ARROW_S3=ON, ARROW_BUILD_FLIGHT=ON, ARROW_BUILD_GANDIVA=ON~
- ~JOB=Build_Debug, GENERATOR=Ninja, CONFIGURATION=Debug~
- ~JOB=MinGW32, MINGW_ARCH=i686, MINGW_PACKAGE_PREFIX=mingw-w64-i686, MINGW_PREFIX=c:\msys64\mingw32, MSYSTEM=MINGW32, USE_CLCACHE=false~
- ~JOB=MinGW64, MINGW_ARCH=x86_64, MINGW_PACKAGE_PREFIX=mingw-w64-x86_64, MINGW_PREFIX=c:\msys64\mingw64, MSYSTEM=MINGW64, USE_CLCACHE=false~
- [x] **JOB=Rust, TARGET=x86_64-pc-windows-msvc, USE_CLCACHE=false**:
- `Rust / AMD64 Windows 2019 Rust nightly-2019-09-25`
- [x] **JOB=C#, APPVEYOR_BUILD_WORKER_IMAGE=Visual Studio 2017, USE_CLCACHE=false**
- `C# / AMD64 Windows 2019 C# 2.2.103`
- [x] **JOB=Go, MINGW_PACKAGE_PREFIX=mingw-w64-x86_64 ...**:
- `Go / AMD64 Windows 2019 Go 1.12`
- ~JOB=R with libarrow, USE_CLCACHE=false, TEST_R_WITH_ARROW=TRUE, RWINLIB_LOCAL=%APPVEYOR_BUILD_FOLDER%\libarrow.zip~
### Github Actions
- [x] **Windows MSVC C++ / Build (Visual Studio 16 2019)**:
- `C++ / AMD64 Windows 2019 C++`: without tests
- [x] **Windows MSVC C++ / Build (Visual Studio 15 2017)**:
- `C++ / AMD64 Windows 2016 C++`: without tests
- [x] **Linux docker-compose / Test (C++ w/ clang-7 & system packages)**: all have llvm for gandiva but the compiler is set to gcc
- `C++ / AMD64 Debian 10 C++`: with GCC 8.3
- `C++ / AMD64 Ubuntu 16.04 C++`: with GCC 5.4
- `C++ / AMD64 Ubuntu 18.04 C++`: with GCC 7.4
- [x] **Linux docker-compose / Test (Rust)**: without rustfmt
- `Rust / AMD64 Debian Rust nightly-2019-09-25`
- [x] **Linux docker-compose / Test (Lint, Release tests)**:
- `Lint / C++, Python, R, Rust, Docker, RAT`
- `Dev / Source Release`
### Nightly Crossbow tests
The packaging builds are out of the scope if this PR, but the nightly **dockerized test** task are in.
Nightly tests:
- [x] docker-r
- [x] docker-r-conda
- [x] docker-r-sanitizer
- [x] docker-rust
- [x] docker-cpp
- [x] docker-cpp-cmake32
- [x] docker-cpp-release
- [x] docker-cpp-static-only
- [x] docker-c_glib
- [x] docker-go
- [x] docker-python-2.7
- [x] docker-python-3.6
- [x] docker-python-3.7
- [x] docker-python-2.7-nopandas
- [x] docker-python-3.6-nopandas
- [x] docker-java
- [x] docker-js
- [x] docker-docs
- [x] docker-lint
- [x] docker-iwyu: included in the lint
- [x] docker-clang-format: included in the lint
- [x] docker-pandas-master
- [x] docker-dask-integration
- [x] docker-hdfs-integration
- [x] docker-spark-integration
- [x] docker-turbodbc-integration
# TODOs left:
- [x] Fix the Apidoc generation for c_glib
- [x] Fix the JNI test for Gandiva and ORC
- [x] Test that crossbow tests are passing
- ~Optionally restore the travis configuration to incrementally decommission old builds~
## Follow-up JIRAs:
- [Archery] Consider porting the docker tool of ursabot to archery
- [Archery] Consider to use archery with or instead of the pre-commit hooks
- [Archery] Create a wrapper script in archery for docker compose in order to run the containers with the host's user and group
- [C++] GCC 5.4.0 has a compile errors, reproduce with UBUNTU=16.04 docker-compose run ubuntu-cpp
- [C++][CI] Test the ported fuzzit integration image
- [C++][CI] Turn off unnecessary features in the integration tests (spark/turbodbc/dask/hdfs)
- [C++][CI] Revisit ASAN UBSAN settings in every C++ based image
- [CI] Consider re-adding the removed debian testing image is removed
- [Go][CI] Pre-install the go dependencies in the dockerfile using go get
- [JS][CI] Pre-install the JS dependencies in the dockerfile
- [Rust][CI] Pre-install the rust dependencies in the dockerfile
- [Java][CI] Pre-install the java dependencies in the dockerfile
- [Ruby][CI] Pre-install the ruby dependencies in the dockerfile and remove it from the test script
- [C#][CI] Pre-install the C# dependencies in the dockerfile
- [R][CI] Fix the r-sanitizer build https://issues.apache.org/jira/browse/ARROW-6957
- [GLIB][MacOS] Fail to execute lua examples (fails to load 'lgi.corelgilua51' despite that lgi is installed)
- [C++][CMake] Automatically set ARROW_GANDIVA_PC_CXX_FLAGS for conda and OSX sdk (see cpp_build.sh)
- [C++][CI] Hiveserver2 instegration test fails to connect to impala container
- [CI][Spark] Support specific Spark version in the integration tet including latest
- [JS][CI] Move nodejs linting from js_build.sh to archery
- [Python][CI] create a docker image for python ASV benchmarks and fix the script
- [CI] Find a short but related prefix for the env vars used for the docker-compose file to prevent collisions
- [C#] the docker container fails to run because of the ubuntu host versions, see https://github.com/dotnet/core/issues/3509
- [C++][Windows] Enable more features on the windows GHA build
- [Doc] document docker-compose usage in the developer sphinx guide
- [CI][C++] Add .ccache to the docker-compose mounts
- [Archery][CI] Refactor the ci/scripts to a sourceable bash functions or to archery directly
- [C++][CI] Use scripts/util_coredump.sh to show automatic backtraces
- [C++] Fix the hanging C++ tests in Windows 2019
- [CI] Ask INFRA to set up the DOCKERHUB_* secrets for GitHub actions
- [C++][CI] Running Gandiva tests fails on Fedora:
Reproduce with: `docker-compose run -e ARROW_GANDIVA=ON fedora-cpp`
```
Running gandiva-internals-test, redirecting output into /build/cpp/build/test-logs/gandiva-internals-test.txt (attempt 1/1)
1364
: CommandLine Error: Option 'x86-experimental-vector-widening-legalization' registered more than once!
1365
LLVM ERROR: inconsistency in registered CommandLine options
1366
/build/cpp/src/gandiva
```
- [JS][CI] NodeJS build fails on Github Actions Windows node
```
> NODE_NO_WARNINGS=1 gulp build
# 'NODE_NO_WARNINGS' is not recognized as an internal or external command,
# operable program or batch file.
# npm ERR! code ELIFECYCLE
# npm ERR! errno 1
# npm ERR! apache-arrow@1.0.0-SNAPSHOT build: `NODE_NO_WARNINGS=1 gulp build`
# npm ERR! Exit status 1
# npm ERR!
# npm ERR! Failed at the apache-arrow@1.0.0-SNAPSHOT build script.
# npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
```
Closes #5589 from kszucs/docker-refactor and squashes the following commits:
|