### Rationale for this change
Python 3.10 is End of Support on October 2026. We can drop support on PyArrow 26.0.0.
See: https://endoflife.date/python
### What changes are included in this PR?
Remove support for 3.10. Update minimum required Python version and update some CI jobs to use the minimum Python supported version.
Some CI jobs using default Python 3.10 (Ubuntu 20.04) have been bumped.
### Are these changes tested?
They will be tested via CI and Archery
### Are there any user-facing changes?
Yes in terms of not supporting older versions of Python. This will be a breaking change for some users but not for the API itself
* GitHub Issue: #50410
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
The CI job is currently failing
### What changes are included in this PR?
Do not set HOME to empty on `build_conda.sh` otherwise ccache fails creating cache folders. Just use default HOME and use WORKDIR
### Are these changes tested?
Yes via the failing job on archery.
### Are there any user-facing changes?
No
* GitHub Issue: #49518
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Move our PyArrow build backend from setuptools and a custom setup.py to scikit-build-core which is just build backend for CMake related projects.
### What changes are included in this PR?
Move from setuptools to scikit-build-core and remove PyArrow setup.py. Update some of the build requirements and minor fixes.
A custom build backend has been also been created in order to wrap scikit-build-core in order to fix problems on License files for monorepos.
pyproject.toml metadata validation expects license files to exist before exercising the build backend that's why we create symlinks. Our thin build backend will just make those symlinks hard-links in order for license and notice files to contain the contents and be added as part of the sdist.
Remove flags that are not used anymore (were only part of setup.py) and documented and validated how the same flags have to be used now.
### Are these changes tested?
Yes all Python CI tests, wheels and sdist are successful.
### Are there any user-facing changes?
Yes, users building PyArrow will now require the new build dependencies to exercise the build and depending on the flags used they might require to use the new documented way of using those flags.
* GitHub Issue: #36411
Lead-authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Co-authored-by: Rok Mihevc <rok@mihevc.org>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Currently we are not building wheels with OpenTelemetry.
### What changes are included in this PR?
- Adding a new feature on our `vcpkg.json` to install required opentelemetry-cpp
- Enabling the opentelemetry feature when downloading dependencies via vcpkg for our wheels.
- Setting `ARROW_WITH_OPENTELEMETRY=ON` on the wheel build scripts
- Minor fix on checking exposed symbols (opentelemetry symbols are exposed)
- Minor improvement to `python/examples/flight/middleware.py` to be able to easily enable opentelemetry middleware tracing for testing.
- Fix on `server_tracing_middleware.cc` to retrieve tracer from `arrow::internal::tracing::GetTracer();` instead of `otel::trace::Provider::GetTracerProvider()->GetTracer("arrow");`
- Expose missing symbols to enable Windows builds
- Fix Windows compatibility adding windows compat header.
- Add `is_opentelemetry_enabled` utility function in pyarrow to validate whether OpenTelemetry was enabled or not when building Arrow C++. Expose this information on `pa.show_info()`.
### Are these changes tested?
Yes, wheels are being successfully built and tested locally with a built wheel that traces are being generated when properly configured.
Added test to validate traces appear on stdout and I've also validated Acero traces are being correctly generated by running the test suite and seeing the opentelemetry traces on Jaeger UI.
### Are there any user-facing changes?
Yes, PyArrow wheel users will be able to enable OpenTelemetry traces.
* GitHub Issue: #49382
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Docker Compose recommends `compose.yaml`:
https://docs.docker.com/compose/intro/compose-application-model/#the-compose-file
> The default path for a Compose file is compose.yaml (preferred) or compose.yml that is placed in the working directory. Compose also supports docker-compose.yaml and docker-compose.yml for backwards compatibility of earlier versions. If both files exist, Compose prefers the canonical compose.yaml.
### What changes are included in this PR?
- Update several `docker-compose.yml` file names to `compose.yaml`
- Update doc and comment references
- Update archery references to the file
- Update workflows and CODEOWNERS to take new file into account.
### Are these changes tested?
Yes via CI and some archery jobs.
### Are there any user-facing changes?
No
* GitHub Issue: #47806
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
Fedora 39 reached EOL on 2024-11-26: https://docs.fedoraproject.org/en-US/releases/eol/
### What changes are included in this PR?
Use Fedora 42 that is the latest release.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
* GitHub Issue: #47045
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
The flag `-a` is not part of the mamba info API anymore and the nightly CI job is currently failing.
### What changes are included in this PR?
Remove the `-a` on mamba info, it still shows relevant information:
```
libmamba version : 2.0.8
mamba version : 2.0.8
curl version : libcurl/8.13.0 OpenSSL/3.4.1 zlib/1.3.1 zstd/1.5.7 libssh2/1.11.1 nghttp2/1.64.0
libarchive version : libarchive 3.7.7 zlib/1.3.1 liblzma/5.6.3 bz2lib/1.0.8 liblz4/1.10.0 libzstd/1.5.6
envs directories : /miniconda-for-arrow/envs
package cache : /miniconda-for-arrow/pkgs
/root/.mamba/pkgs
environment : base
env location : /miniconda-for-arrow
user config files : /root/.mambarc
populated config files : /miniconda-for-arrow/.condarc
virtual packages : __unix=0=0
__linux=6.11.0=0
__glibc=2.38=0
__archspec=1=x86_64_v4
channels : https://conda.anaconda.org/conda-forge/linux-64https://conda.anaconda.org/conda-forge/noarch
base environment : /miniconda-for-arrow
platform : linux-64
```
### Are these changes tested?
Yes via Archery
### Are there any user-facing changes?
No
* GitHub Issue: #46259
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
We want to upgrade our CMake version to 3.25 as discussed on the ML:
https://lists.apache.org/thread/h8jp16ktrj11fmjmjhlg6xvkvv9wzvjk
### What changes are included in this PR?
- Bump minimal CMake version to 3.25
- Manually install CMake on distributions where CMake < 3.25 was installed via package repositories
- Minor fixes to CI in order to have passing builds everywhere
### Are these changes tested?
Yes, via CI.
### Are there any user-facing changes?
Yes, the minimum CMake version to be used to build Arrow is bumped to 3.25.
**This PR includes breaking changes to build systems.**
* GitHub Issue: #44950
Lead-authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Co-authored-by: Bryce Mecum <petridish@gmail.com>
Co-authored-by: Sutou Kouhei <kou@clear-code.com>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Co-authored-by: Neal Richardson <neal.p.richardson@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
This is a follow-on to https://github.com/apache/arrow/pull/43566 which was merged earlier today. See https://github.com/apache/arrow/issues/43570 for more detail on the entire set of changes.
### What changes are included in this PR?
- Replace instances where we call or document calling `docker-compose` with `docker compose`.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
This PR includes the require user-facing changes to document things.
* GitHub Issue: #43570
Lead-authored-by: Bryce Mecum <petridish@gmail.com>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
The C++ classes `FlightInfo` and `FlightEndpoint` have attributes that are not available via the Python API.
### What changes are included in this PR?
Make the following attributes available in Python:
- `FlightInfo.ordered`
- `FlightInfo.app_metadata`
- `FlightEndpoint.expiration_time`
- `FlightEndpoint.app_metadata`
Also makes existing attributes optional in constructor:
- `FlightInfo.total_records`
- `FlightInfo.total_bytes`
### Are these changes tested?
Existing tests that test existing attributes are extended.
### Are there any user-facing changes?
Yes, changes are backward compatible.
* GitHub Issue: #36954
Lead-authored-by: Enrico Minack <github@enrico.minack.dev>
Co-authored-by: Adam Reeve <adreeve@gmail.com>
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Co-authored-by: David Li <li.davidm96@gmail.com>
Signed-off-by: David Li <li.davidm96@gmail.com>
### Rationale for this change
python/examples/minimal_build/Dockerfile.fedora:20 DL3040 warning: `dnf clean all` missing after dnf command.
python/examples/minimal_build/Dockerfile.fedora:20 DL3041 warning: Specify version with `dnf install -y <package>-<version>`.
python/examples/minimal_build/Dockerfile.fedora:32 DL3042 warning: Avoid use of cache directory with pip. Use `pip install --no-cache-dir <package>`
python/examples/minimal_build/Dockerfile.ubuntu:38 DL3042 warning: Avoid use of cache directory with pip. Use `pip install --no-cache-dir <package>`
### What changes are included in this PR?
* Ignore "DL3041 warning: Specify version with `dnf install -y <package>-<version>`." because we don't specify version.
* Suppress other warnings.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
* GitHub Issue: #40443
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
Fedora 39 is the latest release and Fedora 38 will reach EOL on 2024-05-14.
### What changes are included in this PR?
Use Fedora 39.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
* Closes: #40011
Lead-authored-by: Sutou Kouhei <kou@clear-code.com>
Co-authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
### What changes are included in this PR?
Spelling fixes to python/
### Are these changes tested?
### Are there any user-facing changes?
* Closes: #38944
Authored-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
### Rationale for this change
The purpose of this pull request is to support modular encryption in the new Dataset API. See [https://docs.google.com/document/d/13EysCNC6-Nu9wnJ8YpdzmD-aMLn4i2KXUJTNqIihy7A/edit#](url) for supporting document.
### What changes are included in this PR?
I made improvements to the C++ and Python code to enable the Dataset API to have per-file settings for each file saved. Previously, the Dataset API applied the same encryption properties to all saved files, but now I've updated the code to allow for greater flexibility. In the Python code, I've added support for the changes by updating the ParquetFormat class to accept DatasetEncryptionConfiguration and DatasetDecryptionConfiguration structures. With these changes, you can pass the format object to the write_dataset function, giving you the ability to set unique encryption properties for each file in your Dataset.
### Are these changes tested?
Yes, unit tests are included. I have also included a python sample project.
### Are there any user-facing changes?
Yes, as stated above the ParquetFormat class has optional parameters for DatasetEncryptionConfiguration and DatasetDecryptionConfiguration through setters and getters. The Dataset now has the option using this to set different file encryption properties per file
* Closes: #29238
Lead-authored-by: Don <tolleybot@gmail.com>
Co-authored-by: Donald Tolley <tolleybot@gmail.com>
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Co-authored-by: anjakefala <anja@voltrondata.com>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Co-authored-by: Weston Pace <weston.pace@gmail.com>
Co-authored-by: Gang Wu <ustcwg@gmail.com>
Co-authored-by: scoder <stefan_ml@behnel.de>
Co-authored-by: Will Jones <willjones127@gmail.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
### Rationale for this change
Datasets and Substrait are heavy subcomponents (as they have non-trivial dependencies), so should not be enabled unwillingly in all CI builds.
### What changes are included in this PR?
Disable Dataset and Substrait in `ci/scripts/cpp_build.sh`, enable them explicitly in relevant builds.
Disable more subcomponents in CI builds that don't need them, especially "minimal" builds.
### Are these changes tested?
Yes, by existing CI jobs.
### Are there any user-facing changes?
No.
* Closes: #37500
Lead-authored-by: Antoine Pitrou <antoine@python.org>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Co-authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
We can't maintain Plasma.
### What changes are included in this PR?
These changes remove all Plasma related codes.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* Closes: #33243
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
ARROW-9171 and ARROW-17231 are also fixed.
Our CMake packages are broken. For example, `find_package(Parquet)`
doesn't work without specifying `CMAKE_MODULE_PATH`.
`find_package(${PACKAGE})` searches
`${PREFIX}/${PACKAGE}/${PACKAGE}Config.cmake` or
`${PREFIX}/${PACKAGE}/Find${PACKAGE}.cmake`. But our .cmake files are
always installed `${PREFIX}/arrow/`. So `find_package(Parquet)` can't find
`${PREFIX}/arrow/FindParquet.cmake` because "`/arrow/`" isn't
"`/${PACKAGE}`".
This change fixes this by installing `${PACKAGE}Config.cmake` to
`${PREFIX}/${PACKAGE}/` instead of `${PREFIX}/arrow/`.
This also removes all `Find${PACKAGE}.cmake`. We only provides
`${PACKAGE}Config.cmake`. Our `Find${PACKAGE}.cmake` can find `${PACKAGE}`
by CMake, pkg-config or manual .so/.h search. But we don't need to
support pkg-config nor manual .so/.h search. We can use
`${PACKAGE}Config.cmake` to support CMake package search. So this
removes all `Find${PACKAGE}.cmake`.
This also introduces namespace to our CMake targets. For example,
`arrow_shared` is exported as `Arrow::arrow_shared` and
`parquet_static` is exported as `Parquet::parquet_static`. But no
namespace targets such as `arrow_shared` and `parquet_static` are
still also exported for keeping backward compatibility. But this
requires CMake 3.18 or later for users because `add_library(ALIAS)`
for non-global `IMPORTED` library is available since CMake 3.18.
(`Plasma::plasma-store-server` target is also added for
`plasma-store-server` executable.)
FYI: We can resolve this problem by using `COMPONENTS` feature of
`find_package()`. For example, `find_package(Arrow COMPONENTS
Parquet)` is used instead of `find_package(Parquet)`. With
`COMPONENTS`, `${PACKAGE}` is always "Arrow". So we can still install
our .cmake files to `${PREFIX}/arrow/`. But this approach breaks
backward compatibility. So I choose `${PREFIX}/${PACKAGE}/*.cmake`
approach.
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
This PR adds new jobs to the nightly tests in order to exercise the existing Python minimal build examples.
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
This PR makes `pyarrow.parquet` a package and moves `pyarrow.parquet_encryption` to `pyarrow.parquet.encryption`
Closes#12837 from raulcd/ARROW-15824
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Alessandro Molina <amol@turbogears.org>
Exposes in PyArrow the high-level C++ API for Parquet encryption that was added in https://github.com/apache/arrow/pull/8023.
Design document: https://docs.google.com/document/d/1i1M5f5azLEmASj9XQZ_aQLl5Fr5F0CvnyPPVu1xaD9U
A test is added for writing and reading encrypted parquet files using a simple in-memory KMS for testing, that is not to be used as an example of a KMS client. In addition, there is an example KMS client using Vault KMS.
This PR handles only the file-level encryption and decryption.
Dataset is handled in separate PRs.
The investigation of the multithreading model of PME is separated into a separate issue, independent of this one.
Closes#10450 from andersonm-ibm/encryption
Lead-authored-by: Maya Anderson <mayaa@il.ibm.com>
Co-authored-by: andersonm-ibm <63074550+andersonm-ibm@users.noreply.github.com>
Co-authored-by: roee88 <roee88@gmail.com>
Co-authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
This PR replaces most of the use of hardcoded paths
to things `#!/bin` and `#!/usr/bin`, so most of the tools
in `dev` and a bunch of the C++ development tools are usable
on systems that don't have bash at this location.
Closes#10953 from cpcloud/arrow-13648
Authored-by: Phillip Cloud <417981+cpcloud@users.noreply.github.com>
Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
According to PEP 632, distutils will be deprecated in Python 3.10 and removed in 3.12.
* switch to `setuptools` for general packaging
* use the `Version` class from the `packaging` project instead of `distutils.LooseVersion`
Closes#9849 from pitrou/ARROW-12068-remove-distutils
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
Our team would like to implement mutual tls authentication for a pyarrow client, hence modifications to the C++ and Python implementations to at a GrpcMTls scheme for connections.
Closes#7130 from erinleeryan/master
Lead-authored-by: Erin Ryan <erin.ryan@ccri.com>
Co-authored-by: David Li <li.davidm96@gmail.com>
Signed-off-by: David Li <li.davidm96@gmail.com>
* Fedora instructions changed to podman with disclaimer added
* venv script modified to use environment variables more allowing user to put all build related files to single directory (before some were in HOME, other in / )
Closes#6845 from JacekPliszka/ARROW-8237
Lead-authored-by: Jacek Pliszka <jacek.pliszka@gmail.com>
Co-authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
This provides Dockerfile setups to build both with Fedora 31 (very recent) and Ubuntu 18.04, both with conda and pip/virtualenv methods
I made the Python developer instructions a bit more minimal. It's unclear that the average developer needs to build Gandiva and Flight, for example. It's fairly clear how to enable these optional components if they are desired.
Closes#6842 from wesm/ARROW-8237
Authored-by: Wes McKinney <wesm+git@apache.org>
Signed-off-by: Wes McKinney <wesm+git@apache.org>
The `get` command wouldn't use the adequate TLS root certs when fetching a Flight from its endpoints.
Also fix style in the Python examples and configure `archery lint` to check them.
Closes#6808 from pitrou/ARROW-8304-py-flight-client-tls
Lead-authored-by: Antoine Pitrou <antoine@python.org>
Co-authored-by: Ravindra Wagh <ravindra.wagh@cambridgesemantics.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
Patch by @ravindra-wagh
(had to redo a new branch and GitHub PR because of git UI crappiness, sorry)
Closes#6479 from pitrou/ARROW-7928-update-py-flight-examples and squashes the following commits:
d54dd7d77 <Antoine Pitrou> Some nits
269590fba <ravindra-wagh> Updated client to upload a csv file to the server
322dace54 <ravindra-wagh> Update server.py
Lead-authored-by: ravindra-wagh <57516622+ravindra-wagh@users.noreply.github.com>
Co-authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: David Li <li.davidm96@gmail.com>
This is likely not fully compatible with Java yet (I'm aware of one edge case where gRPC-Java does something different than gRPC-C++) - will address in a follow up with integration testing ([ARROW-5875](https://issues.apache.org/jira/browse/ARROW-5875)).
Compared to the Java implementation, this one can make guarantees about which thread things are run on (so you can and must use thread-locals to communicate with application code from middleware).
This uses the experimental gRPC interceptor interface to implement client middleware.
Travis build: https://travis-ci.com/lihalite/arrow/builds/129755428
AppVeyor build: https://ci.appveyor.com/project/lihalite/arrow/builds/27777244Closes#5552 from lihalite/flight-middleware-cpp and squashes the following commits:
03921d455 <David Li> Fix warnings/tests
69505a8db <David Li> Update middleware example
564783797 <David Li> Require headers to be ASCII
e47d1709f <David Li> Don't rely on thread locals for server middleware
f6d314bd1 <David Li> Remove unnecessary files
d036c74d5 <David Li> Get rid of header iterator
7716ade3b <David Li> Address review feedback
99cf60f19 <David Li> Fix Flight tests on Windows
362341ef2 <David Li> Fix lints
65459ad97 <David Li> Add middleware to Flight docs
0be4009db <David Li> Fix Python TLS test
9e23952b9 <David Li> Fix Python middleware
a1d9b5da7 <David Li> Add example of Flight/Python middleware
260deb84b <David Li> Rework C++ client middleware to require infallability
530e6fcc8 <David Li> Rework C++ middleware to require infalliability
53646bc6c <David Li> ARROW-6074: Flight middleware
Authored-by: David Li <li.davidm96@gmail.com>
Signed-off-by: Wes McKinney <wesm+git@apache.org>
Should fix the sporadic crashes in test_flight.py.
Author: Antoine Pitrou <antoine@python.org>
Closes#4891 from pitrou/ARROW-5930-core-flight and squashes the following commits:
842f1013c <Antoine Pitrou> ARROW-5930: Make Flight server init phase explicit
Also covers [ARROW-4627](https://issues.apache.org/jira/browse/ARROW-4627).
This is quite an enormous change, if preferred, I can do my best to try and separate changes.
Author: David Li <li.davidm96@gmail.com>
Closes#4282 from lihalite/arrow-4626-application-metadata and squashes the following commits:
6f1cd8db7 <David Li> Rework interface for accessing server-sent metadata during DoPut
8fd99cd9b <David Li> Inline CompletableFuture in Flight acceptPut
4cebc543a <David Li> Mark flaky Flight test
c551d8527 <David Li> Fix new CheckStyle violations
85e21699c <David Li> Fix Flight integration tests using metadata
eff22393c <David Li> Use FlightStreamChunk in Flight/C++
72c2a3fa0 <David Li> Try to always close FlightStream after acceptPut
1718d9b42 <David Li> Make FlightStream cancellable from acceptPut
7ac44df79 <David Li> Make Netty version consistent with gRPC
1225b67af <David Li> Use ArrowBuf instead of byte for Flight metadata
ccfef2d1e <David Li> Disable Flight cancellation tests in CI
0484c333c <David Li> Pass Flight context to ListActions in Python
b0f71d967 <David Li> Replace ARROW_EXPORT with ARROW_FLIGHT_EXPORT
fdaa76e99 <David Li> Add client-side cancelation of DoGet operations
b4dbc445e <David Li> Enable non-nested dictionary batches in Flight integration tests
f7631a2fd <David Li> Add basic Arrow Flight docs
a8ac27fb3 <David Li> Implement application metadata in Flight
86f4789ab <David Li> Add application metadata field to FlightData message
Also improve the example a bit.
Author: Antoine Pitrou <antoine@python.org>
Closes#4376 from pitrou/ARROW-5398-fix-py-flight-tests and squashes the following commits:
a15313f60 <Antoine Pitrou> ARROW-5398: Fix Flight tests
I haven't changed the client/server construction interfaces to really take advantage of this. I would rather follow up with creating builders instead, as proposed, as that would be cleaner than special-casing the logic in the current constructors.
Author: David Li <li.davidm96@gmail.com>
Closes#4047 from lihalite/flight-uris and squashes the following commits:
870f6ebb8 <David Li> Add more builder options for Java Flight servers
5c127631b <David Li> Make Python Flight bindings more complete
675acc9bc <David Li> Introduce builder for C++ Flight servers
474419667 <David Li> Use builder for Flight server in Java
460442cea <David Li> Use URIs for Flight locations
Right now the only option is a timeout on an overall call, which will be useful to limit hangs when connecting to a Flight service that might have gone down. (gRPC is rather aggressive about retrying for a while before giving up.)
You could imagine having a health check action in DoAction, for instance, which fails if the call doesn't complete within a short time.
I would also like a stream timeout, where the timeout only fires if there is a long delay between messages on a DoGet or DoAction, but this is hard to implement as-is in gRPC. Implementing an async client interface might be a better way to achieve this.
Author: David Li <li.davidm96@gmail.com>
Closes#4144 from lihalite/arrow-5136-call-options and squashes the following commits:
f20181c90 <David Li> Use varargs for Java Flight CallOptions
a573d2c61 <David Li> Make FlightCallOptions a POD type
d4bef2781 <David Li> Add tests for Flight call options in Python
0e6111154 <David Li> Don't hold GIL during Flight server shutdown
a1829b32c <David Li> implement call options for Flight
81e3c7908 <David Li> add call options to Flight
Also sets up the server-side APIs for [ARROW-5136](https://issues.apache.org/jira/browse/ARROW-5136).
I mostly mirrored the existing Java APIs in each language, except I didn't add the BasicAuth specialization.
Author: David Li <li.davidm96@gmail.com>
Author: Antoine Pitrou <antoine@python.org>
Closes#4125 from lihalite/arrow-5137-auth and squashes the following commits:
067ff3217 <Antoine Pitrou> Avoid making I/O methods const
48d42485f <David Li> Implement auth API for Flight
14774082f <David Li> Implement auth API for Flight
02a9663ae <David Li> Implement auth API for Flight
Not a high priority for review - fine to land for 0.14
Author: David Li <David.M.Li@twosigma.com>
Closes#3958 from lihalite/no-redundant-parameter and squashes the following commits:
52ec0b4f5 <David Li> Remove redundant schema parameter to Flight client DoGet
Author: Antoine Pitrou <antoine@python.org>
Closes#3770 from pitrou/ARROW-4687-python-flight-sigint and squashes the following commits:
5e548839f <Antoine Pitrou> Use std::atomic<>
8a28061be <Antoine Pitrou> ARROW-4687: Stop Flight server on incoming signals
Author: David Li <David.M.Li@twosigma.com>
Closes#3712 from lihalite/flight-python-server and squashes the following commits:
10dccb674 <David Li> Use 'except *' to handle exceptions in Cython
445eb87cd <David Li> Copy Flight out parameter values instead of moving in Python
9839e0f14 <David Li> Lint Python Flight code
f33b51f74 <David Li> Guard against Flight server methods not filling out parameter
422b4bcd5 <David Li> Bind Flight action endpoints to Python
c09b138de <David Li> Bind Flight ListFlights server call to Python
4a5cbaf52 <David Li> Guard against Flight server returning null listing
033734df6 <David Li> Bind Flight DoGet server call to Python
35c62f694 <David Li> Treat null FlightDataStream as error
- [ ] Add docs
- [ ] Format code
- [ ] Include Python in integration tests (requires binding the JSON reader/writer from C++)
- [ ] Validate performance?
- [ ] Complete server bindings if approach makes sense
Author: David Li <David.M.Li@twosigma.com>
Author: Wes McKinney <wesm+git@apache.org>
Closes#3566 from lihalite/flight-python and squashes the following commits:
ac29ab88 <David Li> Clean up to-be-implemented parts of Flight Python bindings
9d5442a0 <David Li> Clarify various RecordBatchStream{Reader,Writer} wrappers
e1c298ad <David Li> Lint CMake files
77644447 <Wes McKinney> Reformat cmake
c6b02aa9 <David Li> Add basic Python bindings for Flight
https://github.com/apache/arrow/issues/3154
This removes unused plasma interfaces Fetch(), Wait(), Transfer() and Info(), which depend on plasma manager which has already been removed from ray.
Author: Philipp Moritz <pcmoritz@gmail.com>
Author: Zhijun Fu <pingfu.fzj@antfin.com>
Author: Robert Nishihara <robertnishihara@gmail.com>
Closes#3167 from zhijunfu/remove-legacy-interfaces and squashes the following commits:
0efb5005f <Philipp Moritz> fix tensorflow op
be92e9085 <Philipp Moritz> fix java client
9da2cd38b <Philipp Moritz> Update _plasma.pyx
16ec63e9a <Robert Nishihara> More updates
e7413f739 <Robert Nishihara> Update _plasma.pyx
21398b5e7 <Zhijun Fu> merge
bcb320400 <Zhijun Fu> address comments
7967aea09 <Philipp Moritz> Merge branch 'master' into remove-legacy-interfaces
583cd97c4 <Zhijun Fu> ARROW-4015: remove unused interfaces for plasma manager
Author: Wes McKinney <wesm+git@apache.org>
Closes#2137 from wesm/ARROW-2395 and squashes the following commits:
d9e6e9f3 <Wes McKinney> Fix Cython flake8 warnings in examples
059a6d6b <Wes McKinney> Make functions in benchmarks/common.py py2-compatible
13791863 <Wes McKinney> Fix flake8 warnings outside of pyarrow/ directory. Check in CI
We may want to refactor to support random writes in `NativeFile`, so I've renamed accordingly
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closes#1014 from wesm/ARROW-837 and squashes the following commits:
0fe22b1 [Wes McKinney] Restore readable assertion check
0d5ebe4 [Wes McKinney] Add public pyarrow.allocate_buffer API. Rename FixedSizeBufferOutputStream to FixedSizeBufferWriter for when we can support seeking at some point
Note that this PR breaks the PlasmaClient API (which is still unstable at this point, so this is acceptable). It renames PlasmaClient.get to PlasmaClient.get_buffers and introduces two new functions, PlasmaClient.put and PlasmaClient.get which can put Python objects into the object store and provide access to their content. The old get was renamed to get_buffers because most users will want to use the new get method and therefore it should have the more concise name.
There is some freedom in designing the API; I tried to make it so there is a unified API between getting one and multiple objects (the latter is supported to limit the number of IPC roundtrips with the plasma store when we get many small objects). I also introduced a special object that is returned if one of the objects was not available within the timeout. We could use "None" here, but then it would be hard to distinguish between getting a "None" object and a timeout.
Author: Philipp Moritz <pcmoritz@gmail.com>
Closes#995 from pcmoritz/plasma-putget and squashes the following commits:
bd24e014 [Philipp Moritz] add documentation
e60ea73e [Philipp Moritz] get_buffer -> get_buffers and update example
8c369037 [Philipp Moritz] support full API
5921148e [Philipp Moritz] move put and get into PlasmaClient
cf4bf24f [Philipp Moritz] add type information
0049c674 [Philipp Moritz] fix flake8 linting
44c3b3dc [Philipp Moritz] fixes
20b119e3 [Philipp Moritz] make it possible to get single objects
36f67d6d [Philipp Moritz] implement ObjectID.from_random
c0449543 [Philipp Moritz] add documentation
eb9694a0 [Philipp Moritz] implement timeouts
3518c71a [Philipp Moritz] fix
e1924a45 [Philipp Moritz] add put and get
44ada470 [Philipp Moritz] export symbols
Raúl Cumplido