### Rationale for this change
Currently we are not building wheels with OpenTelemetry.
### What changes are included in this PR?
- Adding a new feature on our `vcpkg.json` to install required opentelemetry-cpp
- Enabling the opentelemetry feature when downloading dependencies via vcpkg for our wheels.
- Setting `ARROW_WITH_OPENTELEMETRY=ON` on the wheel build scripts
- Minor fix on checking exposed symbols (opentelemetry symbols are exposed)
- Minor improvement to `python/examples/flight/middleware.py` to be able to easily enable opentelemetry middleware tracing for testing.
- Fix on `server_tracing_middleware.cc` to retrieve tracer from `arrow::internal::tracing::GetTracer();` instead of `otel::trace::Provider::GetTracerProvider()->GetTracer("arrow");`
- Expose missing symbols to enable Windows builds
- Fix Windows compatibility adding windows compat header.
- Add `is_opentelemetry_enabled` utility function in pyarrow to validate whether OpenTelemetry was enabled or not when building Arrow C++. Expose this information on `pa.show_info()`.
### Are these changes tested?
Yes, wheels are being successfully built and tested locally with a built wheel that traces are being generated when properly configured.
Added test to validate traces appear on stdout and I've also validated Acero traces are being correctly generated by running the test suite and seeing the opentelemetry traces on Jaeger UI.
### Are there any user-facing changes?
Yes, PyArrow wheel users will be able to enable OpenTelemetry traces.
* GitHub Issue: #49382
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
The C++ classes `FlightInfo` and `FlightEndpoint` have attributes that are not available via the Python API.
### What changes are included in this PR?
Make the following attributes available in Python:
- `FlightInfo.ordered`
- `FlightInfo.app_metadata`
- `FlightEndpoint.expiration_time`
- `FlightEndpoint.app_metadata`
Also makes existing attributes optional in constructor:
- `FlightInfo.total_records`
- `FlightInfo.total_bytes`
### Are these changes tested?
Existing tests that test existing attributes are extended.
### Are there any user-facing changes?
Yes, changes are backward compatible.
* GitHub Issue: #36954
Lead-authored-by: Enrico Minack <github@enrico.minack.dev>
Co-authored-by: Adam Reeve <adreeve@gmail.com>
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Co-authored-by: David Li <li.davidm96@gmail.com>
Signed-off-by: David Li <li.davidm96@gmail.com>
Our team would like to implement mutual tls authentication for a pyarrow client, hence modifications to the C++ and Python implementations to at a GrpcMTls scheme for connections.
Closes#7130 from erinleeryan/master
Lead-authored-by: Erin Ryan <erin.ryan@ccri.com>
Co-authored-by: David Li <li.davidm96@gmail.com>
Signed-off-by: David Li <li.davidm96@gmail.com>
The `get` command wouldn't use the adequate TLS root certs when fetching a Flight from its endpoints.
Also fix style in the Python examples and configure `archery lint` to check them.
Closes#6808 from pitrou/ARROW-8304-py-flight-client-tls
Lead-authored-by: Antoine Pitrou <antoine@python.org>
Co-authored-by: Ravindra Wagh <ravindra.wagh@cambridgesemantics.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
Patch by @ravindra-wagh
(had to redo a new branch and GitHub PR because of git UI crappiness, sorry)
Closes#6479 from pitrou/ARROW-7928-update-py-flight-examples and squashes the following commits:
d54dd7d77 <Antoine Pitrou> Some nits
269590fba <ravindra-wagh> Updated client to upload a csv file to the server
322dace54 <ravindra-wagh> Update server.py
Lead-authored-by: ravindra-wagh <57516622+ravindra-wagh@users.noreply.github.com>
Co-authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: David Li <li.davidm96@gmail.com>
This is likely not fully compatible with Java yet (I'm aware of one edge case where gRPC-Java does something different than gRPC-C++) - will address in a follow up with integration testing ([ARROW-5875](https://issues.apache.org/jira/browse/ARROW-5875)).
Compared to the Java implementation, this one can make guarantees about which thread things are run on (so you can and must use thread-locals to communicate with application code from middleware).
This uses the experimental gRPC interceptor interface to implement client middleware.
Travis build: https://travis-ci.com/lihalite/arrow/builds/129755428
AppVeyor build: https://ci.appveyor.com/project/lihalite/arrow/builds/27777244Closes#5552 from lihalite/flight-middleware-cpp and squashes the following commits:
03921d455 <David Li> Fix warnings/tests
69505a8db <David Li> Update middleware example
564783797 <David Li> Require headers to be ASCII
e47d1709f <David Li> Don't rely on thread locals for server middleware
f6d314bd1 <David Li> Remove unnecessary files
d036c74d5 <David Li> Get rid of header iterator
7716ade3b <David Li> Address review feedback
99cf60f19 <David Li> Fix Flight tests on Windows
362341ef2 <David Li> Fix lints
65459ad97 <David Li> Add middleware to Flight docs
0be4009db <David Li> Fix Python TLS test
9e23952b9 <David Li> Fix Python middleware
a1d9b5da7 <David Li> Add example of Flight/Python middleware
260deb84b <David Li> Rework C++ client middleware to require infallability
530e6fcc8 <David Li> Rework C++ middleware to require infalliability
53646bc6c <David Li> ARROW-6074: Flight middleware
Authored-by: David Li <li.davidm96@gmail.com>
Signed-off-by: Wes McKinney <wesm+git@apache.org>
Should fix the sporadic crashes in test_flight.py.
Author: Antoine Pitrou <antoine@python.org>
Closes#4891 from pitrou/ARROW-5930-core-flight and squashes the following commits:
842f1013c <Antoine Pitrou> ARROW-5930: Make Flight server init phase explicit
Also covers [ARROW-4627](https://issues.apache.org/jira/browse/ARROW-4627).
This is quite an enormous change, if preferred, I can do my best to try and separate changes.
Author: David Li <li.davidm96@gmail.com>
Closes#4282 from lihalite/arrow-4626-application-metadata and squashes the following commits:
6f1cd8db7 <David Li> Rework interface for accessing server-sent metadata during DoPut
8fd99cd9b <David Li> Inline CompletableFuture in Flight acceptPut
4cebc543a <David Li> Mark flaky Flight test
c551d8527 <David Li> Fix new CheckStyle violations
85e21699c <David Li> Fix Flight integration tests using metadata
eff22393c <David Li> Use FlightStreamChunk in Flight/C++
72c2a3fa0 <David Li> Try to always close FlightStream after acceptPut
1718d9b42 <David Li> Make FlightStream cancellable from acceptPut
7ac44df79 <David Li> Make Netty version consistent with gRPC
1225b67af <David Li> Use ArrowBuf instead of byte for Flight metadata
ccfef2d1e <David Li> Disable Flight cancellation tests in CI
0484c333c <David Li> Pass Flight context to ListActions in Python
b0f71d967 <David Li> Replace ARROW_EXPORT with ARROW_FLIGHT_EXPORT
fdaa76e99 <David Li> Add client-side cancelation of DoGet operations
b4dbc445e <David Li> Enable non-nested dictionary batches in Flight integration tests
f7631a2fd <David Li> Add basic Arrow Flight docs
a8ac27fb3 <David Li> Implement application metadata in Flight
86f4789ab <David Li> Add application metadata field to FlightData message
Also improve the example a bit.
Author: Antoine Pitrou <antoine@python.org>
Closes#4376 from pitrou/ARROW-5398-fix-py-flight-tests and squashes the following commits:
a15313f60 <Antoine Pitrou> ARROW-5398: Fix Flight tests
I haven't changed the client/server construction interfaces to really take advantage of this. I would rather follow up with creating builders instead, as proposed, as that would be cleaner than special-casing the logic in the current constructors.
Author: David Li <li.davidm96@gmail.com>
Closes#4047 from lihalite/flight-uris and squashes the following commits:
870f6ebb8 <David Li> Add more builder options for Java Flight servers
5c127631b <David Li> Make Python Flight bindings more complete
675acc9bc <David Li> Introduce builder for C++ Flight servers
474419667 <David Li> Use builder for Flight server in Java
460442cea <David Li> Use URIs for Flight locations
Right now the only option is a timeout on an overall call, which will be useful to limit hangs when connecting to a Flight service that might have gone down. (gRPC is rather aggressive about retrying for a while before giving up.)
You could imagine having a health check action in DoAction, for instance, which fails if the call doesn't complete within a short time.
I would also like a stream timeout, where the timeout only fires if there is a long delay between messages on a DoGet or DoAction, but this is hard to implement as-is in gRPC. Implementing an async client interface might be a better way to achieve this.
Author: David Li <li.davidm96@gmail.com>
Closes#4144 from lihalite/arrow-5136-call-options and squashes the following commits:
f20181c90 <David Li> Use varargs for Java Flight CallOptions
a573d2c61 <David Li> Make FlightCallOptions a POD type
d4bef2781 <David Li> Add tests for Flight call options in Python
0e6111154 <David Li> Don't hold GIL during Flight server shutdown
a1829b32c <David Li> implement call options for Flight
81e3c7908 <David Li> add call options to Flight
Also sets up the server-side APIs for [ARROW-5136](https://issues.apache.org/jira/browse/ARROW-5136).
I mostly mirrored the existing Java APIs in each language, except I didn't add the BasicAuth specialization.
Author: David Li <li.davidm96@gmail.com>
Author: Antoine Pitrou <antoine@python.org>
Closes#4125 from lihalite/arrow-5137-auth and squashes the following commits:
067ff3217 <Antoine Pitrou> Avoid making I/O methods const
48d42485f <David Li> Implement auth API for Flight
14774082f <David Li> Implement auth API for Flight
02a9663ae <David Li> Implement auth API for Flight
Not a high priority for review - fine to land for 0.14
Author: David Li <David.M.Li@twosigma.com>
Closes#3958 from lihalite/no-redundant-parameter and squashes the following commits:
52ec0b4f5 <David Li> Remove redundant schema parameter to Flight client DoGet
Author: Antoine Pitrou <antoine@python.org>
Closes#3770 from pitrou/ARROW-4687-python-flight-sigint and squashes the following commits:
5e548839f <Antoine Pitrou> Use std::atomic<>
8a28061be <Antoine Pitrou> ARROW-4687: Stop Flight server on incoming signals
Author: David Li <David.M.Li@twosigma.com>
Closes#3712 from lihalite/flight-python-server and squashes the following commits:
10dccb674 <David Li> Use 'except *' to handle exceptions in Cython
445eb87cd <David Li> Copy Flight out parameter values instead of moving in Python
9839e0f14 <David Li> Lint Python Flight code
f33b51f74 <David Li> Guard against Flight server methods not filling out parameter
422b4bcd5 <David Li> Bind Flight action endpoints to Python
c09b138de <David Li> Bind Flight ListFlights server call to Python
4a5cbaf52 <David Li> Guard against Flight server returning null listing
033734df6 <David Li> Bind Flight DoGet server call to Python
35c62f694 <David Li> Treat null FlightDataStream as error
- [ ] Add docs
- [ ] Format code
- [ ] Include Python in integration tests (requires binding the JSON reader/writer from C++)
- [ ] Validate performance?
- [ ] Complete server bindings if approach makes sense
Author: David Li <David.M.Li@twosigma.com>
Author: Wes McKinney <wesm+git@apache.org>
Closes#3566 from lihalite/flight-python and squashes the following commits:
ac29ab88 <David Li> Clean up to-be-implemented parts of Flight Python bindings
9d5442a0 <David Li> Clarify various RecordBatchStream{Reader,Writer} wrappers
e1c298ad <David Li> Lint CMake files
77644447 <Wes McKinney> Reformat cmake
c6b02aa9 <David Li> Add basic Python bindings for Flight
Raúl Cumplido