### Rationale for this change
Demonstrate how to deal with async Flight from Python.
### What changes are included in this PR?
Add bindings for GetFlightInfo.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes, new APIs.
* Closes: #37093
Lead-authored-by: David Li <li.davidm96@gmail.com>
Co-authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Cython 3 has some breaking changes:
* `&&` is no longer supported
* `nogil` must appear at the end of the line
In addition we had to work around a bug: https://github.com/cython/cython/issues/5333
### What changes are included in this PR?
Minor changes to the cython code.
### Are these changes tested?
No, and they probably should be at some point. However, Cython 3 has not yet released and so it would be impractical to update our CI to run Cython 3 until that happens.
### Are there any user-facing changes?
No
* Closes: #34564
Authored-by: Weston Pace <weston.pace@gmail.com>
Signed-off-by: Weston Pace <weston.pace@gmail.com>
In complex scenarios, the global signal-receiving StopSource may have already been created.
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
This PR moves all the `arrow/python` definitions to its own `libarrow_python.pxd` file as requested on ARROW-16062
Closes#12873 from raulcd/ARROW-16062
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Use `signal.set_wakeup_fd` to wait reliably for the signal to get to the Python interpreter.
Closes#12260 from pitrou/ARROW-15454-csv-cancellation
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
Address all docstrings to make sure they pass `archery numpydoc --allow-rule PR01`
Closes#11245 from amol-/ARROW-13637
Authored-by: Alessandro Molina <amol@turbogears.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
Some Python versions have a bug where `signal.getsignal` creates a reference cycle holding execution frames alive (https://bugs.python.org/issue42248).
This would cause excessive lifetimes of the PyArrow table returned by `read_csv`.
Closes#10609 from pitrou/ARROW-13187-signal-refcycle
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
An uninitialized StopToken caused segfaults if you ever called read_csv with cancellation disabled or when not on the main thread (e.g. if used in a Flight server). If we have a 4.0.1 I think this qualifies as a regression.
Closes#10227 from lidavidm/arrow-12622
Authored-by: David Li <li.davidm96@gmail.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
In this model, a StopSource is instantiated by the consumer, which passes a corresponding StopToken to producer API(s).
Closes#9528 from pitrou/ARROW-8732-cancel-v2
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
This is some routine scrubbing that deals with some remnants of the prior dual py2/3 codebase.
Closes#7421 from wesm/ARROW-9030
Authored-by: Wes McKinney <wesm@apache.org>
Signed-off-by: Wes McKinney <wesm@apache.org>
Only the serial version is implemented for now.
Closes#6764 from pitrou/ARROW-3410-streaming-csv
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: François Saint-Jacques <fsaintjacques@gmail.com>
When an Arrow IO function fails, it should be turned into the appropriate
Python exception subclass, e.g. FileNotFoundError in case the requested
file or directory is missing.
Closes#5960 from pitrou/ARROW-7282-python-ioerror-subclasses and squashes the following commits:
262616e29 <Antoine Pitrou> ARROW-7282: IO functions should raise the right exceptions
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
If the `include_columns` option is not empty:
- columns are included in that order
- columns not in `include_columns` are ignored
- columns in `include_columns` but not in the CSV file produce a null column
Closes#5026 from pitrou/ARROW-5977-csv-include-columns and squashes the following commits:
0f198ef0b <Antoine Pitrou> ARROW-5977: Allow specifying which columns to include
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
This provides less "pluggability" but I think still offers a clean model for extension (subsystems can wrap the constructor for there purposes, and provide external static methods to check for particular types of errors).
Author: Micah Kornfield <emkornfield@gmail.com>
Author: Antoine Pitrou <antoine@python.org>
Closes#4484 from emkornfield/status_code_proposal and squashes the following commits:
4d1ab8d1d <Micah Kornfield> don't import plasma errors directly into top level pyarrow module
a66f999f8 <Micah Kornfield> make format
040216d48 <Micah Kornfield> fixes for comments outside python
729bba1ff <Antoine Pitrou> Fix Py2 issues (hopefully)
ea56d1e6a <Antoine Pitrou> Fix PythonErrorDetail to store Python error state (and restore it in check_status())
21e1b95ac <Micah Kornfield> fix compilation
9c905b094 <Micah Kornfield> fix lint
74d563cd7 <Micah Kornfield> fixes
85786efb1 <Micah Kornfield> change messages
3626a9016 <Micah Kornfield> try removing message
a4e6a1ff2 <Micah Kornfield> add logging for debug
4586fd1e2 <Micah Kornfield> fix typo
8f011b329 <Micah Kornfield> fix status propagation
317ea9c66 <Micah Kornfield> fix complie
9f5916070 <Micah Kornfield> don't make_shared inline
484b3a232 <Micah Kornfield> style fix
14e3467b5 <Micah Kornfield> dont rely on rtti
cd22df64d <Micah Kornfield> format
dec458506 <Micah Kornfield> not-quite pluggable error codes
Also our first API to return a `Result<>`.
Includes appropriate massaging in the test helpers so that ASSERT_OK and ASSERT_RAISES can handle `Result<>` instances.
Author: Antoine Pitrou <antoine@python.org>
Closes#4707 from pitrou/ARROW-4788-struct-array-factory and squashes the following commits:
ae5435857 <Antoine Pitrou> ARROW-4788: Less verbose API for constructing StructArray
Now emits
- TypeError when indices are not integral
- KeyError when an index is out of bounds
Author: Benjamin Kietzman <bengilgit@gmail.com>
Closes#4317 from bkietz/5311-Return-more-specific-invalid-Status-in-T and squashes the following commits:
098afd1ed <Benjamin Kietzman> Fix copypasta doccomments, add IndexError to ConvertPyError
3253f3c25 <Benjamin Kietzman> remove duplicate switch case
dd49bc876 <Benjamin Kietzman> use IndexError in Take()
144745f1d <Benjamin Kietzman> add IndexError status code
eec1536c9 <Benjamin Kietzman> adjust expected errors in tests of take consumers
b3ede9ce6 <Benjamin Kietzman> use more specific error status types in take
This error signals an attempt to exceed capacity of a buffer or container. I initially thought I'd call this `OverflowError` but `CapacityError` makes it clearer that it's not about overflow on arithmetic operations, iMHO.
Author: Antoine Pitrou <antoine@python.org>
Closes#1991 from pitrou/ARROW-2389-capacity-error and squashes the following commits:
afc037d <Antoine Pitrou> ARROW-2389: Add CapacityError
This fixes the serialize and deserialize methods to throw the right exceptions.
Author: Philipp Moritz <pcmoritz@gmail.com>
Closes#1029 from pcmoritz/pyarrow-serialization-error and squashes the following commits:
249b38ed [Philipp Moritz] formatting
2d872a4c [Philipp Moritz] fixes
caad76d1 [Philipp Moritz] add test for deserialization (needs fixing)
57772124 [Philipp Moritz] fix python serialization error propagation
Author: Philipp Moritz <pcmoritz@gmail.com>
Closes#1019 from pcmoritz/plasma-serialization-context and squashes the following commits:
caf85165 [Philipp Moritz] fix
8e8e9011 [Philipp Moritz] add error handling
1a4fade0 [Philipp Moritz] add tests and fixes
2389659e [Philipp Moritz] Pull serialization context through PlasmaClient put and get
This does drop the vector append to the builder that was there before. I'm going to do some local benchmarking to make sure this doesn't degrade performance unacceptably, will report back here
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closes#956 from wesm/ARROW-1342 and squashes the following commits:
f2ebeba8 [Wes McKinney] Fix cpplint issue
f403f9dd [Wes McKinney] Fix test case to be platform independent, note ARROW-1345. Improve quality of error message
f4f44c18 [Wes McKinney] Fix test case where inferred list type is null
ae5c8312 [Wes McKinney] Drop striding check
b4aecd3a [Wes McKinney] Support strided ndarrays in pandas conversion from nested lists
This PR introduces a Cython API to Plasma, a FindPlasma.cmake to make it easier to integrate Plasma with CMake projects and sets up packaging with pyarrow.
Author: Philipp Moritz <pcmoritz@gmail.com>
Author: Robert Nishihara <robertnishihara@gmail.com>
Closes#797 from pcmoritz/plasma-cython and squashes the following commits:
d8319fc [Philipp Moritz] get for of PlasmaClient.connect
d14ab87 [Philipp Moritz] get rid of MutableBuffer
08f24a5 [Philipp Moritz] fix typos and move FixedSizeBufferOutputStream
e33443d [Philipp Moritz] fix setup.py develop for plasma
5f7b779 [Philipp Moritz] changes needed to make Ray work with Plasma in Arrow
b9e2dee [Philipp Moritz] fix windows build
3e4a84d [Philipp Moritz] fix segfault
0bea267 [Philipp Moritz] debug
23fe5f5 [Philipp Moritz] make plasma store binary part of the pyarrow package for tests
b863d13 [Philipp Moritz] fix
997de1e [Philipp Moritz] fix
47dc739 [Philipp Moritz] fixes
47033e7 [Philipp Moritz] switch to pytest
ed84c53 [Philipp Moritz] partial fixes
9bc5c15 [Philipp Moritz] implement wait and fetch for the client
45f338f [Philipp Moritz] test plasma on macOS
8b53618 [Philipp Moritz] fix
54f595e [Philipp Moritz] try fixing python 2 tests
2c6d652 [Philipp Moritz] convert docs to numpy format
3270628 [Philipp Moritz] try to get documentation up
44d1a55 [Philipp Moritz] cleanups and release GIL
a9f6502 [Philipp Moritz] more fixes
1ff88e7 [Philipp Moritz] fix travix ci
348f9bf [Philipp Moritz] fixes
4ae1a27 [Philipp Moritz] fix
fd80203 [Philipp Moritz] Plasma Python extension packaging: It compiles!
3b69973 [Robert Nishihara] Fixed minor python linting.
c9f6bcf [Robert Nishihara] Fix indentation and line lengths in plasma.pyx.
67b0951 [Robert Nishihara] Fix long lines in plasma/test/test.py.
e26527c [Robert Nishihara] Convert plasma test.py from 2 space indentation to 4 space indentation.
acc71d2 [Philipp Moritz] add round trip test for dataframes
2b7f949 [Philipp Moritz] implement mutable arrow python buffers
c06f1b5 [Philipp Moritz] fix test
1d7928f [Philipp Moritz] add arrow roundtrip test
6371e2e [Philipp Moritz] fix tests
3021d59 [Philipp Moritz] make ObjectID pickleable
dd5a7d8 [Philipp Moritz] fix tests
777e9c7 [Philipp Moritz] introduce plasma namespace
a4a9628 [Philipp Moritz] fix c++ tests
924888b [Philipp Moritz] update
f970df3 [Philipp Moritz] reduce logging
2ff2480 [Philipp Moritz] workaround for python visibility
d4934a9 [Philipp Moritz] update
cba92c1 [Philipp Moritz] setup.py for plasma
066d0ea [Philipp Moritz] test
1aea320 [Philipp Moritz] run plasma tests
3c4de52 [Philipp Moritz] use cmake to build the cython extension
bf39297 [Philipp Moritz] build and install pyarrow for plasma tests
5bf722a [Philipp Moritz] fix plasma path
1c5434c [Philipp Moritz] fix formatting
187cc24 [Philipp Moritz] add travis tests
c3d462d [Philipp Moritz] remove Python C extension
d9261b4 [Philipp Moritz] add documentation and license
db2d09a [Philipp Moritz] get all python tests in place
78d08ac [Philipp Moritz] make eviction work in Cython
18e0ac4 [Philipp Moritz] get tests
bc681ca [Philipp Moritz] port some python tests
f8e05f2 [Philipp Moritz] implement plasma.get in the cython client
d590c8a [Philipp Moritz] update
5178ee7 [Philipp Moritz] update
9044a01 [Philipp Moritz] initial plasma cython client commit
I have been looking at LXML's approach to creating both a public Cython API and C++ API
https://github.com/lxml/lxml
While this may seem like a somewhat radical reorganization of the code, putting all of the main symbols in a single Cython extension makes generating a C++ API for them significantly simpler. By using `.pxi` files we can break the codebase into as small pieces as we like (as long as there are no circular dependencies). As a convenient side effect, the build times are shorter.
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closes#680 from wesm/ARROW-819 and squashes the following commits:
9e6ee246 [Wes McKinney] Fix up optional extensions
cff757de [Wes McKinney] Expose pyarrow C API in arrow/python/pyarrow.h
b39d19cd [Wes McKinney] Fix test suite. Move _config into lib
ff1b5e51 [Wes McKinney] Rename things a bit
d4a83912 [Wes McKinney] Reorganize Cython code in the style of lxml so make declaring a public C API easier
