### Rationale for this change
Expose remaining csv write options to pyarrow.
### What changes are included in this PR?
Adding `eol` and `null_string` to csv.WriteOptions.
### Are these changes tested?
Yes, testing of setters included.
### Are there any user-facing changes?
* GitHub Issue: #34577
Lead-authored-by: Martin Nowak <code@dawg.eu>
Co-authored-by: AlenkaF <frim.alenka@gmail.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
Add an optional default_column_type parameter to the CSV reading API (C++ and Python) to provide a fallback type when per-column types aren’t specified, improving schema consistency and complementing the existing column_types logic.
### What changes are included in this PR?
- c++: new convert option "default_column_type" to augment logic around column_types parameter
- 3 reader tests: DefaultColumnTypePartialDefault, DefaultColumnTypeAllStringsWithHeader, DefaultColumnTypeAllStringsNoHeader). The last two tests are inspired by https://github.com/pandas-dev/pandas/pull/62242 and https://github.com/pandas-dev/pandas/issues/57666
- python: corresponding changes to make cpp change consumable from python
- python: extended test_convert_options test - include. default_column_type
- python: added new test "test_default_column_type" which tests how the field impacts schema; also test implicitly verifies leading zero preservation
- relevant documentation update for python component;
### Are these changes tested?
Yes. Existing and new tests are passing.
C++:
> [==========] Running 3 tests from 1 test suite.
> [----------] Global test environment set-up.
> [----------] 3 tests from ReaderTests
> [ RUN ] ReaderTests.DefaultColumnTypePartialDefault
> [ OK ] ReaderTests.DefaultColumnTypePartialDefault (3 ms)
> [ RUN ] ReaderTests.DefaultColumnTypeAllStringsWithHeader
> [ OK ] ReaderTests.DefaultColumnTypeAllStringsWithHeader (0 ms)
> [ RUN ] ReaderTests.DefaultColumnTypeAllStringsNoHeader
> [ OK ] ReaderTests.DefaultColumnTypeAllStringsNoHeader (0 ms)
> [----------] 3 tests from ReaderTests (4 ms total)
>
> [----------] Global test environment tear-down
> [==========] 3 tests from 1 test suite ran. (4 ms total)
> [ PASSED ] 3 tests.
All:
> [==========] 264 tests from 46 test suites ran. (452 ms total)
> [ PASSED ] 264 tests.
pyarrow:
New tests are passing.
### Are there any user-facing changes?
I believe this change is backward compatible. Parameter is optional and its default value doesn't change the existing behavior; All the existing rests are passing.
Maybe relevant: https://github.com/apache/arrow/issues/22232
Relates to https://github.com/apache/arrow/issues/47502
* GitHub Issue: #47502
* GitHub Issue: #22232
Lead-authored-by: Vlad Borovtsov <borovcov.vlad@gmail.com>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
CSV and JSON options lack a nice repr/str dunder method
### What changes are included in this PR?
Add both these methods
### Are these changes tested?
Will be once it's ready for review
### Are there any user-facing changes?
No
* GitHub Issue: #47389
Lead-authored-by: Nic Crane <thisisnic@gmail.com>
Co-authored-by: AlenkaF <frim.alenka@gmail.com>
Co-authored-by: Alenka Frim <AlenkaF@users.noreply.github.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
Expose CSV writer option quoting_header for pyarrow. Addresses https://github.com/apache/arrow/issues/47575
### What changes are included in this PR?
Cython changes for parsing quoting_header option in a manner similar to quoting_style
### Are these changes tested?
Yes, added a unit test under test_csv.py
### Are there any user-facing changes?
Add QuotingStyle quoting_header option in WriteOptions for pyarrow
* GitHub Issue: #47575
Authored-by: Ayush Bansal <abansal1008@gmail.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
The C++ arrow has a JSON streaming reader which is not exposed on the Python interface.
### What changes are included in this PR?
This PR is based on #33761. It adds the `open_json` method to open a streaming reader for a JSON file.
### Are these changes tested?
Yes
### Are there any user-facing changes?
Yes. A new `open_json` method has been added to the Python interface, located at `pyarrow.json.open_json`, and its parameters are the same as the `pyarrow.json.read_json`
* GitHub Issue: #14932
Lead-authored-by: pxc <panxuchen.pxc@alibaba-inc.com>
Co-authored-by: Akshay Subramanian <asubramanian@grailbio.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
A deadlock can happen in a C++ destructor in the following case:
* the C++ destructor is called from Python, holding the GIL
* the C++ destructor waits for a threaded task to finish
* the threaded task has invoked some Python code which is waiting to acquire the GIL
### What changes are included in this PR?
To reliably present such a deadlock, introduce `std::shared_ptr` and `std::unique_ptr` wrappers that release the GIL when deallocating the embedded pointer.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
* Closes: #38676
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Fix the typo found in #34340 which copied from this file.
### What changes are included in this PR?
Minor typo fixes.
### Are these changes tested?
No. Minor documentation changes only.
### Are there any user-facing changes?
No.
Authored-by: SHIMA Tatsuya <ts1s1andn@gmail.com>
Signed-off-by: Alenka Frim <frim.alenka@gmail.com>
This exposes the QuotingStyle option to Python. I tested this with the v10 tag since I wasn't able to build the master branch for some reason.
I'm happy to use enums instead of strings. Both seem to be used in Python and I wasn't sure which one was preferred.
(no Jira ticket since I don't have an account and can't create one)
Lead-authored-by: Frederick Jansen <frederick.jansen@gmail.com>
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
This PR moves all the `arrow/python` definitions to its own `libarrow_python.pxd` file as requested on ARROW-16062
Closes#12873 from raulcd/ARROW-16062
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
This PR modifies the WriteOptions for CSV to allow changing the default delimiter.
Closes#12426 from sanjibansg/csv_delimiter
Authored-by: Sanjiban Sengupta <sanjiban.sg@gmail.com>
Signed-off-by: Yibo Cai <yibo.cai@arm.com>
Address all docstrings to make sure they pass `archery numpydoc --allow-rule PR01`
Closes#11245 from amol-/ARROW-13637
Authored-by: Alessandro Molina <amol@turbogears.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
Allow quoted nulls if quoted_strings_can_be_null is true
Closes#10897 from westonpace/bugfix/ARROW-13580--c-quoted_strings_can_be_null-only-applied-to-s
Lead-authored-by: Weston Pace <weston.pace@gmail.com>
Co-authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
This refactors the CSV write support to expose an explicit CSV writer class, and adds Python bindings and Datasets support.
Closes#10230 from lidavidm/arrow-12512
Authored-by: David Li <li.davidm96@gmail.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
The option is only applicable to string and binary columns.
Closes#10503 from pitrou/ARROW-10115-csv-quoted-nulls
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
Add a new csv reader option which allows the reader to skip rows after reading the column names from the csv.
Closes#10255 from n3world/ARROW-12661-skip_after_names
Lead-authored-by: Nate Clark <nate@neworld.us>
Co-authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
arrow::csv::ConvertOptions has stl container data members which are
not guaranteed to be standard layout type. It causes invalid-offsetof
warning when compiled with clang-12.
This patch changes directly embedded CCSVConvertOptions to unique_ptr
to fix the issue. Parse and Read options are also updated.
Closes#10154 from cyb70289/12240-invalid-offsetof
Lead-authored-by: Yibo Cai <yibo.cai@arm.com>
Co-authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
Users may be surprised that type inference is done on the first block, and values can fail converting afterwards.
Closes#10132 from pitrou/ARROW-12482-csv-streaming-doc
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
This adds ReadOptions to CsvFileFormat and exposes ReadOptions, ConvertOptions, and CsvFragmentScanOptions to Python.
ReadOptions was added to CsvFileFormat as its options can affect the discovered schema. For the block size, which does not need to be global, a field was added to CsvFragmentScanOptions.
Closes#9725 from lidavidm/arrow-8631
Authored-by: David Li <li.davidm96@gmail.com>
Signed-off-by: Benjamin Kietzman <bengilgit@gmail.com>
In this model, a StopSource is instantiated by the consumer, which passes a corresponding StopToken to producer API(s).
Closes#9528 from pitrou/ARROW-8732-cancel-v2
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
The `io::IOContext` class allows passing various settings such as the MemoryPool used for allocation and the Executor for async methods.
Closes#9474 from pitrou/ARROW-10420-io-context
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Benjamin Kietzman <bengilgit@gmail.com>
This offers possibly performance naive CSV writer with
limited options to keep the initial PR down.
Obvious potential improvements to this approach
are:
- Smarter casts for dictionaries
- Arena allocation for intermediate cast results
The implementation also means that for all primitive type
support we might have to fill in gaps in our cast function.
Closes#9504 from emkornfield/csv
Lead-authored-by: Micah Kornfield <emkornfield@gmail.com>
Co-authored-by: emkornfield <micahk@google.com>
Co-authored-by: Antoine Pitrou <antoine@python.org>
Co-authored-by: Micah Kornfield <micahk@google.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
The goal is to have a standardized ABI to communicate streams of homogeneous arrays or record batches (for example for database result sets).
The trickiest part is error reporting. This proposal tries to strike a compromise between simplicity (an integer error code mapping to errno values) and expressivity (an optional description string for application-specific and context-specific details).
Closes#8052 from pitrou/ARROW-9761-c-array-stream
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
By default, Cython extension classes (defined with "cdef class") don't have a weakref slot, so add one to all of them.
This adds just one memory word to each object, which IMHO is acceptable.
Closes#7758 from pitrou/ARROW-9469-py-weakrefable-objects
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
Also add a C++ InputStream wrapper that transforms data from the stream according to an arbitrary callable.
Closes#7456 from pitrou/ARROW-9106-csv-transcoding
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Wes McKinney <wesm@apache.org>
This is some routine scrubbing that deals with some remnants of the prior dual py2/3 codebase.
Closes#7421 from wesm/ARROW-9030
Authored-by: Wes McKinney <wesm@apache.org>
Signed-off-by: Wes McKinney <wesm@apache.org>
Only the serial version is implemented for now.
Closes#6764 from pitrou/ARROW-3410-streaming-csv
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: François Saint-Jacques <fsaintjacques@gmail.com>
Closes#6463 from jorisvandenbossche/ARROW-7628-csv-doc and squashes the following commits:
19e6d3049d <Joris Van den Bossche> update phrasing
fdfbeb9857 <Joris Van den Bossche> ARROW-7628: Clarify docs of csv reader skip_rows and nulls in strings
Authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: François Saint-Jacques <fsaintjacques@gmail.com>
Closes#5936 from pitrou/ARROW-7236-csv-result-migration and squashes the following commits:
a49de2c69 <Antoine Pitrou> ARROW-7236: Add Result<T> APIs to arrow/csv
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Wes McKinney <wesm+git@apache.org>
This is tied to type inference, and only triggers on string or binary columns.
Each chunk is dict-encoded up to a certain cardinality, after which the whole
column falls back on plain encoding.
Closes#5785 from pitrou/ARROW-3408-csv-auto-dict-encode and squashes the following commits:
8d909fdcd <Antoine Pitrou> ARROW-3408: Add CSV option to automatically attempt dict encoding
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Wes McKinney <wesm+git@apache.org>
Add an option that, if enabled, will autogenerate column names
for the target table rather than read them from the CSV file.
Closes#5206 from pitrou/ARROW-6231-autogenerate-column-names and squashes the following commits:
cc7fdc699 <Antoine Pitrou> ARROW-6231: Allow generating CSV column names
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: François Saint-Jacques <fsaintjacques@gmail.com>
If the `include_columns` option is not empty:
- columns are included in that order
- columns not in `include_columns` are ignored
- columns in `include_columns` but not in the CSV file produce a null column
Closes#5026 from pitrou/ARROW-5977-csv-include-columns and squashes the following commits:
0f198ef0b <Antoine Pitrou> ARROW-5977: Allow specifying which columns to include
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
