### Rationale for this change
Expose remaining csv write options to pyarrow.
### What changes are included in this PR?
Adding `eol` and `null_string` to csv.WriteOptions.
### Are these changes tested?
Yes, testing of setters included.
### Are there any user-facing changes?
* GitHub Issue: #34577
Lead-authored-by: Martin Nowak <code@dawg.eu>
Co-authored-by: AlenkaF <frim.alenka@gmail.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
Add an optional default_column_type parameter to the CSV reading API (C++ and Python) to provide a fallback type when per-column types aren’t specified, improving schema consistency and complementing the existing column_types logic.
### What changes are included in this PR?
- c++: new convert option "default_column_type" to augment logic around column_types parameter
- 3 reader tests: DefaultColumnTypePartialDefault, DefaultColumnTypeAllStringsWithHeader, DefaultColumnTypeAllStringsNoHeader). The last two tests are inspired by https://github.com/pandas-dev/pandas/pull/62242 and https://github.com/pandas-dev/pandas/issues/57666
- python: corresponding changes to make cpp change consumable from python
- python: extended test_convert_options test - include. default_column_type
- python: added new test "test_default_column_type" which tests how the field impacts schema; also test implicitly verifies leading zero preservation
- relevant documentation update for python component;
### Are these changes tested?
Yes. Existing and new tests are passing.
C++:
> [==========] Running 3 tests from 1 test suite.
> [----------] Global test environment set-up.
> [----------] 3 tests from ReaderTests
> [ RUN ] ReaderTests.DefaultColumnTypePartialDefault
> [ OK ] ReaderTests.DefaultColumnTypePartialDefault (3 ms)
> [ RUN ] ReaderTests.DefaultColumnTypeAllStringsWithHeader
> [ OK ] ReaderTests.DefaultColumnTypeAllStringsWithHeader (0 ms)
> [ RUN ] ReaderTests.DefaultColumnTypeAllStringsNoHeader
> [ OK ] ReaderTests.DefaultColumnTypeAllStringsNoHeader (0 ms)
> [----------] 3 tests from ReaderTests (4 ms total)
>
> [----------] Global test environment tear-down
> [==========] 3 tests from 1 test suite ran. (4 ms total)
> [ PASSED ] 3 tests.
All:
> [==========] 264 tests from 46 test suites ran. (452 ms total)
> [ PASSED ] 264 tests.
pyarrow:
New tests are passing.
### Are there any user-facing changes?
I believe this change is backward compatible. Parameter is optional and its default value doesn't change the existing behavior; All the existing rests are passing.
Maybe relevant: https://github.com/apache/arrow/issues/22232
Relates to https://github.com/apache/arrow/issues/47502
* GitHub Issue: #47502
* GitHub Issue: #22232
Lead-authored-by: Vlad Borovtsov <borovcov.vlad@gmail.com>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
CSV and JSON options lack a nice repr/str dunder method
### What changes are included in this PR?
Add both these methods
### Are these changes tested?
Will be once it's ready for review
### Are there any user-facing changes?
No
* GitHub Issue: #47389
Lead-authored-by: Nic Crane <thisisnic@gmail.com>
Co-authored-by: AlenkaF <frim.alenka@gmail.com>
Co-authored-by: Alenka Frim <AlenkaF@users.noreply.github.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
Fixes https://github.com/apache/arrow/issues/36889
When writing CSV from a table where the first batch is empty, the header gets written twice:
```python
table = pa.table({"col1": ["a", "b", "c"]})
combined = pa.concat_tables([table.schema.empty_table(), table])
write_csv(combined, buf)
# Result: "col1"\n"col1"\n"a"\n"b"\n"c"\n <-- header appears twice
```
### What changes are included in this PR?
The bug happens because:
1. Header is written to `data_buffer_` and flushed during `CSVWriterImpl` initialization
2. The buffer is not cleared after flush
3. When the next batch is empty, `TranslateMinimalBatch` returns early without modifying `data_buffer_`
4. The write loop then writes `data_buffer_` which still contains stale content
The fix introduces a `WriteAndClearBuffer()` helper that writes the buffer to sink and clears it. This helper is used in all write paths:
- `WriteHeader()`
- `WriteRecordBatch()`
- `WriteTable()`
This ensures the buffer is always clean after any flush, making it impossible for stale content to be written again.
### Are these changes tested?
Yes. Added C++ tests in `writer_test.cc` and Python tests in `test_csv.py`:
- Empty batch at start of table
- Empty batch in middle of table
### Are there any user-facing changes?
No API changes. This is a bug fix that prevents duplicate headers when writing CSV from tables with empty batches.
* GitHub Issue: #36889
Lead-authored-by: Ruiyang Wang <ruiyang@anthropic.com>
Co-authored-by: Ruiyang Wang <56065503+rynewang@users.noreply.github.com>
Co-authored-by: Gang Wu <ustcwg@gmail.com>
Signed-off-by: Gang Wu <ustcwg@gmail.com>
### Rationale for this change
Expose CSV writer option quoting_header for pyarrow. Addresses https://github.com/apache/arrow/issues/47575
### What changes are included in this PR?
Cython changes for parsing quoting_header option in a manner similar to quoting_style
### Are these changes tested?
Yes, added a unit test under test_csv.py
### Are there any user-facing changes?
Add QuotingStyle quoting_header option in WriteOptions for pyarrow
* GitHub Issue: #47575
Authored-by: Ayush Bansal <abansal1008@gmail.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
The C++ arrow has a JSON streaming reader which is not exposed on the Python interface.
### What changes are included in this PR?
This PR is based on #33761. It adds the `open_json` method to open a streaming reader for a JSON file.
### Are these changes tested?
Yes
### Are there any user-facing changes?
Yes. A new `open_json` method has been added to the Python interface, located at `pyarrow.json.open_json`, and its parameters are the same as the `pyarrow.json.read_json`
* GitHub Issue: #14932
Lead-authored-by: pxc <panxuchen.pxc@alibaba-inc.com>
Co-authored-by: Akshay Subramanian <asubramanian@grailbio.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Python 3.8 is End of Support on 31st October 2023. We can drop support for pyarrow 18.0.0.
See: https://endoflife.date/python
### What changes are included in this PR?
Remove support for 3.8. Update minimum required Python version and update some CI jobs to use the minimum Python supported version
### Are these changes tested?
They will be tested via Archery
### Are there any user-facing changes?
Yes in terms of not supporting older versions of Python. This will be a breaking change for some users but not for the API itself
* GitHub Issue: #43518
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
### Rationale for this change
Being able to run pyarrow without requiring numpy.
### What changes are included in this PR?
If numpy is not present we are able to import pyarrow and run functionality.
A new CI job has been created to run some basic tests without numpy.
### Are these changes tested?
Yes via CI.
### Are there any user-facing changes?
Yes, NumPy can be removed from the user installation and pyarrow functionality still works
* GitHub Issue: #25118
Lead-authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Signed-off-by: Antoine Pitrou <antoine@python.org>
We hit this in conda-forge on some runners:
```
=================================== FAILURES ===================================
__________________ TestThreadedCSVTableRead.test_cancellation __________________
[...]
# Interruption should have arrived timely
> assert last_duration <= 1.0
E assert 1.2137842178344727 <= 1.0
pyarrow/tests/test_csv.py:1473: AssertionError
----------------------------- Captured stdout call -----------------------------
workload size: 100000
workload size: 300000
workload size: 900000
workload size: 2700000
workload size: 8100000
```
Give a little bit more time for the expected duration so conda-forge's test suite doesn't fail spuriously (and then have to be restarted manually).
Authored-by: H. Vetinari <h.vetinari@gmx.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
When writing the CSV reader, we thought that the parser not finding the same line limits as the chunker should never happen, hence the terse "chunker out of sync" error message.
It turns out that, if the input contains multiline cell values and the `newlines_in_values` option was not enabled, the chunker can happily delimit a block on a newline that's inside a quoted string. The parser will then see truncated data and will stop parsing, yielding a parsed size that's smaller than the first block (see added comment in the code).
### What changes are included in this PR?
* Add some parser tests that showcase the condition encountered in GH-39857
* Improve error message to guide users towards the solution
### Are these changes tested?
There's no functional change, the error message itself isn't tested.
### Are there any user-facing changes?
No.
* Closes: #39857
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
A deadlock can happen in a C++ destructor in the following case:
* the C++ destructor is called from Python, holding the GIL
* the C++ destructor waits for a threaded task to finish
* the threaded task has invoked some Python code which is waiting to acquire the GIL
### What changes are included in this PR?
To reliably present such a deadlock, introduce `std::shared_ptr` and `std::unique_ptr` wrappers that release the GIL when deallocating the embedded pointer.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
* Closes: #38676
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Cloudpickle was not tested in most parts of the pyarrow test suite. Improving this coverage will make the Cython 3.0.0 upgrade cleaner as cloudpickle was failing in a few places where the default pickle module was not. This has been verified using Cython 0.29.36.
### What changes are included in this PR?
* `__reduce__` methods that need to pass kwargs have been changed from classmethod to staticmethod
* All pytests that pickle objects are parameterized to use both `pickle` and `cloudpickle`
### Are these changes tested?
Yes, pytests run successfully with Cython 0.29.36
### Are there any user-facing changes?
No.
* Closes: #37254
Authored-by: Dane Pitkin <dane@voltrondata.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
Fix archery lint issues with new version of flake8 dependency.
### What changes are included in this PR?
* Lint errors.
### Are these changes tested?
Yes.
```
$ flake8 --version
6.1.0 (mccabe: 0.7.0, pycodestyle: 2.11.0, pyflakes: 3.1.0) CPython 3.11.4 on Darwin
$ archery lint --python
INFO:archery:Running Python formatter (autopep8)
INFO:archery:Running Python linter (flake8)
INFO:archery:Running Cython linter (cython-lint)
$
```
### Are there any user-facing changes?
No
* Closes: #36973
Authored-by: Dane Pitkin <dane@voltrondata.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
This exposes the QuotingStyle option to Python. I tested this with the v10 tag since I wasn't able to build the master branch for some reason.
I'm happy to use enums instead of strings. Both seem to be used in Python and I wasn't sure which one was preferred.
(no Jira ticket since I don't have an account and can't create one)
Lead-authored-by: Frederick Jansen <frederick.jansen@gmail.com>
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
This PR modifies the WriteOptions for CSV to allow changing the default delimiter.
Closes#12426 from sanjibansg/csv_delimiter
Authored-by: Sanjiban Sengupta <sanjiban.sg@gmail.com>
Signed-off-by: Yibo Cai <yibo.cai@arm.com>
Use `signal.set_wakeup_fd` to wait reliably for the signal to get to the Python interpreter.
Closes#12260 from pitrou/ARROW-15454-csv-cancellation
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
Size the workload according to the machine speed, to increase the probability of actually triggering cancellation.
Closes#11598 from pitrou/ARROW-14437-csv-cancellation-test
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
When the infering column decoder encounters an empty batch just return an empty array immediatly and do not consider the batch as the first batch.
Closes#10794 from n3world/ARROW-13441-First_batch_with_rows
Lead-authored-by: Nate Clark <nate@neworld.us>
Co-authored-by: Weston Pace <weston.pace@gmail.com>
Co-authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
* Allow reading CSV columns as time32 and time64
* Automatically infer "hh:mm" and "hh:mm:ss" as time32[s]
Closes#10782 from pitrou/ARROW-11243-csv-times
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
This converts the parser & decoder into map functions and then creates the streaming CSV reader as an async generator. Parallel readahead is then added on top of the parser/decoder to allow for parallel reads.
One thing that is lost at the moment is the ability to encounter a parsing error and then continue. There was a python test that read in the first block, failed to convert the second block, and then successfully read in a third block. I'm not sure if that restart behavior is important but if it is I can look into adding it.
Another thing that could be investigated in the future is combining the file readers and table readers more. They already share some components but the parsing and decoding logic, while basically the same, is handled very differently. The only real difference is that the table reader saves all the parsed blocks for re-parsing and the streaming reader does not.
Closes#10568 from westonpace/feature/ARROW-11889--c-add-parallelism-to-streaming-csv-reader
Authored-by: Weston Pace <weston.pace@gmail.com>
Signed-off-by: Weston Pace <weston.pace@gmail.com>
When rows are skipped but there are still more rows to skip set partial to the remainder of buffer.
Closes#10649 from n3world/ARROW-13246-block_middle_of_skip
Authored-by: Nate Clark <nate@neworld.us>
Signed-off-by: David Li <li.davidm96@gmail.com>
This refactors the CSV write support to expose an explicit CSV writer class, and adds Python bindings and Datasets support.
Closes#10230 from lidavidm/arrow-12512
Authored-by: David Li <li.davidm96@gmail.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
Some Python versions have a bug where `signal.getsignal` creates a reference cycle holding execution frames alive (https://bugs.python.org/issue42248).
This would cause excessive lifetimes of the PyArrow table returned by `read_csv`.
Closes#10609 from pitrou/ARROW-13187-signal-refcycle
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
The option is only applicable to string and binary columns.
Closes#10503 from pitrou/ARROW-10115-csv-quoted-nulls
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
This uses a stop token to let interactive users interrupt a long-running Flight operation. It's not perfect: the operation won't be cancelled until the server delivers a message, so this doesn't protect against very slow servers. (In that case, we'd need some way for the stop source to call TryCancel() on the gRPC RPC object, which would be tricky.) But so long as the server is being responsive, this means Ctrl-C should do what people expect in Python.
Closes#10318 from lidavidm/arrow-12050
Authored-by: David Li <li.davidm96@gmail.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
Add a new csv reader option which allows the reader to skip rows after reading the column names from the csv.
Closes#10255 from n3world/ARROW-12661-skip_after_names
Lead-authored-by: Nate Clark <nate@neworld.us>
Co-authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
For serial CSV readers track the absolute row number and report it in errors encountered during parsing or converting.
I did try to get row numbers for the parallel reader but the only way I thought that could work would be to add delimiter counting to the Chunker but that seemed to add more complexity than I wanted to.
Closes#10321 from n3world/ARROW-12675-report_rows
Authored-by: Nate Clark <nate@neworld.us>
Signed-off-by: Antoine Pitrou <antoine@python.org>
An uninitialized StopToken caused segfaults if you ever called read_csv with cancellation disabled or when not on the main thread (e.g. if used in a Flight server). If we have a 4.0.1 I think this qualifies as a regression.
Closes#10227 from lidavidm/arrow-12622
Authored-by: David Li <li.davidm96@gmail.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
arrow::csv::ConvertOptions has stl container data members which are
not guaranteed to be standard layout type. It causes invalid-offsetof
warning when compiled with clang-12.
This patch changes directly embedded CCSVConvertOptions to unique_ptr
to fix the issue. Parse and Read options are also updated.
Closes#10154 from cyb70289/12240-invalid-offsetof
Lead-authored-by: Yibo Cai <yibo.cai@arm.com>
Co-authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
* Make the interruptible workload larger
* Ensure KeyboardInterrupt doesn't stop the test suite
Closes#9920 from pitrou/ARROW-12241-csv-interrupt-test
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
- implement support for inferring CSV timestamps columns with fractional seconds
- use timestamp[ns] parser which also covers [ms] and [us] to
avoid testing 2 additional timestamp types
Closes#9761 from MartinNowak/ARROW-12031
Authored-by: Martin Nowak <code@dawg.eu>
Signed-off-by: Antoine Pitrou <antoine@python.org>
In this model, a StopSource is instantiated by the consumer, which passes a corresponding StopToken to producer API(s).
Closes#9528 from pitrou/ARROW-8732-cancel-v2
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
This offers possibly performance naive CSV writer with
limited options to keep the initial PR down.
Obvious potential improvements to this approach
are:
- Smarter casts for dictionaries
- Arena allocation for intermediate cast results
The implementation also means that for all primitive type
support we might have to fill in gaps in our cast function.
Closes#9504 from emkornfield/csv
Lead-authored-by: Micah Kornfield <emkornfield@gmail.com>
Co-authored-by: emkornfield <micahk@google.com>
Co-authored-by: Antoine Pitrou <antoine@python.org>
Co-authored-by: Micah Kornfield <micahk@google.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
Also implement conversion to dictionaries of numbers and decimals.
Closes#8160 from pitrou/ARROW-7302-csv-dict
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
* Add a ValidateUTF8() method to StringArray and LargeStringArray
* Automatically call ValidateUTF8() when ValidateFull() is called on
one of those types
Closes#7596 from pitrou/ARROW-9163-validate-utf8
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Wes McKinney <wesm@apache.org>
TODOs:
- [x] split PRs into two, one with simplified python to arrow conversions with benchmarks
- [x] implement union scalar on the C++ side
- [x] store the index value for dictionary scalar
- [x] more tests
Closes ARROW-9017, ARROW-9153
Closes#7519 from kszucs/ARROW-9153
Authored-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
Signed-off-by: Wes McKinney <wesm@apache.org>
Also add a C++ InputStream wrapper that transforms data from the stream according to an arbitrary callable.
Closes#7456 from pitrou/ARROW-9106-csv-transcoding
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Wes McKinney <wesm@apache.org>
`archery lint --flake8` becomes `archery lint --python` and now recognizes the `--fix` option.
Reformatting involves running `autopep8`.
Closes#7215 from pitrou/ARROW-8846-archery-autopep8
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
