This refactors the CSV write support to expose an explicit CSV writer class, and adds Python bindings and Datasets support.
Closes#10230 from lidavidm/arrow-12512
Authored-by: David Li <li.davidm96@gmail.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
This offers possibly performance naive CSV writer with
limited options to keep the initial PR down.
Obvious potential improvements to this approach
are:
- Smarter casts for dictionaries
- Arena allocation for intermediate cast results
The implementation also means that for all primitive type
support we might have to fill in gaps in our cast function.
Closes#9504 from emkornfield/csv
Lead-authored-by: Micah Kornfield <emkornfield@gmail.com>
Co-authored-by: emkornfield <micahk@google.com>
Co-authored-by: Antoine Pitrou <antoine@python.org>
Co-authored-by: Micah Kornfield <micahk@google.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
Only the serial version is implemented for now.
Closes#6764 from pitrou/ARROW-3410-streaming-csv
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: François Saint-Jacques <fsaintjacques@gmail.com>
CSV conversion now has distinct paths for string and binary columns. String columns are UTF8-validated by default, but it can be disabled by setting the `check_utf8` option in `ConvertOptions`.
CSV type inference now first attempts string conversion and falls back on binary if UTF8 validation fails (if it's not disabled).
As for performance, on pure ASCII columns single-threaded reading slows down by ~10% (which can be avoided by setting `check_utf8` to false). Multi-threaded reading does not seem affected here.
Based on PR #2916.
Author: Antoine Pitrou <antoine@python.org>
Closes#2924 from pitrou/ARROW-3407-csv-utf8-conversion and squashes the following commits:
26a812c5c <Antoine Pitrou> ARROW-3407: Add UTF8 handling to CSV conversion
This includes:
- a CSV table reader written in C++
- a Python wrapper around the CSV table reader
- simple type inference for CSV values (null -> int64 -> float64 -> binary)
- generic null parsing using Pandas defaults as a baseline
("NA", "N/A", "NaN"...)
- some simple syntax parameters for CSV parsing
Not included:
- conversion and typing options
- performance tuning
Author: Antoine Pitrou <antoine@python.org>
Closes#2576 from pitrou/ARROW-25-csv-reader and squashes the following commits:
4ae93b2d5 <Antoine Pitrou> ARROW-25: Implement CSV reader
Antoine Pitrou