Summary:
- Document that quoted numeric CSV values are still subject to type inference.
- Add a doctest-backed example showing how `ConvertOptions.column_types` preserves leading zeros.
- AI assistance was used for issue triage and drafting this documentation example.
Validation:
- `/tmp/arrow-doccheck-46853/bin/python -m pytest --doctest-glob="*.rst" docs/source/python/csv.rst -q` — passed, `1 passed`
- `git diff --check HEAD~1 HEAD` — passed
* GitHub Issue: #46853
Authored-by: kalyanamdewri <kalyanampriyam@gmail.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
Add an optional default_column_type parameter to the CSV reading API (C++ and Python) to provide a fallback type when per-column types aren’t specified, improving schema consistency and complementing the existing column_types logic.
### What changes are included in this PR?
- c++: new convert option "default_column_type" to augment logic around column_types parameter
- 3 reader tests: DefaultColumnTypePartialDefault, DefaultColumnTypeAllStringsWithHeader, DefaultColumnTypeAllStringsNoHeader). The last two tests are inspired by https://github.com/pandas-dev/pandas/pull/62242 and https://github.com/pandas-dev/pandas/issues/57666
- python: corresponding changes to make cpp change consumable from python
- python: extended test_convert_options test - include. default_column_type
- python: added new test "test_default_column_type" which tests how the field impacts schema; also test implicitly verifies leading zero preservation
- relevant documentation update for python component;
### Are these changes tested?
Yes. Existing and new tests are passing.
C++:
> [==========] Running 3 tests from 1 test suite.
> [----------] Global test environment set-up.
> [----------] 3 tests from ReaderTests
> [ RUN ] ReaderTests.DefaultColumnTypePartialDefault
> [ OK ] ReaderTests.DefaultColumnTypePartialDefault (3 ms)
> [ RUN ] ReaderTests.DefaultColumnTypeAllStringsWithHeader
> [ OK ] ReaderTests.DefaultColumnTypeAllStringsWithHeader (0 ms)
> [ RUN ] ReaderTests.DefaultColumnTypeAllStringsNoHeader
> [ OK ] ReaderTests.DefaultColumnTypeAllStringsNoHeader (0 ms)
> [----------] 3 tests from ReaderTests (4 ms total)
>
> [----------] Global test environment tear-down
> [==========] 3 tests from 1 test suite ran. (4 ms total)
> [ PASSED ] 3 tests.
All:
> [==========] 264 tests from 46 test suites ran. (452 ms total)
> [ PASSED ] 264 tests.
pyarrow:
New tests are passing.
### Are there any user-facing changes?
I believe this change is backward compatible. Parameter is optional and its default value doesn't change the existing behavior; All the existing rests are passing.
Maybe relevant: https://github.com/apache/arrow/issues/22232
Relates to https://github.com/apache/arrow/issues/47502
* GitHub Issue: #47502
* GitHub Issue: #22232
Lead-authored-by: Vlad Borovtsov <borovcov.vlad@gmail.com>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
In many places in the Python User Guide the code exampels are written with IPython directive (elsewhere code-block is used). IPython directives are converted to IPython format (`In` and `Out` during the doc build). This can lead to slower builds.
### What changes are included in this PR?
IPython directives are converted to runnable code-block (with `>>>` and `...`) and pytest doctest support for `.rst` files is added to the `conda-python-docs` CI job. This means the code in the Python User Guide is tested separately to the building of the documentation.
### Are these changes tested?
Yes, with the CI.
### Are there any user-facing changes?
Changes to the Python User Guide examples will have to be tested with `pytest --doctest-glob='*.rst' docs/source/python/file.rst`
* GitHub Issue: #28859
Lead-authored-by: AlenkaF <frim.alenka@gmail.com>
Co-authored-by: Alenka Frim <AlenkaF@users.noreply.github.com>
Co-authored-by: tadeja <tadeja@users.noreply.github.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
### Rationale for this change
Currently, the Arrow C++ CSV converter does not support parsing strings into `duration` types. This limits CSV ingestion capabilities when handling datasets with time-based intervals represented as numeric strings (e.g., `1000`, `2000000`). This PR adds support for parsing such strings into Arrow's `DurationType`.
Note: Human-readable duration formats such as `1s`, `2m`, or `3h` are not supported in this PR. Support for those formats may be considered in a future enhancement.
### What changes are included in this PR?
- Added `DurationValueDecoder` using `StringConverter<DurationType>`
- Registered support in both standard and dictionary converters
- Added unit tests covering:
- Basic parsing across all time units (s, ms, µs, ns)
- Null and custom null values
- Whitespace handling and error cases
### Are these changes tested?
Yes, conversion logic is fully covered by new tests in `converter_test.cc`.
### Are there any user-facing changes?
Yes, users can now convert duration strings in CSV files to Arrow `duration` arrays by specifying the appropriate schema type.
* GitHub Issue: #40278
Lead-authored-by: Zihan Qi <zihan.qi@tum.de>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Signed-off-by: Antoine Pitrou <antoine@python.org>
* Allow reading CSV columns as time32 and time64
* Automatically infer "hh:mm" and "hh:mm:ss" as time32[s]
Closes#10782 from pitrou/ARROW-11243-csv-times
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
Users may be surprised that type inference is done on the first block, and values can fail converting afterwards.
Closes#10132 from pitrou/ARROW-12482-csv-streaming-doc
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
- implement support for inferring CSV timestamps columns with fractional seconds
- use timestamp[ns] parser which also covers [ms] and [us] to
avoid testing 2 additional timestamp types
Closes#9761 from MartinNowak/ARROW-12031
Authored-by: Martin Nowak <code@dawg.eu>
Signed-off-by: Antoine Pitrou <antoine@python.org>
Also add a C++ InputStream wrapper that transforms data from the stream according to an arbitrary callable.
Closes#7456 from pitrou/ARROW-9106-csv-transcoding
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Wes McKinney <wesm@apache.org>
Only the serial version is implemented for now.
Closes#6764 from pitrou/ARROW-3410-streaming-csv
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: François Saint-Jacques <fsaintjacques@gmail.com>
Author: Antoine Pitrou <antoine@python.org>
Closes#4521 from pitrou/ARROW-5556-py-json-docs and squashes the following commits:
40064de97 <Antoine Pitrou> Address review comments
65f357a6b <Antoine Pitrou> ARROW-5556: Document JSON reader
Author: Antoine Pitrou <antoine@python.org>
Closes#3224 from pitrou/ARROW-3620-document-cpu-count and squashes the following commits:
15fda9ba <Antoine Pitrou> ARROW-3620: Document pa.cpu_count() in Sphinx API docs
Author: Uwe L. Korn <uwelk@xhochy.com>
Author: Korn, Uwe <Uwe.Korn@blue-yonder.com>
Author: Wes McKinney <wesm+git@apache.org>
Closes#2856 from xhochy/doc-merge and squashes the following commits:
5b687ffa7 <Wes McKinney> Add simple README for the format/ directory
071d16ad7 <Uwe L. Korn> Move format specifications back to /format/
337088bfd <Uwe L. Korn> Review comments
fbe99c96c <Uwe L. Korn> Add more C++ docs
78a5eaf4b <Uwe L. Korn> Fix Python docs build
0b4dd3341 <Uwe L. Korn> Rename doc to docs
918e76244 <Uwe L. Korn> Convert format docs to reST
7aeff65ef <Uwe L. Korn> Add doc generation to docker-compose
185cba8fb <Uwe L. Korn> Add pre-commit check for RAT
671d24466 <Uwe L. Korn> Fix references to format documents
bdd824c87 <Uwe L. Korn> Move doc to top-level
985d42806 <Uwe L. Korn> Move Sphinx doc to top-level directory
f7d5e92d9 <Uwe L. Korn> Build C++ API docs
7850db8a6 <Uwe L. Korn> Add breathe as a requirement
d4cf54259 <Uwe L. Korn> Fix linter issues
fd75660e6 <Korn, Uwe> Fix Sphinx build for sphinx>=1.8
9be6fbe6f <Korn, Uwe> Merge C++ and Python documentation
Kalyanam Dewri