* csv: validate dialect options
Resolve each dialect once and validate the merged options before
constructing readers and writers. Handle Unicode character parsing
consistently and enable the corresponding CPython CSV tests.
Assisted-by: Tau:gpt-5.6-luna
* cargo fmt
* Validate csv dialect conflicts with line terminators
- Reject dialect characters that overlap any lineterminator character
- Validate Dialect construction consistently and allow non-ASCII terminators
- Unskip csv invalid character coverage now that validation matches behavior
* Remove stale CSV rejection snippet
Removes a RustPython-only assertion that contradicts the
CPython-compatible dialect validation added by this branch.
Scope: test-only
AI-Assisted-By: Codex
* Revert "Remove stale CSV rejection snippet"
This reverts commit 3a145dbba4.
* Reapply "Remove stale CSV rejection snippet"
This reverts commit 0754731a2e.
* fix(csv): complete dialect merge resolution
Restore dialect validation and remove stale quoting logic.
Assisted-by: Codex: GPT-5
* csv: port CPython reader state machine
Replace the csv-core reader and per-item quote scanner with one Rust
implementation of CPython's nine-state reader parser. Keep parser state
across iterator items and distinguish virtual item boundaries from true
iterator exhaustion.
Centralize field completion so quote provenance, empty-field None conversion,
float conversion, strict parsing, field limits, blank rows, and escaped or
quoted newlines share one path. Apply the existing reentrant-iterator
generation check to every iterator item consumed by a record.
Also accept empty ASCII line terminators like CPython, report a missing
csv-core writer sentinel as _csv.Error instead of panicking, and name the
FSM entry point process_parser_input.
Remove the expected-failure markers from the eleven reader tests that now
pass.
Assisted-by: Codex:gpt-5.6-sol
* csv: test CRLF boundaries with custom lineterminator
* csv: support multi-character lineterminator in writer
Store the dialect line terminator as an owned String instead of the
single-byte csv_core::Terminator, dropping PyDialect's Copy derive. The
manual writer paths emit the full terminator; the csv-core-backed
QUOTE_ALL/QUOTE_NONNUMERIC paths emit a sentinel byte (preserving
csv-core's quote/empty-record bookkeeping) and append the real
terminator. field_needs_quotes/escape now quote a field containing any
terminator byte. The reader ignores lineterminator and always uses CRLF,
matching CPython and avoiding mid-UTF-8 record splits.
* csv: unmark now-passing test_write_lineterminator
* csv: address review feedback
- Remove the redundant per-branch sentinel terminator setup in to_writer;
the unconditional terminator call after the match is the single source.
- Reword the empty-lineterminator errors to "must not be empty" (the
constraint is non-empty, not single-character) on both entry points.
- Promote the sentinel invariant check in writerow from debug_assert_eq!
to assert_eq! so it also guards release builds.
- Add a snippet case for a field containing a line-break byte to ensure
the csv-core path drops only the trailing terminator.
* csv: reject non-ASCII lineterminator
The writer decides what to quote and escape by comparing raw bytes, so a
non-ASCII terminator quoted a field that merely shared a UTF-8 lead byte,
and QUOTE_NONE escaped individual bytes of the terminator and then failed to
decode the record back to a string.
Reject non-ASCII terminators, including lone surrogates, as csv.Error when
the dialect is parsed, and leave code-point-wise handling to a follow-up
(#8310). The non-string error message now matches CPython as well.
* csv: handle empty fields with skipinitialspace
skipinitialspace preprocessed records by trimming both ends of every split
field. Fields containing only spaces could therefore produce an invalid
slice, while trailing spaces were removed even though CPython preserves them.
Trim only field prefixes in the csv-core path and make the remaining
all-space trimming helper return an empty slice safely.
QUOTE_NOTNULL and QUOTE_STRINGS add another distinction: an unquoted empty
field becomes None, but a quoted empty field remains an empty string. Both
forms have identical decoded bytes, so extend the existing QUOTE_NONE custom
reader into one shared path that records whether each field started quoted.
Use that metadata only for the null conversion while retaining QUOTE_NONE
escape behavior.
Unmark Test_Csv.test_read_skipinitialspace now that its standard,
QUOTE_NOTNULL, and QUOTE_STRINGS cases pass.
This keeps the existing one-item reader lifecycle and does not add the larger
strict or multiline parser state machine.
Assisted-by: Codex:gpt-5-sol
* csv: make skipinitialspace quote-aware
The csv-core preprocessing path split each raw iterator item on every
delimiter before trimming field prefixes. Delimiters inside quoted fields were
therefore treated as separators, so spaces after them could be removed before
csv-core parsed the record.
Extract the quote, escape, delimiter, and field-start transitions from
read_quote_record() into a small per-item scanner. Reuse its events in the
skipinitialspace preprocessor and discard only InitialSpace events while
copying all other source bytes unchanged.
This keeps the custom quote-mode reader and preprocessing logic aligned
without introducing persistent multiline or strict parser state. Remove the
delimiter cache that was only needed by the old split-and-join path and add a
regression for spaces following a delimiter inside a quoted field.
Assisted-by: Codex:gpt-5-sol
* csv: reject reentrant reader advancement
Snapshot the reader generation before calling the input iterator. If a nested
call completes a record and advances the same reader, reject the outer call
with csv.Error instead of continuing with stale parser state.
Increment the generation on each successful record path, matching CPython's
handling of the existing reentrant-reader regression. Enable that test by
removing its expectedFailure marker.
Assisted-by: Codex:gpt-5.6-sol
* csv: invalidate outer reads after reentry
Advance the reader generation immediately after the reentrancy check, before
validating or parsing the returned input item. This invalidates the outer call
even when a nested read later fails validation or parsing.
Remove the per-return generation updates now that the common path advances it
exactly once.
Assisted-by: Codex:gpt-5.6-sol
* csv: generation type conversion(usize to u64)
On 32-bit targets (like WebAssembly, which RustPython supports), usize is 32-bit.
Csv with over 32-bit rows can raise panic on these platform.
type conversion from usize to u64.
* csv: quote embedded CR/LF under QUOTE_MINIMAL with custom lineterminator
QUOTE_MINIMAL rows were serialized by csv-core, which only quotes bytes
registered for the configured terminator, so fields containing '\r' or
'\n' were left unquoted when the lineterminator was not CR/LF (e.g. '!'
or '\0'). Route QUOTE_MINIMAL through a hand-written path that uses
field_needs_quotes, which always treats '\r'/'\n' as needing quotes.
A single empty field is quoted and an empty row emits only the
terminator, matching CPython. QUOTE_ALL / QUOTE_NONNUMERIC still use
csv-core.
Assisted-by: Claude Code:claude-opus-4-8
* csv: unmark now-passing writer tests
test_write_iterable and test_write_empty_fields pass with the
QUOTE_MINIMAL writer fix (empty rows no longer emit an empty field).
Assisted-by: Claude Code:claude-opus-4-8
* csv: map QUOTE_MINIMAL to Necessary quote style
QuoteStyle::Minimal was mapped to csv_core's Always, which quoted every
field even under QUOTE_MINIMAL. Map it to Necessary so only fields that
require quoting are quoted.
Unmarks Test_Csv.test_write_quoting.
Assisted-by: Claude Code:claude-opus-4-8
* csv: apply dialect quoting and lineterminator in writer
csv.writer ignored the quoting and lineterminator from a dialect (whether
passed by name or object), always using the defaults. Resolve both from the
dialect via get_quoting()/get_lineterminator(), mirroring get_delimiter(), so
an explicit keyword argument still overrides the dialect.
Unmarks TestDialectUnix.test_simple_writer.
Assisted-by: Claude Code:claude-opus-4-8
* Support array 'w' (Py_UCS4) typecode
Python 3.13 added the 'w' typecode holding fixed 4-byte unicode code
points, unlike 'u' which is platform-dependent wchar_t. Add a Ucs4Char
element type and route it through the existing unicode-array paths:
constructor from str, fromunicode()/tounicode(), __reduce_ex__, repr,
byteswap, and the machine-format code (Utf32). Also register 'w' as a
struct format char in vm::buffer.
Remove the now-passing expectedFailure markers for 'w' tests in
test_array.py.
Assisted-by: Claude
* Reject non-native byte order for the 'w' format char
FormatType::parse accepted 'w' after '<', '>', '!', or '=', but the
non-native branch of FormatType::info has no Ucs4Char arm and hits
unreachable!(), so struct.calcsize('>w') panicked the interpreter.
Add Ucs4Char to the existing native-only filter in FormatCode::parse
(alongside SSizeT/SizeT/VoidP) so those formats fail with the regular
"bad char in struct format" error instead. Native 'w' and
memoryview(array('w', ...)) are unaffected.
Assisted-by: Claude
* Unmark 'w'-typecode tests in test_re and test_csv
test_re's test_empty_array and test_csv's test_char_write construct
array('w', ...) and were marked expectedFailure while the typecode was
unsupported. Now that 'w' works they pass, and regrtest counts the
unexpected successes as failures in CI. Remove the two markers, which
restores both tests to their CPython v3.14.6 form; test_re, test_csv,
and test_array all pass.
Assisted-by: Claude
* csv: apply dialect escapechar when reading
ReaderBuilder only received escapechar when it was passed directly as a
keyword argument. A dialect object's or registered dialect's escapechar was
visible through reader.dialect but not used by csv_core.
Pass escapechar through for named, object, and default dialect paths. This
makes escaped delimiters inside quoted fields follow the configured dialect,
for example '"abc\,def"' reads as 'abc,def'.
Unmark TestQuotedEscapedExcel.test_read_escape_fieldsep.
Assisted-by: Codex
* csv: handle escaped delimiters with QUOTE_NONE
csv_core applies escapechar while reading quoted fields, but its unquoted
field state treats an escaped delimiter as a delimiter. Consequently,
QUOTE_NONE with escapechar split 'abc\,def' into 'abc\' and 'def'.
Add a small QUOTE_NONE record parser for dialects with escapechar. It keeps
the byte following an escape character as field data, so escaped delimiters
do not end a field while ordinary delimiters and record terminators retain
their usual meaning.
Unmark TestEscapedExcel.test_read_escape_fieldsep.
Assisted-by: Codex
* csv: respect skipinitialspace with QUOTE_NONE
The QUOTE_NONE parser returned before the normal reader's whitespace
preprocessing, so it ignored skipinitialspace when escapechar was set.
Track whether parsing has just followed a delimiter and skip only ordinary
spaces in that position. Escaped spaces remain field data.
Add a stdlib_csv snippet regression test for QUOTE_NONE with escapechar and
skipinitialspace.
Assisted-by: Codex:gpt-5.6-terra
* csv: deduplicate reader dialect configuration
to_reader configured ReaderBuilder separately for named, object, and default
dialects even though the configuration was identical.
Resolve the PyDialect once, then apply delimiter, double_quote, escapechar,
and quotechar in one place. Preserve the existing unregistered-name fallback
and required excel dialect lookup behavior.
Assisted-by: Codex:gpt-5.6-terra
* Prevent csv QUOTE_NONE writer panics without quotechar
Handle csv.writer rows with QUOTE_NONE through a RustPython-owned unquoted writer path so quotechar=None no longer reaches the unfinished csv_core configuration branch. Escape delimiter, newline, quotechar, and escapechar bytes according to the active dialect, and preserve CPython's single-empty-field error behavior.
Constraint: Match CPython csv.writer behavior without changing Lib/ copied stdlib files.
Rejected: Relying on csv_core QuoteStyle::Never | it does not model CPython QUOTE_NONE escaping with quotechar=None.
Confidence: high
Scope-risk: narrow
Directive: Keep QUOTE_NONE writer behavior separate unless csv_core gains matching CPython semantics.
Tested: prek run --all-files; cargo test --workspace --exclude rustpython_wasm --exclude rustpython-venvlauncher; cargo build --release --features sqlite; pytest -v in extra_tests
Assisted-by: Codex:gpt-5.5
* Keep csv parity tests aligned with QUOTE_NONE support
Remove the stale RustPython expected-failure marker for the CPython escaped-field writer test that now passes, and cover CR/LF escaping in the snippet regression requested during review.
Constraint: RustPython test policy allows removing expectedFailure markers only when the upstream test passes.
Rejected: Refactoring writer row boilerplate in this follow-up | review marked it as a heavy-lift nitpick and it would broaden the CI fix.
Confidence: high
Scope-risk: narrow
Tested: cargo run --release --features sqlite -- -m test test_csv; cargo run -- extra_tests/snippets/stdlib_csv.py; PATH=/tmp/pyshim:$PATH prek run --all-files; git diff --check; cargo test --workspace --exclude rustpython_wasm --exclude rustpython-venvlauncher; pytest -v in extra_tests
Assisted-by: Codex:gpt-5.5
* Keep csv quote-style dispatch coherent
Review feedback pointed out that adjacent branches selected behavior from the same quoting discriminator. Use one match so future quote-style additions are routed in a single place.
Constraint: RustPython review requested merging adjacent quote-style checks.
Confidence: high
Scope-risk: narrow
Tested: cargo fmt --check
Tested: target/release/rustpython extra_tests/snippets/stdlib_csv.py
Tested: target/release/rustpython -m test test_csv
Tested: PATH=/tmp/pyshim:$PATH prek run --all-files
Tested: RUST_TEST_THREADS=1 cargo test --workspace --exclude rustpython_wasm --exclude rustpython-venvlauncher -- --test-threads=1
Tested: cargo clippy --workspace --exclude rustpython_wasm --exclude rustpython-venvlauncher --all-targets
Assisted-by: Codex:gpt-5.5
* Use `ast.unparse` for decorator generation and every ut_method
* Ensure ut_method type for external patches
* use textwrap
* Apply patches to `test_os.py`
* Apoly on `test_xml_etree.py`
* Run on some test files
* Update `test_str.py`
* Update `test_logging.py` from 3.13.7
Shahar Naveh