* common: fix decompose_float for subnormal inputs
Subnormals carry a biased exponent of 0 and no implicit leading mantissa
bit, so the bit-twiddling frexp misread them: hash(5e-324) returned
8404992 instead of CPython's 16777216. Scale subnormals into the normal
range by 2**54 (exact) before decomposing and fold the shift back into
the returned exponent. Adds a frexp-contract unit test over normals,
subnormals, and boundary values.
Assisted-by: Claude
* common: add HashSecret::from_keys
HashSecret's fields are private and the only constructor derives k0/k1
from a u32 seed, so an embedder cannot reproduce a fixed keying (e.g. a
deterministic PYTHONHASHSEED=0-style run). Add a const constructor
taking explicit SipHash keys; the seeded path is unchanged.
Assisted-by: Claude
* common: fix float format grouping and empty presentation type
Thousands grouping split the magnitude only on '.', so exponent and
percent tails received separators (format(1e20, ',e') gave
'1e+,20'-class corruption). Group only the leading integer digits for
interval-3 output; hex/octal/binary keep whole-magnitude grouping.
Empty presentation type: with a precision, clamp it to at least 1 and
restore a trailing '.0' on integer-looking results (Py_DTSF_ADD_DOT_0);
without one, alternate form now inserts the forced decimal point before
an exponent instead of appending it.
Verified byte-identical to CPython 3.14 over a hash/format differential
battery; test_format, test_float, test_fstring, test_hash pass.
Assisted-by: Claude
* common: satisfy rustfmt and clippy in new hash/float tests
- float_ops: use core::f64::consts::PI in the frexp-contract battery
(clippy::approx_constant)
- hash: rewrite the keyed_hash test via BuildHasher::hash_one
(clippy::manual_hash_one)
- rustfmt the new float_ops/hash/format test modules
Assisted-by: Claude
* common: add DTSF to format.rs spell-checker ignores
Assisted-by: Claude
* Extract unicodedata core into rustpython-unicode crate
Move the vm-free Unicode character database access — the generated UCD 3.2.0 /
latest tables, their build-time generation, and the icu4x/unicode_names2 lookups
for category/bidirectional/combining/east_asian_width/mirrored/decomposition/
normalize/is_normalized/digit/decimal/numeric/name/lookup — into a new leaf crate
over char/CodePoint/&Wtf8. stdlib/unicodedata.rs keeps the UCD pyclass binding: it
extracts the code point, boxes results, and maps errors. The unicode data files,
build.rs table generation, and the icu_properties/icu_normalizer/unicode_names2
dependencies move out of stdlib.
Assisted-by: Claude
* Route str predicates, casefold, and printable through rustpython-unicode
Add classify/case/identifier modules to the shared crate over char/&str/&Wtf8:
isalpha/isalnum/isdecimal/isdigit/isnumeric/isspace/isprintable classification,
XID identifier predicates, and full-mapping casefold. vm/builtins/str.rs and
literal/char.rs now call these instead of icu_properties/icu_casemap directly.
String-level iteration with final-sigma handling (lower/upper/title/capitalize,
islower/isupper) stays in the runtime.
Assisted-by: Claude
* Route sre_engine char classes through rustpython-unicode
Add the regex module with the SRE character-class and case predicates
(is_word/is_space/is_digit, is_uni_* Unicode variants, ascii/locale/unicode case
folding). sre_engine/string.rs now forwards to it, keeping its public API and
behavior identical — including the ASCII-only is_uni_digit and the hardcoded
is_uni_space table.
Assisted-by: Claude
* Route \N{} name lookups through rustpython-unicode
codegen/string_parser.rs (\N{...} escapes) and common/encodings.rs (the
namereplace error handler) now resolve character names via rustpython-unicode
instead of depending on unicode_names2 directly, leaving the crate as the sole
owner of the name database.
Assisted-by: Claude
* Add differential Unicode sweep and shared-crate snippet test
tests/differential.rs sweeps the full 0..0x110000 range and compares each str
classification predicate against a committed CPython 3.14 reference dataset
(tests/data/cpython3.14_predicates.txt, produced by generate_reference.py). Code
points that differ only because the Rust std / icu4x build ships a later Unicode
release than CPython 3.14's 16.0.0 are recorded in
tests/data/version_skew_cpython3.14.txt, regenerable via
RUSTPYTHON_UNICODE_REGEN_SKEW=1; the regen refuses to record any
cpython=true/crate=false divergence, so only newly-assigned code points are
allowed. Any other divergence fails. Both data files use a run-length
`predicate start:end,...` encoding.
extra_tests/snippets/stdlib_unicode_shared.py exercises the routed surface end to
end (str predicates, casefold, identifiers, unicodedata, normalize, \N{}, and re
character classes) and passes identically on CPython 3.14 and RustPython.
Assisted-by: Claude
* Match regex \d to Unicode decimal digits
is_uni_digit previously matched only ASCII 0-9, so re's \d in Unicode mode
missed decimal digits like ٥ and ५. SRE_UNI_IS_DIGIT matches Py_UNICODE_ISDECIMAL
(category Nd), so route it through classify::is_decimal. Unmasks test_bug_6561 in
test_re and extends the shared-crate snippet with the Nd/Nl/No cases.
Assisted-by: Claude
* Replace shallow Unicode wrappers with re-exports
Address review feedback on the extraction.
Functions that only forwarded to the shared crate become `use ... as`
re-exports rather than hand-written wrappers:
- literal: is_printable is dropped; escape.rs calls
rustpython_unicode::classify::is_repr_printable directly.
- unicode::identifier::is_continue re-exports is_xid_continue.
- unicode::data::lookup_character re-exports unicode_names2::character.
- Drop the unused repr(u8) on DecompositionType.
The SRE character-class and case predicates move back into
sre_engine::string (their pre-extraction home) instead of living in a
unicode::regex module that only sre_engine and the vm _sre binding used;
is_uni_digit/is_uni_alnum still delegate to rustpython_unicode::classify.
engine.rs and _sre consume them from sre_engine::string again.
The unicode crate's tests move from a single root tests module in lib.rs
into per-module test submodules (case, classify, data, identifier,
normalize).
Assisted-by: Claude
* no-std
* Address review feedback: test module, dead variant, WTF-8 normalization
- differential.rs: move the sweep tests into a `mod tests` block and drop
the file-level allow of clippy::tests_outside_test_module and
std_instead_of_alloc; the test uses alloc collections instead.
- data.rs: remove the never-constructed DecompositionType::Canonical variant
and its allow(unused); compatibility decomposition never produces it and
canonical decomposition is handled through icu4x.
- normalize.rs: is_normalized now takes &Wtf8 and checks each UTF-8 run,
skipping lone surrogates, matching normalize's run-wise behavior.
- unicodedata.rs: pass as_wtf8() to is_normalized.
Assisted-by: Claude
* Re-export mmap libc constants through host_env
Move direct libc MADV_*, MAP_*, PROT_*, EOVERFLOW references in
stdlib::mmap to host_env::mmap re-exports. Replace libc::c_int with
core::ffi::c_int.
* host_env/mmap: gate EOVERFLOW re-export on cfg(windows)
Its only consumer is the cfg(windows) named-mapping overflow check in
stdlib::mmap; the re-export was gated cfg(unix), so it was configured
out on Windows and the reference failed to resolve.
Assisted-by: Claude
* Move tuple/frozenset hash algorithms into common::hash
Move the pure tuplehash fold (xxHash-based) into `hash::hash_tuple`, taking
a fallible iterator of element hashes, and the frozenset XOR-fold into a
`hash::FrozenSetHash` streaming accumulator. `builtins::tuple` and
`builtins::set` keep only the element-hash loop and delegate the arithmetic.
Assisted-by: Claude
* common/hash: rename acc_pyhash to acc_py_hash
Drop the now-unused `// cspell:ignore pyhash` from tuple.rs left behind
by the tuple hash move; the only remaining occurrence was the
acc_pyhash local. Renaming it lets cspell pass without an ignore.
Assisted-by: Claude
* Prevent csv QUOTE_NONE writer panics without quotechar
Handle csv.writer rows with QUOTE_NONE through a RustPython-owned unquoted writer path so quotechar=None no longer reaches the unfinished csv_core configuration branch. Escape delimiter, newline, quotechar, and escapechar bytes according to the active dialect, and preserve CPython's single-empty-field error behavior.
Constraint: Match CPython csv.writer behavior without changing Lib/ copied stdlib files.
Rejected: Relying on csv_core QuoteStyle::Never | it does not model CPython QUOTE_NONE escaping with quotechar=None.
Confidence: high
Scope-risk: narrow
Directive: Keep QUOTE_NONE writer behavior separate unless csv_core gains matching CPython semantics.
Tested: prek run --all-files; cargo test --workspace --exclude rustpython_wasm --exclude rustpython-venvlauncher; cargo build --release --features sqlite; pytest -v in extra_tests
Assisted-by: Codex:gpt-5.5
* Keep csv parity tests aligned with QUOTE_NONE support
Remove the stale RustPython expected-failure marker for the CPython escaped-field writer test that now passes, and cover CR/LF escaping in the snippet regression requested during review.
Constraint: RustPython test policy allows removing expectedFailure markers only when the upstream test passes.
Rejected: Refactoring writer row boilerplate in this follow-up | review marked it as a heavy-lift nitpick and it would broaden the CI fix.
Confidence: high
Scope-risk: narrow
Tested: cargo run --release --features sqlite -- -m test test_csv; cargo run -- extra_tests/snippets/stdlib_csv.py; PATH=/tmp/pyshim:$PATH prek run --all-files; git diff --check; cargo test --workspace --exclude rustpython_wasm --exclude rustpython-venvlauncher; pytest -v in extra_tests
Assisted-by: Codex:gpt-5.5
* Keep csv quote-style dispatch coherent
Review feedback pointed out that adjacent branches selected behavior from the same quoting discriminator. Use one match so future quote-style additions are routed in a single place.
Constraint: RustPython review requested merging adjacent quote-style checks.
Confidence: high
Scope-risk: narrow
Tested: cargo fmt --check
Tested: target/release/rustpython extra_tests/snippets/stdlib_csv.py
Tested: target/release/rustpython -m test test_csv
Tested: PATH=/tmp/pyshim:$PATH prek run --all-files
Tested: RUST_TEST_THREADS=1 cargo test --workspace --exclude rustpython_wasm --exclude rustpython-venvlauncher -- --test-threads=1
Tested: cargo clippy --workspace --exclude rustpython_wasm --exclude rustpython-venvlauncher --all-targets
Assisted-by: Codex:gpt-5.5
Python supports `mkdir` on Windows. I deferred to calling Rust's
implementation for now. However, like `rename`, Python's implementation
supports additional features that are currently unsupported on
RustPython. I added a note for future reference.
Redox supports `mkdirat` which significantly cleans up the
implementation.
Enable `abi3t` support in PyO3
Add left over c-api functions
Fix PyLong_AsUnsignedLongLongMask
Update pyo3 to `0.29`
Add `PyInt::as_u64_mask`
Fix `PyInt::as_u32_mask`
* Align compiler and AST behavior with CPython
Update the compiler pipeline, AST module support, and related VM compile/eval paths toward CPython 3.14 behavior.
Use upstream Ruff parser crates directly and keep RustPython-specific syntax preflight handling in the compile path.
Refresh opcode metadata, snapshots, and targeted tests for the aligned bytecode and AST behavior.
* fix
* Fix PR 8138 CI failures
* Fix match mapping wildcard scan complexity
* fixes
* Use Ruff runtime AST fields
* Minimize residual patch diff
* refactor error handling
* Remove _ast conversion context wrappers
* Clean up _ast conversion glue and compiler warning escalation
Rename the Node conversion parameter `ctx` to `vm` throughout the _ast
module.
Remove the unused compile_program, compile_program_single,
compile_block_expression, and compile_expression forwarders; compile_top
stays as the entry point.
Stash an escalated compiler SyntaxWarning so a non-SyntaxWarning category
propagates unchanged instead of being rewritten to SyntaxError, matching
PyErr_ExceptionMatches(SyntaxWarning) in compiler_warn. Drop the now-unused
CompileWarningError::into_codegen_error.
Replace the PositionalArguments two-variant enum with a struct holding a
shared range and a PositionalArgumentsKind enum.
Assisted-by: Claude
* Represent compiler flags as a CompilerFlags bitflags
Replace the loose PY_CF_*/CO_* i32 constants in compile_mode with a
CompilerFlags bitflags covering the PyCF_* flags and __future__
CO_FUTURE_* bits. The Python-visible ast.PyCF_* attribute values are
derived from it via bits(); parser input modes stay plain consts.
builtins::compile and the compile_string flag handling test flags
through CompilerFlags::contains instead of raw bit math.
Assisted-by: Claude
* Fix PR 8138 test regressions
* Fix PR 8138 CI failures
* Avoid shell interpolation in Linux deps action
* Fix review findings in compiler/AST changes
- frame.rs: compute the MATCH_CLASS class name lazily so a successful
match no longer takes the type name lock or allocates; the
__match_args__-not-a-tuple branch reuses it instead of recomputing.
- _ast/constant.rs: add the missing leading space to the with_recursion
label ("maximum recursion depth exceeded during compilation").
- sys.set_int_max_str_digits: restore the "maxdigits must be 0 or larger
than {n}" error text.
- codegen: set CO_MAXBLOCKS to 20 in both ir.rs and compile.rs.
- _ast/exception.rs: extract the duplicated excepthandler node guard
into ensure_excepthandler_node.
- codegen: correct the validate_keywords doc comment.
- Drop redundant "CPython" qualifiers from comments added in this branch.
Assisted-by: Claude
Jeong, YunWon