to_string's normal-magnitude branch and complex::component_to_string
returned Rust's shortest formatting directly, which can pick the odd-digit
neighbour of a rounding tie where repr() picks the even one (e.g. bits
0x42e26687db6b9b04 formats as 161852602146008.13 vs repr's ...08.12).
Route both through prefer_cpython_tie_repr, generalized to fixed-notation
strings. Add unit tests and float/complex repr snippet tests.
Assisted-by: Claude
parse_str split the token then parsed each part with float::parse_str,
which tolerates surrounding whitespace, so "1 +2j" parsed as (1+2j).
Reject whitespace inside the token after stripping optional parentheses.
Add unit tests and complex() snippet tests.
Assisted-by: Claude
* Extract unicodedata core into rustpython-unicode crate
Move the vm-free Unicode character database access — the generated UCD 3.2.0 /
latest tables, their build-time generation, and the icu4x/unicode_names2 lookups
for category/bidirectional/combining/east_asian_width/mirrored/decomposition/
normalize/is_normalized/digit/decimal/numeric/name/lookup — into a new leaf crate
over char/CodePoint/&Wtf8. stdlib/unicodedata.rs keeps the UCD pyclass binding: it
extracts the code point, boxes results, and maps errors. The unicode data files,
build.rs table generation, and the icu_properties/icu_normalizer/unicode_names2
dependencies move out of stdlib.
Assisted-by: Claude
* Route str predicates, casefold, and printable through rustpython-unicode
Add classify/case/identifier modules to the shared crate over char/&str/&Wtf8:
isalpha/isalnum/isdecimal/isdigit/isnumeric/isspace/isprintable classification,
XID identifier predicates, and full-mapping casefold. vm/builtins/str.rs and
literal/char.rs now call these instead of icu_properties/icu_casemap directly.
String-level iteration with final-sigma handling (lower/upper/title/capitalize,
islower/isupper) stays in the runtime.
Assisted-by: Claude
* Route sre_engine char classes through rustpython-unicode
Add the regex module with the SRE character-class and case predicates
(is_word/is_space/is_digit, is_uni_* Unicode variants, ascii/locale/unicode case
folding). sre_engine/string.rs now forwards to it, keeping its public API and
behavior identical — including the ASCII-only is_uni_digit and the hardcoded
is_uni_space table.
Assisted-by: Claude
* Route \N{} name lookups through rustpython-unicode
codegen/string_parser.rs (\N{...} escapes) and common/encodings.rs (the
namereplace error handler) now resolve character names via rustpython-unicode
instead of depending on unicode_names2 directly, leaving the crate as the sole
owner of the name database.
Assisted-by: Claude
* Add differential Unicode sweep and shared-crate snippet test
tests/differential.rs sweeps the full 0..0x110000 range and compares each str
classification predicate against a committed CPython 3.14 reference dataset
(tests/data/cpython3.14_predicates.txt, produced by generate_reference.py). Code
points that differ only because the Rust std / icu4x build ships a later Unicode
release than CPython 3.14's 16.0.0 are recorded in
tests/data/version_skew_cpython3.14.txt, regenerable via
RUSTPYTHON_UNICODE_REGEN_SKEW=1; the regen refuses to record any
cpython=true/crate=false divergence, so only newly-assigned code points are
allowed. Any other divergence fails. Both data files use a run-length
`predicate start:end,...` encoding.
extra_tests/snippets/stdlib_unicode_shared.py exercises the routed surface end to
end (str predicates, casefold, identifiers, unicodedata, normalize, \N{}, and re
character classes) and passes identically on CPython 3.14 and RustPython.
Assisted-by: Claude
* Match regex \d to Unicode decimal digits
is_uni_digit previously matched only ASCII 0-9, so re's \d in Unicode mode
missed decimal digits like ٥ and ५. SRE_UNI_IS_DIGIT matches Py_UNICODE_ISDECIMAL
(category Nd), so route it through classify::is_decimal. Unmasks test_bug_6561 in
test_re and extends the shared-crate snippet with the Nd/Nl/No cases.
Assisted-by: Claude
* Replace shallow Unicode wrappers with re-exports
Address review feedback on the extraction.
Functions that only forwarded to the shared crate become `use ... as`
re-exports rather than hand-written wrappers:
- literal: is_printable is dropped; escape.rs calls
rustpython_unicode::classify::is_repr_printable directly.
- unicode::identifier::is_continue re-exports is_xid_continue.
- unicode::data::lookup_character re-exports unicode_names2::character.
- Drop the unused repr(u8) on DecompositionType.
The SRE character-class and case predicates move back into
sre_engine::string (their pre-extraction home) instead of living in a
unicode::regex module that only sre_engine and the vm _sre binding used;
is_uni_digit/is_uni_alnum still delegate to rustpython_unicode::classify.
engine.rs and _sre consume them from sre_engine::string again.
The unicode crate's tests move from a single root tests module in lib.rs
into per-module test submodules (case, classify, data, identifier,
normalize).
Assisted-by: Claude
* no-std
* Address review feedback: test module, dead variant, WTF-8 normalization
- differential.rs: move the sweep tests into a `mod tests` block and drop
the file-level allow of clippy::tests_outside_test_module and
std_instead_of_alloc; the test uses alloc collections instead.
- data.rs: remove the never-constructed DecompositionType::Canonical variant
and its allow(unused); compatibility decomposition never produces it and
canonical decomposition is handled through icu4x.
- normalize.rs: is_normalized now takes &Wtf8 and checks each UTF-8 run,
skipping lone surrogates, matching normalize's run-wise behavior.
- unicodedata.rs: pass as_wtf8() to is_normalized.
Assisted-by: Claude
Redo of #7800.
Bumping the random and crypto crates is not trivial. Both sets of crates
have had API changes which require manual updates.
AI use:
I wrote all of the code myself but used AI to figure out the new APIs.
AI explained SHAKE's block size and why the block_size() function
doesn't exist in the new crate. I confirmed the block size, which is
actually represented in the new types for SHAKE. `Shake128` is actually
`Shake<168>` - 168 is the block size. I cited Wikipedia as it lists the
SHAKE block size in bits.
I also used AI to figure out HMAC for SHA3.
Assisted-by: Codex:gpt-5.4
* Fix process abort on large float format precision
Formatting a float with large precision (>= ~65535) aborted the
interpreter instead of raising a Python exception. CPython handles
the same input by returning a clean string.
# Before
./rustpython -c "print(f'{1.5:.1000000}')"
thread 'main' panicked at crates/literal/src/float.rs:135:
Formatting argument out of range (exit 101, abort)
# After
./rustpython -c "print(f'{1.5:.1000000}')"
1.5
Root cause: Rust's `format!("{:.*}", n, x)` panics when `n`
exceeds the fmt runtime's internal precision limit. `format_fixed`
already caps `n` at u16::MAX, but `format_general` and
`format_exponent` (and the `%` branch in `crates/common/src/format.rs`)
passed user-supplied precision straight through to `format!`.
Fix:
* Introduce `FMT_MAX_PRECISION` + `clamp_fmt_precision()` in
crates/literal/src/float.rs. Cap is `u16::MAX - 1` because
`{:.*e}` hits a second panic (`ndigits > 0` in core flt2dec)
at exactly u16::MAX; the smaller value covers both paths.
* Apply the helper to `format_fixed` (replacing the existing
ad-hoc cap), `format_exponent` (entry), and `format_general`
(three separate format! calls with saturating arithmetic on
derived precision values).
* Apply the helper in the `FormatType::Percentage` branch in
crates/common/src/format.rs.
This is harmless for all normal inputs — f64 carries only ~17
significant digits, so precision beyond 65K is padding zeros at
best. Complex-number and old-style `%`-formatting paths transitively
benefit because they dispatch to the same library functions.
Verified:
* cargo run -- -m test test_float test_fstring test_format:
144 passed, 0 regressed.
* extra_tests/snippets/builtin_format.py: all assertions pass,
including 7 new regression cases covering e / E / g / G / f /
% at precision 1_000_000.
* Probed with 10 magnitude values (0, ±1.5, ±inf, nan, 1e-300,
1e300, f64::MAX, 5e-324) x 4 format types = 40 combinations,
plus precision 0/1/2 boundary, complex formatting, old-style
`%` formatting, and combined specs (fill/align/sign/grouping/
zero-pad). All return clean strings; no process abort.
* Address CodeRabbit review: split cap + drop redundant clamp
Two refinements after CodeRabbit review:
1. Drop the redundant `format!("{:.*}", precision + 1, base)` in
`format_general`'s scientific branch. It was a no-op pre-fix
(magnitude is `.abs()`-ed at the caller, so `base` has no sign
and its length was exactly `precision + 1`), but after I added
the cap it turned into an active truncate — dropping 1 char of
precision at the cap boundary. Reuse `base` directly and extract
`exp_precision` for reuse by `decimal_point_or_empty`.
2. Split the cap into two helpers.
`FMT_MAX_PRECISION = u16::MAX` — for plain `{:.*}` (format_fixed,
%-branch, format_general's
non-scientific branch).
`FMT_MAX_EXP_PRECISION = u16::MAX - 1` — for `{:.*e}` (format_exponent,
format_general's scientific
entry).
The second value is one lower because `{:.*e}` trips an additional
`ndigits > 0` assertion in `core::num::flt2dec` at exactly
`u16::MAX`. The first commit used the tighter cap uniformly,
which silently regressed `format_fixed` by 1 char at
`precision == u16::MAX` (it previously capped at exactly that
value). Two helpers restore byte-identical CPython parity for
fixed / percent / general-non-scientific paths up through
`precision == u16::MAX`.
Verification:
* precision 5 .. 65534: 360 outputs byte-identical to CPython
across 8 magnitudes x 9 precisions x 5 types.
* precision == 65535: f / g / G / % now match CPython (0 diff).
e / E remain 1 char shorter — unavoidable
within the `u16::MAX - 1` exp cap.
* precision > 65535: output stops at cap; CPython emits full
padding — same design divergence as before.
* No panic regression: f-string default, e/E, g/G, %, f at
precision 1_000_000 all return cleanly.
* Test suite: test_float + test_fstring + test_format,
162 passed, 0 regressed.
* Fix ruff format: single-line precision clamp
* Address @youknowone review: byte-identical CPython parity at boundary
Per review comment on `extra_tests/snippets/builtin_format.py:209`:
the patch declares `FMT_MAX_PRECISION = u16::MAX`, so the tests must
cover 65535 and 65536 and demonstrate CPython parity at the boundary.
The previous version only avoided panic — at the cap it silently
truncated 1 char short of CPython for e / E, and thousands of chars
short for f / % at precision beyond the cap. This commit restores
byte-identical CPython output at every precision up to the format-
spec parser's own `i32::MAX` ceiling.
Fix: pad the Rust-format result with '0's up to the user-requested
precision.
Why this is correct, not a workaround: IEEE 754 double has at most
~767 significant decimal digits; past that, every digit is
deterministically '0' in both CPython and the native Rust output.
Our cap (65534 for exp, 65535 for plain) sits far above 767, so
appending zeros reconstructs precisely what CPython would have
produced. Verified on hard inputs: `1e-100`, `5e-324` (subnormal
boundary), `f64::MAX`, mixed magnitudes — the last 100 chars of
Rust-format output at precision 65534 are all '0' for every case.
Changes:
* `format_fixed`: after format!(), extend with (precision - capped)
'0' chars before appending the optional decimal point.
* `format_exponent`: same, applied to the parsed mantissa before
reassembling with the exponent marker.
* `FormatType::Percentage` branch: same. Also fixed a bug the
boundary audit surfaced: the finite-input overflow guard used
`return Ok("inf%")`, which bypasses the outer sign handler.
Changed to a match-arm value so `format_sign_and_align` still
runs and produces "-inf%" for `-f64::MAX`, matching CPython.
Verification:
* 7 magnitudes × 5 precisions × 6 format types = 210 comparisons
against CPython at precisions {65534, 65535, 65536, 100000,
200000}. All 210 byte-identical.
* Gap audit (complex formatting, old-style % formatting, negative
magnitudes, -0.0, combined specs with fill / sign / alternate /
grouping) at boundary precisions. All but 20 byte-identical.
The 20 remaining diffs all stem from a pre-existing
complex-imaginary-part repr bug (`1e100j` expands to 100 '0's
in RustPython vs CPython's `1e+100j`) which reproduces on
upstream main without any part of this patch and is out of
scope here.
* `cargo run -- -m test test_float test_fstring test_format`:
162 passed, 0 regressed.
* `extra_tests/snippets/builtin_format.py` now pins exact
expected strings at 65534 / 65535 / 65536 / 1_000_000 for
every format type, plus the `f64::MAX × 100 → 'inf%'`
overflow case.
* `cargo fmt --check`: pass.
* Clarify boundary test labels + add past-cap depth assertions
Rename the boundary-test section so the three precision points per
format type are labeled below / at / past the cap inline, making the
"past MAX_PRECISION" unhappy-case coverage explicit. Add len-based
assertions at precision 1_000_000 for f, e, and % to exercise the
cap-then-pad path at a depth far beyond the boundary.
* Fix complex repr to use scientific notation for large integer-valued components
repr of a complex number whose real or imaginary part is an integer-valued
float with |x| >= 1e16 emitted the full decimal expansion instead of
scientific notation, diverging from CPython:
Before (RustPython):
repr(1e100 + 1e100j)
(10000000000000000000000000000000000000000000000000000000000
0000000000000000000000000000000000000000000+1000000000000000
000000000000000000000000000000000000000000000000000000000000
00000000000000000000000000000000000000j)
After / CPython:
(1e+100+1e+100j)
Root cause in crates/literal/src/complex.rs::to_string — it bifurcated
each component by .fract() == 0.0:
if im.fract() == 0.0 { im.to_string() } // Rust's default Display
else { float::to_string(im) } // scientific for large/small
Rust's Display never uses scientific notation, so any integer-valued f64
(including 1e16, 1e17, 1e100 which are exactly representable as integers)
routed through the wrong branch and produced the full decimal expansion.
Non-integer magnitudes reached float::to_string and rendered correctly.
The fix is to use one helper per component that implements CPython's
actual PyOS_double_to_string(format='r') rule: scientific notation when
|x| < 1e-4 or |x| >= 1e16, otherwise Rust's default Display (which drops
the trailing '.0' for integer-valued floats — matching CPython's
(1+2j) convention rather than (1.0+2.0j)). The threshold matches
float::to_string; the only behavioral difference is that complex
components render 1.0 as "1" rather than "1.0".
Verified:
* 29 CPython reference cases (normal / boundary / extremes / special /
signed-zero) — all byte-identical after fix.
* 18 additional edge cases (subnormal 5e-324, f64::MAX, MIN_POSITIVE,
DBL_EPSILON, threshold-straddling values) — all byte-identical.
* Lib/test/test_complex.py::test_repr_str /
test_negative_zero_repr_str / test_repr_roundtrip — all pass.
* cargo run -- -m test test_complex — 37 passed.
* cargo run -- -m test test_float test_long — 101 passed.
* ast.unparse() round-trip of source containing complex literals
(e.g. 1e100 + 1e-100j, 1e17 + 1j) produces CPython-identical output.
* extra_tests/snippets/builtin_complex.py — 20+ new regression cases.
* Address CodeRabbit review: clarify threshold boundary test comment
The comment claimed all three assertions stay in non-scientific form,
but the 1e-5 case explicitly verifies scientific notation (since
|1e-5| < 1e-4 falls outside the decimal-form range). Reworded the
header to describe the axis being tested (threshold boundary) and
added per-case inline notes indicating each assertion's expected
form.
* * Added alloc_instead_of_core, std_instead_of_alloc, and std_instead_of_core clippy rules
* Manually changed part of the code to use core/alloc
* use clippy --fix to fix issues in stdlib
* * Used clippy --fix to fix issues in vm
* Imported Range in vm/src/anystr.rs
* * Used clippy --fix to fix issues in common
Joshua Megnauth