2 Commits

Include renames

Author	SHA1	Message	Date
Jeong, YunWon	797d931f7a	Complete rustpython-unicode isolation: case mapping, casing predicates, sre parity (#8237) ... * Move str casing into rustpython-unicode::case and drop icu from vm Add code-point simple mappings (simple_lowercase/uppercase/titlecase/fold), casing predicates (is_lowercase/is_uppercase/is_titlecase/is_cased/ is_case_ignorable), and the string-level capitalize/title/swapcase/casefold helpers to crates/unicode/src/case.rs, moving the titlecase segmentation and final-sigma logic out of vm/builtins/str.rs verbatim. str.rs and anystr.rs now call through the crate; the is_cased kernel takes plain fn(char)->bool predicates instead of icu BinaryProperty generics. Remove icu_casemap/icu_locale/icu_properties/writeable from crates/vm and the now-unused VecFmtWriter helper. crates/unicode is the only workspace member with a direct icu dependency. str.lower/upper keep using Wtf8::to_lowercase/ to_uppercase. Assisted-by: Claude * Route sre is_uni_space through unicode::classify::is_space SRE_UNI_IS_SPACE is Py_UNICODE_ISSPACE. Replace the hand-rolled BMP code-point list with classify::is_space, which is differential-tested against CPython. A full-range sweep confirms the two agree on every code point. Assisted-by: Claude * Use simple case mappings and the Cased property for sre IGNORECASE lower_unicode/upper_unicode took the first char of the full case mapping, so code points with a full mapping but no simple one were miscased (e.g. upper_unicode('ß') returned 'S'). Route them through case::simple_lowercase/ simple_uppercase, matching Py_UNICODE_TOLOWER/TOUPPER. _sre.unicode_iscased derived casedness from those mappings, which only held while the full mapping was used; with simple mappings a cased code point that maps to itself (e.g. the ſt/st ligatures, U+FB05/06) read as uncased and lost its _casefix equivalence. Query the Cased property directly via case::is_cased. Assisted-by: Claude * Sweep casing predicates and simple lowercase mapping against CPython Extend the differential harness to cover is_lowercase/is_uppercase/is_titlecase/ is_cased over the full scalar range, sourced from str.islower/isupper, the Lt category, and _sre.unicode_iscased. Add a parallel sweep of the simple lowercase mapping (Py_UNICODE_TOLOWER via _sre.unicode_tolower) with its own version-skew allow-list; CPython exposes no simple-uppercase oracle, so toupper stays on the SRE unit tests. Record the U+0295 Ll->Lo recategorization (Unicode 16.0.0 -> 17.0.0) as a known reverse-direction divergence so the skew regenerator still rejects genuine regressions. Assisted-by: Claude	2026-07-08 19:18:44 +09:00
Jeong, YunWon	55c012cffd	unicode crate (#8211) ... * Extract unicodedata core into rustpython-unicode crate Move the vm-free Unicode character database access — the generated UCD 3.2.0 / latest tables, their build-time generation, and the icu4x/unicode_names2 lookups for category/bidirectional/combining/east_asian_width/mirrored/decomposition/ normalize/is_normalized/digit/decimal/numeric/name/lookup — into a new leaf crate over char/CodePoint/&Wtf8. stdlib/unicodedata.rs keeps the UCD pyclass binding: it extracts the code point, boxes results, and maps errors. The unicode data files, build.rs table generation, and the icu_properties/icu_normalizer/unicode_names2 dependencies move out of stdlib. Assisted-by: Claude * Route str predicates, casefold, and printable through rustpython-unicode Add classify/case/identifier modules to the shared crate over char/&str/&Wtf8: isalpha/isalnum/isdecimal/isdigit/isnumeric/isspace/isprintable classification, XID identifier predicates, and full-mapping casefold. vm/builtins/str.rs and literal/char.rs now call these instead of icu_properties/icu_casemap directly. String-level iteration with final-sigma handling (lower/upper/title/capitalize, islower/isupper) stays in the runtime. Assisted-by: Claude * Route sre_engine char classes through rustpython-unicode Add the regex module with the SRE character-class and case predicates (is_word/is_space/is_digit, is_uni_* Unicode variants, ascii/locale/unicode case folding). sre_engine/string.rs now forwards to it, keeping its public API and behavior identical — including the ASCII-only is_uni_digit and the hardcoded is_uni_space table. Assisted-by: Claude * Route \N{} name lookups through rustpython-unicode codegen/string_parser.rs (\N{...} escapes) and common/encodings.rs (the namereplace error handler) now resolve character names via rustpython-unicode instead of depending on unicode_names2 directly, leaving the crate as the sole owner of the name database. Assisted-by: Claude * Add differential Unicode sweep and shared-crate snippet test tests/differential.rs sweeps the full 0..0x110000 range and compares each str classification predicate against a committed CPython 3.14 reference dataset (tests/data/cpython3.14_predicates.txt, produced by generate_reference.py). Code points that differ only because the Rust std / icu4x build ships a later Unicode release than CPython 3.14's 16.0.0 are recorded in tests/data/version_skew_cpython3.14.txt, regenerable via RUSTPYTHON_UNICODE_REGEN_SKEW=1; the regen refuses to record any cpython=true/crate=false divergence, so only newly-assigned code points are allowed. Any other divergence fails. Both data files use a run-length `predicate start:end,...` encoding. extra_tests/snippets/stdlib_unicode_shared.py exercises the routed surface end to end (str predicates, casefold, identifiers, unicodedata, normalize, \N{}, and re character classes) and passes identically on CPython 3.14 and RustPython. Assisted-by: Claude * Match regex \d to Unicode decimal digits is_uni_digit previously matched only ASCII 0-9, so re's \d in Unicode mode missed decimal digits like ٥ and ५. SRE_UNI_IS_DIGIT matches Py_UNICODE_ISDECIMAL (category Nd), so route it through classify::is_decimal. Unmasks test_bug_6561 in test_re and extends the shared-crate snippet with the Nd/Nl/No cases. Assisted-by: Claude * Replace shallow Unicode wrappers with re-exports Address review feedback on the extraction. Functions that only forwarded to the shared crate become `use ... as` re-exports rather than hand-written wrappers: - literal: is_printable is dropped; escape.rs calls rustpython_unicode::classify::is_repr_printable directly. - unicode::identifier::is_continue re-exports is_xid_continue. - unicode::data::lookup_character re-exports unicode_names2::character. - Drop the unused repr(u8) on DecompositionType. The SRE character-class and case predicates move back into sre_engine::string (their pre-extraction home) instead of living in a unicode::regex module that only sre_engine and the vm _sre binding used; is_uni_digit/is_uni_alnum still delegate to rustpython_unicode::classify. engine.rs and _sre consume them from sre_engine::string again. The unicode crate's tests move from a single root tests module in lib.rs into per-module test submodules (case, classify, data, identifier, normalize). Assisted-by: Claude * no-std * Address review feedback: test module, dead variant, WTF-8 normalization - differential.rs: move the sweep tests into a `mod tests` block and drop the file-level allow of clippy::tests_outside_test_module and std_instead_of_alloc; the test uses alloc collections instead. - data.rs: remove the never-constructed DecompositionType::Canonical variant and its allow(unused); compatibility decomposition never produces it and canonical decomposition is handled through icu4x. - normalize.rs: is_normalized now takes &Wtf8 and checks each UTF-8 run, skipping lone surrogates, matching normalize's run-wise behavior. - unicodedata.rs: pass as_wtf8() to is_normalized. Assisted-by: Claude	2026-07-07 09:19:25 +09:00