mirror of
https://github.com/RustPython/RustPython.git
synced 2026-10-29 03:04:38 +08:00
main
3
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 youknowdot
|
64330b80b9 |
Implement Unicode 17 grapheme iteration and properties (#9145)
* Implement native Unicode 17 grapheme iteration Add the CPython extended grapheme state machine, full-codepoint properties, and traced Segment/iterator types with positional index handling. Refresh canonical unicodedata tests through update_lib while retaining pending markers. Include a losslessly range-compressed exact-final property oracle and differential test. Source-reviewed implementation; candidate runtime validation remains pending. Reference property generation completed against the pinned CPython Unicode extension. Assisted-by: OpenAI Codex:model version unavailable * Bind grapheme Segment methods to the Python object Use the required Py<Segment> receiver for native descriptors; preserve payload ownership and representation behavior. Assisted-by: OpenAI Codex:model version unavailable * Use explicit object receivers for Segment descriptors Keep the inherent impl on the local payload and receive &Py<Self>, as required for stdlib classes outside the VM crate. Assisted-by: OpenAI Codex:model version unavailable * Mark grapheme property lookups as must-use Satisfy strict Clippy for the three pure property functions without changing their behavior. Assisted-by: OpenAI Codex:model version unavailable * Enable validated Unicode grapheme canonical tests Remove seven stale expected-failure markers through update_lib metadata after native property parity, canonical methods, full Unicode 17 grapheme corpus, and reference-cycle GC validation. Assisted-by: OpenAI Codex:model version unavailable * Accept platform line endings in grapheme oracle header Compare the complete provenance line without its line ending so the same exact Unicode 17 reference is accepted after Windows checkout conversion. Keep the complete code-point property sweep unchanged. Assisted-by: OpenAI Codex:model version unavailable * Keep grapheme property regression in test module Move the existing complete Unicode 17 property regression into the existing cfg(test) module to satisfy strict Clippy. Preserve the fixture, assertions, code-point coverage, and platform-independent provenance line check unchanged. Assisted-by: OpenAI Codex:model version unavailable * Qualify grapheme iterator return type for generated bindings Assisted-by: OpenAI Codex:model version unavailable * Fix deprecated PyO3 code input import in benchmarks Use the canonical code module path introduced in PyO3 0.30 without suppressing deprecation warnings. Assisted-by: OpenAI Codex:model version unavailable |
||
|
Jeong, YunWon
|
797d931f7a |
Complete rustpython-unicode isolation: case mapping, casing predicates, sre parity (#8237)
* Move str casing into rustpython-unicode::case and drop icu from vm
Add code-point simple mappings (simple_lowercase/uppercase/titlecase/fold),
casing predicates (is_lowercase/is_uppercase/is_titlecase/is_cased/
is_case_ignorable), and the string-level capitalize/title/swapcase/casefold
helpers to crates/unicode/src/case.rs, moving the titlecase segmentation and
final-sigma logic out of vm/builtins/str.rs verbatim. str.rs and anystr.rs now
call through the crate; the is_cased kernel takes plain fn(char)->bool
predicates instead of icu BinaryProperty generics.
Remove icu_casemap/icu_locale/icu_properties/writeable from crates/vm and the
now-unused VecFmtWriter helper. crates/unicode is the only workspace member
with a direct icu dependency. str.lower/upper keep using Wtf8::to_lowercase/
to_uppercase.
Assisted-by: Claude
* Route sre is_uni_space through unicode::classify::is_space
SRE_UNI_IS_SPACE is Py_UNICODE_ISSPACE. Replace the hand-rolled BMP code-point
list with classify::is_space, which is differential-tested against CPython. A
full-range sweep confirms the two agree on every code point.
Assisted-by: Claude
* Use simple case mappings and the Cased property for sre IGNORECASE
lower_unicode/upper_unicode took the first char of the full case mapping, so
code points with a full mapping but no simple one were miscased (e.g.
upper_unicode('ß') returned 'S'). Route them through case::simple_lowercase/
simple_uppercase, matching Py_UNICODE_TOLOWER/TOUPPER.
_sre.unicode_iscased derived casedness from those mappings, which only held
while the full mapping was used; with simple mappings a cased code point that
maps to itself (e.g. the ſt/st ligatures, U+FB05/06) read as uncased and lost its
_casefix equivalence. Query the Cased property directly via case::is_cased.
Assisted-by: Claude
* Sweep casing predicates and simple lowercase mapping against CPython
Extend the differential harness to cover is_lowercase/is_uppercase/is_titlecase/
is_cased over the full scalar range, sourced from str.islower/isupper, the Lt
category, and _sre.unicode_iscased. Add a parallel sweep of the simple lowercase
mapping (Py_UNICODE_TOLOWER via _sre.unicode_tolower) with its own version-skew
allow-list; CPython exposes no simple-uppercase oracle, so toupper stays on the
SRE unit tests.
Record the U+0295 Ll->Lo recategorization (Unicode 16.0.0 -> 17.0.0) as a known
reverse-direction divergence so the skew regenerator still rejects genuine
regressions.
Assisted-by: Claude
|
||
|
Jeong, YunWon
|
55c012cffd |
unicode crate (#8211)
* Extract unicodedata core into rustpython-unicode crate
Move the vm-free Unicode character database access — the generated UCD 3.2.0 /
latest tables, their build-time generation, and the icu4x/unicode_names2 lookups
for category/bidirectional/combining/east_asian_width/mirrored/decomposition/
normalize/is_normalized/digit/decimal/numeric/name/lookup — into a new leaf crate
over char/CodePoint/&Wtf8. stdlib/unicodedata.rs keeps the UCD pyclass binding: it
extracts the code point, boxes results, and maps errors. The unicode data files,
build.rs table generation, and the icu_properties/icu_normalizer/unicode_names2
dependencies move out of stdlib.
Assisted-by: Claude
* Route str predicates, casefold, and printable through rustpython-unicode
Add classify/case/identifier modules to the shared crate over char/&str/&Wtf8:
isalpha/isalnum/isdecimal/isdigit/isnumeric/isspace/isprintable classification,
XID identifier predicates, and full-mapping casefold. vm/builtins/str.rs and
literal/char.rs now call these instead of icu_properties/icu_casemap directly.
String-level iteration with final-sigma handling (lower/upper/title/capitalize,
islower/isupper) stays in the runtime.
Assisted-by: Claude
* Route sre_engine char classes through rustpython-unicode
Add the regex module with the SRE character-class and case predicates
(is_word/is_space/is_digit, is_uni_* Unicode variants, ascii/locale/unicode case
folding). sre_engine/string.rs now forwards to it, keeping its public API and
behavior identical — including the ASCII-only is_uni_digit and the hardcoded
is_uni_space table.
Assisted-by: Claude
* Route \N{} name lookups through rustpython-unicode
codegen/string_parser.rs (\N{...} escapes) and common/encodings.rs (the
namereplace error handler) now resolve character names via rustpython-unicode
instead of depending on unicode_names2 directly, leaving the crate as the sole
owner of the name database.
Assisted-by: Claude
* Add differential Unicode sweep and shared-crate snippet test
tests/differential.rs sweeps the full 0..0x110000 range and compares each str
classification predicate against a committed CPython 3.14 reference dataset
(tests/data/cpython3.14_predicates.txt, produced by generate_reference.py). Code
points that differ only because the Rust std / icu4x build ships a later Unicode
release than CPython 3.14's 16.0.0 are recorded in
tests/data/version_skew_cpython3.14.txt, regenerable via
RUSTPYTHON_UNICODE_REGEN_SKEW=1; the regen refuses to record any
cpython=true/crate=false divergence, so only newly-assigned code points are
allowed. Any other divergence fails. Both data files use a run-length
`predicate start:end,...` encoding.
extra_tests/snippets/stdlib_unicode_shared.py exercises the routed surface end to
end (str predicates, casefold, identifiers, unicodedata, normalize, \N{}, and re
character classes) and passes identically on CPython 3.14 and RustPython.
Assisted-by: Claude
* Match regex \d to Unicode decimal digits
is_uni_digit previously matched only ASCII 0-9, so re's \d in Unicode mode
missed decimal digits like ٥ and ५. SRE_UNI_IS_DIGIT matches Py_UNICODE_ISDECIMAL
(category Nd), so route it through classify::is_decimal. Unmasks test_bug_6561 in
test_re and extends the shared-crate snippet with the Nd/Nl/No cases.
Assisted-by: Claude
* Replace shallow Unicode wrappers with re-exports
Address review feedback on the extraction.
Functions that only forwarded to the shared crate become `use ... as`
re-exports rather than hand-written wrappers:
- literal: is_printable is dropped; escape.rs calls
rustpython_unicode::classify::is_repr_printable directly.
- unicode::identifier::is_continue re-exports is_xid_continue.
- unicode::data::lookup_character re-exports unicode_names2::character.
- Drop the unused repr(u8) on DecompositionType.
The SRE character-class and case predicates move back into
sre_engine::string (their pre-extraction home) instead of living in a
unicode::regex module that only sre_engine and the vm _sre binding used;
is_uni_digit/is_uni_alnum still delegate to rustpython_unicode::classify.
engine.rs and _sre consume them from sre_engine::string again.
The unicode crate's tests move from a single root tests module in lib.rs
into per-module test submodules (case, classify, data, identifier,
normalize).
Assisted-by: Claude
* no-std
* Address review feedback: test module, dead variant, WTF-8 normalization
- differential.rs: move the sweep tests into a `mod tests` block and drop
the file-level allow of clippy::tests_outside_test_module and
std_instead_of_alloc; the test uses alloc collections instead.
- data.rs: remove the never-constructed DecompositionType::Canonical variant
and its allow(unused); compatibility decomposition never produces it and
canonical decomposition is handled through icu4x.
- normalize.rs: is_normalized now takes &Wtf8 and checks each UTF-8 run,
skipping lone surrogates, matching normalize's run-wise behavior.
- unicodedata.rs: pass as_wtf8() to is_normalized.
Assisted-by: Claude
|