mirror of
https://github.com/RustPython/RustPython.git
synced 2026-10-29 03:04:38 +08:00
Jeong, YunWon
85e3ea8869
* Extract unicodedata core into rustpython-unicode crate
Move the vm-free Unicode character database access — the generated UCD 3.2.0 /
latest tables, their build-time generation, and the icu4x/unicode_names2 lookups
for category/bidirectional/combining/east_asian_width/mirrored/decomposition/
normalize/is_normalized/digit/decimal/numeric/name/lookup — into a new leaf crate
over char/CodePoint/&Wtf8. stdlib/unicodedata.rs keeps the UCD pyclass binding: it
extracts the code point, boxes results, and maps errors. The unicode data files,
build.rs table generation, and the icu_properties/icu_normalizer/unicode_names2
dependencies move out of stdlib.
Assisted-by: Claude
* Route str predicates, casefold, and printable through rustpython-unicode
Add classify/case/identifier modules to the shared crate over char/&str/&Wtf8:
isalpha/isalnum/isdecimal/isdigit/isnumeric/isspace/isprintable classification,
XID identifier predicates, and full-mapping casefold. vm/builtins/str.rs and
literal/char.rs now call these instead of icu_properties/icu_casemap directly.
String-level iteration with final-sigma handling (lower/upper/title/capitalize,
islower/isupper) stays in the runtime.
Assisted-by: Claude
* Route sre_engine char classes through rustpython-unicode
Add the regex module with the SRE character-class and case predicates
(is_word/is_space/is_digit, is_uni_* Unicode variants, ascii/locale/unicode case
folding). sre_engine/string.rs now forwards to it, keeping its public API and
behavior identical — including the ASCII-only is_uni_digit and the hardcoded
is_uni_space table.
Assisted-by: Claude
* Route \N{} name lookups through rustpython-unicode
codegen/string_parser.rs (\N{...} escapes) and common/encodings.rs (the
namereplace error handler) now resolve character names via rustpython-unicode
instead of depending on unicode_names2 directly, leaving the crate as the sole
owner of the name database.
Assisted-by: Claude
* Add differential Unicode sweep and shared-crate snippet test
tests/differential.rs sweeps the full 0..0x110000 range and compares each str
classification predicate against a committed CPython 3.14 reference dataset
(tests/data/cpython3.14_predicates.txt, produced by generate_reference.py). Code
points that differ only because the Rust std / icu4x build ships a later Unicode
release than CPython 3.14's 16.0.0 are recorded in
tests/data/version_skew_cpython3.14.txt, regenerable via
RUSTPYTHON_UNICODE_REGEN_SKEW=1; the regen refuses to record any
cpython=true/crate=false divergence, so only newly-assigned code points are
allowed. Any other divergence fails. Both data files use a run-length
`predicate start:end,...` encoding.
extra_tests/snippets/stdlib_unicode_shared.py exercises the routed surface end to
end (str predicates, casefold, identifiers, unicodedata, normalize, \N{}, and re
character classes) and passes identically on CPython 3.14 and RustPython.
Assisted-by: Claude
* Match regex \d to Unicode decimal digits
is_uni_digit previously matched only ASCII 0-9, so re's \d in Unicode mode
missed decimal digits like ٥ and ५. SRE_UNI_IS_DIGIT matches Py_UNICODE_ISDECIMAL
(category Nd), so route it through classify::is_decimal. Unmasks test_bug_6561 in
test_re and extends the shared-crate snippet with the Nd/Nl/No cases.
Assisted-by: Claude
* Replace shallow Unicode wrappers with re-exports
Address review feedback on the extraction.
Functions that only forwarded to the shared crate become `use ... as`
re-exports rather than hand-written wrappers:
- literal: is_printable is dropped; escape.rs calls
rustpython_unicode::classify::is_repr_printable directly.
- unicode::identifier::is_continue re-exports is_xid_continue.
- unicode::data::lookup_character re-exports unicode_names2::character.
- Drop the unused repr(u8) on DecompositionType.
The SRE character-class and case predicates move back into
sre_engine::string (their pre-extraction home) instead of living in a
unicode::regex module that only sre_engine and the vm _sre binding used;
is_uni_digit/is_uni_alnum still delegate to rustpython_unicode::classify.
engine.rs and _sre consume them from sre_engine::string again.
The unicode crate's tests move from a single root tests module in lib.rs
into per-module test submodules (case, classify, data, identifier,
normalize).
Assisted-by: Claude
* no-std
* Address review feedback: test module, dead variant, WTF-8 normalization
- differential.rs: move the sweep tests into a `mod tests` block and drop
the file-level allow of clippy::tests_outside_test_module and
std_instead_of_alloc; the test uses alloc collections instead.
- data.rs: remove the never-constructed DecompositionType::Canonical variant
and its allow(unused); compatibility decomposition never produces it and
canonical decomposition is handled through icu4x.
- normalize.rs: is_normalized now takes &Wtf8 and checks each UTF-8 run,
skipping lone surrogates, matching normalize's run-wise behavior.
- unicodedata.rs: pass as_wtf8() to is_normalized.
Assisted-by: Claude
94 lines
3.6 KiB
Python
94 lines
3.6 KiB
Python
# Exercises the Unicode semantics routed through the shared rustpython-unicode
|
||
# crate: str predicates, casefold, identifier rules, unicodedata queries,
|
||
# normalization, \N{} escapes, and re character classes.
|
||
|
||
import re
|
||
import unicodedata
|
||
|
||
# --- str classification predicates ---------------------------------------
|
||
|
||
# Numeric_Type chain: isdecimal ⊂ isdigit ⊂ isnumeric
|
||
assert "5".isdecimal() and "5".isdigit() and "5".isnumeric()
|
||
assert not "²".isdecimal() # SUPERSCRIPT TWO: digit but not decimal
|
||
assert "²".isdigit() and "²".isnumeric()
|
||
assert not "⅓".isdigit() # VULGAR FRACTION ONE THIRD: numeric only
|
||
assert "⅓".isnumeric()
|
||
|
||
assert "abc".isalpha()
|
||
assert "abc123".isalnum()
|
||
assert not "abc123".isalpha()
|
||
assert "あ".isalpha() # HIRAGANA LETTER A
|
||
|
||
assert " \t\n".isspace()
|
||
assert " ".isspace() # IDEOGRAPHIC SPACE
|
||
assert "hello world".isprintable()
|
||
assert not "\x00".isprintable()
|
||
assert " ".isprintable() # ASCII space is printable
|
||
|
||
# identifier rules (XID_Start / XID_Continue, plus leading underscore)
|
||
assert "_var".isidentifier()
|
||
assert "유니코드".isidentifier() # Hangul identifier
|
||
assert not "1abc".isidentifier()
|
||
assert not "a b".isidentifier()
|
||
|
||
# --- case mapping / casefold ---------------------------------------------
|
||
|
||
assert "ABC".lower() == "abc"
|
||
assert "abc".upper() == "ABC"
|
||
# casefold uses full mappings, unlike lower()
|
||
assert "ß".casefold() == "ss" # LATIN SMALL LETTER SHARP S
|
||
assert "Σ".casefold() == "σ" # GREEK CAPITAL SIGMA -> small sigma
|
||
assert "Straße".casefold() == "strasse"
|
||
|
||
# lone-surrogate safety: casefold must not panic on surrogates
|
||
surrogate = "\ud800"
|
||
assert surrogate.casefold() == surrogate
|
||
|
||
# --- unicodedata ----------------------------------------------------------
|
||
|
||
assert unicodedata.category("A") == "Lu"
|
||
assert unicodedata.category("1") == "Nd"
|
||
assert unicodedata.bidirectional("A") == "L"
|
||
assert unicodedata.decimal("٥") == 5 # ARABIC-INDIC DIGIT FIVE
|
||
assert unicodedata.digit("²") == 2
|
||
assert abs(unicodedata.numeric("⅓") - (1 / 3)) < 1e-6
|
||
assert unicodedata.name("☃") == "SNOWMAN"
|
||
assert unicodedata.lookup("SNOWMAN") == "☃"
|
||
assert unicodedata.combining("́") == 230 # COMBINING ACUTE ACCENT
|
||
assert unicodedata.mirrored("(") == 1
|
||
assert unicodedata.east_asian_width("あ") == "W"
|
||
|
||
# ucd_3_2_0 legacy view (used by stringprep)
|
||
assert unicodedata.ucd_3_2_0.unidata_version == "3.2.0"
|
||
|
||
# --- normalization --------------------------------------------------------
|
||
|
||
composed = "é" # é
|
||
decomposed = "é"
|
||
assert unicodedata.normalize("NFC", decomposed) == composed
|
||
assert unicodedata.normalize("NFD", composed) == decomposed
|
||
assert unicodedata.is_normalized("NFC", composed)
|
||
assert not unicodedata.is_normalized("NFD", composed)
|
||
|
||
# --- \N{} escapes (compiler) ---------------------------------------------
|
||
|
||
assert "\N{SNOWMAN}" == "☃"
|
||
assert "\N{GREEK SMALL LETTER ALPHA}" == "α"
|
||
|
||
# --- re character classes -------------------------------------------------
|
||
|
||
assert re.fullmatch(r"\w+", "abc_123") is not None
|
||
assert re.fullmatch(r"\w+", "유니코드") is not None # \w is Unicode-aware
|
||
assert re.fullmatch(r"\d+", "123") is not None
|
||
# \d matches Unicode decimal digits (category Nd), not just ASCII
|
||
assert re.fullmatch(r"\d", "٥") is not None # ARABIC-INDIC DIGIT FIVE
|
||
assert re.fullmatch(r"\d", "५") is not None # DEVANAGARI DIGIT FIVE
|
||
assert re.fullmatch(r"\d", "²") is None # SUPERSCRIPT TWO (No), not decimal
|
||
assert re.fullmatch(r"\s+", " \t\n") is not None
|
||
# ASCII flag restricts \w to ASCII
|
||
assert re.fullmatch(r"\w+", "유", re.ASCII) is None
|
||
# case-insensitive matching routes through the shared case helpers
|
||
assert re.fullmatch(r"straße", "STRAßE", re.IGNORECASE) is not None
|
||
|
||
print("stdlib_unicode_shared: OK")
|