mirror of
https://github.com/RustPython/RustPython.git
synced 2026-10-28 18:59:04 +08:00
Jeong, YunWon
55c012cffd
* Extract unicodedata core into rustpython-unicode crate
Move the vm-free Unicode character database access — the generated UCD 3.2.0 /
latest tables, their build-time generation, and the icu4x/unicode_names2 lookups
for category/bidirectional/combining/east_asian_width/mirrored/decomposition/
normalize/is_normalized/digit/decimal/numeric/name/lookup — into a new leaf crate
over char/CodePoint/&Wtf8. stdlib/unicodedata.rs keeps the UCD pyclass binding: it
extracts the code point, boxes results, and maps errors. The unicode data files,
build.rs table generation, and the icu_properties/icu_normalizer/unicode_names2
dependencies move out of stdlib.
Assisted-by: Claude
* Route str predicates, casefold, and printable through rustpython-unicode
Add classify/case/identifier modules to the shared crate over char/&str/&Wtf8:
isalpha/isalnum/isdecimal/isdigit/isnumeric/isspace/isprintable classification,
XID identifier predicates, and full-mapping casefold. vm/builtins/str.rs and
literal/char.rs now call these instead of icu_properties/icu_casemap directly.
String-level iteration with final-sigma handling (lower/upper/title/capitalize,
islower/isupper) stays in the runtime.
Assisted-by: Claude
* Route sre_engine char classes through rustpython-unicode
Add the regex module with the SRE character-class and case predicates
(is_word/is_space/is_digit, is_uni_* Unicode variants, ascii/locale/unicode case
folding). sre_engine/string.rs now forwards to it, keeping its public API and
behavior identical — including the ASCII-only is_uni_digit and the hardcoded
is_uni_space table.
Assisted-by: Claude
* Route \N{} name lookups through rustpython-unicode
codegen/string_parser.rs (\N{...} escapes) and common/encodings.rs (the
namereplace error handler) now resolve character names via rustpython-unicode
instead of depending on unicode_names2 directly, leaving the crate as the sole
owner of the name database.
Assisted-by: Claude
* Add differential Unicode sweep and shared-crate snippet test
tests/differential.rs sweeps the full 0..0x110000 range and compares each str
classification predicate against a committed CPython 3.14 reference dataset
(tests/data/cpython3.14_predicates.txt, produced by generate_reference.py). Code
points that differ only because the Rust std / icu4x build ships a later Unicode
release than CPython 3.14's 16.0.0 are recorded in
tests/data/version_skew_cpython3.14.txt, regenerable via
RUSTPYTHON_UNICODE_REGEN_SKEW=1; the regen refuses to record any
cpython=true/crate=false divergence, so only newly-assigned code points are
allowed. Any other divergence fails. Both data files use a run-length
`predicate start:end,...` encoding.
extra_tests/snippets/stdlib_unicode_shared.py exercises the routed surface end to
end (str predicates, casefold, identifiers, unicodedata, normalize, \N{}, and re
character classes) and passes identically on CPython 3.14 and RustPython.
Assisted-by: Claude
* Match regex \d to Unicode decimal digits
is_uni_digit previously matched only ASCII 0-9, so re's \d in Unicode mode
missed decimal digits like ٥ and ५. SRE_UNI_IS_DIGIT matches Py_UNICODE_ISDECIMAL
(category Nd), so route it through classify::is_decimal. Unmasks test_bug_6561 in
test_re and extends the shared-crate snippet with the Nd/Nl/No cases.
Assisted-by: Claude
* Replace shallow Unicode wrappers with re-exports
Address review feedback on the extraction.
Functions that only forwarded to the shared crate become `use ... as`
re-exports rather than hand-written wrappers:
- literal: is_printable is dropped; escape.rs calls
rustpython_unicode::classify::is_repr_printable directly.
- unicode::identifier::is_continue re-exports is_xid_continue.
- unicode::data::lookup_character re-exports unicode_names2::character.
- Drop the unused repr(u8) on DecompositionType.
The SRE character-class and case predicates move back into
sre_engine::string (their pre-extraction home) instead of living in a
unicode::regex module that only sre_engine and the vm _sre binding used;
is_uni_digit/is_uni_alnum still delegate to rustpython_unicode::classify.
engine.rs and _sre consume them from sre_engine::string again.
The unicode crate's tests move from a single root tests module in lib.rs
into per-module test submodules (case, classify, data, identifier,
normalize).
Assisted-by: Claude
* no-std
* Address review feedback: test module, dead variant, WTF-8 normalization
- differential.rs: move the sweep tests into a `mod tests` block and drop
the file-level allow of clippy::tests_outside_test_module and
std_instead_of_alloc; the test uses alloc collections instead.
- data.rs: remove the never-constructed DecompositionType::Canonical variant
and its allow(unused); compatibility decomposition never produces it and
canonical decomposition is handled through icu4x.
- normalize.rs: is_normalized now takes &Wtf8 and checks each UTF-8 run,
skipping lone surrogates, matching normalize's run-wise behavior.
- unicodedata.rs: pass as_wtf8() to is_normalized.
Assisted-by: Claude
57 lines
2.0 KiB
Rust
57 lines
2.0 KiB
Rust
// spell-checker:ignore ossl osslconf
|
|
|
|
fn main() {
|
|
println!(r#"cargo::rustc-check-cfg=cfg(osslconf, values("OPENSSL_NO_COMP"))"#);
|
|
println!(r#"cargo::rustc-check-cfg=cfg(openssl_vendored)"#);
|
|
|
|
#[allow(
|
|
clippy::unusual_byte_groupings,
|
|
reason = "hex groups follow OpenSSL version field boundaries"
|
|
)]
|
|
let ossl_vers = [
|
|
(0x1_00_01_00_0, "ossl101"),
|
|
(0x1_00_02_00_0, "ossl102"),
|
|
(0x1_01_00_00_0, "ossl110"),
|
|
(0x1_01_00_07_0, "ossl110g"),
|
|
(0x1_01_00_08_0, "ossl110h"),
|
|
(0x1_01_01_00_0, "ossl111"),
|
|
(0x1_01_01_04_0, "ossl111d"),
|
|
(0x3_00_00_00_0, "ossl300"),
|
|
(0x3_01_00_00_0, "ossl310"),
|
|
(0x3_02_00_00_0, "ossl320"),
|
|
(0x3_03_00_00_0, "ossl330"),
|
|
];
|
|
|
|
for (_, cfg) in ossl_vers {
|
|
println!("cargo::rustc-check-cfg=cfg({cfg})");
|
|
}
|
|
|
|
#[cfg(feature = "ssl-openssl")]
|
|
{
|
|
#[allow(
|
|
clippy::unusual_byte_groupings,
|
|
reason = "OpenSSL version number is parsed with grouped hex fields"
|
|
)]
|
|
if let Ok(v) = std::env::var("DEP_OPENSSL_VERSION_NUMBER") {
|
|
println!("cargo:rustc-env=OPENSSL_API_VERSION={v}");
|
|
// cfg setup from openssl crate's build script
|
|
let version = u64::from_str_radix(&v, 16).unwrap();
|
|
for (ver, cfg) in ossl_vers {
|
|
if version >= ver {
|
|
println!("cargo:rustc-cfg={cfg}");
|
|
}
|
|
}
|
|
}
|
|
if let Ok(v) = std::env::var("DEP_OPENSSL_CONF") {
|
|
for conf in v.split(',') {
|
|
println!("cargo:rustc-cfg=osslconf=\"{conf}\"");
|
|
}
|
|
}
|
|
// it's possible for openssl-sys to link against the system openssl under certain conditions,
|
|
// so let the ssl module know to only perform a probe if we're actually vendored
|
|
if std::env::var("DEP_OPENSSL_VENDORED").is_ok_and(|s| s == "1") {
|
|
println!("cargo::rustc-cfg=openssl_vendored")
|
|
}
|
|
}
|
|
}
|