mirror of
https://github.com/RustPython/RustPython.git
synced 2026-10-28 18:59:04 +08:00
Jeong, YunWon
d04318efac
* sre_engine: document that StrDrive positions are character indices
StrDrive carried no documentation, so nothing recorded that a cursor's
position is a character index rather than a byte offset. Every implementation
satisfies it -- skip(n) advances position by exactly n in all three -- and the
engine depends on it: _count bounds a repeat with position + max_count and
reports the repeat length as a difference of positions, ASSERT compares
position against a lookbehind width, and search_info recovers a match start as
position - (len - 1). A drive over a variable-width encoding that stored byte
offsets would leave those type-correct and wrong, and would index a lookbehind
out of bounds.
Write the invariant down on StringCursor and on each trait method.
Also walk the group's own cursor in GROUPREF instead of counting to the
group's width, so the loop bound is the thing being stepped. Generated
machine code is unchanged.
Assisted-by: Claude
* _sre: drive an all-ASCII str subject over its bytes
with_sre_str handed every str subject to the &Wtf8 drive, which answers
count() by counting every code point and create_cursor(n) by stepping over the
first n. Both run once per Request, so a scan that restarts at successive
positions walked the subject again on every call: finditer over an ASCII
subject of n tokens was quadratic in n.
PyStr already records whether it is ASCII -- StrKind is decided when the
string is built -- and for ASCII a character index is a byte index, so the
&[u8] drive's cursor arithmetic already applies: count() is the byte length
and create_cursor() is a pointer offset. Add AsciiStr, which delegates every
StrDrive method to that impl and differs only in slice(), which reslices the
span and returns str rather than bytes.
Matching is unaffected: StrDrive carries no unicode semantics, because the
engine keys every unicode decision on the compiled pattern's opcode rather
than on the subject type.
Also bind the subject once in with_sre_str, so callers passing a temporary
(`&x.clone()`) build it once rather than per arm.
finditer over an ASCII subject, n tokens, this machine:
n before after
5000 41.60ms 1.37ms
10000 217.30ms 2.68ms
20000 1206.28ms 5.17ms
40000 5016.65ms 10.31ms
Per-doubling x5.22/x5.55/x4.16 becomes x1.95/x1.93/x1.99. Collecting
m.group(0) for every match goes 5061.02ms -> 17.80ms at n=40000.
test.test_re is unchanged at 166 tests, OK (skipped=14, expected failures=6),
and a 2731-line differential over the is_ascii() boundary -- findall,
finditer spans and groups, sub, split, match, search, fullmatch across
subjects that are empty, ASCII, non-ASCII, or mixed -- is byte-identical to
CPython 3.14.6. Introducing an off-by-one in AsciiStr::slice moves 1224 of
those lines, so the comparison reaches the new code.
Assisted-by: Claude