CPython's defaultdict.__missing__ (Modules/_collectionsmodule.c::defdict_missing)
calls default_factory() first; if the factory's recursion already populated
self[key] while running, the existing value is preserved instead of being
overwritten.
RustPython ships a Python fallback at Lib/collections/_defaultdict.py
(the C _collections.defaultdict is not available). That fallback
unconditionally executed self[key] = val after the factory returned,
overwriting any value the recursive call had already stored.
Add a 'if key in self: return self[key]' guard before the assignment.
dict.__contains__ does not invoke __missing__, so there's no recursion
risk; in the common non-reentrant case the check is False and behavior
is unchanged.
Unmasks test_defaultdict.TestDefaultDict.test_factory_conflict_with_set_value.
CPython rejects digit-only format-string field names that overflow
Py_ssize_t at parse time with ValueError: Too many decimal digits in
format string (Python/string_parser.c::get_integer). RustPython's
FieldName::parse accepted any digit string usize::from_str could parse,
producing IndexError or KeyError at lookup instead.
Cap the parsed index at isize::MAX (Py_ssize_t::MAX on every platform)
inside FieldName::parse. Also reject digits-only strings whose value
overflows usize itself (caught when parse_usize returns None on an
all-digit input). A new FormatParseError::TooManyDecimalDigits maps to
the byte-identical CPython wording.
Unmasks test_str.StrTest.test_format_huge_item_number.
CPython rejects format-spec widths that exceed Py_ssize_t::MAX with
ValueError: Too many decimal digits in format string. RustPython's
FormatSpec::_parse only capped precision (via parse_precision); width
was accepted up to usize::MAX, so values like sys.maxsize + 1 silently
produced an effectively-ignored width.
Reject any width above i32::MAX with FormatSpecError::DecimalDigitsTooMany,
matching the existing precision cap and producing the byte-identical
ValueError wording.
Unmasks test_str.StrTest.test_format_huge_width.
CPython 3.14 fully removed the long-deprecated fallback from int() to
__trunc__. Classes that want int(x) support must implement __int__ or
__index__; math.trunc(x) continues to use __trunc__.
Delete the __trunc__ branch in PyObject::try_int so a class with only
__trunc__ now reaches the existing TypeError arm, matching CPython.
Unmasks test_int.IntTestCases.test_intconversion.
Closes#7450.
CPython's unicode_new_impl returns the PyObject_Str result as-is when
type == &PyUnicode_Type, only invoking unicode_subtype_new for actual
str subclasses. RustPython's PyStr::Constructor stripped the result via
Self::from(s.as_wtf8().to_owned()) and re-materialized through
into_ref_with_type, dropping the subclass type even when cls is exactly
str.
Add a slot_new branch that returns input.str(vm)? directly when cls is
str_type with no encoding. Subtype construction and the bytes-decoding
path are unchanged.
Unmasks test_str.StrTest.test_conversion (11 assertTypedEqual cases).
CPython's _abc._abc_init validates that __abc_tpflags__ doesn't combine
Py_TPFLAGS_SEQUENCE and Py_TPFLAGS_MAPPING (Modules/_abc.c::_abc_init).
RustPython's check_abc_tpflags copied the bits onto slots.flags without
that validation, so a class declaring both produced an inconsistent
type silently.
Validate the masked __abc_tpflags__ value against COLLECTION_FLAGS and
return a String error (which the existing new_heap_inner caller turns
into a TypeError) when both bits are present. Move the inheritance-skip
check after the self-attrs branch so a child overriding __abc_tpflags__
is still validated even when the parent already set one of the bits.
deque.__reduce__ passed None as the unpickle state, so a deque
subclass's __dict__ and __slots__ values were dropped across a pickle
round-trip. CPython's deque___reduce___impl
(Modules/_collectionsmodule.c::deque___reduce___impl) calls
_PyObject_GetState, which returns the dict (or a (dict, slots) tuple)
so subclass attributes survive.
Replace the placeholder with the result of __getstate__() on the
instance. object.__getstate__ already implements the dict / dict+slots
protocol, matching _PyObject_GetState.
* Enforce int_max_str_digits on int-to-str conversions
The str-to-int direction already enforced sys.get_int_max_str_digits()
via bytes_to_int; the int-to-str direction did not. CPython 3.14 enforces
both per PEP 644.
Adds check_int_to_str_digits helper in builtins::int (bit-count fast path
+ digit upper-bound from log10(2)), wired into the four Python-level
entry points: repr, the str fast path in protocol::object, int.__format__
(decimal/n/empty spec only — binary bases x/o/b are exempt per CPython),
and the DecimalD/I/U branches of vm::cformat for both str % and bytes %.
Unmasks 8 expectedFailure tests across test_int (max_str_digits, DoS
prevention, int_from_other_bases — each mirrored in IntSubclass),
test_ast (test_repr_large_input_crash) and test_reprlib (test_numbers).
Boundary cases (4299/4300/4301 digits at limit=4300) match CPython 3.14.4.
* Skip int-to-str DoS test on platforms without time.get_clock_info
The test_denial_of_service_prevented_int_to_str regression test uses
support.Stopwatch, which calls time.get_clock_info('monotonic'). In
RustPython that function is gated to unix/windows targets only, so on
wasm32-wasip1 it surfaces as AttributeError and breaks the wasm-wasi CI.
Guard the test with skipUnless(hasattr(time, 'get_clock_info'), ...) so
it runs everywhere it can and is skipped on wasm.
Also narrow is_decimal_int_format to Number(Case::Lower): 'N' is rejected
by format_int as UnknownFormatCode, so excluding it preserves that error
path instead of intercepting it with the digit-limit check.
* Add TODO: RUSTPYTHON marker to skipUnless reason
scripts/update_lib uses TODO: RUSTPYTHON markers inside unittest
decorator reason strings to identify and migrate custom RustPython
patches across CPython library updates.
* Use expectedFailureIf for wasm get_clock_info gap
skipUnless silently hides the test forever; expectedFailureIf surfaces
unexpected success once RustPython implements time.get_clock_info on
wasm, prompting marker removal.
The behavior already matched CPython (the slot conflict is detected,
the __doc__ delete is rejected); only the message text drifted.
- "__slots__ conflicts with a class variable" -> drop the stray "a"
to match CPython's "conflicts with class variable".
- "cannot delete '__doc__' attribute of type 'X'" -> insert "immutable"
before "type" to match CPython's wording (CPython surfaces the same
phrase even for user-defined classes since the descriptor refuses
the delete unconditionally).
CPython's staticmethod and classmethod set __func__ and copy wrapper
attributes (__doc__, __name__, etc.) only inside __init__
(Objects/funcobject.c::sm_init / cm_init). RustPython did this work in
slot_new and again in __init__, so subclasses that override __init__
without calling super().__init__() saw __func__ pointing at the
original callable instead of None.
Move the callable assignment and the wrapper-attribute copy into
Initializer::init; slot_new now just validates the signature and stores
None for the callable, matching the CPython contract.
* Report invalid \uXXXX escape position at the u character
CPython's json decoder reports the position of the `u` specifier
when a \uXXXX escape fails to parse, but RustPython was reporting
the preceding `\`. For surrogate-pair cases (\uXXXX\uYYYY) the
second call was passing char_offset + next_char_i + 1, which
lands on the first hex digit of the first escape -- unrelated to
the actual failure site.
Pass next_char_i (position of the primary `u`) to the primary
decode_unicode call, and capture the second `u`'s char index from
the next_tuple peek to pass to the surrogate-pair decode_unicode
call.
Verified: 13 targeted probes across invalid-hex, short, and pair
cases now all match CPython positions. test.test_json 214 tests
pass with no regressions.
* Add regression test for invalid \uXXXX escape position
* Use raise AssertionError instead of assert False (B011)
* Match CPython error type for non-ASCII struct format arguments
Struct() raised the wrong exception type when the format argument
contained non-ASCII characters:
- str input with non-ASCII char: RustPython raised UnicodeDecodeError
with an empty message; CPython raises UnicodeEncodeError as if
format.encode('ascii') had been called directly.
- bytes input with non-ASCII byte: same wrong UnicodeDecodeError;
CPython passes the bytes through to the format parser, which then
errors with struct.error("bad char in struct format").
Restructure IntoStructFormatBytes::try_from_object to:
- raise UnicodeEncodeError("ascii", s, start, start+1, "ordinal not
in range(128)") for non-ASCII str, with start computed as the
first non-ASCII code point position (matching CPython's natural
encoding-error format);
- raise struct.error("bad char in struct format") for non-ASCII bytes,
produced via the existing new_struct_error helper.
Probed byte-identical with CPython 3.14.4 for both cases. Full
test.test_struct (43 tests) passes with no regressions. Sanity-tested
all standard format/pack/unpack/calcsize call shapes remain unchanged.
* Add regression test for non-ASCII format string error types
* Use raise AssertionError instead of assert False (B011)
Match CPython's "field 'X' is required for Y" error format when
required scalar AST fields receive None during ast_from_object
conversion. Previously these produced TypeError or a generic
"None disallowed in expression list" ValueError.
Fields covered (Alias.name, Arg.arg, Comprehension.target/iter,
Keyword.value, MatchCase.pattern, WithItem.context_expr,
YieldFrom.value): all now raise the CPython-compatible message
verbatim.
Add a get_node_field_required helper alongside the existing
get_node_field / get_node_field_opt and switch the eight call
sites that read these required scalar fields. The helper rejects
None with the proper ValueError before the value flows into the
type converter where Rust's strong typing would otherwise force a
generic catch-all error. Optional fields (read via
get_node_field_opt) and fields where None is valid (e.g.
Constant.value) are unaffected.
This complements the existing AST validator pass at
crates/vm/src/stdlib/_ast/validate.rs (hooked at _ast.rs:763):
the validator handles post-conversion semantic invariants
(ExprContext, empty body, cross-field rules) but cannot handle
required-scalar-field None because Rust's non-Option types reject
None at conversion time, before validation runs.
Verified byte-identical with CPython 3.14.4 for all seven probe
cases. test.test_ast.test_ast: 227 tests, expected failures
38 -> 36 (test_empty_yield_from and test_none_checks unmasked,
no regressions). test.test_compile: 0 regressions.
The _json decoder had two failure modes when a Python str value would
contain a lone surrogate (legal per the Python 3 str model):
1. Boundary UnicodeEncodeError: JsonScanner::Callable::call rejected
any input str with surrogates via try_into_utf8 before scanning
began.
2. Silent U+FFFD corruption: call_scan_once and parse_object's key
path called .to_string() on scanstring's Wtf8Buf output, which
routes through Wtf8::Display (lossy). Array values and dict keys
decoded from JSON \uXXXX escapes silently became U+FFFD.
Switch JsonScanner's five PyUtf8StrRef signatures to PyStrRef, drop
the entry-point try_into_utf8 call, and feed Wtf8Buf directly to
new_str instead of going through .to_string(). Key memoization now
uses HashMap<Wtf8Buf, PyStrRef> so surrogate-bearing keys survive
interning. parse_number takes &[u8] since JSON numbers are ASCII.
Extends the WTF-8 refactor pattern established in #7673 to the
decoder. machinery::scanstring already returns Wtf8Buf and is
unchanged.
Unmasks test_single_surrogate_decode. 214 tests in test.test_json
pass with no regressions. Decoder output verified byte-identical to
CPython 3.13.4 over 10,000 random fuzz cases (JSON docs containing
random surrogate escapes at root/list/dict positions, compared via
json.dumps(..., ensure_ascii=True, sort_keys=True)).
encode_basestring/encode_basestring_ascii took PyUtf8StrRef, so
json.dumps(str_with_lone_surrogate) raised UnicodeEncodeError at the
Python/Rust boundary before write_json_string ran. CPython's encoder
emits \uXXXX under ensure_ascii=True and passes raw WTF-8 otherwise.
Switch to PyStrRef + s.as_wtf8(), matching scanstring in the same file.
Rewrite write_json_string to accept &Wtf8 and iterate
code_point_indices, emitting \uXXXX for surrogates in ascii mode and
passing their bytes through otherwise. Stop escaping 0x7F in the
ensure_ascii=False path (matches py_encode_basestring). Return Wtf8Buf
via the checked from_bytes so invariant breaks panic instead of UB.
Fuzzing also exposed two pre-existing ESCAPE_CHARS typos: 0x0B was
"\u000" and 0x1B was "\u001" (both missing trailing 'b'). Fixed here.
Verified byte-identical with CPython 3.13.4 over 16 manual + 10,000
random fuzz cases. Full test.test_json: 214 tests, 0 failures, 0
unexpected successes. Unmasks test_ascii_non_printable_encode and
test_single_surrogate_encode. Decoder path is a follow-up.
All three @unittest.skip("TODO: RUSTPYTHON") markers in
datetimetester.py are stale — the tests pass as written on current
main. Probed every assertion in each test body against both CPython
3.13.4 and RustPython: outputs are byte-identical for all unicode
passthrough cases (emoji, lone surrogates, surrogate pairs, null
bytes) in test_strftime_special and for all 11 FixedOffset
comparisons (utcoffset, tzname, dst, hash equality) in test_zones.
Full test.test_datetime run after unmasking: 1803 tests, 0 failures,
0 unexpected successes, 463 skipped (was 469 — 6 test-class inherited
runs now execute successfully).
Part of the skip-test inventory in #7611.
Sibling fix to #7656 (which handled Vec<Expr>). The same catch-all
TypeError ("expected some sort of stmt, but got {}") in
Stmt::ast_from_object silently swallowed None, so compile() on an
AST with None in a statement-list field (ClassDef.body, Try.body,
For.body, etc.) raised TypeError where CPython raises
ValueError("None disallowed in statement list").
Add an is_none check before the catch-all, matching the Expr-side
shape introduced in #7656. Option<Stmt> positions are unaffected —
they short-circuit None earlier in node.rs.
Unmasks test_classdef (body=[None] sub-case). Full
test.test_ast.test_ast run: 227 tests, 0 unexpected successes, 38
expected failures (was 39).
Clear four of the five ShellCheck findings reported in #7653. The
fifth finding is in upgrade-pylib.lock.yml — a gh-aw-generated file
marked "DO NOT EDIT" whose source .md does not contain the flagged
pattern — and is out of scope here; it will resolve via the in-flight
gh-aw version bump (#7650).
lib-deps-check.yaml:
* SC2076: replace `=~ " $module "` (quoted regex interpreted
literally) with `== *" $module "*` glob match. Same intent
(literal substring), same semantics across regex metachars.
* SC2086: quote `$GITHUB_OUTPUT` in the output redirect.
update-doc-db.yml:
* SC2129: collapse the eight sequential `echo ... >> $OUTPUT_FILE`
lines (plus one `cat ... >> $OUTPUT_FILE`) into a single grouped
redirect `{ ...; } > "$OUTPUT_FILE"`. Drops the now-redundant
`echo -n '' > $OUTPUT_FILE` truncate.
* SC2016: add `# shellcheck disable=SC2016` above the block; the
backticks in the auto-generated-header comment are literal
Markdown, not command substitution.
Verified locally with shellcheck 0.11.0: both modified blocks
produce no ShellCheck output. Semantic equivalence of the
lib-deps-check change confirmed across six test inputs including
regex metachars and glob-meaningful characters.
This PR fixes a regression from my last islower/isupper patch.
Python's Bytes doesn't assume an encoding, so methods like islower
should only consider ASCII casing.
I updated islower/isupper for UTF-8 and WTF-8 to match CPython more
closely. The two functions now use the same properties as CPython and
now match CPython exactly.
I updated the unit tests to pass on Python 3.15. Unicode updates
sometimes cause properties to shift. I previously tested everything on
Python 3.14, but that lead to failures that I assumed were bugs but were
actually due to Unicode differences. For example, U+0295 is a lower case
letter in older Unicode versions but is NOT in newer versions.
One of the new tests is disabled on Python 3.14 for now because it will
fail in CI till CI is bumped to 3.15.
fanninpm