* Align CFG cleanup bytecode with CPython
* Bytecode parity: fblock unwind, fstring join, folding, scope
- compile.rs: unwind_fblock_stack returns whether a finally ran so
return-statement emission can adjust location handling; restructure
try/except/finally cleanup to preserve or drop boundary NOPs based on
whether the body falls through; rework f-string lowering with
count/join helpers; remove the per-collection-type heuristic for
AST-level folding and defer to flowgraph passes; add several folding
helpers and a ComprehensionLoopControl enum.
- ir.rs: re-run unary/binop folding around tuple folding, add
reorder_conditional_scope_exit_and_jump_back_blocks and several block
classification helpers, add MAX_STR_SIZE, change is_exit_without_lineno
to take the block list.
- symboltable.rs: in analyze_cells, remove names owned as cells in
function-like scopes from the parent's free set; mark lambda scope
type explicitly.
* Refine CFG scope-exit backedge ordering
Five related CPython parity gaps in `str` formatting and construction:
1. **`str(bytes, errors=...)` triggers decode mode.** Previously, only
`encoding=` triggered decode; passing only `errors=` fell back to
`repr()`. CPython's behavior: presence of `encoding` OR `errors`
triggers decode mode (default UTF-8 when only `errors` is given).
2. **`'{...}'.format() IndexError wording.** Generic Rust "tuple index
out of range" replaced with CPython's "Replacement index N out of
range for positional args tuple".
3. **`{0:3.2s}.format('abc')` → 'ab '.** String format spec applied
precision after width padding; CPython truncates BEFORE padding.
Reorder the operations.
4. **`%x` / `%o` / `%X` / `%c` accept `__index__` objects.** Previously
only `PyInt` downcast was attempted. Mirror CPython's
PyNumber_Index dispatch via `try_index_opt`.
5. **`%d` / `%u` / `%i` error wording.** "a number is required" →
"a real number is required" (matches CPython).
Also adds `not <type>` suffix to `%c` error messages so the type is
visible in TypeError text (matches CPython structure even without
fully-qualified names).
Verified byte-identical with CPython 3.14.4 across 25+ probes covering
the format/spec/constructor combinations. Unmasks
`test_str.test_constructor_keyword_args` and
`test_str.test_constructor_defaults`. test_str/test_bytes/test_format/
test_codecs/test_io/test_unicode_identifiers — 1,429 tests pass, 0
regressions. All 188 `extra_tests/snippets/*.py` pass under the CI
feature set.
`test_str.test_format` and `test_str.test_formatting` markers retained:
`test_format` still trips on `'{0:08s}'.format('result')` (numeric
zero-pad treated as fill+left-align by CPython for str type — separate
format-spec parser concern). `test_formatting` still trips on
`%c` error message expecting fully qualified `module.qualname` (RP
returns bare class name — separate broader concern).
Two CPython parity gaps in `compile()` argument handling:
1. **filename rejects only non-buffer-protocol types.** CPython's
`PyUnicode_FSDecoder` accepts `str`, `bytes`, and objects with
`__fspath__` only — `bytearray` / `memoryview` / `array.array` raise
TypeError. RustPython's `FsPath::TryFromObject` impl falls back to
any buffer-protocol object and silently converts to bytes, accepting
them.
```python
>>> compile('pass', bytearray(b'file.py'), 'exec')
# CPython 3.14.4: TypeError
# RustPython main: <code object> ❌
```
Change `CompileArgs::filename` from `FsPath` (which uses the
permissive `TryFromObject` impl) to `PyObjectRef`, then call the
strict `FsPath::try_from_path_like` at the top of `compile()`. Other
`FsPath` consumers are unchanged.
2. **dont_inherit strict-checks the `bool` type.** CPython routes
`dont_inherit` through `PyObject_IsTrue`, calling `__bool__` on
arbitrary objects and propagating exceptions raised there. RustPython
typed it `OptionalArg<bool>`, which rejects subclass-less objects at
binding time before `__bool__` runs.
```python
>>> class EvilBool:
... def __bool__(self): raise ValueError('hi')
>>> compile('pass', 'f', 'exec', dont_inherit=EvilBool())
# CPython 3.14.4: ValueError('hi')
# RustPython main: TypeError('Expected type bool, not EvilBool') ❌
```
Switch to `OptionalArg<ArgIntoBool>` — already used elsewhere in
`builtins` (`all`, `any`, `print(..., flush=...)`).
Verified byte-identical with CPython 3.14.4 across:
- 6 filename types (str, bytes, bytearray, memoryview, list, int)
- 7 dont_inherit values (True/False/1/0/""/"yes"/None) plus the
`__bool__`-raises-ValueError case
Unmasks `Lib/test/test_compile.py`:
- `test_compile_filename`
- `test_compile_filename_refleak`
`cargo run --release -- -m test test_compile test_builtin test_compileall
test_codeop test_ast` — 703 tests pass, 0 regressions. All 188
`extra_tests/snippets/*.py` pass under the CI feature set.
* Round float at the decimal level to match CPython's _Py_dg_dtoa
CPython's `float.__round__` (Objects/floatobject.c) routes through
`_Py_dg_dtoa` and rounds at the decimal level. The previous
`round_float_digits` multiplied by 10**ndigits and rounded at the
IEEE 754 binary level, which diverges for values that aren't exactly
representable. For example, 2.675 stores as 2.67499...; dtoa correctly
rounds it down to 2.67, but `(2.675 * 100.0).round() / 100.0` lands on
2.68 because the multiplication produces a phantom 267.5 tie that
round-half-to-even snaps up.
Rust's `{:.*}` float formatting uses dtoa-style algorithms (Grisu3 +
Dragon4 fallback) and matches CPython's `_Py_dg_dtoa` byte-for-byte.
Replace the multiply-then-round step with `format!` + `parse` for
ndigits >= 0. The ndigits < 0 path is unchanged because dividing
typical inputs by 10**|ndigits| produces genuine ties rather than
synthesizing them.
Verified byte-identical with CPython 3.14.4 over a 108-case random
fuzz plus targeted half-tie probes. Unmasks
`test_float.RoundTestCase.test_matches_float_format` and
`test_previous_round_bugs`.
* Use #[expect] with reason for float_cmp suppression
Co-authored-by: ShaharNaveh <50263213+ShaharNaveh@users.noreply.github.com>
---------
Co-authored-by: ShaharNaveh <50263213+ShaharNaveh@users.noreply.github.com>
* Fix function attribute tests: __defaults__ del, __code__ free vars check, bound method dir()
- Support `del func.__defaults__` and `del func.__kwdefaults__` by changing
setter signatures to `PySetterValue<Option<...>>` so Delete is treated as
setting the value to None (matching CPython behaviour)
- Validate free-variable count when assigning to `func.__code__`: raise
ValueError when the new code object's freevars count doesn't match the
existing closure size
- Add `__dir__` to `PyBoundMethod` that delegates to the underlying
function's dir(), so attributes set on the function show up in
`dir(bound_method)`
- Remove `@unittest.expectedFailure` from the four tests that now pass:
test_blank_func_defaults, test_func_default_args, test___code__,
test_dir_includes_correct_attrs
* Remove expectedFailure markers for now-passing tests
* Update `test_unicodedata.py` to 3.14.4
* Add test asset
* Mark failing tests
* Add `expectedSuccess` docorator
* Add support for nee decorator in patch_spec
* Reapply patches
The uppercase/titlecase branch of PyStr::title() pushed characters
unchanged when starting a new word, which left Latin Extended-B
digraphs (U+01F1 'DZ', U+01C4 'DŽ', etc.) in their uppercase form
instead of mapping them to their distinct titlecase counterparts
(U+01F2 'Dz', U+01C5 'Dž'). For ASCII letters and characters where
to_titlecase is identity this had no effect, hiding the bug for the
common case.
Mirror the lowercase branch — which already calls to_titlecase()
when starting a new word — so both branches symmetrically apply
the titlecase mapping. char::to_titlecase is identity for already-
titlecase and ASCII-uppercase characters, so existing cases stay
correct.
Also unmasks test_unicodedata.UnicodeMiscTest.test_bug_4971, which
asserts exactly this behavior (`'DŽ'.title() == 'Dž'` etc.)
and was marked expectedFailure with reason `+ Dž`.
Closes#7527 (the only example from that issue still failing on
3.14.4; the other four examples already pass on current main).
CPython splits PathLike resolution across two layers: _io.FileIO
preserves the raw argument on .name (low-level contract), while
the high-level open() wrapper resolves it via os.fspath before
delegating. RustPython's io_open skipped that step, so
builtins.open(PathLike).name returned the PathLike object instead
of the resolved str/bytes.
Match Lib/_pyio.py:193-194 by resolving non-int inputs at the
io_open boundary, leaving _io.FileIO unchanged.
* typealias reviews
* Bytecode parity - try/except block order, CFG reorder
Reorder try/except/else/finally to emit else+finally before
except handlers matching CPython layout. Add set_no_location
for cleanup blocks. Extend CFG reorder pass to handle true-path
jump-back for generators, break/continue, and assert in loops.
Add stop-iteration error handler awareness to block protection.
* Align CFG cleanup bytecode with CPython
* Unmark test_dis.test_findlabels expected failure
* Compute target predecessor flags in single pass
remove_nops and remove_redundant_nops_in_blocks repeated
has_jump_predecessor / has_plain_jump_predecessor / target lookups
per block, scanning all blocks each time. With ~200,000 if blocks
this became O(B^2 * I) and timed out test_compile.test_stack_overflow.
Fold the three flags into one O(B * I) pass via
compute_target_predecessor_flags.
* Address review feedback on slice folding, fallthrough, attr-chain
- try_fold_constant_slice now delegates to try_fold_constant_expr so
slice bounds accept the same constants other folding paths do
(unary-folded values, __debug__, etc.).
- remove_nops resolves fallthrough predecessors through empty blocks
via next_nonempty_block before checking ends_with_for_cleanup.
- should_deopt_borrowed_attr_chain ReturnIter matcher now accepts
Instruction::CallKw alongside Instruction::Call, matching the
Call/CallKw treatment in the surrounding deopt trigger check.
* remove test
* Fix del obj.__dict__ to match CPython behavior (issue #5355)
* Address CodeRabbit concerns: fix GC clearing and improve thread safety of lazy __dict__ recreation
* Fix del obj.__dict__: improve GC safety, implement lazy re-creation in setattr, and enable passing CPython tests
* Restore expectedFailure for test_has_inline_values
* Fix ObjExt::new call site to include has_dict parameter
* Remove stray test.py to avoid CI syntax errors
* Remove debug txt files and clean test_class.py comments
* Delete Lib/test/test_class.py
* Restore test_class.py with correct changes (remove expectedFailure, no deletion)
* Fix clippy warnings: remove unused into_inner, collapse nested if-let
* Fix rustfmt formatting and ruff PEP 8 E302 blank line
* Align __dict__ error messages and ensure safety for function/partial objects
* Fix compilation errors: change &self to &Py<Self> in __dict__ methods
* Fix compilation errors: resolve borrow-after-move and replace transpose on PySetterValue
- type.rs: Replace invalid .transpose() on PySetterValue with explicit
match on Assign/Delete variants in subtype_set_dict
- function.rs: Fix borrow-after-move in set___dict__ by capturing class
name before downcast; use as_object() for instance_dict/set_dict calls
- _functools.rs: Same borrow-after-move fix and as_object() calls for
PyPartial's __dict__ getter/setter
* Fix compilation errors: resolve borrow-after-move and replace transpose on PySetterValue
* Fix snippet formatting and mark test_remote as expected failure
* Fix test_remote by removing HAS_DICT flag from function type
* Fix lint formatting error
* Remove unnecessary print statement in test_del_dict
* Fix trailing newlines in snippet test
* Trigger CI
* Align __dict__ generic setter behavior
* Move __dict__ deletion tests to relevant snippets
---------
Co-authored-by: Jeong, YunWon <jeong@youknowone.org>
* Short-circuit identity in rich_compare_bool for Eq/Ne (PyObject_RichCompareBool parity)
CPython distinguishes two comparison entry points (Objects/object.c):
- PyObject_RichCompare returns the raw __eq__ / __ne__ result; no
identity short-circuit
- PyObject_RichCompareBool returns bool; identity implies equality
(and inequality is false on identity), short-circuiting before
dispatch
Collection membership / equality (x in [x], [nan] == [nan], set/dict
comparisons) go through the bool variant and rely on the short-circuit.
RustPython's rich_compare_bool skipped the identity check, so a buggy
or raising __eq__ propagated even when the operand was the same object.
Add an identity short-circuit at the top of rich_compare_bool for Eq
(returns true) and Ne (returns false). Ordering ops fall through to
_cmp because Python does not guarantee reflexivity for </<=/>/>=.
_cmp itself is untouched, so == / != operators continue to invoke
__eq__ / __ne__ exactly as before.
Unmasks test_dictviews.TestDictViews.test_compare_error.
Verified byte-identical with CPython 3.14.4 across 53 scenarios in 10
categories (collection membership / equality / ordering ops / NaN /
hash collision / dict views / list-set-dict ops). 14-module regression
sweep ~2,402 tests passes with no regressions.
* Route proxy comparisons through PyObject_RichCompare
The identity short-circuit added to rich_compare_bool exposed a latent
bug in three proxy types (weakref, weakproxy, mappingproxy): they were
delegating their __eq__ to the bool variant on referents, while CPython
uses PyObject_RichCompare so the referent's __eq__ runs even when the
referents share identity.
Fixes test_weak_keyed_cascading_deletes which depends on key __eq__
firing during dict deletion to trigger a side-effect that mutates the
key list.
PEP 479 wraps a StopIteration raised from a generator/coroutine into a
RuntimeError. CPython sets both __cause__ and __context__ of the new
RuntimeError to the original StopIteration (the same object). RustPython's
wrapping sites set __cause__ only, leaving __context__ as None.
set___cause__ already toggles __suppress_context__ to true, so the
user-facing traceback is unchanged — only assertions that inspect
__context__ directly observe the gap.
Three sites all wrap StopIteration / StopAsyncIteration into RuntimeError
for PEP 479. Set __context__ to the same exception instance before
setting __cause__ at each:
- frame.rs::StopIterationError intrinsic (generator yield-expression path)
- coroutine.rs generator __next__ StopIteration branch
- coroutine.rs async-generator __anext__ StopAsyncIteration branch
PyBaseExceptionRef is Arc-backed, so e.clone() is a refcount bump;
'cause is context' holds, matching CPython.
Unmasks:
- test_generator_stop.TestPEP479.test_stopiteration_wrapping_context
- test_yield_from.TestInterestingEdgeCases.test_close_and_throw_work
- test_yield_from.TestInterestingEdgeCases.test_close_and_throw_raise_stop_iteration
- test_yield_from.TestInterestingEdgeCases.test_close_and_throw_yield
CPython's PySequence_Contains (Objects/abstract.c::_PySequence_IterSearch)
catches the TypeError from PyObject_GetIter when neither __contains__ nor
__iter__ is available, and re-raises with membership-test wording.
RustPython's PySequence::contains propagated the get_iter error verbatim,
so '1 in obj' produced 'X' object is not iterable — a literal but less
specific message.
Wrap the get_iter call in map_err: when the failure is a TypeError,
re-raise with byte-identical CPython wording. Other exception types pass
through unchanged.
Unmasks test_contains.TestContains.test_common_tests.
CPython's compile() (Python/Python-ast.c) accepts any object with
__index__ for the flags and optimize arguments. RustPython's CompileArgs
typed both fields as OptionalArg<PyIntRef>, so a class with only
__index__ raised 'TypeError: Expected type int but X found' during arg
binding.
bpo-36907's regression test (test_call.test_fastcall_clearing_dict)
exercises exactly this: an IntWithDict.__index__ that mutates
self.kwargs mid-call. CPython parses both args via __index__ and doesn't
crash even when the kwargs dict is cleared between binding and use.
Switch flags and optimize to OptionalArg<ArgPrimitiveIndex<i32>>, the
same helper already used by range, slice, bytes.__mul__, hex, oct, etc.
ArgPrimitiveIndex calls try_index (= __index__ protocol) and converts
to the requested primitive in one step, so the three call sites in
compile() simplify from .map_or(Ok(d), |v| v.try_to_primitive(vm))? to
.map_or(d, |v| v.value).
Unmasks test_call.FastCallTests.test_fastcall_clearing_dict.
* Make mappingproxy's nb_or slot symmetric for dict | mp
CPython's mappingproxy_or (Objects/descrobject.c) unwraps a mappingproxy
on either side of '|' to its underlying mapping and delegates back to
PyNumber_Or. This makes 'dict | mp', 'mp | dict', and 'mp | mp' all
produce a plain dict result without dict's own nb_or having to know
about mappingproxy.
RustPython's 'or' slot in 'AsNumber for PyMappingProxy' only handled
the case where the left operand is a mappingproxy. When the slot fired
with a=dict, b=mp (because dict.nb_or returned NotImplemented), it
returned NotImplemented again — so 'dict | mp' raised TypeError. This
propagated to 'UserDict | mappingproxy' (which calls self.data | other).
Rewrite the slot to unwrap a mappingproxy on either side (or both)
before delegating to vm._or on the underlying mappings. The
inplace_or slot is unchanged — '|=' is still rejected.
Unmasks test_userdict.UserDictTest.test_mixed_or.
* Unmask test_types.MappingProxyTests.test_union
The slot fix in this PR also enables this test, which was marked
expectedFailure for the same dict|mappingproxy TypeError. Caught as
UNEXPECTED SUCCESS in CI.
CPython's defaultdict.__missing__ (Modules/_collectionsmodule.c::defdict_missing)
calls default_factory() first; if the factory's recursion already populated
self[key] while running, the existing value is preserved instead of being
overwritten.
RustPython ships a Python fallback at Lib/collections/_defaultdict.py
(the C _collections.defaultdict is not available). That fallback
unconditionally executed self[key] = val after the factory returned,
overwriting any value the recursive call had already stored.
Add a 'if key in self: return self[key]' guard before the assignment.
dict.__contains__ does not invoke __missing__, so there's no recursion
risk; in the common non-reentrant case the check is False and behavior
is unchanged.
Unmasks test_defaultdict.TestDefaultDict.test_factory_conflict_with_set_value.
CPython rejects digit-only format-string field names that overflow
Py_ssize_t at parse time with ValueError: Too many decimal digits in
format string (Python/string_parser.c::get_integer). RustPython's
FieldName::parse accepted any digit string usize::from_str could parse,
producing IndexError or KeyError at lookup instead.
Cap the parsed index at isize::MAX (Py_ssize_t::MAX on every platform)
inside FieldName::parse. Also reject digits-only strings whose value
overflows usize itself (caught when parse_usize returns None on an
all-digit input). A new FormatParseError::TooManyDecimalDigits maps to
the byte-identical CPython wording.
Unmasks test_str.StrTest.test_format_huge_item_number.
CPython rejects format-spec widths that exceed Py_ssize_t::MAX with
ValueError: Too many decimal digits in format string. RustPython's
FormatSpec::_parse only capped precision (via parse_precision); width
was accepted up to usize::MAX, so values like sys.maxsize + 1 silently
produced an effectively-ignored width.
Reject any width above i32::MAX with FormatSpecError::DecimalDigitsTooMany,
matching the existing precision cap and producing the byte-identical
ValueError wording.
Unmasks test_str.StrTest.test_format_huge_width.
CPython 3.14 fully removed the long-deprecated fallback from int() to
__trunc__. Classes that want int(x) support must implement __int__ or
__index__; math.trunc(x) continues to use __trunc__.
Delete the __trunc__ branch in PyObject::try_int so a class with only
__trunc__ now reaches the existing TypeError arm, matching CPython.
Unmasks test_int.IntTestCases.test_intconversion.
Closes#7450.
CPython's unicode_new_impl returns the PyObject_Str result as-is when
type == &PyUnicode_Type, only invoking unicode_subtype_new for actual
str subclasses. RustPython's PyStr::Constructor stripped the result via
Self::from(s.as_wtf8().to_owned()) and re-materialized through
into_ref_with_type, dropping the subclass type even when cls is exactly
str.
Add a slot_new branch that returns input.str(vm)? directly when cls is
str_type with no encoding. Subtype construction and the bytes-decoding
path are unchanged.
Unmasks test_str.StrTest.test_conversion (11 assertTypedEqual cases).
CPython's _abc._abc_init validates that __abc_tpflags__ doesn't combine
Py_TPFLAGS_SEQUENCE and Py_TPFLAGS_MAPPING (Modules/_abc.c::_abc_init).
RustPython's check_abc_tpflags copied the bits onto slots.flags without
that validation, so a class declaring both produced an inconsistent
type silently.
Validate the masked __abc_tpflags__ value against COLLECTION_FLAGS and
return a String error (which the existing new_heap_inner caller turns
into a TypeError) when both bits are present. Move the inheritance-skip
check after the self-attrs branch so a child overriding __abc_tpflags__
is still validated even when the parent already set one of the bits.
deque.__reduce__ passed None as the unpickle state, so a deque
subclass's __dict__ and __slots__ values were dropped across a pickle
round-trip. CPython's deque___reduce___impl
(Modules/_collectionsmodule.c::deque___reduce___impl) calls
_PyObject_GetState, which returns the dict (or a (dict, slots) tuple)
so subclass attributes survive.
Replace the placeholder with the result of __getstate__() on the
instance. object.__getstate__ already implements the dict / dict+slots
protocol, matching _PyObject_GetState.
* Enforce int_max_str_digits on int-to-str conversions
The str-to-int direction already enforced sys.get_int_max_str_digits()
via bytes_to_int; the int-to-str direction did not. CPython 3.14 enforces
both per PEP 644.
Adds check_int_to_str_digits helper in builtins::int (bit-count fast path
+ digit upper-bound from log10(2)), wired into the four Python-level
entry points: repr, the str fast path in protocol::object, int.__format__
(decimal/n/empty spec only — binary bases x/o/b are exempt per CPython),
and the DecimalD/I/U branches of vm::cformat for both str % and bytes %.
Unmasks 8 expectedFailure tests across test_int (max_str_digits, DoS
prevention, int_from_other_bases — each mirrored in IntSubclass),
test_ast (test_repr_large_input_crash) and test_reprlib (test_numbers).
Boundary cases (4299/4300/4301 digits at limit=4300) match CPython 3.14.4.
* Skip int-to-str DoS test on platforms without time.get_clock_info
The test_denial_of_service_prevented_int_to_str regression test uses
support.Stopwatch, which calls time.get_clock_info('monotonic'). In
RustPython that function is gated to unix/windows targets only, so on
wasm32-wasip1 it surfaces as AttributeError and breaks the wasm-wasi CI.
Guard the test with skipUnless(hasattr(time, 'get_clock_info'), ...) so
it runs everywhere it can and is skipped on wasm.
Also narrow is_decimal_int_format to Number(Case::Lower): 'N' is rejected
by format_int as UnknownFormatCode, so excluding it preserves that error
path instead of intercepting it with the digit-limit check.
* Add TODO: RUSTPYTHON marker to skipUnless reason
scripts/update_lib uses TODO: RUSTPYTHON markers inside unittest
decorator reason strings to identify and migrate custom RustPython
patches across CPython library updates.
* Use expectedFailureIf for wasm get_clock_info gap
skipUnless silently hides the test forever; expectedFailureIf surfaces
unexpected success once RustPython implements time.get_clock_info on
wasm, prompting marker removal.
The behavior already matched CPython (the slot conflict is detected,
the __doc__ delete is rejected); only the message text drifted.
- "__slots__ conflicts with a class variable" -> drop the stray "a"
to match CPython's "conflicts with class variable".
- "cannot delete '__doc__' attribute of type 'X'" -> insert "immutable"
before "type" to match CPython's wording (CPython surfaces the same
phrase even for user-defined classes since the descriptor refuses
the delete unconditionally).
CPython's staticmethod and classmethod set __func__ and copy wrapper
attributes (__doc__, __name__, etc.) only inside __init__
(Objects/funcobject.c::sm_init / cm_init). RustPython did this work in
slot_new and again in __init__, so subclasses that override __init__
without calling super().__init__() saw __func__ pointing at the
original callable instead of None.
Move the callable assignment and the wrapper-attribute copy into
Initializer::init; slot_new now just validates the signature and stores
None for the callable, matching the CPython contract.
Match CPython's "field 'X' is required for Y" error format when
required scalar AST fields receive None during ast_from_object
conversion. Previously these produced TypeError or a generic
"None disallowed in expression list" ValueError.
Fields covered (Alias.name, Arg.arg, Comprehension.target/iter,
Keyword.value, MatchCase.pattern, WithItem.context_expr,
YieldFrom.value): all now raise the CPython-compatible message
verbatim.
Add a get_node_field_required helper alongside the existing
get_node_field / get_node_field_opt and switch the eight call
sites that read these required scalar fields. The helper rejects
None with the proper ValueError before the value flows into the
type converter where Rust's strong typing would otherwise force a
generic catch-all error. Optional fields (read via
get_node_field_opt) and fields where None is valid (e.g.
Constant.value) are unaffected.
This complements the existing AST validator pass at
crates/vm/src/stdlib/_ast/validate.rs (hooked at _ast.rs:763):
the validator handles post-conversion semantic invariants
(ExprContext, empty body, cross-field rules) but cannot handle
required-scalar-field None because Rust's non-Option types reject
None at conversion time, before validation runs.
Verified byte-identical with CPython 3.14.4 for all seven probe
cases. test.test_ast.test_ast: 227 tests, expected failures
38 -> 36 (test_empty_yield_from and test_none_checks unmasked,
no regressions). test.test_compile: 0 regressions.
The _json decoder had two failure modes when a Python str value would
contain a lone surrogate (legal per the Python 3 str model):
1. Boundary UnicodeEncodeError: JsonScanner::Callable::call rejected
any input str with surrogates via try_into_utf8 before scanning
began.
2. Silent U+FFFD corruption: call_scan_once and parse_object's key
path called .to_string() on scanstring's Wtf8Buf output, which
routes through Wtf8::Display (lossy). Array values and dict keys
decoded from JSON \uXXXX escapes silently became U+FFFD.
Switch JsonScanner's five PyUtf8StrRef signatures to PyStrRef, drop
the entry-point try_into_utf8 call, and feed Wtf8Buf directly to
new_str instead of going through .to_string(). Key memoization now
uses HashMap<Wtf8Buf, PyStrRef> so surrogate-bearing keys survive
interning. parse_number takes &[u8] since JSON numbers are ASCII.
Extends the WTF-8 refactor pattern established in #7673 to the
decoder. machinery::scanstring already returns Wtf8Buf and is
unchanged.
Unmasks test_single_surrogate_decode. 214 tests in test.test_json
pass with no regressions. Decoder output verified byte-identical to
CPython 3.13.4 over 10,000 random fuzz cases (JSON docs containing
random surrogate escapes at root/list/dict positions, compared via
json.dumps(..., ensure_ascii=True, sort_keys=True)).
encode_basestring/encode_basestring_ascii took PyUtf8StrRef, so
json.dumps(str_with_lone_surrogate) raised UnicodeEncodeError at the
Python/Rust boundary before write_json_string ran. CPython's encoder
emits \uXXXX under ensure_ascii=True and passes raw WTF-8 otherwise.
Switch to PyStrRef + s.as_wtf8(), matching scanstring in the same file.
Rewrite write_json_string to accept &Wtf8 and iterate
code_point_indices, emitting \uXXXX for surrogates in ascii mode and
passing their bytes through otherwise. Stop escaping 0x7F in the
ensure_ascii=False path (matches py_encode_basestring). Return Wtf8Buf
via the checked from_bytes so invariant breaks panic instead of UB.
Fuzzing also exposed two pre-existing ESCAPE_CHARS typos: 0x0B was
"\u000" and 0x1B was "\u001" (both missing trailing 'b'). Fixed here.
Verified byte-identical with CPython 3.13.4 over 16 manual + 10,000
random fuzz cases. Full test.test_json: 214 tests, 0 failures, 0
unexpected successes. Unmasks test_ascii_non_printable_encode and
test_single_surrogate_encode. Decoder path is a follow-up.
All three @unittest.skip("TODO: RUSTPYTHON") markers in
datetimetester.py are stale — the tests pass as written on current
main. Probed every assertion in each test body against both CPython
3.13.4 and RustPython: outputs are byte-identical for all unicode
passthrough cases (emoji, lone surrogates, surrogate pairs, null
bytes) in test_strftime_special and for all 11 FixedOffset
comparisons (utcoffset, tzname, dst, hash equality) in test_zones.
Full test.test_datetime run after unmasking: 1803 tests, 0 failures,
0 unexpected successes, 463 skipped (was 469 — 6 test-class inherited
runs now execute successfully).
Part of the skip-test inventory in #7611.
Sibling fix to #7656 (which handled Vec<Expr>). The same catch-all
TypeError ("expected some sort of stmt, but got {}") in
Stmt::ast_from_object silently swallowed None, so compile() on an
AST with None in a statement-list field (ClassDef.body, Try.body,
For.body, etc.) raised TypeError where CPython raises
ValueError("None disallowed in statement list").
Add an is_none check before the catch-all, matching the Expr-side
shape introduced in #7656. Option<Stmt> positions are unaffected —
they short-circuit None earlier in node.rs.
Unmasks test_classdef (body=[None] sub-case). Full
test.test_ast.test_ast run: 227 tests, 0 unexpected successes, 38
expected failures (was 39).
* Raise ValueError for None in Expr lists to match CPython validator
CPython raises ValueError("None disallowed in expression list") when
compile() encounters None as an element in a Vec<Expr> position
(Set/List/Tuple.elts, BoolOp.values, Call.args, comprehension generators,
etc.). RustPython reached the catch-all TypeError("expected some sort of
expr, but got {}") branch instead, failing a dozen test_ast validator
tests.
Add an explicit is_none() check in Expr::ast_from_object before the
catch-all so None short-circuits with the CPython-compatible ValueError.
Option<Expr> positions (e.g. Dict.keys for **unpack) are unaffected
since node.rs Option<T>::ast_from_object handles None earlier.
Unmasks 12 skip / expectedFailure markers in
Lib/test/test_ast/test_ast.py:
- test_boolop, test_set, test_list, test_tuple
- test_call, test_delete, test_assign
- test_dict, test_dictcomp, test_generatorexp, test_listcomp, test_setcomp
Addresses part of the skip-test inventory in #7611.
The Stmt-level analog (body=[None] -> "None disallowed in statement list",
used by test_classdef) needs a sibling change in statement.rs; out of
scope here.
* Drop redundant .to_owned() on new_value_error argument
Matches the other new_value_error call sites in the file.
Co-authored-by: Shahar Naveh <50263213+ShaharNaveh@users.noreply.github.com>
---------
Co-authored-by: Shahar Naveh <50263213+ShaharNaveh@users.noreply.github.com>
* Fix stack overflow on deeply-nested JSON in json.loads()
json.loads() on a deeply-nested array or object payload (e.g.
'[' * 50000 + ']' * 50000) overflowed the native Rust stack and
crashed the interpreter process with SIGSEGV. CPython raises
RecursionError on the same input via _Py_EnterRecursiveCall in
Modules/_json.c.
The recursion lives in the mutual call chain:
JsonScanner::parse_object / parse_array
-> JsonScanner::call_scan_once
-> JsonScanner::parse_object / parse_array
Every descent funnels through call_scan_once, so wrapping its body
with vm.with_recursion covers both '{' and '[' paths (and their
mixed nesting) with a single guard.
Before:
./rustpython -c "import json; json.loads('[' * 50000 + ']' * 50000)"
-> SIGSEGV (exit 139)
After:
-> RecursionError: maximum recursion depth exceeded while
decoding a JSON object from a string
Verified:
- extra_tests/snippets/stdlib_json.py: all assertions pass
(includes 3 new regression cases: array, object, alternating
nesting at depth 100000)
- cargo run -- -m test test_json: 214 passed, 0 regressed
(9 skipped, 13 expected failures, all pre-existing)
- depth 500000 no longer crashes (RecursionError)
- shallow parsing unchanged
* Enable test_highly_nested_objects_decoding
Per @ShaharNaveh's review on #7632: this test was previously marked
`@unittest.skip("TODO: RUSTPYTHON; crashes")` because json.loads
would SIGSEGV on the 500_000-deep input. The recursion-guard added
in this PR makes it raise RecursionError like CPython, so the skip
decorator can be removed.
$ cargo run -- -m unittest \
test.test_json.test_recursion.TestCRecursion.test_highly_nested_objects_decoding \
test.test_json.test_recursion.TestPyRecursion.test_highly_nested_objects_decoding
...
Ran 2 tests in 0.825s
OK
$ cargo run -- -m test test_json
Ran 214 tests (7 skipped, 13 expected failures) — all pass.
Shahar Naveh