* Align compiler futures and annotations with CPython
Handle Barry parsing consistently across AST, type-comment, REPL, and WASM paths. Preserve deferred annotation source and scope metadata, and make _symtable conversion cached and linear-time.
Assisted-by: Codex:gpt-5.4
* Fix f-string debug ranges and update Barry/annotate test expectations
Compute the debug f-string source range from the raw leading and
trailing text instead of the comment-stripped text, so a comment inside
a multi-line replacement field no longer shifts the emitted LOAD_CONST
location.
Restrict the obsolete-operator location shift to `<>` by also requiring
a `>` at the reported offset; without it any `ExpectedExpression` that
followed a `<` (`2 <;`) was reported one column to the left.
Add `CodeFlags::FUTURE_MASK` and use it in builtins, the shell, and the
WASM VM instead of repeating the eight-flag union, and collapse the
duplicated Barry diagnostic sequences into `BarrySource::diagnostic`.
Update the `__annotate__` disassembly snapshot to LOAD_FAST_BORROW and
replace the RustPython-only `barry_as_FLUFL` co_flags assertion in
extra_tests with the CPython-matching one.
Assisted-by: Claude Code:claude-opus-5
* Restore the annotated function in __annotate__ qualnames
Signature annotation scopes are compiled in the scope enclosing the
annotated function, so the function name has to be folded back into the
qualname: `f.__annotate__`, `C.m.__annotate__`,
`outer.<locals>.inner.__annotate__`. test_type_annotations
test_annotate_qualname covers this.
test_pyrepl test_future_barry_as_flufl now passes, so drop its
expectedFailure marker.
Assisted-by: Claude Code:claude-opus-5
* Pass safepoints in the attached-thread interpreter tests
`main_and_subinterpreter_run_sections_overlap` parked its workers on a
condvar while attached, and `busy_main_interpreter_does_not_block_subinterpreter`
looped over protocol calls that never reach a safepoint. An attached
thread that never reaches a safepoint cannot be suspended, so a
concurrent process-wide stop-the-world (from the gc test running in
parallel) never completes and the already-stopped sibling interpreter
stays stopped; the second worker then waits in `wait_while_suspended`
until the 30s deadline.
Both loops now call `vm.check_signals()` each iteration, and the condvar
wait uses `wait_timeout` so the lock is released between safepoints.
Reproduced on Linux (4 CPUs): the two tests with `--test-threads=2`
failed 2/40 before and 0/40 after; the full `rustpython-vm` lib binary
failed 1/30 before and 0/30 after.
Assisted-by: Claude Code:claude-opus-5
* Report the obsolete `<>` operator from one place
`<>` was located twice: `BarrySource` recorded the token ranges it
rewrote to `!=`, and `cpython_parse_diagnostic_override` separately
peeked at the bytes around an `ExpectedExpression` location. Both
produced "invalid syntax" spanning the whole operator.
`prepare_barry_as_flufl_source` now records the operator ranges in the
non-Barry case too -- by plain text search, since nothing is rewritten
there -- so `BarrySource::invalid_legacy_operator` covers both, and the
byte-peeking helper is gone. It also defers to an unclosed bracket
earlier in the source, which the diagnostic chain it used to sit in did
for it.
A `<>` that starts a statement now spans both characters: the parser
stops at the `<` rather than one character in, which the byte peek did
not handle.
Assisted-by: Claude Code:claude-opus-5
* Drop a py object's payload before its class
`PyInner` declares `typ` before `payload`, so `drop_in_place` dropped the
class first and `PyAtomicRef::drop` left it null. A `PyWeak` payload is
still linked into the weakref list of the object it points at until its
own `Drop` unlinks it under the stripe lock, which happens after that.
`WeakRefList::add` walks the same list looking for a proxy to reuse and
reads `node.class()` off every node it passes, so it could dereference
the null class of a node on its way out.
`PyInner::dealloc` now runs the two destructors in the other order at all
three of its exit paths, with a compile-time check that no other field
has one.
Reproduced by the new test in about 0.1s, as the panic CI hit in the
c-api `weakrefobject::tests::new_proxy` run:
panicked at crates/vm/src/object/ext.rs:312:
unsafe precondition(s) violated: hint::unreachable_unchecked must
never be reached
Assisted-by: Claude Code:claude-opus-5
* Remove the private assignment-expression marker
Why:
extend_namedexpr_scope() already propagates assignment-expression
targets and records DEF_GLOBAL or DEF_NONLOCAL in comprehension scopes.
ASSIGNED_IN_COMPREHENSION remained only for conflict checks. Its private
bit overlaps CPython's packed LOCAL scope bit.
Changes:
- Register assignment-expression targets as ordinary assignments.
- Detect later target conflicts from existing declaration flags.
Assisted-by: Codex:gpt-5.6-sol
* Remove the private comprehension iterator flag
Why:
ITER duplicates the DEF_LOCAL and DEF_COMP_ITER facts already recorded
for comprehension targets. Its private bit also overlaps CPython's
packed scope field.
Changes:
- Use those flags for rebinding checks and local restoration around
inlined comprehensions.
Assisted-by: Codex:gpt-5.6-sol
* Preserve source names in named-expression diagnostics
Why:
Assignment-expression diagnostics exposed the mangled keys used for
symbol-table lookup. CPython reports identifiers as written in source.
Changes:
- Use source names in both comprehension conflict diagnostics.
- Enable test_named_expression_invalid_mangled_class_variables.
Assisted-by: Codex:gpt-5.6-sol
* Fix SymbolUsge::Iter
* fix os.rename
* fix sre
* Re-export rename from crate::posix on Windows
posix_windows.rs is mounted as the `posix` module on Windows, so
`crate::posix_windows` does not resolve.
Assisted-by: Claude
* Apply rustfmt to sre_engine tests
Assisted-by: Claude
* Update test expectations for the sre and tokenize fixes
- test_re: drop expectedFailure from test_word_boundaries,
test_possessive_quantifiers and test_bug_gh101955
- test_inspect, test_pydoc: drop expectedFailure from the signature
tests that now pass
- test_pdb: drop expectedFailure from the two file-modification tests
and +EXPECTED_FAILURE from the test_post_mortem_chained and
test_pdb_asynctask doctests
- test_unittest: mark test_autospec_on_bound_builtin_function as
expectedFailure; inspect.signature() now succeeds on time.ctime
because of its auto-generated __text_signature__
Assisted-by: Claude
* Fix clippy warning
* Fix clippy warnings surfaced by newer clippy
- exceptions.rs: {b:#02x} -> {b:#04x} (unused_format_specs); output
unchanged for decode-error bytes (always >= 0x80)
- binascii.rs: [b'\r', b'\n'] -> b"\r\n" (byte_str_slice)
- host_env, _io.rs, ssl.rs, pyexpat.rs: expect std_instead_of_core where
the suggested core::io items (ErrorKind, Cursor) are unstable (core_io);
use expect so the suppression is flagged for removal once core::io
stabilizes. build_posix_spawn_attrs co-gates the expect with the cfg
block so it is not left unfulfilled on platforms compiling it out.
Assisted-by: Claude
* Fix more
---------
Co-authored-by: Jeong YunWon <jeong@youknowone.org>
* Extract unicodedata core into rustpython-unicode crate
Move the vm-free Unicode character database access — the generated UCD 3.2.0 /
latest tables, their build-time generation, and the icu4x/unicode_names2 lookups
for category/bidirectional/combining/east_asian_width/mirrored/decomposition/
normalize/is_normalized/digit/decimal/numeric/name/lookup — into a new leaf crate
over char/CodePoint/&Wtf8. stdlib/unicodedata.rs keeps the UCD pyclass binding: it
extracts the code point, boxes results, and maps errors. The unicode data files,
build.rs table generation, and the icu_properties/icu_normalizer/unicode_names2
dependencies move out of stdlib.
Assisted-by: Claude
* Route str predicates, casefold, and printable through rustpython-unicode
Add classify/case/identifier modules to the shared crate over char/&str/&Wtf8:
isalpha/isalnum/isdecimal/isdigit/isnumeric/isspace/isprintable classification,
XID identifier predicates, and full-mapping casefold. vm/builtins/str.rs and
literal/char.rs now call these instead of icu_properties/icu_casemap directly.
String-level iteration with final-sigma handling (lower/upper/title/capitalize,
islower/isupper) stays in the runtime.
Assisted-by: Claude
* Route sre_engine char classes through rustpython-unicode
Add the regex module with the SRE character-class and case predicates
(is_word/is_space/is_digit, is_uni_* Unicode variants, ascii/locale/unicode case
folding). sre_engine/string.rs now forwards to it, keeping its public API and
behavior identical — including the ASCII-only is_uni_digit and the hardcoded
is_uni_space table.
Assisted-by: Claude
* Route \N{} name lookups through rustpython-unicode
codegen/string_parser.rs (\N{...} escapes) and common/encodings.rs (the
namereplace error handler) now resolve character names via rustpython-unicode
instead of depending on unicode_names2 directly, leaving the crate as the sole
owner of the name database.
Assisted-by: Claude
* Add differential Unicode sweep and shared-crate snippet test
tests/differential.rs sweeps the full 0..0x110000 range and compares each str
classification predicate against a committed CPython 3.14 reference dataset
(tests/data/cpython3.14_predicates.txt, produced by generate_reference.py). Code
points that differ only because the Rust std / icu4x build ships a later Unicode
release than CPython 3.14's 16.0.0 are recorded in
tests/data/version_skew_cpython3.14.txt, regenerable via
RUSTPYTHON_UNICODE_REGEN_SKEW=1; the regen refuses to record any
cpython=true/crate=false divergence, so only newly-assigned code points are
allowed. Any other divergence fails. Both data files use a run-length
`predicate start:end,...` encoding.
extra_tests/snippets/stdlib_unicode_shared.py exercises the routed surface end to
end (str predicates, casefold, identifiers, unicodedata, normalize, \N{}, and re
character classes) and passes identically on CPython 3.14 and RustPython.
Assisted-by: Claude
* Match regex \d to Unicode decimal digits
is_uni_digit previously matched only ASCII 0-9, so re's \d in Unicode mode
missed decimal digits like ٥ and ५. SRE_UNI_IS_DIGIT matches Py_UNICODE_ISDECIMAL
(category Nd), so route it through classify::is_decimal. Unmasks test_bug_6561 in
test_re and extends the shared-crate snippet with the Nd/Nl/No cases.
Assisted-by: Claude
* Replace shallow Unicode wrappers with re-exports
Address review feedback on the extraction.
Functions that only forwarded to the shared crate become `use ... as`
re-exports rather than hand-written wrappers:
- literal: is_printable is dropped; escape.rs calls
rustpython_unicode::classify::is_repr_printable directly.
- unicode::identifier::is_continue re-exports is_xid_continue.
- unicode::data::lookup_character re-exports unicode_names2::character.
- Drop the unused repr(u8) on DecompositionType.
The SRE character-class and case predicates move back into
sre_engine::string (their pre-extraction home) instead of living in a
unicode::regex module that only sre_engine and the vm _sre binding used;
is_uni_digit/is_uni_alnum still delegate to rustpython_unicode::classify.
engine.rs and _sre consume them from sre_engine::string again.
The unicode crate's tests move from a single root tests module in lib.rs
into per-module test submodules (case, classify, data, identifier,
normalize).
Assisted-by: Claude
* no-std
* Address review feedback: test module, dead variant, WTF-8 normalization
- differential.rs: move the sweep tests into a `mod tests` block and drop
the file-level allow of clippy::tests_outside_test_module and
std_instead_of_alloc; the test uses alloc collections instead.
- data.rs: remove the never-constructed DecompositionType::Canonical variant
and its allow(unused); compatibility decomposition never produces it and
canonical decomposition is handled through icu4x.
- normalize.rs: is_normalized now takes &Wtf8 and checks each UTF-8 run,
skipping lone surrogates, matching normalize's run-wise behavior.
- unicodedata.rs: pass as_wtf8() to is_normalized.
Assisted-by: Claude
* Align compiler and AST behavior with CPython
Update the compiler pipeline, AST module support, and related VM compile/eval paths toward CPython 3.14 behavior.
Use upstream Ruff parser crates directly and keep RustPython-specific syntax preflight handling in the compile path.
Refresh opcode metadata, snapshots, and targeted tests for the aligned bytecode and AST behavior.
* fix
* Fix PR 8138 CI failures
* Fix match mapping wildcard scan complexity
* fixes
* Use Ruff runtime AST fields
* Minimize residual patch diff
* refactor error handling
* Remove _ast conversion context wrappers
* Clean up _ast conversion glue and compiler warning escalation
Rename the Node conversion parameter `ctx` to `vm` throughout the _ast
module.
Remove the unused compile_program, compile_program_single,
compile_block_expression, and compile_expression forwarders; compile_top
stays as the entry point.
Stash an escalated compiler SyntaxWarning so a non-SyntaxWarning category
propagates unchanged instead of being rewritten to SyntaxError, matching
PyErr_ExceptionMatches(SyntaxWarning) in compiler_warn. Drop the now-unused
CompileWarningError::into_codegen_error.
Replace the PositionalArguments two-variant enum with a struct holding a
shared range and a PositionalArgumentsKind enum.
Assisted-by: Claude
* Represent compiler flags as a CompilerFlags bitflags
Replace the loose PY_CF_*/CO_* i32 constants in compile_mode with a
CompilerFlags bitflags covering the PyCF_* flags and __future__
CO_FUTURE_* bits. The Python-visible ast.PyCF_* attribute values are
derived from it via bits(); parser input modes stay plain consts.
builtins::compile and the compile_string flag handling test flags
through CompilerFlags::contains instead of raw bit math.
Assisted-by: Claude
* Fix PR 8138 test regressions
* Fix PR 8138 CI failures
* Avoid shell interpolation in Linux deps action
* Fix review findings in compiler/AST changes
- frame.rs: compute the MATCH_CLASS class name lazily so a successful
match no longer takes the type name lock or allocates; the
__match_args__-not-a-tuple branch reuses it instead of recomputing.
- _ast/constant.rs: add the missing leading space to the with_recursion
label ("maximum recursion depth exceeded during compilation").
- sys.set_int_max_str_digits: restore the "maxdigits must be 0 or larger
than {n}" error text.
- codegen: set CO_MAXBLOCKS to 20 in both ir.rs and compile.rs.
- _ast/exception.rs: extract the duplicated excepthandler node guard
into ensure_excepthandler_node.
- codegen: correct the validate_keywords doc comment.
- Drop redundant "CPython" qualifiers from comments added in this branch.
Assisted-by: Claude
* General nitpicks
* Move Compiler::constant_truthiness to a ConstantData method
* More code cleanup
* bytecode::CodeFlags -> CodeFlags
* More code clenaups
* Share marshal ref table between code object and its internals
read_marshal_bytes, _str, _str_vec, _name_tuple, and _const_tuple now
take a shared ref table and resolve TYPE_REF / register FLAG_REF
entries. deserialize_code is split into a public wrapper and an inner
function that receives the ref table; deserialize_value_depth opens a
fresh inner ref space when it hits Type::Code, mirroring CPython's
behaviour of putting the code object itself at ref slot 0. Nested code
objects inside const tuples reuse the surrounding code's ref space via
the new read_const_value helper.
* Align PYC magic number, FORMAT_VERSION, and header check with CPython 3.14
PYC_MAGIC_NUMBER changes from 2994 to 3627, matching CPython 3.14's
pyc_magic_number_token (0x0a0d0e2b). marshal FORMAT_VERSION drops from
5 to 4 (the encoder/marshal.version value; the decoder already accepts
both). check_pyc_magic_number_bytes now compares all four magic bytes
instead of the first two.
* Accept CPython-tagged .pyc as read-only bytecode source
SourceFileLoader.get_code now also looks for .pyc files using
_RP_FALLBACK_CACHE_TAGS (currently ('cpython-314',)) in addition to
sys.implementation.cache_tag. The matched .pyc is only used for
reading; recompilation still writes to the RustPython-tagged path, so
CPython's .pyc is never overwritten. Source-stat / hash / timestamp
validation logic is unchanged.
* Apply rustfmt to marshal helpers
* Marshal PySlice from format version 4 instead of 5
CPython's marshal supports TYPE_SLICE from format version 4 onwards
and that is the default version. Rejecting slice dumps below version
5 made marshal.dumps(slice(...)) fail with the default version and
broke test.test_marshal.SliceTestCase.test_slice.
* Revert "Accept CPython-tagged .pyc as read-only bytecode source"
Lib/importlib/_bootstrap_external.py is CPython's own code copied
verbatim; local patches here defeat compatibility tracking. The
cpython-XX cache_tag fallback needs to live on the RustPython side
(Rust code or sys.implementation.cache_tag policy), not as edits to
the imported standard library.
This reverts commit 1fc426d0fb5fcdb50d35cad13bbb43e8f6ce1c7f.
* Set marshal FORMAT_VERSION to 5 to match CPython 3.14.5
Py_MARSHAL_VERSION is 5 in CPython 3.14.5 (Include/marshal.h:16) and
TYPE_SLICE serialization rejects version < 5 (Python/marshal.c:720).
Restore the same threshold and constant so marshal.version and the
slice-marshal gate match CPython.
* Thread marshal recursion depth through nested code objects
Code objects embedded in const-tuples reset the depth budget on each
recursion, so a hostile or pathological marshal stream of code-in-tuple-
in-code can blow the stack despite MAX_MARSHAL_STACK_DEPTH. Pass the
current depth through deserialize_code_inner and read_marshal_const_tuple
and decrement at each code-object/tuple boundary.
Also route dict keys through deserialize_value_after_header so TYPE_CODE
keys decode instead of failing with BadType.
* align compiler to CPython
* Align codegen with CPython compile.c
Rename CFG helpers and accessors to the names used in CPython's
compile.c (basicblock_next_instr, basicblock_last_instr,
basicblock_append_instructions, bb_has_fallthrough, is_jump,
make_cfg_traversal_stack, mark_warm/mark_cold, etc.). Drop the unused
boolop-folding gate, mark_cpython_cfg_label_block helper, and
ComprehensionLoopControl::iter_range field.
Track an is_coroutine flag on SymbolTable, set in async def, await, and
async comprehensions, and propagate it through non-generator
comprehensions per symtable_handle_comprehension().
Mark SetupCleanup/SetupFinally/SetupWith as has_arg pseudo-ops, mark
ForIter as a terminator, and add has_arg/has_const on AnyInstruction.
Fix Instruction::stack_effect_jump to delegate to the opcode's
stack_effect_jump rather than stack_effect.
* Align codegen IR with CPython CFG structures
* Match CPython CFG annotation offset arithmetic
* Propagate CPython CFG label translation errors
* Align CPython exception target labeling flow
* Propagate CPython CFG traversal stack allocation errors
* Match CPython optimize_load_fast allocation flow
* Propagate CPython basicblock allocation errors
* Propagate CPython redundant NOP cleanup errors
* Propagate CPython unused const cleanup errors
* Propagate CPython const folding errors
* Propagate CPython swaptimize allocation errors
* Match CPython list-to-tuple fold allocation
* Skip const folding on CPython allocation failures
* Skip subscript folding on CPython allocation failures
* Propagate CPython assembler allocation errors
* Propagate CPython localsplus allocation errors
* Propagate CPython localsplus setup allocation errors
* Propagate CPython jump label map allocation errors
* Propagate CPython instruction sequence allocation errors
* Propagate CPython instruction label allocation errors
* Guard CPython codegen block allocation
* Guard CPython label shadow allocation
* Propagate CPython label shadow allocation errors
* Align CPython c-array allocation updates
* Align CPython ref stack growth
* Match CPython CFG builder debug check
* Avoid Rust-only CFG append clone
* Reuse CPython cleared block slots
* Use CPython block append in copy_basicblock
* Match CPython cfg builder creation order
* Clear label map after CPython apply pass
* Match CPython cfg builder allocation check
* Propagate CPython c-array size errors
* Drop Rust-only label uniqueness check
* Drop Rust-only label shadow debug checks
* Propagate CFG block index overflow
* Propagate CFG label oparg overflow
* Model CPython basicblock instruction storage
* Drop Rust-only recorded CFG precheck
* Model CPython instruction sequence storage
* Propagate instruction sequence offset overflow
* Model CPython instruction sequence labels
* Match CPython jump offset arithmetic
* Match CPython exception table arithmetic
* Match CPython label index arithmetic
* Match CPython instruction offset casts
* Match CPython jump offset indexing
* Match CPython oparg locals casts
* Match CPython localsplus offset arithmetic
* Match CPython cell prefix indexing
* Match CPython C array growth arithmetic
* Match CPython label map allocation arithmetic
* Match CPython label map size tracking
* Use CPython label map size in sequence passes
* Assert CPython label map clearing invariants
* Match CPython label oparg assignment
* Match CPython compiler direct arithmetic
* Match CPython load fast local casts
* Match CPython load fast depth assert
* Match CPython resume depth flagging
* Match CPython stack depth arithmetic
* Drop Rust-only stack overflow error
* Return CPython stack depth directly
* Match CPython C array growth errors
* Match CPython instruction insert asserts
* Match CPython unreachable pseudo jump
* Match CPython CFG size guard
* Match CPython superinstruction assert
* Match CPython redundant jump assert
* Match CPython stackdepth errors
* Match CPython jump offset flow
* Match CPython assembler buffer defaults
* Match CPython bytecode emit growth
* Match CPython assembler entry growth
* Match CPython assembler growth overflow check
* Match CPython remove_unreachable structure
* Match CPython static swap flow
* Inline CPython code unit preprocessing
* Match CPython C array growth checks
* Match CPython label map size guard
* Match CPython load-fast flow
* Simplify CPython CFG condition flow
* Align exception fallthrough propagation
* Match CPython pseudo target table
* Match CPython annotations CFG assert
* Match CPython inverted op assert
* Match CPython many-locals guard
* Reject deopt opcodes in CFG stack effects
* Match CPython invalid stack effect error
* Test CPython deopt stack effect guard
* Match CPython load-fast extended-arg assert
* Match CPython instruction allocation asserts
* Match CPython basicblock last-instr asserts
* Match CPython opcode range asserts
* Assert CPython fallthrough line propagation invariant
* Assert CPython CFG target offset sign
* Assert CPython exception fallthrough invariant
* Assert CPython exception stack bounds
* Assert CPython traversal stack allocation
* Match CPython label-map allocation in shadow
* Mirror CPython label-map sentinel fill
* Match CPython CFG builder allocation asserts
* Match CPython exception stack structure
* Match CPython ref stack structure
* Match CPython CFG traversal stack structure
* Mirror CPython CFG traversal stack pointer
* Use CPython fixed exception handler stack
* Mirror CPython ref stack capacity field
* Match CPython swap optimizer scratch stack
* Align static swap helpers with CPython blocks
* Align swaptimize signature with CPython
* Match CPython redundant pair pass result
* Match CPython inline pass result
* Match CPython redundant NOP pass results
* Fix bytecode metadata after upstream rebase
* Match CPython opcode stack metadata
* Add CPython identifiers to cspell dictionary
Add CNOTAB, LNOTAB, ialloc, ioffset, iused, nblocks, ncellsused,
ncellvars, nextop, noffsets, nvars, swaptimize, untargeted to
.cspell.dict/cpython.txt for the new CFG/assembler code in
crates/codegen/src/ir.rs.
* Fix CI failures from bytecode-parity work
- clippy: drop redundant `test_` prefix on three test functions and
remove an unnecessary `u32` cast in basicblock_clear_reuses_cpython_spare_slots_in_offset_order
- insta: regenerate nested_double_async_with snapshot to match the new
CFG output that drops unreferenced labels after the redundant-NOP pass
- regrtest: drop `@expectedFailure` markers from test_func_args,
test_meth_args (test_compile), test_disassemble_with,
test_disassemble_try_finally (test_dis), and test_except_star
(test_monitoring) which now pass
* Resync generated opcode metadata
Empty conf.toml since WithExceptStart and Setup{Cleanup,Finally,With}
stack effects already match CPython, so the TODO override entries are
stale and only cause CI hook diffs.
Regenerate opcode_metadata.rs and drop the matching SetupCleanup/
SetupFinally/SetupWith assertions on PseudoOpcode::has_arg(); their
`HAS_ARG` flag comes from pseudo definitions in bytecodes.c that the
upstream analyzer does not propagate through PseudoInstruction.properties,
so the generated has_arg() excludes them. has_target() still covers
these block-push pseudos via is_block_push().
* Drop is_block_push has_arg invariant
The CPython invariant `assert(OPCODE_HAS_ARG(op) || !IS_BLOCK_PUSH(op))`
relies on SETUP_{FINALLY,CLEANUP,WITH} carrying `HAS_ARG_FLAG` in
CPython's metadata. The autogen tool reads pseudo-opcode properties from
target instructions and does not propagate the pseudo's own
HAS_ARG flag, so PseudoOpcode::has_arg() omits these three opcodes.
Drop the debug_assert that fired inside py_freeze proc-macro expansion.
* Auto-generate has_eval_break and route AnyInstruction has_arg/has_const via macro
Add fn_has_eval_break to generate_rs_opcode_metadata.py using CPython's
Properties.eval_breaker, removing the hand-written matches! body for
Opcode::has_eval_break and PseudoOpcode::has_eval_break.
Forward has_arg/has_const from Instruction and PseudoInstruction to
their opcode, so AnyInstruction can use either_real_pseudo! like the
other has_* accessors instead of an open-coded match.
* Add `rapidhash` to list of dependencies
* Use `rapidhash::quality::RandomState` in `codegen` crate
* Use `rapidhash::quality::RandomState` in `stdlib` crate
* Use `rapidhash::quality::RandomState` in `vm` crate
* Remove `ahash` from lists of dependencies
* Align codegen CFG cleanup with CPython
* Align codegen bytecode with CPython 3.14
* Remove remove_jump_target_line_nops
The pass dropped a same-line NOP at the start of any jump-targeted block
whose body advanced to a later line, but CPython's basicblock_remove_redundant_nops
only consults the previous/next instruction inside the same block. The
extra pass deleted the else-body line trace anchor exercised by
test_nested_double_async_with.
Ignore test_conditional_break_finally_does_not_keep_break_cleanup_nop;
the break NOP lands in a separate block from the inlined finally body,
so same-line successor elision does not apply here.
* Keep except_handler blocks reachable in eliminate_unreachable_blocks
After convert_pseudo_ops lowers SETUP_FINALLY to a plain NOP, the only
remaining link from the try body to the except_handler block was the
per-instruction except_handler annotation. When earlier passes had
already removed every NOP that carried that annotation (e.g. an empty
try body with `pass`), the handler block became unreachable from the
entry block and its instructions were cleared, dropping the handler
entirely. Seed reachability with blocks already marked except_handler
so handler dispatch survives independent of the in-block annotation.
Also drop two expectedFailure markers in test_patma whose match-tracing
expectations now pass.
* Align bytecode CFG cleanup with CPython
* Add tests for Mode::BlockExpr and VirtualMachine::run_block_expr
test_block_expr_return_const() panics currently.
* Fix "tried to pop from empty stack" with BlockExpr and const
Constant expressions without side effects are optimized away unless in
an interactive mode[1]. This causes panic when last statement in a block
is a constant and Mode::BlockExpr is used as this constant value must be
returned to caller.
Example panic:
---- vm::python_run::tests::test_block_expr_return_const stdout ----
[crates/vm/src/frame.rs:9653:9] self = ExecutingFrame {
code: code: <code object <module> at ??? file "<embedded>", line 1>,
stack_len: 0,
}
thread 'vm::python_run::tests::test_block_expr_return_const' (2640752) panicked at crates/vm/src/frame.rs:9410:18:
tried to pop from empty stack
stack backtrace:
0: __rustc::rust_begin_unwind
at /rustc/59807616e1fa2540724bfbac14d7976d7e4a3860/library/std/src/panicking.rs:689:5
1: core::panicking::panic_fmt
at /rustc/59807616e1fa2540724bfbac14d7976d7e4a3860/library/core/src/panicking.rs:80:14
2: rustpython_vm::frame::ExecutingFrame::fatal
at ./src/frame.rs:9654:9
3: rustpython_vm::frame::ExecutingFrame::pop_stackref_opt
at ./src/frame.rs:9410:18
4: rustpython_vm::frame::ExecutingFrame::pop_stackref
at ./src/frame.rs:9420:18
5: rustpython_vm::frame::ExecutingFrame::pop_value
at ./src/frame.rs:9435:14
6: rustpython_vm::frame::ExecutingFrame::execute_instruction
at ./src/frame.rs:3391:34
7: rustpython_vm::frame::ExecutingFrame::run
at ./src/frame.rs:1623:31
8: rustpython_vm::frame::<impl rustpython_vm::object::core::Py<rustpython_vm::frame::Frame>>::run::{{closure}}
at ./src/frame.rs:1098:44
9: rustpython_vm::frame::<impl rustpython_vm::object::core::Py<rustpython_vm::frame::Frame>>::with_exec
at ./src/frame.rs:1093:9
10: rustpython_vm::frame::<impl rustpython_vm::object::core::Py<rustpython_vm::frame::Frame>>::run
at ./src/frame.rs:1098:14
11: rustpython_vm::vm::VirtualMachine::run_frame::{{closure}}
at ./src/vm/mod.rs:1201:44
12: rustpython_vm::vm::VirtualMachine::with_frame_impl::{{closure}}::{{closure}}
at ./src/vm/mod.rs:1610:60
13: rustpython_vm::vm::VirtualMachine::dispatch_traced_frame
at ./src/vm/mod.rs:1689:22
14: rustpython_vm::vm::VirtualMachine::with_frame_impl::{{closure}}
at ./src/vm/mod.rs:1610:22
15: rustpython_vm::vm::VirtualMachine::with_recursion
at ./src/vm/mod.rs:1547:9
16: rustpython_vm::vm::VirtualMachine::with_frame_impl
at ./src/vm/mod.rs:1572:14
17: rustpython_vm::vm::VirtualMachine::with_frame
at ./src/vm/mod.rs:1555:14
18: rustpython_vm::vm::VirtualMachine::run_frame
at ./src/vm/mod.rs:1201:20
19: rustpython_vm::vm::VirtualMachine::run_code_obj
at ./src/vm/mod.rs:1120:14
20: rustpython_vm::vm::python_run::<impl rustpython_vm::vm::VirtualMachine>::run_block_expr
at ./src/vm/python_run.rs:51:14
21: rustpython_vm::vm::python_run::tests::test_block_expr_return_const::{{closure}}
at ./src/vm/python_run.rs:190:47
22: rustpython_vm::vm::interpreter::Interpreter::enter::{{closure}}
at ./src/vm/interpreter.rs:366:39
23: rustpython_vm::vm::thread::set_current_vm::{{closure}}
at ./src/vm/thread.rs:107:9
24: std::thread::local::LocalKey<T>::try_with
at /home/im/.rustup/toolchains/stable-aarch64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/local.rs:513:12
25: std::thread::local::LocalKey<T>::with
at /home/im/.rustup/toolchains/stable-aarch64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/local.rs:477:20
26: rustpython_vm::vm::thread::set_current_vm
at ./src/vm/thread.rs:102:14
27: rustpython_vm::vm::thread::enter_vm
at ./src/vm/thread.rs:132:5
28: rustpython_vm::vm::interpreter::Interpreter::enter
at ./src/vm/interpreter.rs:366:9
29: rustpython_vm::vm::python_run::tests::test_block_expr_return_const
at ./src/vm/python_run.rs:188:23
30: rustpython_vm::vm::python_run::tests::test_block_expr_return_const::{{closure}}
at ./src/vm/python_run.rs:187:38
31: core::ops::function::FnOnce::call_once
at /home/im/.rustup/toolchains/stable-aarch64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:250:5
32: <fn() -> core::result::Result<(), alloc::string::String> as core::ops::function::FnOnce<()>>::call_once
at /rustc/59807616e1fa2540724bfbac14d7976d7e4a3860/library/core/src/ops/function.rs:250:5
[1] https://github.com/RustPython/RustPython/blob/883ce9d273db98b435eb3f51128925a1255ba88d/crates/codegen/src/compile.rs#L2990-L2997
* Align LOAD_FAST_BORROW analysis with CPython chain shape
Three changes that bring optimize_load_fast_borrow closer to CPython's
optimize_load_fast in flowgraph.c:
* ir.rs: split mark_cold into the CPython-style two passes. Phase 1
propagates "warm" from the entry block, phase 2 propagates "cold" from
except_handler blocks. Blocks reached by neither phase keep cold=false
and stay in their original b_next position, matching CPython's handling
of empty placeholders left by remove_unreachable (e.g. the inner_end of
a nested try/except whose incoming jumps were re-routed by optimize_cfg).
* ir.rs: in optimize_load_fast_borrow, push the fall-through successor
only when the current block has a last instruction (is_some_and).
Empty blocks now terminate fall-through propagation, matching the
`term != NULL` check in optimize_load_fast.
* compile.rs: add switch_to_new_or_reuse_empty() helper and use it in
compile_while. The helper reuses the current block when it is empty
and unlinked, mirroring USE_LABEL absorption in
cfg_builder_maybe_start_new_block. This stops a stray empty block
from appearing between e.g. a try/except end_block and the following
while loop header.
Four codegen tests that depended on the previous fall-through-through-
empty behavior are marked #[ignore] with TODO comments.
Also includes a handful of dictionary entries in .cspell.dict picked up
during the work.
* Interleave const fold passes per-block to match CPython
Mirror CPython's optimize_basic_block() (flowgraph.c) by walking each
block once in instruction order and trying tuple, list, set, unary, and
binop folding at each position before advancing. This replaces the
previous global-pass sequence where every fold_unary_constants pattern
in the whole CFG was registered before any tuple constant, leaving
negated literals like `-1` at co_consts positions earlier than CPython
produces (e.g. snippets.py: -1 at idx 280 vs CPython idx 726).
Changes:
- Extract `fold_unary_constant_at` and `fold_binop_constant_at` per-
position helpers from the existing global passes; the global passes
now call the helpers in a loop.
- Add `fold_constants_per_block` that walks each block to a fixed point,
trying all five folds at each instruction position.
- Call the new walker before the legacy global passes in
optimize_finalize so co_consts insertion order matches CPython's.
Measured on the full Lib tree: differing files 270 → 269; the only
newly matching file is `test/test_ast/snippets.py`, the case raised in
youknowone/RustPython#28.
* Inline small fast-return blocks only through unconditional jumps
`inline_small_fast_return_blocks` previously appended the target
`LOAD_FAST(_BORROW)/RETURN_VALUE` block's instructions onto any
predecessor whose fall-through eventually reached it, in addition to the
unconditional-jump case CPython handles in
`inline_small_or_no_lineno_blocks` (flowgraph.c:1210). CPython only
inlines through unconditional jumps, leaving fall-through predecessors
to reach the shared return block via the natural CFG layout. The extra
fall-through branch duplicated the return tail (e.g. `if/elif/return`
emitted two adjacent `LOAD_FAST_BORROW x; RETURN_VALUE` sequences).
Remove the fall-through inlining branch and keep only the
unconditional-jump path.
Measured on the full Lib tree: differing files 270 → 239 (-31), no new
regressions. Files newly matching include copy.py, argparse.py,
dataclasses.py, logging/__init__.py, pathlib/__init__.py, etc.
* Allow scope-exit/jump-back reorder within a shared except handler
`reorder_conditional_scope_exit_and_jump_back_blocks` previously skipped
any reorder where the conditional, scope-exit, or jump-back block had
an `except_handler` attached, even when all three shared the same
handler. CPython reorders these regardless of try/except context, as
long as the blocks stay within the same protected region. The over-
conservative guard left patterns like `try: for: if cond: return` with
the loop body's scope-exit ahead of the backedge, while CPython places
the backedge first and inverts the conditional.
Replace the `block_is_protected` triple-check with a single
`mismatched_protection` test: skip only when the three blocks do not
share the same `except_handler`. Same-handler reorders preserve the
protected range because every instruction's `except_handler` field
stays attached as `.next` pointers shift.
Measured on the full Lib tree: differing files 239 → 237; no new
regressions.
* Skip jump-over-cleanup reorder when target restarts an exception scope
reorder_jump_over_exception_cleanup_blocks was swapping a small
scope-exit target with a preceding cold cleanup chain even when the
target block began a fresh try (SETUP_FINALLY/SETUP_CLEANUP/SETUP_WITH).
The swap moved the next try's setup ahead of the prior handler's
cleanup_end/next_handler/cleanup_block, making the cleanup_body's
JUMP_FORWARD fall through directly to the cleanup_end and get elided as
redundant. The bytecode then lacked the JUMP_FORWARD that skips the
cleanup blocks and matched the prior handler's borrow tail incorrectly.
Skip the reorder when the target block contains any block-push pseudo
op so a new try's setup stays in source order. Re-enables the four
named/typed except-cleanup borrow tests that were marked #[ignore] in
commit 7481459ea.
* fix if block_idx == BlockIdx::NULL
* Remove unused rust impl for formatting dis output
* remove `examples/dis.rs`
* Added tests
* Update lock
* Try to set snapshot dir
* Remove verbose flag
* Regenerate snapshots after #7711
* Revert "Bump insta from 1.46.3 to 1.47.2 (#7706)"
This reverts commit e6d9ea6bfe.
* Debug info
* Show diff as well
* Show debug faster
* CI: true env
* Recert CI
* Add `CI: true` in ci emv
* Reapply "Bump insta from 1.46.3 to 1.47.2 (#7706)"
This reverts commit 693ca8cbe4d7885a81162a9be31e8bb567db885a.
* simplify macro
* trim on function side
* Force insta workspace root
* fix merge
CPython convention: top-level module / interactive / expression code
does not carry CO_NEWLOCALS or CO_OPTIMIZED. The per-scope mapping at
enter_scope::CompilerScope::Module already returns empty flags, but
Compiler::new seeded the root CodeInfo with CodeFlags::NEWLOCALS,
forcing module code into the NEWLOCALS arm of frame.rs:725-731 so
locals were allocated as a fresh empty dict instead of being bound to
globals (the correct semantics for exec(code, globals)).
Restore the seed to empty() so it matches the per-scope mapping and
CPython's compiler_enter_scope for module scope.
Shahar Naveh