38 Commits
Author SHA1 Message Date
Jeong, YunWonanddot 2a09ee02bd Align compiler and VM bytecode contracts with CPython 3.15 (#9105)
* Align compiler and VM with final CPython 3.15 bytecode

Reconstruct the compiler-only baseline from validated487ae47 on the current main and frozen mapping prerequisitec78d811. Preserve the full source delta while dropping obsolete prerequisite ancestry; detailed old-to-new provenance is retained in the recovery artifact.

Assisted-by: OpenAI Codex: model identifier unavailable

* Fix final script validation, exception cleanup and private cache contracts

Replay reviewed compiler CI repairs 2191a9d, 1f1fe04, a2dacaf on the frozen current prerequisite. Intermediate compile/diagnostic corrections are combined where they address the same failure.

Assisted-by: OpenAI Codex: model identifier unavailable

* Align canonical compiler fixtures and generator frame state

Replay reviewed compiler CI repairs 476bb74 on the frozen current prerequisite. Intermediate compile/diagnostic corrections are combined where they address the same failure.

Assisted-by: OpenAI Codex: model identifier unavailable

* Preserve code-name owners and final warning and constant-fold contracts

Replay reviewed compiler CI repairs 6482adf on the frozen current prerequisite. Intermediate compile/diagnostic corrections are combined where they address the same failure.

Assisted-by: OpenAI Codex: model identifier unavailable

* Skip unavailable monitoring C API tests before tool registration

Replay reviewed compiler CI repairs 246c309 on the frozen current prerequisite. Intermediate compile/diagnostic corrections are combined where they address the same failure.

Assisted-by: OpenAI Codex: model identifier unavailable

* Assert exact final constant-list iterable folding

Replay reviewed compiler CI repairs 7ecedcf on the frozen current prerequisite. Intermediate compile/diagnostic corrections are combined where they address the same failure.

Assisted-by: OpenAI Codex: model identifier unavailable

* Align local monitoring events and exception lifecycle state with final 3.15

Replay reviewed compiler CI repairs ec206b1, 191fe58 on the frozen current prerequisite. Intermediate compile/diagnostic corrections are combined where they address the same failure.

Assisted-by: OpenAI Codex: model identifier unavailable

* Align 3.15 code objects and codegen with CPython

- Store co_consts, co_names, co_localsplusnames, co_localspluskinds,
  co_linetable, co_exceptiontable and co_name as objects on PyCode;
  build co_code and the varnames/cellvars/freevars tuples lazily in
  _co_cached, with co_code deoptimized through deopt_code.
- Move the compiler constant cache (_PyCode_ConstantKey,
  const_cache_insert) to builtins/code/const_cache.rs.
- Write every code field with w_object in marshal and pass the read
  field objects to MarshalBag::make_code through CodeFields.
- Emit RETURN_GENERATOR/POP_TOP in enter_scope and place SETUP_CLEANUP
  after them; drop the generator part of insert_prefix_instructions.
- Add PyCode::first_traceable (_co_firsttraceable) for
  _PyFrame_IsIncomplete and sys.monitoring.
- Follow the CPython DISABLE branches in sys.monitoring, the
  check_matched signature in warnings, the 3.15 % format error
  messages for str and bytes, code_returns_only_none, and the FOR_ITER
  stack analysis.
- Compute IMPORT_NAME flags with in_exception_handler.
- Replace the linetable/exceptiontable byte tests in codegen with
  dis(show_positions=True) insta snapshots in _opcode.
- Remove the opcode metadata verification tooling, byte-exact
  scripts, private cache tests and redundant snippets.
- Restore test_code.py and test_monitoring.py to v3.15.0 plus
  RUSTPYTHON markers.

Assisted-by: Claude Code:claude-opus-5-5

* Remove generator snippet checks covered by Lib/test

Drop extra_tests/snippets/generator_iterator_prefix.py. Its loop,
delegation, async, f_back and never-awaited checks are covered by
test_generators, test_coroutines, test_asyncgen, test_yield_from and
test_monitoring. Move the remaining checks into existing files:
genexpr prefix tracing and monitoring into stdlib_sys.py, and the
caller frame seen from the eager iterator into stdlib_sys_getframe.py.

Assisted-by: Claude Code:claude-opus-5-5

* Emit the generator prefix for top-level await modules

Module code compiled with PyCF_ALLOW_TOP_LEVEL_AWAIT and marked
ste_coroutine now starts with RETURN_GENERATOR; POP_TOP. The prefix and
RESUME emission move into emit_scope_prefix, shared by enter_scope and
the module entry points, as in codegen_enter_scope. Fixes
test_builtin.test_compile_top_level_await.

Assisted-by: Claude Code:claude-opus-5-5

* Allow the c_long cast in _opcode.stack_effect

c_long is i32 on Windows, where clippy reports the cast to i32 as
unnecessary.

Assisted-by: Claude Code:claude-opus-5-5

* Update malachite crates to 0.14.0

pymath requires malachite-bigint ^0, so the example projects, which
have no lock file, resolve it to 0.14.0 while the workspace pinned
0.13.0, and fail to build.

Assisted-by: Claude Code:claude-opus-5-5

---------

Co-authored-by: dot <dot@youknowone.org>
2026-10-14 22:38:42 +09:00
Shahar Naveh 26ff16c762 Update Lib/support to 3.14.7 + some tests (#8578)
* Update `test/support` to 3.14.7

* `test_compile.py`

* Update some tests
2026-08-22 20:56:18 +09:00
Jeong, YunWon a1ba6a662c Align compiler and AST behavior with CPython (#8138)
* Align compiler and AST behavior with CPython

Update the compiler pipeline, AST module support, and related VM compile/eval paths toward CPython 3.14 behavior.

Use upstream Ruff parser crates directly and keep RustPython-specific syntax preflight handling in the compile path.

Refresh opcode metadata, snapshots, and targeted tests for the aligned bytecode and AST behavior.

* fix

* Fix PR 8138 CI failures

* Fix match mapping wildcard scan complexity

* fixes

* Use Ruff runtime AST fields

* Minimize residual patch diff

* refactor error handling

* Remove _ast conversion context wrappers

* Clean up _ast conversion glue and compiler warning escalation

Rename the Node conversion parameter `ctx` to `vm` throughout the _ast
module.

Remove the unused compile_program, compile_program_single,
compile_block_expression, and compile_expression forwarders; compile_top
stays as the entry point.

Stash an escalated compiler SyntaxWarning so a non-SyntaxWarning category
propagates unchanged instead of being rewritten to SyntaxError, matching
PyErr_ExceptionMatches(SyntaxWarning) in compiler_warn. Drop the now-unused
CompileWarningError::into_codegen_error.

Replace the PositionalArguments two-variant enum with a struct holding a
shared range and a PositionalArgumentsKind enum.

Assisted-by: Claude

* Represent compiler flags as a CompilerFlags bitflags

Replace the loose PY_CF_*/CO_* i32 constants in compile_mode with a
CompilerFlags bitflags covering the PyCF_* flags and __future__
CO_FUTURE_* bits. The Python-visible ast.PyCF_* attribute values are
derived from it via bits(); parser input modes stay plain consts.

builtins::compile and the compile_string flag handling test flags
through CompilerFlags::contains instead of raw bit math.

Assisted-by: Claude

* Fix PR 8138 test regressions

* Fix PR 8138 CI failures

* Avoid shell interpolation in Linux deps action

* Fix review findings in compiler/AST changes

- frame.rs: compute the MATCH_CLASS class name lazily so a successful
  match no longer takes the type name lock or allocates; the
  __match_args__-not-a-tuple branch reuses it instead of recomputing.
- _ast/constant.rs: add the missing leading space to the with_recursion
  label ("maximum recursion depth exceeded during compilation").
- sys.set_int_max_str_digits: restore the "maxdigits must be 0 or larger
  than {n}" error text.
- codegen: set CO_MAXBLOCKS to 20 in both ir.rs and compile.rs.
- _ast/exception.rs: extract the duplicated excepthandler node guard
  into ensure_excepthandler_node.
- codegen: correct the validate_keywords doc comment.
- Drop redundant "CPython" qualifiers from comments added in this branch.

Assisted-by: Claude
2026-06-26 18:22:08 +09:00
Jeong, YunWon 5add3d6a69 Align codegen passes and opcode metadata with CPython (#7987)
* Share marshal ref table between code object and its internals

read_marshal_bytes, _str, _str_vec, _name_tuple, and _const_tuple now
take a shared ref table and resolve TYPE_REF / register FLAG_REF
entries. deserialize_code is split into a public wrapper and an inner
function that receives the ref table; deserialize_value_depth opens a
fresh inner ref space when it hits Type::Code, mirroring CPython's
behaviour of putting the code object itself at ref slot 0. Nested code
objects inside const tuples reuse the surrounding code's ref space via
the new read_const_value helper.

* Align PYC magic number, FORMAT_VERSION, and header check with CPython 3.14

PYC_MAGIC_NUMBER changes from 2994 to 3627, matching CPython 3.14's
pyc_magic_number_token (0x0a0d0e2b). marshal FORMAT_VERSION drops from
5 to 4 (the encoder/marshal.version value; the decoder already accepts
both). check_pyc_magic_number_bytes now compares all four magic bytes
instead of the first two.

* Accept CPython-tagged .pyc as read-only bytecode source

SourceFileLoader.get_code now also looks for .pyc files using
_RP_FALLBACK_CACHE_TAGS (currently ('cpython-314',)) in addition to
sys.implementation.cache_tag. The matched .pyc is only used for
reading; recompilation still writes to the RustPython-tagged path, so
CPython's .pyc is never overwritten. Source-stat / hash / timestamp
validation logic is unchanged.

* Apply rustfmt to marshal helpers

* Marshal PySlice from format version 4 instead of 5

CPython's marshal supports TYPE_SLICE from format version 4 onwards
and that is the default version. Rejecting slice dumps below version
5 made marshal.dumps(slice(...)) fail with the default version and
broke test.test_marshal.SliceTestCase.test_slice.

* Revert "Accept CPython-tagged .pyc as read-only bytecode source"

Lib/importlib/_bootstrap_external.py is CPython's own code copied
verbatim; local patches here defeat compatibility tracking. The
cpython-XX cache_tag fallback needs to live on the RustPython side
(Rust code or sys.implementation.cache_tag policy), not as edits to
the imported standard library.

This reverts commit 1fc426d0fb5fcdb50d35cad13bbb43e8f6ce1c7f.

* Set marshal FORMAT_VERSION to 5 to match CPython 3.14.5

Py_MARSHAL_VERSION is 5 in CPython 3.14.5 (Include/marshal.h:16) and
TYPE_SLICE serialization rejects version < 5 (Python/marshal.c:720).
Restore the same threshold and constant so marshal.version and the
slice-marshal gate match CPython.

* Thread marshal recursion depth through nested code objects

Code objects embedded in const-tuples reset the depth budget on each
recursion, so a hostile or pathological marshal stream of code-in-tuple-
in-code can blow the stack despite MAX_MARSHAL_STACK_DEPTH. Pass the
current depth through deserialize_code_inner and read_marshal_const_tuple
and decrement at each code-object/tuple boundary.

Also route dict keys through deserialize_value_after_header so TYPE_CODE
keys decode instead of failing with BadType.

* align compiler to CPython

* Align codegen with CPython compile.c

Rename CFG helpers and accessors to the names used in CPython's
compile.c (basicblock_next_instr, basicblock_last_instr,
basicblock_append_instructions, bb_has_fallthrough, is_jump,
make_cfg_traversal_stack, mark_warm/mark_cold, etc.). Drop the unused
boolop-folding gate, mark_cpython_cfg_label_block helper, and
ComprehensionLoopControl::iter_range field.

Track an is_coroutine flag on SymbolTable, set in async def, await, and
async comprehensions, and propagate it through non-generator
comprehensions per symtable_handle_comprehension().

Mark SetupCleanup/SetupFinally/SetupWith as has_arg pseudo-ops, mark
ForIter as a terminator, and add has_arg/has_const on AnyInstruction.
Fix Instruction::stack_effect_jump to delegate to the opcode's
stack_effect_jump rather than stack_effect.

* Align codegen IR with CPython CFG structures

* Match CPython CFG annotation offset arithmetic

* Propagate CPython CFG label translation errors

* Align CPython exception target labeling flow

* Propagate CPython CFG traversal stack allocation errors

* Match CPython optimize_load_fast allocation flow

* Propagate CPython basicblock allocation errors

* Propagate CPython redundant NOP cleanup errors

* Propagate CPython unused const cleanup errors

* Propagate CPython const folding errors

* Propagate CPython swaptimize allocation errors

* Match CPython list-to-tuple fold allocation

* Skip const folding on CPython allocation failures

* Skip subscript folding on CPython allocation failures

* Propagate CPython assembler allocation errors

* Propagate CPython localsplus allocation errors

* Propagate CPython localsplus setup allocation errors

* Propagate CPython jump label map allocation errors

* Propagate CPython instruction sequence allocation errors

* Propagate CPython instruction label allocation errors

* Guard CPython codegen block allocation

* Guard CPython label shadow allocation

* Propagate CPython label shadow allocation errors

* Align CPython c-array allocation updates

* Align CPython ref stack growth

* Match CPython CFG builder debug check

* Avoid Rust-only CFG append clone

* Reuse CPython cleared block slots

* Use CPython block append in copy_basicblock

* Match CPython cfg builder creation order

* Clear label map after CPython apply pass

* Match CPython cfg builder allocation check

* Propagate CPython c-array size errors

* Drop Rust-only label uniqueness check

* Drop Rust-only label shadow debug checks

* Propagate CFG block index overflow

* Propagate CFG label oparg overflow

* Model CPython basicblock instruction storage

* Drop Rust-only recorded CFG precheck

* Model CPython instruction sequence storage

* Propagate instruction sequence offset overflow

* Model CPython instruction sequence labels

* Match CPython jump offset arithmetic

* Match CPython exception table arithmetic

* Match CPython label index arithmetic

* Match CPython instruction offset casts

* Match CPython jump offset indexing

* Match CPython oparg locals casts

* Match CPython localsplus offset arithmetic

* Match CPython cell prefix indexing

* Match CPython C array growth arithmetic

* Match CPython label map allocation arithmetic

* Match CPython label map size tracking

* Use CPython label map size in sequence passes

* Assert CPython label map clearing invariants

* Match CPython label oparg assignment

* Match CPython compiler direct arithmetic

* Match CPython load fast local casts

* Match CPython load fast depth assert

* Match CPython resume depth flagging

* Match CPython stack depth arithmetic

* Drop Rust-only stack overflow error

* Return CPython stack depth directly

* Match CPython C array growth errors

* Match CPython instruction insert asserts

* Match CPython unreachable pseudo jump

* Match CPython CFG size guard

* Match CPython superinstruction assert

* Match CPython redundant jump assert

* Match CPython stackdepth errors

* Match CPython jump offset flow

* Match CPython assembler buffer defaults

* Match CPython bytecode emit growth

* Match CPython assembler entry growth

* Match CPython assembler growth overflow check

* Match CPython remove_unreachable structure

* Match CPython static swap flow

* Inline CPython code unit preprocessing

* Match CPython C array growth checks

* Match CPython label map size guard

* Match CPython load-fast flow

* Simplify CPython CFG condition flow

* Align exception fallthrough propagation

* Match CPython pseudo target table

* Match CPython annotations CFG assert

* Match CPython inverted op assert

* Match CPython many-locals guard

* Reject deopt opcodes in CFG stack effects

* Match CPython invalid stack effect error

* Test CPython deopt stack effect guard

* Match CPython load-fast extended-arg assert

* Match CPython instruction allocation asserts

* Match CPython basicblock last-instr asserts

* Match CPython opcode range asserts

* Assert CPython fallthrough line propagation invariant

* Assert CPython CFG target offset sign

* Assert CPython exception fallthrough invariant

* Assert CPython exception stack bounds

* Assert CPython traversal stack allocation

* Match CPython label-map allocation in shadow

* Mirror CPython label-map sentinel fill

* Match CPython CFG builder allocation asserts

* Match CPython exception stack structure

* Match CPython ref stack structure

* Match CPython CFG traversal stack structure

* Mirror CPython CFG traversal stack pointer

* Use CPython fixed exception handler stack

* Mirror CPython ref stack capacity field

* Match CPython swap optimizer scratch stack

* Align static swap helpers with CPython blocks

* Align swaptimize signature with CPython

* Match CPython redundant pair pass result

* Match CPython inline pass result

* Match CPython redundant NOP pass results

* Fix bytecode metadata after upstream rebase

* Match CPython opcode stack metadata

* Add CPython identifiers to cspell dictionary

Add CNOTAB, LNOTAB, ialloc, ioffset, iused, nblocks, ncellsused,
ncellvars, nextop, noffsets, nvars, swaptimize, untargeted to
.cspell.dict/cpython.txt for the new CFG/assembler code in
crates/codegen/src/ir.rs.

* Fix CI failures from bytecode-parity work

- clippy: drop redundant `test_` prefix on three test functions and
  remove an unnecessary `u32` cast in basicblock_clear_reuses_cpython_spare_slots_in_offset_order
- insta: regenerate nested_double_async_with snapshot to match the new
  CFG output that drops unreferenced labels after the redundant-NOP pass
- regrtest: drop `@expectedFailure` markers from test_func_args,
  test_meth_args (test_compile), test_disassemble_with,
  test_disassemble_try_finally (test_dis), and test_except_star
  (test_monitoring) which now pass

* Resync generated opcode metadata

Empty conf.toml since WithExceptStart and Setup{Cleanup,Finally,With}
stack effects already match CPython, so the TODO override entries are
stale and only cause CI hook diffs.

Regenerate opcode_metadata.rs and drop the matching SetupCleanup/
SetupFinally/SetupWith assertions on PseudoOpcode::has_arg(); their
`HAS_ARG` flag comes from pseudo definitions in bytecodes.c that the
upstream analyzer does not propagate through PseudoInstruction.properties,
so the generated has_arg() excludes them. has_target() still covers
these block-push pseudos via is_block_push().

* Drop is_block_push has_arg invariant

The CPython invariant `assert(OPCODE_HAS_ARG(op) || !IS_BLOCK_PUSH(op))`
relies on SETUP_{FINALLY,CLEANUP,WITH} carrying `HAS_ARG_FLAG` in
CPython's metadata. The autogen tool reads pseudo-opcode properties from
target instructions and does not propagate the pseudo's own
HAS_ARG flag, so PseudoOpcode::has_arg() omits these three opcodes.
Drop the debug_assert that fired inside py_freeze proc-macro expansion.

* Auto-generate has_eval_break and route AnyInstruction has_arg/has_const via macro

Add fn_has_eval_break to generate_rs_opcode_metadata.py using CPython's
Properties.eval_breaker, removing the hand-written matches! body for
Opcode::has_eval_break and PseudoOpcode::has_eval_break.

Forward has_arg/has_const from Instruction and PseudoInstruction to
their opcode, so AnyInstruction can use either_real_pseudo! like the
other has_* accessors instead of an open-coded match.
2026-05-28 09:19:11 +09:00
Jeong, YunWon 6440e64355 Align codegen metadata with CPython (#7952) 2026-05-23 20:16:03 +09:00
Changjoon 29d6f53ae8 Validate compile() filename type and dont_inherit __bool__ protocol (#7768)
Two CPython parity gaps in `compile()` argument handling:

1. **filename rejects only non-buffer-protocol types.** CPython's
   `PyUnicode_FSDecoder` accepts `str`, `bytes`, and objects with
   `__fspath__` only — `bytearray` / `memoryview` / `array.array` raise
   TypeError. RustPython's `FsPath::TryFromObject` impl falls back to
   any buffer-protocol object and silently converts to bytes, accepting
   them.

   ```python
   >>> compile('pass', bytearray(b'file.py'), 'exec')
   # CPython 3.14.4: TypeError
   # RustPython main: <code object>  ❌
   ```

   Change `CompileArgs::filename` from `FsPath` (which uses the
   permissive `TryFromObject` impl) to `PyObjectRef`, then call the
   strict `FsPath::try_from_path_like` at the top of `compile()`. Other
   `FsPath` consumers are unchanged.

2. **dont_inherit strict-checks the `bool` type.** CPython routes
   `dont_inherit` through `PyObject_IsTrue`, calling `__bool__` on
   arbitrary objects and propagating exceptions raised there. RustPython
   typed it `OptionalArg<bool>`, which rejects subclass-less objects at
   binding time before `__bool__` runs.

   ```python
   >>> class EvilBool:
   ...     def __bool__(self): raise ValueError('hi')
   >>> compile('pass', 'f', 'exec', dont_inherit=EvilBool())
   # CPython 3.14.4: ValueError('hi')
   # RustPython main: TypeError('Expected type bool, not EvilBool')  ❌
   ```

   Switch to `OptionalArg<ArgIntoBool>` — already used elsewhere in
   `builtins` (`all`, `any`, `print(..., flush=...)`).

Verified byte-identical with CPython 3.14.4 across:
- 6 filename types (str, bytes, bytearray, memoryview, list, int)
- 7 dont_inherit values (True/False/1/0/""/"yes"/None) plus the
  `__bool__`-raises-ValueError case

Unmasks `Lib/test/test_compile.py`:
- `test_compile_filename`
- `test_compile_filename_refleak`

`cargo run --release -- -m test test_compile test_builtin test_compileall
test_codeop test_ast` — 703 tests pass, 0 regressions. All 188
`extra_tests/snippets/*.py` pass under the CI feature set.
2026-05-04 10:27:53 +09:00
Jeong, YunWon ea6b608265 compiler parity for typealias (#7645) 2026-04-21 23:59:20 +09:00
Jeong, YunWon 9ee32a1b38 Bytecode parity - boolop, comprehension, CFG passes (#7631)
* Bytecode parity - boolop, comprehension, CFG passes

- Flatten nested same-op BoolOp and add IfExpr to jump_if
- Simplify is_name_imported to module-level only
- Enable inlined comprehensions in module/class scope
- Add TweakInlinedComprehensionScopes with fast_hidden tracking
- Track fb_range in FBlockInfo for with-statement line info
- Add constant subscript folding and unary Not folding
- Add emit_return_const_no_location for implicit returns
- Use JumpNoInterrupt for ternary/except jumps
- Add STACK_USE_GUIDELINE threshold for collection building
- Reorder CFG pipeline: inline small blocks earlier, resolve
  line numbers before cold block extraction
- Add redirect_empty_unconditional_jump_targets,
  reorder_conditional_chain_and_jump_back_blocks,
  materialize_empty_conditional_exit_targets,
  duplicate_shared_jump_back_targets passes
- Add borrow deoptimization for multi-handler, named-except,
  protected conditional tail, and protected import joins
- Run folding/optimization passes twice around peephole

* Bytecode parity - slice augassign, async comp inline

- Augmented assignment with two-part slices uses BINARY_SLICE/STORE_SLICE
- Inline async comprehensions (remove await/async guards)
- Inlined comprehension cleanup jump uses JumpNoInterrupt
- Class firstlineno uses first decorator line when decorated
- Recurse into nested functions for static attribute collection
- Fold unary positive complex constants (+0.0j)
- Add deoptimize_borrow_for_folded_nonliteral_exprs pass
- Add inline_single_predecessor_artificial_expr_exit_blocks pass
- Skip shared artificial expr-stmt exit blocks in small-block inlining
- Mark folded boolop tail as folded_from_nonliteral_expr

* apply reviews
2026-04-21 00:10:00 +09:00
Jeong, YunWon ce0e41de9d Bytecode parity - folding, class prologue, except cleanup
Constant folding:
- Add string/bytes multiply and bytes concat folding in IR
- Add constant subscript folding (str, bytes, tuple indexing)
- Delegate list/set constant folding to IR passes
- Stream big non-const list/set via BUILD+LIST_APPEND

Class/generic compilation:
- Reorder class body prologue: __type_params__ before __classdict__
- Build class function before .generic_base in generic classes
- Register .type_params/.generic_base symbols in proper scopes
- Use load_name/store_name helpers for synthetic variables

Return block handling:
- Only duplicate return-None epilogues, not arbitrary returns
- Add inline_pop_except_return_blocks pass
- Add duplicate_named_except_cleanup_returns pass

Other fixes:
- Fix eliminate_dead_stores to only collapse adjacent duplicates
- Skip STORE_FAST_LOAD_FAST superinstruction in generators after FOR_ITER
- Thread jumps through NOP-only blocks
- Transfer NOP line info to following unconditional jumps
- Extract scope_needs_conditional_annotations_cell helper
- Register __conditional_annotations__ for module future annotations
2026-04-18 15:17:56 +09:00
Jeong, YunWon a04189fa34 Align bytecode codegen structure with CPython 3.14 (#7588)
* Align bytecode codegen structure with CPython 3.14

* Bytecode parity - constant folding, annotation ordering, superinstruction alignment

- Add BoolOp constant folding with short-circuit semantics in compile_expression
- Add constant truthiness evaluation for assert statement optimization
- Disable const collection/boolop folding in starred unpack and assignment contexts
- Move annotation block generation after body with AnnotationsPlaceholder splicing
- Reorder insert_superinstructions to run before push_cold_blocks (matching flowgraph.c)
- Lower LOAD_CLOSURE after superinstructions to avoid false LOAD_FAST_LOAD_FAST
- Add ToBool before PopJumpIf in comparisons and chained compare cleanup blocks
- Unify annotation dict building to always use incremental BuildMap + StoreSubscr
- Add TrueDivide constant folding for integer operands
- Fold constant sets to Frozenset (not Tuple) in try_fold_constant_collection
- Add PyVmBag for frozenset constant materialization in code objects
- Add remove_redundant_const_pop_top_pairs pass and peephole const+branch folding
- Emit Nop for skipped constant expressions and constant-true asserts
- Preserve comprehension local ordering by source-order bound name collection
- Simplify annotation scanning in symboltable (remove simple-name gate)

* Fix CI regressions in marshal and fast-local ops

* impl more

* Align bytecode codegen with CPython structure

* Bytecode parity - comprehension/except scope ordering, load_fast_borrow fixes

- Reorder comprehension symbol-table walk so the outermost iterator
  registers its sub_tables in the enclosing scope before the comp
  scope, and rescan elt/ifs in CPython's order. Codegen peeks past the
  outermost iterator's nested scopes to find the comprehension table.
- For plain try/except, emit handler sub_tables before the else block
  so codegen's linear sub_table cursor stays aligned.
- Rename `collect_simple_annotations` to `collect_annotations` and
  evaluate non-simple annotations during __annotate__ compilation to
  preserve source-order side effects while keeping the simple-name
  index stable.
- Dedupe equivalent code constants in `arg_constant` and add a
  structural equality check on `CodeObject`.
- Disable LOAD_FAST_BORROW for the tail end block when a try has a
  bare `except:` clause, and have `new_block` inherit the flag from
  the current block.
- Remove `cfg!(debug_assertions)` guard around the
  `optimize_load_fast_borrow` start-depth check so mismatches are
  handled (return instead of assert) in release builds.
- Collapse nop-only blocks that precede a return epilogue and hoist
  the prior line number into the next real instruction so the
  line table matches.
- Unmark now-passing `test_consts_in_conditionals`,
  `test_load_fast_unknown_simple`,
  `test_load_fast_known_because_already_loaded`, and PEP 646 f3/f4
  annotation checks.

* Bytecode parity - try/except line tracking, assert 0 shape

- In `compile_try_except`, drop the leading Nop and set the end
  block's source range from the last orelse/body statement so line
  events after the try fall on the right line.
- Recognise constant-false asserts as the direct-raise shape (no
  ToBool/PopJumpIfFalse) and flip the test assertion accordingly.
- Extend `remove_redundant_nops_in_blocks` to also look through a
  trailing nop before a return-epilogue pair (LoadConst/ReturnValue
  or LoadSmallInt/ReturnValue) so the epilogue keeps the correct
  line number.
- Rename `preds` to `predecessor_blocks` in the LOAD_FAST_BORROW
  disable pass and add a test-only `debug_late_cfg_trace` helper.
- Regenerate the `nested_double_async_with` snapshot: the tail
  reference to `stop_exc` now emits LOAD_FAST instead of
  LOAD_FAST_BORROW.

* Bytecode parity - iter folding, break/continue line, cold inlining

- Fold a constant list iterable into a constant tuple in for-loop
  iterable position, matching the CPython optimizer, and strip a
  redundant LIST_TO_TUPLE immediately before GET_ITER in the IR
  peephole pass.
- Emit a Nop at the break/continue source range before unwinding
  so line events land on the break/continue statement instead of
  the following instruction.
- Drop `propagate_disable_load_fast_borrow`; the forward propagation
  was over-zealous and the per-block inheritance in `new_block` plus
  the bare-except marker are enough.
- Relax `inline_small_or_no_lineno_blocks` so small exit blocks at
  the tail of a cold block are always inlined, not just return
  epilogues.
- Add codegen tests covering the LIST_TO_TUPLE/GET_ITER peephole and
  the late-CFG trace helper for a for-loop list-literal iterable.
2026-04-18 09:19:11 +09:00
Shahar Naveh 38f5cbc9a9 Update test_compile.py from 3.14.4 (#7607) 2026-04-15 14:15:48 +09:00
Jeong, YunWon a293e11238 Bytecode parity (#7481)
* Bytecode parity

 Compiler changes:
    - Remove PUSH_NULL from decorator cal
ls, use CALL 0
    - Collect __static_attributes__ from self.xxx = patterns
    - Sort __static_attributes__ alphabetically
    - Move __classdict__ init before __doc__ in class prologue
    - Fold unary negative constants
    - Fold constant list/set literals (3+ elements)
    - Use BUILD_MAP 0 + MAP_ADD for 16+ dict pairs
    - Always run peephole optimizer for s
uperinstructions
    - Emit RETURN_GENERATOR for generator
 functions
    - Add is_generator flag to SymbolTabl
e

* Fix formatting and collapsible_if clippy warnings in compile.rs

* Fix clippy, fold_unary_negative chaining, and generator line tracing

- Replace irrefutable if-let with let for ExceptHandler
- Remove folded UNARY_NEGATIVE instead of replacing with NOP,
  enabling chained negation folding
- Initialize prev_line to def line for generators/coroutines
  to suppress spurious LINE events from preamble instructions
- Remove expectedFailure markers for now-passing tests

* Fix JIT StoreFastStoreFast, format, and remove expectedFailure markers

- Add StoreFastStoreFast handling in JIT instructions
- Fix cargo fmt in frame.rs
- Remove 11 expectedFailure markers for async jump tests in
  test_sys_settrace that now pass

* Fix peephole optimizer: use NOP replacement instead of remove()

Using remove() shifts instruction indices and corrupts subsequent
references, causing "pop stackref but null found" panics at runtime.
Replace folded/combined instructions with NOP instead, which are
cleaned up by the existing remove_nops pass.

* Revert peephole_optimize to use remove() for chaining support

NOP replacement broke chaining of peephole optimizations (e.g.
LOAD_CONST+TO_BOOL then LOAD_CONST+UNARY_NOT for 'not True').
The remove() approach is used by upstream and works correctly here;
fold_unary_negative keeps NOP replacement since it doesn't need chaining.

* Fix StoreFastStoreFast to handle NULL from LoadFastAndClear

StoreFast uses pop_value_opt() to allow NULL values from
LoadFastAndClear in inlined comprehension cleanup paths.
StoreFastStoreFast must do the same, otherwise the peephole
optimizer's fusion of two StoreFast instructions panics when
restoring unbound locals after an inlined comprehension.
2026-03-25 16:10:19 +09:00
CPython Developers d77c7f2519 Update test_compile from v3.14.3 2026-02-27 09:06:56 +09:00
Jeong, YunWon a059bd2958 impl sys.monitoring 2026-02-27 09:06:55 +09:00
Jeong, YunWon 553c26b64c Move sys.flags thread_inherit_context/context_aware_warnings to getset 2026-02-21 22:58:12 +09:00
Jeong YunWon 4b823ebaf5 mark test_dis failures 2026-01-26 10:40:23 +09:00
Jeong YunWon 8c73f3cb30 emit RESUME 2026-01-26 10:40:22 +09:00
Jeong, YunWonandgithub-actions[bot] 82a8f67c71 Replace custom opcodes with CPython standard sequences (#6794)
* Replace custom opcodes with standard sequences

Remove RustPython-specific opcodes (BuildListFromTuples,
BuildMapForCall, BuildSetFromTuples, BuildTupleFromTuples)
and replace their usage with CPython 3.14 standard opcode
sequences:

- BuildListFromTuples → BUILD_LIST + LIST_EXTEND loop
- BuildSetFromTuples → BUILD_SET + SET_UPDATE loop
- BuildTupleFromTuples → BUILD_LIST + LIST_EXTEND + CALL_INTRINSIC_1(ListToTuple)
- BuildMapForCall → DICT_MERGE loop

Implement missing opcodes:
- ListExtend: Extend list with iterable elements
- SetUpdate: Add iterable elements to set
- DictMerge: Merge dict with duplicate key checking

* Auto-generate: generate_opcode_metadata.py

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2026-01-20 00:34:46 +09:00
Terry Tianlin LuanandJeong, YunWon 609dbb1439 Downgraded skips in tests (#6716)
* Downgraded skips in tests

* Fixed failing tests

* Fixed test_ftplib + test_socket + test_ssl + test_threaded_import failures

* Removed comments about which tests are run in which environment

* Addressed PR comments

* Annotated skips on failing tests

* Removed unneeded tests

* Removed unneeded sys import from test_ftplib

* Added annotation to test_ftplib

* Readded skipIf to test_cleanup_with_symlink_modes with a more general ENV_POLLUTING_TESTS_WINDOWS

* Addressed PR comments

* Made changes to minimize diff in PR

* Apply suggestion from @youknowone

---------

Co-authored-by: Jeong, YunWon <69878+youknowone@users.noreply.github.com>
2026-01-15 21:43:22 +09:00
Jeong, YunWon eee360822c Rework compiler about exception handling (#6638)
* asyncgen

* fix coroutine

* rewrite compiler

* tests

* set pyc magic number
2026-01-04 09:15:38 +09:00
Jeong, YunWon 4a6e8fb29e Add except* support (#6530) 2025-12-26 21:54:23 +09:00
Jeong, YunWon 152d10bfea hybrid dis.py from CPython 3.13.10 2025-12-12 21:44:00 +09:00
Jeong, YunWon b666c52df9 code object linetable (#6150)
* Code.replace

* implement linetable
2025-09-16 21:49:54 +09:00
Jeong YunWon f8891ffe3a Fix test_compile 2025-08-01 09:17:08 +09:00
ilp-sysandDimitrisJim 8dbf7b27ef Fix test_path_like_objects (#4773)
Co-authored-by: DimitrisJim <d.f.hilliard@gmail.com>
2023-06-13 00:10:50 +09:00
Andrey Maltsev f5d4c9fc1c Update test_compile from Cpython v3.11.2 2023-04-02 13:04:58 +00:00
Jim Fasarakis-Hilliard 17c5361d43 Merge pull request #4554 from howjmay/co_stacksize
Add co_stacksize to code objects
2023-03-10 17:12:30 +02:00
Yang Hau 95e863cb6d Add co_stacksize to code objects 2023-03-10 21:28:56 +08:00
DimitrisJim 8405693325 Always load __debug__ as a const. 2023-02-26 09:58:27 +02:00
Nick Liu 58ff3e8aaf remove expectedFailure decorator in test_argument_handling 2022-12-29 22:49:26 +08:00
Jeong Yunwon d09209b2a8 mark failing test skip 2022-03-22 02:22:40 +09:00
Dean Li 5ee4fb899b test: use os_helper 2021-11-28 20:51:32 +08:00
Jeong YunWon 913b78ca44 Revert "Merge pull request #3433 from deantvv/test-update"
This reverts commit 9fa5c5ac66, reversing
changes made to e7fa32c687.
2021-11-17 17:06:51 +09:00
Dean Li 49a5805d11 test: use os_helper 2021-11-13 02:18:33 +00:00
jfh dce45825ae Allow any mapping for locals. 2021-10-15 17:34:52 +03:00
Padraic Fanning b7a79eb002 Mark erroring/failing tests 2021-10-02 18:34:23 -04:00
Padraic Fanning 56058844fe Skip panicking test 2021-10-02 16:16:00 -04:00
Padraic Fanning b95edf676f Add test_compile from CPython 3.8 2021-10-02 16:14:42 -04:00