mirrors/RustPython
0
0 Fork 0
mirror of https://github.com/RustPython/RustPython.git synced 2026-10-28 18:59:04 +08:00
Code Activity

Align bytecode codegen structure with CPython 3.14 (#7588)

Browse Source 
* Align bytecode codegen structure with CPython 3.14

* Bytecode parity - constant folding, annotation ordering, superinstruction alignment

- Add BoolOp constant folding with short-circuit semantics in compile_expression
- Add constant truthiness evaluation for assert statement optimization
- Disable const collection/boolop folding in starred unpack and assignment contexts
- Move annotation block generation after body with AnnotationsPlaceholder splicing
- Reorder insert_superinstructions to run before push_cold_blocks (matching flowgraph.c)
- Lower LOAD_CLOSURE after superinstructions to avoid false LOAD_FAST_LOAD_FAST
- Add ToBool before PopJumpIf in comparisons and chained compare cleanup blocks
- Unify annotation dict building to always use incremental BuildMap + StoreSubscr
- Add TrueDivide constant folding for integer operands
- Fold constant sets to Frozenset (not Tuple) in try_fold_constant_collection
- Add PyVmBag for frozenset constant materialization in code objects
- Add remove_redundant_const_pop_top_pairs pass and peephole const+branch folding
- Emit Nop for skipped constant expressions and constant-true asserts
- Preserve comprehension local ordering by source-order bound name collection
- Simplify annotation scanning in symboltable (remove simple-name gate)

* Fix CI regressions in marshal and fast-local ops

* impl more

* Align bytecode codegen with CPython structure

* Bytecode parity - comprehension/except scope ordering, load_fast_borrow fixes

- Reorder comprehension symbol-table walk so the outermost iterator
  registers its sub_tables in the enclosing scope before the comp
  scope, and rescan elt/ifs in CPython's order. Codegen peeks past the
  outermost iterator's nested scopes to find the comprehension table.
- For plain try/except, emit handler sub_tables before the else block
  so codegen's linear sub_table cursor stays aligned.
- Rename `collect_simple_annotations` to `collect_annotations` and
  evaluate non-simple annotations during __annotate__ compilation to
  preserve source-order side effects while keeping the simple-name
  index stable.
- Dedupe equivalent code constants in `arg_constant` and add a
  structural equality check on `CodeObject`.
- Disable LOAD_FAST_BORROW for the tail end block when a try has a
  bare `except:` clause, and have `new_block` inherit the flag from
  the current block.
- Remove `cfg!(debug_assertions)` guard around the
  `optimize_load_fast_borrow` start-depth check so mismatches are
  handled (return instead of assert) in release builds.
- Collapse nop-only blocks that precede a return epilogue and hoist
  the prior line number into the next real instruction so the
  line table matches.
- Unmark now-passing `test_consts_in_conditionals`,
  `test_load_fast_unknown_simple`,
  `test_load_fast_known_because_already_loaded`, and PEP 646 f3/f4
  annotation checks.

* Bytecode parity - try/except line tracking, assert 0 shape

- In `compile_try_except`, drop the leading Nop and set the end
  block's source range from the last orelse/body statement so line
  events after the try fall on the right line.
- Recognise constant-false asserts as the direct-raise shape (no
  ToBool/PopJumpIfFalse) and flip the test assertion accordingly.
- Extend `remove_redundant_nops_in_blocks` to also look through a
  trailing nop before a return-epilogue pair (LoadConst/ReturnValue
  or LoadSmallInt/ReturnValue) so the epilogue keeps the correct
  line number.
- Rename `preds` to `predecessor_blocks` in the LOAD_FAST_BORROW
  disable pass and add a test-only `debug_late_cfg_trace` helper.
- Regenerate the `nested_double_async_with` snapshot: the tail
  reference to `stop_exc` now emits LOAD_FAST instead of
  LOAD_FAST_BORROW.

* Bytecode parity - iter folding, break/continue line, cold inlining

- Fold a constant list iterable into a constant tuple in for-loop
  iterable position, matching the CPython optimizer, and strip a
  redundant LIST_TO_TUPLE immediately before GET_ITER in the IR
  peephole pass.
- Emit a Nop at the break/continue source range before unwinding
  so line events land on the break/continue statement instead of
  the following instruction.
- Drop `propagate_disable_load_fast_borrow`; the forward propagation
  was over-zealous and the per-block inheritance in `new_block` plus
  the bare-except marker are enough.
- Relax `inline_small_or_no_lineno_blocks` so small exit blocks at
  the tail of a cold block are always inlined, not just return
  epilogues.
- Add codegen tests covering the LIST_TO_TUPLE/GET_ITER peephole and
  the late-CFG trace helper for a for-loop list-literal iterable.
This commit is contained in:
Jeong, YunWon authored and GitHub committed 2026-04-18 09:19:11 +09:00
1 parent a61d2e81c0
commit a04189fa34
26 files changed
+4651 -1097
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

Lib/test/test_compile.py
Vendored
-2
View File
@@ -1084,7 +1084,6 @@ class TestSpecifics(unittest.TestCase):
self.assertEqual('RETURN_VALUE', opcodes[-1].opname)
self.assertEqual(None, opcodes[-1].argval)
@unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: 3 != 8
def test_false_while_loop(self):
def break_in_while():
while False:
@@ -1103,7 +1102,6 @@ class TestSpecifics(unittest.TestCase):
self.assertEqual('RETURN_VALUE', opcodes[-1].opname)
self.assertEqual(None, opcodes[1].argval)
@unittest.expectedFailure # TODO: RUSTPYTHON
def test_consts_in_conditionals(self):
def and_true(x):
return True and x
Lib/test/test_grammar.py
Vendored
-1
View File
@@ -304,7 +304,6 @@ class GrammarTests(unittest.TestCase):
" nonlocal x\n"
" x: int\n")
@unittest.expectedFailure # TODO: RUSTPYTHON
def test_var_annot_basic_semantics(self):
# execution order
with self.assertRaises(ZeroDivisionError):
Lib/test/test_patma.py
Vendored
-4
View File
@@ -3448,7 +3448,6 @@ class TestTracing(unittest.TestCase):
self.assertListEqual(self._trace(f, "go x"), [1, 2, 4, 5])
self.assertListEqual(self._trace(f, "spam"), [1, 2, 4, 6, 7])
@unittest.expectedFailure # TODO: RUSTPYTHON
def test_default_capture(self):
def f(command): # 0
match command.split(): # 1
@@ -3463,7 +3462,6 @@ class TestTracing(unittest.TestCase):
self.assertListEqual(self._trace(f, "go x"), [1, 2, 4, 5])
self.assertListEqual(self._trace(f, "spam"), [1, 2, 4, 6, 7])
@unittest.expectedFailure # TODO: RUSTPYTHON
def test_no_default(self):
def f(command): # 0
match command.split(): # 1
@@ -3476,7 +3474,6 @@ class TestTracing(unittest.TestCase):
self.assertListEqual(self._trace(f, "go x"), [1, 2, 4, 5])
self.assertListEqual(self._trace(f, "spam"), [1, 2, 4])
@unittest.expectedFailure # TODO: RUSTPYTHON
def test_only_default_wildcard(self):
def f(command): # 0
match command.split(): # 1
@@ -3487,7 +3484,6 @@ class TestTracing(unittest.TestCase):
self.assertListEqual(self._trace(f, "go x"), [1, 2, 3])
self.assertListEqual(self._trace(f, "spam"), [1, 2, 3])
@unittest.expectedFailure # TODO: RUSTPYTHON
def test_only_default_capture(self):
def f(command): # 0
match command.split(): # 1
Lib/test/test_peepholer.py
Vendored
+1 -5
View File
@@ -441,6 +441,7 @@ class TestTranforms(BytecodeTestCase):
self.check_lnotab(code)
@unittest.expectedFailure # TODO: RUSTPYTHON
def test_constant_folding_remove_nop_location(self):
sources = [
"""
@@ -785,7 +786,6 @@ class TestTranforms(BytecodeTestCase):
c, b, a = a, b, c
self.assertNotInBytecode(f, "SWAP")
@unittest.expectedFailure # TODO: RUSTPYTHON
def test_static_swaps_match_mapping(self):
for a, b, c in product("_a", "_b", "_c"):
pattern = f"{{'a': {a}, 'b': {b}, 'c': {c}}}"
@@ -793,7 +793,6 @@ class TestTranforms(BytecodeTestCase):
code = compile_pattern_with_fast_locals(pattern)
self.assertNotInBytecode(code, "SWAP")
@unittest.expectedFailure # TODO: RUSTPYTHON
def test_static_swaps_match_class(self):
forms = [
"C({}, {}, {})",
@@ -808,7 +807,6 @@ class TestTranforms(BytecodeTestCase):
code = compile_pattern_with_fast_locals(pattern)
self.assertNotInBytecode(code, "SWAP")
@unittest.expectedFailure # TODO: RUSTPYTHON
def test_static_swaps_match_sequence(self):
swaps = {"*_, b, c", "a, *_, c", "a, b, *_"}
forms = ["{}, {}, {}", "{}, {}, *{}", "{}, *{}, {}", "*{}, {}, {}"]
@@ -863,7 +861,6 @@ class TestMarkingVariablesAsUnKnown(BytecodeTestCase):
y = x + x
self.assertInBytecode(f, 'LOAD_FAST_BORROW_LOAD_FAST_BORROW')
@unittest.expectedFailure # TODO: RUSTPYTHON; RETURN_VALUE
def test_load_fast_unknown_simple(self):
def f():
if condition():
@@ -906,7 +903,6 @@ class TestMarkingVariablesAsUnKnown(BytecodeTestCase):
self.assertInBytecode(f5, 'LOAD_FAST_BORROW')
self.assertNotInBytecode(f5, 'LOAD_FAST_CHECK')
@unittest.expectedFailure # TODO: RUSTPYTHON; RETURN_VALUE
def test_load_fast_known_because_already_loaded(self):
def f():
if condition():
Lib/test/test_pep646_syntax.py
Vendored
+2 -2
View File
@@ -305,11 +305,11 @@ is *not* valid syntax.)
{'args': StarredB}
>>> def f3(*args: *b, arg1: int): pass
>>> f3.__annotations__ # TODO: RUSTPYTHON # doctest: +EXPECTED_FAILURE
>>> f3.__annotations__
{'args': StarredB, 'arg1': <class 'int'>}
>>> def f4(*args: *b, arg1: int = 2): pass
>>> f4.__annotations__ # TODO: RUSTPYTHON # doctest: +EXPECTED_FAILURE
>>> f4.__annotations__
{'args': StarredB, 'arg1': <class 'int'>}
>>> def f5(*args: *b = (1,)): pass # TODO: RUSTPYTHON # doctest: +EXPECTED_FAILURE
Lib/test/test_sys_settrace.py
Vendored
-4
View File
@@ -1957,8 +1957,6 @@ class JumpTestCase(unittest.TestCase):
finally:
output.append(5)
# TODO: RUSTPYTHON
@unittest.expectedFailure
@jump_test(1, 5, [], (ValueError, "into an 'except'"))
def test_no_jump_into_bare_except_block(output):
output.append(1)
@@ -1967,8 +1965,6 @@ class JumpTestCase(unittest.TestCase):
except:
output.append(5)
# TODO: RUSTPYTHON
@unittest.expectedFailure
@jump_test(1, 5, [], (ValueError, "into an 'except'"))
def test_no_jump_into_qualified_except_block(output):
output.append(1)
crates/codegen/src/compile.rs
+2310 -486
View File
File diff suppressed because it is too large. Load diff
crates/codegen/src/ir.rs
+1930 -350
View File
File diff suppressed because it is too large. Load diff
crates/codegen/src/snapshots/rustpython_codegen__compile__tests__bare_function_annotations_check_attribute_and_subscript_expressions.snap
Generated
+67
View File
@@ -0,0 +1,67 @@
---
source: crates/codegen/src/compile.rs
assertion_line: 12138
expression: "compile_exec(\"\\\ndef f(one: int):\n int.new_attr: int\n [list][0].new_attr: [int, str]\n my_lst = [1]\n my_lst[one]: int\n return my_lst\n\")"
---
1 0 RESUME (0)
1 LOAD_CONST (<code object __annotate__ at ??? file "source_path", line 1>): 1 0 RESUME (0)
1 LOAD_FAST_BORROW (0, format)
2 LOAD_SMALL_INT (2)
>> 3 COMPARE_OP (>)
4 CACHE
5 POP_JUMP_IF_FALSE (3)
6 CACHE
7 NOT_TAKEN
8 LOAD_COMMON_CONSTANT (NotImplementedError)
9 RAISE_VARARGS (Raise)
10 LOAD_CONST ("one")
11 LOAD_GLOBAL (0, int)
12 CACHE
13 CACHE
14 CACHE
15 CACHE
16 BUILD_MAP (1)
17 RETURN_VALUE
2 MAKE_FUNCTION
3 LOAD_CONST (<code object f at ??? file "source_path", line 1>): 1 0 RESUME (0)
2 1 LOAD_GLOBAL (0, int)
2 CACHE
3 CACHE
4 CACHE
5 CACHE
6 POP_TOP
3 7 LOAD_GLOBAL (2, list)
8 CACHE
9 CACHE
10 CACHE
11 CACHE
12 BUILD_LIST (1)
13 LOAD_SMALL_INT (0)
14 BINARY_OP ([])
15 CACHE
16 CACHE
17 CACHE
18 CACHE
19 CACHE
20 POP_TOP
4 21 LOAD_SMALL_INT (1)
22 BUILD_LIST (1)
23 STORE_FAST (1, my_lst)
5 24 LOAD_FAST_BORROW (1, my_lst)
25 POP_TOP
26 LOAD_FAST_BORROW (0, one)
27 POP_TOP
6 28 LOAD_FAST_BORROW (1, my_lst)
29 RETURN_VALUE
4 MAKE_FUNCTION
5 SET_FUNCTION_ATTRIBUTE(Annotate)
6 STORE_NAME (0, f)
7 LOAD_CONST (None)
8 RETURN_VALUE
crates/codegen/src/snapshots/rustpython_codegen__compile__tests__constant_true_if_pass_keeps_line_anchor_nop.snap
Generated
+10
View File
@@ -0,0 +1,10 @@
---
source: crates/codegen/src/compile.rs
assertion_line: 12222
expression: "compile_exec(\"\\\nif 1:\n pass\n\")"
---
1 0 RESUME (0)
1 NOP
2 2 LOAD_CONST (None)
3 RETURN_VALUE
crates/codegen/src/snapshots/rustpython_codegen__compile__tests__if_ands.snap
Generated
+3 -18
View File
@@ -1,23 +1,8 @@
---
source: crates/codegen/src/compile.rs
assertion_line: 10890
assertion_line: 11413
expression: "compile_exec(\"\\\nif True and False and False:\n pass\n\")"
---
1 0 RESUME (0)
1 LOAD_CONST (True)
2 POP_JUMP_IF_FALSE (11)
>> 3 CACHE
4 NOT_TAKEN
5 LOAD_CONST (False)
6 POP_JUMP_IF_FALSE (7)
>> 7 CACHE
8 NOT_TAKEN
9 LOAD_CONST (False)
10 POP_JUMP_IF_FALSE (3)
>> 11 CACHE
12 NOT_TAKEN
2 13 LOAD_CONST (None)
14 RETURN_VALUE
15 LOAD_CONST (None)
16 RETURN_VALUE
1 LOAD_CONST (None)
2 RETURN_VALUE
crates/codegen/src/snapshots/rustpython_codegen__compile__tests__if_mixed.snap
Generated
+3 -22
View File
@@ -1,27 +1,8 @@
---
source: crates/codegen/src/compile.rs
assertion_line: 10900
assertion_line: 11423
expression: "compile_exec(\"\\\nif (True and False) or (False and True):\n pass\n\")"
---
1 0 RESUME (0)
1 LOAD_CONST (True)
2 POP_JUMP_IF_FALSE (5)
>> 3 CACHE
4 NOT_TAKEN
>> 5 LOAD_CONST (False)
6 POP_JUMP_IF_TRUE (9)
>> 7 CACHE
8 NOT_TAKEN
>> 9 LOAD_CONST (False)
10 POP_JUMP_IF_FALSE (7)
11 CACHE
12 NOT_TAKEN
13 LOAD_CONST (True)
14 POP_JUMP_IF_FALSE (3)
15 CACHE
16 NOT_TAKEN
2 17 LOAD_CONST (None)
18 RETURN_VALUE
19 LOAD_CONST (None)
20 RETURN_VALUE
1 LOAD_CONST (None)
2 RETURN_VALUE
crates/codegen/src/snapshots/rustpython_codegen__compile__tests__if_ors.snap
Generated
+4 -17
View File
@@ -1,23 +1,10 @@
---
source: crates/codegen/src/compile.rs
assertion_line: 10880
assertion_line: 11039
expression: "compile_exec(\"\\\nif True or False or False:\n pass\n\")"
---
1 0 RESUME (0)
1 LOAD_CONST (True)
2 POP_JUMP_IF_TRUE (9)
>> 3 CACHE
4 NOT_TAKEN
>> 5 LOAD_CONST (False)
6 POP_JUMP_IF_TRUE (5)
7 CACHE
8 NOT_TAKEN
>> 9 LOAD_CONST (False)
10 POP_JUMP_IF_FALSE (3)
11 CACHE
12 NOT_TAKEN
1 NOP
2 13 LOAD_CONST (None)
14 RETURN_VALUE
15 LOAD_CONST (None)
16 RETURN_VALUE
2 2 LOAD_CONST (None)
3 RETURN_VALUE
crates/codegen/src/snapshots/rustpython_codegen__compile__tests__nested_double_async_with.snap
Generated
+85 -82
View File
@@ -1,6 +1,6 @@
---
source: crates/codegen/src/compile.rs
assertion_line: 10936
assertion_line: 11626
expression: "compile_exec(\"\\\nasync def test():\n for stop_exc in (StopIteration('spam'), StopAsyncIteration('ham')):\n with self.subTest(type=type(stop_exc)):\n try:\n async with egg():\n raise stop_exc\n except Exception as ex:\n self.assertIs(ex, stop_exc)\n else:\n self.fail(f'{stop_exc} was suppressed')\n\")"
---
1 0 RESUME (0)
@@ -13,7 +13,7 @@ expression: "compile_exec(\"\\\nasync def test():\n for stop_exc in (StopIter
>> 5 CACHE
6 CACHE
7 CACHE
8 LOAD_CONST ("spam")
>> 8 LOAD_CONST ("spam")
9 CALL (1)
>> 10 CACHE
11 CACHE
@@ -34,7 +34,7 @@ expression: "compile_exec(\"\\\nasync def test():\n for stop_exc in (StopIter
26 CACHE
27 STORE_FAST (0, stop_exc)
3 >> 28 LOAD_GLOBAL (4, self)
3 28 LOAD_GLOBAL (4, self)
29 CACHE
30 CACHE
31 CACHE
@@ -51,10 +51,10 @@ expression: "compile_exec(\"\\\nasync def test():\n for stop_exc in (StopIter
42 CACHE
43 LOAD_GLOBAL (9, NULL + type)
44 CACHE
45 CACHE
>> 45 CACHE
46 CACHE
47 CACHE
>> 48 LOAD_FAST (0, stop_exc)
48 LOAD_FAST_BORROW (0, stop_exc)
49 CALL (1)
50 CACHE
51 CACHE
@@ -105,7 +105,7 @@ expression: "compile_exec(\"\\\nasync def test():\n for stop_exc in (StopIter
94 END_SEND
95 POP_TOP
6 96 LOAD_FAST (0, stop_exc)
6 96 LOAD_FAST_BORROW (0, stop_exc)
97 RAISE_VARARGS (Raise)
2 98 END_FOR
@@ -139,115 +139,118 @@ expression: "compile_exec(\"\\\nasync def test():\n for stop_exc in (StopIter
125 POP_TOP
126 POP_TOP
127 POP_TOP
128 JUMP_FORWARD (48)
128 JUMP_FORWARD (3)
129 COPY (3)
130 POP_EXCEPT
131 RERAISE (1)
132 PUSH_EXC_INFO
132 NOP
7 133 LOAD_GLOBAL (12, Exception)
10 133 LOAD_GLOBAL (4, self)
134 CACHE
135 CACHE
136 CACHE
137 CACHE
138 CHECK_EXC_MATCH
139 POP_JUMP_IF_FALSE (28)
138 LOAD_ATTR (13, fail, method=true)
139 CACHE
140 CACHE
141 NOT_TAKEN
142 STORE_FAST (1, ex)
8 143 LOAD_GLOBAL (4, self)
141 CACHE
142 CACHE
143 CACHE
144 CACHE
145 CACHE
146 CACHE
147 CACHE
148 LOAD_ATTR (15, assertIs, method=true)
149 CACHE
150 CACHE
151 CACHE
152 CACHE
148 LOAD_FAST (0, stop_exc)
149 FORMAT_SIMPLE
150 LOAD_CONST (" was suppressed")
151 BUILD_STRING (2)
152 CALL (1)
153 CACHE
154 CACHE
155 CACHE
156 CACHE
157 CACHE
158 LOAD_FAST_LOAD_FAST (ex, stop_exc)
159 CALL (2)
156 POP_TOP
157 JUMP_FORWARD (45)
158 PUSH_EXC_INFO
7 159 LOAD_GLOBAL (14, Exception)
160 CACHE
161 CACHE
162 CACHE
163 POP_TOP
164 POP_EXCEPT
165 LOAD_CONST (None)
166 STORE_FAST (1, ex)
167 DELETE_FAST (1, ex)
168 JUMP_FORWARD (32)
169 RERAISE (0)
170 LOAD_CONST (None)
171 STORE_FAST (1, ex)
172 DELETE_FAST (1, ex)
173 RERAISE (1)
174 COPY (3)
175 POP_EXCEPT
176 RERAISE (1)
163 CACHE
164 CHECK_EXC_MATCH
165 POP_JUMP_IF_FALSE (32)
166 CACHE
167 NOT_TAKEN
168 STORE_FAST (1, ex)
10 177 LOAD_GLOBAL (4, self)
8 169 LOAD_GLOBAL (4, self)
170 CACHE
171 CACHE
172 CACHE
173 CACHE
174 LOAD_ATTR (17, assertIs, method=true)
175 CACHE
176 CACHE
177 CACHE
178 CACHE
179 CACHE
180 CACHE
181 CACHE
182 LOAD_ATTR (17, fail, method=true)
182 CACHE
183 CACHE
184 CACHE
185 CACHE
>> 186 CACHE
184 LOAD_FAST_LOAD_FAST (ex, stop_exc)
185 CALL (2)
186 CACHE
187 CACHE
188 CACHE
189 CACHE
190 CACHE
191 CACHE
192 LOAD_FAST_BORROW (0, stop_exc)
193 FORMAT_SIMPLE
194 LOAD_CONST (" was suppressed")
195 BUILD_STRING (2)
196 CALL (1)
197 CACHE
198 CACHE
199 CACHE
200 POP_TOP
>> 188 CACHE
189 POP_TOP
190 POP_EXCEPT
191 LOAD_CONST (None)
192 STORE_FAST (1, ex)
193 DELETE_FAST (1, ex)
194 JUMP_FORWARD (8)
195 LOAD_CONST (None)
196 STORE_FAST (1, ex)
197 DELETE_FAST (1, ex)
198 RERAISE (1)
199 RERAISE (0)
200 COPY (3)
201 POP_EXCEPT
202 RERAISE (1)
3 201 LOAD_CONST (None)
>> 202 LOAD_CONST (None)
203 LOAD_CONST (None)
204 CALL (3)
205 CACHE
206 CACHE
3 203 LOAD_CONST (None)
204 LOAD_CONST (None)
>> 205 LOAD_CONST (None)
206 CALL (3)
207 CACHE
208 POP_TOP
209 JUMP_BACKWARD (186)
210 CACHE
211 PUSH_EXC_INFO
212 WITH_EXCEPT_START
213 TO_BOOL
214 CACHE
215 CACHE
208 CACHE
209 CACHE
210 POP_TOP
211 JUMP_BACKWARD (188)
212 CACHE
213 PUSH_EXC_INFO
214 WITH_EXCEPT_START
215 TO_BOOL
216 CACHE
217 POP_JUMP_IF_TRUE (2)
217 CACHE
218 CACHE
219 NOT_TAKEN
220 RERAISE (2)
221 POP_TOP
222 POP_EXCEPT
219 POP_JUMP_IF_TRUE (2)
220 CACHE
221 NOT_TAKEN
222 RERAISE (2)
223 POP_TOP
224 POP_TOP
224 POP_EXCEPT
225 POP_TOP
226 JUMP_BACKWARD_NO_INTERRUPT(202)
227 COPY (3)
228 POP_EXCEPT
229 RERAISE (1)
226 POP_TOP
227 POP_TOP
228 JUMP_BACKWARD (205)
229 CACHE
230 COPY (3)
231 POP_EXCEPT
232 RERAISE (1)
2 230 CALL_INTRINSIC_1 (StopIterationError)
231 RERAISE (1)
2 233 CALL_INTRINSIC_1 (StopIterationError)
234 RERAISE (1)
2 MAKE_FUNCTION
3 STORE_NAME (0, test)
crates/codegen/src/symboltable.rs
+71 -57
View File
@@ -571,10 +571,13 @@ impl SymbolTableAnalyzer {
}
newfree.extend(child_free);
}
if let Some(ann_free) = annotation_free {
// Propagate annotation-scope free names to this scope so
// implicit class-scope cells (__classdict__/__conditional_annotations__)
// can be materialized by drop_class_free when needed.
if let Some(ann_free) = annotation_free
&& symbol_table.typ == CompilerScope::Class
{
// Annotation-only free variables should not leak into function
// bodies. We only need to propagate them through class scopes so
// drop_class_free() can materialize implicit class cells when
// annotation scopes reference them.
newfree.extend(ann_free);
}
@@ -1569,48 +1572,42 @@ impl SymbolTableBuilder {
}) => {
// https://github.com/python/cpython/blob/main/Python/symtable.c#L1233
match &**target {
Expr::Name(ast::ExprName { id, .. }) if *simple => {
Expr::Name(ast::ExprName { id, .. }) => {
let id_str = id.as_str();
self.check_name(id_str, ExpressionContext::Store, *range)?;
if *simple {
self.check_name(id_str, ExpressionContext::Store, *range)?;
self.register_name(id_str, SymbolUsage::AnnotationAssigned, *range)?;
// PEP 649: Register annotate function in module/class scope
let current_scope = self.tables.last().map(|t| t.typ);
match current_scope {
Some(CompilerScope::Module) => {
self.register_name("__annotate__", SymbolUsage::Assigned, *range)?;
self.register_name(id_str, SymbolUsage::AnnotationAssigned, *range)?;
// PEP 649: Register annotate function in module/class scope
let current_scope = self.tables.last().map(|t| t.typ);
match current_scope {
Some(CompilerScope::Module) => {
self.register_name(
"__annotate__",
SymbolUsage::Assigned,
*range,
)?;
}
Some(CompilerScope::Class) => {
self.register_name(
"__annotate_func__",
SymbolUsage::Assigned,
*range,
)?;
}
_ => {}
}
Some(CompilerScope::Class) => {
self.register_name(
"__annotate_func__",
SymbolUsage::Assigned,
*range,
)?;
}
_ => {}
} else if value.is_some() {
self.check_name(id_str, ExpressionContext::Store, *range)?;
self.register_name(id_str, SymbolUsage::Assigned, *range)?;
}
}
_ => {
self.scan_expression(target, ExpressionContext::Store)?;
}
}
// Only scan annotation in annotation scope for simple name targets.
// Non-simple annotations (subscript, attribute, parenthesized) are
// never compiled into __annotate__, so scanning them would create
// sub_tables that cause mismatch in the annotation scope's sub_table index.
let is_simple_name = *simple && matches!(&**target, Expr::Name(_));
if is_simple_name {
self.scan_ann_assign_annotation(annotation)?;
} else {
// Still validate annotation for forbidden expressions
// (yield, await, named) even for non-simple targets.
let was_in_annotation = self.in_annotation;
self.in_annotation = true;
let result = self.scan_expression(annotation, ExpressionContext::Load);
self.in_annotation = was_in_annotation;
result?;
}
self.scan_ann_assign_annotation(annotation)?;
if let Some(value) = value {
self.scan_expression(value, ExpressionContext::Load)?;
}
@@ -1639,6 +1636,10 @@ impl SymbolTableBuilder {
let saved_in_conditional_block = self.in_conditional_block;
self.in_conditional_block = true;
self.scan_statements(body)?;
// Preserve source-order symbol analysis so `global`/`nonlocal`
// semantics match CPython, but reorder child scope storage to
// match the codegen order for plain try/except/else.
let body_subtables_len = self.tables.last().unwrap().sub_tables.len();
for handler in handlers {
let ExceptHandler::ExceptHandler(ast::ExceptHandlerExceptHandler {
type_,
@@ -1654,7 +1655,22 @@ impl SymbolTableBuilder {
}
self.scan_statements(body)?;
}
self.scan_statements(orelse)?;
if finalbody.is_empty() {
let handler_subtables = self
.tables
.last_mut()
.unwrap()
.sub_tables
.split_off(body_subtables_len);
self.scan_statements(orelse)?;
self.tables
.last_mut()
.unwrap()
.sub_tables
.extend(handler_subtables);
} else {
self.scan_statements(orelse)?;
}
self.scan_statements(finalbody)?;
self.in_conditional_block = saved_in_conditional_block;
}
@@ -2160,6 +2176,13 @@ impl SymbolTableBuilder {
});
}
assert!(!generators.is_empty());
let outermost = &generators[0];
// CPython evaluates the outermost iterator in the enclosing scope
// before entering the comprehension scope.
self.scan_expression(&outermost.iter, ExpressionContext::IterDefinitionExp)?;
// Comprehensions are compiled as functions, so create a scope for them:
self.enter_scope(
scope_name,
@@ -2195,36 +2218,27 @@ impl SymbolTableBuilder {
// Register the passed argument to the generator function as the name ".0"
self.register_name(".0", SymbolUsage::Parameter, range)?;
self.scan_expression(elt1, ExpressionContext::Load)?;
if let Some(elt2) = elt2 {
self.scan_expression(elt2, ExpressionContext::Load)?;
self.scan_expression(&outermost.target, ExpressionContext::Iter)?;
for if_expr in &outermost.ifs {
self.scan_expression(if_expr, ExpressionContext::Load)?;
}
let mut is_first_generator = true;
for generator in generators {
// Set flag for INNER_LOOP_CONFLICT check (only for inner loops, not the first)
if !is_first_generator {
self.in_comp_inner_loop_target = true;
}
for generator in &generators[1..] {
self.in_comp_inner_loop_target = true;
self.scan_expression(&generator.target, ExpressionContext::Iter)?;
self.in_comp_inner_loop_target = false;
if is_first_generator {
is_first_generator = false;
} else {
self.scan_expression(&generator.iter, ExpressionContext::IterDefinitionExp)?;
}
self.scan_expression(&generator.iter, ExpressionContext::IterDefinitionExp)?;
for if_expr in &generator.ifs {
self.scan_expression(if_expr, ExpressionContext::Load)?;
}
}
self.leave_scope();
if let Some(elt2) = elt2 {
self.scan_expression(elt2, ExpressionContext::Load)?;
}
self.scan_expression(elt1, ExpressionContext::Load)?;
// The first iterable is passed as an argument into the created function:
assert!(!generators.is_empty());
self.scan_expression(&generators[0].iter, ExpressionContext::IterDefinitionExp)?;
self.leave_scope();
Ok(())
}
crates/compiler-core/src/bytecode/instruction.rs
+2 -2
View File
@@ -1405,11 +1405,11 @@ impl InstructionMetadata for PseudoInstruction {
/// SETUP_FINALLY: +1 (exc)
/// SETUP_CLEANUP: +2 (lasti + exc)
/// SETUP_WITH: +1 (pops __enter__ result, pushes lasti + exc)
fn stack_effect_jump(&self, _oparg: u32) -> i32 {
fn stack_effect_jump(&self, oparg: u32) -> i32 {
match self {
Self::SetupFinally { .. } | Self::SetupWith { .. } => 1,
Self::SetupCleanup { .. } => 2,
_ => self.stack_effect(_oparg),
_ => self.stack_effect(oparg),
}
}
crates/vm/src/builtins/code.rs
+99 -9
View File
@@ -1,6 +1,6 @@
//! Infamous code object. The python class `code`
use super::{PyBytesRef, PyStrRef, PyTupleRef, PyType};
use super::{PyBytesRef, PyStrRef, PyTupleRef, PyType, set::PyFrozenSet};
use crate::common::lock::PyMutex;
use crate::{
AsObject, Context, Py, PyObject, PyObjectRef, PyPayload, PyRef, PyResult, VirtualMachine,
@@ -267,13 +267,6 @@ impl<'a> AsBag for &'a Context {
}
}
impl<'a> AsBag for &'a VirtualMachine {
type Bag = PyObjBag<'a>;
fn as_bag(self) -> PyObjBag<'a> {
PyObjBag(&self.ctx)
}
}
#[derive(Clone, Copy)]
pub struct PyObjBag<'a>(pub &'a Context);
@@ -348,6 +341,87 @@ impl ConstantBag for PyObjBag<'_> {
}
}
#[derive(Clone, Copy)]
pub struct PyVmBag<'a>(pub &'a VirtualMachine);
impl ConstantBag for PyVmBag<'_> {
type Constant = Literal;
fn make_constant<C: Constant>(&self, constant: BorrowedConstant<'_, C>) -> Self::Constant {
let vm = self.0;
let ctx = &vm.ctx;
let obj = match constant {
BorrowedConstant::Integer { value } => ctx.new_bigint(value).into(),
BorrowedConstant::Float { value } => ctx.new_float(value).into(),
BorrowedConstant::Complex { value } => ctx.new_complex(value).into(),
BorrowedConstant::Str { value } if value.len() <= 20 => {
ctx.intern_str(value).to_object()
}
BorrowedConstant::Str { value } => ctx.new_str(value).into(),
BorrowedConstant::Bytes { value } => ctx.new_bytes(value.to_vec()).into(),
BorrowedConstant::Boolean { value } => ctx.new_bool(value).into(),
BorrowedConstant::Code { code } => {
PyCode::new_ref_with_bag(vm, code.map_clone_bag(self)).into()
}
BorrowedConstant::Tuple { elements } => {
let elements = elements
.iter()
.map(|constant| self.make_constant(constant.borrow_constant()).0)
.collect();
ctx.new_tuple(elements).into()
}
BorrowedConstant::Slice { elements } => {
let [start, stop, step] = elements;
let start_obj = self.make_constant(start.borrow_constant()).0;
let stop_obj = self.make_constant(stop.borrow_constant()).0;
let step_obj = self.make_constant(step.borrow_constant()).0;
use crate::builtins::PySlice;
PySlice {
start: Some(start_obj),
stop: stop_obj,
step: Some(step_obj),
}
.into_ref(ctx)
.into()
}
BorrowedConstant::Frozenset { elements } => {
let elements = elements
.iter()
.map(|constant| self.make_constant(constant.borrow_constant()).0);
PyFrozenSet::from_iter(vm, elements)
.unwrap()
.into_ref(ctx)
.into()
}
BorrowedConstant::None => ctx.none(),
BorrowedConstant::Ellipsis => ctx.ellipsis.clone().into(),
};
Literal(obj)
}
fn make_name(&self, name: &str) -> &'static PyStrInterned {
self.0.ctx.intern_str(name)
}
fn make_int(&self, value: BigInt) -> Self::Constant {
Literal(self.0.ctx.new_int(value).into())
}
fn make_tuple(&self, elements: impl Iterator<Item = Self::Constant>) -> Self::Constant {
Literal(
self.0
.ctx
.new_tuple(elements.map(|lit| lit.0).collect())
.into(),
)
}
fn make_code(&self, code: CodeObject) -> Self::Constant {
Literal(PyCode::new_ref_with_bag(self.0, code).into())
}
}
pub type CodeObject = bytecode::CodeObject<Literal>;
pub trait IntoCodeObject {
@@ -427,6 +501,22 @@ impl PyCode {
Ordering::Relaxed,
);
}
pub fn new_ref_with_bag(vm: &VirtualMachine, code: CodeObject) -> PyRef<Self> {
PyRef::new_ref(PyCode::new(code), vm.ctx.types.code_type.to_owned(), None)
}
pub fn new_ref_from_bytecode(vm: &VirtualMachine, code: bytecode::CodeObject) -> PyRef<Self> {
Self::new_ref_with_bag(vm, code.map_bag(PyVmBag(vm)))
}
pub fn new_ref_from_frozen<B: AsRef<[u8]>>(
vm: &VirtualMachine,
code: frozen::FrozenCodeObject<B>,
) -> PyRef<Self> {
Self::new_ref_with_bag(vm, code.decode(PyVmBag(vm)))
}
pub fn from_pyc_path(path: &std::path::Path, vm: &VirtualMachine) -> PyResult<PyRef<Self>> {
let name = match path.file_stem() {
Some(stem) => stem.display().to_string(),
@@ -1379,7 +1469,7 @@ impl ToPyObject for CodeObject {
impl ToPyObject for bytecode::CodeObject {
fn to_pyobject(self, vm: &VirtualMachine) -> PyObjectRef {
vm.ctx.new_code(self).into()
PyCode::new_ref_from_bytecode(vm, self).into()
}
}
crates/vm/src/frame.rs
+8 -7
View File
@@ -3451,15 +3451,16 @@ impl ExecutingFrame<'_> {
Ok(None)
}
Instruction::StoreFastLoadFast { var_nums } => {
let value = self.pop_value();
let locals = self.localsplus.fastlocals_mut();
// pop_value_opt: allows NULL from LoadFastAndClear restore paths.
let value = self.pop_value_opt();
let oparg = var_nums.get(arg);
let (store_idx, load_idx) = oparg.indexes();
locals[store_idx] = Some(value);
let load_value = locals[load_idx]
.clone()
.expect("StoreFastLoadFast: load slot should have value after store");
self.push_value(load_value);
let load_value = {
let locals = self.localsplus.fastlocals_mut();
locals[store_idx] = value;
locals[load_idx].clone()
};
self.push_value_opt(load_value);
Ok(None)
}
Instruction::StoreFastStoreFast { var_nums } => {
crates/vm/src/import.rs
+7 -2
View File
@@ -72,7 +72,7 @@ pub fn make_frozen(vm: &VirtualMachine, name: &str) -> PyResult<PyRef<PyCode>> {
vm.ctx.new_utf8_str(name),
)
})?;
Ok(vm.ctx.new_code(frozen.code))
Ok(PyCode::new_ref_from_frozen(vm, frozen.code))
}
pub fn import_frozen(vm: &VirtualMachine, module_name: &str) -> PyResult {
@@ -82,7 +82,12 @@ pub fn import_frozen(vm: &VirtualMachine, module_name: &str) -> PyResult {
vm.ctx.new_utf8_str(module_name),
)
})?;
let module = import_code_obj(vm, module_name, vm.ctx.new_code(frozen.code), false)?;
let module = import_code_obj(
vm,
module_name,
PyCode::new_ref_from_frozen(vm, frozen.code),
false,
)?;
debug_assert!(module.get_attr(identifier!(vm, __name__), vm).is_ok());
let origname = resolve_frozen_alias(module_name);
module.set_attr("__origname__", vm.ctx.new_utf8_str(origname), vm)?;
crates/vm/src/stdlib/_ast.rs
+1 -1
View File
@@ -776,7 +776,7 @@ pub(crate) fn compile(
let source_file = SourceFileBuilder::new(filename, text).finish();
let code = codegen::compile::compile_top(ast, source_file, mode, opts)
.map_err(|err| vm.new_syntax_error(&err.into(), None))?; // FIXME source
Ok(vm.ctx.new_code(code).into())
Ok(crate::builtins::PyCode::new_ref_from_bytecode(vm, code).into())
}
#[cfg(feature = "codegen")]
crates/vm/src/stdlib/_ctypes/simple.rs
+1 -1
View File
@@ -309,7 +309,7 @@ impl PyCSimpleType {
// Float types: accept numbers
Some(tc @ ("f" | "d" | "g"))
if (value.try_float(vm).is_ok() || value.try_int(vm).is_ok()) =>
if value.try_float(vm).is_ok() || value.try_int(vm).is_ok() =>
{
return create_simple_with_value(tc, &value);
}
crates/vm/src/stdlib/_imp.rs
+2 -2
View File
@@ -261,11 +261,11 @@ mod _imp {
name.clone().into_wtf8(),
)
};
let bag = crate::builtins::code::PyObjBag(&vm.ctx);
let bag = crate::builtins::code::PyVmBag(vm);
let code =
rustpython_compiler_core::marshal::deserialize_code(&mut &contiguous[..], bag)
.map_err(|_| invalid_err())?;
return Ok(vm.ctx.new_code(code));
return Ok(PyCode::new_ref_with_bag(vm, code));
}
import::make_frozen(vm, name.as_str())
}
crates/vm/src/stdlib/marshal.rs
+4 -4
View File
@@ -3,7 +3,7 @@ pub(crate) use decl::module_def;
#[pymodule(name = "marshal")]
mod decl {
use crate::builtins::code::{CodeObject, Literal, PyObjBag};
use crate::builtins::code::{CodeObject, Literal, PyVmBag};
use crate::class::StaticType;
use crate::common::wtf8::Wtf8;
use crate::{
@@ -382,7 +382,7 @@ mod decl {
impl<'a> marshal::MarshalBag for PyMarshalBag<'a> {
type Value = PyObjectRef;
type ConstantBag = PyObjBag<'a>;
type ConstantBag = PyVmBag<'a>;
fn make_bool(&self, value: bool) -> Self::Value {
self.0.ctx.new_bool(value).into()
@@ -412,7 +412,7 @@ mod decl {
self.0.ctx.new_tuple(elements.collect()).into()
}
fn make_code(&self, code: CodeObject) -> Self::Value {
self.0.ctx.new_code(code).into()
crate::builtins::PyCode::new_ref_with_bag(self.0, code).into()
}
fn make_stop_iter(&self) -> Result<Self::Value, marshal::MarshalError> {
Ok(self.0.ctx.exceptions.stop_iteration.to_owned().into())
@@ -472,7 +472,7 @@ mod decl {
.into())
}
fn constant_bag(self) -> Self::ConstantBag {
PyObjBag(&self.0.ctx)
PyVmBag(self.0)
}
}
crates/vm/src/types/slot.rs
+2 -2
View File
@@ -1394,9 +1394,9 @@ impl PyType {
SlotAccessor::SqLength => {
update_sub_slot!(as_sequence, length, sequence_len_wrapper, SeqLength)
}
// Sequence concat uses sq_concat slot - no generic wrapper needed
// (handled by number protocol fallback)
SlotAccessor::SqConcat | SlotAccessor::SqInplaceConcat if !ADD => {
// Sequence concat uses sq_concat slot - no generic wrapper needed
// (handled by number protocol fallback)
accessor.inherit_from_mro(self);
}
SlotAccessor::SqRepeat => {
crates/vm/src/vm/compile.rs
+2 -2
View File
@@ -25,8 +25,8 @@ impl VirtualMachine {
source_path: String,
opts: CompileOpts,
) -> Result<PyRef<PyCode>, CompileError> {
let code =
compiler::compile(source, mode, &source_path, opts).map(|code| self.ctx.new_code(code));
let code = compiler::compile(source, mode, &source_path, opts)
.map(|code| PyCode::new_ref_from_bytecode(self, code));
#[cfg(feature = "parser")]
if code.is_ok() {
self.emit_string_escape_warnings(source, &source_path);
scripts/dis_dump.py
+37 -15
View File
@@ -11,6 +11,8 @@ Usage:
"""
import argparse
import ast
import builtins
import dis
import json
import os
@@ -38,6 +40,7 @@ _JUMP_OPNAMES = frozenset(
"JUMP_IF_TRUE_OR_POP",
"JUMP_IF_FALSE_OR_POP",
"FOR_ITER",
"END_ASYNC_FOR",
"SEND",
}
)
@@ -93,6 +96,14 @@ def _normalize_argrepr(argrepr):
argrepr = re.sub(r"\\u([0-9a-fA-F]{4})", _unescape, argrepr)
argrepr = re.sub(r"\\U([0-9a-fA-F]{8})", _unescape, argrepr)
if argrepr.startswith("frozenset({") and argrepr.endswith("})"):
try:
values = ast.literal_eval(argrepr[len("frozenset(") : -1])
except Exception:
return argrepr
if isinstance(values, set):
parts = sorted(_normalize_argrepr(repr(value)) for value in values)
return f"frozenset({{{', '.join(parts)}}})"
return argrepr
@@ -100,10 +111,33 @@ _IS_RUSTPYTHON = (
hasattr(sys, "implementation") and sys.implementation.name == "rustpython"
)
if _IS_RUSTPYTHON and hasattr(dis, "_common_constants"):
common_constants = list(dis._common_constants)
while len(common_constants) < 7:
common_constants.append(
(builtins.list, builtins.set)[len(common_constants) - 5]
)
dis._common_constants = common_constants
# RustPython's ComparisonOperator enum values → operator strings
_RP_CMP_OPS = {0: "<", 1: "<=", 2: "==", 3: "!=", 4: ">", 5: ">="}
def _resolve_localsplus_name(code, arg):
if not isinstance(arg, int) or arg < 0:
return arg
nlocals = len(code.co_varnames)
if arg < nlocals:
return code.co_varnames[arg]
varnames_set = set(code.co_varnames)
nonparam_cells = [v for v in code.co_cellvars if v not in varnames_set]
extra = nonparam_cells + list(code.co_freevars)
idx = arg - nlocals
if 0 <= idx < len(extra):
return extra[idx]
return arg
def _resolve_arg_fallback(code, opname, arg):
"""Resolve a raw argument to its human-readable form.
@@ -113,8 +147,7 @@ def _resolve_arg_fallback(code, opname, arg):
return arg
try:
if "FAST" in opname:
if 0 <= arg < len(code.co_varnames):
return code.co_varnames[arg]
return _resolve_localsplus_name(code, arg)
elif opname == "LOAD_CONST":
if 0 <= arg < len(code.co_consts):
return _normalize_argrepr(repr(code.co_consts[arg]))
@@ -125,18 +158,7 @@ def _resolve_arg_fallback(code, opname, arg):
"LOAD_CLOSURE",
"MAKE_CELL",
):
# arg is localsplus index:
# 0..nlocals-1 = varnames (parameter cells reuse these slots)
# nlocals.. = non-parameter cells + freevars
nlocals = len(code.co_varnames)
if arg < nlocals:
return code.co_varnames[arg]
varnames_set = set(code.co_varnames)
nonparam_cells = [v for v in code.co_cellvars if v not in varnames_set]
extra = nonparam_cells + list(code.co_freevars)
idx = arg - nlocals
if 0 <= idx < len(extra):
return extra[idx]
return _resolve_localsplus_name(code, arg)
elif opname in (
"LOAD_NAME",
"STORE_NAME",
@@ -237,7 +259,7 @@ def _extract_instructions(code):
# 1. argval not in offset_to_idx (not a valid byte offset)
# 2. argval == arg (raw arg returned as-is, not resolved to offset)
# 3. For backward jumps: argval should be < current offset
is_backward = "BACKWARD" in inst.opname
is_backward = "BACKWARD" in inst.opname or inst.opname == "END_ASYNC_FOR"
argval_is_raw = inst.argval == inst.arg and inst.arg is not None
if target_idx is None or argval_is_raw:
target_idx = None # force recalculation