* Share marshal ref table between code object and its internals
read_marshal_bytes, _str, _str_vec, _name_tuple, and _const_tuple now
take a shared ref table and resolve TYPE_REF / register FLAG_REF
entries. deserialize_code is split into a public wrapper and an inner
function that receives the ref table; deserialize_value_depth opens a
fresh inner ref space when it hits Type::Code, mirroring CPython's
behaviour of putting the code object itself at ref slot 0. Nested code
objects inside const tuples reuse the surrounding code's ref space via
the new read_const_value helper.
* Align PYC magic number, FORMAT_VERSION, and header check with CPython 3.14
PYC_MAGIC_NUMBER changes from 2994 to 3627, matching CPython 3.14's
pyc_magic_number_token (0x0a0d0e2b). marshal FORMAT_VERSION drops from
5 to 4 (the encoder/marshal.version value; the decoder already accepts
both). check_pyc_magic_number_bytes now compares all four magic bytes
instead of the first two.
* Accept CPython-tagged .pyc as read-only bytecode source
SourceFileLoader.get_code now also looks for .pyc files using
_RP_FALLBACK_CACHE_TAGS (currently ('cpython-314',)) in addition to
sys.implementation.cache_tag. The matched .pyc is only used for
reading; recompilation still writes to the RustPython-tagged path, so
CPython's .pyc is never overwritten. Source-stat / hash / timestamp
validation logic is unchanged.
* Apply rustfmt to marshal helpers
* Marshal PySlice from format version 4 instead of 5
CPython's marshal supports TYPE_SLICE from format version 4 onwards
and that is the default version. Rejecting slice dumps below version
5 made marshal.dumps(slice(...)) fail with the default version and
broke test.test_marshal.SliceTestCase.test_slice.
* Revert "Accept CPython-tagged .pyc as read-only bytecode source"
Lib/importlib/_bootstrap_external.py is CPython's own code copied
verbatim; local patches here defeat compatibility tracking. The
cpython-XX cache_tag fallback needs to live on the RustPython side
(Rust code or sys.implementation.cache_tag policy), not as edits to
the imported standard library.
This reverts commit 1fc426d0fb5fcdb50d35cad13bbb43e8f6ce1c7f.
* Set marshal FORMAT_VERSION to 5 to match CPython 3.14.5
Py_MARSHAL_VERSION is 5 in CPython 3.14.5 (Include/marshal.h:16) and
TYPE_SLICE serialization rejects version < 5 (Python/marshal.c:720).
Restore the same threshold and constant so marshal.version and the
slice-marshal gate match CPython.
* Thread marshal recursion depth through nested code objects
Code objects embedded in const-tuples reset the depth budget on each
recursion, so a hostile or pathological marshal stream of code-in-tuple-
in-code can blow the stack despite MAX_MARSHAL_STACK_DEPTH. Pass the
current depth through deserialize_code_inner and read_marshal_const_tuple
and decrement at each code-object/tuple boundary.
Also route dict keys through deserialize_value_after_header so TYPE_CODE
keys decode instead of failing with BadType.
* align compiler to CPython
* Align codegen with CPython compile.c
Rename CFG helpers and accessors to the names used in CPython's
compile.c (basicblock_next_instr, basicblock_last_instr,
basicblock_append_instructions, bb_has_fallthrough, is_jump,
make_cfg_traversal_stack, mark_warm/mark_cold, etc.). Drop the unused
boolop-folding gate, mark_cpython_cfg_label_block helper, and
ComprehensionLoopControl::iter_range field.
Track an is_coroutine flag on SymbolTable, set in async def, await, and
async comprehensions, and propagate it through non-generator
comprehensions per symtable_handle_comprehension().
Mark SetupCleanup/SetupFinally/SetupWith as has_arg pseudo-ops, mark
ForIter as a terminator, and add has_arg/has_const on AnyInstruction.
Fix Instruction::stack_effect_jump to delegate to the opcode's
stack_effect_jump rather than stack_effect.
* Align codegen IR with CPython CFG structures
* Match CPython CFG annotation offset arithmetic
* Propagate CPython CFG label translation errors
* Align CPython exception target labeling flow
* Propagate CPython CFG traversal stack allocation errors
* Match CPython optimize_load_fast allocation flow
* Propagate CPython basicblock allocation errors
* Propagate CPython redundant NOP cleanup errors
* Propagate CPython unused const cleanup errors
* Propagate CPython const folding errors
* Propagate CPython swaptimize allocation errors
* Match CPython list-to-tuple fold allocation
* Skip const folding on CPython allocation failures
* Skip subscript folding on CPython allocation failures
* Propagate CPython assembler allocation errors
* Propagate CPython localsplus allocation errors
* Propagate CPython localsplus setup allocation errors
* Propagate CPython jump label map allocation errors
* Propagate CPython instruction sequence allocation errors
* Propagate CPython instruction label allocation errors
* Guard CPython codegen block allocation
* Guard CPython label shadow allocation
* Propagate CPython label shadow allocation errors
* Align CPython c-array allocation updates
* Align CPython ref stack growth
* Match CPython CFG builder debug check
* Avoid Rust-only CFG append clone
* Reuse CPython cleared block slots
* Use CPython block append in copy_basicblock
* Match CPython cfg builder creation order
* Clear label map after CPython apply pass
* Match CPython cfg builder allocation check
* Propagate CPython c-array size errors
* Drop Rust-only label uniqueness check
* Drop Rust-only label shadow debug checks
* Propagate CFG block index overflow
* Propagate CFG label oparg overflow
* Model CPython basicblock instruction storage
* Drop Rust-only recorded CFG precheck
* Model CPython instruction sequence storage
* Propagate instruction sequence offset overflow
* Model CPython instruction sequence labels
* Match CPython jump offset arithmetic
* Match CPython exception table arithmetic
* Match CPython label index arithmetic
* Match CPython instruction offset casts
* Match CPython jump offset indexing
* Match CPython oparg locals casts
* Match CPython localsplus offset arithmetic
* Match CPython cell prefix indexing
* Match CPython C array growth arithmetic
* Match CPython label map allocation arithmetic
* Match CPython label map size tracking
* Use CPython label map size in sequence passes
* Assert CPython label map clearing invariants
* Match CPython label oparg assignment
* Match CPython compiler direct arithmetic
* Match CPython load fast local casts
* Match CPython load fast depth assert
* Match CPython resume depth flagging
* Match CPython stack depth arithmetic
* Drop Rust-only stack overflow error
* Return CPython stack depth directly
* Match CPython C array growth errors
* Match CPython instruction insert asserts
* Match CPython unreachable pseudo jump
* Match CPython CFG size guard
* Match CPython superinstruction assert
* Match CPython redundant jump assert
* Match CPython stackdepth errors
* Match CPython jump offset flow
* Match CPython assembler buffer defaults
* Match CPython bytecode emit growth
* Match CPython assembler entry growth
* Match CPython assembler growth overflow check
* Match CPython remove_unreachable structure
* Match CPython static swap flow
* Inline CPython code unit preprocessing
* Match CPython C array growth checks
* Match CPython label map size guard
* Match CPython load-fast flow
* Simplify CPython CFG condition flow
* Align exception fallthrough propagation
* Match CPython pseudo target table
* Match CPython annotations CFG assert
* Match CPython inverted op assert
* Match CPython many-locals guard
* Reject deopt opcodes in CFG stack effects
* Match CPython invalid stack effect error
* Test CPython deopt stack effect guard
* Match CPython load-fast extended-arg assert
* Match CPython instruction allocation asserts
* Match CPython basicblock last-instr asserts
* Match CPython opcode range asserts
* Assert CPython fallthrough line propagation invariant
* Assert CPython CFG target offset sign
* Assert CPython exception fallthrough invariant
* Assert CPython exception stack bounds
* Assert CPython traversal stack allocation
* Match CPython label-map allocation in shadow
* Mirror CPython label-map sentinel fill
* Match CPython CFG builder allocation asserts
* Match CPython exception stack structure
* Match CPython ref stack structure
* Match CPython CFG traversal stack structure
* Mirror CPython CFG traversal stack pointer
* Use CPython fixed exception handler stack
* Mirror CPython ref stack capacity field
* Match CPython swap optimizer scratch stack
* Align static swap helpers with CPython blocks
* Align swaptimize signature with CPython
* Match CPython redundant pair pass result
* Match CPython inline pass result
* Match CPython redundant NOP pass results
* Fix bytecode metadata after upstream rebase
* Match CPython opcode stack metadata
* Add CPython identifiers to cspell dictionary
Add CNOTAB, LNOTAB, ialloc, ioffset, iused, nblocks, ncellsused,
ncellvars, nextop, noffsets, nvars, swaptimize, untargeted to
.cspell.dict/cpython.txt for the new CFG/assembler code in
crates/codegen/src/ir.rs.
* Fix CI failures from bytecode-parity work
- clippy: drop redundant `test_` prefix on three test functions and
remove an unnecessary `u32` cast in basicblock_clear_reuses_cpython_spare_slots_in_offset_order
- insta: regenerate nested_double_async_with snapshot to match the new
CFG output that drops unreferenced labels after the redundant-NOP pass
- regrtest: drop `@expectedFailure` markers from test_func_args,
test_meth_args (test_compile), test_disassemble_with,
test_disassemble_try_finally (test_dis), and test_except_star
(test_monitoring) which now pass
* Resync generated opcode metadata
Empty conf.toml since WithExceptStart and Setup{Cleanup,Finally,With}
stack effects already match CPython, so the TODO override entries are
stale and only cause CI hook diffs.
Regenerate opcode_metadata.rs and drop the matching SetupCleanup/
SetupFinally/SetupWith assertions on PseudoOpcode::has_arg(); their
`HAS_ARG` flag comes from pseudo definitions in bytecodes.c that the
upstream analyzer does not propagate through PseudoInstruction.properties,
so the generated has_arg() excludes them. has_target() still covers
these block-push pseudos via is_block_push().
* Drop is_block_push has_arg invariant
The CPython invariant `assert(OPCODE_HAS_ARG(op) || !IS_BLOCK_PUSH(op))`
relies on SETUP_{FINALLY,CLEANUP,WITH} carrying `HAS_ARG_FLAG` in
CPython's metadata. The autogen tool reads pseudo-opcode properties from
target instructions and does not propagate the pseudo's own
HAS_ARG flag, so PseudoOpcode::has_arg() omits these three opcodes.
Drop the debug_assert that fired inside py_freeze proc-macro expansion.
* Auto-generate has_eval_break and route AnyInstruction has_arg/has_const via macro
Add fn_has_eval_break to generate_rs_opcode_metadata.py using CPython's
Properties.eval_breaker, removing the hand-written matches! body for
Opcode::has_eval_break and PseudoOpcode::has_eval_break.
Forward has_arg/has_const from Instruction and PseudoInstruction to
their opcode, so AnyInstruction can use either_real_pseudo! like the
other has_* accessors instead of an open-coded match.
* sqlite3: fix Blob.__setitem__ value range validation
Previously, assigning an out-of-range integer (negative or > 255) or an
integer too large for i64 (e.g. 2**65) to a Blob index raised OverflowError
instead of ValueError.
Mirror CPython's ass_subscript_index logic:
- Convert the value via to_i64(), treating any overflow as -1
- Validate the result is in [0, 255], raising ValueError("byte must be in range(0, 256)") otherwise
- Separate deletion error messages: "item deletion" for index, "slice deletion" for slice
* sqlite3: fix Blob.__setitem__ negative-step slice write
In the step != 1 branch of Blob.ass_subscript, the loop used
i_in_temp += step as usize
where step is isize. For negative steps (e.g. step = -2),
(-2isize) as usize = 18446744073709551614
causing an out-of-bounds panic whenever slice_len >= 2.
Fix: use SaturatedSliceIter (already used by the read path) to iterate
over the correct absolute blob indices, then map each index back to a
temp buffer offset via abs_idx - range_start.
Also fix a Clippy lint: replace
val < 0 || val > 255
with the idiomatic
!(0..=255).contains(&val)
Add a regression test in extra_tests/snippets/stdlib_sqlite.py that
exercises blob[9:0:-2] (negative step, slice_len=5).
* fix: guard blob negative-step snippet from CPython 3.11 bug
* style: add blank line after import sys in stdlib_sqlite snippet (ruff)
* Update extra_tests/snippets/stdlib_sqlite.py
---------
Co-authored-by: Jeong, YunWon <69878+youknowone@users.noreply.github.com>
* Update tests
* Add basic audit support
* Add audit for `time.sleep`
* Add some for `socket`
* Some syslog
* Some sys related audits
* some marshal
* monitoring callback
* Mark failing tests
* clippy
* Clippy
* clippy
* mark failing test
* mark more
* Update `test_sys_setprofile.py` to 3.14.5
* Mark failing tests
* Convert host_env Windows path/argv params from raw *const u16 to &WideCStr
* Migrate remaining winapi raw u16 pointer signatures to typed references
* Migrate winreg pub unsafe fn string parameters to typed references
* Add ToPyException impls for host_env error types (PyPy wrap_oserror analog)
* Add CheckLibcResult helper and apply to socket/fcntl/shm/posix_wasi
* Add Win32 BOOL/HANDLE check helpers; apply check helpers across host_env
* Apply Win32/libc check helpers to overlapped/testconsole/os.rs
* Apply Win32 check helpers to winapi.rs (partial)
* Apply Win32 check helpers across more winapi.rs functions
* Apply Win32 check helpers to nt.rs (partial)
* Add CheckWin32Sentinel helper; apply to nt.rs INVALID_HANDLE_VALUE/INVALID_FILE_ATTRIBUTES patterns
* Add OwnedHandle / HandleToOwned helper; apply to mmap create_named_mapping leak path
* Use OwnedHandle RAII in nt::pipe to eliminate manual cleanup on error path
* Use OwnedHandle in nt::chmod_follow; hoist HandleToOwned import
* Drop rustix dependency from vm crate
Remove unused IntoPyException impl for rustix::io::Errno and the
rustix entry in crates/vm/Cargo.toml. rustix is now only depended on
by host_env.
* Fix CI failures: cross-platform regressions
- winapi.rs: pass None to create_event_w; the recent Option<&WideCStr>
migration left one call site still passing a raw null pointer.
- exceptions.rs: gate ToPyException for LockfError with
cfg(any(unix, target_os = "wasi")), matching host_env::fcntl's own
cfg. The previous cfg let it compile on wasm32-unknown-unknown where
host_env::fcntl does not exist.
- io_unsupported.rs: derive Eq on FileMode alongside PartialEq to
satisfy clippy::derive_partial_eq_without_eq.
* Fix CI failures: cfg gates and unused imports
- exceptions.rs: gate ToPyException for LockfError with
cfg(all(unix, not(target_os = "redox"))) to match the type's own
cfg in host_env/src/fcntl.rs (LockfError is not built on wasi).
- signal.rs: CheckLibcResult is only used in unix-gated functions;
split import so it is not pulled in for windows.
- mmap.rs: remove CheckWin32Handle from imports; no longer used after
switching to HandleToOwned-based RAII.
- overlapped.rs: remove INVALID_HANDLE_VALUE from connect_pipe import;
the call now uses .check_valid().
* Fix CI failures: rustfmt and windows unused import
- signal.rs: reorder cfg-gated imports per rustfmt.
- socket.rs: gate ToPyException import to cfg(all(unix, not(target_os = "redox")));
it is only used inside sendmsg which has the same gate, so it was unused
on windows.
* Push remaining libc/extern callsites from vm into host_env
Add host_env wrappers and replace the corresponding vm call sites:
- host_env::errno::strerror_string for libc::strerror
- host_env::io::write_stderr_raw for libc::write(STDERR_FILENO,...)
- host_env::locale::localeconv_data reused from vm::format
- host_env::os::abort for the inline abort extern
- host_env::os::urandom wraps getrandom; getrandom moves from vm to host_env
- host_env::posix::lchmod for the macOS/BSD lchmod extern
- host_env::posix::fcopyfile for the macOS fcopyfile extern
- host_env::nt::wputenv for the Windows _wputenv extern
vm/format.rs's get_locale_info now uses host_env on both unix and windows
instead of the unix-only libc::localeconv path.
* Move time tz state and winsound FFI into host_env
- host_env::time::tz: wraps the libc tzset/timezone/daylight/tzname
globals on non-msvc, non-wasm32 targets. vm::stdlib::time now reads
these via the typed wrappers instead of declaring its own externs.
- host_env::winsound (windows): exposes PlaySoundW (via a typed
PlaySoundSource enum), Beep, and MessageBeep. vm::stdlib::winsound
drops its inline FFI block and routes through host_env.
* Migrate unsetenv to host_env::nt::wputenv; rustfmt
- vm::stdlib::os::unsetenv had a second _wputenv call site that still
referenced the removed inline extern. Route it through
host_env::nt::wputenv like putenv.
- rustfmt fixups in exceptions.rs (boolean chain layout) and the two
winsound files.
* Address PR review comments
- host_env::winapi::create_process: assert that the command_line buffer
is NUL-terminated and that the env block ends with a double-NUL,
matching the Win32 CreateProcessW contract.
- stdlib::overlapped CreateEvent: replace WideCString::from_str_truncate
with the fallible from_str(), so embedded NULs in the event name
surface as ValueError instead of being silently truncated.
- vm::exceptions::ReadlinkError::NotSymbolicLink now maps to OSError
(matches Win32 ERROR_NOT_A_REPARSE_POINT semantics) rather than
ValueError.
- winreg::ConnectRegistry: route the non-zero return through the
existing os_error_from_windows_code helper so the resulting exception
carries the real winerror/message instead of a generic OSError.
* Fix CI failures and address review follow-ups
CI failures:
- rustfmt cleanup in exceptions.rs after the ReadlinkError change.
- vm/stdlib/os.rs: drop unused ToWideString import that the wputenv
migration left behind.
- vm/stdlib/winsound.rs: replace explicit `&*buf` with `&buf` to
satisfy clippy::explicit_auto_deref.
- Lib/test/test_format.py, Lib/test/test_types.py: drop the now-stale
expectedFailureIfWindows decorators on the locale-format tests; the
Windows path now reads real `localeconv` data via host_env so these
tests pass.
Review follow-ups:
- host_env::winapi::create_process: switch the new buffer terminator
checks from `assert!` to fallible validators returning
`io::ErrorKind::InvalidInput`, so bad inputs stay recoverable at
the API boundary.
- host_env::winsound::play_sound: reject `Memory(_)` together with
`SND_ASYNC` (lifetime-unsafe) and `SND_MEMORY` without a
`Memory(_)` source. Expand `PlaySoundError` into a variant enum.
- vm::stdlib::_winapi::CreateProcess: route the Win32 path/argv strings
through `as_wtf8().to_wide_cstring()` like the rest of the Windows
API surface; `expect_str()` could panic on Python strings containing
lone surrogates.
* Align codegen CFG cleanup with CPython
* Align codegen bytecode with CPython 3.14
* Remove remove_jump_target_line_nops
The pass dropped a same-line NOP at the start of any jump-targeted block
whose body advanced to a later line, but CPython's basicblock_remove_redundant_nops
only consults the previous/next instruction inside the same block. The
extra pass deleted the else-body line trace anchor exercised by
test_nested_double_async_with.
Ignore test_conditional_break_finally_does_not_keep_break_cleanup_nop;
the break NOP lands in a separate block from the inlined finally body,
so same-line successor elision does not apply here.
* Keep except_handler blocks reachable in eliminate_unreachable_blocks
After convert_pseudo_ops lowers SETUP_FINALLY to a plain NOP, the only
remaining link from the try body to the except_handler block was the
per-instruction except_handler annotation. When earlier passes had
already removed every NOP that carried that annotation (e.g. an empty
try body with `pass`), the handler block became unreachable from the
entry block and its instructions were cleared, dropping the handler
entirely. Seed reachability with blocks already marked except_handler
so handler dispatch survives independent of the in-block annotation.
Also drop two expectedFailure markers in test_patma whose match-tracing
expectations now pass.
* Align bytecode CFG cleanup with CPython
The tests for swapcase() were failing for two reasons. The first is
'𐐧' casing which should be fixed with modern Unicode tables. The second
failure is due to CPython's sigma override, which I implemented in
PR #7717.
Shahar Naveh