4 Commits

Include renames

Author	SHA1	Message	Date
Jeong, YunWonandLee Dogeon	d02d81e740	perf: VM and runtime hot paths from pyperformance (#8737) ... * time: use CLOCK_UPTIME_RAW for monotonic/perf_counter on Apple CLOCK_MONOTONIC keeps advancing during system sleep on macOS, unlike CPython's mach_absolute_time()-backed perf_counter/monotonic. Benchmarks that spanned a laptop sleep recorded thousands of seconds for a single iteration (pyperformance float/fannkuch showed ~1000x slowdowns). Use CLOCK_UPTIME_RAW (equivalent to mach_absolute_time) on Apple targets and report it as such in time.get_clock_info(). Assisted-by: Claude Code:claude-fable-5-1 * vm: derive f_lineno lazily instead of tracking prev_line per instruction The dispatch loop wrote prev_line (a locations lookup plus a Cell store) on every instruction only so f_lineno could read it. f_lineno now derives the line from lasti and code.locations, prev_line is only maintained on the cold tracing path for 'line' event de-duplication, the two use_tracing reads per instruction are merged into one, and the pre-dispatch lasti reload is replaced by the value just stored. When a trace function is installed on an already-running frame (bdb stepping back into an untraced caller, frame.f_trace assignment, a local trace returned from the 'call' event on a resumed generator) prev_line is synced from lasti so no spurious 'line' event fires. Assisted-by: Claude Code:claude-fable-5-1 * vm: coalesce per-call thread-locals into one FRAME_SLOT_CACHE set_current_frame ran on every frame enter/exit and touched three separate thread_local! statics (each a _tlv_get_addr call on macOS). Merge them into a single thread-local struct so the hot path does one TLS access. Assisted-by: Claude Code:claude-fable-5-1 * vm: fold stop/finalizing checks into EVAL_BREAKER eval_breaker_tripped ran three checks per bytecode: the finalizing flag, a thread-local stop_requested lookup, and the EVAL_BREAKER load. Add STOP_BIT and FINALIZING_BIT so the per-instruction check is a single relaxed atomic load; the slow path (check_signals) still re-derives the exact per-thread answer. The bits are set alongside the per-thread flags and cleared under the stop-the-world exclusion. Assisted-by: Claude Code:claude-fable-5-1 * gc: set the TRACKED bit with a plain store for freshly allocated objects gc_bits is zero for a fresh allocation (both the freelist reuse path and PyInner::new zero it), so track_new_object can initialise the bit with a relaxed store instead of a fetch_or. Re-tracking resurrected objects keeps the read-modify-write variant. Assisted-by: Claude Code:claude-fable-5-1 * vm: skip thread exception publish when pushing/popping a None exc slot Generator/coroutine resume pushes and pops an exc_info slot on every send. When that slot is None the value returned by topmost_exception cannot change (it skips None entries), so the thread-local exception publish in push_exception/pop_exception is unnecessary. Assisted-by: Claude Code:claude-fable-5-1 * vm: allocation-free keyword-call fast path and super() MRO walk Keyword calls to plain functions where every parameter is filled exactly once are reordered in place into the fast-locals layout (no IndexMap, no keyword name clones); any mismatch falls back to the existing slow path so error messages are unchanged. super().attr now walks the type's MRO by reference instead of cloning every class into two Vecs per lookup. Assisted-by: Claude Code:claude-fable-5-1 * builtins: cheaper enumerate and zip iteration enumerate kept its counter as a BigInt behind a write lock and did bignum arithmetic on every item; use a machine integer that promotes to BigInt only for huge/negative starts or on overflow. zip pre-sizes its per-item Vec so the common two-iterable case allocates once. Assisted-by: Claude Code:claude-fable-5-1 * _asyncio: cache module objects and use native set/dict operations Task registration re-resolved _asyncio module attributes and re-imported contextvars on every task; the eager path additionally went through generic call_method for set add/discard. Cache the module objects per thread (validated by identity), split the contextvars cache from the task containers, and use PySet/PyDict methods directly. Assisted-by: Claude Code:claude-fable-5-1 * perf: use mimalloc as global allocator for rustpython CLI Adds mimalloc (0.1.52, default-features = false) as the #[global_allocator] for the rustpython binary, gated behind a new `mimalloc` feature enabled by default on non-wasm targets (dep declared under target.'cfg(not(target_arch = "wasm32"))'.dependencies, so wasm builds never pull it in regardless of default-features). Profiling a `t = (a, b, c)` loop showed malloc/free at ~40% of samples on macOS with the system allocator. Switching to mimalloc gives consistent wins across small-object-heavy workloads, medians over 5-9 interleaved A/B rounds (release build, this machine): tuple (5M `(a,b,c)`) 1.020s -> 0.863s (1.18x) 4-thread tuple (5M total) 1.643s -> 1.382s (1.19x) class instances (2M) 0.874s -> 0.797s (1.10x) dict-heavy (200k x 20) 2.196s -> 1.992s (1.10x) string-heavy (join x 20) 0.662s -> 0.592s (1.12x) nqueens (pyperformance) 0.327s -> 0.286s (1.14x) chaos (pyperformance) 0.204s -> 0.191s (1.06x) richards (pyperformance) 0.154s -> 0.151s (1.02x) go (pyperformance) 0.411s -> 0.405s (1.01x) listcomp / fib ~neutral (<=1%) Peak RSS (/usr/bin/time -l): instances workload roughly unchanged (29.0MB -> 29.1MB); dict-heavy workload actually drops (155MB -> 106MB), likely because mimalloc reclaims freed pages more aggressively than the system allocator across repeated large-dict churn. `cargo build` and `cargo build --release` are warning-free with the default feature set, and `cargo build --release --no-default-features --features stdlib,threading,importlib,stdio,host_env` (mimalloc excluded) also builds cleanly. CI's wasm32-wasip1 and other --no-default-features build matrices in .github/workflows already omit `mimalloc` from their feature lists and target wasm via the cfg(not(wasm32)) dependency gate, so no workflow changes needed. `rustpython -m test -j4 test_gc test_weakref test_threading test_dict test_list test_tuple test_str test_bytes test_memoryview test_ctypes test_mmap test_array` passes identically (2541 tests, 241 skipped) on both the baseline and mimalloc binaries. Assisted-by: Claude Code:claude-fable-5-1 * vm: cap dict/set table shrink and compact holes on resize asyncio's eager task path (and its per-task bookkeeping sets in general) repeatedly adds and discards distinct task objects from a handful of long-lived sets (_eager_tasks, _scheduled_tasks). Two dict_inner issues turned that steady-state churn into the dominant cost of async_tree_eager: 1. resize() picked a target size of `used * 2` with no floor, so once a churny set settled around 1-2 live entries it shrank to a table that the very next insert would already overflow, forcing a resize on almost every single add. Clamping to the original minimum table size (8) restores the headroom dict/set start with. 2. `entries` only ever grows (holes from deletions are never reclaimed), so resize()'s rebuild pass scanned every entry ever inserted into the dict's lifetime, not just the live ones - a set with a stable working set size but heavy churn grew this scan, and the dict's memory, without bound. resize() now compacts holes out of `entries` first, matching CPython's compacting hash table and keeping the shape-stamp fast path (which already requires a hole-free layout) reachable more often. Measured with a standalone async_tree micro-benchmark (6 levels x 6 branches, pyperformance's bm_async_tree "none"/"eager" shape) run inside one process for 10 asyncio.run() iterations, comparing medians of interleaved binary runs on an otherwise idle machine: - eager: ~2.91s -> ~0.86s median (about 3.4x) - none: ~1.98s -> ~1.23s median (about 1.6x) A single fresh-process run (no in-process accumulation) still improves by roughly 15-20% for eager, since the resize-storm from point 1 hits even a single run's worth of task churn. test_asyncio (34 files, all passing), test_dict, test_set, test_weakref, test_weakset, test_ordered_dict, test_gc, test_descr, test_class, and `cargo test -p rustpython-vm -p rustpython-stdlib` all pass. Assisted-by: Claude Code:claude-fable-5-1 * _functools: add native _lru_cache_wrapper RustPython had no native _functools._lru_cache_wrapper, so functools.lru_cache always fell back to the pure-Python implementation in Lib/functools.py (RLock, linked-list bookkeeping, closures). That fallback dominates typing.Protocol's __instancecheck__ path: it calls inspect.getattr_static, whose _shadowed_dict is itself lru_cache'd, so every isinstance() check against a runtime-checkable protocol paid for several pure-Python cache-wrapper calls plus lock acquisitions. Add a native pyclass mirroring CPython's C _lru_cache_wrapper: a reentrant lock scoped only around the cache lookup/insert steps (not held across the wrapped call, matching CPython's threading behavior), LRU ordering piggybacked on the cache dict's own insertion order (move-to-end via del+reinsert, oldest evicted from the front) so no separate key-comparison structure is needed, and __copy__/__deepcopy__ returning self to match CPython since __reduce__ (used only for pickling) requires __qualname__, which wrapped non-function callables such as functools.partial don't have. Measured with a standalone script reproducing pyperformance's bm_typing_runtime_protocols isinstance() loop (300 loops, medians of 5 interleaved runs): 0.260s -> 0.182s, about 1.43x faster. In isolation, a cache-hit through lru_cache went from ~0.31s to ~0.07s for 200k calls, now faster than calling the underlying function uncached (matching CPython's C accelerator). Verified with test_typing, test_abc, test_inspect, test_functools, test_weakref, test_isinstance, and test_descr (all pass, matching the pre-existing baseline, including test_functools's already-present TestLRUC suite which now actually exercises the native implementation). Assisted-by: Claude Code:claude-fable-5-1 * sre_engine: ASCII fast path for \d \s \w unicode category checks is_uni_digit/is_uni_space/is_uni_alnum went through icu_properties' GeneralCategory table lookup for every character, even the common ASCII case. Add a `ch < 0x80` fast path that resolves directly via std's ascii predicates, falling back to the full table only for non-ASCII code points (Latin-1 supplement chars like 'A-umlaut' still route through the table since they are alphanumeric but not ASCII). Measured on a tight `re.findall(r"\w+", ...)` / `re.sub` loop over mostly-ASCII text (interleaved A/B, 4 rounds, medians): re.sub: 1.178s -> 1.134s (~3.7% faster) re.findall: 1.110s -> 1.062s (~4.3% faster) Assisted-by: Claude Code:claude-fable-5-1 * vm: skip VM attach for no-op generator/coroutine __del__ Generator, coroutine, and async-generator dealloc always went through drop_slow_inner's call_slot_del, which unconditionally attaches to a VM (with_vm -> begin_interpreter_section -> per-interpreter thread-slot hashmap lookup) before running del(), even though del() itself is a documented no-op once the object is already closed or running - the common case for a generator/coroutine consumed to completion. Add PyTypeSlots::del_needed, an optional VM-free fast predicate checked before call_slot_del/with_vm; generator, coroutine, and async_generator install a check mirroring their own del() no-op branches, letting drop_slow_inner and try_call_finalizer skip the VM attach entirely for finished objects. Standalone microbenchmarks (median of 5 interleaved runs): - create+drain a 3-yield generator, 2M times: 1.099s -> 0.997s (-9%) - pyperformance bm_generators shape (yield-from tree traversal): -4% - pyperformance bm_coroutines shape (recursive fibonacci coroutines): -13% Assisted-by: Claude Code:claude-fable-5-1 * _asyncio: skip full __import__ dispatch on module-cache hot path with_asyncio_cache/with_contextvars_cache called vm.import("_asyncio"/ "contextvars", 0) on every Task/Future registration just to validate their module-identity cache, even on the (overwhelming) hit path. That still pays the full __import__ builtin dispatch (attribute lookup on builtins, FuncArgs construction, native slot_call, ImportArgs parsing, import_module_level) instead of a cheap sys.modules lookup. Added fast_import_cached(), which checks sys.modules directly and only falls back to vm.import on a miss (module missing/reloaded). Measured with a standalone asyncio_tree-style microbenchmark (6 levels x 6 branches, several asyncio.run() iterations per process, median of 5 interleaved A/B rounds): none: 1.045s -> 0.895s (~14% faster) eager: 0.716s -> 0.591s (~17% faster) eager_tg: 1.023s -> 0.849s (~17% faster) test_asyncio (all files), test_contextvars (module absent from this checkout, pre-existing), test_weakref, test_weakset all pass/unchanged. Assisted-by: Claude Code:claude-fable-5-1 * vm: skip VM-attach section in with_vm when already on the current interpreter Object teardown (drop_slow_inner -> call_slot_del, try_call_finalizer, weakref callbacks) calls with_vm(), which unconditionally went through set_current_vm -> begin_interpreter_section: an INTERP_THREAD_SLOTS HashMap lookup plus an Arc clone, even in the overwhelmingly common case where the dropping thread is already attached to the exact interpreter that owns the object (a __del__ object or weakref target going out of scope mid-bytecode). with_vm now checks whether the object's owning interpreter is already the one on top of VM_STACK. Only the top of VM_STACK is ever ATTACHED on a thread (every push is preceded by an attach, every pop restores the enclosing attach), so when it matches we call f(vm) directly and skip begin_interpreter_section/set_current_vm entirely. The rare not-attached/different-interpreter cases (nested subinterpreters, thread outside any VM, shutdown) fall through to the original attach/detach path unchanged. Interleaved release-build measurements (5 rounds, medians), plain-Rust micro-scripts, no pyperf: - 2M create+drop of a __del__ object: ~0.55s -> ~0.46s (~16% faster) - 2M create+drop with a weakref callback: ~1.14s -> ~1.06s (~7% faster) - 2M io.StringIO() create+drop: ~0.734s -> ~0.697s (~5% faster) - fib(30) and a 5M-iteration tuple-unpack loop: unchanged (noise-level) `sample` profiles of the __del__ benchmark confirm begin_interpreter_section/ensure_thread_slot frames disappear after the change. Verified: `cargo test -p rustpython-vm --release`, and `-m test -j4 test_gc test_weakref test_finalization test_threading test_thread test_concurrent_futures test_os test_signal test_generators test_io test_atexit test_sys` (test_interpreters module absent in this tree, pre-existing). Also a 4-thread stress script hammering __del__ object creation against a concurrent gc.collect() loop for 10s, and a threading+queue handoff dropping a __del__ object after cross-thread transfer, both pass. Assisted-by: Claude Code:claude-fable-5-1 * vm: general sys.modules fast path for vm.import VirtualMachine::import always resolved and called builtins.__import__ (FuncArgs construction, native slot_call, ImportArgs::from_args, then import_module_level) even when the module was already cached in sys.modules. ~63 Rust stdlib call sites hit this on hot paths (copyreg, pickle's reduce_newobj, warnings, asyncio, etc). Add VirtualMachine::try_import_cached: for a plain absolute import (level 0, empty from-list), if builtins.__import__ is still the original import function (compared by identity against vm.import_func) and the module -- or, for a dotted name, its top-level package -- is already present and fully initialized in sys.modules, return it directly, skipping the __import__ dispatch and import_module_level entirely. Falls through to the existing slow path whenever the module is uncached, still initializing, or __import__ has been overridden by user code, so test_import/test_importlib/test_builtin overrides still work. Exposed import::is_module_initializing (refactored out of import_ensure_initialized) so the fast path can check __spec__._initializing the same way the slow path does. Removes _asyncio's local fast_import_cached workaround (added in cb2064f0e) now that vm.import itself has the same fast path generally. Measured (interleaved A/B, 5 rounds, pickle.dumps of a plain object, which hits copyreg import via reduce_newobj): ~2% wall-clock improvement per call (baseline ~4.75s / 100k dumps vs ~4.66s candidate), consistent across all rounds. Verified: overridden __import__ still invoked, a replaced sys.modules is read through the same accessor, and a self-importing/initializing module is not handed out early -- all match baseline behavior. test suite (test_import test_importlib test_builtin test_pkg test_pkgutil test_runpy test_zipimport test_site test_warnings test_copy test_pickle test_asyncio.test_tasks test_threaded_import test_sys): 13/14 files pass; test_threaded_import fails identically on baseline (ModuleNotFoundError, pre-existing, unrelated to this change). Assisted-by: Claude Code:claude-fable-5-1 * _asyncio: bypass generic context.run dispatch in task_eager_start task_eager_start ran the first step via context.run(coro.send, None), which builds a bound `send` method object and then re-enters the generic Callable/FuncArgs dispatch just to invoke context.run's own generic call. Since eager tasks in async_tree_eager complete almost entirely synchronously (no event-loop scheduling needed), this per-task dispatch overhead dominated. Enter/exit the native PyContext directly and call coro.send via a single call_method instead, falling back to the generic path for non-native Context subclasses. async_tree_eager median: ~705ms -> ~697ms across 5 interleaved A/B rounds (~1% eager-only win); async_tree (none) unaffected (noise-level delta). test_asyncio (all files), test_weakref, test_weakset pass; pre-existing test_contextvars module-not-found failure unchanged from baseline. Assisted-by: Claude Code:claude-fable-5-1 * vm: drop scopeguard::defer! from generator/coroutine resume hot path resume_gen_frame restored owner/exc-slot/frame-chain state via a scopeguard::defer! closure that ran on every exit (Ok, Err, or panic). Since the closure's body is the function's only tail expression, a plain captured result plus explicit cleanup (the same shape with_frame already uses for the analogous call path) covers both normal exit paths identically; only the recursion-depth decrement keeps a scopeguard, matching with_frame's precedent that this codebase does not otherwise guarantee generator/frame state consistency across a Rust panic (panic=unwind is not used to implement any Python-level control flow here). Delegation-chain benchmark (chain.py, forwarding N values through a yield-from chain) measures a small but consistent win: ~2-5% less wall time per forwarded value at depths 1/4/16 across interleaved A/B rounds. Single-generator create+finish cost is a wash. Assisted-by: Claude Code:claude-fable-5-1 * vm: run generator bodies through the call trampoline `Py<FrameObject>::resume` built its `ExecutingFrame` with `tailcall_enabled: false`, because a `TailCall` result had nowhere to go: the trampoline was reachable only from `run_frame_fast`, whose entry frame lives on the data stack and can never yield. So every ordinary Python call made from inside a generator or coroutine body re-entered `run_frame_fast` recursively instead of being flattened. Generalize the trampoline to drive a heap-resident entry frame whose enter/exit bookkeeping belongs to its caller, and whose `Yield` is a valid terminal result rather than a panic. `FrameKind` now says, per frame, what the trampoline owes it: `Entry` (exit it, caller frees the storage), `GenEntry` (`resume_gen_frame` linked it and will unlink it; a `Yield` is the trampoline's result) or `Callee` (exit it and free its storage). The three near-identical dispatch arms collapse into one `Step::Finished` / `Step::Deliver` loop. The suspended-frame stack moves onto the VM and is reused across invocations: a generator body enters the trampoline once per resume, so a `Vec::with_capacity(8)` there cost a malloc/free per yield (+27 ns/resume when first measured). Invocations nest strictly LIFO, so each owns the region above the length it finds on entry. Neutral on its own — a generator body's first call was the only one not already flattened, since the callee's own frame runs under the trampoline either way. It is what lets SEND/FOR_ITER hand a generator frame to the same loop. Measured (chain of `yield from`, 1-yield generator create+finish, bm_generators, bm_coroutines, async_tree none/eager): within noise, except generator create+finish 625 ns -> 603 ns. Assisted-by: Claude Code:claude-fable-5-1 * vm: run a `yield from` chain in one eval loop `SEND` on a suspended generator recursed: `Coro::send` -> `resume_gen_frame` -> a fresh `ExecutingFrame::run` per level, and the value came back the same way. Measured at ~55 ns per level of delegation, against CPython's ~5.8 ns. Only about 5 ns of that is the resume bookkeeping itself (measured by stubbing out the claim, the exception slot, the owner swap and the recursion checks). The rest is what a delegating frame does per value: leave and re-enter the eval loop, and dispatch `RESUME`, `JUMP_BACKWARD_NO_INTERRUPT`, `SEND` and `YIELD_VALUE` — four instructions that, in the canonical `yield from` / `await` shape, only hand a value along. So the trampoline now runs the chain itself. `SendGen` parks its frame and returns `ExecutionResult::GenResume`; the trampoline claims and links the sub-generator (`coroutine::flat_resume_enter`, the first half of what `Coro::send` does around a resume) and runs its frame in the same loop. On the way down, every frame it finds already suspended at a `yield from` whose delegate can be resumed is parked without being run at all (`yield_from_delegate`); on the way back up, a yielded value is re-yielded for each such frame by parking its `lasti` past the `YIELD_VALUE` (`park_after_yield_from`) instead of running it. A whole chain therefore costs one claim/link/unlink per level and no instruction dispatch at all, with `Py<FrameObject>::resume` extending the same treatment to the outermost level, which is where `next()` and asyncio's task step enter a chain. Each level is still claimed, linked and unlinked exactly as a recursive `Coro::send` would, so `gi_running`, `f_back`, `gi_frame`, `gi_yieldfrom`, `sys._getframe`, tracebacks, `throw`, `close`, PEP 479 and the `RecursionError` depth are unchanged; `resume_gen_frame` and the flat path share `gen_frame_link`/`gen_frame_unlink` and the invariants are documented on `FlatResume`. Everything else keeps the recursive path: tracing or monitoring on (both would see the skipped instructions), `throw`/`close`, async generators, subclasses, non-generator iterators, `FOR_ITER`, a `SEND` that is not a `yield from`, and a delegate that has not started or is not itself delegating — the last two because one lone level costs more to hand to the trampoline than to send into from the eval loop the caller is in. Measured (median of 5, ns/value or seconds): chain of `yield from`, depth 16 911 -> 479 -47% chain of `yield from`, depth 4 212 -> 158 -26% bm_generators 0.272 -> 0.187 -31% richards 0.772 -> 0.771 ~0 bm_coroutines 0.0129 -> 0.0130 +0.8% async_tree none ~neutral chain depth 1 (no delegation) 81.1 -> 83.6 +3.1% fib(24), calls through trampoline 0.0180 -> 0.0183 +1.7% Per level of delegation: 55.3 ns -> 26.4 ns (CPython 3.14: 5.8 ns). Assisted-by: Claude Code:claude-fable-5-1 * vm: reset the oparg prefix when a signal unwinds a frame An `EXTENDED_ARG` leaves its prefix in `arg_state` for the instruction that follows it. The signal path in the eval loop unwinds to a handler and `continue`s straight back to the top, skipping the loop's own `arg_state.reset()`, so the handler's first instruction was decoded with the interrupted instruction's prefix still applied. Every other path that jumps somewhere new — the `set_f_lineno` restart just above — resets first. Reachable whenever a signal arrives on the instruction right after an `EXTENDED_ARG` in a frame that has a handler: a `KeyboardInterrupt` into a `yield from` chain panicked with `InvalidBytecode` in roughly half of the runs of a 16-deep chain, and now does not in ten out of ten. Assisted-by: Claude Code:claude-fable-5-1 * bytecode: fetch a code unit with one 16-bit acquire load The dispatch loop read the opcode with an Acquire `AtomicU8` load and the arg byte with a separate Relaxed one. The acquire load sits between the two, so LLVM had to re-load `self.code` and `code.instructions` for the second access: four extra instructions and two extra dependent loads on the instruction-fetch path of every bytecode instruction. `CodeUnits::read_unit` reads both bytes as a single aligned Acquire `AtomicU16`, which pairs with `replace_op`'s Release store exactly as `read_op` did. `CodeUnit` gains `align(2)` so the 16-bit access is guaranteed aligned rather than relying on the allocator. 5.91 -> 5.76 ns/bytecode-instruction on a straight-line LOAD_FAST/STORE_FAST loop (-2.5%); range loop -2%. Assisted-by: Claude Code:claude-fable-5-1 * vm: compute the cache-entry count after dispatch, not before `op.cache_entries()` is a byte-table lookup used only to skip inline cache units after an instruction falls through. Computing it before the handler kept it live across the whole (very large) inlined match, so every bytecode instruction paid an address materialization, the table load, a stack spill and a reload of the count. Moving it next to its only use lets it be computed on the fall-through path with the value in a register. Measured together with the following commit (the eval-breaker outline): 5.76 -> 5.55 ns/bytecode-instruction on a straight-line LOAD_FAST/STORE_FAST loop, range loop -1.5%, fib(27) -2.5%. Assisted-by: Claude Code:claude-fable-5-1 * vm: move the eval-breaker body out of the dispatch loop The eval-breaker check itself stays inline on the per-instruction path (moving it off regresses badly), but everything it guards -- the finalizing/main-thread check, stop-the-world suspend, the QSBR checkpoint, signal delivery and the scheduled GC -- was inlined into the loop body too. That code is never executed on the hot path yet its address materializations and live values competed for registers with every opcode handler, so the loop head spilled more than it needed to. `eval_breaker_work` is `#[cold] #[inline(never)]`, leaving a single `ldrb`/`cbz` pair inline for the common "nothing pending" case. Measured with the preceding commit: 5.76 -> 5.55 ns/bytecode-instruction on a straight-line LOAD_FAST/STORE_FAST loop. Assisted-by: Claude Code:claude-fable-5-1 * vm: always inline the eval stack push/pop primitives `pop_value`, `pop_value_opt`, `top_value` and friends are a handful of instructions each, but `ExecutingFrame::run` is ~50 KB of inlined opcode handlers, so LLVM's inline budget refused them and emitted real calls: STORE_FAST and POP_TOP each began with `bl ...pop_value`, which also forced spills around the call. `#[inline(always)]` puts the four or five instructions they actually are back into the handlers. 5.55 -> 5.29 ns/bytecode-instruction on a straight-line LOAD_FAST/STORE_FAST loop (-4.8%); nbody -3.5%, fib(27) -3.5%, `s += i` loop -3.5%, `for _ in range(n)` -4%. Assisted-by: Claude Code:claude-fable-5-1 * vm: carry the instruction fetch state in loop locals Instruction fetch went through memory three times per bytecode instruction: `self.lasti` (a pointer field) was re-loaded, the program counter was stored to it and read back at the top of the next iteration, and `self.code` -> `code.instructions` was re-loaded to address the code unit. LLVM cannot keep any of them in registers because the handlers write through `&mut self`, so the fetch chain started with a store-to-load round trip followed by two dependent pointer loads. `lasti_cell` and `instructions` are now copied into locals once (both are plain shared references with the frame's lifetime), and the program counter is carried in `idx`: after a handler that did not jump the next index is already known from `lasti_before` plus the cache count. The frame's `lasti` is still stored before every instruction, so tracebacks, `f_lineno`, signals and `sys._getframe` see exactly what they did; only the read-back is gone. The paths that unwind (signal, exception handler, `set_f_lineno`) re-read it, since they jump to a new position. Also collapses the eval-breaker `if`, which clippy flagged after the previous commit moved its body out of line. range loop -3.3%, richards -3.1%, nbody -2.9%, `s += i` loop -2.2%, go -1.7%, raytrace -1.2%; mean -1.1% over the 13-benchmark set. The straight-line LOAD_FAST/STORE_FAST micro is unchanged at 5.28 ns/bytecode-instruction (it has no cache entries and never jumps, so it was already the best case for the old code). Assisted-by: Claude Code:claude-fable-5-1 * bytecode: hoist the instruction array base out of the dispatch loop Addressing a code unit still went `self.code` -> `code.instructions` -> the `Box<[CodeUnit]>` data pointer on every instruction, two dependent loads before the acquire load that fetches the unit itself, because the handlers write through `&mut self` and LLVM has to assume the fields could change. The instruction array is allocated once when the code object is built and never reallocated -- specialization rewrites units in place -- so `CodeUnits::units_ptr` may be read once before the loop and reused. `read_unit_from` is the same 16-bit acquire load taken from that base; `read_unit` now delegates to it, and the index is unchecked exactly as it was before. nqueens -1.9%, raytrace -1.6%, range loop -1.3%, deltablue -1.1%, nbody -0.8%, dispatch micro -0.7%; mean -0.7% over the 13-benchmark set. Assisted-by: Claude Code:claude-fable-5-1 * common: an immortal bit in the refcount word Atomic reference counting is about a third of what a bytecode costs now that the dispatch loop is trimmed: built `--no-default-features`, where `PyAtomic` is a `Cell`, the straight-line dispatch micro runs at 3.5 ns/instruction against 5.4. Every `TO_BOOL`/`POP_JUMP_IF_*` pair and every `None` or small-int push pays a relaxed `ldadd` and a `Release` `ldaddl` on an object that can never die. PEP 683's answer is to say so in the word. A fourth flag bit, `IMMORTAL`, takes a bit from the strong count (leaving 60 on a 64-bit target, 28 on a 32-bit one, both far past what a program reaches); `make_immortal` sets it and parks the count at CPython's `_Py_IMMORTAL_REFCNT`, and `inc`, `inc_by`, `dec` and `safe_inc` read the bit and return without touching the counter. The parked count is what keeps the rest of the tree honest without a special case: it is far too large for a `strong_count() == 1` in-place-mutation fast path to fire on, and `start_gc_refs` already reads a count that large as `GC_REACHABLE`, so an immortal object is a permanent root that keeps its referents alive. Two details pay for the branch the mortal path now runs, which cost 4% on object-heavy benchmarks before they went in: * `leak` sets `IMMORTAL` alongside `LEAKED`. An interned object could already never reach zero, so `dec` no longer needs the test it made on the result of the decrement: the immortal test at the top decides both. * `inc`'s three exceptional cases — a destructed word, a count at the ceiling, a count at zero — fold into one comparison against the masked word, with the handling moved to a cold helper. What is left is one instruction on `dec` and roughly nothing on `inc`, and the mortal path measures at parity. Assisted-by: Claude Code:claude-fable-5-1 * vm: make the singletons, small ints, interned strings and static types immortal Everything the `Context` builds once and keeps — `None`, `True`, `False`, `Ellipsis`, `NotImplemented`, the empty tuple, frozenset, bytes and str, the -5..256 int cache and the interned one-character strings — is now marked immortal, as is every static type object, which its `static_cell` holds for the life of the process. So is every entry of the string pool: `mark_intern` already stopped those refcounts from reaching zero, so immortality frees no memory that would otherwise have been freed; it only takes the atomic read-modify-write off every incref and decref of an attribute name, a dict key or a docstring. Straight-line dispatch micro (200 LOAD_FAST/STORE_FAST pairs over cached ints, median of 5 interleaved runs): 5.39 -> 4.01 ns/instruction, -26%. A four-thread `if flag:` loop, where the contention on the shared bool was real: 2.36 s -> 2.14 s, -10%. pyperformance bench functions run standalone, medians of 5 interleaved rounds against a matched build: fannkuch -7.8%, nqueens -3.8%, chaos -3.0%, raytrace +1.0%, go +1.2%, richards +4.2%, nine of them together -2.9%. Per-benchmark deltas under about 4% sit inside this tree's build-to-build ThinLTO layout noise, which reaches that size between two builds of identical source. Observable: `sys.getrefcount` of an immortal object reports 4294967295, CPython's `_Py_IMMORTAL_REFCNT`, and stops moving — what `test_sys.test_refcount` expects of `None`. `is_interned`, and the dict pointer-equality key fast path behind it, keep answering from the separate `LEAKED` bit. Assisted-by: Claude Code:claude-fable-5-1 * codecs: fast-path utf-8 decode, skipping the codec registry round trip bytes.decode('utf-8') / str(bytes, 'utf-8', ...) always went through the generic codec registry: a per-call attribute lookup for _is_text_encoding, then a full call into the pure-Python encodings.utf_8.decode wrapper, which called back into the native utf_8_decode and unpacked a 2-tuple result. CPython's PyUnicode_Decode special-cases a handful of built-in encoding names (utf-8 among them) and decodes them directly, bypassing the codec registry entirely -- a codecs.register() override does not affect str(b"...", "utf-8") there either. This adds the same fast path here: recognized utf-8 aliases go straight to the native decoder. Measured with a pure-Python `str(bytes, 'utf-8', 'surrogatepass')` loop: ~723ns -> ~401ns per call (1.8x). On the pyperformance unpickle benchmark (which decodes short strings this way for every SHORT_BINUNICODE opcode): ~14.4ms -> ~13.2ms per loop, an ~8.3% win. Assisted-by: Claude Code:claude-fable-5-1 * codecs: fast-path utf-8/ascii/latin-1 encode and ascii/latin-1 decode Extend the utf-8 decode fast path from 3fe684629 to the symmetric encode direction (str.encode()/bytes(s, enc)), and add the same registry-skipping fast path for ascii and latin-1 encode/decode, all of which are simple native codecs.rs functions already. For utf-8 encode with no surrogates in the string (the common case), encoding is just a copy of the string's existing UTF-8 bytes, so that's the fastest path; strings containing surrogates and non-strict error handlers still go through the native encoder + ErrorsHandler, matching custom codecs.register_error handlers. Measured (median of interleaved runs vs baseline, 2M iterations): str.encode("utf-8"): 351ns -> 195ns (-44%) str.encode(): 449ns -> 291ns (-35%) bytes(s, "utf-8"): 542ns -> 382ns (-30%) str.encode("ascii"): 436ns -> 302ns (-31%) str.encode("latin-1"): 487ns -> 333ns (-32%) bytes.decode("ascii"): 519ns -> 359ns (-31%) bytes.decode("latin-1"):590ns -> 417ns (-29%) bytes.decode("utf-8")/str(b,"utf-8"): unchanged (already fast-pathed) Assisted-by: Claude Code:claude-fable-5-1 * vm: make the runtime safe for borrowed stack entries `PyStackRef` has carried a borrow tag since the stack was unified with the fastlocals, but only `LOAD_SMALL_INT` -- which borrows immortal `Context` objects -- ever set it, so nothing in the runtime had to be right about borrowed entries. Turning `LOAD_FAST_BORROW` into a real borrow makes three places load-bearing, and this fixes them ahead of the switch. `Traverse for PyStackRef` handed every entry to the tracer, borrowed ones included. The cycle collector subtracts one from an object's `gc_refs` per edge it is shown, so a borrowed entry -- which owns no strong count -- would have pushed a live object's count to zero and freed it out from under the frame. Borrowed entries are now skipped; nothing becomes unreachable, because the fastlocals slot the entry borrows from is on the same traversal. `park_after_yield_from` stands in for a `YIELD_VALUE` that never runs, so the `promote_stack()` in that handler never ran either; it now promotes there too, and the debug assertion after it stops being vacuous. `STORE_FAST`, `DELETE_FAST` and the fused stores now call `LocalsPlus::debug_audit_local_release`, which in debug builds panics if the slot is about to drop the last count of an object a borrowed entry still points at -- the one shape codegen's `optimize_load_fast` exists to prevent, and the one that would be a use-after-free if the analysis or a modelled stack effect were wrong. `BORROW_LOCAL_LOADS` (off) is the switch for the borrow itself, and `LOAD_FAST_BORROW`/`LOAD_FAST_BORROW_LOAD_FAST_BORROW` now go through `push_local`, which is a plain clone while it is off. The invariant the switch depends on is written out where `PyStackRef` is defined. No measurable change: with the switch off the only behavioral difference is the traversal fix, which no current bytecode can reach. Assisted-by: Claude Code:claude-fable-5-1 * vm: read consumed stack entries in place instead of owning them `pop_value()` turns whatever is on the stack into an owned `PyObjectRef`, so an instruction that only reads its operand still pays a reference count round trip -- and, once `LOAD_FAST_BORROW` starts pushing borrows, would pay an atomic increment to promote a borrow it is about to throw away. This moves every consumer that never needs ownership onto `pop_stackref()`, which hands back the tagged entry and costs nothing for a borrow. Discarding pops (62 of them, across the `LOAD_ATTR`, `TO_BOOL`, `BINARY_OP`, `COMPARE_OP`, `CONTAINS_OP` and `CALL` specializations, which read their operands through `top_value()`/`nth_value()` and then drop them) become `pop_stackref()`. So do the read-only operands of `POP_TOP`, `POP_JUMP_IF_*`, `TO_BOOL`, `BINARY_OP`, `COMPARE_OP`, `CONTAINS_OP`, `IS_OP`, `STORE_ATTR`, `DELETE_ATTR`, `STORE_SUBSCR`, `DELETE_SUBSCR` and `UNPACK_SEQUENCE`. Two protocol entry points had to move from an owned receiver to `&PyObject` to make that reachable: `try_to_bool`, which never used the ownership it took, and `rich_compare`, which forwards to the `&self` `_cmp` anyway. Both keep working unchanged for `PyObjectRef` callers through deref. `LOAD_ATTR`'s method specializations popped the receiver and pushed it straight back below the bound function; they now move the entry across with its tag intact, which is how CPython keeps a method call's receiver borrowed from `LOAD_ATTR` all the way through `CALL`. The plain attribute read reads through the receiver rather than owning it. Neutral while `BORROW_LOCAL_LOADS` is off -- for an owned entry `pop_stackref()` and `pop_value()` compile to the same thing. Assisted-by: Claude Code:claude-fable-5-1 * vm: turn LOAD_FAST_BORROW into an actual borrow `LOAD_FAST_BORROW` accounts for 21.5% of executed instructions on nbody+richards+nqueens, and until now it did exactly what `LOAD_FAST` does: an atomic increment on the way onto the stack and, from the instruction that consumed it, an atomic release decrement on the way off. Neither is needed. The fastlocals slot holds a count for as long as the entry is on the stack -- codegen's `optimize_load_fast` only rewrites a `LOAD_FAST` when it can prove the entry is consumed before anything can release that slot -- so the entry can be a borrow that owns nothing, which is what CPython 3.14 does with a tagged `_PyStackRef`. Flipping `BORROW_LOCAL_LOADS` on is the whole change; the two preceding commits are what make it safe (cycle-collector traversal, yield-point promotion, a debug audit of the invariant) and what make it pay (consumers that read through the entry rather than promoting it back to owned). Medians of 7 interleaved rounds, this machine, against the pre-series binary: straight-line while loop 6.32 -> 6.01 ns/instruction (-5.0%) x = a + b loop -6.3% if flag: loop -3.8% attribute-read loop -5.2% method-call loop -3.3% subscript loop -6.7% fib(27) -0.8% nbody -2.6% fannkuch -1.7% nqueens +1.3% richards -3.0% deltablue -2.3% go -3.0% raytrace -1.7% float -3.2% chaos -2.6% spectral_norm -2.5% mean -2.1% About half of that is the consumer rewrite alone (mean -1.0% with the switch still off) and the rest is the borrow. nqueens is the one benchmark that goes the other way, by less than the roughly 4% of build-to-build layout noise this tree has, and it moves with the consumer rewrite rather than the switch. Full suite green (418 files, 42,110 tests). A build with debug assertions on runs the suite with no audit failure, and trips the audit exactly when `LOAD_FAST` is forced to borrow in a shape codegen refuses to convert (`f(x, (x := None))`). Adversarial scripts -- swap loops and reassignment under `__del__` tracking, argument evaluation that deletes its sibling, `a[a.pop()]`, closures over a rebound cell, generators dropped while suspended, tracebacks whose `f_locals` are read after unwinding, `settrace` writing `f_locals` mid-expression, a method receiver whose local is cleared while the call is being built, `__add__`/`__lt__`/`__bool__` that clear the local their operand came from, threads sharing objects through locals, and `gc.collect()` from a finalizer -- all agree with CPython 3.13 under `gc.set_threshold(1, 1, 1)`, in both builds. Assisted-by: Claude Code:claude-fable-5-1 * vm: exact-dict fast path for DICT_MERGE and CALL_FUNCTION_EX kwargs `g(**d)` on an exact dict compiled to BUILD_MAP+DICT_MERGE+CALL_FUNCTION_EX and paid for two Python-level `keys()`/`__getitem__` round trips per key at both the DICT_MERGE handler and collect_ex_args' iterate_mapping_keys, even though the source is a plain dict. Add an exact-dict (downcast + type check, so subclasses still go through the mapping protocol) fast path to both sites that iterates entries natively via PyDict::items_vec, matching CPython's PyDict_Check fast paths in PyDict_Merge / CALL_FUNCTION_EX. Measured (1M calls, release build): g(**d) 1320ns -> 665ns, f(**d) 1470ns -> 900ns, c.m(**d) 1690ns -> 1135ns, f(x=1,**d) 1600ns -> 1035ns, f(**d1,**d2) 1700ns -> 1013ns. bm_django_template ~7% faster, bm_argparse (subparsers) ~7% faster. Assisted-by: Claude Code:claude-fable-5-1 * vm: skip the recursion guard hashing exact ints inside a tuple tuple.__hash__ hashed every element through PyObject::hash, which wraps each call in with_recursion's native-stack-depth check. That guard exists for elements whose hash can recurse into arbitrary Python code, but int.__hash__ is a leaf computation - it can neither recurse nor call back into Python - so the check is pure overhead for the overwhelmingly common case of tuples of plain ints (e.g. namedtuple-shaped dict keys). pyperformance's mdp benchmark builds huge tables keyed by tuples of deeply nested namedtuples whose leaves are ints; profiling with `sample` showed PyObject::hash/with_recursion/tuple's slot_hash dominating the CPU time. Hashing exact-int elements directly via hash::hash_bigint cuts mdp's wall time from ~6.79s to ~4.81s (median of 5 interleaved runs), about 29% faster, with no change to hash values or recursion-guard behavior for any other element type. Assisted-by: Claude Code:claude-fable-5-1 * vm: wire slot descriptors from __slots__ into C-level type slots Classes that put a dunder name (e.g. "__setitem__") into __slots__ get a per-instance member descriptor for it, which is a legitimate way to override a special method per-instance (chameleon's tal.RepeatDict does exactly this). PyType::new_verbose ran init_slots() to compute mp_ass_subscript/mp_subscript/etc. before these member descriptors were added to the type's attribute namespace, so the slot lookup never saw them and __setitem__/__getitem__ silently fell back to "does not support item assignment". Fix by calling update_slot for each dunder-shaped __slots__ member right after it is installed on the type, mirroring what type.__setattr__ already does for regular attribute updates. Assisted-by: Claude Code:claude-fable-5-1 * frame: stop allocating cold data to close a generator `clear_locals_and_stack`, which runs whenever a generator or coroutine comes to an end, reached for the frame's lazily-allocated `FrameColdData` through `cold()`. That allocates it on first access, so every finishing generator paid a `Box<FrameColdData>` malloc and free just to find `f_extra_locals`, `f_locals_cache` and `f_overwritten_fast_locals` all empty -- cold data is only ever populated by tracing and frame introspection. Ask for it with `cold_opt()` instead and skip the three reads when it was never allocated. Medians of interleaved A/B runs, ns per generator create+finish over a plain call: `for _ in g(): pass` 265 -> 231 (-13%), a coroutine that completes synchronously 642 -> 617 (-4%), `sum(x for x in range(3))` 1033 -> 997 (-3.5%). Assisted-by: Claude Code:claude-fable-5-1 * function: bind exact args straight into a generator's frame Calling a generator, coroutine or async generator function took the general argument-binding path even when the call site was the simplest possible one. `invoke_exact_args_slots` and the vectorcall fast path both recognised the shape -- every parameter positional, filled exactly once -- and then handed generator-like code back to `invoke()`, which built a `FuncArgs` (a `Vec` plus an `IndexMap`) only for `fill_locals_from_args_inner` to walk varargs packing, keyword matching and defaults that this shape cannot reach. Add `make_generator_exact_args`, the counterpart of `prepare_exact_args_frame` for the one call shape it refuses: it builds the heap frame the generator will own, moves the arguments straight into its fastlocals and returns the generator. Both fast paths now use it. The vectorcall one matters on its own for generator expressions, whose fresh `MAKE_FUNCTION` on every evaluation keeps them out of the call-site specialization that would otherwise catch them. Medians of interleaved A/B runs, ns per create+finish over a plain call: `for _ in g(): pass` 231 -> 218 (-6%), a synchronously-completing coroutine 616 -> 598 (-3%), `sum(x for x in range(3))` 1007 -> 967 (-4%). Assisted-by: Claude Code:claude-fable-5-1 * frame: drain a dead frame in place instead of moving it out Three allocations and a large move on the path a finishing generator takes to release its frame. `Traverse::clear` opened with `self.iframe.get_mut().take()`, moving the whole `InterpreterFrame` -- a couple of hundred bytes -- out of its slot and onto the stack, only to drop it there. Drain it through `as_mut()` and empty the slot afterwards. `LocalsPlus::clear_into` collected the evaluation stack into a `Vec` that existed only to be extended into the caller's; push into that one directly. And `clear_stack_and_cells` built (and dropped) a `Vec` for a stack it had not looked at -- a frame that reached its `return`, which is every generator that simply finished, left nothing there. Look first. Medians of 5 interleaved rounds, ns per create+finish: `for _ in g(): pass` 307 -> 297 (-3.4%), a synchronously-completing coroutine 600 -> 585 (-2.5%), `list(g())` 565 -> 558; a plain call is unchanged. Assisted-by: Claude Code:claude-fable-5-1 * vm: don't raise a TypeError to ask an iterator for its length `length_hint_opt` opened with `iter.length()`, whose whole job on a type with no length slot is to build the `TypeError` that says so -- which this caller then catches, recognises and discards. Every iterator and every generator takes that path, and `list()` calls it on whatever it is handed, so a `list(gen)` paid for an exception object per call. `length_opt` answers the same question with `None`. CPython's `PyObject_LengthHint` gates the call on the slots for the same reason. Median of 5 interleaved rounds: `list(g())` x200k for a 1-yield generator 556 -> 407 ns (-27%). pyperformance generators, coroutines, comprehensions and async_tree are unchanged, as are nqueens, richards and chaos. Assisted-by: Claude Code:claude-fable-5-1 * gc: track a generator and its frame in one pass Making a generator tracks two objects that are born together and die together -- the generator and the frame it owns -- and each one took the gen0 list lock and a thread-local read of the running interpreter for itself. Give the pair one entry point: `PyGenerator`, `PyCoroutine` and `PyAsyncGen` opt out of tracking at allocation, the way frames already do, and `make_generator_or_coro` links both into gen0 under a single lock, with one owner read and one collection check. Medians of 5 interleaved rounds: pyperformance coroutines -2.5%, generators -0.9%; `for _ in g(): pass` over a plain call 205 -> 200 ns. Assisted-by: Claude Code:claude-fable-5-1 * dict_inner: fast-path exact-int equality in key_eq Mirrors CPython's lookdict: two exact `int` keys now compare by BigInt value directly instead of going through the generic rich-compare dispatch (identical_or_equal -> bool_eq -> rich_compare_bool -> __eq__ slot lookup). Falls back to bool_eq whenever either side isn't exactly `int` (subclasses may override __eq__), so behavior is unchanged there. Measured (min-of-6 rounds, 2M-iteration micros, this tree's build has ~4% layout noise so min is used as the noise floor): - dict_int_lookup (`d[i]`, int keys): 70.9ns -> 62.0ns (-12.5%) - dict_str_lookup / dict_tuple_lookup / set_str_contains: unchanged (within noise), confirming no regression on non-int keys. pyperformance sanity checks (single round each, base vs candidate): bm_mdp median ~5027ms/loop vs ~5003ms/loop (no regression, hashing isn't the dominant cost there); bm_nqueens, bm_go, bm_deltablue, bm_hexiom, bm_chaos all within ~1% (noise). Hash/equality values cross-checked bit-identical against CPython 3.14 for 0, 1, -1, True, False, 2**61-1, 2**61, -(2**63), 2**100, -(2**100), and int/float/bool/Decimal/Fraction(1) hash-equality invariants. Also tried an exact-type fast dispatch in PyObject::hash (skipping the slots.hash vtable call + with_recursion for exact int/str) - it only moved hash_small_int/hash_large_int by ~2%, under the 3% bar, so it was not committed. Tests: `rustpython -m test -j4 test_hash test_int test_long test_tuple test_dict test_set test_dictviews test_ordered_dict test_collections test_fractions test_numeric_tower test_float test_bool test_str test_userdict test_weakref` all pass (1796 run, 72 skipped). `cargo test -p rustpython-vm --release -p rustpython-common` passes. clippy clean on the touched lines. Assisted-by: Claude Code:claude-fable-5-1 * vm: let the modulus type handle a three-argument pow CPython's ternary dispatch offers the operation to the third operand's type whenever its slot differs from the two already tried, including the case where neither operand had one. The port required both of those slots to exist, so `pow(10, 2, Decimal(7))` never reached `Decimal` and raised TypeError. Assisted-by: Claude Code:claude-fable-5-1 * vm: add a feature-gated opcode-pair histogram Counts `(prev_op, op)` in the dispatch loop and dumps the nonzero pairs at exit, so superinstruction and specialization candidates can be ranked from what pyperformance-shaped code actually dispatches rather than guessed at. Entirely behind the non-default `opcode-histogram` feature: the module is `#[cfg]`-ed out and the loop's two-line hook with it, so a default build is byte-identical to one without the patch. Assisted-by: Claude Code:claude-fable-5-1 * vm: let boolean producers run the branch that follows them `TO_BOOL*`, `COMPARE_OP*`, `CONTAINS_OP*` and `IS_OP` materialised a bool only for the `POP_JUMP_IF_*` one dispatch later to pop and test it, and the compiler's `NOT_TAKEN` marker cost a third dispatch on the fall-through edge. Generalise the `COMPARE_OP_INT`-only jump fusion so every boolean producer resolves the jump itself, and step over `NOT_TAKEN` from there and from the plain `POP_JUMP_IF_*` handlers. Both back off to ordinary dispatch when the successor is an instrumented opcode or while tracing, so `sys.monitoring` branch events and `sys.settrace` opcode events are unaffected. Dispatches over 13 standalone pyperformance benchmarks: -5.0% total (richards -11.7%, unpickle_pure_python -8.9%, sympy -8.4%, hexiom -8.1%, django_template -7.1%, go -6.1%; nbody unchanged). Times, best of 7 interleaved rounds: deltablue -4.0%, hexiom -3.5%, go -3.4%, unpickle_pure_python -3.3%, richards -3.2%, raytrace -3.0%, nqueens -1.9%, geomean -1.0%; a branch-heavy microbenchmark -4.7% against a straight-line control at 0%. Assisted-by: Claude Code:claude-fable-5-1 * Adapt yield-from resume to optional generator frames Coro.frame is now PyAtomicRef<Option<FrameObject>>, so the flattened resume path uses frame_opt/clear_except_code instead of a direct frame. Assisted-by: Grok:4.6 * Fix clippy and cr_origin after try_to_bool/f_lineno changes try_to_bool now takes &self, so drop the leftover clones. Derive cr_origin lines from lasti instead of prev_line, and satisfy -Dwarnings on the immortal-refcount and del_needed types. Assisted-by: Grok:4.6 * Convert fd filenames without the os ToPyObject impl host_env-free builds (sandbox check, wasm) do not compile os.rs, so OsPathOrFd::filename cannot use Borrowed's ToPyObject. Use as_raw() instead, which is the same conversion. Assisted-by: Grok:4.6 * Gate fd path helpers on the host_env feature WASM clippy builds rustpython-vm without host_env, so OsPathOrFd and the fspath/fd helpers are unused and fail -Dwarnings. Assisted-by: Grok:4.6 * Drop redundant pickle try_to_bool clones after rebase Assisted-by: Grok:4.6 * Document why enumerate keeps a Counter beside BigInt Assisted-by: Grok:4.6 * Move fd path helpers into ospath_fd under host_env Assisted-by: Grok:4.6 * Group threading eval-breaker bits in one module Assisted-by: Grok:4.6 * Make InterpreterFrame construction unsafe run_iframe stays safe: a live frame already has valid identity pointers. Assisted-by: Grok:4.6 * Move OsPathOrFd re-export to the top of ospath.rs Assisted-by: Grok:4.6 --------- Co-authored-by: Lee Dogeon <dev.moreal@gmail.com>	2026-09-20 00:32:26 +09:00
Jeong, YunWonandLee Dogeon	358f6a875a	native _json, _pickle, pyexpat, and _elementtree (#8741) ... * _json: add native make_encoder (c_make_encoder) json.encoder fell back to the pure-Python _make_iterencode because _json had no make_encoder, making json.dumps ~25x slower than CPython. Implement the C encoder semantics natively: str/None/bool/int/float/list/tuple/dict ordering, key coercion and skipkeys, sort_keys via the VM's fallible sort (comparison errors propagate instead of panicking), circular reference detection, indent handling, allow_nan, default() fallback with recursion guard, and CPython's add_note annotations. Removes the now-passing expectedFailure markers in test_json. Assisted-by: Claude Code:claude-fable-5-1 * pyexpat: add UseForeignDTD, fix silently swallowed handler exceptions genshi's XML template loader (bm_genshi) called xmlparser.UseForeignDTD(), which pyexpat.rs didn't implement, raising AttributeError. Add it as an accepted no-op: the pyexpat module here wraps the pure-Rust xml-rs crate (not real libexpat), which has no hook for splicing in a foreign DTD. Root-causing the resulting bug also uncovered why genshi's template generate()/render() produced empty output instead of erroring: do_parse's invoke_handler swallowed every exception raised by a Python callback (handler.call(...).ok()), and genshi's expat callbacks fail immediately by reading `parser.CurrentLineNumber`/`CurrentColumnNumber`, which pyexpat.rs never exposed. The AttributeError was silently discarded, so no start/end/text events ever reached genshi's queue. Fix both: propagate handler exceptions out of Parse()/ParseFile() (matching libexpat/CPython semantics), and add CurrentLineNumber/CurrentColumnNumber/ CurrentByteIndex as read-only attributes backed by xml-rs's per-event TextPosition (byte index is derived from it and is exact for ASCII documents). This also fixes test_minidom's testExceptionOnSpacesInXMLNSValue, previously an expected failure because the ValueError from an invalid xmlns value was swallowed the same way. PositionTest in test_pyexpat.py checks byte-exact expat semantics (token *start* position) that xml-rs's *current parse cursor* position can't reproduce exactly, so that one test is now marked an expected failure. Assisted-by: Claude Code:claude-fable-5-1 * pyexpat: support multi-chunk Parse()/ParseFile() feeds bm_xml_etree failed with AssertionError: missing end tags from TreeBuilder.close(), because xml.etree.ElementTree reads and feeds large documents to XMLParser.feed() in ~64KB chunks (and xml.sax's IncrementalParser and ElementTree's XMLPullParser do the same). Each Parse()/ParseFile() call built a brand-new xml-rs EventReader from only the bytes passed in that call, discarding all parser state between calls. When a chunk boundary fell in the middle of an element, the fragment parsed by itself was invalid/incomplete XML, so xml-rs silently errored out (matching this module's existing "ignore stricter xml-rs errors" behavior) and the corresponding start/end events were dropped entirely -- not just delayed. xml-rs has no API to resume a parser across separate Read sources, so instead each call now appends the new chunk to a persistent buffer and re-parses the buffer as a whole, replaying only the events at or beyond the count already dispatched in previous calls (dispatched_events). This also makes position tracking (CurrentLineNumber/Column/ByteIndex) correct regardless of how the input was chunked. Trade-off: each Parse() call is now O(size fed so far), so a document delivered in many small chunks is parsed several times over; acceptable for correctness but noticeably slower than a true incremental parser for pathological chunk sizes. This also fixes several previously-xfailed tests that depend on incremental feeding: test_xml_etree.py's XMLPullParserTest (test_events, test_feed_while_iterating, test_simple_xml, test_simple_xml_chunk_22, test_simple_xml_with_ns) and ElementTreeTest.test_parseliteral, plus test_sax.py's ExpatReaderTest (test_expat_entityresolver_default, test_expat_incremental, test_expat_incremental_reset) -- their expectedFailure markers are removed. test_simple_xml_chunk_1/_chunk_5 (1- and 5-byte chunks) still fail and remain xfail: at that granularity xml-rs's stricter-than-expat tokenizer errors on some intermediate prefixes that libexpat tolerates. Assisted-by: Claude Code:claude-fable-5-1 * test: drop stale expectedFailure markers fixed by the pyexpat changes test_plistlib's four invalid-document tests and test_xmlrpc's test_loads_unsupported pass now that pyexpat no longer swallows handler exceptions and keeps parser state across chunked feeds. Assisted-by: Claude Code:claude-fable-5-1 * pyexpat: drive a single background EventReader for true incremental Parse() 8802451f3 fixed multi-chunk Parse()/ParseFile() feeds by accumulating every fed byte and re-parsing the whole buffer from scratch on each call, which is O(total bytes) per chunk -- quadratic in document size. xml-rs's EventReader has no API to pause/resume across separate Read sources, so instead one EventReader is now driven to completion on a dedicated background thread fed through a condvar-backed queue (FeedBuffer/FeedReader); Parse()/ParseFile() push their chunk, wake the thread, and drain (on the calling thread only) whatever events that unblocks, stopping once the thread reports it is blocked waiting for more input. Handlers always run on the calling thread. A handler exception stops dispatch without tearing down the stream, so the next Parse() resumes where it left off, mirroring libexpat. CurrentByteIndex is now computed incrementally per line (with an amortized per-query cursor, needed since real documents are often a single very long line) instead of scanning the whole buffer. ParserCreate() never spawns a thread; one is spawned lazily by the first Parse()/ParseFile() call, and dropping the parser always stops and joins it. Fixes test_xml_etree{,_c}'s test_events_comment, previously xfailed because the old whole-buffer replay lost top-level comment events across feeds. Measured (M4 Pro, release build): - Scaling (ElementTree.parse, newline-separated doc): new backend 1/4/16 MB = 0.15/0.61/2.35s (linear); CPython 3.14 same file = 0.015/0.055/0.23s. - pyperformance's bm_xml_etree doc (156KB, effectively one line): old backend single etree.parse() = 1.99s, single iterparse() = 3.53s; new backend = 0.025s / 0.031s (~80-110x). Full bench_parse/iterparse/generate/ process (30/10/10/1 reps) now finish in ~1.5s/0.6s/0.4s/0.3s instead of timing out past pyperformance's 400s limit. - 10k ParserCreate() (never fed / partially fed / fully fed and dropped): 0.32s, no residual native threads, no RSS growth. test -j4 test_pyexpat test_xml_etree test_xml_etree_c test_sax test_minidom test_xml_dom_minicompat test_pulldom test_plistlib test_xmlrpc test_html test_htmlparser test_threading: all pass (1350 tests, 89 skipped). Assisted-by: Claude Code:claude-fable-5-1 * pyexpat: add a Sync fallback backend for targets without threads 2f58e9531 made pyexpat always drive its EventReader on a background thread (ParserStream::spawn), panicking via .expect() if the thread couldn't be spawned. pyexpat is registered unconditionally, including on wasm32, where std::thread::spawn always fails at runtime, so the first Parse() call panicked there. Introduce a Backend enum: Backend::Threaded (the existing ParserStream) on targets that can spawn threads, and a new Backend::Sync, the accumulate-and-reparse implementation from 8802451f3, used on wasm32 and as a runtime fallback if spawning a thread ever fails. Both share LineTracker for identical CurrentLineNumber/CurrentColumnNumber/ CurrentByteIndex and handler/error semantics. Fixed two correctness bugs while porting the Sync backend forward: marking the parser permanently "finished" as soon as a from-scratch reparse happened to reach EndDocument (which can be a false positive on a not-yet-complete buffer, e.g. right before epilog content arrives) rather than only on the caller's real isfinal signal; and counting that same EndDocument towards the dispatched-event index, which could make a real event land on an index a later, longer reparse had already skipped past. Verified by force-selecting Backend::Sync under a test-only env var (removed) and diffing its output against Backend::Threaded for the xml/sax/etree/plistlib/xmlrpc test suites and a 2MB document fed in 1KB chunks. Assisted-by: Claude Code:claude-fable-5-1 * _pickle: native Pickler, Unpickler and PickleBuffer Adds `crates/stdlib/src/pickle.rs`, a native `_pickle` implementing Pickler, Unpickler, PickleBuffer, PicklerMemoProxy/UnpicklerMemoProxy, dump/dumps/load/loads and the PickleError family, so `Lib/pickle.py` stops falling back to `_Pickler`/`_Unpickler`. Protocols 0-5 are supported and produce byte-identical output to the pure-Python pickler, including framing for protocol >= 4, memoization, `fast` mode, dispatch_table, reducer_override, persistent_id / persistent_load, find_class with `_compat_pickle` fix_imports, buffer callbacks and out-of-band protocol 5 buffers. This also turns on `pickle.PickleBuffer`, which RustPython did not have before. Speedups over the pure-Python implementation (best of 5, us/op): pickle 835 -> 21 (39x), pickle_list 525 -> 4.4 (119x), pickle_dict 4018 -> 32 (125x), unpickle 630 -> 24 (26x), unpickle_list 317 -> 5.0 (63x). CPython 3.14's C accelerator runs the same workloads 1.5-2.9x faster than this. test_pickle now runs 1008 tests (was 470) since the C-implementation test classes are enabled; test_pickle, test_pickletools, test_picklebuffer are green and 22 obsolete expectedFailure markers are removed. Assisted-by: Claude Code:claude-fable-5-1 * _pickle: raise the pickle.find_class audit event Unpickler.find_class now raises `sys.audit('pickle.find_class', module, name)` like CPython, so audit hooks can veto global lookups (test_audit.test_pickle). Also drops four expectedFailure markers that the native module makes pass: test_pydoc's builtin-method description tests, test_memoryview.test_picklebuffer_reference_loop and test_pyclbr.test_others. Full suite: 456 test files / 42,648 tests green. Assisted-by: Claude Code:claude-fable-5-1 * _pickle: avoid temporary buffers on the save fast paths Opcode plus payload is now appended straight to the output buffer (`write_pair`) instead of building a `Vec` per int, str, bytes and memo opcode, `save_tuple` no longer copies the element slice, and the `when serializing ...` note text is only formatted when an error is actually raised. bm_pickle "pickle" workload: 21.4 -> 17.0 us/op. Assisted-by: Claude Code:claude-fable-5-1 * pyexpat: make the threaded backend fork-safe fork() only keeps the calling thread. test_platform's test_mac_ver_with_fork crashed the child with SIGTRAP: plistlib parses a plist through pyexpat (spawning a background parser thread) before os.fork(), then parses again in the child. Bisecting away every other explanation (a plain threading.Thread survives the same fork fine, since it goes through this VM's own fork-safe thread bookkeeping) showed the crash is not from touching the *inherited* ParserStream (already joined by refcounting before the fork in this repro) but from spawning a *brand new* parser thread from the child, which reliably crashed during the new thread's own startup. Register a pthread_atfork child hook (lazily, the first time a parser thread is spawned) that sets a permanent, process-wide flag the instant this process is ever the child of a fork(). Once set: - ParserStream::try_spawn refuses to spawn any further thread and Backend::new falls back to Backend::Sync instead, exactly as on targets that cannot spawn threads at all. - Feeding an inherited Backend::Threaded raises a clear ExpatError instead of touching its FeedBuffer/channel (whose mutex/condvar may look held by a thread that no longer exists), and dropping it leaks its JoinHandle/buffer/channel via ManuallyDrop rather than joining or destroying them, since even destroying a POSIX mutex that looks held is undefined behavior. The parent is unaffected: the pthread_atfork hook only runs in the child. Verified with a 200-iteration fork stress test (fresh parsers and parsers forked mid-parse) and the full test_platform/xml/fork/ subprocess/multiprocessing_fork test modules. Assisted-by: Claude Code:claude-fable-5-1 * pyexpat: implement name interning via self.intern, matching libexpat self.intern (an always-present dict by default) was stored but never consulted; every StartElement/EndElement/attribute name allocated a fresh PyStr even when the same tag or attribute name repeated across many sibling events. Wire it up the way libexpat does (PyDict_SetDefault(self->intern, name, name)): element and attribute *names* are now memoized so a repeated name reuses the same PyStr object (and its cached hash) instead of reallocating and rehashing. Attribute values are left out of the cache, same as libexpat. This fixes test_pyexpat's previously-xfailed InterningTest.test, which asserts object identity across repeated StartElementHandler/ EndElementHandler calls. Measured on the pyperformance xml_etree parse/iterparse/generate/ process variants: no measurable wall-clock change (the dominant cost there is Python-level TreeBuilder/Element bytecode execution and thread hand-off, not name allocation), so this is a correctness/ feature-completion fix rather than a throughput win for that benchmark; kept because it is a real, previously-inert compatibility gap and is otherwise free. Assisted-by: Claude Code:claude-fable-5-1 * _elementtree: native Element, SubElement and ParseError Adds crates/stdlib/src/elementtree.rs providing the accelerator names xml/etree/ElementTree.py picks up at import time. Element carries the full API (tag/text/tail/attrib, indexing and slicing, find*/iter/ itertext, copy/deepcopy/pickle state) with CPython's lazily created attrib dict and deferred text join, is subclassable, weak-referenceable and GC-traversable, and drops every displaced object after releasing its lock so a __del__ can re-enter safely. xml_etree, best of 3 (s): parse 1.64 -> 1.14, iterparse 0.68 -> 0.54, generate 0.44 -> 0.25, process 0.27 -> 0.16. Assisted-by: Claude Code:claude-fable-5-1 * _elementtree: native TreeBuilder and XMLParser TreeBuilder keeps CPython's element/data/tail bookkeeping in Rust, including the pull-parser event hooks _setevents installs. XMLParser drives a pyexpat parser whose handlers are this type's own native methods, so a start or end tag reaches the builder without a Python frame: the universal-name cache, attribute dict and TreeBuilder call all happen in Rust when the target is our own builder, while foreign targets still go through the documented start/end/data/comment/pi protocol. ElementTree.parse() also picks up the _parse_whole fast path. xml_etree, best of 3 (s): parse 1.14 -> 0.53, iterparse 0.54 -> 0.28, generate 0.25 -> 0.24, process 0.16 -> 0.16. Against the pure-Python baseline that is parse 3.1x, iterparse 2.4x, generate 1.8x, process 1.7x. Assisted-by: Claude Code:claude-fable-5-1 * pyexpat: stop rescanning the whole line per position query LineTracker converted a TextPosition column to a byte offset by validating and decoding its entire buffered line tail on every query, and appended fed bytes to that tail one at a time. On a document that is one long line -- what xml.etree.tostring() writes, and what the xml_etree benchmark parses -- the tail is the whole read-ahead, so a per-event lookup became quadratic. Walk only the characters the query actually asks for, over the deque's two halves in place, and record fed bytes a line at a time. Byte indices are unchanged. xml_etree, best of 3 (s): parse 0.52 -> 0.38, iterparse 0.28 -> 0.25. Assisted-by: Claude Code:claude-fable-5-1 * _elementtree: charge Element.__deepcopy__ to the recursion budget A chain of elements is copied by native recursion, so a deep enough tree overflowed the real stack instead of raising. Wrap the copy in with_recursion, as CPython does, and drop the skip on test_deeply_nested_deepcopy for the accelerated run. Assisted-by: Claude Code:claude-fable-5-1 * pyexpat: don't let a stale NeedMore end a Parse() early The parser thread reports NeedMore when it finds the feed buffer empty and parks. If it reached that point before the chunk Parse() was about to push became visible -- which it does whenever the thread happens to be scheduled first, so more often under load -- pump() consumed that report as if it answered the new chunk and handed control back to Python having dispatched nothing, silently losing the whole document. Stamp each NeedMore with the push count observed when the buffer was found empty and ignore reports older than the push being pumped. The push notifies the condvar under the same mutex the report was sent from, so the parser thread is guaranteed to wake and report again. Fixes an intermittent failure of test_pyexpat's InterningTest under parallel load (about 5 losses per 20000 parses before, none after). Assisted-by: Claude Code:claude-fable-5-1 * Restore xml.etree test markers instead of skipTest in bodies Assisted-by: Grok:4.6 * Clone state before try_to_bool in pickle load_build try_to_bool takes the object by value, and load_build still needs the state dict afterwards. Assisted-by: Grok:4.6 * Drop decorative section-separator comments Assisted-by: Grok:4.6 * Represent pickle opcodes as an enum Assisted-by: Grok:4.6 * Use full paths for thread-only types in pyexpat Assisted-by: Grok:4.6 --------- Co-authored-by: Lee Dogeon <dev.moreal@gmail.com>	2026-09-19 19:21:32 +09:00
Jeong, YunWon	210ccc889c	Thread-safe type operations: type lock, QSBR type-cache reclamation, GC stop-the-world, and interpreter optimizations (#7416) ... * type lock * Drop old PyObjectRef outside type lock to prevent deadlock Dropping values inside with_type_lock can trigger weakref callbacks, which may access attributes (LOAD_ATTR specialization) and re-acquire the non-reentrant type mutex, causing deadlock. Return old values from lock closures so they drop after lock release. * Align type lock behavior with CPython * Add PUBLISHED flag bit to RefCount state word Assisted-by: Claude * Add QSBR module for deferred memory reclamation Assisted-by: Claude * Defer memory free of cache-published objects via QSBR Assisted-by: Claude * Wire QSBR checkpoints into thread lifecycle and eval breaker Assisted-by: Claude * Publish type-cache values to QSBR and bypass cache without a VM Mark cached type-method values as QSBR-published before storing their pointer in TYPE_CACHE, so racing readers never try-incref freed memory. Debug-assert that the current thread is ATTACHED whenever a lock-free cache read happens. Delete find_name_in_mro_without_vm and its unsound direct SeqLock read outside any VM/thread registration; find_name_in_mro now falls back to the locked, uncached MRO walk when no VM is current. This also removes assign_version_tag()'s last caller, so its unlocked fallback is deleted along with it (version_for_specialization keeps the locked path). VirtualMachine::initialize() runs Python bytecode (e.g. importing codecs/encodings) before any enter_vm scope exists, which left the bootstrap thread not ATTACHED for cache reads during startup. Add thread::VmBootstrapGuard, an RAII counterpart to enter_vm usable across statements that need &mut VirtualMachine, and wrap initialize() with it. Assisted-by: Claude * Use try-incref reads and QSBR-backed swaps in specialization cache Assisted-by: Claude * Reset QSBR thread registry after fork in the child Add Qsbr::reset_after_fork, which clears the registered thread slots before draining the retire queue. Call it in py_os_after_fork_child, right after type_cache_after_fork() and before _thread::after_fork_child() re-registers the surviving thread. Dead parent threads' Arc<QsbrSlot> handles live on in the child's copied memory with no destructor ever running, so without this the registry keeps them 'online' forever and future grace periods never complete. Assisted-by: Claude * Add threaded stress test for type cache mutation races Assisted-by: Claude * Restrict qsbr internals to pub(crate) and fix clippy lints Downgrade pub items in the private qsbr::threading module (QsbrSlot, Qsbr, QSBR, and its methods) to pub(crate), since they were never reachable outside the crate. Downgrade ThreadSlot::qsbr accordingly to avoid a private-interfaces warning. Import Arc/Weak from alloc instead of std to match clippy::std_instead_of_alloc. Rewrite the process() comment: goals are not strictly ordered by push order since advance() and the queue push aren't atomic together, so concurrent free_delayed calls can interleave; the drained prefix stays sound because each item individually passed poll, with reordering only affecting reclamation latency. Assisted-by: Claude * Gate per-instruction QSBR check behind a global pending flag eval_breaker_tripped() and check_signals() called thread::qsbr_break_requested() on every bytecode instruction, which does a thread-local lookup, RefCell borrow, and atomic load even when no QSBR retirement is pending. This caused a measured 10-18% slowdown on tight interpreter loops. Add Qsbr::pending, a global AtomicBool set under the queue lock while free_delayed() holds a retired allocation, and cleared under the same lock by process()/drain_all() once the queue empties. The hot path now checks Qsbr::break_pending() (a single relaxed static load) before touching thread-local state, and only pays the TLS cost while a retirement is actually in flight. Assisted-by: Claude * Merge signal and QSBR eval-breaker flags into one atomic word Replace the separate ANY_TRIGGERED AtomicBool and QSBR pending AtomicBool checks in the per-instruction eval-breaker path with a single AtomicU8 (EVAL_BREAKER) holding a bit per source. The global QSBR instance mirrors its pending state into the QSBR bit under the same queue-lock critical sections that already toggle pending, while local QSBR instances used by unit tests are left untouched. eval_breaker_tripped() and check_signals() now read/act on the merged word (eval_breaker_pending(), qsbr_bit_set()) instead of issuing two separate atomic loads per instruction. set_triggered() switches from store to fetch_or so a signal handler never clobbers the QSBR bit. Removed the now-unreferenced is_triggered() helper; break_pending() is kept for unit tests only and marked allow(dead_code) outside test builds. Assisted-by: Claude * Skip freelist reuse for published objects in default_dealloc Freelist-eligible payloads (tuple, int, list, dict, float, ...) were routed to T::freelist_push before ever reaching PyInner::dealloc, so a published object (e.g. a tuple cached as a type attribute) bypassed the is_published -> free_delayed QSBR hook entirely. That let PyRef::new_ref reuse the slot and rewrite the refcount word with a non-atomic core::ptr::write racing a reader's atomic try-incref, and let FreeList::drop call alloc::dealloc directly while a reader could still be mid-read. Gate the freelist branch in default_dealloc on RefCount::is_published(), checked before any teardown. A published object now always falls through to PyInner::dealloc, whose existing hook defers the memory free via QSBR. Document the non-unix (e.g. Windows) reclamation-latency limitation in the QSBR design doc's Lifecycle integration section (previously only noted in a code comment in vm/thread.rs). Extend the type-cache stress test so the mutator also churns a freelist-eligible published tuple class attribute (C.shape), read by readers each inner loop and deleted on the same cadence as C.m, to exercise the fixed bypass. Assisted-by: Claude * Use itoa for PyInt::to_str_radix_10 i64 fast path Format small integers with itoa::Buffer instead of i64::to_string() in PyInt::to_str_radix_10(), which backs both str(int)/repr(int) for exact int and PyObject::str()'s exact-int fast path. Adds the itoa crate as a workspace dependency. Assisted-by: Claude * Make object attribute dunders wrapper descriptors Remove the __getattribute__, __setattr__ and __delattr__ #[pymethod]s from PyBaseObject so add_operators installs PyWrapper slot wrappers for them from the #[pyslot] functions instead of method descriptors. lookup_slot_in_mro now classifies these entries as NativeSlot, so heap types without Python-level overrides keep the native getattro/setattro slot functions, which lets LOAD_ATTR/STORE_ATTR specialization apply to instances of Python-defined classes. Since __setattr__ and __delattr__ share the setattro slot, resolve both names together in update_one_slot: overriding only one of them no longer lets the other name's native resolution overwrite the dispatching wrapper. Assisted-by: Claude * Resolve TpSetattro pair on attribute deletion, not just addition update_one_slot's TpSetattro branch only ran the combined __setattr__/__delattr__ resolution (added in the previous commit) when ADD was true. On deletion (e.g. del C.__setattr__), it fell back to inherit_from_mro, which copies the base class's slot and discards the class's own remaining Python-level override of the other name. lookup_slot_in_mro reads the current attribute dicts, so the same resolution is correct for both addition and deletion; the ADD-only guard is removed and the logic now always runs. Assisted-by: Claude * Generate Opcode::cache_entries/deopt as table lookups generate_rs_opcode_metadata.py now emits a 256-entry const array for each function instead of a chained match over opcode names. Opcode::deopt() indexes an [Option<Opcode>; 256] table built from the existing specialization-to-base mapping, and Opcode::cache_entries() indexes a [u8; 256] table that bakes in the same deopt/to_base composition deoptimize() previously performed before its own match. Regenerated opcode_metadata.rs from the updated script; PseudoOpcode's bodies are unchanged since it has no cache/deopt entries to table. Added an exhaustive test in instruction.rs that checks every valid u8 opcode against a frozen copy of the old chained-match bodies. Assisted-by: Claude * Make Opcode/Instruction numeric conversions O(1) Opcode::as_numeric, Opcode::as_instruction and Instruction::as_opcode were each a per-variant match over ~230 arms in the define_opcodes! macro. Opcode now carries the same explicit $op_id discriminants and #[repr($typ)] as Instruction, which makes the conversions sound as: - Opcode::as_numeric: a plain `self as $typ` identity cast. - Opcode::as_instruction / Instruction::as_opcode: a mem::transmute, relying on both enums sharing one-$typ-wide layout (Instruction's payload fields are all the zero-sized Arg<T> marker, checked by a new per-instantiation size_of assertion). try_from_numeric is left as a match: $op_id values have gaps (specialized/instrumented opcode ranges), so a range check can't replace it. Adds #[inline] to the rewritten conversions plus deoptimize, and to the small generated wrapper/table-lookup functions from the previous opcode-metadata change (as_u8/as_u16, cache_entries, deopt, to_base) via generate_rs_opcode_metadata.py, then regenerates opcode_metadata.rs. Extends the instruction.rs equivalence-test pattern with two exhaustive tests (u8 and u16 instantiations) checking the new identity-cast/transmute conversions against the untouched try_from_numeric/TryFrom paths. Assisted-by: Claude * Pop CallIsinstance arguments directly instead of collecting into Vecs The specialized handler built two heap-allocated Vecs per call (pop_multiple().collect() plus a with_capacity(2) buffer) before invoking is_instance. Pop the class and instance straight off the stack for the effective two-argument case, removing the per-call allocations. Assisted-by: Claude * Inherit tp_new instead of always installing new_wrapper update_one_slot's TpNew branch stored new_wrapper for every ADD, so all heap types had slots.new == new_wrapper and the CallAllocAndEnterInit specialization never fired. Now __new__ is resolved through the MRO dicts: an own or Python-level definition installs new_wrapper, while a builtin __new__ entry (or none) inherits slots.new from the solid base. The now-reachable CallAllocAndEnterInit path gains the missing guards: the specializer requires co_argcount == oparg + 1 (via can_specialize_call) and rejects generator-like __init__, and the handler re-checks the argcount against the class-level init cache. Without the argcount check, __init__ defaults broke re._parser (UnboundLocalError in SubPattern.__init__). Remove the now-unused is_simple_for_call_specialization. Assisted-by: Claude * Skip locals dict allocation for the init-cleanup shim frame Mark the shim code object NEWLOCALS and create the shim frame with no locals mapping so FrameLocals::lazy() is used. The shim only executes ExitInitCheck/ReturnValue and never touches locals, so this removes one dict allocation per specialized instantiation. Assisted-by: Claude * Guard CALL_ALLOC_AND_ENTER_INIT with cached function version The init specialization cache stored only the type version, so swapping __init__.__code__ (which zeroes func_version) did not deopt the warmed call site and stale code assumptions (argcount shape, no varargs/kwonly, not a generator) were applied to the new code object. Store get_version_for_current_state() in the specialization cache when caching __init__ and re-check func_version() in the handler before using the cached function, deopting to the generic call path on mismatch, the same way the getitem specialization cache does. Assisted-by: Claude * Apply rustfmt and ruff-format fixes to files from earlier commits Assisted-by: Claude * Add frame object freelist Frame.iframe becomes FrameUnsafeCell<Option<InterpreterFrame>> so a dead frame can be left as an empty husk. Traverse::clear extracts all owned child references (including localsplus values via LocalsPlus::clear_into) and empties the cell. Dead frame allocations are cached in a thread-local FreeList<Frame> (up to 200 entries) and reused by PyRef::new_ref. clear_generator returns early when the frame was already cleared by cycle collection; generator drops can run after tp_clear of their frame. gc: skip tp_clear for objects saved to gc.garbage by DEBUG_SAVEALL, since they remain reachable from Python. Assisted-by: Claude * Drop frame children directly in tp_clear instead of extracting Pushing the frame's ~10 child references into the clear buffer grew a Vec on every frame deallocation. Take the interpreter frame out of the cell and drop it in place; LocalsPlus::clear_into is removed. Assisted-by: Claude * Untrack objects before tp_clear and guard cross-thread f_locals The cycle collector called tp_clear on unreachable objects while they were still linked into the GC generation lists, so another thread could obtain a strong reference to an already-cleared object (e.g. a frame husk with iframe == None) via gc.get_objects() and access the cleared payload. Untrack the dead set before the clear phase, mirroring the untrack-then-clear ordering of the refcount dealloc path. Objects that gained an external reference before untracking are found by comparing each object's strong count against the references coming from within the dead set; such late-resurrected objects and everything reachable from them are re-tracked and skipped, letting a later collection retry once the external references are released. Also reject f_locals access for a frame currently executing on another thread. Reading fastlocals of such a frame races with the executing thread overwriting the slots (torn reads of dropped values). Access from the executing thread itself (locals(), trace callbacks) is still allowed via the current-frame chain check. Assisted-by: Claude * Skip localsplus heap copy for uniquely referenced dying frames release_datastack_frame now checks the frame's strong count after untracking it from the GC generation lists. When the caller holds the only reference, localsplus values are dropped in place and the data stack storage is released without the heap copy; escaped frames (traceback, sys._getframe, trace callbacks) keep the copy path. The three inline materialize+pop call sites in function.rs are routed through release_datastack_frame. Assisted-by: Claude * Stage exact-args call arguments in stack slot buffers CallPyExactArgs, CallBoundMethodExactArgs and CallAllocAndEnterInit no longer collect popped arguments into a per-call Vec (plus a second Vec for self-prepending). Arguments are popped into a fixed-size Option<PyObjectRef> slot buffer (CallArgBuffer, 8 inline slots, heap fallback for larger arities) and moved into fastlocals via the new invoke_exact_args_slots / a take() iterator. - prepare_exact_args_frame now accepts an ExactSizeIterator of args - specialization_run_init_cleanup_shim takes the slot buffer and fills slot 0 with new_obj, dropping one redundant clone - LoadAttrGetattributeOverridden, LoadAttrProperty and BinaryOpSubscrGetitem pass inline slot arrays instead of vec![] Assisted-by: Claude * Reduce allocations in generic call paths - FuncArgs::prepend_arg: use reserve instead of reserve_exact so exact-capacity vectors do not realloc on every prepend. - IntoFuncArgs::into_method_args: build the final args vec once with capacity len + 1 instead of prepending into a full vector. - PyType::call: skip cloning FuncArgs when no init slot can run after slot_new (no init slot, not `type`, and slot_new is not new_wrapper). Assisted-by: Claude * Skip redundant exc_info restore on frame exit with_frame_impl saves the current exc_info slot value and restores it when the frame exits. When the slot still holds the same value, the restore rewrites the slot and recomputes the thread-exception mirror for no effect. Add restore_exception, which compares the slot against the saved value by pointer identity and skips the store when they match. The saved value is a strong reference held for the whole frame scope, so the pointed-to object cannot be freed and its address reused while the frame runs, making the pointer comparison free of ABA. Assisted-by: Claude * Apply rustfmt to gc_state.rs Assisted-by: Claude * Avoid empty FuncArgs clone in PyType::call Cloning args for the init call clones the kwargs map even when args is empty, which shows up in no-argument instantiation profiles. Prepare the init-call args before slot_new: a default FuncArgs when args is empty (indistinguishable from a clone of empty args), the existing clone otherwise. The empty check also keeps the init-slot load of the clone-elision path off the no-argument call path. Assisted-by: Claude * Push freelist husks only after tp_clear and child drops default_dealloc pushed the object onto the thread-local freelist before running clear_fn. Payloads that drop children inside clear (Frame) can run __del__ there, and a reentrant allocation could pop the husk and overwrite its payload while clear_fn still held a &mut borrow of it. Tuples had the same window through the extracted-edges drop, which ran after the push. Reorder default_dealloc to clear, drop the extracted children, and only then attempt the freelist push, so the husk becomes reusable only when no borrows into the payload remain. The tuple freelist bucketed husks by element count, which required reading the payload during push; after this reordering the elements are already cleared. PyInner<PyTuple> is a fixed-size allocation (the elements box is dropped and replaced on reuse), so replace the per-size buckets with the shared single-list FreeList. Remove the now-unused pyinner_layout helper. Assisted-by: Claude * Resolve __set__ and __delete__ together for the descr_set slot The descr_set slot serves both __set__ and __delete__, but the TpDescrSet accessor resolved only the single modified name and fell back to MRO inheritance on delete. Deleting one of the pair could disable the surviving operation: after `del D.__set__`, an instance delete stopped calling the remaining __delete__. Resolve both names together like the TpSetattro accessor does: any Python-level definition selects the dispatching wrapper, matching native functions are stored directly, a single native function with the other name absent is stored directly, and the slot is inherited from the MRO only when neither name resolves. Assisted-by: Claude * Update type base on __bases__ assignment set_bases replaced bases and mro but never updated the stored base, so __base__ kept reporting the old value after reassignment and slot resolution (tp_new inheritance, set_new/set_alloc during init_slots) read the stale solid base. Change the base field to PyAtomicRef<Option<PyType>> so it can be swapped under the type lock while remaining lock-free for readers, and recompute it in set_bases with the same best_base validation type creation uses (BASETYPE flag, instance layout conflict among bases). The swapped-out base is parked in the frame's temporary refs; other references released inside the critical section are dropped after the lock is released. Also in set_bases: - remove this class from the old bases' subclass lists (pruning dead entries), so repeated assignment no longer accumulates duplicates - roll back bases, base, and all updated mros when the recursive mro update fails, instead of leaving the type half-reparented - fix the misformatted empty-tuple error message Remove the expectedFailure marker from test_unsubclassable_types, which now passes. Assisted-by: Claude * Fix rollback order, dead weakrefs, and lock-held drops in set_bases - Restore recorded mros in reverse on rollback so a class visited multiple times through diamond inheritance ends with its original mro instead of an intermediate one. - Skip dead weakrefs in the subclass list during the recursive mro update instead of panicking on upgrade. - Retire the replaced mros on the success path instead of dropping them while the type lock is held. Assisted-by: Claude * Reify number sub-slot wrapper once in update_one_slot The update_sub_slot! macro expanded the Python-method wrapper closure at both the own and inherited store sites. Each expansion is a distinct fn item, so a base and a non-overriding subclass stored wrappers with distinct addresses in unmerged debug builds. binary_op1 compares slot fn addresses to detect whether a subclass overrides the operator, so the inherited slot was misread as an override and C() // E() dispatched to __rfloordiv__ instead of __floordiv__. Bind the wrapper store in a single closure and call it from both branches so the fn item is reified once and the slot value stays identical across a base and its subclasses. Document the conservative-on-mismatch nature of the new_wrapper address guard in call_wrapped. Assisted-by: Claude * Restrict type_cache_after_fork visibility and guard type-cache reads type_cache_after_fork is only called from within the crate, so mark it pub(crate) to resolve the unreachable-pub warning. Add debug_assert_current_thread_attached at the has_name_in_mro lock-free type-cache read site, matching lookup_ref_and_version_interned, and gate the function on debug_assertions so it is not compiled unused in release builds where the call sites are elided. Assisted-by: Claude * Guard BinaryOpSubscrGetitem with the cached type version The specialized BINARY_OP_SUBSCR_GETITEM handler used the cached __getitem__ after checking only the function version, unlike the sibling specialized handlers which revalidate the type version tag first. Store the type version in the inline cache at specialization time and revalidate owner.class().tp_version_tag against it before using the cached function, deopting to the generic subscript path on mismatch. Assisted-by: Claude * Call __abstractmethods__ __len__ once in object.__new__ The abstract-method check invoked the user-visible __len__ twice: once via length_opt for the count and again while materializing the method names. Derive the count from the materialized list instead, so __len__ runs once. Assisted-by: Claude * Use i64 fast paths for specialized int add/sub/mul Rewrite execute_binary_op_int to box results through new_int via i64 checked arithmetic instead of raw BigInt ops with new_bigint. Add an int_mul helper and a shared int_fast_op that computes the i64 result and falls back to the BigInt operation on to_i64 or checked-op failure. Wire int_mul into both the specialized BinaryOpMultiplyInt handler and the generic execute_bin_op Multiply/InplaceMultiply path, gated on exact int operands so subclasses keep dispatching through _mul/_imul. Assisted-by: Claude * Use i64 fast paths for exact int floordiv and remainder Add floordiv_i64/mod_i64 computing i64 floor-division and divisor-signed remainder, guarding zero divisor and i64::MIN overflow by returning None. Wire them through int_floordiv/int_mod (shared int_div_fast_op boxes via new_int) into the generic execute_bin_op FloorDivide/Remainder and their Inplace variants, gated on exact int operands so subclasses and zero divisors fall through to the existing _floordiv/_mod/_ifloordiv/_imod slow path. Assisted-by: Claude * Skip trashcan and untrack for non-GC-tracked objects in dealloc default_dealloc read is_gc_tracked() only to decide untracking, and entered the trashcan recursion guard unconditionally. Non-GC objects (int, float, str, ...) own no child references that recurse during deallocation, so they need neither the trashcan nor untracking. Read is_gc_tracked() once and gate both trashcan begin/end and untrack on it, removing three thread-local accesses per non-GC object deallocation. Assisted-by: Claude * Merge trashcan depth and defer queue into one thread-local struct trashcan::end accessed two separate thread-locals (DEALLOC_DEPTH and DEALLOC_QUEUE) at the outermost deallocation. Combine both into a single `Trashcan` thread-local holding Cell-based depth and queue fields, so begin and end each reach their state through one thread-local access. The queue is set back before each deferred dealloc call so reentrant begin/end during draining never holds an outstanding borrow. Assisted-by: Claude * Check object layout compatibility on __bases__ and __class__ assignment Add a shared compatible_for_assignment helper that walks each type to the base that fixes its instance layout and rejects the assignment when the old and new layouts differ (basicsize, itemsize, member count, __dict__, __weakref__, and __slots__ names). Wire it into set_bases, which previously performed no layout check, and replace the inline check in __class__ assignment, which compared the two types directly and over-rejected a subclass that adds no layout. Enable test_descr.test_builtin_bases, which the new set_bases check passes. Assisted-by: Claude * Stop the world around GC pointer-reading phases The cycle collector reads each tracked object's interpreter state during reference subtraction, the reachability walk and the strong-reference snapshot, including the localsplus of frames other threads are executing. Those slots are written without synchronization, so the reads are a data race under threading. collect_inner now stops the world before taking the generation read locks and restarts it after the strong-reference snapshot, before the finalizer/weakref/tp_clear phases. A debug assertion checks that no frame on any thread's call stack was classified unreachable. Automatic collections from maybe_collect are deferred to the next bytecode safepoint via a new eval-breaker GC bit instead of running synchronously: a synchronous collection can hold an internal lock (e.g. the lazy frame locals cell) that another thread is blocked on with no way to reach a safepoint, deadlocking the stop. Hardens the stop-the-world machinery for these frequent, concurrent GC stops: attach_thread now honors a pending stop after re-attaching so a thread doing rapid allow_threads calls cannot run past the requester; stop_the_world completion is level-triggered on all-threads-suspended rather than an edge-triggered countdown; and the thread-start started/ready handshakes detach while waiting so the waiter is parkable. Updates the Frame and PyRwLock/PyMutex traversal SAFETY comments to state the actual invariant. Assisted-by: Claude * Add threaded GC vs executing-frame stress snippet Workers churn frame state (recursion, generators, frame cycles) while a collector loops gc.collect() and an introspector walks live frame objects. Exercises the stop-the-world barrier around GC traversal. Assisted-by: Claude * Serialize fork and GC stop-the-world requesters fork() (posix before/after-fork) and the cycle collector both drive the single StopTheWorldState. With no mutual exclusion their requester word and suspension countdown could interleave and be clobbered, so the completion check never converged and a requester waited on itself forever (reproducible parent-side hang when forking with GC enabled). Add one exclusion held for the whole stop->start span of either requester: stop_the_world acquires it before any stop bookkeeping and start_the_world/reset_after_fork release it. The acquire is park-friendly (poll try-lock and honor a pending suspend between tries) so a requester blocked behind an active stop can still be force-parked instead of deadlocking the active requester. The acquirer holds no other lock while spinning. Also correct the PyRwLock traverse SAFETY note: a failed try_read may mean a force-parked thread holds the write lock; skipping is safe because under-traversal only over-approximates liveness. Document the residual gc.collect()-under-a-non-generation-lock exposure at the collect_inner barrier. Assisted-by: Claude * Add fork under concurrent GC stop-the-world snippet Worker threads allocate cyclic garbage with GC enabled while the main thread forks repeatedly and each child collects. Exercises the fork/GC stop-the-world exclusion; the allocation rate is throttled so a collection stays cheap in unoptimized builds. Assisted-by: Claude * Detach while acquiring the import lock The global import lock is held across bytecode by the importlib bootstrap, so its holder can be parked at a safepoint mid-hold. Acquiring it while attached let another thread block attached on the lock, so a stop-the-world requester could wait forever for that attached thread to suspend while the holder stayed parked. Wrap IMP_LOCK acquisition in allow_threads in both _imp.acquire_lock and the pre-fork acquire_imp_lock_for_fork so the wait honors stop-the-world requests. Correct the acquire_exclusion comment: a spinning requester may hold IMP_LOCK (fork) or the collecting mutex (GC); safety relies on those never being acquired attached-blocking by another thread. Assisted-by: Claude * Add concurrent-import vs GC deadlock snippet Two threads re-import modules (contending the import lock) while a third storms the cycle collector and a fourth allocates cyclic garbage. Regressed as a hang before the import lock acquisition was made park-friendly. Assisted-by: Claude * Create call frames untracked and track generators explicitly Add PyPayload::NEW_REF_UNTRACKED (default false, true for Frame) so PyRef::new_ref skips auto-tracking ordinary call frames in the GC. Generator/coroutine/async-generator frames are tracked explicitly in invoke_with_locals before their generator back-reference is installed. run_frame debug-asserts that a datastack frame is untracked on entry. Assisted-by: Claude * Track escaped call frames lazily at datastack release Rewrite release_datastack_frame around the invariant that a datastack frame is never GC-tracked while it runs: strong_count() == 1 means the frame never escaped, so drop localsplus in place without touching the GC; strong_count() > 1 means it escaped, so materialize localsplus onto the heap and then track the frame. This removes the untrack and the untrack-recheck dance from the common non-escaping path. Assisted-by: Claude * Guard the tracked-frame localsplus invariant Document at Frame::traverse that references to a frame are always recorded as graph edges, so the collector reads a frame's localsplus only when the frame is itself a tracked candidate. Add debug assertions at both frame track sites (escaped datastack frames and generator frames) that a frame is heap-backed before it is tracked, so a collector never reads data-stack-resident, still-mutating storage. Assisted-by: Claude * Reword frame/dealloc invariant comments and drop needless borrow - frame.rs: replace the task codename in the tracked-frame localsplus invariant comment with a description of the collector-vs-executing-frame behavior. - object.rs: remove needless borrow on current_cls in the __class__ assignment compatibility check. - core.rs: rewrite the default_dealloc trashcan/untrack-skip comment to state the actual invariant, which now covers untracked non-escaped frames releasing at interpreter depth with bounded recursion. Assisted-by: Claude * Move type SetAttr slot rewrite inside the type lock and tighten cache reads Run update_slot inside the same with_type_lock transaction as modified_inner and the attributes dict mutation, so the version invalidation, dict change, and slot-table rewrite are published together. Load init_version and getitem_version with Acquire in the specialization cache readers to pair with the Release stores in the writers. Assisted-by: Claude * Capture the type version before reading mutable slots in specializers specialize_load_attr, specialize_store_attr, specialize_to_bool, and the CallAllocAndEnterInit path of specialize_call read the version tag with version_for_specialization before inspecting getattro, setattro, the bool/len slots, and tp_new/tp_alloc, so a concurrent install of the corresponding dunder invalidates the version the specialization is cached against. In the BinaryOpSubscr __getitem__ path, check the HEAPTYPE and eval-frame gates before lookup_ref_and_version_interned, which takes the global type lock and may allocate a version tag. Assisted-by: Claude * Use _get_method_dict for the test_type check in the test runner Replace the direct __func__.__dict__ access with the _get_method_dict helper so plain functions without __func__ do not raise AttributeError. Assisted-by: Claude * Fix lint hooks: cspell dictionary entries, spelling fix, formatting Add qsbr to the rustpython dictionary and reborrows/reparenting to the top-level word list. Rename oldto/newto locals to old_to/new_to. Fix Stabilise -> Stabilize typo in a frame.rs comment. Apply cargo fmt to object.rs, type.rs, and frame.rs, and ruff format/check fixes to two extra_tests snippet files. Assisted-by: Claude * Rebuild all slots for type and descendants on __bases__ reassignment Add PyType::update_all_slots, which invalidates version tags and iterates the full SLOT_DEFS name table calling update_slot::<true> for each distinct name. Unlike init_slots, which is additive and driven only by dunder names present in the current MRO, this resets a slot whose method left the MRO instead of leaving a stale dispatcher pointer. Call it from set_bases in place of the previous modified_inner + init_slots pair, so reassigning __bases__ rebuilds slots for the type and every descendant. Add extra_tests/snippets/type_bases_slot_rebuild.py covering removed-method resets on zelf and deep descendants, the wrong-target switch, __getattr__, the added-method mirror, and a swap-away-and-back round trip. Assisted-by: Claude * Derive the thread-local frame stack from the frame chain Remove the per-VM `frames` Vec. current_frame, sys._getframe, sys._getframemodulename, frame.f_back, sys.monitoring re-instrumentation, gc.get_referrers, faulthandler stack dumps and the post-fork slot rebuild now walk the signal-safe CURRENT_FRAME/`previous` chain instead. Add `Py::from_payload_ptr` to recover a frame object from a chain pointer. The cross-thread `ThreadSlot::frames` registry (sys._current_frames, cross-thread f_back, GC stop-the-world assertion) is unchanged. Assisted-by: Claude * Inherit sub-slot fallback into the field being resolved update_one_slot's number/sequence/mapping fallback inherited via the accessor's default field. Left and right binary ops (add/right_add) share one accessor but occupy distinct fields, so resolving an absent right op or deleting it reset the left op's field, dropping a still-defined __add__ dispatcher. Inherit the exact field under resolution instead. Assisted-by: Claude * Gate type_cache_after_fork to its fork caller and use Self in downcast type_cache_after_fork is only called from the unix fork path, so gate it with all(feature = "host_env", unix) to match the caller and avoid a dead-code error on non-unix targets. Replace the explicit PyType with Self in the modified_inner downcast. Assisted-by: Claude * Unmark test_attr and test_method_call_error in test_monitoring Both TestLoadSuperAttr tests now pass; remove their expectedFailure markers. Assisted-by: Claude * Run tp_new specialization __init__ without a trampoline frame CallAllocAndEnterInit ran __init__ inside a synthetic init-cleanup shim frame whose code carried the __init__ name. Since the thread frame stack is derived from the frame chain, that shim frame was visible to sys._getframe, f_back walks, traceback construction and inspect.stack / inspect.trace. Its code object has empty co_positions, so a stack walk that reached it raised StopIteration inside inspect._get_code_position, cascading into unrelated failures (test_inspect trace/stack/frame, asyncio source traceback). Call __init__ directly via run_frame, enforce the __init__() should return None contract inline, and drop the shim: the init-cleanup code object, its builder, with_frame_untraced, monitoring_disabled_for_code, and the extra-frame datastack/recursion budget. Removing the second frame per construction also cuts the specialization's per-call cost. Assisted-by: Claude * Replace per-call thread-frame mutex with an atomic top-of-stack (unix) On unix threading builds, publish each thread's top Python frame in a single relaxed AtomicPtr store from set_current_frame instead of pushing onto a parking_lot::Mutex<Vec<FramePtr>> per call. Cross-thread readers (sys._current_frames, cross-thread f_back, faulthandler.dump_traceback, the GC unreachable debug-assert) run under stop-the-world and walk the published top frame down the Frame::previous chain; the owning thread is then parked at a safepoint, so the pointer and the frames it reaches are quiescent and alive. The faulthandler watchdog is a plain OS thread that cannot stop-the-world, so it walks the chain lock-free and best-effort. Non-unix threading builds have no stop-the-world and keep the existing mutex-guarded frame stack unchanged. Assisted-by: Claude * Skip exc_info save/restore for callees that never touch the slot with_frame saves and restores the shared exc_info slot around every Python call to contain frames that leave it unbalanced. Cache a has_exc_handling bit on PyCode at creation, set when the bytecode contains any opcode that calls vm.set_exception (PushExcInfo, PopExcept, CheckEgMatch, EndAsyncFor, InstrumentedEndAsyncFor). A callee whose code has none of these cannot mutate the slot, so the save and restore are skipped for it. Generators go through resume_gen_frame and are unaffected. Assisted-by: Claude * Return a write-through FrameLocalsProxy from frame.f_locals Optimized (function) frames now expose `f_locals` as a `FrameLocalsProxy` implementing PEP 667 semantics instead of a cached snapshot dict: - reads go live through the fast-local slots; each access mints a fresh proxy; keys that do not name a fast local are stored in a per-frame `f_extra_locals` side dict and folded into `locals()`. - writes to a fast-local key store into the slot (or its cell) in place; deleting a fast local raises ValueError; extra keys delete normally. - full mapping protocol: keys/values/items (lists), get/pop/setdefault, update (dict or FrameLocalsProxy only), __or__/__ior__/__ror__ (dict result), copy (plain dict), __reduce__ blocks pickling/copy, repr with recursion guard, mapping-pattern and Mapping ABC support. Class/module/exec frames keep returning their namespace mapping directly. Cross-thread access to a frame running on another thread still raises RuntimeError. A closed generator now keeps its frame locals when a durable frame reference escaped (f_locals proxy, sys._getframe, f_back), matching take_ownership; the escape is tracked with a per-frame flag. The snapshot-then-fold locals_to_fast/locals_dirty write-back is retired since proxy writes reach the slots directly. Assisted-by: Claude * Retain the caller frame so f_back resolves after it returns When a frame escapes its execution (referenced through a traceback, `sys._getframe`, `f_locals`, ...), capture a strong reference to its caller at release time. `f_back` consults it once the caller has left the live frame chain, so the Python-visible frame chain survives return. The retained reference is a GC-traversed edge and is cleared by `frame.clear()`, so ancestor chains stay collectable. Assisted-by: Claude * Fix debug-build native stack overflow on deep recursion Make check_c_stack_overflow one-sided (trip whenever the stack pointer is below the soft limit) so a single native frame larger than the margin cannot step past the danger band undetected. Raise the debug STACK_MARGIN_BYTES from 4096 to 16384 words so the margin exceeds a single debug interpreter frame, leaving headroom to raise RecursionError. Release margin unchanged. Clamp the soft-limit margin to half the stack so small explicit thread stacks do not get a soft limit above their stack top. Assisted-by: Claude * Allocate exception instance __dict__ lazily Exception construction went through into_ref_with_type, which eagerly allocated an empty instance dict for every HAS_DICT type. Add into_ref_with_type_lazy_dict, which builds the instance with an unallocated dict slot, and route the four exception construction sites (PyBaseException, PyOSError, OSErrorBuilder, PyBaseExceptionGroup) through it. The dict now materializes on first attribute write or __dict__ access via the existing get_or_insert path. add_note and PyImportError::slot_init now obtain the dict through object_get_dict so they materialize it instead of assuming it exists. A freshly constructed exception no longer reports an empty dict in gc.get_referents, matching the reference interpreter. Assisted-by: Claude * Allocate exception __dict__ lazily in vm.new_exception Route vm.new_exception() through into_ref_with_type_lazy_dict so internally raised exceptions (new_type_error, new_value_error, etc.) start without an instance dict, matching the slot_new path. The dict is materialized on the first attribute write or __dict__ access. Assisted-by: Claude * Validate FrameLocalsProxy.update() arguments Reject keyword arguments and require exactly one positional argument, raising TypeError with the "takes no keyword arguments" and "takes exactly one argument (N given)" messages. Assisted-by: Claude * Address review nits in frame and faulthandler - Drop stale vm.frames reference from the release_datastack_frame uniqueness argument. - Assert has_exc_handling when unwinding an Except-typed stack slot, documenting the invariant that guards the shared exc_info write. - Truncate the specialized __init__ return-type name to 200 chars, matching the unspecialized wrapper. - Reword two comments to describe behavior without prose references to CPython. Assisted-by: Claude * Use scopeguard for start_the_world in faulthandler dump_all_threads Wrap the unix stop-the-world registry walk in a scope with scopeguard::defer! so start_the_world runs on panic, matching the f_back and get_all_current_frames sites. Add scopeguard to the stdlib dependencies. Also correct the restore_exception doc comment to name with_frame after the rename. Assisted-by: Claude * Gitignore docs/superpowers Assisted-by: Claude * Rename type_bases_slot_rebuild.py to builtin_type_bases.py Match the builtin_* naming convention of extra_tests/snippets. Assisted-by: Claude * Apply rustfmt, ruff, and cspell lint fixes Reformat with rustfmt, fix import spacing in builtin_type_bases.py with ruff, and add "pointee" to the cspell word list. Assisted-by: Claude * Gate unix-only QSBR methods to their call sites `online`, `drain_all`, and `reset_after_fork` are called only from unix code (thread attach/detach, post-fork reset), and `offline` from unix code plus a unit test. Gate them with matching cfg so `-D dead_code` does not fire on non-unix targets. Assisted-by: Claude * Visit the function closure tuple as a GC edge PyFunction::traverse visited the cells inside the closure tuple instead of the tuple object itself, so the tuple's reference from the function was never subtracted during cycle collection. A closure tuple that reached back to its function (or, through a frame retained by f_back, to a Thread) was stranded as a false GC root and never collected, leaking the whole cycle. Visit the tuple itself, matching clear(). Assisted-by: Claude * Apply formatting hook fix to builtin_type_bases.py Assisted-by: Claude * Unmark asyncgen finalization-by-gc tests in test_base_events test_asyncgen_finalization_by_gc and test_asyncgen_finalization_by_gc_in_other_thread now pass; GC finalizes the async generators. Assisted-by: Claude * Unmark test_sni_callback_refcycle in test_ssl The servername-callback reference cycle is now collected by GC. Assisted-by: Claude * Widen GC stop-the-world gates from unix-only to all threading builds Change `cfg(all(unix, feature = "threading"))` to `cfg(feature = "threading")` on the stop-the-world machinery so it also compiles and runs on non-unix threading builds: - StopTheWorldState, its stats, stw_trace, and the stop_the_world field - ThreadSlot state/stop_requested/thread fields and their initializers - wait_while_suspended/attach_thread/detach_thread/suspend_if_needed/do_suspend, allow_threads, stop_requested_for_current_thread, and the enter_vm / VmBootstrapGuard / attach_current_thread / release_current_thread / cleanup attach-state wiring - eval_breaker_tripped, check_signals, run_scheduled_gc, signal GC_BIT / schedule_gc / take_gc_scheduled, and the frame.rs safepoint call - CollectStopTheWorld and its use in collect_inner - QSBR::online/offline, now called from attach/detach on all threading builds - debug_assert_current_thread_attached and its type-cache call sites maybe_collect defers auto-collection to the bytecode safepoint on every threading build instead of only unix; non-threading builds keep the inline collect. stw_trace writes to std stderr on non-unix. top_frame publishing (CURRENT_TOP_FRAME_SLOT, set_current_frame), the frame-walk debug assert in collect_inner, and the fork reinit helpers remain unix-only; non-unix keeps ThreadSlot::frames for introspection. Assisted-by: Claude * Detach current thread around blocking _winapi/_overlapped waits Wrap the blocking Windows wait calls in `vm.allow_threads` so the calling thread transitions ATTACHED -> DETACHED for the duration of the wait: - _winapi: WaitForSingleObject, WaitForMultipleObjects, BatchedWaitForMultipleObjects, ConnectNamedPipe, ReadFile, Overlapped.GetOverlappedResult - _overlapped: Overlapped.getresult These previously blocked while ATTACHED, so a stop-the-world requester could never suspend the thread and spun in its wait loop indefinitely. * Format WaitForMultipleObjects allow_threads closure * Treat concurrent sni_callback removal as no-op in invoke_sni_callback * Assert capi refcount on a fresh mortal list instead of the int type object The refcount test asserted exact incref/decref deltas on PyInt's shared type object. That object's reference count is perturbed by the other capi tests running in parallel, and is immortal under some interpreter configurations, so the deltas were not reliably +1/-1. Assert them on a freshly created, uniquely owned list whose reference count is private to the test and mortal. Assisted-by: Claude * Traverse and GC-track PyOSError instances The `#[pyexception]` struct macro now forwards a `traverse` option to the generated `#[pyclass]`, and `ExceptionItemMeta` accepts the `traverse` key. `PyOSError` is marked `traverse = "manual"`, so `HAS_TRAVERSE` is true and OSError-family instances are tracked at creation and traversed by the collector. `PyOSError::traverse` now visits the underlying `PyBaseException` (traceback, cause, context, args) instead of `PyException::try_traverse`, which was a no-op because `PyException` has `HAS_TRAVERSE = false`. * Unmark test_blockingioerror in test_io The BlockingIOError reference cycle is now collected by GC. Assisted-by: Claude	2026-07-08 15:21:41 +09:00
Jeong, YunWon	60c1f77288	update cspell config (#7452) ... * add allowCompoundWords * cspell dict	2026-03-17 20:18:31 +09:00