* Reinit IO buffer locks after fork to prevent deadlocks
BufferedReader/Writer/TextIOWrapper use PyThreadMutex internally.
If a parent thread held one of these locks during fork(), the child
would deadlock on any IO operation.
Add reinit_after_fork() to RawThreadMutex and call it on sys.stdin/
stdout/stderr in the child process fork handler, analogous to
CPython's _PyIO_Reinit().
* Address review: unsafe fn + decoder lock reinit
- Mark reinit_std_streams_after_fork as unsafe fn to encode
fork-only precondition, update call site in posix.rs
- Reinit IncrementalNewlineDecoder's PyThreadMutex via
TextIOWrapper's decoder field to prevent child deadlocks
* Auto-format: cargo fmt --all
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* Add PyType vectorcall and use vectorcall in all specialized call fallbacks
* Add vectorcall slot for PyMethodDescriptor and PyWrapper
* Add FuncArgs::from_vectorcall_owned and simplify vectorcall fallback paths
Remove has_vectorcall checks from execute_call_vectorcall and
execute_call_kw_vectorcall. The invoke_vectorcall fallback now uses
from_vectorcall_owned to move args instead of cloning.
* Add CALL_ALLOC_AND_ENTER_INIT specialization
Optimizes user-defined class instantiation MyClass(args...)
when tp_new == object.__new__ and __init__ is a simple
PyFunction. Allocates the object directly and calls __init__
via invoke_exact_args, bypassing the generic type.__call__
dispatch path.
* Invalidate JIT cache when __code__ is reassigned
Change jitted_code from OnceCell to PyMutex<Option<CompiledCode>> so
it can be cleared on __code__ assignment. The setter now sets the
cached JIT code to None to prevent executing stale machine code.
* Atomic operations for specialization cache
- range iterator: deduplicate fast_next/next_fast
- Replace raw pointer reads/writes in CodeUnits with atomic
operations (AtomicU8/AtomicU16) for thread safety
- Add read_op (Acquire), read_arg (Relaxed), compare_exchange_op
- Use Release ordering in replace_op to synchronize cache writes
- Dispatch loop reads opcodes atomically via read_op/read_arg
- Fix adaptive counter access: use read/write_adaptive_counter
instead of read/write_cache_u16 (was reading wrong bytes)
- Add pre-check guards to all specialize_* functions to prevent
concurrent specialization races
- Move modified() before attribute changes in type.__setattr__
to prevent use-after-free of cached descriptors
- Use SeqCst ordering in modified() for version invalidation
- Add Release fence after quicken() initialization
* Fix slot wrapper override for inherited attributes
For __getattribute__: only use getattro_wrapper when the type
itself defines the attribute; otherwise inherit native slot from
base class via MRO.
For __setattr__/__delattr__: only store setattro_wrapper when
the type has its own __setattr__ or __delattr__; otherwise keep
the inherited base slot.
* Fix StoreAttrSlot cache overflow corrupting next instruction
write_cache_u32 at cache_base+3 writes 2 code units (positions 3 and 4),
but STORE_ATTR only has 4 cache entries (positions 0-3). This overwrites
the next instruction with the upper 16 bits of the slot offset.
Changed to write_cache_u16/read_cache_u16 since member descriptor offsets
fit within u16 (max 65535 bytes).
* Exclude method_descriptor from has_python_cmp check
has_python_cmp incorrectly treated method_descriptor as Python-level
comparison methods, causing richcompare slot to use wrapper dispatch
instead of inheriting the native slot.
* Fix CompareOpFloat NaN handling
partial_cmp returns None for NaN comparisons. is_some_and incorrectly
returned false for all NaN comparisons, but NaN != x should be true
per IEEE 754 semantics.
* Fix invoke_exact_args borrow in CallAllocAndEnterInit
* Distinguish Python method vs not-found in slot MRO lookup
Change lookup_slot_in_mro to return a 3-state SlotLookupResult
enum (NativeSlot/PythonMethod/NotFound) instead of Option<T>.
Previously, both "found a Python-level method" and "found nothing"
returned None, causing incorrect slot inheritance. For example,
class Test(Mixin, TestCase) would inherit object.slot_init from
Mixin via inherit_from_mro instead of using init_wrapper to
dispatch TestCase.__init__.
Apply this fix consistently to all slot update sites:
update_main_slot!, update_sub_slot!, TpGetattro, TpSetattro,
TpDescrSet, TpHash, TpRichcompare, SqAssItem, MpAssSubscript.
* Extract specialization helper functions to reduce boilerplate
- deoptimize() / deoptimize_at(): replace specialized op with base op
- adaptive(): decrement warmup counter or call specialize function
- commit_specialization(): replace op on success, backoff on failure
- execute_binary_op_int() / execute_binary_op_float(): typed binary ops
Removes 10 duplicate deoptimize_* functions, consolidates 13 adaptive
counter blocks, 6 binary op handlers, and 7 specialize tail patterns.
Also replaces inline deopt blocks in LoadAttr/StoreAttr handlers.
* Improve specialization guards and fix mark_stacks
- CONTAINS_OP_SET: add frozenset support in handler and specialize
- TO_BOOL_ALWAYS_TRUE: cache type version instead of checking slots
- LOAD_GLOBAL_BUILTIN: cache builtins dict version alongside globals
- mark_stacks: deoptimize specialized opcodes for correct reachability
* Auto-format: cargo fmt --all
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Add missing reinit_mutex_after_fork for vm.state.monitoring
PyMutex in reinit_locks_after_fork.
Use Mutex::raw() accessor in reinit_parking_lot_mutex instead
of pointer cast from struct start for layout safety.
* Remove unsafe ctx cast in make_class
Use Context::genesis() directly in make_class to obtain
&'static Context, eliminating the raw pointer cast.
* make_class -> make_static_type
* Add vectorcall (PEP 590) dispatch for function calls
Add VectorCallFunc slot to PyTypeSlots and vectorcall dispatch path
in the interpreter loop for Call and CallKw instructions.
Implement vectorcall for PyFunction (with fast path for simple
positional-only calls that fills fastlocals directly), PyBoundMethod
(avoids prepend_arg O(n) shift), and PyNativeFunction.
Add FuncArgs::from_vectorcall helper for fallback conversion.
Vectorcall slot is inherited with call slot and cleared when
__call__ is overridden in Python subclasses.
* Optimize vectorcall: move args instead of clone, use vectorcall in specialized paths
- invoke_exact_args takes Vec by value and uses drain() to move args
into fastlocals instead of cloning (eliminates refcount overhead)
- CallPyGeneral and CallBoundMethodGeneral now call vectorcall_function
directly instead of going through FuncArgs + prepend_arg + invoke
- CallKwPy and CallKwBoundMethod use vectorcall_function with kwnames
- vectorcall_bound_method uses insert(0) on existing Vec instead of
allocating a second Vec
* Auto-format: cargo fmt --all
* Fix vectorcall_native_function kwarg slice out-of-bounds
When needs_self was true and kwargs were present, pos_args only
contained positional args (self + original positionals) but
from_vectorcall expected kwarg values to follow in the slice.
Build the full args array (self + all original args including kwarg
values) before passing to from_vectorcall.
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
4096-entry direct-mapped cache keyed by (tp_version_tag, interned_name_ptr)
using lock-free SeqLock pattern for concurrent access.
- find_name_in_mro() checks cache before MRO walk
- has_name_in_mro() avoids cloning on cache hit
- Auto-assigns version tags on cache miss
- Invalidation via modified() (tp_version_tag=0) with value drops
- TYPE_CACHE_CLEARING flag suppresses re-population during GC drops
- Clear method cache during GC collection to break reference cycles
* Reduce per-instruction overhead in interpreter loop
- Defer prev_line update to when tracing is active
- Inline skip_caches_if_fallthrough, compute cache_entries once per instruction
* Unmark test_pdb_set_trace EXPECTED_FAILURE
prev_line deferral fix corrects pdb line tracking, making
these doctests pass.
* Use Mutex::raw() accessor in reinit_mutex_after_fork
Use lock_api's Mutex::raw() to access the underlying RawMutex
instead of casting &PyMutex<T> directly. This avoids layout
assumptions about lock_api::Mutex<R, T> field ordering.
* Replace force_unlock with reinit_*_after_fork
Replace all force_unlock() + try_lock() patterns with zero-based
reinit that bypasses parking_lot internals entirely. After fork(),
the child is single-threaded so reinited locks won't contend.
Add reinit_rwlock_after_fork to common::lock alongside the existing
reinit_mutex_after_fork. Replace force_unlock_after_fork methods in
codecs, intern, and gc_state with reinit_after_fork equivalents.
This fixes after_fork_child silently dropping thread handles when
try_lock() failed on per-handle Arc<Mutex> locks.
Change the code field from PyMutex<PyRef<PyCode>> to PyAtomicRef<PyCode>,
eliminating mutex lock/unlock on every function call. The setter uses
swap_to_temporary_refs for safe deferred drop of the old code object.
Defer locals dict allocation for function frames until first access.
Most function frames only use fastlocals and never touch the locals
dict, so this skips one dict heap allocation per function call.
Also remove a redundant code.clone() in invoke_with_locals.
Func call ~23% faster, method call ~15% faster in benchmarks.
* Fix _at_fork_reinit to write INIT directly instead of calling unlock()
unlock() goes through unlock_slow() which accesses parking_lot's
global hash table to unpark waiters. After fork(), this hash table
contains stale entries from dead parent threads, making unlock_slow()
unsafe. Writing INIT directly bypasses parking_lot internals entirely.
* Add import lock (IMP_LOCK) reinit after fork
The import lock is a ReentrantMutex that was never reinit'd after
fork(). If a parent thread held it during fork, the child would
deadlock on any import. Only reset if the owner is a dead thread;
if the surviving thread held it, normal unlock still works.
* Revert __class__ lookup skip in object_isinstance
The optimization to skip __class__ lookup based on getattro check
was incorrect: a class can override __class__ as a property while
still using standard __getattribute__. Revert to always performing
the lookup, matching CPython's object_isinstance behavior.
* Collapse nested if in object_isinstance for clippy
* Fix symbol table sub_table desync for non-simple annotation targets
Non-simple annotations (subscript/attribute/parenthesized targets like
`a[0]: expr`) were scanned in the annotation scope during symbol table
analysis, creating sub_tables for any comprehensions. But codegen only
compiles simple name annotations into __annotate__, so those sub_tables
were never consumed. This caused subsequent simple annotations'
comprehension sub_tables to get the wrong index, resulting in
"the symbol 'X' must be present in the symbol table" errors.
Fix: skip entering annotation scope for non-simple annotations since
they are never compiled into __annotate__.
* Validate forbidden expressions in non-simple annotation targets
Fix cspell "desynchronize" warning and validate yield/await/named/async
comprehension expressions in non-simple annotations without creating
annotation scopes.
* Restore in_annotation flag before propagating error
* Implement LOAD_ATTR inline caching with adaptive specialization
Add type version counter (tp_version_tag) to PyType with subclass
invalidation cascade. Add cache read/write methods (u16/u32/u64)
to CodeUnits. Implement adaptive specialization in load_attr that
replaces the opcode with specialized variants on first execution:
- LoadAttrMethodNoDict: cached method lookup for slotted types
- LoadAttrMethodWithValues: cached method with dict shadow check
- LoadAttrInstanceValue: direct dict lookup skipping descriptors
Specialized opcodes guard on type_version_tag and deoptimize back
to generic LOAD_ATTR with backoff counter on cache miss.
* Add BINARY_OP and CALL adaptive specialization
BINARY_OP: Specialize int add/subtract/multiply and float
add/subtract/multiply with type guards and deoptimization.
CALL: Add func_version to PyFunction, specialize simple
function calls (CallPyExactArgs, CallBoundMethodExactArgs)
with invoke_exact_args fast path that skips FuncArgs
allocation and fill_locals_from_args.
* Lazy quickening for adaptive specialization counters
Move counter initialization from compile-time to RESUME execution,
matching CPython's _PyCode_Quicken pattern. Store counter in CACHE
entry's arg byte to preserve op=Instruction::Cache for dis/JIT.
Add PyCode.quickened flag for one-time initialization.
* Add Instruction::deoptimize() and CodeUnits::original_bytes()
- deoptimize() maps specialized opcodes back to their base adaptive variant
- original_bytes() produces deoptimized bytecode with zeroed CACHE entries
- co_code now returns deoptimized bytes, _co_code_adaptive returns current bytes
- Marshal serialization uses original_bytes() instead of raw transmute
* Fix monitoring and specialization interaction
- cache_entries() returns correct count for instrumented opcodes
- deoptimize() maps instrumented opcodes back to base
- quicken() skips adaptive counter for instrumented opcodes
- instrument_code Phase 3 deoptimizes specialized opcodes and
clears CACHE entries to prevent stale pointer dereferences
* Address review: bounds checks, UB fix, version overflow, error handling
- Add bounds checks to read_cache_u16/u32/u64
- Fix quicken() aliasing UB by using &mut directly
- Add JumpBackwardJit/JumpBackwardNoJit to deoptimize()
- Guard can_specialize_call with NEWLOCALS flag check
- Use compare_exchange_weak for version tag to prevent wraparound
- Propagate dict lookup errors in LoadAttrMethodWithValues
- Apply adaptive backoff on version tag assignment failure
- Remove duplicate imports in frame.rs
* Remove intermediate Vec allocation in unpack_sequence fast path
Push elements directly from tuple/list slice in reverse order
instead of cloning into a temporary Vec first.
* Use read-only atomic load before swap in check_signals
Add Relaxed load guard before the Acquire swap to avoid cache-line
invalidation on every instruction dispatch when no signal is pending.
* Cache builtins downcast in ExecutingFrame for LOAD_GLOBAL
Pre-compute builtins.downcast_ref::<PyDict>() at frame entry and reuse
the cached reference in load_global_or_builtin and LoadBuildClass.
Also add get_chain_exact to skip redundant exact_dict type checks.
* Add number Add slot to PyStr for direct str+str dispatch
binary_op1 can now resolve str+str addition directly via the number
slot instead of falling through to the sequence concat path.
* Guard FastLocals access in locals() with try_lock on state mutex
Address CodeRabbit review: f_locals() could access fastlocals without
synchronization when called from another thread. Use try_lock on the
state mutex so concurrent access is properly serialized.
* Use exact type check for builtins_dict cache
downcast_ref::<PyDict>() matches dict subclasses, causing
get_chain_exact to bypass custom __getitem__ overrides.
Use downcast_ref_if_exact to only fast-path exact dict types.
* Consolidate with_recursion in _cmp to single guard
Move the recursion depth check to wrap the entire _cmp body
instead of each individual call_cmp direction, reducing Cell
read/write pairs and scopeguard overhead per comparison.
* Add opcode-level fast paths for FOR_ITER, COMPARE_OP, BINARY_OP
- FOR_ITER: detect PyRangeIterator and bypass generic iterator
protocol (atomic slot load + indirect call)
- COMPARE_OP: inline int/float comparison for exact types,
skip rich_compare dispatch and with_recursion overhead
- BINARY_OP: inline int add/sub with i64 checked arithmetic
to avoid BigInt heap allocation and binary_op1 dispatch
* Also check globals is exact dict for LOAD_GLOBAL fast path
get_chain_exact bypasses __missing__ on dict subclasses.
Move get_chain_exact to PyExact<PyDict> impl with debug_assert,
and have get_chain delegate to it. Store builtins_dict as
Option<&PyExact<PyDict>> to enforce exact type at compile time.
Use PyRangeIterator::next_fast() instead of pub(crate) fields.
Fix comment style issues.
* Add str(int) and repr(int) fast path using i64 conversion
- Skip __str__ method resolution for exact PyInt in PyObject::str()
- Use i64::to_string() for small integers, BigInt::to_string() for large ones
- ~36% improvement in str(int) benchmarks
* Extract PyInt::to_str_radix_10() to deduplicate i64 fast path logic
In object_isinstance(), when is_subtype() returns false, the __class__
attribute lookup via get_attribute_opt is redundant for objects using
standard __getattribute__, since __class__ is a data descriptor on
object that always returns obj.class().
* [update_lib] fast date lookup for todo
* add deps
* Auto-format: ruff format
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* Implement PyStackRef borrowed stack references for LOAD_FAST_BORROW
* Fix PyStackRef: use NonZeroUsize for niche optimization, revert borrowed refs
- Change PyStackRef.bits from usize to NonZeroUsize so Option<PyStackRef>
has the same size as Option<PyObjectRef> via niche optimization
- Revert LoadFastBorrow to use clone instead of actual borrowed refs
to avoid borrowed refs remaining on stack at yield points
- Add static size assertions for Option<PyStackRef>
- Add stackref, fastlocal to cspell dictionary
- Remove debug eprintln statements
- Fix clippy warning for unused push_borrowed
* Add borrowed ref debug_assert to InstrumentedYieldValue, clean up comments
Shahar Naveh