* apply more allow_threads
* Simplify STW thread state transitions
- Fix park_detached_threads: successful CAS no longer sets
all_suspended=false, avoiding unnecessary polling rounds
- Replace park_timeout(50µs) with park() in wait_while_suspended
- Remove redundant self-suspension in attach_thread and detach_thread;
the STW controller handles DETACHED→SUSPENDED via park_detached_threads
- Add double-check under mutex before condvar wait to prevent lost wakes
- Remove dead stats_detach_wait_yields field and add_detach_wait_yields
* Representable for ThreadHandle
* Set ThreadHandle state to Running in parent thread after spawn
Like CPython's ThreadHandle_start, set RUNNING state in the parent
thread immediately after spawn() succeeds, rather than in the child.
This eliminates a race where join() could see Starting state if called
before the child thread executes.
Also reverts the macOS skip for test_start_new_thread_failed since the
root cause is fixed.
* Set ThreadHandle state to Running in parent thread after spawn
* Add debug_assert for thread state in start_the_world
* Unskip now-passing test_get_event_loop_thread and test_start_new_thread_at_finalization
* Wrap IO locks and file ops in allow_threads
Add lock_wrapped to ThreadMutex for detaching thread state
while waiting on contended locks. Use it for buffered and
text IO locks. Wrap FileIO read/write in allow_threads via
crt_fd to prevent STW hangs on blocking file operations.
* Use std::sync for thread start/ready events
Replace parking_lot Mutex/Condvar with std::sync (pthread-based)
for started_event and handle_ready_event. This prevents hangs
in forked children where parking_lot's global HASHTABLE may be
corrupted.
On Windows, SimpleQueue skips write locking because pipe
writes are assumed atomic. Without GIL, PipeConnection.
_send_bytes races on _send_ov when multiple threads call
send_bytes concurrently (e.g. _terminate_pool vs workers).
Wait for pending overlapped writes inside WriteFile before
returning to Python, so ERROR_IO_PENDING is never exposed
and the _send_ov assignment window is negligible.
* Suspend Python threads before fork()
Add stop-the-world thread suspension around fork() to prevent
deadlocks from locks held by dead parent threads in the child.
- Thread states: DETACHED / ATTACHED / SUSPENDED with atomic CAS
transitions matching _PyThreadState_{Attach,Detach,Suspend}
- stop_the_world / start_the_world: park all non-requester threads
before fork, resume after (parent) or reset (child)
- allow_threads (Py_BEGIN/END_ALLOW_THREADS): detach around blocking
syscalls (os.read/write, waitpid, Lock.acquire, time.sleep) so
stop_the_world can force-park via CAS
- Acquire/release import lock around fork lifecycle
- zero_reinit_after_fork: generic lock reset for parking_lot types
- gc_clear_raw: detach dict instead of clearing entries
- Lock-free double-check for descriptor cache reads (no read-side
seqlock); write-side seqlock retained for writer serialization
- fork() returns PyResult, checks PythonFinalizationError, calls
sys.audit
* Object header slimming: prefix allocation for ObjExt
Extract dict, weak_list, and slots fields from PyInner<T> into a
separate ObjExt struct allocated as a prefix before PyInner using
Layout::extend(). Objects that don't need these fields (int, str,
float, list, tuple, dict, etc.) skip the prefix entirely.
- Add HAS_WEAKREF flag to PyTypeFlags for per-type weakref control
- Add HAS_EXT bit to GcBits indicating prefix presence
- Define ObjExt struct with dict, weak_list, slots
- Shrink PyInner header from ~80-88 bytes to ~32 bytes for lightweight objects
- Update all accessor methods to go through ext_ref()
- Update bootstrap type hierarchy to use prefix allocation
- Add __weakref__ getset descriptor for heap types
- Set HAS_WEAKREF on builtin types that support weak references
- Remove test_weak_keyed_bad_delitem expectedFailure (now passes)
* Add HAS_WEAKREF to _asyncio Future/Task, rename weakref helpers
- Add HAS_WEAKREF flag to PyFuture and PyTask (matches CPython)
- Rename subtype_getweakref/setweakref to subtype_get_weakref/set_weakref
to fix cspell unknown word lint
* Add HAS_WEAKREF to array, deque, _grouper; remove expectedFailure markers
- Add HAS_WEAKREF to PyArray and PyDeque (matches CPython)
- Add HAS_WEAKREF to PyItertoolsGrouper (internal use by groupby)
- Remove 6 expectedFailure markers from test_dataclasses for weakref/slots
tests that now pass
* Add HAS_WEAKREF to code, union, partial, lock, IO, mmap, sre, sqlite3, typevar types
Add HAS_WEAKREF flag to built-in types that support weakref:
- PyCode, PyUnion, PyPartial, Lock, RLock
- All IO base/concrete classes (_IOBase, _RawIOBase, _BufferedIOBase,
_TextIOBase, BufferedReader, BufferedWriter, BufferedRandom,
BufferedRWPair, TextIOWrapper, StringIO, BytesIO, FileIO,
WindowsConsoleIO)
- PyMmap, sre Pattern, sqlite3 Connection/Cursor
- TypeVar, ParamSpec, ParamSpecArgs, ParamSpecKwargs, TypeVarTuple
Remove 3 expectedFailure markers from test_descr for now-passing tests.
* Add HAS_DICT to type flags and handle non-METHOD/CLASS in descr_get
- Add HAS_DICT flag to PyType (type metaclass) alongside existing
HAS_WEAKREF. All type objects are instances of type and need dict
support, matching CPython's PyType_Type.
- Replace unimplemented!() in PyMethodDescriptor::descr_get with
fallback to bind obj directly, matching CPython's method_get which
uses PyCFunction_NewEx for non-METH_METHOD methods.
* Fix ext detection, HeapMethodDef ownership, WASM error
- Remove HAS_EXT gc_bits flag; detect ext from type flags
using raw pointer reads to avoid Stacked Borrows violations
- Store HeapMethodDef owner in payload instead of dict hack
- Clear dict entries in gc_clear_raw to break cycles
- Add WASM error fallback when exception serialization fails
* Implement locale-aware 'n' format specifier for int, float, complex
Add LocaleInfo struct and locale-aware formatting methods to FormatSpec.
The 'n' format type now reads thousands_sep, decimal_point, and grouping
from C localeconv() and applies proper locale-based number grouping.
Remove @unittest.skip from test_format.test_locale.
* Fix complex 'n' format and remove locale expectedFailure markers
Rewrite format_complex_locale to reuse format_complex_re_im, matching
formatter_unicode.c: add_parens=0 and skip_re=0 for 'n' type.
Remove @expectedFailure from test_float__format__locale and
test_int__format__locale in test_types.py.
* Auto-format: cargo fmt --all
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* Remove cells_frees duplicate storage from Frame
Cell/free variable objects were stored in both a separate
`Box<[PyCellRef]>` (cells_frees field) and in the localsplus
fastlocals array. Remove the redundant cells_frees field and
access cell objects directly through localsplus, eliminating
one Box allocation and N clone operations per frame creation.
* Extract InterpreterFrame from Frame with Deref wrapper
Introduce InterpreterFrame struct containing all execution state
fields previously on Frame. Frame now wraps InterpreterFrame via
FrameUnsafeCell and implements Deref for transparent field access.
localsplus and prev_line are plain fields on InterpreterFrame
(no longer individually wrapped in FrameUnsafeCell) since the
entire InterpreterFrame is wrapped at the Frame level.
* Auto-format: cargo fmt --all
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* Fix thread-safety in GC, type cache, and instruction cache
GC / refcount:
- Add safe_inc() check for strong()==0 in RefCount
- Add try_to_owned() to PyObject for atomic refcount acquire
- Replace strong_count()+to_owned() with try_to_owned() in GC
collection and weakref callback paths to prevent TOCTOU races
Type cache:
- Add proper SeqLock (sequence counter) to TypeCacheEntry
- Readers spin-wait on odd sequence, validate after read
- Writers bracket updates with begin_write/end_write
- Use try_to_owned + pointer revalidation on read path
- Call modified() BEFORE attribute modification in set_attr
Instruction cache:
- Add pointer_cache (AtomicUsize array) to CodeUnits for
single atomic pointer load/store (prevents torn reads)
- Add try_read_cached_descriptor with try_to_owned + pointer
and version revalidation after increment
- Add write_cached_descriptor with version-bracketed writes
RLock:
- Fix release() to check is_owned_by_current_thread
- Add _release_save/_acquire_restore methods
* Fix RLock _acquire_restore tuple handling and unxfail threading test
* Align type cache seqlock writer protocol with CPython
* RLock: use single parking_lot level, track recursion manually
Instead of calling lock()/unlock() N times for recursion depth N,
keep parking_lot at 1 level and manage the count ourselves.
This makes acquire/release O(1) and matches CPython's
_PyRecursiveMutex approach (lock once + set level directly).
* Add try_to_owned_from_ptr to avoid &PyObject on stale ptrs
Use addr_of! to access ref_count directly from a raw pointer
without forming &PyObject first. Applied in type cache and
instruction cache hit paths where the pointer may be stale.
* Fix CI: spelling typo and xfail flaky test_thread_safety
- Fix "minimising" -> "minimizing" for cspell
- xfail test_thread_safety: dict iteration races with
concurrent GC mutations in _finalizer_registry
* Add freelists for PyComplex, PyInt, PyBoundMethod, PyRange
- PyComplex(100), PyInt(100), PyBoundMethod(20), PyRange(6)
- Use try_with instead of with in all freelist push/pop to prevent
panic on thread-local access during thread teardown
* Use alloc::dealloc in FreeList::Drop to avoid thread-local access panic
During thread teardown, Box::from_raw triggers cascading destructors
that may access already-destroyed thread-local storage (GC state, other
freelists). Use raw dealloc instead to free memory without running
destructors.
* Auto-format: cargo fmt --all
* Fix clippy: use core::alloc::Layout instead of alloc::alloc::Layout
* Address review: PyBoundMethod clear=false, update FreeList doc comment
- Set clear=false on PyBoundMethod (tp_clear=NULL in classobject.c)
- Update FreeList doc comment to match actual Drop behavior (raw dealloc)
* Remove PyBoundMethod freelist to fix refcount/weakref test failures
Non-Option PyObjectRef fields retain references in freelist, causing
weakref and refcount assertions to fail in test_unittest, test_multiprocessing,
and test_socket.
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Raise SystemExit in check_signals() for non-main threads
once the finalizing flag is set. Without GIL, this serves
as the checkpoint where daemon threads detect shutdown,
analogous to CPython's _PyThreadState_MustExit in take_gil.
* Replace GC tracking HashSet with intrusive linked list
Replace per-generation HashSet<GcObjectPtr> with intrusive doubly-linked
lists for GC object tracking. Each PyInner now carries gc_pointers
(prev/next) and gc_generation fields, enabling O(1) track/untrack
without hashing.
- Add gc_pointers (Pointers<PyObject>) and gc_generation (u8) to PyInner
- Implement GcLink trait for intrusive list integration
- Replace generation_objects/permanent_objects/tracked_objects/finalized_objects
HashSets with generation_lists/permanent_list LinkedLists
- Use GcBits::FINALIZED flag instead of finalized_objects HashSet
- Change default_dealloc to untrack directly before memory free
- Hold both src/dst list locks in promote_survivors to prevent race
conditions with concurrent untrack_object calls
- Add pop_front to LinkedList for freeze/unfreeze operations
Move unreachable_refs creation before drop(gen_locks) so that raw
pointer dereferences and refcount increments happen while generation
list read locks are held. Previously, after dropping read locks, other
threads could untrack and free objects, causing use-after-free when
creating strong references from the raw GcPtr pointers.
* Remove PyMutex<FrameState> from Frame, use UnsafeCell fields directly
Move stack, cells_frees, prev_line out of the mutex-protected FrameState
into Frame as FrameUnsafeCell fields. This eliminates mutex lock/unlock
overhead on every frame execution (with_exec).
Safety relies on the same single-threaded execution guarantee that
FastLocals already uses.
* Add thread-local DataStack for bump-allocating frame data
Introduce DataStack with linked chunks (16KB initial, doubling) and
push/pop bump allocation. Add datastack field to VirtualMachine.
Not yet wired to frame creation.
* Unify FastLocals and BoxVec stack into LocalsPlus
Replace separate FastLocals (Box<[Option<PyObjectRef>]>) and
BoxVec<Option<PyStackRef>> with a single LocalsPlus struct that
stores both in a contiguous Box<[usize]> array. The first
nlocalsplus slots are fastlocals and the rest is the evaluation
stack. Typed access is provided through transmute-based methods.
Remove BoxVec import from frame.rs.
* Use DataStack for LocalsPlus in non-generator function calls
Normal function calls now bump-allocate LocalsPlus data from the
per-thread DataStack instead of a separate heap allocation.
Generator/coroutine frames continue using heap allocation since
they outlive the call.
On frame exit, data is copied to the heap (materialize_to_heap)
to preserve locals for tracebacks, then the DataStack is popped.
VirtualMachine.datastack is wrapped in UnsafeCell for interior
mutability (safe because frame allocation is single-threaded LIFO).
* Fix clippy: import Layout from core::alloc instead of alloc::alloc
* Fix vectorcall compatibility with LocalsPlus API
Update vectorcall dispatch functions to use localsplus stack
accessors instead of direct stack field access. Add
stack_truncate method to LocalsPlus. Update vectorcall fast
path in function.rs to use datastack and fastlocals_mut().
* Add datastack, nlocalsplus, ncells, tstate to cspell dictionary
* Fix DataStack pop() for non-monotonic allocation addresses
Check both bounds of the current chunk when determining if a
pop base is in the current chunk. The previous check (base >=
chunk_start) fails on Windows where newer chunks may be
allocated at lower addresses than older ones.
* Fix stale comments: release_datastack -> materialize_localsplus
* Fix non-threading mode for parallel test execution
Two fixes for Cell-based types used in static items under non-threading
mode, which cause data races when Rust test runner uses parallel threads:
1. LazyLock: use std::sync::LazyLock when std is available instead of
wrapping core::cell::LazyCell with a false `unsafe impl Sync`.
The LazyCell wrapper is kept only for no-std (truly single-threaded).
2. gc_state: use static_cell! (thread-local in non-threading mode)
instead of OnceLock, so each thread gets its own GcState with
Cell-based PyRwLock/PyMutex that are not accessed concurrently.
* Fix CallAllocAndEnterInit to use LocalsPlus stack API
* Use checked arithmetic in LocalsPlus and DataStack allocators
* Address code review: checked arithmetic, threading feature deps, Send gate
- Use checked arithmetic for nlocalsplus in Frame::new
- Add "std" to threading feature dependencies in rustpython-common
- Gate GcState Send impl with #[cfg(feature = "threading")]
* Clean up comments: remove redundant/stale remarks, fix CPython references
* Add object freelist infrastructure and PyFloat freelist
Add freelist_push/freelist_pop hooks to PyPayload trait with default
no-op implementations. Modify default_dealloc to offer dead objects
to the freelist before freeing memory. Modify PyRef::new_ref to try
popping from the freelist before allocating.
Implement thread-local freelist for PyFloat (max 100 entries).
Float objects are ideal candidates: fixed-size payload (f64),
no GC tracking, no instance dict, trivial Drop.
Benchmark improvement on float-heavy workloads:
float_arith: ~12% faster
float_list: ~17% faster
* Add PyList freelist and GC-safe freelist infrastructure
Add HAS_FREELIST const to PyPayload trait. For freelist types, GC
untracking is done immediately (not deferred) during dealloc to
prevent race conditions when the object is reused.
Restructure new_ref so freelist-popped objects also get GC tracked,
enabling freelist support for GC-tracked types like PyList.
Add drop_in_place for old payload before writing new one, required
for types with non-trivial Drop (e.g. PyList's RwLock<Vec>).
Implement thread-local freelist for PyList (max 80 entries).
* Add PyDict freelist and exact-type guard for freelist push
Add thread-local freelist for PyDict (max 80 entries).
Add heaptype check in default_dealloc to prevent subclass instances
from entering the freelist. Subclass objects have a different Python
type than the base type, so reusing them would return objects with
the wrong __class__.
Skip PyTuple freelist: structseq types (stat_result, struct_time)
are static subtypes sharing the same Rust payload, making type-safe
reuse impractical with heaptype check alone.
* Add PySlice freelist (max 1, matching PySlice_MAXFREELIST)
* Add FreeList<T> wrapper to drain cached objects on thread teardown
Replace Cell<Vec<*mut PyObject>> with Cell<FreeList<T>> which implements
Drop to properly deallocate PyInner<T> when threads exit.
Also fix freelist_pop safety doc to match actual contract.
* Add manual Traverse impl for PySlice with clear() and fix comment
* Deduplicate allocation fallback in PyRef::new_ref
* Reinit IO buffer locks after fork to prevent deadlocks
BufferedReader/Writer/TextIOWrapper use PyThreadMutex internally.
If a parent thread held one of these locks during fork(), the child
would deadlock on any IO operation.
Add reinit_after_fork() to RawThreadMutex and call it on sys.stdin/
stdout/stderr in the child process fork handler, analogous to
CPython's _PyIO_Reinit().
* Address review: unsafe fn + decoder lock reinit
- Mark reinit_std_streams_after_fork as unsafe fn to encode
fork-only precondition, update call site in posix.rs
- Reinit IncrementalNewlineDecoder's PyThreadMutex via
TextIOWrapper's decoder field to prevent child deadlocks
* Auto-format: cargo fmt --all
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* Add PyType vectorcall and use vectorcall in all specialized call fallbacks
* Add vectorcall slot for PyMethodDescriptor and PyWrapper
* Add FuncArgs::from_vectorcall_owned and simplify vectorcall fallback paths
Remove has_vectorcall checks from execute_call_vectorcall and
execute_call_kw_vectorcall. The invoke_vectorcall fallback now uses
from_vectorcall_owned to move args instead of cloning.
* Add CALL_ALLOC_AND_ENTER_INIT specialization
Optimizes user-defined class instantiation MyClass(args...)
when tp_new == object.__new__ and __init__ is a simple
PyFunction. Allocates the object directly and calls __init__
via invoke_exact_args, bypassing the generic type.__call__
dispatch path.
* Invalidate JIT cache when __code__ is reassigned
Change jitted_code from OnceCell to PyMutex<Option<CompiledCode>> so
it can be cleared on __code__ assignment. The setter now sets the
cached JIT code to None to prevent executing stale machine code.
* Atomic operations for specialization cache
- range iterator: deduplicate fast_next/next_fast
- Replace raw pointer reads/writes in CodeUnits with atomic
operations (AtomicU8/AtomicU16) for thread safety
- Add read_op (Acquire), read_arg (Relaxed), compare_exchange_op
- Use Release ordering in replace_op to synchronize cache writes
- Dispatch loop reads opcodes atomically via read_op/read_arg
- Fix adaptive counter access: use read/write_adaptive_counter
instead of read/write_cache_u16 (was reading wrong bytes)
- Add pre-check guards to all specialize_* functions to prevent
concurrent specialization races
- Move modified() before attribute changes in type.__setattr__
to prevent use-after-free of cached descriptors
- Use SeqCst ordering in modified() for version invalidation
- Add Release fence after quicken() initialization
* Fix slot wrapper override for inherited attributes
For __getattribute__: only use getattro_wrapper when the type
itself defines the attribute; otherwise inherit native slot from
base class via MRO.
For __setattr__/__delattr__: only store setattro_wrapper when
the type has its own __setattr__ or __delattr__; otherwise keep
the inherited base slot.
* Fix StoreAttrSlot cache overflow corrupting next instruction
write_cache_u32 at cache_base+3 writes 2 code units (positions 3 and 4),
but STORE_ATTR only has 4 cache entries (positions 0-3). This overwrites
the next instruction with the upper 16 bits of the slot offset.
Changed to write_cache_u16/read_cache_u16 since member descriptor offsets
fit within u16 (max 65535 bytes).
* Exclude method_descriptor from has_python_cmp check
has_python_cmp incorrectly treated method_descriptor as Python-level
comparison methods, causing richcompare slot to use wrapper dispatch
instead of inheriting the native slot.
* Fix CompareOpFloat NaN handling
partial_cmp returns None for NaN comparisons. is_some_and incorrectly
returned false for all NaN comparisons, but NaN != x should be true
per IEEE 754 semantics.
* Fix invoke_exact_args borrow in CallAllocAndEnterInit
* Distinguish Python method vs not-found in slot MRO lookup
Change lookup_slot_in_mro to return a 3-state SlotLookupResult
enum (NativeSlot/PythonMethod/NotFound) instead of Option<T>.
Previously, both "found a Python-level method" and "found nothing"
returned None, causing incorrect slot inheritance. For example,
class Test(Mixin, TestCase) would inherit object.slot_init from
Mixin via inherit_from_mro instead of using init_wrapper to
dispatch TestCase.__init__.
Apply this fix consistently to all slot update sites:
update_main_slot!, update_sub_slot!, TpGetattro, TpSetattro,
TpDescrSet, TpHash, TpRichcompare, SqAssItem, MpAssSubscript.
* Extract specialization helper functions to reduce boilerplate
- deoptimize() / deoptimize_at(): replace specialized op with base op
- adaptive(): decrement warmup counter or call specialize function
- commit_specialization(): replace op on success, backoff on failure
- execute_binary_op_int() / execute_binary_op_float(): typed binary ops
Removes 10 duplicate deoptimize_* functions, consolidates 13 adaptive
counter blocks, 6 binary op handlers, and 7 specialize tail patterns.
Also replaces inline deopt blocks in LoadAttr/StoreAttr handlers.
* Improve specialization guards and fix mark_stacks
- CONTAINS_OP_SET: add frozenset support in handler and specialize
- TO_BOOL_ALWAYS_TRUE: cache type version instead of checking slots
- LOAD_GLOBAL_BUILTIN: cache builtins dict version alongside globals
- mark_stacks: deoptimize specialized opcodes for correct reachability
* Auto-format: cargo fmt --all
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Add missing reinit_mutex_after_fork for vm.state.monitoring
PyMutex in reinit_locks_after_fork.
Use Mutex::raw() accessor in reinit_parking_lot_mutex instead
of pointer cast from struct start for layout safety.
* Remove unsafe ctx cast in make_class
Use Context::genesis() directly in make_class to obtain
&'static Context, eliminating the raw pointer cast.
* make_class -> make_static_type
* Add vectorcall (PEP 590) dispatch for function calls
Add VectorCallFunc slot to PyTypeSlots and vectorcall dispatch path
in the interpreter loop for Call and CallKw instructions.
Implement vectorcall for PyFunction (with fast path for simple
positional-only calls that fills fastlocals directly), PyBoundMethod
(avoids prepend_arg O(n) shift), and PyNativeFunction.
Add FuncArgs::from_vectorcall helper for fallback conversion.
Vectorcall slot is inherited with call slot and cleared when
__call__ is overridden in Python subclasses.
* Optimize vectorcall: move args instead of clone, use vectorcall in specialized paths
- invoke_exact_args takes Vec by value and uses drain() to move args
into fastlocals instead of cloning (eliminates refcount overhead)
- CallPyGeneral and CallBoundMethodGeneral now call vectorcall_function
directly instead of going through FuncArgs + prepend_arg + invoke
- CallKwPy and CallKwBoundMethod use vectorcall_function with kwnames
- vectorcall_bound_method uses insert(0) on existing Vec instead of
allocating a second Vec
* Auto-format: cargo fmt --all
* Fix vectorcall_native_function kwarg slice out-of-bounds
When needs_self was true and kwargs were present, pos_args only
contained positional args (self + original positionals) but
from_vectorcall expected kwarg values to follow in the slice.
Build the full args array (self + all original args including kwarg
values) before passing to from_vectorcall.
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
4096-entry direct-mapped cache keyed by (tp_version_tag, interned_name_ptr)
using lock-free SeqLock pattern for concurrent access.
- find_name_in_mro() checks cache before MRO walk
- has_name_in_mro() avoids cloning on cache hit
- Auto-assigns version tags on cache miss
- Invalidation via modified() (tp_version_tag=0) with value drops
- TYPE_CACHE_CLEARING flag suppresses re-population during GC drops
- Clear method cache during GC collection to break reference cycles
Jeong, YunWon