* Add per-size tuple freelist (20 buckets × 2000 each)
Implement PyTuple freelist matching tuples[PyTuple_MAXSAVESIZE]:
- TupleFreeList with 20 per-size buckets (sizes 1..=20, 2000 capacity each)
- freelist_push uses pre-clear size hint for correct bucket selection
- freelist_pop takes &Self payload to select bucket by size
- Type guard in new_ref handles structseq types sharing PyTuple vtable
- Add pyinner_layout<T>() helper for custom freelist Drop impls
- Update freelist_pop/push signatures across all freelist types
* freelist: exact type check at pop call-site
Move exact-type filtering from freelist_pop implementations to the
single call-site in new_ref. This prevents structseq and other subtypes
from popping tuple freelist entries entirely, rather than popping and
then deallocating on type mismatch.
Add Context::try_genesis() that returns None during bootstrap to avoid
deadlock when genesis() is called during Context initialization.
* Move exact type check from freelist_push to call-site in default_dealloc
Remove typ parameter from freelist_push trait signature. The exact type
check is now done once at the call-site alongside the heaptype check,
simplifying all freelist_push implementations.
* Remove freelist_hint; call freelist_push before tp_clear
By calling freelist_push before tp_clear, the payload is still intact
and can be read directly (e.g. tuple element count for bucket selection).
This eliminates freelist_hint and the hint parameter entirely.
* Introduce TimeoutSeconds utility type for timeout parameters
Follow-up refactoring from #7237.
Python timeout parameters typically accept both float and int. Several
places in the codebase used Either<f64, i64> for this, each repeating
the same match-and-convert boilerplate. This extracts that into a
TimeoutSeconds type in vm::function::number.
Refactored sites:
- _sqlite3::ConnectArgs.timeout
- _thread::AcquireArgs.timeout
- _thread::ThreadHandle::join timeout
Either<f64, i64> in time.rs (6 sites) left unchanged: those are
timestamp values with per-branch logic (floor, range checks, etc).
Either<f64, isize> in select.rs also left unchanged (different type).
* Validate timeout in ThreadHandle::join to prevent panic
* refactor: move TimeoutSeconds from number to time module
- Remove weak_list from ObjExt, allocate WeakRefList as its own
prefix slot before PyInner
- Add MANAGED_WEAKREF flag (1 << 3) to PyTypeFlags
- Normalize MANAGED_WEAKREF from HAS_WEAKREF after flag assembly
- Use Layout::extend offsets in bootstrap allocator
The method cache (TYPE_CACHE) was storing strong references (Arc
clones) to cached attribute values, which inflated sys.getrefcount().
This caused intermittent test_memoryview failures where refcount
assertions would fail depending on GC collection timing.
Store borrowed raw pointers instead. Safety is guaranteed because:
- type_cache_clear() nullifies all entries during GC collection,
before the collector breaks cycles
- type_cache_clear_version() nullifies entries when a type is
modified, before the source dict entry is removed
- Readers use try_to_owned_from_ptr (safe_inc) to atomically
validate and increment the refcount on cache hit
* apply more allow_threads
* Simplify STW thread state transitions
- Fix park_detached_threads: successful CAS no longer sets
all_suspended=false, avoiding unnecessary polling rounds
- Replace park_timeout(50µs) with park() in wait_while_suspended
- Remove redundant self-suspension in attach_thread and detach_thread;
the STW controller handles DETACHED→SUSPENDED via park_detached_threads
- Add double-check under mutex before condvar wait to prevent lost wakes
- Remove dead stats_detach_wait_yields field and add_detach_wait_yields
* Representable for ThreadHandle
* Set ThreadHandle state to Running in parent thread after spawn
Like CPython's ThreadHandle_start, set RUNNING state in the parent
thread immediately after spawn() succeeds, rather than in the child.
This eliminates a race where join() could see Starting state if called
before the child thread executes.
Also reverts the macOS skip for test_start_new_thread_failed since the
root cause is fixed.
* Set ThreadHandle state to Running in parent thread after spawn
* Add debug_assert for thread state in start_the_world
* Unskip now-passing test_get_event_loop_thread and test_start_new_thread_at_finalization
* Wrap IO locks and file ops in allow_threads
Add lock_wrapped to ThreadMutex for detaching thread state
while waiting on contended locks. Use it for buffered and
text IO locks. Wrap FileIO read/write in allow_threads via
crt_fd to prevent STW hangs on blocking file operations.
* Use std::sync for thread start/ready events
Replace parking_lot Mutex/Condvar with std::sync (pthread-based)
for started_event and handle_ready_event. This prevents hangs
in forked children where parking_lot's global HASHTABLE may be
corrupted.
On Windows, SimpleQueue skips write locking because pipe
writes are assumed atomic. Without GIL, PipeConnection.
_send_bytes races on _send_ov when multiple threads call
send_bytes concurrently (e.g. _terminate_pool vs workers).
Wait for pending overlapped writes inside WriteFile before
returning to Python, so ERROR_IO_PENDING is never exposed
and the _send_ov assignment window is negligible.
* Suspend Python threads before fork()
Add stop-the-world thread suspension around fork() to prevent
deadlocks from locks held by dead parent threads in the child.
- Thread states: DETACHED / ATTACHED / SUSPENDED with atomic CAS
transitions matching _PyThreadState_{Attach,Detach,Suspend}
- stop_the_world / start_the_world: park all non-requester threads
before fork, resume after (parent) or reset (child)
- allow_threads (Py_BEGIN/END_ALLOW_THREADS): detach around blocking
syscalls (os.read/write, waitpid, Lock.acquire, time.sleep) so
stop_the_world can force-park via CAS
- Acquire/release import lock around fork lifecycle
- zero_reinit_after_fork: generic lock reset for parking_lot types
- gc_clear_raw: detach dict instead of clearing entries
- Lock-free double-check for descriptor cache reads (no read-side
seqlock); write-side seqlock retained for writer serialization
- fork() returns PyResult, checks PythonFinalizationError, calls
sys.audit
* Object header slimming: prefix allocation for ObjExt
Extract dict, weak_list, and slots fields from PyInner<T> into a
separate ObjExt struct allocated as a prefix before PyInner using
Layout::extend(). Objects that don't need these fields (int, str,
float, list, tuple, dict, etc.) skip the prefix entirely.
- Add HAS_WEAKREF flag to PyTypeFlags for per-type weakref control
- Add HAS_EXT bit to GcBits indicating prefix presence
- Define ObjExt struct with dict, weak_list, slots
- Shrink PyInner header from ~80-88 bytes to ~32 bytes for lightweight objects
- Update all accessor methods to go through ext_ref()
- Update bootstrap type hierarchy to use prefix allocation
- Add __weakref__ getset descriptor for heap types
- Set HAS_WEAKREF on builtin types that support weak references
- Remove test_weak_keyed_bad_delitem expectedFailure (now passes)
* Add HAS_WEAKREF to _asyncio Future/Task, rename weakref helpers
- Add HAS_WEAKREF flag to PyFuture and PyTask (matches CPython)
- Rename subtype_getweakref/setweakref to subtype_get_weakref/set_weakref
to fix cspell unknown word lint
* Add HAS_WEAKREF to array, deque, _grouper; remove expectedFailure markers
- Add HAS_WEAKREF to PyArray and PyDeque (matches CPython)
- Add HAS_WEAKREF to PyItertoolsGrouper (internal use by groupby)
- Remove 6 expectedFailure markers from test_dataclasses for weakref/slots
tests that now pass
* Add HAS_WEAKREF to code, union, partial, lock, IO, mmap, sre, sqlite3, typevar types
Add HAS_WEAKREF flag to built-in types that support weakref:
- PyCode, PyUnion, PyPartial, Lock, RLock
- All IO base/concrete classes (_IOBase, _RawIOBase, _BufferedIOBase,
_TextIOBase, BufferedReader, BufferedWriter, BufferedRandom,
BufferedRWPair, TextIOWrapper, StringIO, BytesIO, FileIO,
WindowsConsoleIO)
- PyMmap, sre Pattern, sqlite3 Connection/Cursor
- TypeVar, ParamSpec, ParamSpecArgs, ParamSpecKwargs, TypeVarTuple
Remove 3 expectedFailure markers from test_descr for now-passing tests.
* Add HAS_DICT to type flags and handle non-METHOD/CLASS in descr_get
- Add HAS_DICT flag to PyType (type metaclass) alongside existing
HAS_WEAKREF. All type objects are instances of type and need dict
support, matching CPython's PyType_Type.
- Replace unimplemented!() in PyMethodDescriptor::descr_get with
fallback to bind obj directly, matching CPython's method_get which
uses PyCFunction_NewEx for non-METH_METHOD methods.
* Fix ext detection, HeapMethodDef ownership, WASM error
- Remove HAS_EXT gc_bits flag; detect ext from type flags
using raw pointer reads to avoid Stacked Borrows violations
- Store HeapMethodDef owner in payload instead of dict hack
- Clear dict entries in gc_clear_raw to break cycles
- Add WASM error fallback when exception serialization fails
* Implement locale-aware 'n' format specifier for int, float, complex
Add LocaleInfo struct and locale-aware formatting methods to FormatSpec.
The 'n' format type now reads thousands_sep, decimal_point, and grouping
from C localeconv() and applies proper locale-based number grouping.
Remove @unittest.skip from test_format.test_locale.
* Fix complex 'n' format and remove locale expectedFailure markers
Rewrite format_complex_locale to reuse format_complex_re_im, matching
formatter_unicode.c: add_parens=0 and skip_re=0 for 'n' type.
Remove @expectedFailure from test_float__format__locale and
test_int__format__locale in test_types.py.
* Auto-format: cargo fmt --all
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* Remove cells_frees duplicate storage from Frame
Cell/free variable objects were stored in both a separate
`Box<[PyCellRef]>` (cells_frees field) and in the localsplus
fastlocals array. Remove the redundant cells_frees field and
access cell objects directly through localsplus, eliminating
one Box allocation and N clone operations per frame creation.
* Extract InterpreterFrame from Frame with Deref wrapper
Introduce InterpreterFrame struct containing all execution state
fields previously on Frame. Frame now wraps InterpreterFrame via
FrameUnsafeCell and implements Deref for transparent field access.
localsplus and prev_line are plain fields on InterpreterFrame
(no longer individually wrapped in FrameUnsafeCell) since the
entire InterpreterFrame is wrapped at the Frame level.
* Auto-format: cargo fmt --all
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* Fix thread-safety in GC, type cache, and instruction cache
GC / refcount:
- Add safe_inc() check for strong()==0 in RefCount
- Add try_to_owned() to PyObject for atomic refcount acquire
- Replace strong_count()+to_owned() with try_to_owned() in GC
collection and weakref callback paths to prevent TOCTOU races
Type cache:
- Add proper SeqLock (sequence counter) to TypeCacheEntry
- Readers spin-wait on odd sequence, validate after read
- Writers bracket updates with begin_write/end_write
- Use try_to_owned + pointer revalidation on read path
- Call modified() BEFORE attribute modification in set_attr
Instruction cache:
- Add pointer_cache (AtomicUsize array) to CodeUnits for
single atomic pointer load/store (prevents torn reads)
- Add try_read_cached_descriptor with try_to_owned + pointer
and version revalidation after increment
- Add write_cached_descriptor with version-bracketed writes
RLock:
- Fix release() to check is_owned_by_current_thread
- Add _release_save/_acquire_restore methods
* Fix RLock _acquire_restore tuple handling and unxfail threading test
* Align type cache seqlock writer protocol with CPython
* RLock: use single parking_lot level, track recursion manually
Instead of calling lock()/unlock() N times for recursion depth N,
keep parking_lot at 1 level and manage the count ourselves.
This makes acquire/release O(1) and matches CPython's
_PyRecursiveMutex approach (lock once + set level directly).
* Add try_to_owned_from_ptr to avoid &PyObject on stale ptrs
Use addr_of! to access ref_count directly from a raw pointer
without forming &PyObject first. Applied in type cache and
instruction cache hit paths where the pointer may be stale.
* Fix CI: spelling typo and xfail flaky test_thread_safety
- Fix "minimising" -> "minimizing" for cspell
- xfail test_thread_safety: dict iteration races with
concurrent GC mutations in _finalizer_registry
* Add freelists for PyComplex, PyInt, PyBoundMethod, PyRange
- PyComplex(100), PyInt(100), PyBoundMethod(20), PyRange(6)
- Use try_with instead of with in all freelist push/pop to prevent
panic on thread-local access during thread teardown
* Use alloc::dealloc in FreeList::Drop to avoid thread-local access panic
During thread teardown, Box::from_raw triggers cascading destructors
that may access already-destroyed thread-local storage (GC state, other
freelists). Use raw dealloc instead to free memory without running
destructors.
* Auto-format: cargo fmt --all
* Fix clippy: use core::alloc::Layout instead of alloc::alloc::Layout
* Address review: PyBoundMethod clear=false, update FreeList doc comment
- Set clear=false on PyBoundMethod (tp_clear=NULL in classobject.c)
- Update FreeList doc comment to match actual Drop behavior (raw dealloc)
* Remove PyBoundMethod freelist to fix refcount/weakref test failures
Non-Option PyObjectRef fields retain references in freelist, causing
weakref and refcount assertions to fail in test_unittest, test_multiprocessing,
and test_socket.
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Raise SystemExit in check_signals() for non-main threads
once the finalizing flag is set. Without GIL, this serves
as the checkpoint where daemon threads detect shutdown,
analogous to CPython's _PyThreadState_MustExit in take_gil.
* Replace GC tracking HashSet with intrusive linked list
Replace per-generation HashSet<GcObjectPtr> with intrusive doubly-linked
lists for GC object tracking. Each PyInner now carries gc_pointers
(prev/next) and gc_generation fields, enabling O(1) track/untrack
without hashing.
- Add gc_pointers (Pointers<PyObject>) and gc_generation (u8) to PyInner
- Implement GcLink trait for intrusive list integration
- Replace generation_objects/permanent_objects/tracked_objects/finalized_objects
HashSets with generation_lists/permanent_list LinkedLists
- Use GcBits::FINALIZED flag instead of finalized_objects HashSet
- Change default_dealloc to untrack directly before memory free
- Hold both src/dst list locks in promote_survivors to prevent race
conditions with concurrent untrack_object calls
- Add pop_front to LinkedList for freeze/unfreeze operations
Move unreachable_refs creation before drop(gen_locks) so that raw
pointer dereferences and refcount increments happen while generation
list read locks are held. Previously, after dropping read locks, other
threads could untrack and free objects, causing use-after-free when
creating strong references from the raw GcPtr pointers.
* Remove PyMutex<FrameState> from Frame, use UnsafeCell fields directly
Move stack, cells_frees, prev_line out of the mutex-protected FrameState
into Frame as FrameUnsafeCell fields. This eliminates mutex lock/unlock
overhead on every frame execution (with_exec).
Safety relies on the same single-threaded execution guarantee that
FastLocals already uses.
* Add thread-local DataStack for bump-allocating frame data
Introduce DataStack with linked chunks (16KB initial, doubling) and
push/pop bump allocation. Add datastack field to VirtualMachine.
Not yet wired to frame creation.
* Unify FastLocals and BoxVec stack into LocalsPlus
Replace separate FastLocals (Box<[Option<PyObjectRef>]>) and
BoxVec<Option<PyStackRef>> with a single LocalsPlus struct that
stores both in a contiguous Box<[usize]> array. The first
nlocalsplus slots are fastlocals and the rest is the evaluation
stack. Typed access is provided through transmute-based methods.
Remove BoxVec import from frame.rs.
* Use DataStack for LocalsPlus in non-generator function calls
Normal function calls now bump-allocate LocalsPlus data from the
per-thread DataStack instead of a separate heap allocation.
Generator/coroutine frames continue using heap allocation since
they outlive the call.
On frame exit, data is copied to the heap (materialize_to_heap)
to preserve locals for tracebacks, then the DataStack is popped.
VirtualMachine.datastack is wrapped in UnsafeCell for interior
mutability (safe because frame allocation is single-threaded LIFO).
* Fix clippy: import Layout from core::alloc instead of alloc::alloc
* Fix vectorcall compatibility with LocalsPlus API
Update vectorcall dispatch functions to use localsplus stack
accessors instead of direct stack field access. Add
stack_truncate method to LocalsPlus. Update vectorcall fast
path in function.rs to use datastack and fastlocals_mut().
* Add datastack, nlocalsplus, ncells, tstate to cspell dictionary
* Fix DataStack pop() for non-monotonic allocation addresses
Check both bounds of the current chunk when determining if a
pop base is in the current chunk. The previous check (base >=
chunk_start) fails on Windows where newer chunks may be
allocated at lower addresses than older ones.
* Fix stale comments: release_datastack -> materialize_localsplus
* Fix non-threading mode for parallel test execution
Two fixes for Cell-based types used in static items under non-threading
mode, which cause data races when Rust test runner uses parallel threads:
1. LazyLock: use std::sync::LazyLock when std is available instead of
wrapping core::cell::LazyCell with a false `unsafe impl Sync`.
The LazyCell wrapper is kept only for no-std (truly single-threaded).
2. gc_state: use static_cell! (thread-local in non-threading mode)
instead of OnceLock, so each thread gets its own GcState with
Cell-based PyRwLock/PyMutex that are not accessed concurrently.
* Fix CallAllocAndEnterInit to use LocalsPlus stack API
* Use checked arithmetic in LocalsPlus and DataStack allocators
* Address code review: checked arithmetic, threading feature deps, Send gate
- Use checked arithmetic for nlocalsplus in Frame::new
- Add "std" to threading feature dependencies in rustpython-common
- Gate GcState Send impl with #[cfg(feature = "threading")]
* Clean up comments: remove redundant/stale remarks, fix CPython references
* Add object freelist infrastructure and PyFloat freelist
Add freelist_push/freelist_pop hooks to PyPayload trait with default
no-op implementations. Modify default_dealloc to offer dead objects
to the freelist before freeing memory. Modify PyRef::new_ref to try
popping from the freelist before allocating.
Implement thread-local freelist for PyFloat (max 100 entries).
Float objects are ideal candidates: fixed-size payload (f64),
no GC tracking, no instance dict, trivial Drop.
Benchmark improvement on float-heavy workloads:
float_arith: ~12% faster
float_list: ~17% faster
* Add PyList freelist and GC-safe freelist infrastructure
Add HAS_FREELIST const to PyPayload trait. For freelist types, GC
untracking is done immediately (not deferred) during dealloc to
prevent race conditions when the object is reused.
Restructure new_ref so freelist-popped objects also get GC tracked,
enabling freelist support for GC-tracked types like PyList.
Add drop_in_place for old payload before writing new one, required
for types with non-trivial Drop (e.g. PyList's RwLock<Vec>).
Implement thread-local freelist for PyList (max 80 entries).
* Add PyDict freelist and exact-type guard for freelist push
Add thread-local freelist for PyDict (max 80 entries).
Add heaptype check in default_dealloc to prevent subclass instances
from entering the freelist. Subclass objects have a different Python
type than the base type, so reusing them would return objects with
the wrong __class__.
Skip PyTuple freelist: structseq types (stat_result, struct_time)
are static subtypes sharing the same Rust payload, making type-safe
reuse impractical with heaptype check alone.
* Add PySlice freelist (max 1, matching PySlice_MAXFREELIST)
* Add FreeList<T> wrapper to drain cached objects on thread teardown
Replace Cell<Vec<*mut PyObject>> with Cell<FreeList<T>> which implements
Drop to properly deallocate PyInner<T> when threads exit.
Also fix freelist_pop safety doc to match actual contract.
* Add manual Traverse impl for PySlice with clear() and fix comment
* Deduplicate allocation fallback in PyRef::new_ref
* Reinit IO buffer locks after fork to prevent deadlocks
BufferedReader/Writer/TextIOWrapper use PyThreadMutex internally.
If a parent thread held one of these locks during fork(), the child
would deadlock on any IO operation.
Add reinit_after_fork() to RawThreadMutex and call it on sys.stdin/
stdout/stderr in the child process fork handler, analogous to
CPython's _PyIO_Reinit().
* Address review: unsafe fn + decoder lock reinit
- Mark reinit_std_streams_after_fork as unsafe fn to encode
fork-only precondition, update call site in posix.rs
- Reinit IncrementalNewlineDecoder's PyThreadMutex via
TextIOWrapper's decoder field to prevent child deadlocks
* Auto-format: cargo fmt --all
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* Add PyType vectorcall and use vectorcall in all specialized call fallbacks
* Add vectorcall slot for PyMethodDescriptor and PyWrapper
* Add FuncArgs::from_vectorcall_owned and simplify vectorcall fallback paths
Remove has_vectorcall checks from execute_call_vectorcall and
execute_call_kw_vectorcall. The invoke_vectorcall fallback now uses
from_vectorcall_owned to move args instead of cloning.
Jeong, YunWon