In the free-threaded build, dict() and dict.update() checked that a
list key/value pair had two items and then read them with borrowed
references without holding its lock, so another thread could empty
the list in between and the items were read from a freed or NULL
array.
Heap types have no static type object so switch from `Py_Is` to an
assert that the shadowed `tp_new` / `tp_init` function is the expected
one to validate the vectorcall is only used with the right type.
Heap types can have zero-argument `__new__` and `__init__` so add
support for that producing the same error message via a new helper
`_PyArg_NoPositionalStack()`.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The classmethod and staticmethod metadata setters dropped their reference to the descriptor dictionary before inserting the new value. A key comparison or value finalizer can replace the dictionary during insertion, leaving the operation using released memory. Keep the dictionary alive until insertion finishes. Backport to 3.15 and 3.14.
On free-threaded builds, list.__sizeof__() reads an allocation header through an atomically loaded storage pointer. The list's owner can resize and free that storage between the load and the header read, producing invalid size data or an invalid memory read. Hold the list lock while inspecting the allocation. Backport to 3.15 and 3.14.
A type alias can be evaluated again before an earlier evaluation finishes, either recursively or in another thread. The later completion overwrites the cached value without releasing it, permanently leaking that value; synchronize cache access and release the replaced value outside the lock. Backport to 3.15 and 3.14.
Bound the shift count by MAX_LONG_DIGITS*PyLong_SHIFT instead of
PY_SSIZE_T_MAX*PyLong_SHIFT, so that the result size computed in
long_lshift1() cannot overflow Py_ssize_t.
* Update vendored mimalloc to v2.5.2
Upgrade mimalloc from v2.1.2 to v2.5.2 as-is from upstream.
v2.1.2: https://github.com/microsoft/mimalloc/releases/tag/v2.1.2
v2.5.2: https://github.com/microsoft/mimalloc/releases/tag/v2.5.2
* Re-apply CPython-specific changes to mimalloc v2.5.2
Most of the previous downstream changes were upstreamed in mimalloc
v2.1.3 - v2.1.7 (heap tags, _mi_heap_init/_mi_tld_init, MI_PRIM_THREAD_ID,
sub-processes, heap walk optimizations, platform fixes) and are dropped.
Remaining CPython-specific changes, all marked with a "CPython" comment:
- QSBR delayed page freeing for free-threaded builds (gh-115103,
gh-145615): mi_page_t/mi_heap_t fields and _PyMem_mi_* hooks in
page.c, heap.c and segment.c.
- debug_offset: preserve the first bytes of a PyObject (ob_tid, refcount
fields) when filling freed or uninitialized memory in debug builds
(gh-115491).
- Atomic mi_block_t.next loads/stores to avoid TSAN reports (gh-129748).
- Relax the owning thread assertion in _mi_free_delayed_block for
free-threaded builds, since the GC traverses heaps of other threads.
- Call _mi_heap_delayed_free_partial() in mi_heap_visit_blocks()
(gh-112532).
- Only count reclaimed pages up to MI_MEDIUM_OBJ_SIZE_MAX as a
"right page" in mi_segment_reclaim (gh-115103).
- Rename the weak _ZSt15get_new_handlerv symbol and disable the
mi_is_in_heap_region debug warning.
- Zero-initialize _mi_page_empty instead of a positional initializer.
* Update the free-threaded build to mimalloc v2.5.2 changes
mimalloc v2.1.4 replaced the list of abandoned segments with per-arena
bitmaps and introduced sub-processes (mi_subproc_t) to keep abandoned
segments of separate interpreters apart. Use them instead of the
CPython-specific mi_abandoned_pool_t:
- Each interpreter owns a mi_subproc_t created in init_interpreter().
- tstate_mimalloc_bind() sets tld->segments.subproc and uses the
upstream _mi_heap_init() signature.
- The GC and sys.getallocatedblocks() visit abandoned segments with
_PyMem_mi_visit_abandoned_blocks(), the upstream cursor API without
the mi_option_visit_abandoned requirement, while STW.
* Update NEWS.d
* fix MSAN
* fix TSAN
* fix test_cext
* Disable THP
* Address code review
* Address code review
* Address code review
* nit
* Address code review
* Address code review
* gh-154318: Guard C recursion when hashing nested tuples and frozendicts
tuple_hash() and frozendict_hash() fold each contained object's hash into
an accumulator with PyObject_Hash() inside an unguarded loop. For a deeply
nested tuple or frozendict this recurses back into the same function once
per level and overflows the C stack (SIGSEGV / STATUS_STACK_OVERFLOW)
instead of raising RecursionError.
Wrap both loops in _Py_EnterRecursiveCall() / _Py_LeaveRecursiveCall(), as
the sibling comparison and repr paths already do. The guard is placed after
the cached-hash early return, so repeated hashing of an already-hashed
tuple is unaffected.
This also covers types.GenericAlias, whose ga_hash() hashes its __args__
tuple and so crashed through tuple_hash().
When a generic alias contains a list of parameters and an object claiming to be an unpacked type-variable tuple returns a non-tuple from `__typing_subst__`, the error path frees the temporary argument tuple before reading it to format a `TypeError`. That use-after-free can crash the interpreter instead of reporting the invalid substitution.
_PyTuple_Resize() did not validate newsize for non-empty tuples.
Raise SystemError for a negative size and MemoryError for a too-large size,
like PyTuple_New() already does for the empty-tuple path.
Read insertion order from the iterator's captured values snapshot instead
of re-reading ma_values and asserting against a concurrently changing
size. Retry under the dictionary lock when a value slot has been cleared.
Make insertion-order writes atomic and publish size changes with release
stores, pairing with the iterator's size load. Keep order-byte accesses
relaxed. Add _Py_atomic_store_uint8_release() and
FT_ATOMIC_STORE_UINT8_RELEASE() for this.
gh-157361: Implement aiter() in Python instead of C
Move both forms of aiter() into the frozen _pybuiltins module and remove
the C async_callable_iterator types. The callable form now returns an
iterator whose __anext__() is a plain coroutine, so call-graph tools can
see through it into the awaited result of the callable.
Release the references still held by the original tuple when
reallocation fails.
Backport to 3.15 and 3.14.
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
delitem_common() unconditionally called lookdict_index() to locate the index slot in dk_indices that points at the entry being removed.
That slot is only used by the combined branch, to overwrite it with DKIX_DUMMY. In split tables the whole probe sequence was redundant work.
Move the call into the combined branch and drop the now-pointless assert() for split tables.
Microbenchmark (free-threaded build, -Og, no PGO; deleting keys from materialized instance dicts; medians over 12 alternating runs):
del d[k] 71.2 -> 69.2 ns/op (-2.8%)
d.pop(k) 98.9 -> 96.1 ns/op (-2.8%)
colliding hashes 78.0 -> 74.4 ns/op (-4.6%)
Setting __name__, __qualname__, __doc__ or __module__ on a function
stopped the world in the free-threaded build since gh-154826 and
gh-154851, making functools.wraps() orders of magnitude slower with
multiple threads.
Use the per-object critical section and _PyObject_XSetRefDelayed()
instead, as gen_set_name() already does. Use acquire loads in the
readers and hold strong references across error reporting, tracing, and
eval-frame recording that may execute Python or detach the thread. The
__code__, __defaults__, __kwdefaults__ and __closure__ setters still
stop the world to clear func_version.
Document synchronization requirements for direct field access and the
lifetime of borrowed function attributes.
_PyObject_LookupSpecial needs a PyErr_Occurred() check to handle
failures while during the lookup, or else it can fail with a "returned
a result with an exception set" error.
Add some missing ones in bytes and weakref/
mpage