AST nodes had an explicit `__dict__` slot, so every node built by
`ast.parse` allocated its own dict. With `Py_TPFLAGS_MANAGED_DICT`,
node attributes are stored as inline values with shared keys instead.
`ast.parse` is ~15.8% faster, walking trees is ~15.9% faster, and
the memory used by the AST trees is reduced by ~51.6%.
In the free-threaded build, compile() of a source string sets
suppress_co_const_immortalization so that its output is the same as in
the default build. The branch for AST objects returned before that, so
constants were replaced by the immortal ones of the interpreter's table
while a tuple of names (such as co_names) which was the same object was
not, and the marshal output depended on the code objects created before
in the process.
- Unify `_Py_ext_module_loader_info_init*` functions. The “core” one was entirely unused, and the difference between “builtin” and “extension” can be expressed in arguments.
- Consolidate getting & checking the module name from spec. (This improves error message wording for `imp.create_builtin` with empty name.)
- Allow non-ASCII names.
Core modules (`sys` and `builtins`) still don't go through the routine.
Co-authored-by: Itamar Oren <itamarost@gmail.com>
CPython copies the trampoline templates into executable memory. Keep the
ELF source in .rodata so BOLT preserves its bytes and boundaries.
Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>
Heap types have no static type object so switch from `Py_Is` to an
assert that the shadowed `tp_new` / `tp_init` function is the expected
one to validate the vectorcall is only used with the right type.
Heap types can have zero-argument `__new__` and `__init__` so add
support for that producing the same error message via a new helper
`_PyArg_NoPositionalStack()`.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
When Python is built with older Linux libc headers, timed semaphore waits use the real-time clock even if `sem_clockwait` is available at runtime. Use a weak reference on supported compilers to select `sem_clockwait` at runtime, falling back to `sem_timedwait` when the symbol is absent. ThreadSanitizer builds continue to use the real-time clock.
Addresses [astral-sh/python-build-standalone#1176](https://github.com/astral-sh/python-build-standalone/issues/1176).
Adds the ability to build Emscripten statically, so that it can be linked in as
an embedded library. Includes a test of the embedding case.
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Russell Keith-Magee <russell@keith-magee.com>
Resetting a ContextVar can finalize the replaced value, whose finalizer may set the variable again. Updating the lookup cache after releasing the old context root overwrites that newer cached value, so get() can return a stale value or use a freed object. Update the cache before the old root can be released. Backport to 3.15 and 3.14.
Concurrent calls to codecs.unregister() can shrink the search path between its length check and indexed lookup in a free-threaded build. The lookup then returns NULL and an unconditional decref crashes the interpreter; hold the registry mutex across the scan and removal. Also reported in gh-157078. Backport to 3.15 and 3.14.
A context watcher callback can clear a later watcher while a context-switch notification is being dispatched. The dispatcher still uses its earlier snapshot of active watchers, so it calls a NULL callback and crashes; check that each callback is still registered before invoking it. Backport to 3.15 and 3.14.
_CALL_LIST_APPEND stole the arg stackref via PyStackRef_AsPyObjectSteal
and passed it to _PyList_AppendTakeRef. When the list's backing array
failed to grow, _PyList_AppendTakeRef decreffed the item, but the stale
stackref remained on the value stack. The exception unwinder then closed
it a second time, causing a double-free / use-after-free.
Fix by giving _PyList_AppendTakeRef a separate reference via Py_NewRef
and closing the stackref explicitly on success. On the error path the
stackref still holds a valid reference, so the exception unwinder can
safely close it.
* Update vendored mimalloc to v2.5.2
Upgrade mimalloc from v2.1.2 to v2.5.2 as-is from upstream.
v2.1.2: https://github.com/microsoft/mimalloc/releases/tag/v2.1.2
v2.5.2: https://github.com/microsoft/mimalloc/releases/tag/v2.5.2
* Re-apply CPython-specific changes to mimalloc v2.5.2
Most of the previous downstream changes were upstreamed in mimalloc
v2.1.3 - v2.1.7 (heap tags, _mi_heap_init/_mi_tld_init, MI_PRIM_THREAD_ID,
sub-processes, heap walk optimizations, platform fixes) and are dropped.
Remaining CPython-specific changes, all marked with a "CPython" comment:
- QSBR delayed page freeing for free-threaded builds (gh-115103,
gh-145615): mi_page_t/mi_heap_t fields and _PyMem_mi_* hooks in
page.c, heap.c and segment.c.
- debug_offset: preserve the first bytes of a PyObject (ob_tid, refcount
fields) when filling freed or uninitialized memory in debug builds
(gh-115491).
- Atomic mi_block_t.next loads/stores to avoid TSAN reports (gh-129748).
- Relax the owning thread assertion in _mi_free_delayed_block for
free-threaded builds, since the GC traverses heaps of other threads.
- Call _mi_heap_delayed_free_partial() in mi_heap_visit_blocks()
(gh-112532).
- Only count reclaimed pages up to MI_MEDIUM_OBJ_SIZE_MAX as a
"right page" in mi_segment_reclaim (gh-115103).
- Rename the weak _ZSt15get_new_handlerv symbol and disable the
mi_is_in_heap_region debug warning.
- Zero-initialize _mi_page_empty instead of a positional initializer.
* Update the free-threaded build to mimalloc v2.5.2 changes
mimalloc v2.1.4 replaced the list of abandoned segments with per-arena
bitmaps and introduced sub-processes (mi_subproc_t) to keep abandoned
segments of separate interpreters apart. Use them instead of the
CPython-specific mi_abandoned_pool_t:
- Each interpreter owns a mi_subproc_t created in init_interpreter().
- tstate_mimalloc_bind() sets tld->segments.subproc and uses the
upstream _mi_heap_init() signature.
- The GC and sys.getallocatedblocks() visit abandoned segments with
_PyMem_mi_visit_abandoned_blocks(), the upstream cursor API without
the mi_option_visit_abandoned requirement, while STW.
* Update NEWS.d
* fix MSAN
* fix TSAN
* fix test_cext
* Disable THP
* Address code review
* Address code review
* Address code review
* nit
* Address code review
* Address code review
In free-threading builds gc.set_debug() stored gcstate->debug with a plain
write while gc.get_debug() and the collector read it without synchronisation,
which ThreadSanitizer reports as a data race. Access the flag with
FT_ATOMIC_STORE/LOAD_INT_RELAXED, consistent with how gc.enable()/disable() already handle gcstate->enabled.
* gh-159399: Fix crash in codecs.encode() on a concurrently unregistered codec
In the free-threaded build, codec_getitem() released the CodecInfo tuple
returned by _PyCodec_Lookup() before taking a reference to the encoder
or decoder it returns. If another thread cleared the lookup cache in
between with codecs.unregister(), the tuple could hold the last
reference to the item, which was then freed before it was increfed.
Take the reference before releasing the tuple, as
codec_getitem_checked() already does.
* Remove the free-threading stress test
Always call pthread_attr_init() in hardware_stack_limits(), not only
on FreeBSD (if pthread_attr_get_np() is available).
Emscripten: add pthread_attr_init() workaround.
gh-157361: Implement aiter() in Python instead of C
Move both forms of aiter() into the frozen _pybuiltins module and remove
the C async_callable_iterator types. The callable form now returns an
iterator whose __anext__() is a plain coroutine, so call-graph tools can
see through it into the awaited result of the callable.
Use pthread_attr_get_np() on FreeBSD to get the stack size (and guard
size) of the current thread.
Check if pthread_attr_get_np() function and <pthread_np.h> header are
available in configure.
Update test_json.test_recursion. On FreeBSD 16 with a default stack size of
512 MB, the json C implementation no longer raises RecursionError in
test_highly_nested_objects_encoding(). Reduce also max_depth from 5,000 to
2,000 so test_highly_nested_objects_encoding() of the Python implementation
only takes 1.6 seconds, instead of 12 seconds.
Setting __name__, __qualname__, __doc__ or __module__ on a function
stopped the world in the free-threaded build since gh-154826 and
gh-154851, making functools.wraps() orders of magnitude slower with
multiple threads.
Use the per-object critical section and _PyObject_XSetRefDelayed()
instead, as gen_set_name() already does. Use acquire loads in the
readers and hold strong references across error reporting, tracing, and
eval-frame recording that may execute Python or detach the thread. The
__code__, __defaults__, __kwdefaults__ and __closure__ setters still
stop the world to clear func_version.
Document synchronization requirements for direct field access and the
lifetime of borrowed function attributes.
Extend the use of pthread_getattr_np in stack size detection from any glibc linux to any glibc *or* android linux. Update tests for android's new recursion limit.
Co-authored-by: Malcolm Smith <smith@chaquo.com>
Starting off with WASI version, WASI SDK version, and whether cooperative multithreading is available. All values are required as WASI SDK is the officially supported compiler according to PEP 11.
A single-phase init function that creates its module using only the
last component of a dotted name (as pybind11 does for submodules) got
a module whose __name__ was the short name. Set the package context
while calling the init function, as the dynamic loader does, so the
module gets the full name from the spec.
Move source access and logical spans into the tokenizer source API, and keep input and encoding ownership in the reader. Export the incremental decoder helper for parser generator extensions.
The compiler merges equal constants of a module through its constant
cache. intern_constants() then ran on each code object and rebuilt a
frozenset (or, in the free-threaded build, a slice) when one of its
strings was replaced by an already interned one, so a constant shared by
several functions was copied for some of them depending on which strings
happened to be interned in the process. The class body's __qualname__
constant and co_qualname had the same problem. marshal then wrote
different bytes for the same source.
Intern the eligible strings of the final constants of each code unit
after optimization and before the code object is created. Tuples are
updated in place; a frozenset or slice is rebuilt once and stored in the
constant cache so that later units get the same object. The same code
is used in both builds.
Which strings are interned does not change, except that identifier-like
strings in slice constants are now interned in the default build too.
TestConstantsIdentity compares the identity graph and the marshal
output of code compiled in differently prepared processes, including
names tuples shared with constants.
The marshal writer decided whether an object can be referenced later
(FLAG_REF) from its reference count, so the output for the same value
depended on what else in the process held a reference to its parts, and
differed between the default and the free-threaded builds' view of the
same objects. This is the remaining cause of gh-129724 and the cause of
gh-156504.
FLAG_REF is now set exactly on the objects which are written more than
once in the dump. The writer still makes one pass: it records the
position of the type byte of each object it remembers and of each
TYPE_REF, and when the dump is complete it sets the flags and renumbers
the references. A reference count of one is only used to skip the
hashtable. Interned strings are no longer flagged unconditionally.
The reader is unchanged and the format version is not bumped: both the
old and the new output are loaded by old and new readers.
PyMarshal_WriteObjectToFile() now builds the data in memory and writes
it with one fwrite(), since the flags are only known at the end.
Nothing is written if the object cannot be marshalled.
Lysandros Nikolaou