A single-phase init function that creates its module using only the
last component of a dotted name (as pybind11 does for submodules) got
a module whose __name__ was the short name. Set the package context
while calling the init function, as the dynamic loader does, so the
module gets the full name from the spec.
Move source access and logical spans into the tokenizer source API, and keep input and encoding ownership in the reader. Export the incremental decoder helper for parser generator extensions.
The compiler merges equal constants of a module through its constant
cache. intern_constants() then ran on each code object and rebuilt a
frozenset (or, in the free-threaded build, a slice) when one of its
strings was replaced by an already interned one, so a constant shared by
several functions was copied for some of them depending on which strings
happened to be interned in the process. The class body's __qualname__
constant and co_qualname had the same problem. marshal then wrote
different bytes for the same source.
Intern the eligible strings of the final constants of each code unit
after optimization and before the code object is created. Tuples are
updated in place; a frozenset or slice is rebuilt once and stored in the
constant cache so that later units get the same object. The same code
is used in both builds.
Which strings are interned does not change, except that identifier-like
strings in slice constants are now interned in the default build too.
TestConstantsIdentity compares the identity graph and the marshal
output of code compiled in differently prepared processes, including
names tuples shared with constants.
The marshal writer decided whether an object can be referenced later
(FLAG_REF) from its reference count, so the output for the same value
depended on what else in the process held a reference to its parts, and
differed between the default and the free-threaded builds' view of the
same objects. This is the remaining cause of gh-129724 and the cause of
gh-156504.
FLAG_REF is now set exactly on the objects which are written more than
once in the dump. The writer still makes one pass: it records the
position of the type byte of each object it remembers and of each
TYPE_REF, and when the dump is complete it sets the flags and renumbers
the references. A reference count of one is only used to skip the
hashtable. Interned strings are no longer flagged unconditionally.
The reader is unchanged and the format version is not bumped: both the
old and the new output are loaded by old and new readers.
PyMarshal_WriteObjectToFile() now builds the data in memory and writes
it with one fwrite(), since the flags are only known at the end.
Nothing is written if the object cannot be marshalled.
Set elements are written in the order of their dumps. They were sorted
as (dump, element) pairs, so elements with the same dump were compared.
That raised TypeError for a set of several complex NaNs, which the
compiler can produce by constant folding, making the module impossible
to marshal.
Sort (dump, index, element) tuples, in which the indexes are all
different. Elements with the same dump stay in the order of the set,
as float NaNs did before.
Marshalling a set element can run Python code (__buffer__, an audit
hook) which changes the set. The number of elements was read before
the loop, so a set which grew was written past the end of the list and
one which shrank left NULL items in the list which was then sorted.
Raise RuntimeError("set changed size during iteration") instead.
This makes it so that in wasi builds if `-DPY_CALL_TRAMPOLINE` is passed, a call trampoline adaptor is used to prevent traps when there is a call signature mismatch between a C slot handler and the signature that the interpreter uses to call it.
This requires Clang >= 22. PEP 11 specifies that builds of Python 3.15 use WASI SDK version 33 which includes clang 23.1.0, so this could be used with Python 3.15 WASI builds but not earlier ones.
It requires a WebAssembly runtime that supports wasm-gc. Wasmtime has supported wasm-gc since version 27.0 released November 25, 2024. Some other runtimes still don't support it, but for those runtimes people can use builds with the trampoline disabled.
This also removed the Emscripten trampoline support for JS runtimes with no wasm-gc support. This removes a lot of implementation complexity and allows WASI and Emscripten to share code. The main concern is that it drops support for some very old iPhones, but every iPhone model released since 2018 is compatible with versions of Safari that support wasm-gc.
Tests were added that define handlers with various wrong numbers of arguments. The tests also run on non-webassembly platforms since they should work there as well.
Destructors can run after finalization has cleared the interpreter's
lazy_modules set and lazy_pending_submodules dict. Declaring or
resolving a lazy import from one of them passed NULL to PySet_Add(),
PySet_Discard() or the pending-submodules lookup.
Treat the bookkeeping as a no-op once the state is gone, as the other
readers of this state already do.
Add a new internal _Py_strerror() function to Python/fileutils.c. It
uses strerror_r() if available, or use strerror() otherwise.
Replace all strerror(code) calls with _Py_strerror(code).
Make os.strerror() thread-safe: use the reentrant strerror_r()
function if available.
* The configure script now checks if strerror_r() is supported.
* Add a stress test to test_free_threading.test_os (new module).
* Add a comment on decode_current_locale() assertion which fails if
the input string is mutated.
* Add an assertion to _Py_DecodeLocale() to detect if the input
string was mutated during the function call.
* Speed up PyObject_CallMethod via _PyObject_GetMethod
PyObject_CallMethod (and _PyObject_CallMethod, PyEval_CallMethod,
_PyObject_CallMethodId and the _SizeT variant) resolved the method with
PyObject_GetAttr, which builds a temporary bound-method object on every call,
then called it.
Resolve the method with _PyObject_GetMethod instead (the same lookup the
interpreter uses for obj.name(...)) and call it directly via
_PyObject_VectorcallPrepend, skipping the bound-method allocation. Behaviour
is unchanged: same attribute semantics, same "attribute of type ... is not
callable" error, and the historical
PyObject_CallMethod(o, m, "O", tuple) -> o.m(*tuple) unpacking.
The shared helper callmethod() and _PyObject_CallMethodFormat() are no longer
needed: their only caller (traceback.c) now uses PyObject_CallFunction, so both
are removed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Use _PyObject_GetMethodStackRef so method calls scale in free-threading
Resolve the method in callmethod_va() via _PyObject_GetMethodStackRef()
instead of _PyObject_GetMethod(). The StackRef variant returns the method
as a deferred reference, avoiding the per-call atomic refcount on the
shared method object that otherwise serializes threads in the
free-threaded build.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* rename to callmethod
* review comments
* review comments
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Kumar Aditya <kumaraditya@python.org>
Co-authored-by: Inada Naoki <songofacandy@gmail.com>
* gh-157407: Use _PyUnicode_AsUTF8NoNUL for AC 's' and 'z' converters
The code was doing the same as the helper added in GH-111957 but inline.
That resulted in significantly more `.text` for most functions. Refactor
the Arguemnt Clinic converters code to deduplicate a bit while keeping
generated code close to original.
On my Linux x64 box this reduces `.text` by ~6200 bytes in a clang
release build (no PGO or LTO). Performance is flat.
The helper was already exported for `_sqlite3`, update the comment that
it is used in AC generated code as well.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Update Include/internal/pycore_unicodeobject.h
Co-authored-by: Pieter Eendebak <pieter.eendebak@gmail.com>
* Remove NEWS, add period
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Pieter Eendebak <pieter.eendebak@gmail.com>
PyBytesWriter_Finish() doesn't need to check the size. It's known to
be valid.
PyBytesWriter_Create() now only calls byteswriter_resize() if the
size is larger than the small buffer.
Remove byteswriter_data() function: call static inline
_PyBytesWriter_GetData() function directly.
* update_seq
* no need for XCHGL, MOVL is enough?
* gh-155811: Retry an inconsistent GC snapshot once
---------
Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>
* Run the test suite with MSan in CI
* Additional fixes
* Add `_Py_MSAN_UNPOISON_STRING`
* Apply Victor's suggestions
Co-authored-by: Victor Stinner <victor.stinner@gmail.com>
* Apply Victor's suggestions
Co-authored-by: Victor Stinner <victor.stinner@gmail.com>
---------
Co-authored-by: Victor Stinner <victor.stinner@gmail.com>
Add output_length to _Py_EncodeLocale(), _Py_EncodeUTF8(),
encode_current_locale() and encode_ascii(). So unicode_encode_locale() and
wstr_to_utf8() can now use the output_length, instead of having to compute
strlen().
* Rename functions:
* _Py_DecodeUTF8Ex() => _Py_DecodeUTF8()
* _Py_EncodeUTF8Ex() => _Py_EncodeUTF8()
* _Py_EncodeLocaleEx() => _Py_EncodeLocale()
* _Py_DecodeLocaleEx() => _Py_DecodeLocale()
* Add tests on Py_DecodeLocale() and Py_EncodeLocale() in test_codecs.
Clarify also their documentation.
* _Py_EncodeLocale() and _Py_DecodeLocale() now sets all output parameters on
error. Add more tests on these output parameters.
* Add encode_current_locale_impl() to simplify encode_current_locale().
* Remove reason parameter of _Py_EncodeLocale() and _Py_DecodeLocale() to
simplify the code. Only the UTF-8 decoder used specific reasons.
* _Py_EncodeUTF8() now uses size_t instead of Py_ssize_t to iterate
on the input string.
* On error, encode_current_locale_impl() now returns ENCODE_ERROR (new
constant) instead of DECODE_ERROR.
* Mark _Py_EncodeLocaleRaw() as static (it's not used outside fileutils.c).
Py_GetVersion() now allocates memory on the heap, instead of using a
static buffer, to no longer truncate the version if it's longer than
299 bytes.
Update Py_GetCompiler() and Py_GetBuildInfo() tests: they are now
always a part of sys.version.
* gh-158140: Resolve lazy siblings through the pending-child registry
Co-authored-by: Petr Viktorin <encukou@gmail.com>
Co-authored-by: T. Wouters <thomas@python.org>
Replace global variables with a new _PyOS_GetOpt structure:
* _PyOS_opterr => getopt->error
* _PyOS_optind => getopt->index
* _PyOS_optarg => getopt->arg
Changes:
* Add tests to test_cmd_line.
* Rename _PyOS_ResetGetOpt() to _PyOS_GetOpt_Init(). Add argc and
argv parameters.
* Don't expose longindex in the structure, it's only used internally.
* Move _PyOS_LongOption structure from the internal C API to
getopt.c.
* Expose two possible builtin JIT gaps
* Add trace and runtime guard for builtin dict
* Check _GUARD_BUILTINS_IS_CANONICAL once per frame
* Add check and fix for globals being folded
Formatting a float or complex with a precision within about 1000 of
INT_MAX could crash or produce incorrect output.
PyOS_double_to_string() now raises ValueError("precision too big") for
such precisions, as the format string parsers already do for
precisions above INT_MAX.
The limit applies regardless of presentation type or value, so a few
calls that previously succeeded (inf, nan, or 'g' with such a
precision) now raise as well.
* Import the module a lazy `import a.b as c` names
`import a.b as c` compiles to `IMPORT_NAME a.b` followed by `IMPORT_FROM b`.
Lazily, IMPORT_NAME leaves a placeholder holding "a.b", and IMPORT_FROM
rewrote it into the placeholder `lazy from a import b` produces. Reification
then imported `a` alone and read `b` off it, so the module `a.b` was never
imported under its own name: an attribute of the package shadowing it answered
instead, and `math.pi`, which no module backs, bound the float where the eager
statement raises ModuleNotFoundError.
Mark the dotted import on the placeholder and keep the whole name on it.
Reification imports that name and then walks its components with IMPORT_FROM,
which is what the eager statement does.
The test pinning `lazy import math.pi as pi` as working is inverted, since the
eager statement raises.
* Stop excluding test_trace from the lazy-imports-all run
It passes now that a lazy `import a.b as c` imports the module: the
KeyError on 'test.tracedmodules.testmod' came from the submodule never
being imported under its own name.
* Rename lz_submodule to lz_dotted_as and trim the comments
The flag means "bind the whole dotted name, not the root", which the old
name did not say, and import.c already has unrelated lazy_pending_submodules
machinery to be confused with.
* Chain lazy IMPORT_FROM placeholders instead of flagging dotted imports
Each deferred IMPORT_FROM off a placeholder without a fromlist now keeps
the previous placeholder in lz_from and the attribute name in lz_attr.
Reification walks back to the placeholder IMPORT_NAME left, runs that
import, and replays the lookups in order with _PyEval_ImportFrom, which
is what the eager bytecode does. This drops the lz_dotted_as flag and
also follows a custom __lazy_import__ that returns a placeholder for a
different module name.
* Chain every deferred IMPORT_FROM onto the previous placeholder
`lazy from a import b` now records its lookup the same way as `import
a.b as c`, so a placeholder holds either the module name and fromlist or
the previous placeholder and an attribute name, and reification has a
single path. The import passes only the name being resolved as the
fromlist, so accessing b still does not import the other names'
submodules.
* Treat an empty fromlist on a lazy import placeholder as no fromlist
__import__("a.b", fromlist=()) returns the top-level package `a`, the same
as fromlist=None, but the placeholder kept the empty tuple. Every consumer
of lz_attr then read it as a real fromlist: reification narrowed it to the
chained attribute and replayed the lookups on `a.b`, _PyEval_LazyImportFrom
took the attribute off sys.modules["a.b"], and the repr named `a.b.attr`.
_PyLazyImport_New already collapses None to NULL for exactly this reason,
so collapse an empty tuple there too and every site follows.
* gh-157757: Preserve empty fromlists for custom import hooks
---------
Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>
* gh-151518: Avoid STW starvation of attaching threads
Free-threaded stop-the-world pauses can otherwise starve a thread trying to reattach after it was suspended while detached. A tight manual gc.collect() loop can release and immediately request the next stop-the-world pause, repeatedly parking the detached thread before it can attach and make progress.
Add a distinct _Py_THREAD_SUSPENDED_DETACHED state for tstates parked from DETACHED. tstate_wait_attach() marks an attach waiter only after observing that detached-origin suspended state, and park_detached_threads() skips only those active waiters on later stop-the-world passes. The ordinary successful tstate_try_attach() path remains the baseline CAS-only path.
Teach the related stop-the-world paths about both suspended states, including start_the_world() and tstate_delete_common(). Keep the new wait flag after the existing hot free-threaded _PyThreadStateImpl fields so their offsets do not move.
Add a free-threaded GC regression test that runs a subprocess with a tight gc.collect() worker and verifies the main thread can reattach after sleeping and stop the worker.
* gh-151518: Track attach waiters in the thread state
Represent active attach waiters with suspended-waiting and
detached-waiting states. Preserve waiter registration when the
world resumes, and keep passive detached threads immediately parkable.
Restore the thread-state padding and retain the single-CAS
uncontended attach path.
* gh-151518: Simplify the GC fairness regression
Use explicit warmup imports and a joined non-daemon collector.
Run the child through script_helper with a faulthandler watchdog
so a stalled attachment still fails with a traceback.
Describe the bug as a fairness issue in the NEWS entry.
* gh-151518: Exercise STW fairness directly
* gh-151518: Preserve attach waiters when resuming BRC suspensions
Share the waiter-aware resume transition between stop-the-world pauses
and biased reference count merging so a concurrently registering waiter
keeps its opportunity to attach.
Restore the support import needed by the new upstream GC regression and
align the thread-state constants.
* gh-151518: Strengthen the STW attach fairness regression
Hold each test pause for 10 ms and repeat pauses in C to reduce the
opportunities for a waiting thread to run between pause requests.
Clarify that parking rechecks the thread state before sleeping.
* gh-151518: Restrict STW test pauses to free-threaded builds
STW is a no-op in GIL builds. Avoid waiting in the helper there,
since WASI cannot perform the blocking futex operation.
* gh-151518: Wake only registered attach waiters
Avoid a parking-lot lookup when resuming a suspended thread that
has not tried to reattach. Use the old state from the successful CAS
so concurrent waiter registration still receives a wakeup.
If a command line option wants to exit Python, move this option
processing from PyConfig_Read() to Py_Main(). So
Py_InitializeFromInitConfig() and Py_InitializeFromConfig() can no
longer return an exit code.
Deprecate PyInitConfig_GetExitCode(), PyStatus_Exit(), and PyStatus_IsExit()
functions, and PyStatus.exitcode member.
Renumber getopt.c long options to start at 1, instead of 0.
Keep 4 variables in the stable ABI:
* Py_FileSystemDefaultEncodeErrors
* Py_FileSystemDefaultEncoding
* Py_HasFileSystemDefaultEncoding
* Py_UTF8Mode
Mark the 4 variables as ABI only in Misc/stable_abi.toml.
Remove Py_SetProgramName(), Py_SetPythonHome(), PySys_SetArgv() and
PySys_SetArgvEx() functions, deprecated since Python 3.11.
Mark the 4 functions as abi_only in stable_abi.toml.
Itamar Oren