Py_GetVersion() now allocates memory on the heap, instead of using a
static buffer, to no longer truncate the version if it's longer than
299 bytes.
Update Py_GetCompiler() and Py_GetBuildInfo() tests: they are now
always a part of sys.version.
* gh-158140: Resolve lazy siblings through the pending-child registry
Co-authored-by: Petr Viktorin <encukou@gmail.com>
Co-authored-by: T. Wouters <thomas@python.org>
Replace global variables with a new _PyOS_GetOpt structure:
* _PyOS_opterr => getopt->error
* _PyOS_optind => getopt->index
* _PyOS_optarg => getopt->arg
Changes:
* Add tests to test_cmd_line.
* Rename _PyOS_ResetGetOpt() to _PyOS_GetOpt_Init(). Add argc and
argv parameters.
* Don't expose longindex in the structure, it's only used internally.
* Move _PyOS_LongOption structure from the internal C API to
getopt.c.
* Expose two possible builtin JIT gaps
* Add trace and runtime guard for builtin dict
* Check _GUARD_BUILTINS_IS_CANONICAL once per frame
* Add check and fix for globals being folded
Formatting a float or complex with a precision within about 1000 of
INT_MAX could crash or produce incorrect output.
PyOS_double_to_string() now raises ValueError("precision too big") for
such precisions, as the format string parsers already do for
precisions above INT_MAX.
The limit applies regardless of presentation type or value, so a few
calls that previously succeeded (inf, nan, or 'g' with such a
precision) now raise as well.
* Import the module a lazy `import a.b as c` names
`import a.b as c` compiles to `IMPORT_NAME a.b` followed by `IMPORT_FROM b`.
Lazily, IMPORT_NAME leaves a placeholder holding "a.b", and IMPORT_FROM
rewrote it into the placeholder `lazy from a import b` produces. Reification
then imported `a` alone and read `b` off it, so the module `a.b` was never
imported under its own name: an attribute of the package shadowing it answered
instead, and `math.pi`, which no module backs, bound the float where the eager
statement raises ModuleNotFoundError.
Mark the dotted import on the placeholder and keep the whole name on it.
Reification imports that name and then walks its components with IMPORT_FROM,
which is what the eager statement does.
The test pinning `lazy import math.pi as pi` as working is inverted, since the
eager statement raises.
* Stop excluding test_trace from the lazy-imports-all run
It passes now that a lazy `import a.b as c` imports the module: the
KeyError on 'test.tracedmodules.testmod' came from the submodule never
being imported under its own name.
* Rename lz_submodule to lz_dotted_as and trim the comments
The flag means "bind the whole dotted name, not the root", which the old
name did not say, and import.c already has unrelated lazy_pending_submodules
machinery to be confused with.
* Chain lazy IMPORT_FROM placeholders instead of flagging dotted imports
Each deferred IMPORT_FROM off a placeholder without a fromlist now keeps
the previous placeholder in lz_from and the attribute name in lz_attr.
Reification walks back to the placeholder IMPORT_NAME left, runs that
import, and replays the lookups in order with _PyEval_ImportFrom, which
is what the eager bytecode does. This drops the lz_dotted_as flag and
also follows a custom __lazy_import__ that returns a placeholder for a
different module name.
* Chain every deferred IMPORT_FROM onto the previous placeholder
`lazy from a import b` now records its lookup the same way as `import
a.b as c`, so a placeholder holds either the module name and fromlist or
the previous placeholder and an attribute name, and reification has a
single path. The import passes only the name being resolved as the
fromlist, so accessing b still does not import the other names'
submodules.
* Treat an empty fromlist on a lazy import placeholder as no fromlist
__import__("a.b", fromlist=()) returns the top-level package `a`, the same
as fromlist=None, but the placeholder kept the empty tuple. Every consumer
of lz_attr then read it as a real fromlist: reification narrowed it to the
chained attribute and replayed the lookups on `a.b`, _PyEval_LazyImportFrom
took the attribute off sys.modules["a.b"], and the repr named `a.b.attr`.
_PyLazyImport_New already collapses None to NULL for exactly this reason,
so collapse an empty tuple there too and every site follows.
* gh-157757: Preserve empty fromlists for custom import hooks
---------
Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>
* gh-151518: Avoid STW starvation of attaching threads
Free-threaded stop-the-world pauses can otherwise starve a thread trying to reattach after it was suspended while detached. A tight manual gc.collect() loop can release and immediately request the next stop-the-world pause, repeatedly parking the detached thread before it can attach and make progress.
Add a distinct _Py_THREAD_SUSPENDED_DETACHED state for tstates parked from DETACHED. tstate_wait_attach() marks an attach waiter only after observing that detached-origin suspended state, and park_detached_threads() skips only those active waiters on later stop-the-world passes. The ordinary successful tstate_try_attach() path remains the baseline CAS-only path.
Teach the related stop-the-world paths about both suspended states, including start_the_world() and tstate_delete_common(). Keep the new wait flag after the existing hot free-threaded _PyThreadStateImpl fields so their offsets do not move.
Add a free-threaded GC regression test that runs a subprocess with a tight gc.collect() worker and verifies the main thread can reattach after sleeping and stop the worker.
* gh-151518: Track attach waiters in the thread state
Represent active attach waiters with suspended-waiting and
detached-waiting states. Preserve waiter registration when the
world resumes, and keep passive detached threads immediately parkable.
Restore the thread-state padding and retain the single-CAS
uncontended attach path.
* gh-151518: Simplify the GC fairness regression
Use explicit warmup imports and a joined non-daemon collector.
Run the child through script_helper with a faulthandler watchdog
so a stalled attachment still fails with a traceback.
Describe the bug as a fairness issue in the NEWS entry.
* gh-151518: Exercise STW fairness directly
* gh-151518: Preserve attach waiters when resuming BRC suspensions
Share the waiter-aware resume transition between stop-the-world pauses
and biased reference count merging so a concurrently registering waiter
keeps its opportunity to attach.
Restore the support import needed by the new upstream GC regression and
align the thread-state constants.
* gh-151518: Strengthen the STW attach fairness regression
Hold each test pause for 10 ms and repeat pauses in C to reduce the
opportunities for a waiting thread to run between pause requests.
Clarify that parking rechecks the thread state before sleeping.
* gh-151518: Restrict STW test pauses to free-threaded builds
STW is a no-op in GIL builds. Avoid waiting in the helper there,
since WASI cannot perform the blocking futex operation.
* gh-151518: Wake only registered attach waiters
Avoid a parking-lot lookup when resuming a suspended thread that
has not tried to reattach. Use the old state from the successful CAS
so concurrent waiter registration still receives a wakeup.
If a command line option wants to exit Python, move this option
processing from PyConfig_Read() to Py_Main(). So
Py_InitializeFromInitConfig() and Py_InitializeFromConfig() can no
longer return an exit code.
Deprecate PyInitConfig_GetExitCode(), PyStatus_Exit(), and PyStatus_IsExit()
functions, and PyStatus.exitcode member.
Renumber getopt.c long options to start at 1, instead of 0.
Keep 4 variables in the stable ABI:
* Py_FileSystemDefaultEncodeErrors
* Py_FileSystemDefaultEncoding
* Py_HasFileSystemDefaultEncoding
* Py_UTF8Mode
Mark the 4 variables as ABI only in Misc/stable_abi.toml.
Remove Py_SetProgramName(), Py_SetPythonHome(), PySys_SetArgv() and
PySys_SetArgvEx() functions, deprecated since Python 3.11.
Mark the 4 functions as abi_only in stable_abi.toml.
PyConfig_Read() and _PyPreConfig_Read() no longer read global config
variables (such as Py_BytesWarningFlag). Instead, PyConfig_Read() now
copies PyPreConfig members (isolated, use_environment and dev_mode).
_PyPreConfig_Read() still reads the last global configuration
variable: Py_UTF8Mode.
No longer use deprecated Py_IgnoreEnvironmentFlag in Py_GETENV().
Instead, use PyConfig.use_environment when Python is initialized.
Modify also PySys_SetArgv() to use PyConfig.isolated when Python is
initialized.
getpagesize() is not an ISO C or POSIX function. Darwin hides the prototype under -std=c11, so the _remote_debugging build fails with -Werror=implicit-function-declaration. That is the failure in the reported x86_64-apple-darwin → arm64-apple-darwin cross build.
Call sysconf(_SC_PAGESIZE) instead, which is already how resource.getpagesize() works. Fall back to 4096 if sysconf() fails.
--without-remote-debug is unchanged. That flag only disables the target-side attach protocol (Py_REMOTE_DEBUG); it does not exclude the _remote_debugging client module.
The value of a @setter is now a parameter, which can use a converter,
and a @getter can define a return converter. Declaring the value is
optional.
The accessors of an attribute are now collected in a Property, so the
entry of PyGetSetDef is identified by the Python name instead of the C
basename, and is composed without the help of the preprocessor unless
some accessor is compiled conditionally.
Previously the marshal writer recorded an error code and converted it into
an exception at the end, replacing the exception which was already raised
with ValueError("unmarshallable object"). Error messages now name the type
of the unsupported object and the required version.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
It was the ANSI code page instead of the encoding of the device the stream
is connected to, as in 3.7.
The stdio encoding is now left undefined in this mode and determined for
every standard stream.
Replace soft deprecated PyBytes_FromStringAndSize() with
PyBytesWriter.
Replace also PyBytes_FromStringAndSize(NULL, 0) with
Py_GetConstant(Py_CONSTANT_EMPTY_BYTES) in other functions.
Replace also PyBytes_FromStringAndSize(NULL, 0) with
PyBytes_FromStringAndSize("", 0) in _Py_GetConstant_Init().
Use free() instead of PyMem_RawFree() in pythread_wrapper() to avoid
a data race if another thread calls PyMem_SetAllocator() in parallel.
Reenable @nomemtest tests on TSAN.
Co-authored-by: Nathan Goldbaum <nathan.goldbaum@gmail.com>
In debug mode at Python exit, check if immutable singleton objects
have been modified by mistake to detect bugs in C extensions.
Add tests corrupting bytes, str, bool and int singleton objects.
Add pycore_global_objects_fini.h internal header file.
pycore_global_objects_fini_generated.h is now fully generated by the
script.
Co-authored-by: Petr Viktorin <encukou@gmail.com>
Make pegen and _tokenize consume tokenizer operations instead of accessing the tokenizer layout. Return token kind, span, location, and raw-string context together, so parser actions use the context of the cached token.
Represent persistent scanner positions as offsets and remove pointer save/restore bookkeeping when input storage grows.
Pengyu Lee