- Unify `_Py_ext_module_loader_info_init*` functions. The “core” one was entirely unused, and the difference between “builtin” and “extension” can be expressed in arguments.
- Consolidate getting & checking the module name from spec. (This improves error message wording for `imp.create_builtin` with empty name.)
- Allow non-ASCII names.
Core modules (`sys` and `builtins`) still don't go through the routine.
Co-authored-by: Itamar Oren <itamarost@gmail.com>
Adds the ability to build Emscripten statically, so that it can be linked in as
an embedded library. Includes a test of the embedding case.
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Russell Keith-Magee <russell@keith-magee.com>
Document that single-phase init modules are added to sys.modules by
the call itself (unlike multi-phase init modules), and that the spec
name identifies the module, so a later call with the same name and a
different init function reuses the cached module without calling the
new init function. Add regression tests for both behaviors.
Co-authored-by: Petr Viktorin <encukou@gmail.com>
A single-phase init function that creates its module using only the
last component of a dotted name (as pybind11 does for submodules) got
a module whose __name__ was the short name. Set the package context
while calling the init function, as the dynamic loader does, so the
module gets the full name from the spec.
The marshal writer decided whether an object can be referenced later
(FLAG_REF) from its reference count, so the output for the same value
depended on what else in the process held a reference to its parts, and
differed between the default and the free-threaded builds' view of the
same objects. This is the remaining cause of gh-129724 and the cause of
gh-156504.
FLAG_REF is now set exactly on the objects which are written more than
once in the dump. The writer still makes one pass: it records the
position of the type byte of each object it remembers and of each
TYPE_REF, and when the dump is complete it sets the flags and renumbers
the references. A reference count of one is only used to skip the
hashtable. Interned strings are no longer flagged unconditionally.
The reader is unchanged and the format version is not bumped: both the
old and the new output are loaded by old and new readers.
PyMarshal_WriteObjectToFile() now builds the data in memory and writes
it with one fwrite(), since the flags are only known at the end.
Nothing is written if the object cannot be marshalled.
If a command line option wants to exit Python, move this option
processing from PyConfig_Read() to Py_Main(). So
Py_InitializeFromInitConfig() and Py_InitializeFromConfig() can no
longer return an exit code.
Deprecate PyInitConfig_GetExitCode(), PyStatus_Exit(), and PyStatus_IsExit()
functions, and PyStatus.exitcode member.
Renumber getopt.c long options to start at 1, instead of 0.
Keep 4 variables in the stable ABI:
* Py_FileSystemDefaultEncodeErrors
* Py_FileSystemDefaultEncoding
* Py_HasFileSystemDefaultEncoding
* Py_UTF8Mode
Mark the 4 variables as ABI only in Misc/stable_abi.toml.
Remove Py_SetProgramName(), Py_SetPythonHome(), PySys_SetArgv() and
PySys_SetArgvEx() functions, deprecated since Python 3.11.
Mark the 4 functions as abi_only in stable_abi.toml.
PyConfig_Read() and _PyPreConfig_Read() no longer read global config
variables (such as Py_BytesWarningFlag). Instead, PyConfig_Read() now
copies PyPreConfig members (isolated, use_environment and dev_mode).
_PyPreConfig_Read() still reads the last global configuration
variable: Py_UTF8Mode.
No longer use deprecated Py_IgnoreEnvironmentFlag in Py_GETENV().
Instead, use PyConfig.use_environment when Python is initialized.
Modify also PySys_SetArgv() to use PyConfig.isolated when Python is
initialized.
* Fix Py_RunMain() to return an exit code, rather than calling
Py_Exit(), when running a script, a command, or the REPL.
* _PyRun_SimpleFile() now logs errors to stderr, for example if setting
__main__.__file__ fails.
* Add tests on Py_RunMain() exitcode.
* Rename functions:
* _PyRun_SimpleStringFlagsWithName() => _PyRun_SimpleString()
* _PyRun_SimpleFileObject() => _PyRun_SimpleFile()
* _PyRun_AnyFileObject() => _PyRun_AnyFile()
* _PyRun_InteractiveLoopObject() => _PyRun_InteractiveLoop()
* Change _PyRun_SimpleString(), _PyRun_SimpleFile(), _PyRun_AnyFile() and
_PyRun_InteractiveLoop() return type to PyObject*.
* pymain_repl() now displays the error if PySys_Audit() or
import _pyrepl failed.
* PyRun_SimpleFileExFlags() and PyRun_AnyFileExFlags() now log
PyUnicode_DecodeFSDefault() error. So these functions can no longer
return -1 with an exception set.
* Add FOR_ITER_VIRTUAL to specialize FOR_ITER for virtual iterators
* Add GET_ITER_SELF to specialize GET_ITER for iterators (including generators)
* Add GET_ITER_VIRTUAL to specialize GET_ITER for iterables as virtual iterators
* Add new (internal) _tp_iteritem function slot to PyTypeObject
* Put limited RESUME at start of genexpr for free-threading. Fix up exception handling in genexpr
## Summary
- Move the `runtime->initialized = 1` store from before `site.py` import to the end of `init_interp_main()`, so `Py_IsInitialized()` only returns true after initialization has fully completed
- Access `initialized` and `core_initialized` through new inline accessors using acquire/release atomics, to also protect from data race undefined behavior
- `PySys_AddAuditHook()` now uses the accessor, so with the flag move it correctly skips audit hook invocation during all init phases (matching the documented "after runtime initialization" behavior) ... We could argue that running these earlier would be good even if the intent was never explicitly expressed, but that'd be its own issue.
## Motivation
`Py_IsInitialized()` returned 1 while `Py_InitializeEx()` was still running — specifically, before `site.py` had been imported. See https://github.com/PyO3/pyo3/issues/5900 where a second thread could acquire the GIL and start executing Python with an incomplete `sys.path` because `site.py` hadn't finished.
The flag was also a plain `int` with no atomic operations, making concurrent reads a C-standard data race, though unlikely to manifest.
## Regression test:
The added test properly fails on `main` with `ERROR: Py_IsInitialized() was true during site import`.
---
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The pymalloc huge page support had two problems. First, on
architectures where the default huge page size exceeds the arena
size (e.g. 32 MiB on PPC, 512 MiB on ARM64 with 64 KB base
pages), mmap with MAP_HUGETLB silently allocates a full huge page
even when the requested size is smaller. The subsequent munmap
with the original arena size then fails with EINVAL, permanently
leaking the entire huge page. Second, huge pages were always
attempted when compiled in, with no way to disable them at
runtime. On Linux, if the huge page pool is exhausted, page
faults including copy-on-write faults after fork deliver SIGBUS
and kill the process.
The arena allocator now queries the system huge page size from
/proc/meminfo and skips MAP_HUGETLB when the arena size is not a
multiple of it. Huge pages also now require explicit opt-in at
runtime via the PYTHON_PYMALLOC_HUGEPAGES environment variable,
which is read through PyConfig and respects -E and -I flags.
The config field pymalloc_hugepages is propagated to the runtime
allocators struct so the low-level arena allocator can check it
without calling getenv directly.
Many functions related to compiling or parsing Python code, such as
compile(), ast.parse(), symtable.symtable(),
and importlib.abc.InspectLoader.source_to_code() now allow to pass
the module name used when filtering syntax warnings.
* FOR_ITER now pushes either the iterator and NULL or leaves the iterable and pushes tagged zero
* NEXT_ITER uses the tagged int as the index into the sequence or, if TOS is NULL, iterates as before.
Switches over to a _Py_thread_local in place of autoTssKey, and also fixes a few other checks regarding PyGILState_Ensure after finalization.
Note that this doesn't fix concurrent use of PyGILState_Ensure with Py_Finalize; I'm pretty sure zapthreads doesn't work at all, and that needs to be fixed seperately.
Incidentally, this also fixed the warning not showing up if a subinterpreter wasn't
cleaned up via _interpreters.destroy. I had to update some of the tests as a result.
Optimize `LOAD_FAST` opcodes into faster versions that load borrowed references onto the operand stack when we can prove that the lifetime of the local outlives the lifetime of the temporary that is loaded onto the stack.
