gh-159603: Use pthread_attr_get_np() on FreeBSD (#159604)
Use pthread_attr_get_np() on FreeBSD to get the stack size (and guard
size) of the current thread.
Check if pthread_attr_get_np() function and <pthread_np.h> header are
available in configure.
Update test_json.test_recursion. On FreeBSD 16 with a default stack size of
512 MB, the json C implementation no longer raises RecursionError in
test_highly_nested_objects_encoding(). Reduce also max_depth from 5,000 to
2,000 so test_highly_nested_objects_encoding() of the Python implementation
only takes 1.6 seconds, instead of 12 seconds.
(cherry picked from commit 71ba39dc05)
gh-159320: Enable stack size detection on Android (GH-159321)
Extend the use of pthread_getattr_np in stack size detection from any glibc linux to any glibc *or* android linux. Update tests for android's new recursion limit.
(cherry picked from commit 8ff41c2ae7)
Co-authored-by: T. Wouters <thomas@python.org>
Co-authored-by: Malcolm Smith <smith@chaquo.com>
Backport the shared lazy import resolver to Python 3.15, regenerate
interpreter files, and update the ABI snapshot.
Co-authored-by: Petr Viktorin <encukou@gmail.com>
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
gh-157757: Fix to make `lazy import a.b as c` import the module `a.b` (#158092)
* Import the module a lazy `import a.b as c` names
`import a.b as c` compiles to `IMPORT_NAME a.b` followed by `IMPORT_FROM b`.
Lazily, IMPORT_NAME leaves a placeholder holding "a.b", and IMPORT_FROM
rewrote it into the placeholder `lazy from a import b` produces. Reification
then imported `a` alone and read `b` off it, so the module `a.b` was never
imported under its own name: an attribute of the package shadowing it answered
instead, and `math.pi`, which no module backs, bound the float where the eager
statement raises ModuleNotFoundError.
Mark the dotted import on the placeholder and keep the whole name on it.
Reification imports that name and then walks its components with IMPORT_FROM,
which is what the eager statement does.
The test pinning `lazy import math.pi as pi` as working is inverted, since the
eager statement raises.
* Stop excluding test_trace from the lazy-imports-all run
It passes now that a lazy `import a.b as c` imports the module: the
KeyError on 'test.tracedmodules.testmod' came from the submodule never
being imported under its own name.
* Rename lz_submodule to lz_dotted_as and trim the comments
The flag means "bind the whole dotted name, not the root", which the old
name did not say, and import.c already has unrelated lazy_pending_submodules
machinery to be confused with.
* Chain lazy IMPORT_FROM placeholders instead of flagging dotted imports
Each deferred IMPORT_FROM off a placeholder without a fromlist now keeps
the previous placeholder in lz_from and the attribute name in lz_attr.
Reification walks back to the placeholder IMPORT_NAME left, runs that
import, and replays the lookups in order with _PyEval_ImportFrom, which
is what the eager bytecode does. This drops the lz_dotted_as flag and
also follows a custom __lazy_import__ that returns a placeholder for a
different module name.
* Chain every deferred IMPORT_FROM onto the previous placeholder
`lazy from a import b` now records its lookup the same way as `import
a.b as c`, so a placeholder holds either the module name and fromlist or
the previous placeholder and an attribute name, and reification has a
single path. The import passes only the name being resolved as the
fromlist, so accessing b still does not import the other names'
submodules.
* Treat an empty fromlist on a lazy import placeholder as no fromlist
__import__("a.b", fromlist=()) returns the top-level package `a`, the same
as fromlist=None, but the placeholder kept the empty tuple. Every consumer
of lz_attr then read it as a real fromlist: reification narrowed it to the
chained attribute and replayed the lookups on `a.b`, _PyEval_LazyImportFrom
took the attribute off sys.modules["a.b"], and the repr named `a.b.attr`.
_PyLazyImport_New already collapses None to NULL for exactly this reason,
so collapse an empty tuple there too and every site follows.
* gh-157757: Preserve empty fromlists for custom import hooks
---------
(cherry picked from commit 1e8ff18a12)
Co-authored-by: Brittany Reynoso <breynoso@meta.com>
gh-153205: Set `MemoryError` in stackref array conversion (GH-153206)
(cherry picked from commit 08759ff101)
Co-authored-by: An Long <aisk@users.noreply.github.com>
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Co-authored-by: sobolevn <mail@sobolevn.me>
gh-151546: Fix stack limits on musl (#151548)
If the thread stack size is set by linker flags, pass the stack size
to Python/ceval.c via the new _Py_LINKER_THREAD_STACK_SIZE variable
to set Py_C_STACK_SIZE macro.
(cherry picked from commit 9a61d1c0c8)
The function__entry and function__return probes stopped working in Python 3.11
when the interpreter was restructured around the new bytecode system. This change
restores these probes by adding DTRACE_FUNCTION_ENTRY() at the start_frame label
in bytecodes.c and DTRACE_FUNCTION_RETURN() in the RETURN_VALUE and YIELD_VALUE
instructions. The helper functions are defined in ceval.c and extract the
filename, function name, and line number from the frame before firing the probe.
This builds on the approach from https://github.com/python/cpython/pull/125019
but avoids modifying the JIT template since the JIT does not currently support
DTrace. The macros are conditionally compiled with WITH_DTRACE and are no-ops
otherwise. The tests have been updated to use modern opcode names (CALL, CALL_KW,
CALL_FUNCTION_EX) and a new bpftrace backend was added for Linux CI alongside
the existing SystemTap tests. Line probe tests were removed since that probe
was never restored after 3.11.
* Make the `PY_UNWIND` monitoring event available as a code-local
event to allow trapping on function exit events when an exception
bubbles up. This complements the PY_RETURN event by allowing to
catch any function exit event.
* Allow `PY_UNWIND` to be `DISABLE`d; disabling it disables the event for the whole code object.
* Do the above for `PY_THROW`, `RAISE`, `EXCEPTION_HANDLED`, and `RERAISE` events.
* Add FOR_ITER_VIRTUAL to specialize FOR_ITER for virtual iterators
* Add GET_ITER_SELF to specialize GET_ITER for iterators (including generators)
* Add GET_ITER_VIRTUAL to specialize GET_ITER for iterables as virtual iterators
* Add new (internal) _tp_iteritem function slot to PyTypeObject
* Put limited RESUME at start of genexpr for free-threading. Fix up exception handling in genexpr
Revert "GH-126910: Make `_Py_get_machine_stack_pointer` return the stack pointer (#147945)"
This reverts commit 255026d9ee,
which broke a tier-1 buildbot.
* Make _Py_get_machine_stack_pointer return the stack pointer (or close to it), not the frame pointer
* Make ``_Py_ReachedRecursionLimit`` inline again
* Remove ``_Py_MakeRecCheck`` relacing its use with ``_Py_ReachedRecursionLimit``
* Move stack swtiching check into ``_Py_CheckRecursiveCall``
Add special cases for classmethod and staticmethod descriptors in
_PyObject_GetMethodStackRef() to avoid calling tp_descr_get, which
avoids reference count contention on the bound method and underlying
callable. This improves scaling when calling classmethods and
staticmethods from multiple threads.
Also refactor method_vectorcall in classobject.c into a new _PyObject_VectorcallPrepend() helper so that it can be used by
PyObject_VectorcallMethod as well.
The positional arguments passed to _PyStack_UnpackDict are already
kept alive by the caller, so we can avoid the extra reference count
operations by using borrowed references instead of creating new ones.
This reduces reference count contention in the free-threaded build
when calling functions with keyword arguments. In particular, this
avoids contention on the type argument to `__new__` when instantiating
namedtuples with keyword arguments.
Add a FRAME_SUSPENDED_YIELD_FROM_LOCKED state that acts as a brief
lock, preventing other threads from transitioning the frame state
while gen_getyieldfrom reads the yield-from object off the stack.
Now that the specializing interpreter works with free threading,
replace ENABLE_SPECIALIZATION_FT with ENABLE_SPECIALIZATION and
replace requires_specialization_ft with requires_specialization.
Also limit the uniquely referenced check to FOR_ITER_RANGE. It's not
necessary for FOR_ITER_GEN and would cause test_for_iter_gen to fail.
Stan Ulbrych