Files
+2 4ceb3faf2c [3.15] Restore deferred backports for 3.15.1, including GH-158874 (#158882)
* [3.15] gh-153364: Make frame, coroutine, and task-waiter chain walks iterative and bounded (GH-153365) (#158813)

gh-153364: Make frame, coroutine, and task-waiter chain walks iterative and bounded (GH-153365)

* let me declare single limit

* use our new limit in process_frame_chain()

* add it in parse_async_frame_chain()

* parse_coro_chain()

* NEWS

* async in the message?

* test

* no race

* process_task_awaited_by

* process_task_awaited_by limit test

* NEWS

* MAX_TASK_WAITER_CHAIN_DEPTH

* TASK_WAITER_CHAIN_DEPTH in test

* TASK_WAITER_CHAIN_DEPTH 256

* prevent the drift with the comment

* better naming, better style

* MAX_TASK_WAITER_CHAIN_DEPTH comment

* task-waiter iterative bfs walk

* iterative coro-walk

* nicer news

* 1 << 14

* comment

* unused read_Py_ssize_t

* fix tombstones

* simplify

* correct msg

* better test

* news for tombstones

* left-over from when testing buggy version

* redundant new line
(cherry picked from commit e0861c6ae7)

Co-authored-by: Maurycy Pawłowski-Wieroński <maurycy@maurycy.com>
(cherry picked from commit 9e401cf7f5)

* [3.15] gh-155811: Add a seqcount to `gc_stats` to prevent torn reads (GH-155828) (#158829)

* update_seq

* no need for XCHGL, MOVL is enough?

* gh-155811: Retry an inconsistent GC snapshot once

---------
(cherry picked from commit 5fecd448bb)

Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit 028124039c)

* [3.15] gh-151292: `_remote_debugging`: Do not corrupt the binary file when hitting `OverflowError` (GH-152892) (#158830)

gh-151292: `_remote_debugging`: Do not corrupt the binary file when hitting `OverflowError` (GH-152892)

* the kolektor

* test

* better test

* news

* =Q, move const to the base, not self.running

* gh-151292: Track binary writer finalization state

---------
(cherry picked from commit f839c061d3)

Co-authored-by: Maurycy Pawłowski-Wieroński <maurycy@maurycy.com>
Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>
(cherry picked from commit 8a7c23f316)

* [3.15] gh-158583: Fix uninitialized memory read in bytes.fromhex() (GH-158584) (#158691)

gh-158583: Fix uninitialized memory read in bytes.fromhex() (GH-158584)
(cherry picked from commit 9d22a5334b)

Co-authored-by: Victor Stinner <vstinner@python.org>
(cherry picked from commit 4f7af46e25)

* [3.15] gh-154194: Degrade frames in Tachyon instead of failing the sample (GH-154195) (#158831)

* gh-154194: Degrade frames in Tachyon instead of failing the sample (#154195)

* degrade gracefully

* news

* better NEWS wording

* do not raise on MAX_REMOTE_STR_READ

* bye MAX_REMOTE_STR_READ

* fix -m asyncio ps|pstree

* test truncation and linetable sentinel

* simpler

* simpler

* redundant now

* respect #157790 in the news

---------

Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>
(cherry picked from commit 7d25916b41)

* Preserve the stable ABI when creating fallback frame names

---------

Co-authored-by: Maurycy Pawłowski-Wieroński <maurycy@maurycy.com>
(cherry picked from commit c27f494061)

* [3.15] Add MSan to CI (GH-158625) (#158832)

Run a job that the test suite with MSan to the CI (#158625)

* Run the test suite with MSan in CI

* Additional fixes

* Add `_Py_MSAN_UNPOISON_STRING`

* Apply Victor's suggestions

* Apply Victor's suggestions

---------

(cherry picked from commit b93fb19a6e)

Co-authored-by: Stan Ulbrych <stan@python.org>
Co-authored-by: Victor Stinner <victor.stinner@gmail.com>
(cherry picked from commit 1bc78dea18)

* [3.15] gh-156810: Write the profiler's collapsed-stack export as UTF-8 (GH-156811) (#156814)

gh-156810: Write the profiler's collapsed-stack export as UTF-8 (GH-156811)
(cherry picked from commit c3706f4c68)

Co-authored-by: tonghuaroot (童话) <tonghuaroot@gmail.com>
(cherry picked from commit fce28da9b7)

* [3.15] gh-158552: Wait for Windows threads to suspend before blocking sampling (GH-158802) (#158845)

gh-158552: Wait for Windows threads to suspend before blocking sampling (GH-158802)

* gh-158552: Wait for Windows threads to suspend before blocking sampling

* Use a named Windows thread enumeration status constant
(cherry picked from commit 1643525f0f)

Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>
(cherry picked from commit 31288da21d)

* [3.15] gh-152721: Fix quadratic RLE replay time in the profiling binary reader (GH-152722) (#158850)

Backport of GH-152722.

Co-authored-by: tonghuaroot <tonghuaroot@gmail.com>
(cherry picked from commit ebaca2ac2d)

* [3.15] gh-156545: Fix flamegraph export RecursionError on deeply recursive programs (GH-156546) (#158851)

Backport of GH-156546.

Co-authored-by: tonghuaroot (童话) <tonghuaroot@gmail.com>
(cherry picked from commit 0ac7217040)

* [3.15] gh-158540: Add the profiled script's directory to sys.path (GH-158548) (#158844)

gh-158540: Add the profiled script's directory to sys.path (GH-158548)

* gh-158540: Add the profiled script's directory to sys.path

When a script is profiled with ``python -m profiling.sampling run
script.py`` from another directory, the script cannot import modules
placed next to it, because ``_sync_coordinator._execute_script()``
executes it with the working directory (added by
``_setup_environment()`` for the module case) as ``sys.path[0]``
instead of the script's own directory.

Make the script's directory importable in ``_execute_script()``,
matching the behavior of ``python script.py``.

Add a regression test that runs the coordinator on a script importing
a sibling module.

* Update _sync_coordinator.py Comment simplified

* gh-158540: Resolve symlinks when adding the script directory to sys.path

``python script.py`` resolves symlinks when computing ``sys.path[0]``, so
a script reached through a symlink (``link.py -> sub/where.py``) imports
modules from the real script's directory.  Apply ``os.path.realpath()``
before taking the directory name, and make sure the result is placed at
the front of ``sys.path`` even if it was already listed.

Add a regression test for a symlinked script.

---------
(cherry picked from commit 3f02aab9fa)

Co-authored-by: he_tao <53343436+hetaozdh@users.noreply.github.com>
Co-authored-by: Eduardo Villalpando Mello <eduardo.villalpando.mello@gmail.com>
(cherry picked from commit f52d831397)

* [3.15] gh-153838: Skip non-regular source files in the heatmap exporter (GH-153839) (#158853)

Backport of GH-153839.

Co-authored-by: tonghuaroot <tonghuaroot@gmail.com>
(cherry picked from commit 48998df251)

* [3.15] gh-158539: Fix exception mode missing handlers in generators/coroutines (GH-158581) (#158852)

* [3.15] gh-158539: Fix exception mode missing handlers in generators/coroutines (GH-158581)

Backport of GH-158581.

Co-authored-by: LucasZhou <donghao.zhou@outlook.com>

* [3.15] gh-158539: Use portable static assertion messages

* [3.15] gh-158539: Keep layout assertions with debug-offset validation

* [3.15] gh-158539: Use the platform guard for in-process inspection tests

---------

Co-authored-by: LucasZhou <donghao.zhou@outlook.com>
(cherry picked from commit d625ecbb8d)

* [3.15] gh-158522: Fix truncated stack for a task whose coroutine recurses (GH-158526) (#158870)

Co-authored-by: Timofei Ivankov <128279579+deadlovelll@users.noreply.github.com>
(cherry picked from commit a904b397ed)

* gh-156545, gh-158539: Fix deep flamegraph export on small C stacks and macOS runtime lookup (#158874)

(cherry picked from commit 114de198c0)

Include the C-stack test helper from main, introduced by
ce5ae29ef9, which the regression test
requires but 3.15 does not yet provide.

---------

Co-authored-by: Miss Islington (bot) <31488909+miss-islington@users.noreply.github.com>
Co-authored-by: Maurycy Pawłowski-Wieroński <maurycy@maurycy.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Victor Stinner <vstinner@python.org>
Co-authored-by: Stan Ulbrych <stan@python.org>
Co-authored-by: Victor Stinner <victor.stinner@gmail.com>
Co-authored-by: tonghuaroot (童话) <tonghuaroot@gmail.com>
Co-authored-by: he_tao <53343436+hetaozdh@users.noreply.github.com>
Co-authored-by: Eduardo Villalpando Mello <eduardo.villalpando.mello@gmail.com>
Co-authored-by: LucasZhou <donghao.zhou@outlook.com>
Co-authored-by: Timofei Ivankov <128279579+deadlovelll@users.noreply.github.com>
2026-10-11 15:42:59 +01:00

327 lines
11 KiB
C

/******************************************************************************
* Remote Debugging Module - Object Reading Functions
*
* This file contains functions for reading Python objects from remote
* process memory, including strings, bytes, and integers.
******************************************************************************/
#include "_remote_debugging.h"
#include <limits.h>
/* ============================================================================
* MEMORY READING FUNCTIONS
* ============================================================================ */
#define DEFINE_MEMORY_READER(type_name, c_type, error_msg) \
int \
read_##type_name(RemoteUnwinderObject *unwinder, uintptr_t address, c_type *result) \
{ \
int res = _Py_RemoteDebug_PagedReadRemoteMemory(&unwinder->handle, address, sizeof(c_type), result); \
if (res < 0) { \
set_exception_cause(unwinder, PyExc_RuntimeError, error_msg); \
return -1; \
} \
return 0; \
}
DEFINE_MEMORY_READER(ptr, uintptr_t, "Failed to read pointer from remote memory")
DEFINE_MEMORY_READER(char, char, "Failed to read char from remote memory")
int
read_py_ptr(RemoteUnwinderObject *unwinder, uintptr_t address, uintptr_t *ptr_addr)
{
if (read_ptr(unwinder, address, ptr_addr)) {
set_exception_cause(unwinder, PyExc_RuntimeError, "Failed to read Python pointer");
return -1;
}
*ptr_addr &= ~Py_TAG_BITS;
return 0;
}
/* ============================================================================
* PYTHON OBJECT READING FUNCTIONS
* ============================================================================ */
PyObject *
read_py_str(
RemoteUnwinderObject *unwinder,
uintptr_t address,
Py_ssize_t max_len
) {
// Read the entire PyUnicodeObject at once; for short strings the data
// is inline right after the header and we'll already have (some of) it.
char unicode_obj[SIZEOF_UNICODE_OBJ];
int res = _Py_RemoteDebug_PagedReadRemoteMemory(
&unwinder->handle,
address,
SIZEOF_UNICODE_OBJ,
unicode_obj
);
if (res < 0) {
set_exception_cause(unwinder, PyExc_RuntimeError, "Failed to read PyUnicodeObject");
return NULL;
}
Py_ssize_t len = GET_MEMBER(Py_ssize_t, unicode_obj, unwinder->debug_offsets.unicode_object.length);
if (len < 0) {
PyErr_Format(PyExc_RuntimeError,
"Invalid string length (%zd) at 0x%lx", len, address);
set_exception_cause(unwinder, PyExc_RuntimeError, "Invalid string length in remote Unicode object");
return NULL;
}
if (len > max_len) {
// gh-157788: a long name must not fail the whole read
len = max_len;
}
// Inspect state to pick the right data offset and character width.
// We rely on the remote process sharing this Python version's
// PyASCIIObject layout, the same assumption already used for `length`.
struct _PyUnicodeObject_state state = GET_MEMBER(
struct _PyUnicodeObject_state,
unicode_obj,
unwinder->debug_offsets.unicode_object.state);
if (!state.compact) {
PyErr_Format(PyExc_RuntimeError,
"Cannot read non-compact Unicode object at 0x%lx", address);
set_exception_cause(unwinder, PyExc_RuntimeError,
"Legacy (non-compact) Unicode objects are not supported");
return NULL;
}
int kind = (int)state.kind;
Py_UCS4 max_char;
switch (kind) {
case PyUnicode_1BYTE_KIND:
max_char = state.ascii ? 0x7F : 0xFF;
break;
case PyUnicode_2BYTE_KIND:
max_char = 0xFFFF;
break;
case PyUnicode_4BYTE_KIND:
max_char = 0x10FFFF;
break;
default:
PyErr_Format(PyExc_RuntimeError,
"Invalid Unicode kind %d at 0x%lx", kind, address);
set_exception_cause(unwinder, PyExc_RuntimeError,
"Invalid kind in remote Unicode object");
return NULL;
}
size_t header_size = state.ascii
? (size_t)unwinder->debug_offsets.unicode_object.asciiobject_size
: (size_t)unwinder->debug_offsets.unicode_object.compactunicodeobject_size;
// len * kind is bounded by max_len * 4 (kind <= 4, len <= max_len), so
// the multiplication can't overflow for any caller-sane max_len, but the
// explicit cap here keeps a corrupted remote `length` from later turning
// into a giant allocation.
size_t nbytes = (size_t)len * (size_t)kind;
if ((size_t)len > (SIZE_MAX / 4) || nbytes > (size_t)max_len * 4) {
PyErr_Format(PyExc_RuntimeError,
"Implausible Unicode byte size %zu at 0x%lx", nbytes, address);
set_exception_cause(unwinder, PyExc_RuntimeError,
"Garbage byte size in remote Unicode object");
return NULL;
}
PyObject *result = PyUnicode_New(len, max_char);
if (result == NULL) {
set_exception_cause(unwinder, PyExc_RuntimeError, "Failed to allocate PyUnicode for remote string");
return NULL;
}
if (nbytes == 0) {
return result;
}
void *data = PyUnicode_DATA(result);
// Reuse data already present in the header read; only round-trip for
// whatever spills past it.
size_t inline_avail = (header_size < SIZEOF_UNICODE_OBJ)
? SIZEOF_UNICODE_OBJ - header_size
: 0;
size_t inline_bytes = nbytes < inline_avail ? nbytes : inline_avail;
if (inline_bytes > 0) {
memcpy(data, unicode_obj + header_size, inline_bytes);
}
if (nbytes > inline_bytes) {
res = _Py_RemoteDebug_PagedReadRemoteMemory(
&unwinder->handle,
address + header_size + inline_bytes,
nbytes - inline_bytes,
(char *)data + inline_bytes);
if (res < 0) {
Py_DECREF(result);
set_exception_cause(unwinder, PyExc_RuntimeError, "Failed to read string data from remote memory");
return NULL;
}
}
return result;
}
PyObject *
read_py_bytes(
RemoteUnwinderObject *unwinder,
uintptr_t address,
Py_ssize_t max_len
) {
PyObject *result = NULL;
char *buf = NULL;
// Read the entire PyBytesObject at once
char bytes_obj[SIZEOF_BYTES_OBJ];
int res = _Py_RemoteDebug_PagedReadRemoteMemory(
&unwinder->handle,
address,
SIZEOF_BYTES_OBJ,
bytes_obj
);
if (res < 0) {
set_exception_cause(unwinder, PyExc_RuntimeError, "Failed to read PyBytesObject");
goto err;
}
Py_ssize_t len = GET_MEMBER(Py_ssize_t, bytes_obj, unwinder->debug_offsets.bytes_object.ob_size);
if (len < 0 || len > max_len) {
PyErr_Format(PyExc_RuntimeError,
"Invalid bytes length (%zd) at 0x%lx", len, address);
set_exception_cause(unwinder, PyExc_RuntimeError, "Invalid bytes length in remote bytes object");
return NULL;
}
buf = (char *)PyMem_RawMalloc(len+1);
if (buf == NULL) {
PyErr_NoMemory();
set_exception_cause(unwinder, PyExc_MemoryError, "Failed to allocate buffer for bytes reading");
return NULL;
}
size_t offset = (size_t)unwinder->debug_offsets.bytes_object.ob_sval;
res = _Py_RemoteDebug_PagedReadRemoteMemory(&unwinder->handle, address + offset, len, buf);
if (res < 0) {
set_exception_cause(unwinder, PyExc_RuntimeError, "Failed to read bytes data from remote memory");
goto err;
}
buf[len] = '\0';
result = PyBytes_FromStringAndSize(buf, len);
if (result == NULL) {
set_exception_cause(unwinder, PyExc_RuntimeError, "Failed to create PyBytes from remote bytes data");
goto err;
}
PyMem_RawFree(buf);
assert(result != NULL);
return result;
err:
if (buf != NULL) {
PyMem_RawFree(buf);
}
return NULL;
}
long
read_py_long(
RemoteUnwinderObject *unwinder,
uintptr_t address
)
{
unsigned int shift = PYLONG_BITS_IN_DIGIT;
// Read the entire PyLongObject at once
char long_obj[SIZEOF_LONG_OBJ];
int bytes_read = _Py_RemoteDebug_PagedReadRemoteMemory(
&unwinder->handle,
address,
(size_t)unwinder->debug_offsets.long_object.size,
long_obj);
if (bytes_read < 0) {
set_exception_cause(unwinder, PyExc_RuntimeError, "Failed to read PyLongObject");
return -1;
}
uintptr_t lv_tag = GET_MEMBER(uintptr_t, long_obj, unwinder->debug_offsets.long_object.lv_tag);
int negative = (lv_tag & 3) == 2;
Py_ssize_t size = lv_tag >> 3;
if (size == 0) {
return 0;
}
// Validate size: reject garbage (negative or unreasonably large)
if (size < 0 || size > MAX_LONG_DIGITS) {
PyErr_Format(PyExc_RuntimeError,
"Invalid PyLong digit count: %zd (expected 0-%d)", size, MAX_LONG_DIGITS);
set_exception_cause(unwinder, PyExc_RuntimeError,
"Invalid PyLong size (corrupted remote memory)");
return -1;
}
// Calculate how many digits fit inline in our local buffer
Py_ssize_t ob_digit_offset = unwinder->debug_offsets.long_object.ob_digit;
Py_ssize_t inline_digits_space = SIZEOF_LONG_OBJ - ob_digit_offset;
Py_ssize_t max_inline_digits = inline_digits_space / (Py_ssize_t)sizeof(digit);
digit *digits = (digit *)PyMem_RawMalloc(size * sizeof(digit));
if (!digits) {
PyErr_NoMemory();
set_exception_cause(unwinder, PyExc_MemoryError, "Failed to allocate digits for PyLong");
return -1;
}
if (size <= max_inline_digits && size <= _PY_NSMALLNEGINTS + _PY_NSMALLPOSINTS) {
memcpy(digits, long_obj + ob_digit_offset, size * sizeof(digit));
} else {
bytes_read = _Py_RemoteDebug_PagedReadRemoteMemory(
&unwinder->handle,
address + (uintptr_t)unwinder->debug_offsets.long_object.ob_digit,
sizeof(digit) * size,
digits
);
if (bytes_read < 0) {
set_exception_cause(unwinder, PyExc_RuntimeError, "Failed to read PyLong digits from remote memory");
goto error;
}
}
unsigned long limit = negative
? (unsigned long)LONG_MAX + 1UL
: (unsigned long)LONG_MAX;
unsigned long value = 0;
for (Py_ssize_t i = size; i-- > 0;) {
if (digits[i] >= PyLong_BASE) {
PyErr_Format(PyExc_RuntimeError,
"Invalid PyLong digit: %u (base %u)", digits[i], PyLong_BASE);
set_exception_cause(unwinder, PyExc_RuntimeError,
"Invalid PyLong digit (corrupted remote memory)");
goto error;
}
if (value > ((limit - (unsigned long)digits[i]) >> shift)) {
PyErr_SetString(PyExc_OverflowError,
"Remote PyLong value does not fit in C long");
set_exception_cause(unwinder, PyExc_OverflowError,
"Remote PyLong value is too large");
goto error;
}
value = (value << shift) | (unsigned long)digits[i];
}
PyMem_RawFree(digits);
if (negative) {
if (value == (unsigned long)LONG_MAX + 1UL) {
return LONG_MIN;
}
return -(long)value;
}
return (long)value;
error:
PyMem_RawFree(digits);
return -1;
}