mirror of
https://github.com/python/cpython.git
synced 2026-10-29 02:45:15 +08:00
+2








4ceb3faf2c
* [3.15] gh-153364: Make frame, coroutine, and task-waiter chain walks iterative and bounded (GH-153365) (#158813) gh-153364: Make frame, coroutine, and task-waiter chain walks iterative and bounded (GH-153365) * let me declare single limit * use our new limit in process_frame_chain() * add it in parse_async_frame_chain() * parse_coro_chain() * NEWS * async in the message? * test * no race * process_task_awaited_by * process_task_awaited_by limit test * NEWS * MAX_TASK_WAITER_CHAIN_DEPTH * TASK_WAITER_CHAIN_DEPTH in test * TASK_WAITER_CHAIN_DEPTH 256 * prevent the drift with the comment * better naming, better style * MAX_TASK_WAITER_CHAIN_DEPTH comment * task-waiter iterative bfs walk * iterative coro-walk * nicer news * 1 << 14 * comment * unused read_Py_ssize_t * fix tombstones * simplify * correct msg * better test * news for tombstones * left-over from when testing buggy version * redundant new line (cherry picked from commite0861c6ae7) Co-authored-by: Maurycy Pawłowski-Wieroński <maurycy@maurycy.com> (cherry picked from commit9e401cf7f5) * [3.15] gh-155811: Add a seqcount to `gc_stats` to prevent torn reads (GH-155828) (#158829) * update_seq * no need for XCHGL, MOVL is enough? * gh-155811: Retry an inconsistent GC snapshot once --------- (cherry picked from commit5fecd448bb) Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> (cherry picked from commit028124039c) * [3.15] gh-151292: `_remote_debugging`: Do not corrupt the binary file when hitting `OverflowError` (GH-152892) (#158830) gh-151292: `_remote_debugging`: Do not corrupt the binary file when hitting `OverflowError` (GH-152892) * the kolektor * test * better test * news * =Q, move const to the base, not self.running * gh-151292: Track binary writer finalization state --------- (cherry picked from commitf839c061d3) Co-authored-by: Maurycy Pawłowski-Wieroński <maurycy@maurycy.com> Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com> (cherry picked from commit8a7c23f316) * [3.15] gh-158583: Fix uninitialized memory read in bytes.fromhex() (GH-158584) (#158691) gh-158583: Fix uninitialized memory read in bytes.fromhex() (GH-158584) (cherry picked from commit9d22a5334b) Co-authored-by: Victor Stinner <vstinner@python.org> (cherry picked from commit4f7af46e25) * [3.15] gh-154194: Degrade frames in Tachyon instead of failing the sample (GH-154195) (#158831) * gh-154194: Degrade frames in Tachyon instead of failing the sample (#154195) * degrade gracefully * news * better NEWS wording * do not raise on MAX_REMOTE_STR_READ * bye MAX_REMOTE_STR_READ * fix -m asyncio ps|pstree * test truncation and linetable sentinel * simpler * simpler * redundant now * respect #157790 in the news --------- Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com> (cherry picked from commit7d25916b41) * Preserve the stable ABI when creating fallback frame names --------- Co-authored-by: Maurycy Pawłowski-Wieroński <maurycy@maurycy.com> (cherry picked from commitc27f494061) * [3.15] Add MSan to CI (GH-158625) (#158832) Run a job that the test suite with MSan to the CI (#158625) * Run the test suite with MSan in CI * Additional fixes * Add `_Py_MSAN_UNPOISON_STRING` * Apply Victor's suggestions * Apply Victor's suggestions --------- (cherry picked from commitb93fb19a6e) Co-authored-by: Stan Ulbrych <stan@python.org> Co-authored-by: Victor Stinner <victor.stinner@gmail.com> (cherry picked from commit1bc78dea18) * [3.15] gh-156810: Write the profiler's collapsed-stack export as UTF-8 (GH-156811) (#156814) gh-156810: Write the profiler's collapsed-stack export as UTF-8 (GH-156811) (cherry picked from commitc3706f4c68) Co-authored-by: tonghuaroot (童话) <tonghuaroot@gmail.com> (cherry picked from commitfce28da9b7) * [3.15] gh-158552: Wait for Windows threads to suspend before blocking sampling (GH-158802) (#158845) gh-158552: Wait for Windows threads to suspend before blocking sampling (GH-158802) * gh-158552: Wait for Windows threads to suspend before blocking sampling * Use a named Windows thread enumeration status constant (cherry picked from commit1643525f0f) Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com> (cherry picked from commit31288da21d) * [3.15] gh-152721: Fix quadratic RLE replay time in the profiling binary reader (GH-152722) (#158850) Backport of GH-152722. Co-authored-by: tonghuaroot <tonghuaroot@gmail.com> (cherry picked from commitebaca2ac2d) * [3.15] gh-156545: Fix flamegraph export RecursionError on deeply recursive programs (GH-156546) (#158851) Backport of GH-156546. Co-authored-by: tonghuaroot (童话) <tonghuaroot@gmail.com> (cherry picked from commit0ac7217040) * [3.15] gh-158540: Add the profiled script's directory to sys.path (GH-158548) (#158844) gh-158540: Add the profiled script's directory to sys.path (GH-158548) * gh-158540: Add the profiled script's directory to sys.path When a script is profiled with ``python -m profiling.sampling run script.py`` from another directory, the script cannot import modules placed next to it, because ``_sync_coordinator._execute_script()`` executes it with the working directory (added by ``_setup_environment()`` for the module case) as ``sys.path[0]`` instead of the script's own directory. Make the script's directory importable in ``_execute_script()``, matching the behavior of ``python script.py``. Add a regression test that runs the coordinator on a script importing a sibling module. * Update _sync_coordinator.py Comment simplified * gh-158540: Resolve symlinks when adding the script directory to sys.path ``python script.py`` resolves symlinks when computing ``sys.path[0]``, so a script reached through a symlink (``link.py -> sub/where.py``) imports modules from the real script's directory. Apply ``os.path.realpath()`` before taking the directory name, and make sure the result is placed at the front of ``sys.path`` even if it was already listed. Add a regression test for a symlinked script. --------- (cherry picked from commit3f02aab9fa) Co-authored-by: he_tao <53343436+hetaozdh@users.noreply.github.com> Co-authored-by: Eduardo Villalpando Mello <eduardo.villalpando.mello@gmail.com> (cherry picked from commitf52d831397) * [3.15] gh-153838: Skip non-regular source files in the heatmap exporter (GH-153839) (#158853) Backport of GH-153839. Co-authored-by: tonghuaroot <tonghuaroot@gmail.com> (cherry picked from commit48998df251) * [3.15] gh-158539: Fix exception mode missing handlers in generators/coroutines (GH-158581) (#158852) * [3.15] gh-158539: Fix exception mode missing handlers in generators/coroutines (GH-158581) Backport of GH-158581. Co-authored-by: LucasZhou <donghao.zhou@outlook.com> * [3.15] gh-158539: Use portable static assertion messages * [3.15] gh-158539: Keep layout assertions with debug-offset validation * [3.15] gh-158539: Use the platform guard for in-process inspection tests --------- Co-authored-by: LucasZhou <donghao.zhou@outlook.com> (cherry picked from commitd625ecbb8d) * [3.15] gh-158522: Fix truncated stack for a task whose coroutine recurses (GH-158526) (#158870) Co-authored-by: Timofei Ivankov <128279579+deadlovelll@users.noreply.github.com> (cherry picked from commita904b397ed) * gh-156545, gh-158539: Fix deep flamegraph export on small C stacks and macOS runtime lookup (#158874) (cherry picked from commit114de198c0) Include the C-stack test helper from main, introduced byce5ae29ef9, which the regression test requires but 3.15 does not yet provide. --------- Co-authored-by: Miss Islington (bot) <31488909+miss-islington@users.noreply.github.com> Co-authored-by: Maurycy Pawłowski-Wieroński <maurycy@maurycy.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: Victor Stinner <vstinner@python.org> Co-authored-by: Stan Ulbrych <stan@python.org> Co-authored-by: Victor Stinner <victor.stinner@gmail.com> Co-authored-by: tonghuaroot (童话) <tonghuaroot@gmail.com> Co-authored-by: he_tao <53343436+hetaozdh@users.noreply.github.com> Co-authored-by: Eduardo Villalpando Mello <eduardo.villalpando.mello@gmail.com> Co-authored-by: LucasZhou <donghao.zhou@outlook.com> Co-authored-by: Timofei Ivankov <128279579+deadlovelll@users.noreply.github.com>
327 lines
11 KiB
C
327 lines
11 KiB
C
/******************************************************************************
|
|
* Remote Debugging Module - Object Reading Functions
|
|
*
|
|
* This file contains functions for reading Python objects from remote
|
|
* process memory, including strings, bytes, and integers.
|
|
******************************************************************************/
|
|
|
|
#include "_remote_debugging.h"
|
|
#include <limits.h>
|
|
|
|
/* ============================================================================
|
|
* MEMORY READING FUNCTIONS
|
|
* ============================================================================ */
|
|
|
|
#define DEFINE_MEMORY_READER(type_name, c_type, error_msg) \
|
|
int \
|
|
read_##type_name(RemoteUnwinderObject *unwinder, uintptr_t address, c_type *result) \
|
|
{ \
|
|
int res = _Py_RemoteDebug_PagedReadRemoteMemory(&unwinder->handle, address, sizeof(c_type), result); \
|
|
if (res < 0) { \
|
|
set_exception_cause(unwinder, PyExc_RuntimeError, error_msg); \
|
|
return -1; \
|
|
} \
|
|
return 0; \
|
|
}
|
|
|
|
DEFINE_MEMORY_READER(ptr, uintptr_t, "Failed to read pointer from remote memory")
|
|
DEFINE_MEMORY_READER(char, char, "Failed to read char from remote memory")
|
|
|
|
int
|
|
read_py_ptr(RemoteUnwinderObject *unwinder, uintptr_t address, uintptr_t *ptr_addr)
|
|
{
|
|
if (read_ptr(unwinder, address, ptr_addr)) {
|
|
set_exception_cause(unwinder, PyExc_RuntimeError, "Failed to read Python pointer");
|
|
return -1;
|
|
}
|
|
*ptr_addr &= ~Py_TAG_BITS;
|
|
return 0;
|
|
}
|
|
|
|
/* ============================================================================
|
|
* PYTHON OBJECT READING FUNCTIONS
|
|
* ============================================================================ */
|
|
|
|
PyObject *
|
|
read_py_str(
|
|
RemoteUnwinderObject *unwinder,
|
|
uintptr_t address,
|
|
Py_ssize_t max_len
|
|
) {
|
|
// Read the entire PyUnicodeObject at once; for short strings the data
|
|
// is inline right after the header and we'll already have (some of) it.
|
|
char unicode_obj[SIZEOF_UNICODE_OBJ];
|
|
int res = _Py_RemoteDebug_PagedReadRemoteMemory(
|
|
&unwinder->handle,
|
|
address,
|
|
SIZEOF_UNICODE_OBJ,
|
|
unicode_obj
|
|
);
|
|
if (res < 0) {
|
|
set_exception_cause(unwinder, PyExc_RuntimeError, "Failed to read PyUnicodeObject");
|
|
return NULL;
|
|
}
|
|
|
|
Py_ssize_t len = GET_MEMBER(Py_ssize_t, unicode_obj, unwinder->debug_offsets.unicode_object.length);
|
|
if (len < 0) {
|
|
PyErr_Format(PyExc_RuntimeError,
|
|
"Invalid string length (%zd) at 0x%lx", len, address);
|
|
set_exception_cause(unwinder, PyExc_RuntimeError, "Invalid string length in remote Unicode object");
|
|
return NULL;
|
|
}
|
|
if (len > max_len) {
|
|
// gh-157788: a long name must not fail the whole read
|
|
len = max_len;
|
|
}
|
|
|
|
// Inspect state to pick the right data offset and character width.
|
|
// We rely on the remote process sharing this Python version's
|
|
// PyASCIIObject layout, the same assumption already used for `length`.
|
|
struct _PyUnicodeObject_state state = GET_MEMBER(
|
|
struct _PyUnicodeObject_state,
|
|
unicode_obj,
|
|
unwinder->debug_offsets.unicode_object.state);
|
|
|
|
if (!state.compact) {
|
|
PyErr_Format(PyExc_RuntimeError,
|
|
"Cannot read non-compact Unicode object at 0x%lx", address);
|
|
set_exception_cause(unwinder, PyExc_RuntimeError,
|
|
"Legacy (non-compact) Unicode objects are not supported");
|
|
return NULL;
|
|
}
|
|
|
|
int kind = (int)state.kind;
|
|
Py_UCS4 max_char;
|
|
switch (kind) {
|
|
case PyUnicode_1BYTE_KIND:
|
|
max_char = state.ascii ? 0x7F : 0xFF;
|
|
break;
|
|
case PyUnicode_2BYTE_KIND:
|
|
max_char = 0xFFFF;
|
|
break;
|
|
case PyUnicode_4BYTE_KIND:
|
|
max_char = 0x10FFFF;
|
|
break;
|
|
default:
|
|
PyErr_Format(PyExc_RuntimeError,
|
|
"Invalid Unicode kind %d at 0x%lx", kind, address);
|
|
set_exception_cause(unwinder, PyExc_RuntimeError,
|
|
"Invalid kind in remote Unicode object");
|
|
return NULL;
|
|
}
|
|
|
|
size_t header_size = state.ascii
|
|
? (size_t)unwinder->debug_offsets.unicode_object.asciiobject_size
|
|
: (size_t)unwinder->debug_offsets.unicode_object.compactunicodeobject_size;
|
|
|
|
// len * kind is bounded by max_len * 4 (kind <= 4, len <= max_len), so
|
|
// the multiplication can't overflow for any caller-sane max_len, but the
|
|
// explicit cap here keeps a corrupted remote `length` from later turning
|
|
// into a giant allocation.
|
|
size_t nbytes = (size_t)len * (size_t)kind;
|
|
if ((size_t)len > (SIZE_MAX / 4) || nbytes > (size_t)max_len * 4) {
|
|
PyErr_Format(PyExc_RuntimeError,
|
|
"Implausible Unicode byte size %zu at 0x%lx", nbytes, address);
|
|
set_exception_cause(unwinder, PyExc_RuntimeError,
|
|
"Garbage byte size in remote Unicode object");
|
|
return NULL;
|
|
}
|
|
|
|
PyObject *result = PyUnicode_New(len, max_char);
|
|
if (result == NULL) {
|
|
set_exception_cause(unwinder, PyExc_RuntimeError, "Failed to allocate PyUnicode for remote string");
|
|
return NULL;
|
|
}
|
|
if (nbytes == 0) {
|
|
return result;
|
|
}
|
|
|
|
void *data = PyUnicode_DATA(result);
|
|
|
|
// Reuse data already present in the header read; only round-trip for
|
|
// whatever spills past it.
|
|
size_t inline_avail = (header_size < SIZEOF_UNICODE_OBJ)
|
|
? SIZEOF_UNICODE_OBJ - header_size
|
|
: 0;
|
|
size_t inline_bytes = nbytes < inline_avail ? nbytes : inline_avail;
|
|
if (inline_bytes > 0) {
|
|
memcpy(data, unicode_obj + header_size, inline_bytes);
|
|
}
|
|
|
|
if (nbytes > inline_bytes) {
|
|
res = _Py_RemoteDebug_PagedReadRemoteMemory(
|
|
&unwinder->handle,
|
|
address + header_size + inline_bytes,
|
|
nbytes - inline_bytes,
|
|
(char *)data + inline_bytes);
|
|
if (res < 0) {
|
|
Py_DECREF(result);
|
|
set_exception_cause(unwinder, PyExc_RuntimeError, "Failed to read string data from remote memory");
|
|
return NULL;
|
|
}
|
|
}
|
|
|
|
return result;
|
|
}
|
|
|
|
PyObject *
|
|
read_py_bytes(
|
|
RemoteUnwinderObject *unwinder,
|
|
uintptr_t address,
|
|
Py_ssize_t max_len
|
|
) {
|
|
PyObject *result = NULL;
|
|
char *buf = NULL;
|
|
|
|
// Read the entire PyBytesObject at once
|
|
char bytes_obj[SIZEOF_BYTES_OBJ];
|
|
int res = _Py_RemoteDebug_PagedReadRemoteMemory(
|
|
&unwinder->handle,
|
|
address,
|
|
SIZEOF_BYTES_OBJ,
|
|
bytes_obj
|
|
);
|
|
if (res < 0) {
|
|
set_exception_cause(unwinder, PyExc_RuntimeError, "Failed to read PyBytesObject");
|
|
goto err;
|
|
}
|
|
|
|
Py_ssize_t len = GET_MEMBER(Py_ssize_t, bytes_obj, unwinder->debug_offsets.bytes_object.ob_size);
|
|
if (len < 0 || len > max_len) {
|
|
PyErr_Format(PyExc_RuntimeError,
|
|
"Invalid bytes length (%zd) at 0x%lx", len, address);
|
|
set_exception_cause(unwinder, PyExc_RuntimeError, "Invalid bytes length in remote bytes object");
|
|
return NULL;
|
|
}
|
|
|
|
buf = (char *)PyMem_RawMalloc(len+1);
|
|
if (buf == NULL) {
|
|
PyErr_NoMemory();
|
|
set_exception_cause(unwinder, PyExc_MemoryError, "Failed to allocate buffer for bytes reading");
|
|
return NULL;
|
|
}
|
|
|
|
size_t offset = (size_t)unwinder->debug_offsets.bytes_object.ob_sval;
|
|
res = _Py_RemoteDebug_PagedReadRemoteMemory(&unwinder->handle, address + offset, len, buf);
|
|
if (res < 0) {
|
|
set_exception_cause(unwinder, PyExc_RuntimeError, "Failed to read bytes data from remote memory");
|
|
goto err;
|
|
}
|
|
buf[len] = '\0';
|
|
|
|
result = PyBytes_FromStringAndSize(buf, len);
|
|
if (result == NULL) {
|
|
set_exception_cause(unwinder, PyExc_RuntimeError, "Failed to create PyBytes from remote bytes data");
|
|
goto err;
|
|
}
|
|
|
|
PyMem_RawFree(buf);
|
|
assert(result != NULL);
|
|
return result;
|
|
|
|
err:
|
|
if (buf != NULL) {
|
|
PyMem_RawFree(buf);
|
|
}
|
|
return NULL;
|
|
}
|
|
|
|
long
|
|
read_py_long(
|
|
RemoteUnwinderObject *unwinder,
|
|
uintptr_t address
|
|
)
|
|
{
|
|
unsigned int shift = PYLONG_BITS_IN_DIGIT;
|
|
|
|
// Read the entire PyLongObject at once
|
|
char long_obj[SIZEOF_LONG_OBJ];
|
|
int bytes_read = _Py_RemoteDebug_PagedReadRemoteMemory(
|
|
&unwinder->handle,
|
|
address,
|
|
(size_t)unwinder->debug_offsets.long_object.size,
|
|
long_obj);
|
|
if (bytes_read < 0) {
|
|
set_exception_cause(unwinder, PyExc_RuntimeError, "Failed to read PyLongObject");
|
|
return -1;
|
|
}
|
|
|
|
uintptr_t lv_tag = GET_MEMBER(uintptr_t, long_obj, unwinder->debug_offsets.long_object.lv_tag);
|
|
int negative = (lv_tag & 3) == 2;
|
|
Py_ssize_t size = lv_tag >> 3;
|
|
|
|
if (size == 0) {
|
|
return 0;
|
|
}
|
|
|
|
// Validate size: reject garbage (negative or unreasonably large)
|
|
if (size < 0 || size > MAX_LONG_DIGITS) {
|
|
PyErr_Format(PyExc_RuntimeError,
|
|
"Invalid PyLong digit count: %zd (expected 0-%d)", size, MAX_LONG_DIGITS);
|
|
set_exception_cause(unwinder, PyExc_RuntimeError,
|
|
"Invalid PyLong size (corrupted remote memory)");
|
|
return -1;
|
|
}
|
|
|
|
// Calculate how many digits fit inline in our local buffer
|
|
Py_ssize_t ob_digit_offset = unwinder->debug_offsets.long_object.ob_digit;
|
|
Py_ssize_t inline_digits_space = SIZEOF_LONG_OBJ - ob_digit_offset;
|
|
Py_ssize_t max_inline_digits = inline_digits_space / (Py_ssize_t)sizeof(digit);
|
|
|
|
digit *digits = (digit *)PyMem_RawMalloc(size * sizeof(digit));
|
|
if (!digits) {
|
|
PyErr_NoMemory();
|
|
set_exception_cause(unwinder, PyExc_MemoryError, "Failed to allocate digits for PyLong");
|
|
return -1;
|
|
}
|
|
|
|
if (size <= max_inline_digits && size <= _PY_NSMALLNEGINTS + _PY_NSMALLPOSINTS) {
|
|
memcpy(digits, long_obj + ob_digit_offset, size * sizeof(digit));
|
|
} else {
|
|
bytes_read = _Py_RemoteDebug_PagedReadRemoteMemory(
|
|
&unwinder->handle,
|
|
address + (uintptr_t)unwinder->debug_offsets.long_object.ob_digit,
|
|
sizeof(digit) * size,
|
|
digits
|
|
);
|
|
if (bytes_read < 0) {
|
|
set_exception_cause(unwinder, PyExc_RuntimeError, "Failed to read PyLong digits from remote memory");
|
|
goto error;
|
|
}
|
|
}
|
|
|
|
unsigned long limit = negative
|
|
? (unsigned long)LONG_MAX + 1UL
|
|
: (unsigned long)LONG_MAX;
|
|
unsigned long value = 0;
|
|
|
|
for (Py_ssize_t i = size; i-- > 0;) {
|
|
if (digits[i] >= PyLong_BASE) {
|
|
PyErr_Format(PyExc_RuntimeError,
|
|
"Invalid PyLong digit: %u (base %u)", digits[i], PyLong_BASE);
|
|
set_exception_cause(unwinder, PyExc_RuntimeError,
|
|
"Invalid PyLong digit (corrupted remote memory)");
|
|
goto error;
|
|
}
|
|
if (value > ((limit - (unsigned long)digits[i]) >> shift)) {
|
|
PyErr_SetString(PyExc_OverflowError,
|
|
"Remote PyLong value does not fit in C long");
|
|
set_exception_cause(unwinder, PyExc_OverflowError,
|
|
"Remote PyLong value is too large");
|
|
goto error;
|
|
}
|
|
value = (value << shift) | (unsigned long)digits[i];
|
|
}
|
|
PyMem_RawFree(digits);
|
|
if (negative) {
|
|
if (value == (unsigned long)LONG_MAX + 1UL) {
|
|
return LONG_MIN;
|
|
}
|
|
return -(long)value;
|
|
}
|
|
return (long)value;
|
|
error:
|
|
PyMem_RawFree(digits);
|
|
return -1;
|
|
}
|