mirror of
https://github.com/RustPython/RustPython.git
synced 2026-10-28 18:59:04 +08:00
stack/python-3.15-8950
108
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 youknowdot
|
29f819ec40 |
Implement the stable ABI bytes formatting functions (#9060)
Assisted-by: OpenAI Codex:model identifier unavailable |
||
|
youknowdot
|
d674ea11e3 |
Drop the legacy abi3-only PyCFunction_Call export (#9059)
Follow up on #9055: RustPython targets abi3t, which does not require this legacy binary-compatibility entry point. Assisted-by: OpenAI Codex:model identifier unavailable |
||
|
youknowdot
|
bc5b9abd75 |
Export the stable ABI PyCFunction_Call shim (#9055)
Assisted-by: OpenAI Codex:model identifier unavailable |
||
|
이광효andJeong YunWon
|
a8e4bde885 |
Reject unsupported native keywords before argument binding (#8957)
* Reject unsupported native keywords before argument binding Add explicit keyword dispatch metadata independently of inferred Rust calling flags. Reject nonempty keywords before binding and conversion, while preserving descriptor self checks and keyword-capable calls. Annotate 31 builtins and list.append/list.clear, derive C-API policies from ml_flags, and group dynamic registration metadata in MethodDefSpec. Add regression coverage and remove the id doctest expected-failure marker. Validation: workspace and C-API tests/clippy, VM without doc, keyword regressions, and test_call/test_extcall/test_builtin/test_list passed. The full snippet suite had 475 passes and 7 failures: six CPython-side expectation mismatches and one missing optional SQLite module. Assisted-by: Codex:GPT-6 * Infer native keyword dispatch from argument metadata Replace no_keywords annotations with registration-time inference from FromArgs::PARAMS, including nested parameter metadata. Forward arguments with keyword-capable parameters to the binder, preserving min/max body validation and descriptor self checks. Add regressions for raw arguments, keyword argument structs, and list.sort. Tested: C-API tests, workspace and C-API Clippy, VM without doc, both keyword regression snippets, and test_call/test_extcall/test_builtin/test_list. Workspace tests reported eight _opcode snapshot failures. The full snippet suite reported 477 passes and seven failures: six CPython-side failures and one RustPython failure due to the missing optional SQLite module. Assisted-by: Codex:GPT-6 * Simplify native function registration metadata Reuse SimpleItemMeta for pyfunction attributes and remove FunctionItemMeta. Remove MethodDefSpec and restore the original new_method_def arguments with an additional keyword_dispatch parameter, updating its callers. Consolidate keyword dispatch regressions in syntax_function_args.py, preserving their assertions and placing them before builtin shadowing. Validation: workspace and C-API tests/clippy, VM without doc, the combined snippet, and test_extcall/test_builtin/test_list/test_call passed. The full snippet suite reported 477 passes and seven existing failures: six CPython-side failures and one missing optional SQLite module. Assisted-by: Codex:GPT-6 * Reject native keywords by calling convention flags Remove `KeywordDispatch`, the `keyword_dispatch` field of `PyMethodDef`, `PyMethodDef::with_keyword_dispatch`, `function::keyword_dispatch` and the derive macros' `keyword_dispatch_tokens`. Native functions and method descriptors reject nonempty keywords when `flags` lacks `KEYWORDS`, and report the unqualified name when `flags` contains `VARARGS`. `Context::new_method_def` takes `(name, f, flags, doc)` again. Add a snippet for `int.__new__` called with a keyword argument. Assisted-by: Claude Code:claude-opus-5-5 --------- Co-authored-by: Jeong YunWon <jeong@youknowone.org> |
||
|
Bas Schoenmaeckers
|
e6cbf3628d |
Add buffer support to c-api (#8071)
* Add buffer support to c-api * Review * Refactor PyBuffer * Manual implement `PyBuffer_GetPointer` * Review * Store in 1 Box |
||
|
Bas Schoenmaeckers
|
5b3e2a40e2 |
Add recursion guard to the c-api (#8796)
* Add recursion guard to the c-api * Do not change with_recursion as this has side effects |
||
|
Bas Schoenmaeckers
|
ada43c4f82 |
Add PyABIInfo_Check to the c-api (#8941)
|
||
|
Bas Schoenmaeckers
|
d81b7c2cb1 |
Add basic c-variadic functions to the c-api (#8933)
* Add basic c-variadic functions to the c-api * Review |
||
|
Jeong, YunWon
|
1f83b1593f |
Fix CI failures and bump version to 0.6.1 (#8940)
* Use AtomicPyTypeFlags in PyType_FromSlots PyType_FromSlots still imported the old PyAtomicTypeFlags name. Assisted-by: Grok:4.7 * Pass CPython test args as a bash array Assisted-by: Grok:4.7 * Bump workspace version to 0.6.1 Assisted-by: Grok:4.7 * Restore the CPython test step in ci.yaml Assisted-by: Grok:4.7 |
||
|
Alessio Attilio
|
ce3f671e10 |
feat(capi): implement PyModule_Add* and module helpers from modsupport.h (#8927)
Assisted-by: Gemini:gemini-3.7-flash |
||
|
Bas Schoenmaeckers
|
b3c84c0fc7 |
Add first draft of PyType_FromSlots (#8918)
|
||
|
Jeong, YunWon
|
1d7d704414 |
Use bitflagset for type flags (#8873)
* Use bitflagset for type flags PyTypeFlags is defined with bitflagset! and the atomic tp_flags word with atomic_bitflagset!, replacing the hand-written PyAtomicTypeFlags. Assisted-by: Grok:4.7 Assisted-by: Claude:claude-opus-5-5 * Use bitflagset 0.0.4 from its git main branch bitflagset 0.0.4 is not on crates.io yet. Patch it to the git main branch in the workspace and in the example projects that depend on rustpython-vm, since each of them is a separate workspace. Assisted-by: Claude:claude-opus-5-5 * Depend on crates.io bitflagset 0.0.4 bitflagset 0.0.4 is published. Drop the git patches from the workspace and the separate example-project workspaces. Assisted-by: Grok:4.7 * Keep empty patch tables in example workspaces The example workspaces already have an empty [patch.crates-io] table. Dropping the bitflagset git line leaves that table in place. Assisted-by: Grok:4.7 * Call bitflagset methods directly for type flags Drop the plain-set has_feature wrapper and the intersects, is_created_with_flags, DEFAULT, and heap_type_flags aliases. Single-bit updates use insert, and flag combinations use from_slice. Assisted-by: Grok:4.7 * OR paired type flags in one update HAS_WEAKREF with MANAGED_WEAKREF, and HAS_DICT with MANAGED_DICT (and INLINE_VALUES when present), are applied with one |=. Assisted-by: Grok:4.7 * Store heap type flag sets as constants HEAPTYPE | BASETYPE, and the dict and immutable variants built from it, are const values instead of from_slice calls at each use. Assisted-by: Grok:4.7 * Test SEQUENCE and MAPPING in one load PyTypeFlags::COLLECTION is the two-bit mask. Callers check it with one is_disjoint instead of two contains loads. Assisted-by: Grok:4.7 * Fix clippy 1.99 lints in the vm Pass closure arguments by value where clippy reports a needless borrow, drop the unfulfilled redundant_else expectation on sum, and return the HRESULT after todo!(). Assisted-by: Grok:4.7 |
||
|
Jeong, YunWon
|
8f84fea377 |
Fix 32-bit member layout assertions and address member descriptor review (#8882)
* Address review findings on member descriptors Member definitions carry an internal `PY_ATOMIC` flag taken from `MemberLayout::ATOMIC`, and loads use an atomic access only for atomic storage; extension members never set it. `XMLParser` publishes its new state, `entity` and `target` only after every handler lookup succeeds. `subs_parameters` reads type names only for the error. The unraisable hook prints `<unknown>` when `__module__` fails or is not a str. The builtin function `__reduce__` decides on the bound object alone. Assisted-by: Grok:4.7 Assisted-by: Claude:claude-opus-5-5 * Fix member layout assertions on 32-bit targets The tp_flags assertion compared AtomicU64 with u64 alignment, which differs on i686. The kevent assertion required usize, isize, c_ulong and c_long to be 64-bit. Both now check only what the member cells need. Assisted-by: Grok:4.7 Assisted-by: Claude:claude-opus-5-5 * Use bitflags for member definition flags PY_READONLY, PY_AUDIT_READ, PY_RELATIVE_OFFSET and PY_ATOMIC become the repr(transparent) PyMemberFlags. C extension flags are read with from_bits_retain. Assisted-by: Grok:4.7 Assisted-by: Claude:claude-opus-5-5 |
||
|
Jeong, YunWon
|
31963fa767 |
Expose more attributes as member descriptors (#8857)
* Read type.__module__ from the dict only for heap types A static type's `__module__` is the text before the last `.` of its name, or `builtins`. A heap type reads its dict and raises `AttributeError` when the entry is missing; `type` and `object` reprs ignore that error. `type()` no longer stores `__module__ = None` when the caller's globals have no `__name__`. `#[pyexception(module = ...)]` now reaches the type name, and static types that relied on a dict `__module__` (ast nodes, pyexpat, ssl, _hashlib and _interpchannels errors, ExceptionGroup) carry their module. Assisted-by: Grok:4.7 Assisted-by: Claude:claude-opus-5-5 * Expose more builtin attributes as member descriptors `code.co_flags`, `co_nlocals`, `co_filename`, `super.__self__`, `__self_class__`, `__thisclass__`, `traceback.tb_frame`, `tb_lasti`, `type.__itemsize__` and `builtin_function_or_method.__module__` are now `#[pymember]` fields instead of getsets. `super` keeps its three references in atomic cells instead of a lock, `tb_lasti` is an `i32`, `co_nlocals` is stored at construction, and the builtin function `__module__` is a writable object cell. Assisted-by: Grok:4.7 Assisted-by: Claude:claude-opus-5-5 * Expose stdlib attributes as member descriptors Decompressor `eof`/`needs_input`/`unused_data` (bz2, lzma, zlib, plus lzma `check`), `ContextVar.name`, `functools.partial` `func`/`args`/ `keywords`, socket `family`/`type`/`proto`, `select.kevent` fields, `SemLock` `handle`/`kind`/`maxvalue`/`name`, `Pickler.bin`/`fast`, `xmlparser.intern`, ElementTree `XMLParser.entity`/`target`, `UCD.unidata_version`, `_ExceptHookArgs` fields, multibytecodec `stream`, and `_io` `Buffered*.raw` and `TextIOWrapper` `buffer`/`encoding`/`line_buffering`/`write_through` are now `#[pymember]` fields instead of getsets. Values that changed under a lock move to atomic cells on the payload. New member kinds `Short`, `Long`, `String`, `UShort`, `ULong`, `LongLong` and `ULongLong` access exactly the width of their C type. Assisted-by: Grok:4.7 Assisted-by: Claude:claude-opus-5-5 * Match bare super and static type itemsize `super.__new__(super)` leaves `__thisclass__`, `__self__` and `__self_class__` empty, and a super with no type takes the zero-argument path when bound. Static types carry their `tp_itemsize` (`type`, `memoryview`, `code`, `frame`, generators, `_sre` objects, `_ctypes` thunks), inherit it from their direct base when they leave it at 0, and a variable-size base rejects any nonempty `__slots__` and a weakref slot. `_thread._ExceptHookArgs` is a struct sequence. Assisted-by: Grok:4.7 Assisted-by: Claude:claude-opus-5-5 * Expose SimpleNamespace.__dict__, __basicsize__ and __flags__ as members `SimpleNamespace.__dict__` addresses the instance dict cell the same way `module.__dict__` does. `slots.basicsize` now holds the full `tp_basicsize` (header plus payload), so `__basicsize__` reads it directly. Type flags are one atomic word: `__abstractmethods__` and `_abc` set their bits in it in place, the separate `abc_tpflags` word is gone, and `__flags__` reads the word as an unsigned long member. Assisted-by: Grok:4.7 Assisted-by: Claude:claude-opus-5-5 |
||
|
Jeong, YunWon
|
be23b2c562 |
Store database docstrings xz-compressed behind an optional doc feature (#8836)
* Skip Python-defined classes when collecting builtin type docs generate.py recorded every object.__subclasses__() entry as builtins.<name> after importing the whole stdlib, so classes written in Python (tkinter, unittest, tarfile, ...) ended up in the doc database. Classes that are mutable heap types without C method descriptors and whose module is not a C extension are now skipped, and their entries are removed from data.inc.rs. Assisted-by: Claude:claude-opus-5-5 Assisted-by: Grok:grok-4.7 * Store database docstrings xz-compressed behind an optional doc feature - rustpython-doc's build script compresses all database docstrings into one xz blob with an offset index; identical texts share one offset. - Macros emit blob references for database docs; vm decompresses the blob on first access. - New `doc` cargo feature (default on) in rustpython, rustpython-vm and rustpython-stdlib. Without it the blob and xz are not linked and database-backed __doc__ is None; __text_signature__ is unaffected. - Native types without a docstring get __doc__ = None in their dict. - Workflows that build with --no-default-features enable `doc` where tests or releases need docstrings. Assisted-by: Claude:claude-opus-5-5 Assisted-by: Grok:grok-4.7 |
||
|
Jeong, YunWon
|
bdc9246b37 |
Address builtin members by field offset (#8816)
* Address builtin members by field offset `#[pymember]` now declares a field of the payload and generates a PyMemberDef whose offset is the payload offset plus the field offset. The getter/setter function form and `MemberAccess::Func` are removed; builtin, slot, and C-API members share one get/set path. The macro checks the field type against the member type. Converted members: property fget/fset/fdel/__doc__, function __globals__/__closure__/__builtins__/__doc__/__module__, staticmethod and classmethod __func__/__wrapped__, complex real/imag (T_DOUBLE), frame f_trace_lines (T_BOOL), defaultdict default_factory, getset and member descriptor __name__/__objclass__, module __dict__. frame.f_trace_opcodes is a getset. Writable member fields are atomic pointer cells. The instance dict pointer is an atomic cell at a fixed offset in front of the object, and dict presence follows the type's HAS_DICT flag. The pickle basicsize check no longer counts dict, weakref, and slot storage, which live outside the payload. Assisted-by: Grok:4.7 Assisted-by: Claude:claude-opus-5-5 * Address member review findings - `#[pymember]` keeps leading `#[cfg]` attributes on the generated registration, and malformed `field`/`offset` values are compile errors. - The `descriptor` module and the items the expansion names are public. - Writable double members are an `AtomicU64` cell, loaded and stored with relaxed atomics. Readonly double members stay plain `f64` reads. - `LOAD_ATTR_SLOT` is not specialized for `Py_AUDIT_READ` members. - `defaultdict.__reduce__` passes no factory argument when `default_factory` is None. Assisted-by: Grok:4.7 Assisted-by: Claude:claude-opus-5-5 * Declare members on payload fields `#[pymember]` goes on a payload struct field, or on the struct with an explicit `offset` when the storage is not a payload field. The struct `#[pyclass]` builds `PyClassDef::MEMBERS`, a static `PyMemberSpec` table, and `extend_class` registers one member descriptor per entry. `path` addresses a subfield, one field may carry several members, and a `#[cfg]` on the field gates its entry. The impl-side const form is removed and now a compile error. Assisted-by: Grok:4.7 Assisted-by: Claude:claude-opus-5-5 * Split nullable and non-null atomic cells The nullable cell API (`new_empty`, `load_owned`, `store`, `compare_exchange_empty`) moves from `PyAtomicRef<PyObject>` to `PyAtomicRef<Option<PyObject>>`, and the typed `PyAtomicRef<Option<T>>` shares it. The non-null `PyAtomicRef<PyObject>` gets `load_owned` and `store` that return the object directly. Slot cells, property fget/fset/fdel/__doc__, and defaultdict default_factory are nullable cells. staticmethod and classmethod callable are non-null cells and are read without `unwrap`. The instance dict cell is `PyAtomicRef<Option<PyDict>>`, so reading it no longer downcasts. Assisted-by: Grok:4.7 Assisted-by: Claude:claude-opus-5-5 * Drop the unused cold trace_lines flag `FrameColdData::trace_lines` had no writer after `f_trace_lines` became a frame object member. A frame without a frame object traces lines by default. Assisted-by: Claude:claude-opus-5-5 * Compute payload offsets from the merged Py header `PyInner` is now `Py`, so the payload offset and the dict member offset are measured from `Py<T>`. Assisted-by: Claude:claude-opus-5-5 * Expose descriptor names and owner types as members method_descriptor, classmethod_descriptor, and wrapper_descriptor expose `__objclass__` and `__name__` as readonly members, and method-wrapper exposes `__self__` as a readonly member. The separate `objclass` field of method and classmethod descriptors always equaled `common.typ` and is removed. Assisted-by: Grok:4.7 Assisted-by: Claude:claude-opus-5-5 * Expose remaining builtin tp_members as members Attributes stored in a payload field become readonly or writable member descriptors: bound method __func__/__self__, code co_argcount/co_posonlyargcount/co_kwonlyargcount/co_stacksize (Py_T_UINT), generic alias, coroutine and async generator members, interpolation and template, range and slice start/stop/step, type __base__, union, and the typing TypeVar, ParamSpec, TypeVarTuple, and TypeAliasType members. `MemberKind` gains `Int` and `Uint` with the `PyMember_GetOne`/`SetOne` range checks. Attributes computed on access or kept behind a lock stay getsets. Assisted-by: Grok:4.7 Assisted-by: Claude:claude-opus-5-5 * Spell a docless member as doc = false `#[pymember(doc = false)]` stores no docstring, matching the `doc = "..."` form, and replaces the separate `no_doc` flag. Assisted-by: Claude:claude-opus-5-5 * Make pymember readonly by default `#[pymember]` declares a readonly member unless `writable` is given. The `readonly` option is removed and every declaration is converted. Assisted-by: Claude:claude-opus-5-5 * Check the receiver of method descriptors Binding or calling a method_descriptor with an object that is not an instance of the owner type raises TypeError before the native function runs. Static and class methods are exempt. Assisted-by: Grok:4.7 Assisted-by: Claude:claude-opus-5-5 * Expose exception attributes as members BaseException.__suppress_context__, StopIteration.value, SystemExit.code, ImportError, NameError, AttributeError, OSError, SyntaxError, BaseExceptionGroup and the UnicodeEncode/Decode/TranslateError attributes are member descriptors backed by payload fields. ImportError, NameError, AttributeError and SyntaxError gain payloads for their fields, and the methods that read those attributes read the fields. Add the Py_ssize_t member kind for UnicodeError start and end. Assisted-by: Grok:4.7 Assisted-by: Claude:claude-opus-5-5 * Avoid same-type casts in unsigned member conversion On targets where c_ulong is u32, casting through c_ulong was a no-op cast that clippy rejects. Assisted-by: Claude:claude-opus-5-5 * Prefer an explicit pymember doc over the doc table doc = "text" is used as the docstring without consulting the stored attribute documentation; only an omitted doc looks it up. Drop the member doc strings that repeated the stored documentation. Assisted-by: Grok:4.7 Assisted-by: Claude:claude-opus-5-5 * Infer pymember kinds from field types MemberLayout gives the member kind of each accepted field type, and the macro reads it from the field or path subfield, so type = "bool" and the other numeric spellings are gone. type only selects object_ex. Writable doubles use a new AtomicF64 cell instead of AtomicU64 bits. MemberLayout and MemberCell replace the per-kind marker traits. Assisted-by: Grok:4.7 Assisted-by: Claude:claude-opus-5-5 * Read nullable atomic cells only through load_owned PyAtomicRef<Option<T>> and PyAtomicRef<Option<PyObject>> lose to_owned, to_owned_ordering and try_to_owned; load_owned, which retries the incref until it applies to the pointer still in the cell, is the one owned read. Callers that kept a deref borrow across VM calls now take a strong reference. Assisted-by: Grok:4.7 Assisted-by: Claude:claude-opus-5-5 * Inherit static type slots from the whole MRO extend_class copied slots only from the direct bases, and tp_init is copied only from a base that defines it, so TabError missed the SyntaxError init that IndentationError inherits. Walk the MRO the same way init_slots does. Assisted-by: Grok:4.7 Assisted-by: Claude:claude-opus-5-5 * Let Rust doc comments win over stored docstrings A /// comment on a Python-exposed item is its docstring; the rustpython-doc entry is used only when there is none. #[pymember] reads the field's /// comment instead of doc = "...". Doc comments that were shadowed by a stored docstring become // comments, so no docstring changes. AGENTS.md describes the rule. Assisted-by: Grok:4.7 Assisted-by: Claude:claude-opus-5-5 * Adapt members to upstream changes _collections._tuplegetter.__doc__ is a writable nullable member on the payload field. The impl-side const form no longer compiles. Assisted-by: Grok:4.7 * Drop docstring text left as comments and inline downcast_cell Comments that only repeated an item's docstring are removed; developer notes stay as // comments. The typed cell methods call downcast_unchecked directly, each with its own safety note. Assisted-by: Grok:4.7 Assisted-by: Claude:claude-opus-5-5 * Read sqlite row factories through load_owned The nullable cell no longer has to_owned; the sqlite module is built only with the sqlite feature, so the default build did not catch it. Assisted-by: Claude:claude-opus-5-5 |
||
|
Jeong, YunWon
|
dc3b93abfb |
Read tuple, bytes, int, float and complex payloads through views on Py<T> (#8819)
* Read PyTuple payloads through as_slice() - Add Py<PyTuple<R>>::as_slice(); it is generic over R, so typed tuples use the same view. - Callers reach slice methods, indexing and slice arguments through as_slice(). - PyTuple's own sequence and mapping slots and marshal's set_marshal_item reach the remaining PyTuple methods through the payload field. - The Py<T>::payload field is pub(crate). Assisted-by: Claude Code:claude-opus-5-5 * Read PyBytes payloads through as_bytes() - Callers read bytes through Py<PyBytes>::as_bytes(). - bytes.rs reaches PyBytesInner and the remaining PyBytes methods through the payload field; fspath and os call payload.contains_nuls(). - DictKey for [u8] falls back to the PyObject key_eq; it previously resolved to its own key_eq through Deref. Assisted-by: Claude Code:claude-opus-5-5 * Copy capi tuple arguments with as_slice().to_vec() Assisted-by: Claude Code:claude-opus-5-5 * Read sqlite3 Row tuples through as_slice() Assisted-by: Claude Code:claude-opus-5-5 * Read platform-specific tuples and bytes through as_slice() and as_bytes() Assisted-by: Claude Code:claude-opus-5-5 * Read PyInt and PyFloat payloads through as_bigint() and to_f64() Add `Py<PyInt>::as_bigint()`, `Py<PyInt>::try_to_primitive()` and `Py<PyFloat>::to_f64()`, which forward to the payload methods of the same name. Reads of the `value` field and primitive conversions go through these views; other PyInt and PyFloat methods are called through the `payload` field inside rustpython-vm and through payload() outside it. Assisted-by: Claude Code:claude-opus-5-5 * Read PyComplex payloads through as_complex() Add Py<PyComplex>::as_complex() and use it at the call sites that read the payload directly. Assisted-by: Claude Code:claude-opus-5-5 * Move PyInt conversion helpers to Py<PyInt> try_to_i64_fast(), to_str_radix_10(), as_u32_mask(), as_u64_mask(), try_to_primitive() and try_to_primitive_raw() only read the integer value, so they are defined on Py<PyInt> and callers no longer go through the payload. PyRange computes its protocol length from the BigInt directly instead of building a PyInt payload. Assisted-by: Claude Code:claude-opus-5-5 * Share one conversion between try_to_primitive() and its OverflowError variant Py<PyInt>::try_to_primitive() and try_to_primitive_in_range(), renamed from try_to_primitive_raw(), both call to_primitive(). They differ only in how a negative value for an unsigned type is reported: ValueError for try_to_primitive(), OverflowError for try_to_primitive_in_range(). The target type is bound with `for<'a> TryFrom<&'a BigInt>`, so the methods no longer tie a lifetime to &self. Assisted-by: Claude Code:claude-opus-5-5 * Move PyInt bitwise operators to Py<PyInt> __xor__, __or__, __and__ and their int_op() helper are defined on Py<PyInt>. PyBool calls them without going through the payload. Assisted-by: Claude Code:claude-opus-5-5 * Move PyBytes operators to Py<PyBytes> __add__, __contains__, __getitem__ and __mod__ are defined on Py<PyBytes>. The subscript, contains and remainder slots call them without going through the payload. Assisted-by: Claude Code:claude-opus-5-5 * Move PyInt general_op() and modpow() to Py<PyInt> The power slot calls them on Py<PyInt> without going through the payload. Assisted-by: Claude Code:claude-opus-5-5 |
||
|
Jeong, YunWon
|
0ebc6570e3 |
Add payload views and Py handle shortcuts (#8812)
* Add payload views and Py handle shortcuts PyTuple::payload_slice, PyBytes::payload_bytes, and PyStr::payload_wtf8 expose the payload data. Common payload methods are forwarded from Py<T>. Assisted-by: Grok CLI:grok-4.7 * Call PyTuple::as_slice from payload_slice `&*self` tripped clippy borrow_deref_ref and needless_borrow. Assisted-by: Grok CLI:grok-4.7 * Use Element.is_empty in the truth-value check len() != 0 trips clippy::len_zero once is_empty is available. Assisted-by: Grok CLI:grok-4.7 * Drop redundant payload view aliases as_bytes, as_wtf8, and as_slice already forward to the payload methods. Assisted-by: Grok CLI:grok-4.7 * Forward more payload methods from Py handles Frame iframe, bytearray buffer, set add, exception args and traceback, and length or lookup helpers are available on Py<T>. Assisted-by: Grok CLI:grok-4.7 * Read type slots through payload in macros Generated subclass checks and MRO slot inheritance no longer use Deref to reach PyType.slots or attributes. Assisted-by: Grok CLI:grok-4.7 * Index a few tuples through as_slice These Py<PyTuple> indexes no longer rely on Deref to [T]. Assisted-by: Grok CLI:grok-4.7 * Read type slots through Py<PyType>::slots obj.class().slots and builtin type slots now call slots() instead of the payload field. Assisted-by: Grok CLI:grok-4.7 * Read type version and heap layout through Py<PyType> Inline-cache checks use tp_version_tag(). Heap-type weakref checks use heaptype_ext(). Assisted-by: Grok CLI:grok-4.7 * Call slots() on Py type handles Type creation, type methods, and frame specialization read slots through Py<PyType> instead of the payload field. Assisted-by: Grok CLI:grok-4.7 * Use as_bytes, as_slice, and traceback accessors Bytes lengths go through as_bytes(). Tuple len, emptiness, and iteration go through as_slice(). Exception handles use traceback(). Assisted-by: Grok CLI:grok-4.7 * Drop shortcuts that only forward payload methods Tuple length and iteration go through as_slice. Dict and list expose len. Set add stays on the payload. Assisted-by: Grok CLI:grok-4.7 * Drop dict and list len shortcuts len without is_empty fails clippy. Tuple element lookup uses as_slice().get instead of iter().nth. Assisted-by: Grok CLI:grok-4.7 |
||
|
Jeong, YunWon
|
d474353be5 |
Address slot members by byte offset (#8811)
* Keep slot values freelist-eligible Member cells now sit in the object allocation ahead of the extension header, so a slot read is one pointer instead of a boxed slice. A value stored in a slot is not marked published. With threading, only the value that leaves the slot is, so its free can wait out a reader. Without threading a slot read is a single incref. Assisted-by: Grok:4.7 * Address slot members by byte offset A slot member's offset is the byte distance from the object to its inline pointer cell. Loads and stores, including the specialized instructions, use that address. A relative C offset is resolved by adding the object header and the type's basic size. Assisted-by: Grok:4.7 * Keep base slot offsets stable on subclasses Cell 0 sits directly in front of the extension header. Further slots extend away from the object, so a member offset computed for a base class still addresses the same cell on a subclass. Assisted-by: Grok:4.7 * Pack slot cells flush against the extension header On 32-bit targets an odd slot count is not a multiple of ObjExt's alignment. The padding now sits in front of the cells, so cell 0 stays immediately before the header. The non-threading slot load is also the function's return value. Assisted-by: Grok:4.7 * Place weakref lists in front of slot cells A subclass that adds a weakref list no longer inserts it between ObjExt and the object, so inherited member offsets stay valid. Assisted-by: Grok:4.7 |
||
|
Bas SchoenmaeckersandJeong, YunWon
|
a1971c1d9d |
Unify PyObject dereferencing in c-api (#8292)
* Add `FfiPtrExt` for uniform pointer dereferences * Panic on NULL ptrs * capi: import type_name from core, not std Assisted-by: Grok:4.6 * Use `assume_owned_or_err` in PyGetSetDef * Update latest code --------- Co-authored-by: Jeong, YunWon <jeong@youknowone.org> |
||
|
Bas Schoenmaeckers
|
3272bee78b |
Add PyNumber_AsSsize_t to the c-api (#8801)
|
||
|
Jeong, YunWon
|
2619bdcca5 | Fix hasattr (#8809) | ||
|
Jeong, YunWon
|
155cea292f |
Store instance slots as atomic object pointers (#8804)
* Store instance slots as atomic object pointers PyMemberDef now carries type, offset, and flags. Slot values are atomic PyObject pointers with try-incref reads, so a concurrent store cannot free the memory under a reader. Objects placed in a slot stay published until QSBR reclaims them. Assisted-by: Grok:4.7 * Collapse nested slot load condition Assisted-by: Grok:4.7 * Reject over-long digit strings before parsing A pure ASCII digit string longer than the int digit limit now fails from a wide digit check. The denial-of-service test no longer spends the failure path scanning and parsing the whole literal. Assisted-by: Grok:4.7 * Collapse nested digit-limit condition Assisted-by: Grok:4.7 * Restore MemberKind for member type codes The VM stores the member type as an enum whose discriminants match the C type codes. The C API still reads the integer and converts it at the boundary. Flags stay integers. Assisted-by: Grok:4.7 * Mark MemberKind::from_i32 as must_use Assisted-by: Grok:4.7 |
||
|
Jeong, YunWon
|
0940849029 |
Borrow tp_descr_get/tp_init arguments and by-value public helpers (#8803)
* Borrow tp_descr_get/tp_init arguments and by-value public helpers - DescrGetFunc and GetDescriptor::descr_get take (&PyObject, Option<&PyObject>, Option<&PyObject>); impls call .to_owned() where they store or return an argument. - InitFunc takes &PyObject and Initializer::init takes &Py<Self>. - VirtualMachine::get_attribute_opt, print_exception, call_get_descriptor(_specific) and other helpers flagged by clippy::needless_pass_by_value take &PyObject / &Py<T>; callers in crates/, examples/ and tests/ drop the clones made for the call. Assisted-by: Claude Code:claude-opus-5-5 Assisted-by: Grok CLI:grok-4.7 * Pass the borrowed zelf to internal_close in WindowsConsoleIO init Assisted-by: Claude Code:claude-opus-5-5 * Fix borrowed signatures missed by the rebase OrderedDict.init, classmethod_descriptor.descr_get, and the warnings get_attribute_opt callers still used owned handles. Assisted-by: Grok CLI:grok-4.7 |
||
|
Jeong, YunWonandLee Dogeon
|
afad92bf1b |
Add OrderedDict, marshal, inspect, and chflags bitflagset (#8780)
* Implement _collections.OrderedDict Extracted from #6799: native OrderedDict as a dict subclass, iterator mutation tracking, and mapping-view set operations. * Add OrderedDict, warnings source lines, and socket sendmsg Implement _collections.OrderedDict as a dict subclass with a linked list of insertion-order nodes. Pack struct F/D as native float/double complex pairs, and report unexpected __replace__ field names as a list repr. Register encodings.latin_1 at startup. Reject NUL in input() prompts. Copy non-exact type dicts in mapping order. Fetch warning source lines through _bless_my_loader and unicode splitlines. Keep the last warnings.filters list after deletion. Snapshot sendmsg sequences before converting buffers, parse ancillary ints with __index__, and dispatch AF_CAN by protocol. Store sys.trace and sys.profile on each ThreadSlot so _settraceallthreads/_setprofileallthreads are visible to sys.gettrace/getprofile on other threads. Use <frozen importlib._bootstrap> as frozen importlib co_filename. Keep HEAPTYPE socket objects GC-tracked. Assisted-by: Grok:grok-4.6 * Load marshal via readinto and dump sets in dumps order marshal.load fills a scratch buffer with readinto on a contiguous writable memoryview, and rejects a returned length other than the requested size. set and frozenset dumps write elements in sorted(v, key=marshal.dumps) order. Assisted-by: Grok:grok-4.6 * Add classmethod_descriptor and ABC IS_ABSTRACT flags Give METH_CLASS methods a distinct classmethod_descriptor type. Store Py_TPFLAGS_IS_ABSTRACT from __abstractmethods__ and look up that attribute only in the type's own dict. Expose defaultdict.default_factory as a member descriptor. Inherit bool's unused number slots from int. Mark set and frozenset __contains__ as METH_COEXIST. Parse classmethod, staticmethod, object.__new__, and heap type text signatures. Register interactive source in linecache. Assisted-by: Grok:grok-4.6 * address review: odict reuse, descr call, new wrapper Reuse odict node slots and walk the live list in next_after. Require both | operands to be dicts. Drop the odict iterator on GC clear. Bind classmethod descriptors through descr_get and update IS_ABSTRACT under the type lock. Give odict keys and items sq_contains. Walk tp_new_wrapper past new_wrapper so HEAPTYPE socket subclasses can call socket.__new__. Split FreeBSD Bluetooth protocol constants from Linux values. Assisted-by: Grok:grok-4.6 * Represent chflags bits with bitflagset Assisted-by: Grok:grok-4.6 * Represent access and mode bits with bitflagset Assisted-by: Grok:grok-4.6 * Continue unfinished REPL suites as incomplete input Assisted-by: Grok:grok-4.6 * Add pickle and os.stat inspect signatures Give _pickle.Pickler the clinic constructor text signature and accept fix_imports and buffer_callback as positional-or-keyword. Allow #[pyfunction(text_signature)] and use it so os.stat reports (path, *, dir_fd=None, follow_symlinks=True). Uncomment import _pickle and drop the pickle inspect expectedFailures. Assisted-by: Grok:grok-4.6 * Add Unpickler, dump/load, and lstat signatures Give _pickle Unpickler, dump, dumps, load, and loads clinic text signatures, and os.lstat (path, *, dir_fd=None). Assisted-by: Grok:grok-4.6 * Complete OrderedDict with insertion-order links Move the native OrderedDict into ordered_dict.rs and track order with a linked list of nodes, keeping mapping-view set operations. Assisted-by: Grok:grok-4.6 * address CI: stat casts, HCI_DATA_DIR, passing xfails Compare stat permission bits as Mode so Linux clippy does not flag a u32-to-u32 cast. Export HCI_DATA_DIR with the Bluetooth socket option values. Drop expectedFailure from tests that now pass. Assisted-by: Grok:grok-4.6 * Fix wasm teardown deadlock and bluetooth constants Assisted-by: Grok:grok-4.6 * Clear process-wide eval breaker bits in STW test Assisted-by: Grok:grok-4.6 * Drop exclusive-held panic that tripped wasm clippy Assisted-by: Grok:grok-4.6 * Fix OrderedDict contains and export Windows BDADDR Assisted-by: Grok:grok-4.6 * Add cmp_to_key and hash-once LRU cache _functools.cmp_to_key now returns KeyWrapper. LRU lookups store the key hash on _lru_list_elem so __hash__ is not called again on hits or eviction. Assisted-by: Grok:grok-4.6 * Allow subclass keywords on list, tuple, and float list.__init__ rejects keywords only when tp_new is still list's. tuple/float.__new__ reject them only when tp_init is still the base's. Assisted-by: Grok:grok-4.6 * Fix CI lint, test re-export, and structseq members Drop the duplicate clear_eval_breaker_for_test re-export. Format structseq imports. Expose struct sequence fields as readonly members so assignment raises "readonly attribute". Assisted-by: Grok:grok-4.6 * Allow subclass keywords on itertools and filter Positional-only constructors ignore extra keywords when a subclass has replaced tp_init, matching list/tuple/float. Assisted-by: Grok:grok-4.6 * Remove text_signature derive parameter Drop the string override on #[pyfunction] and #[pyclass]. Signatures are generated from the Rust parameter list only. Restore inspect expectedFailures that depended on the override. Assisted-by: Grok:grok-4.6 * Rename set contains_method to __contains__ Keep a borrowed contains helper for slots and C-API. The coexist pymethod is named __contains__. Assisted-by: Grok:grok-4.6 * Use MonitoringEvent instead of EVENT_* aliases Drop the unused MonitoringEvents bitflags and the EVENT_PY_START style mask constants. Call sites use the enum and .mask(). Assisted-by: Grok:grok-4.6 * Store warnings filters as PyListRef The list already serializes its elements. Drop the extra mutex around the cached list pointer. Assisted-by: Grok:grok-4.6 * Re-export clear_eval_breaker_for_test after rebase Keep the STW test helper visible without bringing back stop_bit_set. Assisted-by: Grok:grok-4.6 * Fix groupby reentrancy and islice arguments Give _grouper its own tgtkey and take currvalue after the key comparison. Accept __index__ on islice bounds, reject step 0, and drain the source up to stop. Assisted-by: Grok:grok-4.6 * SETREF warnings filters, onceregistry, and defaultaction Keep the last list, dict, or str after the module attributes are deleted by swapping the interpreter cache pointer. Do not store a None fallback for _warnings_context. Assisted-by: Grok:grok-4.6 * Iterate OrderedDict inner dict via next_entry &PyDict is not an iterator after the borrowed-ref signature change. Assisted-by: Grok:grok-4.6 * Remove expectedFailure from testBluetoothConstants Linux now exports BTPROTO_HCI and the rest of the set. Assisted-by: Grok:grok-4.6 * Borrow get_filter arguments as PyObject Match the warn_explicit call site that passes borrowed handles after the internal-helper borrow change. Assisted-by: Grok:grok-4.6 * Fix bool slots and SemHandle safety docs Leave unused bool number slots unset so inspect reports __add__ on int. Document SemHandle::from_raw safety. Assisted-by: Grok:grok-4.6 * Keep sendmsg errors and drop duplicate linecache Replace sendmsg iterable TypeError only. Register REPL source in linecache once per compiled input. Assisted-by: Grok:grok-4.6 * Nest REPL parse or-patterns for clippy Assisted-by: Grok:grok-4.6 * Mark pydoc os.stat signature test expectedFailure Assisted-by: Grok:grok-4.6 --------- Co-authored-by: Lee Dogeon <dev.moreal@gmail.com> |
||
|
Jeong, YunWon
|
30ff0329a2 |
Borrow handles in internal helpers that only borrowed them (#8798)
Internal functions that took PyObjectRef or PyRef<T> by value but only borrowed it (reported by clippy::needless_pass_by_value) now take &PyObject or &Py<T>. Callers pass a borrow instead of an owned handle; 61 caller-side .clone()/.to_owned() calls made only to pass the argument are removed. Where a converted body already cloned the parameter, that .clone() became .to_owned(). Functions with #[py*] attributes, trait methods, and functions whose signature is fixed by a function pointer, getset or slot wrapper type are unchanged. PyFloat_FromString in crates/capi passes the borrowed object to parse_float_from_string without taking an owned reference. Assisted-by: Claude Code:claude-opus-5-5 Assisted-by: Grok CLI:grok-4.7 |
||
|
Bas Schoenmaeckers
|
15c9bdd48f |
Impl PyBytes_FromString PyBytes_FromObject PyBytes_AsStringAndSize (#8795)
* Impl `PyBytes_FromString` `PyBytes_FromObject` `PyBytes_AsStringAndSize` * Do not copy in PyBytes_FromObject when already a bytes obj |
||
|
Bas Schoenmaeckers
|
43a595ed65 |
Add SeqIter and CallIter support to c-api (#8794)
|
||
|
Jeong, YunWon
|
803520930a |
capi: add PyFrame_GetLocals, GetGlobals, GetBuiltins (#8761)
Wire the C-API getters to frame.f_locals, f_globals, and f_builtins, and cover write-through vs snapshot locals in a capi test. Assisted-by: Grok:4.6 |
||
|
Jeong, YunWon
|
7bc3ef6c0a |
ExceptionGroup fields, OSError use-init, and live f_lineno (#8738)
* Add ExceptionGroup fields and OSError use-init BaseExceptionGroup stores msg, excs, and excs_str; args keep the original sequence. OSError defers parse when a subclass overrides only __init__. characters_written can be deleted. Exception args/traceback/cause/context reject invalid setattr/delattr. UnicodeError.object is a deletable field. except* remainder projection copies traceback/context/cause. SyntaxError.__init__ clears omitted location attributes. Assisted-by: Grok:grok-4.6 * Fix OSError payload init and UnicodeError construction new_payload_exception now runs slot_new so BlockingIOError parses characters_written. new_exception falls back to invoke_exception for extra-payload types. Mixed tab/space UnexpectedIndentation is TabError. AttributeError pickle omits obj. Assisted-by: Grok:grok-4.6 * address review: classify TabError from error line Assisted-by: Grok:grok-4.6 * Add new_simple_exception; fill SyntaxError and f_lineno new_exception constructs extra-payload types; BaseException-sized types use new_simple_exception. Non-UTF-8 SyntaxError sets location and text. Codegen errors load text via ProgramText. Keep Missing parentheses capitalization. f-string adjacent atoms report a missing comma. f_lineno may be None; lasti is not restored on unwind. Assisted-by: Grok:grok-4.6 * address review: skip f-string continuation keywords Assisted-by: Grok:grok-4.6 * Fix clippy map_unwrap_or in Non-UTF-8 offset Assisted-by: Grok:grok-4.6 * Rename encode error ctor and exception setters new_unicode_encode_error_real is new_unicode_encode_error. set___traceback__/__cause__/__context__ are pygetset setters; Rust uses set_traceback/set_cause/set_context. Assisted-by: Grok:grok-4.6 * Use set_context in native pickle Assisted-by: Grok:grok-4.6 * Collapse nested if in asyncio traceback restore Assisted-by: Grok:grok-4.6 * Use current instruction for f_lineno addr2line lasti is stored as the next instruction index. addr2line of that byte offset is past RETURN_VALUE, so f_lineno was None and stdlib_threading_current_frames hung on leftover workers. Assisted-by: Grok:grok-4.6 * Look up format args, dir class, and frame builtins Consume printf args before rejecting unknown conversions. object.__dir__ uses getattr of __class__/__dict__. _PyEval_GetBuiltin looks up iter/__import__ on the frame. Assisted-by: Grok:grok-4.6 * Fix builtin lookups after rebase and unxfail passing tests Assisted-by: Grok:grok-4.6 * Restore RERAISE lasti and SyntaxError locations RERAISE with oparg restores lasti from the exception table before unwind, so leftover f_lineno matches the original raise. Handler lookup still uses the raising instruction. Indentation errors outrank print/exec missing-parentheses. BOM-with-non-utf8, bytes literals, and global+nonlocal now report the same offsets as the parser/symtable. object.__dir__ keeps string keys; type.__type_params__ skips the getset wrapper stored on builtin types. Assisted-by: Grok:grok-4.6 * Unxfail syntax/import tests and bind jump locals IMPORT_NAME passes fromlist None instead of an empty tuple. Jumping f_lineno binds None into unbound localsplus slots. str lstrip/rstrip copy subclasses to exact str when unchanged. Assisted-by: Grok:grok-4.6 * Fix import_from callers and rename frame lineno sqlite3.iterdump now passes fromlist as a PyObject. FrameObject::lineno is the i32 helper; f_lineno is the getset. Assisted-by: Grok:grok-4.6 * Pass sqlite fromlist without Into conversion from_list.into() could not infer Into<PyObjectRef> at the import_from call. The typed tuple already converts. Assisted-by: Grok:grok-4.6 |
||
|
Bas Schoenmaeckers
|
da694300b8 |
Add Py_IncRef, Py_DecRef, Py_XNewRef (#8726)
|
||
|
Bas Schoenmaeckers
|
dacf514210 |
Add long export/writer to c-api (#8161)
* Add long export/writer c-api * Add `PyLong_AsNativeBytes` * Fix PyLongWriter_Create errors * Set PyLongWriter_Create digits to null on err |
||
|
Roshan Ramani
|
1345a82443 |
Report bytes out-of-range elements as CPython does (#8679)
* Report `bytes` out-of-range elements as CPython does `collect_bytes` is shared by the `bytes` and `bytearray` entry points and hardcoded one element error, so `bytes([256])` said "byte must be in range(0, 256)" where CPython says "bytes". Every `bytearray` path does say "byte", so only the one caller was wrong. The element error is now a parameter, alongside the type error this function already varies per caller. Verified against CPython 3.14 for all seven paths. Assisted-by: Claude Code:claude-opus-5 * Keep the bytearray wording in PyByteArray_FromObject It called `bytes_from_object`, so the previous commit made it report "bytes must be in range(0, 256)" where CPython's `PyByteArray_FromObject` says "byte". It is the C-level bytearray constructor, so it now uses `bytearray_from_object`, which carries that wording and the constructor's unsized iteration. Assisted-by: Claude Code:claude-opus-5 |
||
|
Jeong, YunWon
|
2c8d3ce50b |
Implement PEP 667 frame locals semantics (#8602)
* Implement PEP 667 frame locals semantics Assisted-by: Codex:GPT-5 * Fix C API test CString import Assisted-by: Codex:GPT-5 * Address PEP 667 review findings Assisted-by: Codex:GPT-5 |
||
|
Jeong, YunWon
|
819b4b86fd |
Fix malformed Unicode error constructors (#8497)
* Fix malformed Unicode error constructors Replace message-only Unicode decode and encode errors with fully initialized exceptions, preserve source objects and failure ranges, and remove the obsolete constructors. Assisted-by: Codex:gpt-5 * Fix Windows Unicode CI regressions Assisted-by: Codex:gpt-5 |
||
|
Jeong, YunWon
|
11de045ef8 |
Per-interpreter runtime state and a process-wide GC stop-the-world (#8517)
* vm: add per-interpreter runtime state and interpreter registry - vm/runtime.rs: process-global interpreter registry (monotonic ids, weak entries), InterpreterWhence/InterpreterInfo, process main id recording via main_interpreter_id(), a threading-gated owner map (store_owned_interpreter/ take_owned_interpreter/is_owned_interpreter/owned_interpreter_count), and the SUPPORTS_ISOLATED_INTERPRETERS constant. - PyGlobalState gains interpreter_id/whence/is_main and is_main_interpreter(); PyConfig/Settings derive Clone so a subinterpreter can clone parent config. - Interpreter: id()/whence()/is_main()/is_process_main(), create_subinterpreter() and create_owned_subinterpreter(); unregister on Drop. - thread.rs: per-interpreter thread slots (INTERP_THREAD_SLOTS), slot swap when switching interpreters on one OS thread, cleanup keyed by interpreter id. - Install signal handlers and init the main-thread ident only on the main interpreter; _thread._is_main_interpreter reflects the current interpreter. - sys.implementation.supports_isolated_interpreters reads the constant. - Guard the registry static for non-threading builds where rc::Weak is !Send. Assisted-by: Claude Code:claude-opus-4-8 * gc: stop every interpreter while collecting The generation lists are process-global, so a collection reads and frees objects owned by every interpreter. CollectStopTheWorld stopped only the collecting interpreter, leaving other interpreters' threads free to mutate the same object graph during the reference-subtraction, reachability and snapshot phases. Stop all live interpreters instead, in runtime id order, and restart them in reverse. The global `collecting` mutex serializes collectors process-wide, so no second collector takes these exclusions in another order; fork acquires a single interpreter's exclusion, so the orders cannot cycle. - runtime: add live_interpreter_states(), ordered by interpreter id. - StopTheWorldState methods take &PyGlobalState instead of &VirtualMachine, so an interpreter's world can be stopped without a VM for it; update the call sites in frame, _thread, posix, faulthandler and capi. - Document in gc_state() that the collector is process-wide: gc.disable(), thresholds, gc.garbage and gc.get_objects() observe process-wide state, and a per-interpreter collector additionally needs untrack_object (called from default_dealloc with no VM in scope) routed to the owning interpreter. Tests: stop_the_world_parks_threads_of_another_interpreter asserts a thread entered in one interpreter parks another interpreter's threads (it fails without this change), plus a collect-while-another-interpreter-churns test. Assisted-by: Claude Code:claude-opus-4-8 * vm: scope the interpreter registry per thread without threading The registry held `rc::Weak` in a process-global `OnceLock` and covered the resulting `!Send`/`!Sync` with an `unsafe impl` justifying it as "non-threading builds are single-threaded". That is not this codebase's model: `static_cell!` is thread-local without the `threading` feature precisely so each OS thread can own its own `Context::genesis()` and `GcState`, so two threads could reach the same `Rc` counts through the registry. Use `static_cell!` for the registry as well, matching `gc_state()`, and drop the `unsafe impl`. Ids are consequently unique per registry rather than per process in non-threading builds, which is documented on `alloc_interpreter_id`. Move the recorded main interpreter id into the registry so it follows the same scoping instead of living in a separate global `OnceLock`. Also make `CollectStopTheWorld::new` accumulate into a live guard: it built a bare `Vec` and only moved it into the restarting `Drop` type after stopping every interpreter, so an unwind partway through the loop left the already stopped interpreters parked forever with their exclusion held. Assisted-by: Claude Code:claude-opus-4-8 * vm: keep the interpreter registry usable across bootstrap, drop and fork Three gaps where the registry did not describe the interpreters that actually exist, each of which hides an interpreter from the collector's stop-the-world. Register before `initialize()`. Registration ran as the last step of `initialize_vm`, so the whole bootstrap — which executes Python bytecode and allocates GC-tracked objects — was invisible to `live_interpreter_states()`. It still cannot run any earlier than this: the init hooks take `PyRc::get_mut` on the state, which fails as soon as the registry holds a weak reference to it. Stop unregistering in `Interpreter::drop`. The handle does not decide the interpreter's lifetime — every `ThreadedVirtualMachine` from `new_thread()` holds its own `PyRc<PyGlobalState>` — so an interpreter with running workers disappeared from the registry while its threads kept mutating the object graph. The entries are weak, so lifetime is already correct without the removal; dead entries are now reaped when registering instead. Interpreters are consequently released rather than unregistered at a fixed point, so the two tests asserting disappearance now wait for it: a collection in progress legitimately holds a reference to every live interpreter. Repair other interpreters after fork. `py_os_after_fork_child` only fixed the forking interpreter, leaving every other one with slots for threads that did not survive (still ATTACHED if they were running bytecode) plus locks and stop-the-world flags held by them. Since a collection stops all interpreters, the child's first collection would wait for threads that no longer exist. Reset their locks, stop-the-world state and thread tables, drop this thread's cached slots for them, and reinit the registry's own locks first, since enumerating interpreters now takes them. Tests: test_gc, test_threading and test_fork1 pass, as do the vm tests in both the threading and default configurations. Assisted-by: Claude Code:claude-opus-4-8 * vm: attach and detach thread slots when switching interpreters `enter_vm` decided whether to attach from `was_outermost` (an empty VM_STACK), which held while a thread could only ever be in one interpreter. With a slot per (thread, interpreter) pair, entering interpreter B from a thread already inside interpreter A's section switched CURRENT_THREAD_SLOT to B's slot but attached nothing: the thread then ran B's bytecode with B's slot DETACHED while A's slot stayed ATTACHED. A collector stopping B force-parks the DETACHED slot and concludes B is stopped, and then walks the object graph this thread is still mutating. Pair the attach/detach with the slot switch instead (≈ `_PyThreadState_Swap`): `begin_interpreter_section` detaches the enclosing interpreter's slot, makes the target slot current and attaches it, and `end_interpreter_section` undoes that and re-attaches the enclosing interpreter. Both live in `set_current_vm`, which every path making a VM current already goes through, so `enter_vm` and `VmBootstrapGuard` no longer track outermost-ness themselves. `nested_enter_of_subinterpreter_is_stoppable` covers this: it runs a subinterpreter nested inside the parent's section and asserts the sub's threads park when the sub's world is stopped. It fails with the previous attach-at-outermost-only behavior. Assisted-by: Claude Code:claude-opus-4-8 * vm: list only the current interpreter's subclasses Interpreters share the context, so `class Foo(int)` in one of them pushes onto the same `int.subclasses` every other one reads, and `int.__subclasses__()` returned types no other interpreter can reach. Record the creating interpreter on `HeapTypeExt` and filter `__subclasses__` by it, the way `lookup_tp_subclasses` reads `tp_subclasses` out of per-interpreter state for static builtin types. Types built before any interpreter exists — the ones the shared context creates, including the exception hierarchy — carry no id and stay visible to every interpreter, which is what `_PyStaticType_InitBuiltin` produces by registering the builtin subclass links once per interpreter. The other walks over `subclasses` (version-tag invalidation, abc flag propagation, mro updates, slot propagation) are left as they are: each starts from a type being mutated, so from a heap type, whose subclasses all live in the interpreter that created it. `subinterpreter_subclasses_are_scoped_to_their_interpreter` covers this and fails without the filter. Assisted-by: Claude Code:claude-opus-5 * gc: give each interpreter its own collector state The generation lists stay process-wide, because an object is untracked from `default_dealloc`, where no interpreter is in scope to route to. What changes is that a collection no longer acts on every interpreter's objects, and the gc module no longer reports one interpreter's state to another. `track_object` stamps the running interpreter into a new `gc_owner` word on the object header, and a collection takes as candidates only the objects carrying its own tag plus the ones carrying none. `gc_owner` fits in the padding the header's alignment already forces, so objects do not grow; an assertion on the header size keeps it that way. Objects allocated with no interpreter running — everything the shared context builds — carry no owner and stay candidates for every interpreter, which is where they were before. Objects that outlive the interpreter that tracked them are adopted the same way by the next full collection, rather than being left to a collector that will never come. `enabled`, the thresholds, the debug flags, the statistics, `gc.garbage` and `gc.callbacks` move onto `PyGlobalState` — the last two off the shared `Context` — so `gc.disable()`, `gc.set_threshold()`, `gc.get_stats()`, `gc.get_objects()` and `gc.garbage` describe the interpreter that asks. The occupancy counts behind `gc.get_count()` and `gc.get_freeze_count()` stay process-wide: they measure how full the shared lists are. Their decrements are saturating now, since a collection zeroes the generations it emptied while another interpreter's objects are still sitting in them. Stop-the-world still stops every interpreter. Unowned objects are candidates and any interpreter can incref one, so the refcounts a collection reads are only stable while all of them are parked. This also fixes a deadlock it exposed: `CollectStopTheWorld` dropped its references to the stopped interpreters while the collection still held the generation read locks, so releasing the last reference to one — which frees its objects, and so untracks them — waited for a write lock behind that read lock. The references are now held until the guard itself drops. `collections_only_reach_the_collecting_interpreter` and `get_objects_only_reports_the_calling_interpreter` cover this and both fail without the owner check. Assisted-by: Claude Code:claude-opus-5 * _queue, _thread: detach before taking locks held across waits `_queue.Semaphore` holds its mutex across the `allow_threads` condvar wait, and `join_internal` holds a thread handle's completion mutex the same way. Stop-the-world can stop a thread while it holds either one. The remaining acquisitions ran attached, so a thread blocking on such a mutex had no safepoint left to reach: the stop never completed, and the holder was never resumed to release it. Route those acquisitions through helpers that detach first. The fork-child reinit paths keep their direct locks. Assisted-by: Claude * _io, _winapi: detach on the remaining stopped-holdable lock takes `TextIOWrapper.__repr__` took `data` directly while every other method takes it through `lock_opt`, which detaches. `Overlapped` holds `inner` across the `allow_threads` in `GetOverlappedResult`, and all four of its takes were direct. A thread stopped by stop-the-world can be holding either mutex, so taking one while attached left the blocked thread with no safepoint to reach. Assisted-by: Claude * gc: size the interpreter owner tag to the header padding `gc_owner` was a u32. With 4-byte pointers its alignment pushed it out of the padding that follows the gc bits and generation, growing every object by a word and tripping the `SIZEOF_PYOBJECT_HEAD` assertion on 32-bit targets. Introduce `GcOwner = u16`, which the `repr(C)` layout places at offset 10 on 32-bit and 18 on 64-bit, leaving the header at 6 words on both. Tags now run out after 65535 interpreters; `alloc_owner` already falls back to `GC_NO_OWNER`, so an interpreter past that collects as it did before tagging. Also widen three test deadlines that measure liveness, not speed. Assisted-by: Claude * vm: reuse cleared frame blocks and shorten interpreter hot paths - `datastack` remembers the most recently popped frame block. `push_frame` reports an exact LIFO reuse, and `setup_datastack_frame` then skips zero-filling localsplus. - Small-int loads push the context's cached int as a borrowed stack reference instead of taking a new one. - Calls to jitted functions go straight to `execute_call_vectorcall`. - Binary-op specialization reads ints through the new `PyInt::try_to_i64_fast` instead of the generic primitive conversion, and `try_to_bool` moves its non-bool path into a `#[cold]` helper. - Dict caches read an entry through a keys-version stamp (`get_index_if_keys_version`) rather than an entry-index hint. - Frame publishing caches a pointer to `ThreadSlot::top_iframe` in a thread-local `Cell` instead of borrowing `CURRENT_THREAD_SLOT`. * gc, vm: address review notes on owner tags and interpreter docs Look up retired owner tags with a sorted binary search instead of a linear scan, which every scanned object paid for once per dropped interpreter. Drop the claim that clearing a tag frees it for reuse; `alloc_owner` only ever hands out new tags. Also correct the tag space it mentions, which is 16-bit since the tag was sized to the header padding. Document `is_main` as "top-level interpreter" rather than "the process main": every top-level interpreter sets it, and only the first registered one becomes the main `main_interpreter_id` reports. Assert membership rather than an exact owned-interpreter count delta; the owned table is process-global and other tests store into it in parallel. Assisted-by: Claude * vm: gate interpreter registration on stop-the-world admission A collection snapshots the registry, stops the interpreters it found, and then reads tracked objects with their threads parked. An interpreter that registered after the snapshot was taken was absent from it, so nothing stopped it and its bootstrap ran Python — allocating and mutating the shared generation lists — underneath that scan. Registration and the stop now share a process-global gate: the collection holds it from before the snapshot until the restart, and registration takes it around the registry insert. The insert runs detached, since a thread that waited for the gate, or re-attached while holding it, would leave the stop it waits for no safepoint to complete at. Assisted-by: Claude * gc: keep the tracking counters off the barrier path Every tracked allocation and every free went through sequentially consistent counter updates, and each free through a `fetch_update` CAS loop. The counters drive only the gen0 threshold and `gc.get_count()`, and the generation locks — not the counters — order the list changes they describe, so they are relaxed now and the decrement is a load plus a conditional `fetch_sub`. `is_enabled` and `threshold`, both read once per allocation, are relaxed for the same reason: an allocation racing `gc.disable()` may use either value. Drop `alloc_count`, which nothing has ever read. Assisted-by: Claude * dict: settle lookups and iteration steps under one read guard A lookup probed under a read guard, dropped it with the matched entry in hand, and then took the lock again to re-find the entry. `lookup_extract` reads the entry while the probe still holds the guard when key identity settles the match, which is the case that cannot run Python. Dict and set iterators took one lock to compare the size and another to read the entry, then cloned both the key and the value even though a keys or values view keeps only one of them. `next_entry_checked` does the size check and the read under one guard, and clones through a per-view projection. A store that missed its inline-cache hint re-probed the dict afterwards only to recover the entry index the store had just computed; `unchecked_push` reports that index instead. Assisted-by: Claude * vm: build a call's argument vector once The specialized CALL handlers collected the positional arguments into one vector and then copied them into a second one to put `self` in front, so every specialized builtin, method-descriptor, class and non-Python call paid two allocations, a copy and two frees. The arguments are already laid out on the value stack in vectorcall order, so `take_call_args` fills one vector by index — the shape `execute_call_vectorcall` already used. `vectorcall_native_function` and the keyword path of `vectorcall_function` then cloned that vector again to build `FuncArgs`; both now move it in through `from_vectorcall_owned`, as the other vectorcall slots do. Assisted-by: Claude * vm: reach an instance dict without cloning it The specialized attribute instructions cloned the instance dict — a rwlock round-trip plus a refcount round-trip — for two things that only look at it: `LoadAttrMethodLazyDict` asking whether the dict exists, and the keys-version stamp check that is the whole of `shadowing_instance_attr`'s fast path. Both now read it borrowed, through `has_instance_dict` / `with_instance_dict`. `generic_getattr_opt` probed the dict with the name's `&Wtf8`, which hashes the name on every lookup and can never match a key by pointer. Passing the `Py<PyStr>` uses the string's cached hash and the interned-key identity check, and drops an allocation when a stored key is not an exact `str`. Assisted-by: Claude * vm: shorten the per-instruction safepoint and the call preamble The dispatch loop asks once per instruction whether stop-the-world wants this thread, and that read went through `CURRENT_THREAD_SLOT` — a `RefCell` borrow, so two stores to thread-local memory around an `Option` test. Cache the `stop_requested` pointer in a plain `Cell` at the same three places the frame pointers are cached, and the safepoint becomes one relaxed load. `lasti` is advanced from the index the loop just read, rather than reloaded to increment it, and `Resume` reads `quickened` before swapping it, so a call to an already-quickened code object costs a load instead of an atomic read-modify-write. An exact-args vectorcall to a Python function built a heap `FrameObject` where the equivalent `invoke` path uses a data stack frame; it now does the same when tracing is off. Assisted-by: Claude * vm: give KwArgs a zero-sized hasher `KwArgs::default()` is built for every call, keyword-less ones included, and it seeded a `RandomState` each time — a thread-local read and 16 bytes in every `FuncArgs`. Keyword names come from the program text, so the map now uses `BuildHasherDefault`, whose `Default` is a zero-init. Assisted-by: Claude * vm: promote borrowed stack refs before a yield `yield <small int>` tripped the "borrowed refs on stack at yield point" assertion: `LoadSmallInt` pushes a borrowed ref, and a yield saves the stack with the frame, which is exactly what that assertion forbids. Promote the stack first, so a suspended frame owns everything it holds. Assisted-by: Claude * vm: run the exact-args vectorcall on a heap frame again `vectorcall_function` took the data stack path for an exact-args call whenever tracing was off. A data stack frame that materializes — one traceback entry is enough — copies its localsplus into the `FrameObject`, so every value it holds gains a reference for as long as the frame runs, and a tail call holds the callee's function object until the caller's next call or return. Both are reachable without this hunk, through the `CallPyExactArgs` specialization, but routing the vectorcall entry point through the data stack made them apply from a call site's first execution: test_enumerate, test_memoryio, test_sys, test_tempfile and test_traceback all read a refcount or a finalization that moved. Assisted-by: Claude * vm: release a tail call's function at the callee's return The trampoline moved the callee's function object into the caller's `SuspendedFrame` and dropped it only once the caller had run again, so a function entered through a tail call kept one reference past its return: `sys.getrefcount` on it read one high for the caller's next stretch of bytecode, and anything the function reached stayed alive that much longer. The callee's frame is released before the `ReturnValue` or `Unwind` action is formed, and a materialized frame object holds its own references, so the owner is dropped where the suspended caller is popped instead. Assisted-by: Claude * vm: stop copying a running frame's locals when it materializes `materialize_slow` cloned the source frame's fast locals into the new `FrameObject`, so every value in a data stack frame gained a reference lasting as long as that frame object: one traceback entry through a frame kept all of its locals alive, and `sys.getrefcount` read one high for each of them. Nothing read that copy while the frame ran. `find_live_source_iframe` already routes reads to the live frame, and `exit_iframe` fills the slots from it at return, so the copy now starts empty as `materialize_slow_chain`'s does, and `snapshot_to_heap` goes with its last caller. Two readers had to be brought to the live frame the same way. `has_active_hidden_locals` read the frame object's own slots, which let `locals()` in a class body write an inlined comprehension's hidden variable into the class namespace; it now shares `live_fastlocals` with the other two. Another thread's frame has no live frame to reach from here, so the six cross-thread sites fill the copy under stop-the-world through `materialize_with_locals`. Assisted-by: Claude * vm: key data stack frame reuse on the exact frame size push_frame and pop_frame compared 16-byte-aligned sizes, so two frames whose sizes differ by less than the alignment matched as a reuse. The caller reads that as "every slot was cleared by the previous frame", but the larger frame's tail slots were never touched by it. Assisted-by: Claude * vm: store the frame object payload address in the cross-thread slot set_current_frame and reinit_frame_slot_after_fork wrote the address of the Py<FrameObject>, while the cross-thread readers hand that pointer to Py::from_payload_ptr, which subtracts the object header from it. Both writers now store the payload address. Assisted-by: Claude * vm: copy a foreign thread's frames instead of linking them sys._current_frames(), _thread._current_exceptions() and the cross-thread f_back path handed back frame objects still linked to the running frames. The owning thread wrote its locals into those same buffers at exit_iframe while the reader was cloning out of them. materialize_detached and materialize_detached_chain copy the values under stop-the-world and leave nothing for the owner to write into; materialize_with_locals is gone. FrameColdData gains attached_tid, the thread still running the frame a frame object was materialized from, which exit_iframe clears once it has written the values in. check_locals_access now rejects a read from any other thread while it is set; the FrameOwner test could not, because materialize_slow forces FrameObject ownership. Also removes with_iframe, which had no callers. Assisted-by: Claude * vm: resolve the two rustdoc links this branch added MAIN_INTERPRETER_ID is not in scope in interpreter.rs, and new_thread only exists in a threading build, so neither link resolved. Assisted-by: Claude |
||
|
Jeong, YunWon
|
d5d19e0611 |
Fix 33 reproduced fuzzing and static-review defects (#8514)
* mmap: guard move() against dest/src past the mapping size
An offset larger than the mapping made `size - dest` underflow, producing
an out-of-range slice index panic. Check the high side first, as write() does.
Assisted-by: Claude
* exceptions: keep ImportError name/path/name_from out of an empty instance dict
ImportError.__reduce__ unwrapped get_arg(0), which is None when the exception
was constructed with no positional argument, aborting on pickle.dumps(ImportError()).
Fall back to the full args tuple there, and expose name/path/name_from as class
attributes defaulting to None so a bare ImportError() reduces to (cls, ()).
Assisted-by: Claude
* asyncio: stop unwrapping the _current_tasks downcast in current_task()
_asyncio._current_tasks is a reassignable module attribute; the three sibling
functions already degrade gracefully when it is not a dict, current_task() did not.
Assisted-by: Claude
* asyncio: raise TypeError when FutureIter.throw()'s exception class returns a non-exception
exc_type.call() goes through a Python-controlled __new__, so its result can be
any object; the downcast was unwrapped. Report it as a TypeError instead.
Assisted-by: Claude
* sequence: use a fallible reservation for sequence repetition
The guard only rejects a repeat whose per-element size crosses MAX_MEMORY_SIZE,
so (1,) * (10**12) still reached Vec::with_capacity and aborted in the allocator.
Reserve fallibly and surface a MemoryError.
Assisted-by: Claude
* collections: reject an oversized deque repetition with MemoryError
deque * sys.maxsize reached the allocator and aborted with a capacity overflow.
Apply the MAX_MEMORY_SIZE guard the other sequences already carry.
Assisted-by: Claude
* itertools: raise OverflowError for an out-of-range r argument
combinations/combinations_with_replacement/permutations narrowed r with
to_usize().unwrap(), so r=2**64 panicked instead of raising.
Assisted-by: Claude
* math: stream the generic sumprod path instead of collecting both iterables
The big-int path collected each argument into a Vec before multiplying, so an
unbounded iterable exhausted memory. Advance both iterators in lockstep and
accumulate, reporting a length mismatch when only one is exhausted.
Assisted-by: Claude
* _imp: raise TypeError for a second positional argument to find_frozen
withdata is keyword-only and unimplemented; passing it positionally hit an
unimplemented!() and aborted.
Assisted-by: Claude
* _typing: check _idfunc arity before indexing args
_typing._idfunc() with no argument indexed args[0] out of bounds.
Assisted-by: Claude
* exceptions: require a sequence for the ExceptionGroup excs argument
The argument was collected before being validated, so an unbounded iterable
such as itertools.count() exhausted memory. Reject a non-sequence up front.
Assisted-by: Claude
* mmap: treat an inverted find/rfind range as empty
find(b"x", 5, 2) built a slice whose start exceeded its end and panicked.
Assisted-by: Claude
* collections: give deque and defaultdict a GC traverse
Neither type opted into traversal, so a reference cycle through a deque or a
defaultdict was never collected.
Assisted-by: Claude
* itertools: opt the iterator types into GC traverse
cycle and its siblings hold Python references but declared no traverse, so a
cycle built through one of them leaked.
Assisted-by: Claude
* _ctypes: mask an out-of-range int instead of panicking
c_char_p(2**64) and pointer item assignment called .expect() on the narrowing
conversion. Wrap the value to the target width, which is what the C
implementation stores.
Assisted-by: Claude
* hashlib: import _hashlib when a hash module is loaded
The hash object type is a static type owned by _hashlib; calling _md5.md5()
without _hashlib imported hit an uninitialized static type and panicked.
Assisted-by: Claude
* _csv: fall back to the built-in dialect defaults when none is registered
The dialect table is empty until csv.py registers 'excel', so _csv.reader([])
unwrapped a missing entry and panicked.
Assisted-by: Claude
* builtins: reject surrogates in compile()/eval() source instead of panicking
expect_str() panics on a str containing surrogates; convert with try_as_utf8
so eval(chr(0xd800)) raises.
Assisted-by: Claude
* _suggestions: require a list for _generate_suggestions candidates
The argument was collected before validation, so an unbounded iterable
exhausted memory.
Assisted-by: Claude
* lzma: size the filter chain before consuming it
filters= was collected into a Vec before the length check, so an unbounded
iterable exhausted memory. Take the length through the sequence protocol first.
Assisted-by: Claude
* classmethod: opt into GC traverse
staticmethod already declares traverse; classmethod did not, so a cycle through
the wrapped callable leaked.
Assisted-by: Claude
* posix: reject unbounded iterables in posix_spawn and setgroups
argv, setsigdef, setsigmask and setgroups bound an ArgIterable and collected it
before validating, so an infinite generator exhausted memory. Require a
list/tuple for argv, validate signals while streaming, and take setgroups
through the sequence protocol.
Assisted-by: Claude
* _ctypes: size Array slice assignment and _argtypes_ before collecting
Both eagerly collected their argument, so an unbounded iterable exhausted
memory before the length check ran.
Assisted-by: Claude
* sys: propagate the breakpointhook warning failure
warn() was unwrapped, so an unimportable $PYTHONBREAKPOINT under
-W error panicked instead of raising.
Assisted-by: Claude
* structseq: require the sequence argument when constructing a struct sequence
A no-argument construction produced an empty backing tuple, and reading any
named field then indexed out of bounds.
Assisted-by: Claude
* Guard the hash slot dispatch against unbounded recursion
`PyObject::hash` invoked the type's hash slot directly, so an element-wise
`__hash__` following a deeply nested object graph recursed one native frame
per level and overflowed the stack. Wrap the dispatch in `with_recursion`,
matching the repr and rich-compare dispatches in the same file, so
`hash(x)` on a nested tuple/GenericAlias/slice raises `RecursionError`.
Assisted-by: Claude
* genericalias: guard the __parameters__ walk against unbounded recursion
`make_parameters_from_slice` recursed into every list/tuple argument with
nothing counting the frames, so `list[L]` for a self-referential or deeply
nested `L` overflowed the native stack at subscript time. Wrap the descent
in `with_recursion`, which makes the walk fallible; `PyGenericAlias::new`,
`from_args` and `make_parameters` now return `PyResult` and every caller
propagates.
Assisted-by: Claude
* Fix the type confusion in PyAtomicRef's Debug impl
`PyAtomicRef<T>` stores a pointer to a `Py<T>`, as `Deref`, `load_raw`,
`swap` and `Drop` all read it, but `Debug` cast it to a bare `T` and
formatted the object header as payload bytes. For `PyFunction`, whose
`code: PyAtomicRef<PyCode>` has a pointer-chasing `Debug`, that
dereferenced header words and segfaulted.
Cast to `PyObject` instead, which is what `Drop` already does and which
also covers the `PyAtomicRef<PyObject>` and `PyAtomicRef<Option<T>>`
instantiations that have no `Py<T>`.
`_asyncio._enter_task` reached this through `{:?}` in its "Cannot enter
into task" message; format the two tasks with their Python repr, which is
what the message is meant to show.
Assisted-by: Claude
* _sre: disallow instantiating Match
`Match` inherited `object.__new__`, so `M.__new__(M)` produced an instance
whose `regs`/`string`/`pattern` were never filled in by a match run; the
mapping subscript path read them and dereferenced garbage. The type has no
public constructor, so mark it `DISALLOW_INSTANTIATION`, which makes
`M.__new__(M)` raise `TypeError: cannot create 're.Match' instances`.
Assisted-by: Claude
* utils: return the empty repr instead of asserting a non-empty collection
`collection_repr` took the first element with an `.expect()` justified by the
caller's preceding non-empty check. Another thread clearing the collection
between that check and the iteration made the iterator yield nothing and
panicked the worker. Fall back to the caller-supplied empty form, which is
the text those callers already produce for an empty collection.
Assisted-by: Claude
* itertools: advance cycle's index atomically
`cycle.__next__` did `fetch_add(1)` and then reset the index to 0 in a
separate store, so two threads replaying the saved items could both read an
index past the end of `saved` and panic on the slice access. Do the advance
and the wrap in one `fetch_update`.
Assisted-by: Claude
* _ctypes: reject a float argument to a foreign call without argtypes
`conv_param`, the conversion used when `argtypes` is not set, converted its
argument with `try_int`, which goes through `__int__` and so accepts a
float. `libc.strlen(1.5)` therefore passed 1 where the callee expects a
`char *` and the callee dereferenced it. Match `ConvParam`, which does a
`PyLong_Check` and converts nothing: take the branch only for an `int` (or
a subclass, so `True` still converts), and let a float fall through to
"Don't know how to convert parameter".
The branch below it converted a float to a C double, but `try_int` claimed
every float before it could run, so it was dead; `CArgValue::Double`
existed only for that branch and both go. Typed doubles are unaffected —
they travel as `CArgValue::Typed` with code 'd'.
Assisted-by: Claude
* Report the iterator itself from PyIter's traverse
`Traverse for PyIter<O>` delegated to the inherent `PyObject::traverse` of
the object it wraps, so it reported that iterator's referents instead of
the iterator. The iterator's own reference to those referents was then
never subtracted during the collector's reference-subtraction pass, the
referents kept a non-zero gc_refs, and every object reachable from them
was classified as a root.
Any cycle running through a type with a `PyIter` field therefore survived
collection: `map`, `filter`, `zip`, `enumerate`, `reversed` and the
`itertools` iterators all leaked, while the same cycle through a `list`,
`tuple` or `list_iterator` collected. Report the wrapped object, as the
`PyObjectRef`, `PyRef<T>` and `PyStackRef` impls do.
`itertools.tee` still leaks: its shared buffer is a `PyRc<PyItertoolsTeeData>`
rather than a Python object, so the collector cannot see through it.
Assisted-by: Claude
* Remove the obsolete expectedFailure on test_code_module.test_unicode_error
Compiling a source string containing a lone surrogate now raises
UnicodeEncodeError, so the test passes.
Assisted-by: Claude
* itertools: reserve the combination indices fallibly
`combinations` and `combinations_with_replacement` built their index vector
with an infallible allocation, so an `r` that passes the ssize_t check but
does not fit in memory aborted the process instead of raising MemoryError.
Assisted-by: Claude
* Apply the struct sequence constructor's dict argument
`structseq(sequence, dict)` discarded its second argument, so the hidden
fields past `n_sequence_fields` — `tm_zone`, `st_atime` and friends — were
always None when constructed directly or restored from a `(sequence, dict)`
pickle, and a non-dict second argument was accepted silently.
Take the dict, require it to be a dict, and fill the hidden slots the
sequence did not cover from it. A key that names a field the sequence
already supplied, or no field at all, is now a
"got duplicate or unexpected field name(s)" TypeError instead of being
dropped. Both arguments are bindable by name, as `sequence` and `dict`.
`os.stat_result` and `os.statvfs_result` did not accept a second argument
at all; they and `time.struct_time` now share the parsing.
Assisted-by: Claude
* _imp: report the argument count in find_frozen's arity error
Assisted-by: Claude
* Add regression tests for the reproduced crashers
One case per catalog entry, each asserting the behavior the fix produces:
recursion guards, the memory-unsafety sites, the overflow and unbounded
allocation guards, the eager-collection rejections, the unwrap sites, and
the cycles the collector now breaks. Every expected value was checked
against CPython 3.14.
Assisted-by: Claude
* Tolerate a changed-size RuntimeError in the set repr stress test
The mutator and reader threads race on purpose; a "changed size during
iteration" RuntimeError is a valid outcome of that race and should not fail
the test. The panic it guards against is not.
Assisted-by: Claude
* Move the crash regression tests into per-module snippets
crash_regressions.py collected every reproduced crasher in one file. Split it
into the snippet for the module each case exercises, and add stdlib_gc.py,
stdlib_asyncio.py, stdlib_lzma.py, stdlib_threading_set_repr.py and
stdlib_threading_itertools_cycle.py for the cases with no existing home.
The suite runs every snippet under the host CPython too, so the checks only
RustPython raises are guarded by sys.implementation.name: the hash and
__parameters__ recursion depth, and the deque repeat overflow. The float
ctypes argument accepts either TypeError or ctypes.ArgumentError.
Assisted-by: Claude
|
||
|
Jeong, YunWon
|
cdf006cfa1 |
Split InterpreterFrame from FrameObject for stack-allocated execution (#8354)
* Add LightFrame for unobserved Python-to-Python calls
Allocate a lightweight LightFrame header on the DataStack instead of a
full Frame PyObject for specialized exact-args call paths. The frame is
materialized lazily only when observed (sys._getframe, traceback, tracing).
Key changes:
- LightFrame struct with borrowed pointers and no refcount overhead
- FrameSource enum (Heavy/Light) replacing the object field on ExecutingFrame
- CURRENT_LIGHT_FRAME TLS for the light frame chain
- Interleaved heavy/light frame walking in frame_at_offset_vm and
current_thread_frame_vm
- invoke_light_slots on PyFunction for the fast call path
Call overhead reduced from ~92ns to ~58ns (~37% reduction).
fib(28) improved from ~152ms to ~111ms (~27% reduction).
Assisted-by: Claude
* Streamline with_frame: inline recursion check, skip tracing dispatch
- Inline check_recursive_call instead of wrapping in with_recursion closure
- Amortize C stack overflow check to every 64th call depth
- Skip dispatch_traced_frame when use_tracing is false (hot path)
- Add recursion_depth_increment/decrement helpers
Assisted-by: Claude
* Address review findings for LightFrame correctness
- Fix materialize_light_frame previous pointer: use payload pointer
(&**frame) instead of Py<Frame> pointer, matching the chain convention
- Fix predecessor ref management: use retained_back instead of
mem::forget to keep materialized predecessors alive
- Fix materialized ref reclaim: use FrameRef::from_raw to drop the
leaked keeper ref without double-incrementing
- Use PyAtomic<u32> for LightFrame.lasti instead of AtomicU32, removing
the unsound cast on non-threading targets
- Add amortized C stack overflow check to light frame path
- Add scopeguard to light frame execution for panic safety
- Remove unnecessary Send/Sync impls from LightFrame
- Make check_c_stack_overflow pub(crate) for light frame path access
Assisted-by: Claude
* Fix CI failures: cspell, stale materialized state, f_back chain gaps
- Fix cspell: 'amortised' → 'amortized'
- Sync lasti, prev_line, and fastlocals when re-observing a materialized
light frame, fixing stale f_locals/f_lineno in test_inspect and others
- In with_frame, materialize any active light frame as the heavy frame's
predecessor and store it in retained_back, so f_back and inspect.stack
see the correct interleaved order (H_new → L_mat → H_old)
- Guard invoke_light_slots on NEWLOCALS | OPTIMIZED code flags
Assisted-by: Claude
* Unify frame chain: replace dual-chain with single FrameChainPtr
Replace the dual-chain frame management (CURRENT_FRAME AtomicPtr<Frame>
+ CURRENT_LIGHT_FRAME Cell<*const LightFrame>) with a single unified
chain using FrameChainPtr, a tagged pointer where bit 0 distinguishes
heavy (*const Frame) from light (*const LightFrame) entries.
Changes:
- Add FrameChainPtr type with tagged pointer encoding
- Change LightFrame: replace previous_light + saved_current_frame with
single previous: FrameChainPtr field
- Change InterpreterFrame.previous from AtomicPtr<Frame> to PyAtomic<usize>
- Change CURRENT_FRAME TLS from AtomicPtr<Frame> to AtomicUsize
- Remove CURRENT_LIGHT_FRAME TLS entirely
- Simplify all chain walkers (frame_at_offset, frame_at_offset_vm,
find_owned_chain_frame, for_each_current_frame, current_thread_frame)
- Remove light-frame materialization block from with_frame
- Update f_back to dispatch on FrameChainPtr tags
- Update faulthandler and gc_state to walk unified chain
- Adaptive C stack check: every call in debug, every 16th in release
Assisted-by: Claude
* Fix light frame materialization: current_frame, f_back, and GC tracking
- current_frame() now uses current_thread_frame_vm() to materialize
light frames, fixing super()/compile()/exec() when called from light
frame context
- materialize_light_frame stores retained_back for heavy predecessors
too, so f_back resolves after the caller returns
- Escaped materialized frames (traceback, sys._getframe) are GC-tracked
at cleanup so reference cycles are collectible
- Restore pre-existing @expectedFailure markers for test_frame proxy tests
Assisted-by: Claude
* Limit light frame materialization to super() only
current_frame() returns heavy-only again to avoid unnecessary
materialization that extends object lifetimes. super() init uses
current_thread_frame_vm() directly since it must inspect the
light frame's code and freevars to find __class__.
Assisted-by: Claude
* Fix remaining CI failures: C stack check, GC tracking threshold, expectedFailure
- Check C stack overflow every call in light frame path (not every
16th) to prevent stack overflow on CI workers with smaller stacks
- Fix GC tracking threshold: escaped materialized frame detection
uses strong_count > 1 (was > 2, missed frames referenced only by
traceback in the result)
- current_frame() materializes light frames so warnings, compile(),
and other callers see the correct frame
- Mark test_futures2 PyFutureTests.test_task_exc_handler_correct_context
as expectedFailure (pre-existing on main: PyTask __del__ timing)
Assisted-by: Claude
* Avoid materialization in hot paths: add light-frame-aware accessors
current_frame() stays heavy-only to avoid costly materialization on
every call. Instead, add current_globals(), current_code(), and
current_builtins() that read directly from light frame raw pointers
without creating frame PyObjects.
Callers migrated:
- PyFunction:🆕 current_builtins() for fallback builtins lookup
- compile()/eval(): current_code() for future feature flags
- type.__new__: current_globals() for __module__ detection
- typevar/typing: current_globals() for caller module name
- _io: current_code() for source path check
- super(): current_thread_frame_vm() for __class__ cell access
- warnings.warn: current_thread_frame_vm() for stack level walk
- import: current_globals() for module resolution
Assisted-by: Claude
* Fix current_locals() to include light frames
current_locals() must materialize light frames so locals() builtin
returns the correct scope. This fixes test_zipimport doctest and
other tests that rely on locals() in light frame context.
Assisted-by: Claude
* Fix clippy, profile tracing, resume_gen_frame, and faulthandler issues
- Fix clippy: collapsible_if in materialize_light_frame, map_unwrap_or in import
- Fix profile breakage: use current_thread_frame_vm in trace_event_inner to
materialize light frames, preventing frame identity mismatch when profiling
is enabled inside a light frame (e.g. profile.Profile.runctx)
- Disable light frames inside trace/profile callbacks (tracing_is_suppressed)
- Fix resume_gen_frame: skip light frames in previous chain to avoid dangling
pointers after generator suspend
- Fix faulthandler: emit '<no Python frame>' when chain has only light frames
Assisted-by: Claude
* Fix materialized frame refcount: transfer localsplus ownership on exit
Remove @expectedFailure from test_futures2: the underlying issue was that
materialized light frames cloned localsplus values (refcount +1 each),
keeping objects alive until the next GC cycle collection.
Fix: after run_light_frame returns, transfer ownership of localsplus
values from the light frame to the materialized frame, replacing the
clones. This eliminates the extra refcount, allowing objects to be
GC'd immediately when their last reference is dropped.
Also stabilize materialized frame's localsplus on the heap in
sync_materialized_on_exit before GC tracking — the data-stack backing
will be popped imminently.
Assisted-by: Claude
* Fix rustfmt, increase stack margin, address review comments
- Fix rustfmt formatting issues (lint CI failure)
- Increase STACK_MARGIN_BYTES from 2048 to 3072 words to prevent C stack
overflow in deep recursion tests on macOS CI (test_functools SIGSEGV)
- Skip dispatch_traced_frame when use_tracing is false (review comment)
- Fix post-fork top_frame initialization: walk past light frames to find
nearest heavy frame (review comment)
Assisted-by: Claude
* Remove redundant test override in test_futures2
The override only called super() without any marker, making it a
no-op that prek flags as a redundant test patch.
Assisted-by: Claude
* Fix test_pdb and test_functools CI failures
- Revert with_frame dispatch_traced_frame skip: tracing can be enabled
mid-execution (pdb.set_trace), so Return events must always be checked.
This fixes 3 test_pdb doctest failures across all platforms.
- Check C stack overflow on every with_frame call instead of every 64th:
the previous sampling approach missed overflows between checks when
native stack frames are large (invoke_light_slots + lru_cache recursion).
- Add #[inline(never)] to invoke_light_slots to prevent the optimizer
from merging its stack frame into callers.
- Add early C stack check before DataStack allocation in invoke_light_slots.
Assisted-by: Claude
* Rename Frame to FrameObject, FrameRef to FrameObjectRef
Prepare for _PyInterpreterFrame/PyFrameObject separation by
renaming the PyObject-backed frame type to FrameObject.
Python-visible name ("frame") is unchanged.
Assisted-by: Claude
* Remove Deref<Target = InterpreterFrame> from FrameObject
Replace implicit field access through Deref with explicit
iframe()/iframe_ref()/iframe_mut() calls. This decouples
FrameObject from InterpreterFrame field layout, enabling
independent modification of InterpreterFrame in future commits.
Assisted-by: Claude
* Move InterpreterFrame identity fields to raw pointers
InterpreterFrame's code/globals/builtins/func_obj fields are now
borrowed raw pointers instead of owned PyRef/PyObjectRef. Ownership
of these references is anchored in FrameObject's new owned_code,
owned_globals, owned_builtins, owned_func_obj fields.
This aligns InterpreterFrame's layout with LightFrame's existing
raw-pointer pattern, enabling future DataStack-based allocation
without PyObject heap allocation.
- Add code()/globals()/builtins()/func_obj() accessor methods
- Add FrameObject::new_ref() for combined create+allocate+patch
- Update ExecutingFrame to use &Py<PyCode> / &Py<PyDict> / &PyObject
- Update monitoring.rs functions to take &Py<PyCode> instead of &PyRef<PyCode>
- Add Send+Sync impls for InterpreterFrame (raw pointers are !Send by default)
Assisted-by: Claude
* Remove LightFrame, FrameChainPtr, and dual-frame-type infrastructure
Delete LightFrame struct, FrameChainPtr tagged pointer, FrameSource enum,
and all supporting functions: materialize_light_frame, sync_light_to_materialized,
transfer_localsplus_to_materialized, sync_materialized_on_exit,
run_light_frame, invoke_light_slots body, LocalsPlus::from_datastack_raw.
All call paths now use a single frame type (FrameObject wrapping
InterpreterFrame). The frame chain is a simple *const FrameObject linked
list through InterpreterFrame.previous.
invoke_light_slots delegates to invoke_exact_args_slots.
ExecutingFrame.frame_source replaced with ExecutingFrame.frame.
Frame chain walking simplified throughout (no more light/heavy dispatch).
Net: -724 lines of dual-frame-type complexity.
Assisted-by: Claude
* Clean up invoke_light_slots and _vm function variants
- Replace invoke_light_slots calls with invoke_exact_args_slots
- Delete the invoke_light_slots delegate method
- Replace current_thread_frame_vm/frame_at_offset_vm with their
non-_vm equivalents (no more light frame materialization needed)
- Remove stale light-frame comment in callable.rs
Assisted-by: Claude
* Fix CI lint issues
- Move github context expressions from run blocks to env blocks
(zizmor/template-injection)
- Add workflow-level permissions: {} to lib-deps-check.yaml
- Suppress zizmor excessive-permissions for lib-deps-check.yaml
(pull_request_target is required for PR comments)
- Apply rustfmt formatting
Assisted-by: Claude
* Stack-allocate InterpreterFrame for non-generator function calls
- Extract InterpreterFrame::new() from FrameObject::new()
- Change frame chain from *const FrameObject to *const InterpreterFrame
- Add InterpreterFrame.materialized field for lazy FrameObject creation
- Add vm.with_iframe()/run_frame_fast() for stack-allocated frame execution
- Move ExecutingFrame.frame to iframe: *const InterpreterFrame
- Implement lazy materialize() for on-demand FrameObject creation
- Update invoke_with_locals/invoke_prepared_exact_args to skip heap allocation
- Update frame chain walking in faulthandler, gc_state, builtins/frame
- super() reads InterpreterFrame directly without materialization
Assisted-by: Claude
* Remove unnecessary refcount and atomic ops from fast call path
- Use raw pointer for func_obj instead of cloning PyObjectRef (saves 2 atomic RMW)
- Skip owner AcqRel swap for stack-allocated frames (always Thread-owned)
Assisted-by: Claude
* Change trace field to Option<PyObjectRef>, remove vm from InterpreterFrame::new
- trace: PyMutex<PyObjectRef> -> PyMutex<Option<PyObjectRef>>
- Eliminates vm.ctx.none() refcount inc/dec per frame init
- Remove vm parameter from InterpreterFrame::new() (no longer needed)
- f_trace getter returns None when trace is unset
- f_trace setter stores None instead of the None singleton
Assisted-by: Claude
* Store current frame pointer on VirtualMachine for fast lookup
- Add current_frame_ptr Cell<usize> field to VirtualMachine
- Hot path (with_iframe) reads/writes vm.current_frame_ptr instead of TLS
- TLS CURRENT_FRAME still maintained for signal-safe traceback walking
- Use set_current_frame_nosave() to skip cross-thread top_frame update
- super(), frame_at_offset, current_thread_frame_materialize use vm Cell
Assisted-by: Claude
* Apply rustfmt
Assisted-by: Claude
* Fix clippy use_self warnings, add inline hints for hot path
Assisted-by: Claude
* Remove code refcount clone from fast call path
Defer code.to_owned() to the generator/coroutine path only. On the fast
path, use &Py<PyCode> directly — the code object is alive via the
PyFunction on the caller's stack.
Assisted-by: Claude
* Remove VM current_frame_ptr Cell (fix SIGSEGV in threading builds)
The Cell<usize> on VirtualMachine could desynchronize from the TLS
CURRENT_FRAME in nested VM scenarios (enter_vm called with different
VMs on the same thread). Revert to TLS-only frame chain management.
Keep set_current_frame_nosave() for the stack-frame fast path: it
skips cross-thread top_frame publication but still does the TLS swap.
Fix clippy not_unsafe_ptr_arg_deref in set_current_frame.
Fix unnecessary Result wrapper in make_generator_or_coro.
Assisted-by: Claude
* Fix CI failures: GC frame assertion, faulthandler Radium import, dead code warning
- Fix SIGSEGV in threading builds: GC debug assertion walked frame
chain via top_frame, which is not updated by set_current_frame_nosave
used by stack-allocated iframes. Walk CURRENT_FRAME TLS chain instead.
- Add InterpreterFrame::get_lasti() accessor, remove direct Radium
import from faulthandler.rs (fixes unused import warning).
- Add cfg_attr allow(dead_code) for from_payload_ptr without threading
(fixes WASM build warning).
Assisted-by: Claude
* Fix materialized frame: copy localsplus, clear stale previous pointers
- materialize_slow: copy fastlocals snapshot to heap instead of empty
localsplus (fixes index-out-of-bounds panic in locals()/f_locals)
- materialize_slow: set previous to null (stack iframes become dangling)
- with_frame/resume_gen_frame: clear iframe.previous on pop (prevents
dangling pointers to freed stack iframes)
- f_back: walk TLS CURRENT_FRAME chain instead of top_frame for
cross-thread lookup (top_frame is stale for stack-allocated iframes)
- check_locals_access: match materialized frames by comparing the
Py<FrameObject> address stored in the stack iframe's materialized field
Assisted-by: Claude
* Fix Windows clippy: gate unix-only imports and variables
FrameObject, Py imports and top_iframe variable are only used in
unix+threading cfg blocks. Gate them properly for non-unix builds.
Assisted-by: Claude
* Fall back to FrameObject when tracing; fix cross-thread frame access
- invoke_with_locals: fall back to heap-allocated FrameObject path
when use_tracing is active, so trace/profile callbacks fire correctly.
This fixes test_trace, test_bdb, test_monitoring regressions.
- set_current_frame: publish top_iframe alongside top_frame in ThreadSlot
so cross-thread readers (sys._current_frames) can materialize
stack-allocated frames from other threads under stop-the-world.
- get_all_current_frames: use TLS CURRENT_FRAME for current thread,
fall back to top_iframe materialization for other threads.
- Use set_current_frame (not nosave) in with_iframe for proper
cross-thread visibility.
Assisted-by: Claude
* Apply rustfmt
Assisted-by: Claude
* Fix Windows: gate unix-only top_iframe_ptr variable
Assisted-by: Claude
* Fix frame chain: f_back, retained_back, owner, faulthandler
- Set materialized frame owner to FrameObject (not Thread) so
frame.clear() works on traceback frames
- Fix f_back for materialized frames: walk TLS chain to find source
iframe and materialize its previous frame
- Fix faulthandler dump_traceback to use dump_live_frames (reads
live lasti from stack iframes instead of stale materialized copies)
- Add retained_back propagation in with_frame cleanup: when a frame
escapes, materialize caller and mark it escaped for chain propagation
- Fix release_datastack_frame: don't overwrite retained_back if
already set by with_frame cleanup
Assisted-by: Claude
* Apply rustfmt
Assisted-by: Claude
* Refine with_frame retained_back: use frame_obj instead of materialize
Avoid materializing caller frames in with_frame cleanup to prevent
creating localsplus snapshots that keep extra refcounts. Only use
existing FrameObjects (frame_obj) for retained_back. Also keep
previous pointer alive in release_datastack_frame since the caller
is still executing at that point.
Assisted-by: Claude
* Fix f_lineno: use Cell<u32> for prev_line, update on every instruction
prev_line was only updated when tracing was active, causing f_lineno
to return stale values when observed mid-CALL (e.g. from warnings.warn
or sys._getframe). Changed prev_line from u32 to Cell<u32> for
interior mutability so it can be read safely while ExecutingFrame
holds a shared reference, and update it on every instruction so
f_lineno always returns the correct line.
Assisted-by: Claude
* Fix f_lineno for materialized frames: read live prev_line from TLS chain
Materialized FrameObjects have a snapshot of prev_line from materialize
time. When f_lineno is called on a materialized frame that is still
executing (its source iframe is on the TLS chain), walk the TLS chain
to find the source iframe and read its live prev_line. This fixes
lineno tracking in warnings, gettext, and other modules that read
frame line numbers during function calls.
Also fixed the pointer comparison: use from_payload_ptr to convert
FrameObject payload address to Py<FrameObject> address for matching
against the materialized field.
Assisted-by: Claude
* Sync live frame state for materialized frames
- Add find_live_source_iframe() to walk TLS chain and find the live
source InterpreterFrame for a materialized FrameObject
- f_lasti: read live lasti from source iframe when available
- f_lineno: use read_volatile for live prev_line from source iframe
- sync_visible_locals_to_mapping: read live localsplus from source iframe
- framelocalsproxy_getval: read live localsplus from source iframe
- with_iframe cleanup: sync localsplus, prev_line, lasti to materialized
FrameObject when the frame returns
- sync_fastlocals_from: new LocalsPlus method to update fastlocals
Fixes test_inspect (stale f_locals, stale lineno in positions),
test_listcomps (stale iteration variable in locals()).
Assisted-by: Claude
* Fix retained_back: use materialize_chain and strong_count check
- Add materialize_chain() that creates a lightweight FrameObject with
empty localsplus for f_back chain building only. This avoids cloning
local variables which would create extra refcounts (fixes test_memoryview).
- with_frame cleanup: use strong_count > 1 instead of escaped flag to
determine if retained_back is needed, ensuring chain propagation.
- with_iframe cleanup: use materialize_chain for retained_back to avoid
extra refcounts from localsplus snapshots.
Fixes test_memoryview (refcount from retained_back chain) and
test_traceback.test_extract_stack (f_back chain depth).
Assisted-by: Claude
* Fix cross-thread f_back chain for sys._current_frames/exceptions
- get_all_current_frames: materialize entire frame chain and link
retained_back during stop-the-world for other threads
- f_back: use stop-the-world to safely materialize cross-thread frame
chains on unix when prev iframe is on another thread
- f_back: check frame_obj() before STW for fast path
Fixes test_sys.test_current_frames and test_sys.test_current_exceptions.
Assisted-by: Claude
* Fix monitoring LINE events and faulthandler thread dump
- Skip all instrumented opcodes in bytecode loop prev_line update to
avoid defeating InstrumentedLine de-duplication
- Add prev_line update in execute_instrumented for non-RESUME/non-LINE
instrumented opcodes
- update_events_mask: walk TLS iframe chain to re-instrument all frames
including stack-allocated ones (fixes missing LINE events for inline
frames already past RESUME)
- faulthandler: use top_iframe fallback for stack-allocated frames in
dump_all_threads and watchdog_thread
- faulthandler: use dump_live_frames for current thread dump
Fixes test_monitoring LINE/CALL tests, test_faulthandler thread dumps.
Assisted-by: Claude
* Fix clippy: remove duplicate Radium import, use previous() helper
Assisted-by: Claude
* Fix materialized frame issues: GC tracking, live locals, f_trace propagation
- Track materialized FrameObjects in GC so cycle collection works
(fixes __del__ not called at exit for exception traceback cycles)
- materialize_chain returns owned PyRef without temporary_refs to
avoid defeating GC cycle detection from non-GC-tracked storage
- framelocalsproxy_setval writes to live source iframe when available
(fixes f_locals proxy writes not affecting executing frame)
- f_lineno uses lasti-based line for non-executing frames instead of
prev_line (fixes PEP 626 wrong line after exception unwind)
- f_trace/f_trace_lines/f_trace_opcodes setters propagate to live
source iframe (fixes pdb set_trace not working through materialized
frames)
- frame.clear() checks find_live_source_iframe to reject clearing a
frame that is backed by a live stack-allocated iframe
- Fix clippy warnings and unused imports in faulthandler
- Remove unsafe top_iframe access from faulthandler watchdog thread
Assisted-by: Claude
* Fix faulthandler crashes, GC tracking timing, and test_generators
- Use top_iframe instead of top_frame for frame chain walking in
faulthandler (dump_all_threads and watchdog), avoiding stale or
GC-cleared FrameObject pointer dereference
- Defer GC tracking of materialized FrameObjects to with_iframe
cleanup (after set_current_frame restores old chain), preventing
premature collection while temporary_refs still holds the only ref
- Add FrameObject::try_iframe() for safe access to potentially
GC-cleared frames
- Remove expectedFailure for test_exhausted_generator_frame_cycle
(now passes with GC tracking fix)
- Add 'noalias' to cspell dictionary
Assisted-by: Claude
* Fix Windows frame dump and remove dead code
- Publish top_iframe on all platforms (not just unix) so Windows
faulthandler and sys._current_frames can see stack-allocated frames
- Use top_iframe instead of frames Mutex in Windows faulthandler
dump and watchdog paths
- Use top_iframe in non-unix get_all_current_frames for sys._current_frames
- Remove unused dump_traceback_thread_chain (replaced by top_iframe walk)
- Mark test_pdb_await_support as expected failure
Assisted-by: Claude
* Fix ThreadSlot init on all platforms, skip test_pdb_await_support
- Add top_iframe to ThreadSlot initializer in init_thread_slot_if_needed
(fixes wasm32, Windows compile errors)
- Skip test_pdb_await_support on RustPython (async pdb exception
callback receives None exc argument)
Assisted-by: Claude
* Remove dead faulthandler code (dump_frame_from_ref, dump_traceback_thread_frames, try_iframe)
These functions are no longer called after switching all frame chain
walking to use top_iframe directly.
Assisted-by: Claude
* Fix Windows sys._current_frames: materialize full frame chain
The non-unix get_all_current_frames path was only materializing the
top iframe without linking retained_back, so f_back chain walking
could not find deeper frames like f123() in test_sys tests.
Replicate the full chain materialization from the unix path.
Assisted-by: Claude
* Use STW-based cross-thread f_back on all platforms
Replace the non-unix frames-mutex fallback with the same
stop-the-world iframe chain materialization used on unix.
Fixes test_current_exceptions on Windows where stack-allocated
iframes were not found in the FrameObject-only frames list.
Assisted-by: Claude
* Fix set_f_lineno to write to live iframe, use STW for cross-thread access
- set_f_lineno: write lasti/pending_stack_pops to the live source
iframe instead of the materialized copy so debugger jumps take
effect on stack-allocated frames
- f_back cross-thread: enter STW before dereferencing the prev
pointer to prevent use-after-free if the owning thread returns
- get_all_current_frames (non-unix): add STW protection when
walking cross-thread iframe chains, matching the unix path
Assisted-by: Claude
* Guard current_location() against lasti==0, always capture retained_back
- current_location(): return first_line_number instead of
panicking when lasti is 0 (before first instruction executes)
- with_frame retained_back: materialize the caller iframe when
needed so f_back always resolves for escaped FrameObjects
Assisted-by: Claude
* Fix format, revert eager retained_back materialization
Remove double blank line in frame.rs (lint failure).
Revert with_frame retained_back to only use already-materialized
caller FrameObjects — eagerly materializing the caller added
refcounts on local variables, preventing timely deallocation and
causing test_io, test_memoryview, and test_futures failures.
Assisted-by: Claude
* Remove rustpython-unicode-isolation-issue.md from gitignore
Assisted-by: Claude
* Remove test_pdb_await_support skip: async pdb now works correctly
Assisted-by: Claude
* Fix STOP_ITERATION monitoring: wrap value in StopIteration instance
fire_stop_iteration was passing the raw iterator return value to
callbacks, but the STOP_ITERATION event callback signature expects
a StopIteration exception instance. Wrap non-StopIteration values
in a new StopIteration(value), matching PyMonitoring_FireStopIterationEvent.
This fixes test_pdb_await_support where bdb's exception_callback
received None instead of a StopIteration instance.
Assisted-by: Claude
|
||
|
박효종
|
6a7d0fd144 |
Allow lone-surrogate keyword keys in f(**d) (#8409)
* Allow lone-surrogate keyword keys in f(**d)
`f(**{'\udc81': 2})` raised `TypeError: keywords must be strings` even
though the key is a valid `str`, because `KwArgs` stored keys as Rust
`String` (strict UTF-8) and `collect_ex_args` narrowed each key to
`PyUtf8Str` (valid UTF-8 required). CPython only checks that the key is a
`str` (`Py_TPFLAGS_UNICODE_SUBCLASS`), never encoding validity.
Change `KwArgs<T>`'s key type from `String` to `Wtf8Buf` (the WTF-8
storage `PyStr` already uses) and relax the keyword-key downcast from
`PyUtf8Str` to `PyStr` in `collect_ex_args`, `from_vectorcall`/
`from_vectorcall_owned` (which previously panicked on surrogate keys),
the `functools.partial` keyword merge, and the C-API `dict_to_kwargs`.
`Wtf8Buf` borrows only as `Wtf8`, so inherent `get`/`contains_key`/
`swap_remove`/`shift_remove(&str)` on `KwArgs` restore the `&str` lookup
interface `String: Borrow<str>` used to provide (via the zero-cost
`Wtf8::new` cast), and a generic `FromIterator<(K: Into<Wtf8Buf>, T)>`
keeps construction sites unchanged. WTF-8 awareness stays localized to
`function/argument.rs`.
Fixes #8228
Assisted-by: Claude Code:claude-opus-4-8
* Avoid cloning kwargs keys in _ast, borrow via as_ref instead
`new_str` only needs a `&Wtf8`, so pass `key.as_ref()` rather than
cloning the key into an owned `Wtf8Buf`. The key is reused afterwards
(error message, `intern_str`), so a borrow is the right fit.
Assisted-by: Claude Code:claude-opus-4-8
|
||
|
Bas Schoenmaeckers
|
4d80a3320f | Fix some pyo3 testcase against the c-api (#8349) | ||
|
Bas Schoenmaeckers
|
04a203c955 | Fix deadlock in c-api when a thread enters a stop_the_world (#8355) | ||
|
Bas Schoenmaeckers
|
455d22b353 |
Add more unicode functions to the c-api (#8272)
* Add more unicode functions Review Add unicode decode functions Add more functions * Remove * Add helper |
||
|
강동안
|
12057b3a03 | Take a borrow in invoke_exception instead of an owned type (#8327) | ||
|
Jeong, YunWon
|
c7d265e8dd |
ctypes: fix Windows pip truststore (#8314)
* ctypes: preserve result when errcheck returns args * ctypes: cover Windows pip truststore * capi: implement PyEval_SaveThread * capi: merge saved thread state into PyThreadState |
||
|
김은빈
|
d26972e3bc |
weakref: reuse proxy objects (#8307)
* weakref: reuse callback-less proxy objects like CPython Make PyWeakProxy a #[repr(transparent)] view sharing PyWeak's payload (manual PyPayload with shared PAYLOAD_TYPE_ID + class check), so a proxy is the weakref-list node itself and add()'s reuse applies to it directly. Port CPython's insert_weakref/get_basic_refs slot discipline: genericref at head, generic proxy right after it, with callback/class re-verificationbefore reuse. Constructor moves to a slot_new override since reuse returns an existing object. Drops the hidden __weakproxy marker subclass and the proxy branch in PyWeakref_GetRef. Assisted-by: Claude Code:claude-opus-4-8 * weakref: always validate class in basic-proxy slot lookup * weakref: rustfmt find_generic_proxy_ptr * weakref: unify reuse check shape |
||
|
Bas Schoenmaeckers
|
c5efd71f2a |
Add codecs support to c-api (#8267)
* Add codecs support to c-apo * Reduce code duplication |
||
|
Bas Schoenmaeckers
|
96fde50f75 | Add AsyncIterator support to c-api (#8285) | ||
|
Bas Schoenmaeckers
|
ac28fff085 |
Add descriptor support to c-api (#8283)
* Add descriptor support to c-api * Review * Extract getset builder * Extract memberdef builder |
||
|
Bas Schoenmaeckers
|
57b630a6c7 | Add str compare functions to c-api (#8268) | ||
|
Bas Schoenmaeckers
|
7f81b2491c |
Add more dict functions to c-api (#8269)
* Add more dict functions to c-api * Review |