mirror of
https://github.com/RustPython/RustPython.git
synced 2026-10-29 03:04:38 +08:00
0.6.0
466
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Jeong YunWon
|
b2626c56de |
Drop parking_lot_core git patch and tkinter deps
Comment out the parking_lot_core git patch and use the crates.io crate. Remove tcl-sys, tk-sys, and the tkinter feature. Assisted-by: Grok:4.7 |
||
|
Jeong, YunWon
|
cb0f3b012b |
Add a native _zoneinfo module (#8915)
* Add a native _zoneinfo module ZoneInfo reads TZif tuples from zoneinfo._common.load_data. The stdlib module implements utcoffset, dst, tzname, fromutc, and the zone cache. whats_left compares that class. Cache tests no longer expect failure. Assisted-by: Grok:grok-4.7 * Fix ZoneInfo layout and keep its cache on the module utcoffset, dst, and tzname take the object. Zone data is boxed so the payload lines up with tzinfo on 32-bit. The strong, weak, and timedelta caches live on each interpreter's _zoneinfo module. Assisted-by: Grok:grok-4.7 |
||
|
Joshua Megnauth
|
3e2d4129fe |
sqlite: Handle C strs at FFI margins, fix lint (#8910)
Some of the FFI wrappers take in a byte slice that's presumed to be NUL capped. It's more ergonomic and safer to take in a &CStr which is impossible to get wrong because it's validated at the margins. Additionally, there were a few areas where a NUL was embedded directly into the source (i.e. "COMMIT(NUL)" which displayed as "COMMIT^@"). This had the interesting effect of breaking grep on the source code. |
||
|
Jeong, YunWon
|
100dac0bbe |
Reject payload receivers on #[pymethod] and #[pygetset] (#8898)
* Report errors from #[pyclass] trait items The trait branch of impl_pyclass_impl built its own ImplContext, so the errors collected while extracting trait items were never emitted: an invalid #[pymethod] or #[pygetset] in a #[pyclass] trait was left out of the class without a compile error. The trait branch now uses the shared context whose errors are emitted. Assisted-by: Claude Code:claude-opus-5-5 * Pass the object to #[pymethod] and #[pygetset] receivers outside vm Outside crates/vm, where impl Py<T> cannot be written, #[pymethod] and #[pygetset] receivers that took &self become zelf: &Py<Self> inside the existing impl T blocks. Where Rust code calls one of these methods on a payload value, the logic stays in a payload method and the Python method forwards to it. Assisted-by: Claude Code:claude-opus-5-5 Assisted-by: Grok:grok-4.7 * Reject payload receivers on #[pymethod] and #[pygetset] A #[pymethod] or #[pygetset] item with a self receiver is now a compile error when self is the payload: in an impl for T rather than Py<T> or PyRef<T>, and in a #[pyclass] trait. The error points to taking zelf: &Py<Self> or zelf: PyRef<Self>, or defining the item in impl Py<T>. Assisted-by: Claude Code:claude-opus-5-5 * Pass the object to the layout test getter The subclass layout test's extra getter is a #[pygetset] on the payload. Take zelf: &Py<Self> so the receiver is the object. Assisted-by: Grok:4.7 |
||
|
1ndahous3
|
ea76251294 |
Cache parsed struct formats per interpreter (#8903)
* Cache parsed struct formats per interpreter Assisted-by: Codex:GPT-6 * Preserve struct cache entries on concurrent misses Assisted-by: Codex:GPT-6 |
||
|
Jeong, YunWon
|
d54a4c5932 |
Set builtin __doc__, __module__, and datetime signatures (#8897)
* Set builtin __doc__, __module__, and datetime signatures Expose empty doc-db member entries as ''. Heap __dict__ and __weakref__ getsets carry their descriptor docs. Member repr uses the qualified type name. __module__ is collections for the deque iterators, _csv for Dialect, ctypes for ArgumentError and CField, _lzma for LZMAError, operator for the getter types, pickle for PickleBuffer, _ssl for MemoryBIO, SSLSession, _SSLContext and _SSLSocket, weakref for the proxy types, and posix (nt on Windows) for times_result and uname_result. unicodedata's copied UCD functions set __module__ to unicodedata. _testinternalcapi function docs and weakproxy.__str__ use the stored text. datetime replace defaults render as unchanged, __replace__ is ($self, /, **changes) and binds through replace, now defaults tz to None, and fromisoformat and tzinfo methods name the parameter object. Assisted-by: Grok:grok-4.7 * Move doctext helpers above the test module db_slice and docs sat after mod tests. Remove expectedFailure from test_html_doc, test_text_doc, test_help_output_redirect, test_method_aliases, and test_long_signatures. Assisted-by: Grok:grok-4.7 |
||
|
1ndahous3
|
f909d2a30d |
Add native _datetime module (#8872)
* Add native _datetime module Assisted-by: Claude Code:claude-opus-5-5 Assisted-by: Codex:GPT-6 * Use pymember and safe timezone storage in datetime Assisted-by: Codex:GPT-6 |
||
|
Jeong, YunWon
|
04eb0ad94c |
Match remaining native docs and signatures to CPython 3.14 (#8877)
* Record native docstrings the doc DB generator skipped - Stop ignoring `__dir__`; skip `__base__`, `__bases__` and `__module__` instead. - Recurse into native classes whose metaclass is not `type`, and also key a class stored under another attribute name by its own `__name__`. - Collect native objects exposed by pure-Python modules, such as the codec error handlers, under their own `__module__` and `__name__`. - Key types found through `object.__subclasses__()` by their own module as well as `builtins`. - Record getset/member descriptors named like class metadata (`coroutine.__name__`), and native types a module exposes only through an instance (`sys.flags`). - Treat a class as native only when it owns its C descriptors, so an enum holding a copied `str.__format__` is skipped. - Keep docstrings unstripped, and import every module in sorted order before walking them. - Apply the resulting key delta, measured on macOS with CPython 3.14.7, to data.inc.rs. Assisted-by: Claude:claude-opus-5-5 Assisted-by: Grok:grok-4.7 * Stop Rust comments on Python items from becoming __doc__ Use `//` instead of `///` on Python-exposed items so their comments are not published as `__doc__`. Assisted-by: Claude:claude-opus-5-5 Assisted-by: Grok:grok-4.7 * Match native names, signatures and doc lookup - Rename the weakref type to `weakref.ReferenceType` and define `__class_getitem__` on `_ctypes.Array` instead of its metaclass. - Give each name of a multi-name `#[pymodule(sub)]` function (`posix.remove`/`unlink`) its database doc. - Fill methods from `with(...)` traits (dict view `__reversed__`, `isdisjoint`) with the implementing class's database doc. - Resolve database docs for `#[pymodule]` classes at macro time, and for the codec error handlers and `_lzma.LZMAError` in const context. - Add `Context::new_exception_type_with_doc`. - Match `posix_spawn` parameter names and defaults, make `dict_keys` and `dict_items` unhashable, add `_CData.__ctypes_from_outparam__`, and omit the Windows-only lines of `sys.__doc__` on other platforms. Assisted-by: Claude:claude-opus-5-5 Assisted-by: Grok:grok-4.7 * Give struct sequence fields their database docs `new_readonly_tuple_member` takes the field doc, looked up in the class's attribute doc table through the new `class_attr_item_doc`. Assisted-by: Claude:claude-opus-5-5 * Look up pymodule(sub) docs by an explicit module name posix and nt share one doc entry. A sub module sets name to the Python module it is merged into, and doc lookup uses that name. Assisted-by: Claude:claude-opus-5-5 Assisted-by: Grok:grok-4.7 * Record pending class doc bodies on PyMethodDef The pymethod derive sets doc_body_pending when a method has no Rust doc body and expansion did not resolve a database span. concat_with_attr_docs copies that span from the class table and clears the flag. Other constructors leave the flag false. Assisted-by: Claude:claude-opus-5-5 Assisted-by: Grok:grok-4.7 * Name argument fields after their Python parameters PosixSpawnArgs.args is renamed to argv, and nt's follow_symlinks field drops a name override equal to the field name. Assisted-by: Claude:claude-opus-5-5 |
||
|
Jeong, YunWon
|
8f84fea377 |
Fix 32-bit member layout assertions and address member descriptor review (#8882)
* Address review findings on member descriptors Member definitions carry an internal `PY_ATOMIC` flag taken from `MemberLayout::ATOMIC`, and loads use an atomic access only for atomic storage; extension members never set it. `XMLParser` publishes its new state, `entity` and `target` only after every handler lookup succeeds. `subs_parameters` reads type names only for the error. The unraisable hook prints `<unknown>` when `__module__` fails or is not a str. The builtin function `__reduce__` decides on the bound object alone. Assisted-by: Grok:4.7 Assisted-by: Claude:claude-opus-5-5 * Fix member layout assertions on 32-bit targets The tp_flags assertion compared AtomicU64 with u64 alignment, which differs on i686. The kevent assertion required usize, isize, c_ulong and c_long to be 64-bit. Both now check only what the member cells need. Assisted-by: Grok:4.7 Assisted-by: Claude:claude-opus-5-5 * Use bitflags for member definition flags PY_READONLY, PY_AUDIT_READ, PY_RELATIVE_OFFSET and PY_ATOMIC become the repr(transparent) PyMemberFlags. C extension flags are read with from_bits_retain. Assisted-by: Grok:4.7 Assisted-by: Claude:claude-opus-5-5 |
||
|
Jeong, YunWon
|
4344a34a3c |
Stop Rust comments on Python items from becoming __doc__ (#8893)
Use `//` instead of `///` on Python-exposed items so their comments are not published as `__doc__`. Assisted-by: Claude:claude-opus-5-5 Assisted-by: Grok:grok-4.7 |
||
|
1ndahous3
|
af37a76c1f |
Floor and ceil exact floats directly in math.floor/math.ceil (#8834)
Assisted-by: Claude Code:claude-opus-5-5 |
||
|
Alessio Attilio
|
7329839fb5 | fix(core): align hashability of dict views, namespace, and typing builtins (#8844) | ||
|
1ndahous3
|
afae370753 |
Fix array search equality and reentrant callbacks (#8870)
Assisted-by: Codex:GPT-6 |
||
|
Alessio Attilio
|
b9fea9c7be |
fix(contextvars): isolate context stack per interpreter and thread on VirtualMachine (#8838)
* fix(contextvars): isolate context stack per interpreter and thread on VirtualMachine * style: format and sort imports in extra_tests/snippets/stdlib_contextvars.py Assisted-by: Antigravity:Gemini-3.8-Flash |
||
|
Alessio Attilio
|
65647e99ee |
fix(csv): quote non-numeric objects in writer QUOTE_NONNUMERIC mode (#8843)
* fix(csv): quote non-numeric objects in writer QUOTE_NONNUMERIC mode * style: format extra_tests/snippets/stdlib_csv.py Assisted-by: Antigravity:Gemini-3.8-Flash |
||
|
Jeong, YunWon
|
31963fa767 |
Expose more attributes as member descriptors (#8857)
* Read type.__module__ from the dict only for heap types A static type's `__module__` is the text before the last `.` of its name, or `builtins`. A heap type reads its dict and raises `AttributeError` when the entry is missing; `type` and `object` reprs ignore that error. `type()` no longer stores `__module__ = None` when the caller's globals have no `__name__`. `#[pyexception(module = ...)]` now reaches the type name, and static types that relied on a dict `__module__` (ast nodes, pyexpat, ssl, _hashlib and _interpchannels errors, ExceptionGroup) carry their module. Assisted-by: Grok:4.7 Assisted-by: Claude:claude-opus-5-5 * Expose more builtin attributes as member descriptors `code.co_flags`, `co_nlocals`, `co_filename`, `super.__self__`, `__self_class__`, `__thisclass__`, `traceback.tb_frame`, `tb_lasti`, `type.__itemsize__` and `builtin_function_or_method.__module__` are now `#[pymember]` fields instead of getsets. `super` keeps its three references in atomic cells instead of a lock, `tb_lasti` is an `i32`, `co_nlocals` is stored at construction, and the builtin function `__module__` is a writable object cell. Assisted-by: Grok:4.7 Assisted-by: Claude:claude-opus-5-5 * Expose stdlib attributes as member descriptors Decompressor `eof`/`needs_input`/`unused_data` (bz2, lzma, zlib, plus lzma `check`), `ContextVar.name`, `functools.partial` `func`/`args`/ `keywords`, socket `family`/`type`/`proto`, `select.kevent` fields, `SemLock` `handle`/`kind`/`maxvalue`/`name`, `Pickler.bin`/`fast`, `xmlparser.intern`, ElementTree `XMLParser.entity`/`target`, `UCD.unidata_version`, `_ExceptHookArgs` fields, multibytecodec `stream`, and `_io` `Buffered*.raw` and `TextIOWrapper` `buffer`/`encoding`/`line_buffering`/`write_through` are now `#[pymember]` fields instead of getsets. Values that changed under a lock move to atomic cells on the payload. New member kinds `Short`, `Long`, `String`, `UShort`, `ULong`, `LongLong` and `ULongLong` access exactly the width of their C type. Assisted-by: Grok:4.7 Assisted-by: Claude:claude-opus-5-5 * Match bare super and static type itemsize `super.__new__(super)` leaves `__thisclass__`, `__self__` and `__self_class__` empty, and a super with no type takes the zero-argument path when bound. Static types carry their `tp_itemsize` (`type`, `memoryview`, `code`, `frame`, generators, `_sre` objects, `_ctypes` thunks), inherit it from their direct base when they leave it at 0, and a variable-size base rejects any nonempty `__slots__` and a weakref slot. `_thread._ExceptHookArgs` is a struct sequence. Assisted-by: Grok:4.7 Assisted-by: Claude:claude-opus-5-5 * Expose SimpleNamespace.__dict__, __basicsize__ and __flags__ as members `SimpleNamespace.__dict__` addresses the instance dict cell the same way `module.__dict__` does. `slots.basicsize` now holds the full `tp_basicsize` (header plus payload), so `__basicsize__` reads it directly. Type flags are one atomic word: `__abstractmethods__` and `_abc` set their bits in it in place, the separate `abc_tpflags` word is gone, and `__flags__` reads the word as an unsigned long member. Assisted-by: Grok:4.7 Assisted-by: Claude:claude-opus-5-5 |
||
|
Jiseok CHOI
|
f87170772e |
sqlite3: restrict Row case matching to ASCII (#8856)
Assisted-by: GitHub Copilot:GPT-5.6 Terra (medium) |
||
|
Jiseok CHOI
|
7897287e94 |
sqlite3: raise IndexError for oversized Row indices (#8855)
Assisted-by: GitHub Copilot:GPT-5.6 Terra (medium) |
||
|
Jiseok CHOI
|
a63a53a15f |
sqlite3: reject text_factory deletion (#8854)
Assisted-by: GitHub Copilot:GPT-5.6 Terra (medium) |
||
|
Jeong, YunWon
|
be23b2c562 |
Store database docstrings xz-compressed behind an optional doc feature (#8836)
* Skip Python-defined classes when collecting builtin type docs generate.py recorded every object.__subclasses__() entry as builtins.<name> after importing the whole stdlib, so classes written in Python (tkinter, unittest, tarfile, ...) ended up in the doc database. Classes that are mutable heap types without C method descriptors and whose module is not a C extension are now skipped, and their entries are removed from data.inc.rs. Assisted-by: Claude:claude-opus-5-5 Assisted-by: Grok:grok-4.7 * Store database docstrings xz-compressed behind an optional doc feature - rustpython-doc's build script compresses all database docstrings into one xz blob with an offset index; identical texts share one offset. - Macros emit blob references for database docs; vm decompresses the blob on first access. - New `doc` cargo feature (default on) in rustpython, rustpython-vm and rustpython-stdlib. Without it the blob and xz are not linked and database-backed __doc__ is None; __text_signature__ is unaffected. - Native types without a docstring get __doc__ = None in their dict. - Workflows that build with --no-default-features enable `doc` where tests or releases need docstrings. Assisted-by: Claude:claude-opus-5-5 Assisted-by: Grok:grok-4.7 |
||
|
Jeong, YunWon
|
bdc9246b37 |
Address builtin members by field offset (#8816)
* Address builtin members by field offset `#[pymember]` now declares a field of the payload and generates a PyMemberDef whose offset is the payload offset plus the field offset. The getter/setter function form and `MemberAccess::Func` are removed; builtin, slot, and C-API members share one get/set path. The macro checks the field type against the member type. Converted members: property fget/fset/fdel/__doc__, function __globals__/__closure__/__builtins__/__doc__/__module__, staticmethod and classmethod __func__/__wrapped__, complex real/imag (T_DOUBLE), frame f_trace_lines (T_BOOL), defaultdict default_factory, getset and member descriptor __name__/__objclass__, module __dict__. frame.f_trace_opcodes is a getset. Writable member fields are atomic pointer cells. The instance dict pointer is an atomic cell at a fixed offset in front of the object, and dict presence follows the type's HAS_DICT flag. The pickle basicsize check no longer counts dict, weakref, and slot storage, which live outside the payload. Assisted-by: Grok:4.7 Assisted-by: Claude:claude-opus-5-5 * Address member review findings - `#[pymember]` keeps leading `#[cfg]` attributes on the generated registration, and malformed `field`/`offset` values are compile errors. - The `descriptor` module and the items the expansion names are public. - Writable double members are an `AtomicU64` cell, loaded and stored with relaxed atomics. Readonly double members stay plain `f64` reads. - `LOAD_ATTR_SLOT` is not specialized for `Py_AUDIT_READ` members. - `defaultdict.__reduce__` passes no factory argument when `default_factory` is None. Assisted-by: Grok:4.7 Assisted-by: Claude:claude-opus-5-5 * Declare members on payload fields `#[pymember]` goes on a payload struct field, or on the struct with an explicit `offset` when the storage is not a payload field. The struct `#[pyclass]` builds `PyClassDef::MEMBERS`, a static `PyMemberSpec` table, and `extend_class` registers one member descriptor per entry. `path` addresses a subfield, one field may carry several members, and a `#[cfg]` on the field gates its entry. The impl-side const form is removed and now a compile error. Assisted-by: Grok:4.7 Assisted-by: Claude:claude-opus-5-5 * Split nullable and non-null atomic cells The nullable cell API (`new_empty`, `load_owned`, `store`, `compare_exchange_empty`) moves from `PyAtomicRef<PyObject>` to `PyAtomicRef<Option<PyObject>>`, and the typed `PyAtomicRef<Option<T>>` shares it. The non-null `PyAtomicRef<PyObject>` gets `load_owned` and `store` that return the object directly. Slot cells, property fget/fset/fdel/__doc__, and defaultdict default_factory are nullable cells. staticmethod and classmethod callable are non-null cells and are read without `unwrap`. The instance dict cell is `PyAtomicRef<Option<PyDict>>`, so reading it no longer downcasts. Assisted-by: Grok:4.7 Assisted-by: Claude:claude-opus-5-5 * Drop the unused cold trace_lines flag `FrameColdData::trace_lines` had no writer after `f_trace_lines` became a frame object member. A frame without a frame object traces lines by default. Assisted-by: Claude:claude-opus-5-5 * Compute payload offsets from the merged Py header `PyInner` is now `Py`, so the payload offset and the dict member offset are measured from `Py<T>`. Assisted-by: Claude:claude-opus-5-5 * Expose descriptor names and owner types as members method_descriptor, classmethod_descriptor, and wrapper_descriptor expose `__objclass__` and `__name__` as readonly members, and method-wrapper exposes `__self__` as a readonly member. The separate `objclass` field of method and classmethod descriptors always equaled `common.typ` and is removed. Assisted-by: Grok:4.7 Assisted-by: Claude:claude-opus-5-5 * Expose remaining builtin tp_members as members Attributes stored in a payload field become readonly or writable member descriptors: bound method __func__/__self__, code co_argcount/co_posonlyargcount/co_kwonlyargcount/co_stacksize (Py_T_UINT), generic alias, coroutine and async generator members, interpolation and template, range and slice start/stop/step, type __base__, union, and the typing TypeVar, ParamSpec, TypeVarTuple, and TypeAliasType members. `MemberKind` gains `Int` and `Uint` with the `PyMember_GetOne`/`SetOne` range checks. Attributes computed on access or kept behind a lock stay getsets. Assisted-by: Grok:4.7 Assisted-by: Claude:claude-opus-5-5 * Spell a docless member as doc = false `#[pymember(doc = false)]` stores no docstring, matching the `doc = "..."` form, and replaces the separate `no_doc` flag. Assisted-by: Claude:claude-opus-5-5 * Make pymember readonly by default `#[pymember]` declares a readonly member unless `writable` is given. The `readonly` option is removed and every declaration is converted. Assisted-by: Claude:claude-opus-5-5 * Check the receiver of method descriptors Binding or calling a method_descriptor with an object that is not an instance of the owner type raises TypeError before the native function runs. Static and class methods are exempt. Assisted-by: Grok:4.7 Assisted-by: Claude:claude-opus-5-5 * Expose exception attributes as members BaseException.__suppress_context__, StopIteration.value, SystemExit.code, ImportError, NameError, AttributeError, OSError, SyntaxError, BaseExceptionGroup and the UnicodeEncode/Decode/TranslateError attributes are member descriptors backed by payload fields. ImportError, NameError, AttributeError and SyntaxError gain payloads for their fields, and the methods that read those attributes read the fields. Add the Py_ssize_t member kind for UnicodeError start and end. Assisted-by: Grok:4.7 Assisted-by: Claude:claude-opus-5-5 * Avoid same-type casts in unsigned member conversion On targets where c_ulong is u32, casting through c_ulong was a no-op cast that clippy rejects. Assisted-by: Claude:claude-opus-5-5 * Prefer an explicit pymember doc over the doc table doc = "text" is used as the docstring without consulting the stored attribute documentation; only an omitted doc looks it up. Drop the member doc strings that repeated the stored documentation. Assisted-by: Grok:4.7 Assisted-by: Claude:claude-opus-5-5 * Infer pymember kinds from field types MemberLayout gives the member kind of each accepted field type, and the macro reads it from the field or path subfield, so type = "bool" and the other numeric spellings are gone. type only selects object_ex. Writable doubles use a new AtomicF64 cell instead of AtomicU64 bits. MemberLayout and MemberCell replace the per-kind marker traits. Assisted-by: Grok:4.7 Assisted-by: Claude:claude-opus-5-5 * Read nullable atomic cells only through load_owned PyAtomicRef<Option<T>> and PyAtomicRef<Option<PyObject>> lose to_owned, to_owned_ordering and try_to_owned; load_owned, which retries the incref until it applies to the pointer still in the cell, is the one owned read. Callers that kept a deref borrow across VM calls now take a strong reference. Assisted-by: Grok:4.7 Assisted-by: Claude:claude-opus-5-5 * Inherit static type slots from the whole MRO extend_class copied slots only from the direct bases, and tp_init is copied only from a base that defines it, so TabError missed the SyntaxError init that IndentationError inherits. Walk the MRO the same way init_slots does. Assisted-by: Grok:4.7 Assisted-by: Claude:claude-opus-5-5 * Let Rust doc comments win over stored docstrings A /// comment on a Python-exposed item is its docstring; the rustpython-doc entry is used only when there is none. #[pymember] reads the field's /// comment instead of doc = "...". Doc comments that were shadowed by a stored docstring become // comments, so no docstring changes. AGENTS.md describes the rule. Assisted-by: Grok:4.7 Assisted-by: Claude:claude-opus-5-5 * Adapt members to upstream changes _collections._tuplegetter.__doc__ is a writable nullable member on the payload field. The impl-side const form no longer compiles. Assisted-by: Grok:4.7 * Drop docstring text left as comments and inline downcast_cell Comments that only repeated an item's docstring are removed; developer notes stay as // comments. The typed cell methods call downcast_unchecked directly, each with its own safety note. Assisted-by: Grok:4.7 Assisted-by: Claude:claude-opus-5-5 * Read sqlite row factories through load_owned The nullable cell no longer has to_owned; the sqlite module is built only with the sqlite feature, so the default build did not catch it. Assisted-by: Claude:claude-opus-5-5 |
||
|
1ndahous3
|
eabba7c1b7 |
Match CPython's overflow limits in math.perm and math.comb (#8841)
Assisted-by: Claude Code:claude-opus-5-5 |
||
|
Jeong, YunWon
|
dc3b93abfb |
Read tuple, bytes, int, float and complex payloads through views on Py<T> (#8819)
* Read PyTuple payloads through as_slice() - Add Py<PyTuple<R>>::as_slice(); it is generic over R, so typed tuples use the same view. - Callers reach slice methods, indexing and slice arguments through as_slice(). - PyTuple's own sequence and mapping slots and marshal's set_marshal_item reach the remaining PyTuple methods through the payload field. - The Py<T>::payload field is pub(crate). Assisted-by: Claude Code:claude-opus-5-5 * Read PyBytes payloads through as_bytes() - Callers read bytes through Py<PyBytes>::as_bytes(). - bytes.rs reaches PyBytesInner and the remaining PyBytes methods through the payload field; fspath and os call payload.contains_nuls(). - DictKey for [u8] falls back to the PyObject key_eq; it previously resolved to its own key_eq through Deref. Assisted-by: Claude Code:claude-opus-5-5 * Copy capi tuple arguments with as_slice().to_vec() Assisted-by: Claude Code:claude-opus-5-5 * Read sqlite3 Row tuples through as_slice() Assisted-by: Claude Code:claude-opus-5-5 * Read platform-specific tuples and bytes through as_slice() and as_bytes() Assisted-by: Claude Code:claude-opus-5-5 * Read PyInt and PyFloat payloads through as_bigint() and to_f64() Add `Py<PyInt>::as_bigint()`, `Py<PyInt>::try_to_primitive()` and `Py<PyFloat>::to_f64()`, which forward to the payload methods of the same name. Reads of the `value` field and primitive conversions go through these views; other PyInt and PyFloat methods are called through the `payload` field inside rustpython-vm and through payload() outside it. Assisted-by: Claude Code:claude-opus-5-5 * Read PyComplex payloads through as_complex() Add Py<PyComplex>::as_complex() and use it at the call sites that read the payload directly. Assisted-by: Claude Code:claude-opus-5-5 * Move PyInt conversion helpers to Py<PyInt> try_to_i64_fast(), to_str_radix_10(), as_u32_mask(), as_u64_mask(), try_to_primitive() and try_to_primitive_raw() only read the integer value, so they are defined on Py<PyInt> and callers no longer go through the payload. PyRange computes its protocol length from the BigInt directly instead of building a PyInt payload. Assisted-by: Claude Code:claude-opus-5-5 * Share one conversion between try_to_primitive() and its OverflowError variant Py<PyInt>::try_to_primitive() and try_to_primitive_in_range(), renamed from try_to_primitive_raw(), both call to_primitive(). They differ only in how a negative value for an unsigned type is reported: ValueError for try_to_primitive(), OverflowError for try_to_primitive_in_range(). The target type is bound with `for<'a> TryFrom<&'a BigInt>`, so the methods no longer tie a lifetime to &self. Assisted-by: Claude Code:claude-opus-5-5 * Move PyInt bitwise operators to Py<PyInt> __xor__, __or__, __and__ and their int_op() helper are defined on Py<PyInt>. PyBool calls them without going through the payload. Assisted-by: Claude Code:claude-opus-5-5 * Move PyBytes operators to Py<PyBytes> __add__, __contains__, __getitem__ and __mod__ are defined on Py<PyBytes>. The subscript, contains and remainder slots call them without going through the payload. Assisted-by: Claude Code:claude-opus-5-5 * Move PyInt general_op() and modpow() to Py<PyInt> The power slot calls them on Py<PyInt> without going through the payload. Assisted-by: Claude Code:claude-opus-5-5 |
||
|
Jeong, YunWonandRoshan Ramani
|
5c875d322f |
Call __index__ in default integer argument conversion (#8828)
* Accept `__index__` objects for integer arguments CPython converts these nine arguments with `__index__`, RustPython required an exact `int`. Uses the existing `ArgSize`/`ArgPrimitiveIndex` converters at each call site rather than widening the blanket `int` conversion, so argument paths CPython keeps strict (ctypes pointers, thread handles) are unaffected, and the range check happens while the argument is bound rather than in the method body. Also widens `maxsplit` before adding one. `sys.maxsize` was already reachable there with a plain `int` and overflowed in a debug build. Assisted-by: Claude Code:claude-opus-5 * Add snippet for `__index__` argument conversion Expectations verified against CPython 3.14. Assisted-by: Claude Code:claude-opus-5 * Call __index__ in primitive integer conversion; add ArgStrictInt The TryFromObject impls for isize, usize and the fixed-width integer types now take an int (or subclass) as-is and otherwise call __index__, instead of accepting only int instances. ArgStrictInt<T> keeps the old int-only conversion. The ctypes address and handle arguments (from_address, PyObj_FromPtr, dlsym, dlclose, call_function, call_cdeclfunction, non-Windows FreeLibrary) use it. _thread._make_thread_handle checks for an int itself and raises "ident must be an integer". Assisted-by: Claude:claude-opus-5-5 Assisted-by: Grok:grok-4.7 * Accept __index__ where arguments were bound as int objects chr, bin, math.factorial, zlib.crc32/adler32, binascii.crc32/crc_hqx, the start/stop of sequence index(), the select event mask and a __len__ result took an int object directly and rejected other objects with __index__. They now go through __index__. math.ldexp still accepts only an int for its exponent and raises "Expected an int as second argument to ldexp." otherwise. Extend protocol_index_argument.py with the newly accepted calls, an int subclass whose __index__ is not called, and the sites that still reject __index__. Assisted-by: Claude:claude-opus-5-5 Assisted-by: Grok:grok-4.7 * Replace ArgPrimitiveIndex and ArgSize with PySsize/PySize aliases The primitive integer conversions call __index__ themselves, so ArgPrimitiveIndex<T> and its alias ArgSize behaved the same as T. They are removed. Arguments that used ArgSize or ArgPrimitiveIndex<isize> now take PySsize (= isize), ArgPrimitiveIndex<usize> takes PySize (= usize), and the other ArgPrimitiveIndex<T> take T. Those with a literal fallback take it as a `default`; those where a missing argument is a distinct state stay OptionalArg. list.insert is a #[pymethod] again instead of going through a wrapper. Assisted-by: Claude:claude-opus-5-5 Assisted-by: Grok:grok-4.7 --------- Co-authored-by: Roshan Ramani <roshanramani.dev@gmail.com> |
||
|
Jeong, YunWon
|
00dfed8ff9 |
Resolve class attribute docs at macro time and drop rustpython-doc from vm (#8830)
* Resolve class attribute docs at macro time and drop rustpython-doc from vm - #[pyclass] and #[pystruct_sequence] emit a sorted PyClassDef::ATTR_DOCS table from the doc database; attribute and slot docs look up that table. - rustpython-vm no longer depends on rustpython-doc. - _operator.attrgetter, itemgetter, methodcaller and object get constructor Args with FromArgs::PARAMS, replacing the hardcoded class signature docs in class.rs. - methodcaller() with no arguments raises "methodcaller needs at least one argument, the method name". Assisted-by: Claude:claude-opus-5-5 Assisted-by: Grok:grok-4.7 * Drop default values that equal the field type's default Bare `default` on integer and bool fields already renders the zero value in the text signature, so `default = 0` and `default = false` are replaced with bare `default`. Assisted-by: Claude:claude-opus-5-5 Assisted-by: Grok:grok-4.7 * Return the empty str singleton from Context::new_str Assisted-by: Claude:claude-opus-5-5 |
||
|
1ndahous3
|
8fa504249e |
Speed up audit events and exec/eval; share audit hooks across threads (#8824)
* Speed up exec/eval and audited builtins when no audit hooks are set Assisted-by: Claude Code:claude-opus-5-5 * move audit hooks from `VirtualMachine` to `PyGlobalState` * addaudithook only appends once the lock * sync with main |
||
|
Jeong, YunWon
|
3c654204f5 |
Match native text signatures to CPython names, kinds and defaults (#8815)
* Render CPython defaults in generated text signatures Arguments whose default rendered as `<unrepresentable>`, while the same callable has a representable default in CPython, now render that default: - Where the Rust value can be a literal, the argument becomes a plain field with `default = <literal>`. - The remaining cases keep their Rust default and add `py_default` with a comment. These are wrappers such as ArgIntoFloat or ArgIntoBool, values that need the VM, enums, and Option where None means another value. Byte string literals render as Python bytes. A function registered under several names gets an internal doc per name, so its __doc__ no longer carries the signature header. Removed expectedFailure from test_pydoc tests that now pass. Assisted-by: Grok:grok-4.7 Assisted-by: Claude:claude-opus-5-5 * Match native callable signatures to CPython names and kinds Rename parameters and adjust positional-only / keyword-only kinds and defaults of native functions and methods so their generated text signatures match CPython. Remove expectedFailure markers from tests that now pass. Assisted-by: Claude:claude-opus-5-5 Assisted-by: Grok:grok-4.7 * Fix Windows console read size and keep type.__prepare__ permissive Assisted-by: Claude:claude-opus-5-5 * Derive signature defaults from typed Rust values Param.default is now a DefaultRepr rendered by the const writer. Literals, integer and bool constants become typed defaults, and other default expressions use the type's const fn py_default. A bare `optional` renders as None, so `default = None` rewrites are reverted. Address review comments: zlib max_length defaults, get_ca_certs binary_form keyword, getpeercert der, WASM ssl certificate keywords, fstatvfs fd-only, and _sre endpos shown as sys.maxsize. Assisted-by: Claude:claude-opus-5-5 Assisted-by: Grok:grok-4.7 * Generate class constructor text signatures at compile time A #[pyclass] impl composes the type's internal doc from the Args of its Initializer or Constructor, so type.__text_signature__ and inspect.signature work for native classes. __doc__ stays the plain doc. Struct sequences get (iterable=(), /) and __reduce__ (self, /). Hand-parsed constructors are converted to FromArgs structs, and _weakref.proxy is split from ProxyType. Assisted-by: Claude:claude-opus-5-5 Assisted-by: Grok:grok-4.7 * Remove expectedFailure from test_hmac tests that now pass Assisted-by: Claude:claude-opus-5-5 * Match remaining native signatures and count missing CPython signatures as matches Give lock.__enter__ and RLock.__enter__ their own no-argument methods, add FromArgs for start_new_thread and start_joinable_thread keeping their errors, take RTLD_GLOBAL/RTLD_LOCAL from libc, and align names, kinds and <unrepresentable> defaults where CPython's signature is invalid (bytes.hex, reduce, warn, link, setlocale, add_done_callback, ...). getaddrinfo accepts keyword arguments. whats_left treats a callable CPython has no signature for as matching. Assisted-by: Claude:claude-opus-5-5 Assisted-by: Grok:grok-4.7 * Drop unfulfilled dead_code expectation on os.link dir_fd args Assisted-by: Claude:claude-opus-5-5 * Distinguish OptionalArg and Option in signature defaults An `optional` field's signature default now comes from its type: Option<T> renders None, while OptionalArg<T> (a missing argument that is not any Python value) renders <unrepresentable>. Fields where a missing argument behaves exactly like None become Option<T>; fields whose missing state differs keep OptionalArg. Redundant py_default = "<unrepresentable>" entries are removed. Assisted-by: Claude:claude-opus-5-5 Assisted-by: Grok:grok-4.7 * Remove expectedFailure from test_rlcompleter.test_attr_matches Assisted-by: Claude:claude-opus-5-5 * Use Option for optional arguments whose missing and None states match OptionalOption and OptionalArg<Option<T>> fields that treat a missing argument like None become Option<T> with #[pyarg(optional)], which renders =None in the text signature. Plain optional parameters of that kind move into small FromArgs structs. Uses where missing and None differ (reduce initial, StringIO newline, TextIOWrapper.reconfigure, set_asyncgen_hooks) keep OptionalOption. Assisted-by: Claude:claude-opus-5-5 Assisted-by: Grok:grok-4.7 * Render ::NAME defaults by name and restore named constant defaults - default = ::NAME writes NAME into the text signature and stores Into::into(NAME) as the Rust value. - A one-field tuple-struct argument pattern such as Fildes(fd) names the parameter after its binding, so fcntl and termios keep destructuring. - zlib, lzma, syslog, mmap and select defaults use their named constants again instead of literals or <unrepresentable>. Assisted-by: Claude:claude-opus-5-5 Assisted-by: Grok:grok-4.7 * Name FromArgs fields after their Python parameters Drop #[pyarg(name = ...)] where the Python name is a valid Rust field name and rename the field instead. The option stays only for keywords (fn, loop, final, type, self) and the non-snake_case memLevel. Assisted-by: Claude:claude-opus-5-5 Assisted-by: Grok:grok-4.7 * Build literal defaults from Python values and type <unrepresentable> - default = LIT on a non-primitive field converts the literal through ToPyObject and TryFromObject when the argument is missing, so object fields no longer need a vm expression plus a matching py_default. - A bare default on a primitive integer, bool or float field renders its zero value. - DefaultRepr::Unrepresentable replaces the <unrepresentable> text, and py_default = "<unrepresentable>" is a compile error. - pyexpat.ParserCreate distinguishes a missing intern from intern=None, mmap AccessMode renders its value, and sqlite blobopen names rowid. Assisted-by: Claude:claude-opus-5-5 Assisted-by: Grok:grok-4.7 * Resolve native docstrings from the doc database at compile time - rustpython-doc stores its table as a sorted const slice with const fn lookups (get, get_attr, class_attr_doc) instead of a phf map, and the update-doc-db workflow emits that form. - Class, method, getset, module function and struct sequence docs, and per-class slot wrapper docs, are chosen by const lookups in the derive output. The database text wins over Rust doc comments. - Macros reach the table through rustpython_vm::__exports, so only the vm depends on rustpython-doc. - Remove expectedFailure markers from doctest and pydoc tests that pass. Assisted-by: Claude:claude-opus-5-5 Assisted-by: Grok:grok-4.7 * Keep time.gmtime, localtime and ctime without a text signature Their [seconds] parameter has no text signature, and unittest.mock relies on inspect.signature failing for them. Take OptionalOption again so the default stays unrepresentable, and treat None like a missing argument. Assisted-by: Claude:claude-opus-5-5 |
||
|
Jeong, YunWon
|
0ebc6570e3 |
Add payload views and Py handle shortcuts (#8812)
* Add payload views and Py handle shortcuts PyTuple::payload_slice, PyBytes::payload_bytes, and PyStr::payload_wtf8 expose the payload data. Common payload methods are forwarded from Py<T>. Assisted-by: Grok CLI:grok-4.7 * Call PyTuple::as_slice from payload_slice `&*self` tripped clippy borrow_deref_ref and needless_borrow. Assisted-by: Grok CLI:grok-4.7 * Use Element.is_empty in the truth-value check len() != 0 trips clippy::len_zero once is_empty is available. Assisted-by: Grok CLI:grok-4.7 * Drop redundant payload view aliases as_bytes, as_wtf8, and as_slice already forward to the payload methods. Assisted-by: Grok CLI:grok-4.7 * Forward more payload methods from Py handles Frame iframe, bytearray buffer, set add, exception args and traceback, and length or lookup helpers are available on Py<T>. Assisted-by: Grok CLI:grok-4.7 * Read type slots through payload in macros Generated subclass checks and MRO slot inheritance no longer use Deref to reach PyType.slots or attributes. Assisted-by: Grok CLI:grok-4.7 * Index a few tuples through as_slice These Py<PyTuple> indexes no longer rely on Deref to [T]. Assisted-by: Grok CLI:grok-4.7 * Read type slots through Py<PyType>::slots obj.class().slots and builtin type slots now call slots() instead of the payload field. Assisted-by: Grok CLI:grok-4.7 * Read type version and heap layout through Py<PyType> Inline-cache checks use tp_version_tag(). Heap-type weakref checks use heaptype_ext(). Assisted-by: Grok CLI:grok-4.7 * Call slots() on Py type handles Type creation, type methods, and frame specialization read slots through Py<PyType> instead of the payload field. Assisted-by: Grok CLI:grok-4.7 * Use as_bytes, as_slice, and traceback accessors Bytes lengths go through as_bytes(). Tuple len, emptiness, and iteration go through as_slice(). Exception handles use traceback(). Assisted-by: Grok CLI:grok-4.7 * Drop shortcuts that only forward payload methods Tuple length and iteration go through as_slice. Dict and list expose len. Set add stays on the payload. Assisted-by: Grok CLI:grok-4.7 * Drop dict and list len shortcuts len without is_empty fails clippy. Tuple element lookup uses as_slice().get instead of iter().nth. Assisted-by: Grok CLI:grok-4.7 |
||
|
Jeong, YunWon
|
e540b4152d |
Borrow exception and dict methods that only read self (#8808)
add_note, __reduce__, and __setstate__ take &Py<BaseException>. dict.__ror__ takes &Py<PyDict>. Callers drop the clone made for add_note. Assisted-by: Grok CLI:grok-4.7 |
||
|
Jeong, YunWon
|
4174a77cac |
Pass owned iterables through; host enumerate on Py<T> (#8806)
* Pass owned iterables through; host enumerate on Py<T> dict update, OrderedDict, defaultdict, itertools, and pickle pass an owned object to PyIter::try_from_object. #[pyclass(...)] impl Py<T> now emits PyClassImpl for T. enumerate's methods live on that impl. A bare impl Py<T> stays an extension. Assisted-by: Grok CLI:grok-4.7 * Host Self-free pyclass impls on Py<T> filter, Ellipsis, NotImplemented, bool, Placeholder, and NoDefault do not use Self as the payload type, so their pyclass impl can be Py<T>. Assisted-by: Grok CLI:grok-4.7 |
||
|
Jeong, YunWon
|
0940849029 |
Borrow tp_descr_get/tp_init arguments and by-value public helpers (#8803)
* Borrow tp_descr_get/tp_init arguments and by-value public helpers - DescrGetFunc and GetDescriptor::descr_get take (&PyObject, Option<&PyObject>, Option<&PyObject>); impls call .to_owned() where they store or return an argument. - InitFunc takes &PyObject and Initializer::init takes &Py<Self>. - VirtualMachine::get_attribute_opt, print_exception, call_get_descriptor(_specific) and other helpers flagged by clippy::needless_pass_by_value take &PyObject / &Py<T>; callers in crates/, examples/ and tests/ drop the clones made for the call. Assisted-by: Claude Code:claude-opus-5-5 Assisted-by: Grok CLI:grok-4.7 * Pass the borrowed zelf to internal_close in WindowsConsoleIO init Assisted-by: Claude Code:claude-opus-5-5 * Fix borrowed signatures missed by the rebase OrderedDict.init, classmethod_descriptor.descr_get, and the warnings get_attribute_opt callers still used owned handles. Assisted-by: Grok CLI:grok-4.7 |
||
|
Jeong, YunWonandLee Dogeon
|
afad92bf1b |
Add OrderedDict, marshal, inspect, and chflags bitflagset (#8780)
* Implement _collections.OrderedDict Extracted from #6799: native OrderedDict as a dict subclass, iterator mutation tracking, and mapping-view set operations. * Add OrderedDict, warnings source lines, and socket sendmsg Implement _collections.OrderedDict as a dict subclass with a linked list of insertion-order nodes. Pack struct F/D as native float/double complex pairs, and report unexpected __replace__ field names as a list repr. Register encodings.latin_1 at startup. Reject NUL in input() prompts. Copy non-exact type dicts in mapping order. Fetch warning source lines through _bless_my_loader and unicode splitlines. Keep the last warnings.filters list after deletion. Snapshot sendmsg sequences before converting buffers, parse ancillary ints with __index__, and dispatch AF_CAN by protocol. Store sys.trace and sys.profile on each ThreadSlot so _settraceallthreads/_setprofileallthreads are visible to sys.gettrace/getprofile on other threads. Use <frozen importlib._bootstrap> as frozen importlib co_filename. Keep HEAPTYPE socket objects GC-tracked. Assisted-by: Grok:grok-4.6 * Load marshal via readinto and dump sets in dumps order marshal.load fills a scratch buffer with readinto on a contiguous writable memoryview, and rejects a returned length other than the requested size. set and frozenset dumps write elements in sorted(v, key=marshal.dumps) order. Assisted-by: Grok:grok-4.6 * Add classmethod_descriptor and ABC IS_ABSTRACT flags Give METH_CLASS methods a distinct classmethod_descriptor type. Store Py_TPFLAGS_IS_ABSTRACT from __abstractmethods__ and look up that attribute only in the type's own dict. Expose defaultdict.default_factory as a member descriptor. Inherit bool's unused number slots from int. Mark set and frozenset __contains__ as METH_COEXIST. Parse classmethod, staticmethod, object.__new__, and heap type text signatures. Register interactive source in linecache. Assisted-by: Grok:grok-4.6 * address review: odict reuse, descr call, new wrapper Reuse odict node slots and walk the live list in next_after. Require both | operands to be dicts. Drop the odict iterator on GC clear. Bind classmethod descriptors through descr_get and update IS_ABSTRACT under the type lock. Give odict keys and items sq_contains. Walk tp_new_wrapper past new_wrapper so HEAPTYPE socket subclasses can call socket.__new__. Split FreeBSD Bluetooth protocol constants from Linux values. Assisted-by: Grok:grok-4.6 * Represent chflags bits with bitflagset Assisted-by: Grok:grok-4.6 * Represent access and mode bits with bitflagset Assisted-by: Grok:grok-4.6 * Continue unfinished REPL suites as incomplete input Assisted-by: Grok:grok-4.6 * Add pickle and os.stat inspect signatures Give _pickle.Pickler the clinic constructor text signature and accept fix_imports and buffer_callback as positional-or-keyword. Allow #[pyfunction(text_signature)] and use it so os.stat reports (path, *, dir_fd=None, follow_symlinks=True). Uncomment import _pickle and drop the pickle inspect expectedFailures. Assisted-by: Grok:grok-4.6 * Add Unpickler, dump/load, and lstat signatures Give _pickle Unpickler, dump, dumps, load, and loads clinic text signatures, and os.lstat (path, *, dir_fd=None). Assisted-by: Grok:grok-4.6 * Complete OrderedDict with insertion-order links Move the native OrderedDict into ordered_dict.rs and track order with a linked list of nodes, keeping mapping-view set operations. Assisted-by: Grok:grok-4.6 * address CI: stat casts, HCI_DATA_DIR, passing xfails Compare stat permission bits as Mode so Linux clippy does not flag a u32-to-u32 cast. Export HCI_DATA_DIR with the Bluetooth socket option values. Drop expectedFailure from tests that now pass. Assisted-by: Grok:grok-4.6 * Fix wasm teardown deadlock and bluetooth constants Assisted-by: Grok:grok-4.6 * Clear process-wide eval breaker bits in STW test Assisted-by: Grok:grok-4.6 * Drop exclusive-held panic that tripped wasm clippy Assisted-by: Grok:grok-4.6 * Fix OrderedDict contains and export Windows BDADDR Assisted-by: Grok:grok-4.6 * Add cmp_to_key and hash-once LRU cache _functools.cmp_to_key now returns KeyWrapper. LRU lookups store the key hash on _lru_list_elem so __hash__ is not called again on hits or eviction. Assisted-by: Grok:grok-4.6 * Allow subclass keywords on list, tuple, and float list.__init__ rejects keywords only when tp_new is still list's. tuple/float.__new__ reject them only when tp_init is still the base's. Assisted-by: Grok:grok-4.6 * Fix CI lint, test re-export, and structseq members Drop the duplicate clear_eval_breaker_for_test re-export. Format structseq imports. Expose struct sequence fields as readonly members so assignment raises "readonly attribute". Assisted-by: Grok:grok-4.6 * Allow subclass keywords on itertools and filter Positional-only constructors ignore extra keywords when a subclass has replaced tp_init, matching list/tuple/float. Assisted-by: Grok:grok-4.6 * Remove text_signature derive parameter Drop the string override on #[pyfunction] and #[pyclass]. Signatures are generated from the Rust parameter list only. Restore inspect expectedFailures that depended on the override. Assisted-by: Grok:grok-4.6 * Rename set contains_method to __contains__ Keep a borrowed contains helper for slots and C-API. The coexist pymethod is named __contains__. Assisted-by: Grok:grok-4.6 * Use MonitoringEvent instead of EVENT_* aliases Drop the unused MonitoringEvents bitflags and the EVENT_PY_START style mask constants. Call sites use the enum and .mask(). Assisted-by: Grok:grok-4.6 * Store warnings filters as PyListRef The list already serializes its elements. Drop the extra mutex around the cached list pointer. Assisted-by: Grok:grok-4.6 * Re-export clear_eval_breaker_for_test after rebase Keep the STW test helper visible without bringing back stop_bit_set. Assisted-by: Grok:grok-4.6 * Fix groupby reentrancy and islice arguments Give _grouper its own tgtkey and take currvalue after the key comparison. Accept __index__ on islice bounds, reject step 0, and drain the source up to stop. Assisted-by: Grok:grok-4.6 * SETREF warnings filters, onceregistry, and defaultaction Keep the last list, dict, or str after the module attributes are deleted by swapping the interpreter cache pointer. Do not store a None fallback for _warnings_context. Assisted-by: Grok:grok-4.6 * Iterate OrderedDict inner dict via next_entry &PyDict is not an iterator after the borrowed-ref signature change. Assisted-by: Grok:grok-4.6 * Remove expectedFailure from testBluetoothConstants Linux now exports BTPROTO_HCI and the rest of the set. Assisted-by: Grok:grok-4.6 * Borrow get_filter arguments as PyObject Match the warn_explicit call site that passes borrowed handles after the internal-helper borrow change. Assisted-by: Grok:grok-4.6 * Fix bool slots and SemHandle safety docs Leave unused bool number slots unset so inspect reports __add__ on int. Document SemHandle::from_raw safety. Assisted-by: Grok:grok-4.6 * Keep sendmsg errors and drop duplicate linecache Replace sendmsg iterable TypeError only. Register REPL source in linecache once per compiled input. Assisted-by: Grok:grok-4.6 * Nest REPL parse or-patterns for clippy Assisted-by: Grok:grok-4.6 * Mark pydoc os.stat signature test expectedFailure Assisted-by: Grok:grok-4.6 --------- Co-authored-by: Lee Dogeon <dev.moreal@gmail.com> |
||
|
Joshua Megnauth
|
86c7fa20c4 |
Fix missing workspace lints & fix lints (#8781)
Some of the workspace packages were missing the `[lints]` table. I enabled the lints then fixed the resulting errors. |
||
|
Jeong, YunWon
|
30ff0329a2 |
Borrow handles in internal helpers that only borrowed them (#8798)
Internal functions that took PyObjectRef or PyRef<T> by value but only borrowed it (reported by clippy::needless_pass_by_value) now take &PyObject or &Py<T>. Callers pass a borrow instead of an owned handle; 61 caller-side .clone()/.to_owned() calls made only to pass the argument are removed. Where a converted body already cloned the parameter, that .clone() became .to_owned(). Functions with #[py*] attributes, trait methods, and functions whose signature is fixed by a function pointer, getset or slot wrapper type are unchanged. PyFloat_FromString in crates/capi passes the borrowed object to parse_float_from_string without taking an owned reference. Assisted-by: Claude Code:claude-opus-5-5 Assisted-by: Grok CLI:grok-4.7 |
||
|
Jeong, YunWon
|
05f1386fca |
host_env: publish remaining Windows socket constants (#8792)
* host_env: publish remaining Windows socket constants Windows SOMAXCONN follows winsock2.h (0x7fffffff), not the Winsock 1.1 value windows-sys still exports. The socket module takes the signed INADDR group constants and the other Winsock names host_env already defines. Assisted-by: Grok * test_socket: expect Bluetooth constants to pass on Windows The Windows names this test reads are published now, so the blanket expectedFailure is an unexpected success there. Other platforms still miss BTPROTO_HCI and the rest of the non-Windows set. Assisted-by: Grok |
||
|
Jeong, YunWon
|
56df93a919 |
Replace borrowed &PyObjectRef and &PyRef<T> with &PyObject and &Py<T> (#8793)
Parameters, return types, closure annotations, Fn and Iterator bounds, trait methods and struct fields that borrowed an owning handle now borrow the object: &PyObjectRef -> &PyObject, &PyRef<T> / &PyXxxRef -> &Py<T>. Existing .clone() calls on those values became .to_owned(). - Move PyTuple::extract_tuple into impl Py<PyTuple>; FromPyTuple::from_pytuple takes &Py<PyTuple>. - DictIter and IntoIterator for &PyDict use &Py<PyDict>; apply_shared_ns takes &Py<PyDict>. - PyExactSizeIterator, calculate_suggestions, collection_repr and the cformat tuple helpers take iterators over &PyObject; callers map with |o| &**o. - Return types changed: TypeNamespace::as_dict, Context::cached_int, struct_error_type, PyBoundMethod::function_obj/self_obj, AsRef for ArgIndex. as_dict, function_obj and self_obj are no longer const fn. - asyncgenerator unbox, csv parse_lineterminator and _ctypes bytes parameter take &Py<T> instead of &T. Assisted-by: Claude Code:claude-opus-5-5 Assisted-by: Grok CLI:grok-4.7 |
||
|
Jeong, YunWon
|
3ac2afd044 | Start socket timeout after the sock snapshot (#8791) | ||
|
Jeong, YunWon
|
8e4f6d2ab8 |
host_env: export SIG*, sysconf, CLOCK_*, unix socket names (#8790)
* host_env: export SIG*, sysconf, CLOCK_*, unix socket names Re-export signal numbers, pathconf/sysconf names, clock ids, and the unix socket integers the VM still took from libc. Assisted-by: Claude * stdlib: take SIG*, sysconf, CLOCK_*, socket names from host_env signal, posix pathconf/sysconf enums, time clock ids, and the shared socket pyattrs re-export host_env instead of libc. Assisted-by: Claude * host_env: export remaining socket integers and Windows SHUT_* Alias WinSock SD_* as SHUT_* so the shared pyattr names resolve. Re-export the rest of the libc socket integers the VM still took from `use libc as c`. Assisted-by: Claude * stdlib: take remaining socket pyattrs from host_env Module integer names come from host_env. Types and syscalls stay on libc / WinSock. Assisted-by: Claude * stdlib: drop unused Windows socket constant imports pyattrs now take those names from host_env. Keep only the syscall names the shared body still reads as c::. Assisted-by: Claude |
||
|
Jeong, YunWon
|
cb3d5dbb1a |
Take &Py<T> in getset closures, try_value_with, slice and ssl helpers (#8787)
- `try_value_with` passes `&Py<T>` to its closure instead of `&T`, and its four callers follow. - structseq and pyexpat getset closures take `&Py<T>`. - `get_jit_args` takes `&Py<PyFunction>`. - `SaturatedSlice::with_slice` takes `&Py<PySlice>`; `to_saturated` moves from `impl PySlice` to `impl Py<PySlice>`. - The ssl compat helpers take `&Py<PySSLSocket>`; `do_handshake`, `read`, `write` and `shutdown` take `zelf: &Py<Self>` so they can pass it on. An inherent `impl Py<PySSLSocket>` is not possible in the stdlib crate. The csv `&PyDialect` sites keep the payload reference: the dialect is stored by value, so no `Py<PyDialect>` is available there. Assisted-by: Claude Opus 5 (1M context) Assisted-by: grok-4.7 |
||
|
Jeong, YunWon
|
2b8c095e11 |
address review: keep sendall deadline; isolate STW test (#8786)
Pass sendall's absolute Deadline through sock_op_timeout_err. Add a test that trips eval_breaker_tripped with STOP_BIT clear. Assisted-by: Grok:4.6 |
||
|
Jeong, YunWon
|
979ea4c4dc |
host_env: export vsock, wasm socket, Darwin flags (#8779)
* host_env: export vsock, wasm socket, Darwin flags Publish vsock/UDPLITE/SOL fallbacks, wasm socket and errno constants, remaining BSD chflags, and S_IFDOOR/PORT/WHT. Assisted-by: Claude * stdlib: take vsock, wasm socket, Darwin flags from host_env _socket, wasm _socket/_io/errno, _stat, and openssl alerts now re-export the host_env names. Assisted-by: Claude * host_env: keep wasm socket/errno off emscripten wasm32-unknown-emscripten is unix, so the stub modules must be unknown-unknown (and WASI) only. Assisted-by: Claude * stdlib: match socket module_def cfg to the module Assisted-by: Claude * host_env: drop duplicate Windows SOL_UDP The IANA fallback also compiled on Windows, colliding with the WinSock definition. Assisted-by: Claude |
||
|
Jeong, YunWon
|
eb1546ae08 |
posix: add os.forkpty so pty.spawn does not hang (#8782)
* posix: add os.forkpty pty.fork prefers forkpty so a nested spawn is a new session with its own controlling tty. The openpty+login_tty fallback hung the jsontests collector. Assisted-by: Claude * select: rebuild fd sets after EINTR select(2) updates the sets in place. Retrying an interrupted call with empty sets and a NULL timeout blocks forever. Assisted-by: Claude * http.server: keep CGI off the body-read timeout A millisecond handler timeout races the client thread writing the request and is inherited across CGI fork, which RSTs the client under load. Assisted-by: Claude * Revert "http.server: keep CGI off the body-read timeout" Lib edits outside a CPython copy are not allowed. Assisted-by: Claude * host_env: note why forkpty is libc not nix Assisted-by: Claude * benches: run deepcopy 10x more iterations Assisted-by: Claude |
||
|
Jeong, YunWon
|
f56bfc9a5c |
Show stdin traceback lines; omit StopIteration None args (#8778)
* Show stdin traceback lines; omit StopIteration None args Register interactive <stdin> source in linecache so REPL tracebacks include the source line and caret. Build StopIteration with empty args when the value is None, and wrap monitoring STOP_ITERATION the same way. Unskip REPL stderr flush tests and pdb Flaky doctests. Assisted-by: Grok:4.6 * Take &Py<PyCode> in register_code_in_linecache Assisted-by: Grok:4.6 * Count Unknown closing braces in tokenize EOF check `$` inside an f-string interpolation is TokenKind::Unknown, so the matching `}` is Unknown rather than Rbrace and generate_tokens raised TokenError. Also convert &Py<PyCode> to PyObjectRef explicitly when registering linecache. Assisted-by: Grok:4.6 * Unskip test_pdb.PdbTestCase.test_zipimport The test now passes, so the expectedFailure is an unexpected success under CI. Assisted-by: Grok:4.6 * Park STW threads; poll sockets after timeout deadline Check per-thread stop_requested in eval_breaker_tripped so workers still park when another interpreter clears STOP_BIT. Start socket timeout after the fd snapshot, and poll once with timeout 0 when the deadline has already passed, so a readable socket is not dropped as timed out. Assisted-by: Grok:4.6 |
||
|
Jeong, YunWon
|
7b018871a3 |
host_env: export remaining stdlib constants (#8760)
* host_env: export winsound, spawn, and remaining SSL_OP flags Assisted-by: Claude * stdlib: take winsound, nt spawn, and SSL_OP names from host_env Assisted-by: Claude * host_env: export ssl alerts, sysexits, socket extras Assisted-by: Claude * stdlib: take remaining host_env and common constants Assisted-by: Claude * Remove unused workspace exitcode dependency Assisted-by: Claude * host_env: export PROTO, access, reparse, wait flags Assisted-by: Claude * stdlib: take PROTO, access, reparse, wait from host_env Assisted-by: Claude |
||
|
Jeong, YunWon
|
54420c4471 |
Drop serial MP CI retries and killpg overlay (#8756)
Multiprocessing fork/spawn/forkserver now run in the main -j $CORES CPython test job. Remove expectedFailureIf for os.killpg now that killpg exists. Assisted-by: Grok:4.6 |
||
|
Jeong, YunWon
|
f759c18a0e |
Take &Py<T> instead of bare payload references (#8753)
* Drop gc_traversal from CodSpeed benches Remove benches/benchmarks/gc_traversal.py so CodSpeed no longer includes that execution benchmark. Assisted-by: Grok:4.6 * Take &Py<T> instead of bare payload references Change `&PyCode`, `&PyInt`, `&PyTuple`, `&PySlice`, `&PyStr`, `&PyFunction`, `&PyList` and `&PyDict` in function parameters, closure types and local type annotations to `&Py<T>` where every caller already holds a `Py<T>`. Assisted-by: Claude Fable 5.1 Assisted-by: grok-4.6 |
||
|
hyoinandout
|
c857f3acfd |
mmap: correct find/rfind for empty subsequences (#8749)
get_find_range clamped end up to start, so an inverted range such as (2, 1) became the empty-but-valid range (2, 2). A real subsequence then found nothing there, which looked right, but the empty subsequence matched and both methods reported a hit where CPython reports -1. rfind also returned the start of the range for an empty subsequence, because its body was copied from find and the returned edge was never adjusted. An empty subsequence matches at the edge the scan begins from, so a reverse scan must report the end. Dropping the clamp lets an inverted range stay inverted and be rejected explicitly. find and rfind now share find_inner, which keeps the two directions from drifting apart again, and it checks the mmap is open before inspecting the subsequence, so find(b"") on a closed mmap raises ValueError instead of returning a position. m = mmap.mmap(-1, 3); m[:] = b"abc" m.rfind(b"") # was 0, now 3 m.rfind(b"", 1, 2) # was 1, now 2 m.find(b"", 2, 1) # was 2, now -1 m.rfind(b"", 2, 1) # was 2, now -1 Verified on macOS arm64 against CPython 3.14.7 with a 19-case table covering both methods over empty and non-empty subsequences, clamped, negative and inverted ranges: 8 mismatches before, none after. The added snippet assertions fail on the unpatched build and pass on CPython 3.14.7 as well. test_mmap, cargo fmt --check and cargo clippy -p rustpython-stdlib -D warnings are clean. Fixes #8722 Assisted-by: Claude Code:claude-opus-5 |
||
|
Jeong, YunWon
|
7bc3ef6c0a |
ExceptionGroup fields, OSError use-init, and live f_lineno (#8738)
* Add ExceptionGroup fields and OSError use-init BaseExceptionGroup stores msg, excs, and excs_str; args keep the original sequence. OSError defers parse when a subclass overrides only __init__. characters_written can be deleted. Exception args/traceback/cause/context reject invalid setattr/delattr. UnicodeError.object is a deletable field. except* remainder projection copies traceback/context/cause. SyntaxError.__init__ clears omitted location attributes. Assisted-by: Grok:grok-4.6 * Fix OSError payload init and UnicodeError construction new_payload_exception now runs slot_new so BlockingIOError parses characters_written. new_exception falls back to invoke_exception for extra-payload types. Mixed tab/space UnexpectedIndentation is TabError. AttributeError pickle omits obj. Assisted-by: Grok:grok-4.6 * address review: classify TabError from error line Assisted-by: Grok:grok-4.6 * Add new_simple_exception; fill SyntaxError and f_lineno new_exception constructs extra-payload types; BaseException-sized types use new_simple_exception. Non-UTF-8 SyntaxError sets location and text. Codegen errors load text via ProgramText. Keep Missing parentheses capitalization. f-string adjacent atoms report a missing comma. f_lineno may be None; lasti is not restored on unwind. Assisted-by: Grok:grok-4.6 * address review: skip f-string continuation keywords Assisted-by: Grok:grok-4.6 * Fix clippy map_unwrap_or in Non-UTF-8 offset Assisted-by: Grok:grok-4.6 * Rename encode error ctor and exception setters new_unicode_encode_error_real is new_unicode_encode_error. set___traceback__/__cause__/__context__ are pygetset setters; Rust uses set_traceback/set_cause/set_context. Assisted-by: Grok:grok-4.6 * Use set_context in native pickle Assisted-by: Grok:grok-4.6 * Collapse nested if in asyncio traceback restore Assisted-by: Grok:grok-4.6 * Use current instruction for f_lineno addr2line lasti is stored as the next instruction index. addr2line of that byte offset is past RETURN_VALUE, so f_lineno was None and stdlib_threading_current_frames hung on leftover workers. Assisted-by: Grok:grok-4.6 * Look up format args, dir class, and frame builtins Consume printf args before rejecting unknown conversions. object.__dir__ uses getattr of __class__/__dict__. _PyEval_GetBuiltin looks up iter/__import__ on the frame. Assisted-by: Grok:grok-4.6 * Fix builtin lookups after rebase and unxfail passing tests Assisted-by: Grok:grok-4.6 * Restore RERAISE lasti and SyntaxError locations RERAISE with oparg restores lasti from the exception table before unwind, so leftover f_lineno matches the original raise. Handler lookup still uses the raising instruction. Indentation errors outrank print/exec missing-parentheses. BOM-with-non-utf8, bytes literals, and global+nonlocal now report the same offsets as the parser/symtable. object.__dir__ keeps string keys; type.__type_params__ skips the getset wrapper stored on builtin types. Assisted-by: Grok:grok-4.6 * Unxfail syntax/import tests and bind jump locals IMPORT_NAME passes fromlist None instead of an empty tuple. Jumping f_lineno binds None into unbound localsplus slots. str lstrip/rstrip copy subclasses to exact str when unchanged. Assisted-by: Grok:grok-4.6 * Fix import_from callers and rename frame lineno sqlite3.iterdump now passes fromlist as a PyObject. FrameObject::lineno is the i32 helper; f_lineno is the getset. Assisted-by: Grok:grok-4.6 * Pass sqlite fromlist without Into conversion from_list.into() could not infer Into<PyObjectRef> at the import_from call. The typed tuple already converts. Assisted-by: Grok:grok-4.6 |
||
|
Jeong, YunWon
|
f1b8b09492 |
Don't hold socket locks across blocking I/O (#8746)
accept/recv/send kept the socket RwLock during the syscall. After fork, the child copied that held lock, so _ResourceSharer._afterfork hung in listener.close(). Unskip fork connection pickling, pool task ref release, and spawn thread-pool terminate. Add extra_tests for fork+accept close and fork after DupFd. Assisted-by: Grok:4.6 |
||
|
Jeong, YunWonandLee Dogeon
|
d02d81e740 |
perf: VM and runtime hot paths from pyperformance (#8737)
* time: use CLOCK_UPTIME_RAW for monotonic/perf_counter on Apple
CLOCK_MONOTONIC keeps advancing during system sleep on macOS, unlike
CPython's mach_absolute_time()-backed perf_counter/monotonic. Benchmarks
that spanned a laptop sleep recorded thousands of seconds for a single
iteration (pyperformance float/fannkuch showed ~1000x slowdowns).
Use CLOCK_UPTIME_RAW (equivalent to mach_absolute_time) on Apple targets
and report it as such in time.get_clock_info().
Assisted-by: Claude Code:claude-fable-5-1
* vm: derive f_lineno lazily instead of tracking prev_line per instruction
The dispatch loop wrote prev_line (a locations lookup plus a Cell store)
on every instruction only so f_lineno could read it. f_lineno now derives
the line from lasti and code.locations, prev_line is only maintained on
the cold tracing path for 'line' event de-duplication, the two use_tracing
reads per instruction are merged into one, and the pre-dispatch lasti
reload is replaced by the value just stored.
When a trace function is installed on an already-running frame (bdb
stepping back into an untraced caller, frame.f_trace assignment, a local
trace returned from the 'call' event on a resumed generator) prev_line is
synced from lasti so no spurious 'line' event fires.
Assisted-by: Claude Code:claude-fable-5-1
* vm: coalesce per-call thread-locals into one FRAME_SLOT_CACHE
set_current_frame ran on every frame enter/exit and touched three separate
thread_local! statics (each a _tlv_get_addr call on macOS). Merge them into
a single thread-local struct so the hot path does one TLS access.
Assisted-by: Claude Code:claude-fable-5-1
* vm: fold stop/finalizing checks into EVAL_BREAKER
eval_breaker_tripped ran three checks per bytecode: the finalizing flag,
a thread-local stop_requested lookup, and the EVAL_BREAKER load. Add
STOP_BIT and FINALIZING_BIT so the per-instruction check is a single
relaxed atomic load; the slow path (check_signals) still re-derives the
exact per-thread answer. The bits are set alongside the per-thread flags
and cleared under the stop-the-world exclusion.
Assisted-by: Claude Code:claude-fable-5-1
* gc: set the TRACKED bit with a plain store for freshly allocated objects
gc_bits is zero for a fresh allocation (both the freelist reuse path and
PyInner::new zero it), so track_new_object can initialise the bit with a
relaxed store instead of a fetch_or. Re-tracking resurrected objects keeps
the read-modify-write variant.
Assisted-by: Claude Code:claude-fable-5-1
* vm: skip thread exception publish when pushing/popping a None exc slot
Generator/coroutine resume pushes and pops an exc_info slot on every
send. When that slot is None the value returned by topmost_exception
cannot change (it skips None entries), so the thread-local exception
publish in push_exception/pop_exception is unnecessary.
Assisted-by: Claude Code:claude-fable-5-1
* vm: allocation-free keyword-call fast path and super() MRO walk
Keyword calls to plain functions where every parameter is filled exactly
once are reordered in place into the fast-locals layout (no IndexMap, no
keyword name clones); any mismatch falls back to the existing slow path
so error messages are unchanged. super().attr now walks the type's MRO
by reference instead of cloning every class into two Vecs per lookup.
Assisted-by: Claude Code:claude-fable-5-1
* builtins: cheaper enumerate and zip iteration
enumerate kept its counter as a BigInt behind a write lock and did bignum
arithmetic on every item; use a machine integer that promotes to BigInt
only for huge/negative starts or on overflow. zip pre-sizes its per-item
Vec so the common two-iterable case allocates once.
Assisted-by: Claude Code:claude-fable-5-1
* _asyncio: cache module objects and use native set/dict operations
Task registration re-resolved _asyncio module attributes and re-imported
contextvars on every task; the eager path additionally went through generic
call_method for set add/discard. Cache the module objects per thread
(validated by identity), split the contextvars cache from the task
containers, and use PySet/PyDict methods directly.
Assisted-by: Claude Code:claude-fable-5-1
* perf: use mimalloc as global allocator for rustpython CLI
Adds mimalloc (0.1.52, default-features = false) as the #[global_allocator]
for the rustpython binary, gated behind a new `mimalloc` feature enabled
by default on non-wasm targets (dep declared under
target.'cfg(not(target_arch = "wasm32"))'.dependencies, so wasm builds
never pull it in regardless of default-features).
Profiling a `t = (a, b, c)` loop showed malloc/free at ~40% of samples on
macOS with the system allocator. Switching to mimalloc gives consistent
wins across small-object-heavy workloads, medians over 5-9 interleaved
A/B rounds (release build, this machine):
tuple (5M `(a,b,c)`) 1.020s -> 0.863s (1.18x)
4-thread tuple (5M total) 1.643s -> 1.382s (1.19x)
class instances (2M) 0.874s -> 0.797s (1.10x)
dict-heavy (200k x 20) 2.196s -> 1.992s (1.10x)
string-heavy (join x 20) 0.662s -> 0.592s (1.12x)
nqueens (pyperformance) 0.327s -> 0.286s (1.14x)
chaos (pyperformance) 0.204s -> 0.191s (1.06x)
richards (pyperformance) 0.154s -> 0.151s (1.02x)
go (pyperformance) 0.411s -> 0.405s (1.01x)
listcomp / fib ~neutral (<=1%)
Peak RSS (/usr/bin/time -l): instances workload roughly unchanged
(29.0MB -> 29.1MB); dict-heavy workload actually drops (155MB -> 106MB),
likely because mimalloc reclaims freed pages more aggressively than the
system allocator across repeated large-dict churn.
`cargo build` and `cargo build --release` are warning-free with the
default feature set, and `cargo build --release --no-default-features
--features stdlib,threading,importlib,stdio,host_env` (mimalloc
excluded) also builds cleanly. CI's wasm32-wasip1 and other
--no-default-features build matrices in .github/workflows already omit
`mimalloc` from their feature lists and target wasm via the
cfg(not(wasm32)) dependency gate, so no workflow changes needed.
`rustpython -m test -j4 test_gc test_weakref test_threading test_dict
test_list test_tuple test_str test_bytes test_memoryview test_ctypes
test_mmap test_array` passes identically (2541 tests, 241 skipped) on
both the baseline and mimalloc binaries.
Assisted-by: Claude Code:claude-fable-5-1
* vm: cap dict/set table shrink and compact holes on resize
asyncio's eager task path (and its per-task bookkeeping sets in general)
repeatedly adds and discards distinct task objects from a handful of
long-lived sets (_eager_tasks, _scheduled_tasks). Two dict_inner issues
turned that steady-state churn into the dominant cost of async_tree_eager:
1. resize() picked a target size of `used * 2` with no floor, so once a
churny set settled around 1-2 live entries it shrank to a table that
the very next insert would already overflow, forcing a resize on
almost every single add. Clamping to the original minimum table size
(8) restores the headroom dict/set start with.
2. `entries` only ever grows (holes from deletions are never reclaimed),
so resize()'s rebuild pass scanned every entry ever inserted into the
dict's lifetime, not just the live ones - a set with a stable working
set size but heavy churn grew this scan, and the dict's memory, without
bound. resize() now compacts holes out of `entries` first, matching
CPython's compacting hash table and keeping the shape-stamp fast path
(which already requires a hole-free layout) reachable more often.
Measured with a standalone async_tree micro-benchmark (6 levels x 6
branches, pyperformance's bm_async_tree "none"/"eager" shape) run inside
one process for 10 asyncio.run() iterations, comparing medians of
interleaved binary runs on an otherwise idle machine:
- eager: ~2.91s -> ~0.86s median (about 3.4x)
- none: ~1.98s -> ~1.23s median (about 1.6x)
A single fresh-process run (no in-process accumulation) still improves
by roughly 15-20% for eager, since the resize-storm from point 1 hits
even a single run's worth of task churn.
test_asyncio (34 files, all passing), test_dict, test_set, test_weakref,
test_weakset, test_ordered_dict, test_gc, test_descr, test_class, and
`cargo test -p rustpython-vm -p rustpython-stdlib` all pass.
Assisted-by: Claude Code:claude-fable-5-1
* _functools: add native _lru_cache_wrapper
RustPython had no native _functools._lru_cache_wrapper, so
functools.lru_cache always fell back to the pure-Python implementation
in Lib/functools.py (RLock, linked-list bookkeeping, closures). That
fallback dominates typing.Protocol's __instancecheck__ path: it calls
inspect.getattr_static, whose _shadowed_dict is itself lru_cache'd, so
every isinstance() check against a runtime-checkable protocol paid for
several pure-Python cache-wrapper calls plus lock acquisitions.
Add a native pyclass mirroring CPython's C _lru_cache_wrapper: a
reentrant lock scoped only around the cache lookup/insert steps (not
held across the wrapped call, matching CPython's threading behavior),
LRU ordering piggybacked on the cache dict's own insertion order
(move-to-end via del+reinsert, oldest evicted from the front) so no
separate key-comparison structure is needed, and __copy__/__deepcopy__
returning self to match CPython since __reduce__ (used only for
pickling) requires __qualname__, which wrapped non-function callables
such as functools.partial don't have.
Measured with a standalone script reproducing pyperformance's
bm_typing_runtime_protocols isinstance() loop (300 loops, medians of
5 interleaved runs): 0.260s -> 0.182s, about 1.43x faster. In
isolation, a cache-hit through lru_cache went from ~0.31s to ~0.07s
for 200k calls, now faster than calling the underlying function
uncached (matching CPython's C accelerator).
Verified with test_typing, test_abc, test_inspect, test_functools,
test_weakref, test_isinstance, and test_descr (all pass, matching the
pre-existing baseline, including test_functools's already-present
TestLRUC suite which now actually exercises the native implementation).
Assisted-by: Claude Code:claude-fable-5-1
* sre_engine: ASCII fast path for \d \s \w unicode category checks
is_uni_digit/is_uni_space/is_uni_alnum went through icu_properties'
GeneralCategory table lookup for every character, even the common
ASCII case. Add a `ch < 0x80` fast path that resolves directly via
std's ascii predicates, falling back to the full table only for
non-ASCII code points (Latin-1 supplement chars like 'A-umlaut' still
route through the table since they are alphanumeric but not ASCII).
Measured on a tight `re.findall(r"\w+", ...)` / `re.sub` loop over
mostly-ASCII text (interleaved A/B, 4 rounds, medians):
re.sub: 1.178s -> 1.134s (~3.7% faster)
re.findall: 1.110s -> 1.062s (~4.3% faster)
Assisted-by: Claude Code:claude-fable-5-1
* vm: skip VM attach for no-op generator/coroutine __del__
Generator, coroutine, and async-generator dealloc always went through
drop_slow_inner's call_slot_del, which unconditionally attaches to a VM
(with_vm -> begin_interpreter_section -> per-interpreter thread-slot
hashmap lookup) before running del(), even though del() itself is a
documented no-op once the object is already closed or running - the
common case for a generator/coroutine consumed to completion.
Add PyTypeSlots::del_needed, an optional VM-free fast predicate checked
before call_slot_del/with_vm; generator, coroutine, and async_generator
install a check mirroring their own del() no-op branches, letting
drop_slow_inner and try_call_finalizer skip the VM attach entirely for
finished objects.
Standalone microbenchmarks (median of 5 interleaved runs):
- create+drain a 3-yield generator, 2M times: 1.099s -> 0.997s (-9%)
- pyperformance bm_generators shape (yield-from tree traversal): -4%
- pyperformance bm_coroutines shape (recursive fibonacci coroutines): -13%
Assisted-by: Claude Code:claude-fable-5-1
* _asyncio: skip full __import__ dispatch on module-cache hot path
with_asyncio_cache/with_contextvars_cache called vm.import("_asyncio"/
"contextvars", 0) on every Task/Future registration just to validate
their module-identity cache, even on the (overwhelming) hit path. That
still pays the full __import__ builtin dispatch (attribute lookup on
builtins, FuncArgs construction, native slot_call, ImportArgs parsing,
import_module_level) instead of a cheap sys.modules lookup. Added
fast_import_cached(), which checks sys.modules directly and only falls
back to vm.import on a miss (module missing/reloaded).
Measured with a standalone asyncio_tree-style microbenchmark
(6 levels x 6 branches, several asyncio.run() iterations per process,
median of 5 interleaved A/B rounds):
none: 1.045s -> 0.895s (~14% faster)
eager: 0.716s -> 0.591s (~17% faster)
eager_tg: 1.023s -> 0.849s (~17% faster)
test_asyncio (all files), test_contextvars (module absent from this
checkout, pre-existing), test_weakref, test_weakset all pass/unchanged.
Assisted-by: Claude Code:claude-fable-5-1
* vm: skip VM-attach section in with_vm when already on the current interpreter
Object teardown (drop_slow_inner -> call_slot_del, try_call_finalizer,
weakref callbacks) calls with_vm(), which unconditionally went through
set_current_vm -> begin_interpreter_section: an INTERP_THREAD_SLOTS
HashMap lookup plus an Arc clone, even in the overwhelmingly common
case where the dropping thread is already attached to the exact
interpreter that owns the object (a __del__ object or weakref target
going out of scope mid-bytecode).
with_vm now checks whether the object's owning interpreter is already
the one on top of VM_STACK. Only the top of VM_STACK is ever ATTACHED
on a thread (every push is preceded by an attach, every pop restores
the enclosing attach), so when it matches we call f(vm) directly and
skip begin_interpreter_section/set_current_vm entirely. The rare
not-attached/different-interpreter cases (nested subinterpreters,
thread outside any VM, shutdown) fall through to the original
attach/detach path unchanged.
Interleaved release-build measurements (5 rounds, medians), plain-Rust
micro-scripts, no pyperf:
- 2M create+drop of a __del__ object: ~0.55s -> ~0.46s (~16% faster)
- 2M create+drop with a weakref callback: ~1.14s -> ~1.06s (~7% faster)
- 2M io.StringIO() create+drop: ~0.734s -> ~0.697s (~5% faster)
- fib(30) and a 5M-iteration tuple-unpack loop: unchanged (noise-level)
`sample` profiles of the __del__ benchmark confirm
begin_interpreter_section/ensure_thread_slot frames disappear after
the change.
Verified: `cargo test -p rustpython-vm --release`, and
`-m test -j4 test_gc test_weakref test_finalization test_threading
test_thread test_concurrent_futures test_os test_signal test_generators
test_io test_atexit test_sys` (test_interpreters module absent in this
tree, pre-existing). Also a 4-thread stress script hammering __del__
object creation against a concurrent gc.collect() loop for 10s, and a
threading+queue handoff dropping a __del__ object after cross-thread
transfer, both pass.
Assisted-by: Claude Code:claude-fable-5-1
* vm: general sys.modules fast path for vm.import
VirtualMachine::import always resolved and called builtins.__import__
(FuncArgs construction, native slot_call, ImportArgs::from_args, then
import_module_level) even when the module was already cached in
sys.modules. ~63 Rust stdlib call sites hit this on hot paths
(copyreg, pickle's reduce_newobj, warnings, asyncio, etc).
Add VirtualMachine::try_import_cached: for a plain absolute import
(level 0, empty from-list), if builtins.__import__ is still the
original import function (compared by identity against vm.import_func)
and the module -- or, for a dotted name, its top-level package -- is
already present and fully initialized in sys.modules, return it
directly, skipping the __import__ dispatch and import_module_level
entirely. Falls through to the existing slow path whenever the module
is uncached, still initializing, or __import__ has been overridden by
user code, so test_import/test_importlib/test_builtin overrides still
work.
Exposed import::is_module_initializing (refactored out of
import_ensure_initialized) so the fast path can check
__spec__._initializing the same way the slow path does.
Removes _asyncio's local fast_import_cached workaround (added in
cb2064f0e) now that vm.import itself has the same fast path generally.
Measured (interleaved A/B, 5 rounds, pickle.dumps of a plain object,
which hits copyreg import via reduce_newobj): ~2% wall-clock
improvement per call (baseline ~4.75s / 100k dumps vs ~4.66s
candidate), consistent across all rounds. Verified: overridden
__import__ still invoked, a replaced sys.modules is read through the
same accessor, and a self-importing/initializing module is not handed
out early -- all match baseline behavior.
test suite (test_import test_importlib test_builtin test_pkg
test_pkgutil test_runpy test_zipimport test_site test_warnings
test_copy test_pickle test_asyncio.test_tasks test_threaded_import
test_sys): 13/14 files pass; test_threaded_import fails identically on
baseline (ModuleNotFoundError, pre-existing, unrelated to this change).
Assisted-by: Claude Code:claude-fable-5-1
* _asyncio: bypass generic context.run dispatch in task_eager_start
task_eager_start ran the first step via context.run(coro.send, None),
which builds a bound `send` method object and then re-enters the
generic Callable/FuncArgs dispatch just to invoke context.run's own
generic call. Since eager tasks in async_tree_eager complete almost
entirely synchronously (no event-loop scheduling needed), this
per-task dispatch overhead dominated. Enter/exit the native PyContext
directly and call coro.send via a single call_method instead,
falling back to the generic path for non-native Context subclasses.
async_tree_eager median: ~705ms -> ~697ms across 5 interleaved A/B
rounds (~1% eager-only win); async_tree (none) unaffected (noise-level
delta). test_asyncio (all files), test_weakref, test_weakset pass;
pre-existing test_contextvars module-not-found failure unchanged from
baseline.
Assisted-by: Claude Code:claude-fable-5-1
* vm: drop scopeguard::defer! from generator/coroutine resume hot path
resume_gen_frame restored owner/exc-slot/frame-chain state via a
scopeguard::defer! closure that ran on every exit (Ok, Err, or panic).
Since the closure's body is the function's only tail expression, a
plain captured result plus explicit cleanup (the same shape with_frame
already uses for the analogous call path) covers both normal exit
paths identically; only the recursion-depth decrement keeps a
scopeguard, matching with_frame's precedent that this codebase does
not otherwise guarantee generator/frame state consistency across a
Rust panic (panic=unwind is not used to implement any Python-level
control flow here).
Delegation-chain benchmark (chain.py, forwarding N values through a
yield-from chain) measures a small but consistent win: ~2-5% less
wall time per forwarded value at depths 1/4/16 across interleaved A/B
rounds. Single-generator create+finish cost is a wash.
Assisted-by: Claude Code:claude-fable-5-1
* vm: run generator bodies through the call trampoline
`Py<FrameObject>::resume` built its `ExecutingFrame` with
`tailcall_enabled: false`, because a `TailCall` result had nowhere to go:
the trampoline was reachable only from `run_frame_fast`, whose entry frame
lives on the data stack and can never yield. So every ordinary Python call
made from inside a generator or coroutine body re-entered `run_frame_fast`
recursively instead of being flattened.
Generalize the trampoline to drive a heap-resident entry frame whose
enter/exit bookkeeping belongs to its caller, and whose `Yield` is a valid
terminal result rather than a panic. `FrameKind` now says, per frame, what
the trampoline owes it: `Entry` (exit it, caller frees the storage),
`GenEntry` (`resume_gen_frame` linked it and will unlink it; a `Yield` is
the trampoline's result) or `Callee` (exit it and free its storage). The
three near-identical dispatch arms collapse into one `Step::Finished` /
`Step::Deliver` loop.
The suspended-frame stack moves onto the VM and is reused across
invocations: a generator body enters the trampoline once per resume, so a
`Vec::with_capacity(8)` there cost a malloc/free per yield (+27 ns/resume
when first measured). Invocations nest strictly LIFO, so each owns the
region above the length it finds on entry.
Neutral on its own — a generator body's first call was the only one not
already flattened, since the callee's own frame runs under the trampoline
either way. It is what lets SEND/FOR_ITER hand a generator frame to the
same loop. Measured (chain of `yield from`, 1-yield generator create+finish,
bm_generators, bm_coroutines, async_tree none/eager): within noise, except
generator create+finish 625 ns -> 603 ns.
Assisted-by: Claude Code:claude-fable-5-1
* vm: run a `yield from` chain in one eval loop
`SEND` on a suspended generator recursed: `Coro::send` -> `resume_gen_frame`
-> a fresh `ExecutingFrame::run` per level, and the value came back the same
way. Measured at ~55 ns per level of delegation, against CPython's ~5.8 ns.
Only about 5 ns of that is the resume bookkeeping itself (measured by
stubbing out the claim, the exception slot, the owner swap and the recursion
checks). The rest is what a delegating frame does per value: leave and
re-enter the eval loop, and dispatch `RESUME`, `JUMP_BACKWARD_NO_INTERRUPT`,
`SEND` and `YIELD_VALUE` — four instructions that, in the canonical
`yield from` / `await` shape, only hand a value along.
So the trampoline now runs the chain itself. `SendGen` parks its frame and
returns `ExecutionResult::GenResume`; the trampoline claims and links the
sub-generator (`coroutine::flat_resume_enter`, the first half of what
`Coro::send` does around a resume) and runs its frame in the same loop. On
the way down, every frame it finds already suspended at a `yield from` whose
delegate can be resumed is parked without being run at all
(`yield_from_delegate`); on the way back up, a yielded value is re-yielded
for each such frame by parking its `lasti` past the `YIELD_VALUE`
(`park_after_yield_from`) instead of running it. A whole chain therefore
costs one claim/link/unlink per level and no instruction dispatch at all,
with `Py<FrameObject>::resume` extending the same treatment to the outermost
level, which is where `next()` and asyncio's task step enter a chain.
Each level is still claimed, linked and unlinked exactly as a recursive
`Coro::send` would, so `gi_running`, `f_back`, `gi_frame`, `gi_yieldfrom`,
`sys._getframe`, tracebacks, `throw`, `close`, PEP 479 and the
`RecursionError` depth are unchanged; `resume_gen_frame` and the flat path
share `gen_frame_link`/`gen_frame_unlink` and the invariants are documented
on `FlatResume`. Everything else keeps the recursive path: tracing or
monitoring on (both would see the skipped instructions), `throw`/`close`,
async generators, subclasses, non-generator iterators, `FOR_ITER`, a `SEND`
that is not a `yield from`, and a delegate that has not started or is not
itself delegating — the last two because one lone level costs more to hand
to the trampoline than to send into from the eval loop the caller is in.
Measured (median of 5, ns/value or seconds):
chain of `yield from`, depth 16 911 -> 479 -47%
chain of `yield from`, depth 4 212 -> 158 -26%
bm_generators 0.272 -> 0.187 -31%
richards 0.772 -> 0.771 ~0
bm_coroutines 0.0129 -> 0.0130 +0.8%
async_tree none ~neutral
chain depth 1 (no delegation) 81.1 -> 83.6 +3.1%
fib(24), calls through trampoline 0.0180 -> 0.0183 +1.7%
Per level of delegation: 55.3 ns -> 26.4 ns (CPython 3.14: 5.8 ns).
Assisted-by: Claude Code:claude-fable-5-1
* vm: reset the oparg prefix when a signal unwinds a frame
An `EXTENDED_ARG` leaves its prefix in `arg_state` for the instruction that
follows it. The signal path in the eval loop unwinds to a handler and
`continue`s straight back to the top, skipping the loop's own
`arg_state.reset()`, so the handler's first instruction was decoded with the
interrupted instruction's prefix still applied. Every other path that jumps
somewhere new — the `set_f_lineno` restart just above — resets first.
Reachable whenever a signal arrives on the instruction right after an
`EXTENDED_ARG` in a frame that has a handler: a `KeyboardInterrupt` into a
`yield from` chain panicked with `InvalidBytecode` in roughly half of the
runs of a 16-deep chain, and now does not in ten out of ten.
Assisted-by: Claude Code:claude-fable-5-1
* bytecode: fetch a code unit with one 16-bit acquire load
The dispatch loop read the opcode with an Acquire `AtomicU8` load and the
arg byte with a separate Relaxed one. The acquire load sits between the
two, so LLVM had to re-load `self.code` and `code.instructions` for the
second access: four extra instructions and two extra dependent loads on
the instruction-fetch path of every bytecode instruction.
`CodeUnits::read_unit` reads both bytes as a single aligned Acquire
`AtomicU16`, which pairs with `replace_op`'s Release store exactly as
`read_op` did. `CodeUnit` gains `align(2)` so the 16-bit access is
guaranteed aligned rather than relying on the allocator.
5.91 -> 5.76 ns/bytecode-instruction on a straight-line
LOAD_FAST/STORE_FAST loop (-2.5%); range loop -2%.
Assisted-by: Claude Code:claude-fable-5-1
* vm: compute the cache-entry count after dispatch, not before
`op.cache_entries()` is a byte-table lookup used only to skip inline
cache units after an instruction falls through. Computing it before the
handler kept it live across the whole (very large) inlined match, so
every bytecode instruction paid an address materialization, the table
load, a stack spill and a reload of the count.
Moving it next to its only use lets it be computed on the fall-through
path with the value in a register.
Measured together with the following commit (the eval-breaker outline):
5.76 -> 5.55 ns/bytecode-instruction on a straight-line
LOAD_FAST/STORE_FAST loop, range loop -1.5%, fib(27) -2.5%.
Assisted-by: Claude Code:claude-fable-5-1
* vm: move the eval-breaker body out of the dispatch loop
The eval-breaker check itself stays inline on the per-instruction path
(moving it off regresses badly), but everything it guards -- the
finalizing/main-thread check, stop-the-world suspend, the QSBR
checkpoint, signal delivery and the scheduled GC -- was inlined into the
loop body too. That code is never executed on the hot path yet its
address materializations and live values competed for registers with
every opcode handler, so the loop head spilled more than it needed to.
`eval_breaker_work` is `#[cold] #[inline(never)]`, leaving a single
`ldrb`/`cbz` pair inline for the common "nothing pending" case.
Measured with the preceding commit: 5.76 -> 5.55 ns/bytecode-instruction
on a straight-line LOAD_FAST/STORE_FAST loop.
Assisted-by: Claude Code:claude-fable-5-1
* vm: always inline the eval stack push/pop primitives
`pop_value`, `pop_value_opt`, `top_value` and friends are a handful of
instructions each, but `ExecutingFrame::run` is ~50 KB of inlined opcode
handlers, so LLVM's inline budget refused them and emitted real calls:
STORE_FAST and POP_TOP each began with `bl ...pop_value`, which also
forced spills around the call. `#[inline(always)]` puts the four or five
instructions they actually are back into the handlers.
5.55 -> 5.29 ns/bytecode-instruction on a straight-line
LOAD_FAST/STORE_FAST loop (-4.8%); nbody -3.5%, fib(27) -3.5%,
`s += i` loop -3.5%, `for _ in range(n)` -4%.
Assisted-by: Claude Code:claude-fable-5-1
* vm: carry the instruction fetch state in loop locals
Instruction fetch went through memory three times per bytecode
instruction: `self.lasti` (a pointer field) was re-loaded, the program
counter was stored to it and read back at the top of the next iteration,
and `self.code` -> `code.instructions` was re-loaded to address the code
unit. LLVM cannot keep any of them in registers because the handlers
write through `&mut self`, so the fetch chain started with a
store-to-load round trip followed by two dependent pointer loads.
`lasti_cell` and `instructions` are now copied into locals once (both are
plain shared references with the frame's lifetime), and the program
counter is carried in `idx`: after a handler that did not jump the next
index is already known from `lasti_before` plus the cache count. The
frame's `lasti` is still stored before every instruction, so tracebacks,
`f_lineno`, signals and `sys._getframe` see exactly what they did; only
the read-back is gone. The paths that unwind (signal, exception handler,
`set_f_lineno`) re-read it, since they jump to a new position.
Also collapses the eval-breaker `if`, which clippy flagged after the
previous commit moved its body out of line.
range loop -3.3%, richards -3.1%, nbody -2.9%, `s += i` loop -2.2%,
go -1.7%, raytrace -1.2%; mean -1.1% over the 13-benchmark set. The
straight-line LOAD_FAST/STORE_FAST micro is unchanged at 5.28
ns/bytecode-instruction (it has no cache entries and never jumps, so it
was already the best case for the old code).
Assisted-by: Claude Code:claude-fable-5-1
* bytecode: hoist the instruction array base out of the dispatch loop
Addressing a code unit still went `self.code` -> `code.instructions` ->
the `Box<[CodeUnit]>` data pointer on every instruction, two dependent
loads before the acquire load that fetches the unit itself, because the
handlers write through `&mut self` and LLVM has to assume the fields
could change.
The instruction array is allocated once when the code object is built
and never reallocated -- specialization rewrites units in place -- so
`CodeUnits::units_ptr` may be read once before the loop and reused.
`read_unit_from` is the same 16-bit acquire load taken from that base;
`read_unit` now delegates to it, and the index is unchecked exactly as
it was before.
nqueens -1.9%, raytrace -1.6%, range loop -1.3%, deltablue -1.1%,
nbody -0.8%, dispatch micro -0.7%; mean -0.7% over the 13-benchmark set.
Assisted-by: Claude Code:claude-fable-5-1
* common: an immortal bit in the refcount word
Atomic reference counting is about a third of what a bytecode costs now
that the dispatch loop is trimmed: built `--no-default-features`, where
`PyAtomic` is a `Cell`, the straight-line dispatch micro runs at
3.5 ns/instruction against 5.4. Every `TO_BOOL`/`POP_JUMP_IF_*` pair and
every `None` or small-int push pays a relaxed `ldadd` and a `Release`
`ldaddl` on an object that can never die.
PEP 683's answer is to say so in the word. A fourth flag bit, `IMMORTAL`,
takes a bit from the strong count (leaving 60 on a 64-bit target, 28 on a
32-bit one, both far past what a program reaches); `make_immortal` sets it
and parks the count at CPython's `_Py_IMMORTAL_REFCNT`, and `inc`,
`inc_by`, `dec` and `safe_inc` read the bit and return without touching
the counter. The parked count is what keeps the rest of the tree honest
without a special case: it is far too large for a `strong_count() == 1`
in-place-mutation fast path to fire on, and `start_gc_refs` already reads
a count that large as `GC_REACHABLE`, so an immortal object is a permanent
root that keeps its referents alive.
Two details pay for the branch the mortal path now runs, which cost 4% on
object-heavy benchmarks before they went in:
* `leak` sets `IMMORTAL` alongside `LEAKED`. An interned object could
already never reach zero, so `dec` no longer needs the test it made on
the result of the decrement: the immortal test at the top decides both.
* `inc`'s three exceptional cases — a destructed word, a count at the
ceiling, a count at zero — fold into one comparison against the masked
word, with the handling moved to a cold helper.
What is left is one instruction on `dec` and roughly nothing on `inc`, and
the mortal path measures at parity.
Assisted-by: Claude Code:claude-fable-5-1
* vm: make the singletons, small ints, interned strings and static types immortal
Everything the `Context` builds once and keeps — `None`, `True`, `False`,
`Ellipsis`, `NotImplemented`, the empty tuple, frozenset, bytes and str,
the -5..256 int cache and the interned one-character strings — is now
marked immortal, as is every static type object, which its `static_cell`
holds for the life of the process. So is every entry of the string pool:
`mark_intern` already stopped those refcounts from reaching zero, so
immortality frees no memory that would otherwise have been freed; it only
takes the atomic read-modify-write off every incref and decref of an
attribute name, a dict key or a docstring.
Straight-line dispatch micro (200 LOAD_FAST/STORE_FAST pairs over cached
ints, median of 5 interleaved runs): 5.39 -> 4.01 ns/instruction, -26%.
A four-thread `if flag:` loop, where the contention on the shared bool was
real: 2.36 s -> 2.14 s, -10%.
pyperformance bench functions run standalone, medians of 5 interleaved
rounds against a matched build: fannkuch -7.8%, nqueens -3.8%, chaos
-3.0%, raytrace +1.0%, go +1.2%, richards +4.2%, nine of them together
-2.9%. Per-benchmark deltas under about 4% sit inside this tree's
build-to-build ThinLTO layout noise, which reaches that size between two
builds of identical source.
Observable: `sys.getrefcount` of an immortal object reports 4294967295,
CPython's `_Py_IMMORTAL_REFCNT`, and stops moving — what
`test_sys.test_refcount` expects of `None`. `is_interned`, and the dict
pointer-equality key fast path behind it, keep answering from the separate
`LEAKED` bit.
Assisted-by: Claude Code:claude-fable-5-1
* codecs: fast-path utf-8 decode, skipping the codec registry round trip
bytes.decode('utf-8') / str(bytes, 'utf-8', ...) always went through the
generic codec registry: a per-call attribute lookup for
_is_text_encoding, then a full call into the pure-Python
encodings.utf_8.decode wrapper, which called back into the native
utf_8_decode and unpacked a 2-tuple result.
CPython's PyUnicode_Decode special-cases a handful of built-in encoding
names (utf-8 among them) and decodes them directly, bypassing the codec
registry entirely -- a codecs.register() override does not affect
str(b"...", "utf-8") there either. This adds the same fast path here:
recognized utf-8 aliases go straight to the native decoder.
Measured with a pure-Python `str(bytes, 'utf-8', 'surrogatepass')` loop:
~723ns -> ~401ns per call (1.8x). On the pyperformance unpickle
benchmark (which decodes short strings this way for every
SHORT_BINUNICODE opcode): ~14.4ms -> ~13.2ms per loop, an ~8.3% win.
Assisted-by: Claude Code:claude-fable-5-1
* codecs: fast-path utf-8/ascii/latin-1 encode and ascii/latin-1 decode
Extend the utf-8 decode fast path from 3fe684629 to the symmetric encode
direction (str.encode()/bytes(s, enc)), and add the same registry-skipping
fast path for ascii and latin-1 encode/decode, all of which are simple
native codecs.rs functions already. For utf-8 encode with no surrogates in
the string (the common case), encoding is just a copy of the string's
existing UTF-8 bytes, so that's the fastest path; strings containing
surrogates and non-strict error handlers still go through the native
encoder + ErrorsHandler, matching custom codecs.register_error handlers.
Measured (median of interleaved runs vs baseline, 2M iterations):
str.encode("utf-8"): 351ns -> 195ns (-44%)
str.encode(): 449ns -> 291ns (-35%)
bytes(s, "utf-8"): 542ns -> 382ns (-30%)
str.encode("ascii"): 436ns -> 302ns (-31%)
str.encode("latin-1"): 487ns -> 333ns (-32%)
bytes.decode("ascii"): 519ns -> 359ns (-31%)
bytes.decode("latin-1"):590ns -> 417ns (-29%)
bytes.decode("utf-8")/str(b,"utf-8"): unchanged (already fast-pathed)
Assisted-by: Claude Code:claude-fable-5-1
* vm: make the runtime safe for borrowed stack entries
`PyStackRef` has carried a borrow tag since the stack was unified with the
fastlocals, but only `LOAD_SMALL_INT` -- which borrows immortal `Context`
objects -- ever set it, so nothing in the runtime had to be right about
borrowed entries. Turning `LOAD_FAST_BORROW` into a real borrow makes three
places load-bearing, and this fixes them ahead of the switch.
`Traverse for PyStackRef` handed every entry to the tracer, borrowed ones
included. The cycle collector subtracts one from an object's `gc_refs` per
edge it is shown, so a borrowed entry -- which owns no strong count -- would
have pushed a live object's count to zero and freed it out from under the
frame. Borrowed entries are now skipped; nothing becomes unreachable, because
the fastlocals slot the entry borrows from is on the same traversal.
`park_after_yield_from` stands in for a `YIELD_VALUE` that never runs, so the
`promote_stack()` in that handler never ran either; it now promotes there too,
and the debug assertion after it stops being vacuous.
`STORE_FAST`, `DELETE_FAST` and the fused stores now call
`LocalsPlus::debug_audit_local_release`, which in debug builds panics if the
slot is about to drop the last count of an object a borrowed entry still
points at -- the one shape codegen's `optimize_load_fast` exists to prevent,
and the one that would be a use-after-free if the analysis or a modelled stack
effect were wrong.
`BORROW_LOCAL_LOADS` (off) is the switch for the borrow itself, and
`LOAD_FAST_BORROW`/`LOAD_FAST_BORROW_LOAD_FAST_BORROW` now go through
`push_local`, which is a plain clone while it is off. The invariant the switch
depends on is written out where `PyStackRef` is defined.
No measurable change: with the switch off the only behavioral difference is
the traversal fix, which no current bytecode can reach.
Assisted-by: Claude Code:claude-fable-5-1
* vm: read consumed stack entries in place instead of owning them
`pop_value()` turns whatever is on the stack into an owned `PyObjectRef`, so
an instruction that only reads its operand still pays a reference count round
trip -- and, once `LOAD_FAST_BORROW` starts pushing borrows, would pay an
atomic increment to promote a borrow it is about to throw away. This moves
every consumer that never needs ownership onto `pop_stackref()`, which hands
back the tagged entry and costs nothing for a borrow.
Discarding pops (62 of them, across the `LOAD_ATTR`, `TO_BOOL`, `BINARY_OP`,
`COMPARE_OP`, `CONTAINS_OP` and `CALL` specializations, which read their
operands through `top_value()`/`nth_value()` and then drop them) become
`pop_stackref()`. So do the read-only operands of `POP_TOP`, `POP_JUMP_IF_*`,
`TO_BOOL`, `BINARY_OP`, `COMPARE_OP`, `CONTAINS_OP`, `IS_OP`, `STORE_ATTR`,
`DELETE_ATTR`, `STORE_SUBSCR`, `DELETE_SUBSCR` and `UNPACK_SEQUENCE`.
Two protocol entry points had to move from an owned receiver to `&PyObject` to
make that reachable: `try_to_bool`, which never used the ownership it took,
and `rich_compare`, which forwards to the `&self` `_cmp` anyway. Both keep
working unchanged for `PyObjectRef` callers through deref.
`LOAD_ATTR`'s method specializations popped the receiver and pushed it
straight back below the bound function; they now move the entry across with
its tag intact, which is how CPython keeps a method call's receiver borrowed
from `LOAD_ATTR` all the way through `CALL`. The plain attribute read reads
through the receiver rather than owning it.
Neutral while `BORROW_LOCAL_LOADS` is off -- for an owned entry
`pop_stackref()` and `pop_value()` compile to the same thing.
Assisted-by: Claude Code:claude-fable-5-1
* vm: turn LOAD_FAST_BORROW into an actual borrow
`LOAD_FAST_BORROW` accounts for 21.5% of executed instructions on
nbody+richards+nqueens, and until now it did exactly what `LOAD_FAST` does:
an atomic increment on the way onto the stack and, from the instruction that
consumed it, an atomic release decrement on the way off. Neither is needed.
The fastlocals slot holds a count for as long as the entry is on the stack --
codegen's `optimize_load_fast` only rewrites a `LOAD_FAST` when it can prove
the entry is consumed before anything can release that slot -- so the entry
can be a borrow that owns nothing, which is what CPython 3.14 does with a
tagged `_PyStackRef`.
Flipping `BORROW_LOCAL_LOADS` on is the whole change; the two preceding
commits are what make it safe (cycle-collector traversal, yield-point
promotion, a debug audit of the invariant) and what make it pay (consumers
that read through the entry rather than promoting it back to owned).
Medians of 7 interleaved rounds, this machine, against the pre-series binary:
straight-line while loop 6.32 -> 6.01 ns/instruction (-5.0%)
x = a + b loop -6.3% if flag: loop -3.8%
attribute-read loop -5.2% method-call loop -3.3%
subscript loop -6.7% fib(27) -0.8%
nbody -2.6% fannkuch -1.7% nqueens +1.3% richards -3.0%
deltablue -2.3% go -3.0% raytrace -1.7% float -3.2%
chaos -2.6% spectral_norm -2.5% mean -2.1%
About half of that is the consumer rewrite alone (mean -1.0% with the switch
still off) and the rest is the borrow. nqueens is the one benchmark that goes
the other way, by less than the roughly 4% of build-to-build layout noise this
tree has, and it moves with the consumer rewrite rather than the switch.
Full suite green (418 files, 42,110 tests). A build with debug assertions on
runs the suite with no audit failure, and trips the audit exactly when
`LOAD_FAST` is forced to borrow in a shape codegen refuses to convert
(`f(x, (x := None))`). Adversarial scripts -- swap loops and reassignment
under `__del__` tracking, argument evaluation that deletes its sibling,
`a[a.pop()]`, closures over a rebound cell, generators dropped while
suspended, tracebacks whose `f_locals` are read after unwinding, `settrace`
writing `f_locals` mid-expression, a method receiver whose local is cleared
while the call is being built, `__add__`/`__lt__`/`__bool__` that clear the
local their operand came from, threads sharing objects through locals, and
`gc.collect()` from a finalizer -- all agree with CPython 3.13 under
`gc.set_threshold(1, 1, 1)`, in both builds.
Assisted-by: Claude Code:claude-fable-5-1
* vm: exact-dict fast path for DICT_MERGE and CALL_FUNCTION_EX kwargs
`g(**d)` on an exact dict compiled to BUILD_MAP+DICT_MERGE+CALL_FUNCTION_EX
and paid for two Python-level `keys()`/`__getitem__` round trips per key at
both the DICT_MERGE handler and collect_ex_args' iterate_mapping_keys, even
though the source is a plain dict. Add an exact-dict (downcast + type
check, so subclasses still go through the mapping protocol) fast path to
both sites that iterates entries natively via PyDict::items_vec, matching
CPython's PyDict_Check fast paths in PyDict_Merge / CALL_FUNCTION_EX.
Measured (1M calls, release build): g(**d) 1320ns -> 665ns, f(**d)
1470ns -> 900ns, c.m(**d) 1690ns -> 1135ns, f(x=1,**d) 1600ns -> 1035ns,
f(**d1,**d2) 1700ns -> 1013ns. bm_django_template ~7% faster,
bm_argparse (subparsers) ~7% faster.
Assisted-by: Claude Code:claude-fable-5-1
* vm: skip the recursion guard hashing exact ints inside a tuple
tuple.__hash__ hashed every element through PyObject::hash, which wraps
each call in with_recursion's native-stack-depth check. That guard exists
for elements whose hash can recurse into arbitrary Python code, but
int.__hash__ is a leaf computation - it can neither recurse nor call back
into Python - so the check is pure overhead for the overwhelmingly common
case of tuples of plain ints (e.g. namedtuple-shaped dict keys).
pyperformance's mdp benchmark builds huge tables keyed by tuples of deeply
nested namedtuples whose leaves are ints; profiling with `sample` showed
PyObject::hash/with_recursion/tuple's slot_hash dominating the CPU time.
Hashing exact-int elements directly via hash::hash_bigint cuts mdp's
wall time from ~6.79s to ~4.81s (median of 5 interleaved runs), about 29%
faster, with no change to hash values or recursion-guard behavior for any
other element type.
Assisted-by: Claude Code:claude-fable-5-1
* vm: wire slot descriptors from __slots__ into C-level type slots
Classes that put a dunder name (e.g. "__setitem__") into __slots__ get
a per-instance member descriptor for it, which is a legitimate way to
override a special method per-instance (chameleon's tal.RepeatDict does
exactly this). PyType::new_verbose ran init_slots() to compute
mp_ass_subscript/mp_subscript/etc. before these member descriptors were
added to the type's attribute namespace, so the slot lookup never saw
them and __setitem__/__getitem__ silently fell back to "does not
support item assignment".
Fix by calling update_slot for each dunder-shaped __slots__ member
right after it is installed on the type, mirroring what
type.__setattr__ already does for regular attribute updates.
Assisted-by: Claude Code:claude-fable-5-1
* frame: stop allocating cold data to close a generator
`clear_locals_and_stack`, which runs whenever a generator or coroutine
comes to an end, reached for the frame's lazily-allocated `FrameColdData`
through `cold()`. That allocates it on first access, so every finishing
generator paid a `Box<FrameColdData>` malloc and free just to find
`f_extra_locals`, `f_locals_cache` and `f_overwritten_fast_locals` all
empty -- cold data is only ever populated by tracing and frame
introspection. Ask for it with `cold_opt()` instead and skip the three
reads when it was never allocated.
Medians of interleaved A/B runs, ns per generator create+finish over a
plain call: `for _ in g(): pass` 265 -> 231 (-13%), a coroutine that
completes synchronously 642 -> 617 (-4%), `sum(x for x in range(3))`
1033 -> 997 (-3.5%).
Assisted-by: Claude Code:claude-fable-5-1
* function: bind exact args straight into a generator's frame
Calling a generator, coroutine or async generator function took the
general argument-binding path even when the call site was the simplest
possible one. `invoke_exact_args_slots` and the vectorcall fast path both
recognised the shape -- every parameter positional, filled exactly once
-- and then handed generator-like code back to `invoke()`, which built a
`FuncArgs` (a `Vec` plus an `IndexMap`) only for
`fill_locals_from_args_inner` to walk varargs packing, keyword matching
and defaults that this shape cannot reach.
Add `make_generator_exact_args`, the counterpart of
`prepare_exact_args_frame` for the one call shape it refuses: it builds
the heap frame the generator will own, moves the arguments straight into
its fastlocals and returns the generator. Both fast paths now use it. The
vectorcall one matters on its own for generator expressions, whose fresh
`MAKE_FUNCTION` on every evaluation keeps them out of the call-site
specialization that would otherwise catch them.
Medians of interleaved A/B runs, ns per create+finish over a plain call:
`for _ in g(): pass` 231 -> 218 (-6%), a synchronously-completing
coroutine 616 -> 598 (-3%), `sum(x for x in range(3))` 1007 -> 967 (-4%).
Assisted-by: Claude Code:claude-fable-5-1
* frame: drain a dead frame in place instead of moving it out
Three allocations and a large move on the path a finishing generator
takes to release its frame.
`Traverse::clear` opened with `self.iframe.get_mut().take()`, moving the
whole `InterpreterFrame` -- a couple of hundred bytes -- out of its slot
and onto the stack, only to drop it there. Drain it through `as_mut()`
and empty the slot afterwards.
`LocalsPlus::clear_into` collected the evaluation stack into a `Vec` that
existed only to be extended into the caller's; push into that one
directly. And `clear_stack_and_cells` built (and dropped) a `Vec` for a
stack it had not looked at -- a frame that reached its `return`, which is
every generator that simply finished, left nothing there. Look first.
Medians of 5 interleaved rounds, ns per create+finish: `for _ in g():
pass` 307 -> 297 (-3.4%), a synchronously-completing coroutine 600 -> 585
(-2.5%), `list(g())` 565 -> 558; a plain call is unchanged.
Assisted-by: Claude Code:claude-fable-5-1
* vm: don't raise a TypeError to ask an iterator for its length
`length_hint_opt` opened with `iter.length()`, whose whole job on a type
with no length slot is to build the `TypeError` that says so -- which
this caller then catches, recognises and discards. Every iterator and
every generator takes that path, and `list()` calls it on whatever it is
handed, so a `list(gen)` paid for an exception object per call.
`length_opt` answers the same question with `None`. CPython's
`PyObject_LengthHint` gates the call on the slots for the same reason.
Median of 5 interleaved rounds: `list(g())` x200k for a 1-yield
generator 556 -> 407 ns (-27%). pyperformance generators, coroutines,
comprehensions and async_tree are unchanged, as are nqueens, richards
and chaos.
Assisted-by: Claude Code:claude-fable-5-1
* gc: track a generator and its frame in one pass
Making a generator tracks two objects that are born together and die
together -- the generator and the frame it owns -- and each one took the
gen0 list lock and a thread-local read of the running interpreter for
itself. Give the pair one entry point: `PyGenerator`, `PyCoroutine` and
`PyAsyncGen` opt out of tracking at allocation, the way frames already
do, and `make_generator_or_coro` links both into gen0 under a single
lock, with one owner read and one collection check.
Medians of 5 interleaved rounds: pyperformance coroutines -2.5%,
generators -0.9%; `for _ in g(): pass` over a plain call 205 -> 200 ns.
Assisted-by: Claude Code:claude-fable-5-1
* dict_inner: fast-path exact-int equality in key_eq
Mirrors CPython's lookdict: two exact `int` keys now compare by BigInt
value directly instead of going through the generic rich-compare
dispatch (identical_or_equal -> bool_eq -> rich_compare_bool -> __eq__
slot lookup). Falls back to bool_eq whenever either side isn't exactly
`int` (subclasses may override __eq__), so behavior is unchanged there.
Measured (min-of-6 rounds, 2M-iteration micros, this tree's build has
~4% layout noise so min is used as the noise floor):
- dict_int_lookup (`d[i]`, int keys): 70.9ns -> 62.0ns (-12.5%)
- dict_str_lookup / dict_tuple_lookup / set_str_contains: unchanged
(within noise), confirming no regression on non-int keys.
pyperformance sanity checks (single round each, base vs candidate):
bm_mdp median ~5027ms/loop vs ~5003ms/loop (no regression, hashing
isn't the dominant cost there); bm_nqueens, bm_go, bm_deltablue,
bm_hexiom, bm_chaos all within ~1% (noise).
Hash/equality values cross-checked bit-identical against CPython 3.14
for 0, 1, -1, True, False, 2**61-1, 2**61, -(2**63), 2**100, -(2**100),
and int/float/bool/Decimal/Fraction(1) hash-equality invariants.
Also tried an exact-type fast dispatch in PyObject::hash (skipping the
slots.hash vtable call + with_recursion for exact int/str) - it only
moved hash_small_int/hash_large_int by ~2%, under the 3% bar, so it was
not committed.
Tests: `rustpython -m test -j4 test_hash test_int test_long test_tuple
test_dict test_set test_dictviews test_ordered_dict test_collections
test_fractions test_numeric_tower test_float test_bool test_str
test_userdict test_weakref` all pass (1796 run, 72 skipped). `cargo
test -p rustpython-vm --release -p rustpython-common` passes. clippy
clean on the touched lines.
Assisted-by: Claude Code:claude-fable-5-1
* vm: let the modulus type handle a three-argument pow
CPython's ternary dispatch offers the operation to the third operand's type
whenever its slot differs from the two already tried, including the case where
neither operand had one. The port required both of those slots to exist, so
`pow(10, 2, Decimal(7))` never reached `Decimal` and raised TypeError.
Assisted-by: Claude Code:claude-fable-5-1
* vm: add a feature-gated opcode-pair histogram
Counts `(prev_op, op)` in the dispatch loop and dumps the nonzero pairs at
exit, so superinstruction and specialization candidates can be ranked from
what pyperformance-shaped code actually dispatches rather than guessed at.
Entirely behind the non-default `opcode-histogram` feature: the module is
`#[cfg]`-ed out and the loop's two-line hook with it, so a default build is
byte-identical to one without the patch.
Assisted-by: Claude Code:claude-fable-5-1
* vm: let boolean producers run the branch that follows them
`TO_BOOL*`, `COMPARE_OP*`, `CONTAINS_OP*` and `IS_OP` materialised a bool
only for the `POP_JUMP_IF_*` one dispatch later to pop and test it, and the
compiler's `NOT_TAKEN` marker cost a third dispatch on the fall-through edge.
Generalise the `COMPARE_OP_INT`-only jump fusion so every boolean producer
resolves the jump itself, and step over `NOT_TAKEN` from there and from the
plain `POP_JUMP_IF_*` handlers. Both back off to ordinary dispatch when the
successor is an instrumented opcode or while tracing, so `sys.monitoring`
branch events and `sys.settrace` opcode events are unaffected.
Dispatches over 13 standalone pyperformance benchmarks: -5.0% total
(richards -11.7%, unpickle_pure_python -8.9%, sympy -8.4%, hexiom -8.1%,
django_template -7.1%, go -6.1%; nbody unchanged). Times, best of 7
interleaved rounds: deltablue -4.0%, hexiom -3.5%, go -3.4%,
unpickle_pure_python -3.3%, richards -3.2%, raytrace -3.0%, nqueens -1.9%,
geomean -1.0%; a branch-heavy microbenchmark -4.7% against a straight-line
control at 0%.
Assisted-by: Claude Code:claude-fable-5-1
* Adapt yield-from resume to optional generator frames
Coro.frame is now PyAtomicRef<Option<FrameObject>>, so the flattened
resume path uses frame_opt/clear_except_code instead of a direct frame.
Assisted-by: Grok:4.6
* Fix clippy and cr_origin after try_to_bool/f_lineno changes
try_to_bool now takes &self, so drop the leftover clones. Derive
cr_origin lines from lasti instead of prev_line, and satisfy
-Dwarnings on the immortal-refcount and del_needed types.
Assisted-by: Grok:4.6
* Convert fd filenames without the os ToPyObject impl
host_env-free builds (sandbox check, wasm) do not compile os.rs, so
OsPathOrFd::filename cannot use Borrowed's ToPyObject. Use as_raw()
instead, which is the same conversion.
Assisted-by: Grok:4.6
* Gate fd path helpers on the host_env feature
WASM clippy builds rustpython-vm without host_env, so OsPathOrFd and
the fspath/fd helpers are unused and fail -Dwarnings.
Assisted-by: Grok:4.6
* Drop redundant pickle try_to_bool clones after rebase
Assisted-by: Grok:4.6
* Document why enumerate keeps a Counter beside BigInt
Assisted-by: Grok:4.6
* Move fd path helpers into ospath_fd under host_env
Assisted-by: Grok:4.6
* Group threading eval-breaker bits in one module
Assisted-by: Grok:4.6
* Make InterpreterFrame construction unsafe
run_iframe stays safe: a live frame already has valid identity pointers.
Assisted-by: Grok:4.6
* Move OsPathOrFd re-export to the top of ospath.rs
Assisted-by: Grok:4.6
---------
Co-authored-by: Lee Dogeon <dev.moreal@gmail.com>
|