956 Commits
Author SHA1 Message Date
Alessio Attilio be3d12e96c implement _imp.create_dynamic and _imp.exec_dynamic (#8919)
Assisted-by: Gemini:gemini-3.7-flash
2026-10-02 09:25:33 +09:00
Jeong, YunWon 1d7d704414 Use bitflagset for type flags (#8873)
* Use bitflagset for type flags

PyTypeFlags is defined with bitflagset! and the atomic tp_flags word with
atomic_bitflagset!, replacing the hand-written PyAtomicTypeFlags.

Assisted-by: Grok:4.7
Assisted-by: Claude:claude-opus-5-5

* Use bitflagset 0.0.4 from its git main branch

bitflagset 0.0.4 is not on crates.io yet. Patch it to the git main
branch in the workspace and in the example projects that depend on
rustpython-vm, since each of them is a separate workspace.

Assisted-by: Claude:claude-opus-5-5

* Depend on crates.io bitflagset 0.0.4

bitflagset 0.0.4 is published. Drop the git patches from the workspace
and the separate example-project workspaces.

Assisted-by: Grok:4.7

* Keep empty patch tables in example workspaces

The example workspaces already have an empty [patch.crates-io] table.
Dropping the bitflagset git line leaves that table in place.

Assisted-by: Grok:4.7

* Call bitflagset methods directly for type flags

Drop the plain-set has_feature wrapper and the intersects,
is_created_with_flags, DEFAULT, and heap_type_flags aliases.
Single-bit updates use insert, and flag combinations use from_slice.

Assisted-by: Grok:4.7

* OR paired type flags in one update

HAS_WEAKREF with MANAGED_WEAKREF, and HAS_DICT with MANAGED_DICT
(and INLINE_VALUES when present), are applied with one |=.

Assisted-by: Grok:4.7

* Store heap type flag sets as constants

HEAPTYPE | BASETYPE, and the dict and immutable variants built from
it, are const values instead of from_slice calls at each use.

Assisted-by: Grok:4.7

* Test SEQUENCE and MAPPING in one load

PyTypeFlags::COLLECTION is the two-bit mask. Callers check it with
one is_disjoint instead of two contains loads.

Assisted-by: Grok:4.7

* Fix clippy 1.99 lints in the vm

Pass closure arguments by value where clippy reports a needless
borrow, drop the unfulfilled redundant_else expectation on sum, and
return the HRESULT after todo!().

Assisted-by: Grok:4.7
2026-10-02 09:20:57 +09:00
1ndahous3 112b7efa80 Promote retained GC objects and batch generation updates (#8930)
Assisted-by: Codex:GPT-6
2026-10-02 01:42:09 +09:00
Bas Schoenmaeckers 468c40d578 Fix clippy errors for latest rust stable (#8934) 2026-10-02 01:40:54 +09:00
Joshua Megnauth ab3d759e74 CI: Run clippy on more features (#8916)
Clippy doesn't run on everything which is why I keep accidentally
uncovering new lints to fix. I updated CI to run clippy on all features,
except tk, as well as all binaries, examples, and tests. That should
cover everything.

Tk currently doesn't build (at least for me), so that feature is still
disabled in CI.

While here, fix some of the lints that are now raised.
2026-10-01 00:59:00 +00:00
1ndahous3 5b578797e7 Link libffi only when the vm's host_env feature is on (#8912)
Assisted-by: Claude Code:claude-opus-5-5
2026-09-30 13:30:03 +00:00
1ndahous3 e8919b9632 Preserve list mutation progress and release removed items outside locks (#8894)
* Preserve list mutation progress and release removed items outside locks

Assisted-by: Codex:GPT-6

* Trim excessive list extension reservations

Assisted-by: Codex:GPT-6
2026-09-30 22:14:47 +09:00
Sumi Jeong c9c78db91c Bound AST nesting so deep source raises instead of crashing (#8827)
* Bound AST nesting so deep source raises instead of crashing

Deeply nested source exhausted the native stack and killed the process
with SIGSEGV, which no Python-level `except` can catch.

The parser grows its own stack when it runs low on one, so it returns
trees deeper than the passes after it can walk. Those passes recurse
without a limit of their own, and the symbol table, which does apply
one, runs last. `compile(..., PyCF_ONLY_AST)` returns before reaching
it at all.

Apply that same limit to the tree right after it is parsed, and count
bracket nesting before parsing rather than after, as CPython's
tokenizer does. Checking afterwards built the tree first, and one
nested that deep exhausts the stack when it is dropped.

Fixes #7655

Assisted-by: Claude Code:Opus 5

* Return Result from too_deeply_nested_error

Lets the four call sites use `?` instead of matching on `Some` and
returning the error by hand, which drops 11 lines. `Result` is already
`#[must_use]`, so the attribute on the function goes away with it.

Assisted-by: Claude Code:Opus 5

* Return Result from pre_parse_source_error too

Same shape as the previous commit, for the other check this PR adds, so
the two are consistent with each other.

Assisted-by: Claude Code:Opus 5

* Drop exceeds_max_nesting for the check beside it

The escape-warning pass needs the bracket check but had no `SourceFile` to
build a `CompileError` from, so it got a `bool` wrapper of its own. Building
the `SourceFile` there lets it call `pre_parse_source_error` instead, which
is shaped like the other checks in that file, and costs nothing measurable.

Assisted-by: Claude Code:Opus 5
2026-09-30 22:13:15 +09:00
Jeong, YunWon 100dac0bbe Reject payload receivers on #[pymethod] and #[pygetset] (#8898)
* Report errors from #[pyclass] trait items

The trait branch of impl_pyclass_impl built its own ImplContext, so the
errors collected while extracting trait items were never emitted: an
invalid #[pymethod] or #[pygetset] in a #[pyclass] trait was left out of
the class without a compile error. The trait branch now uses the shared
context whose errors are emitted.

Assisted-by: Claude Code:claude-opus-5-5

* Pass the object to #[pymethod] and #[pygetset] receivers outside vm

Outside crates/vm, where impl Py<T> cannot be written, #[pymethod] and
#[pygetset] receivers that took &self become zelf: &Py<Self> inside the
existing impl T blocks. Where Rust code calls one of these methods on a
payload value, the logic stays in a payload method and the Python method
forwards to it.

Assisted-by: Claude Code:claude-opus-5-5
Assisted-by: Grok:grok-4.7

* Reject payload receivers on #[pymethod] and #[pygetset]

A #[pymethod] or #[pygetset] item with a self receiver is now a compile
error when self is the payload: in an impl for T rather than Py<T> or
PyRef<T>, and in a #[pyclass] trait. The error points to taking
zelf: &Py<Self> or zelf: PyRef<Self>, or defining the item in impl Py<T>.

Assisted-by: Claude Code:claude-opus-5-5

* Pass the object to the layout test getter

The subclass layout test's extra getter is a #[pygetset] on the
payload. Take zelf: &Py<Self> so the receiver is the object.

Assisted-by: Grok:4.7
2026-09-30 22:12:16 +09:00
Jeong, YunWon 536737f4a0 Stabilize whats_left signature defaults and weakref repr (#8911)
Mask sys.executable, sys.version_info, the install scheme, and set order in signatures. Compare decimal and zoneinfo with the pure-Python fallback. Print the referent type's module in weakref repr.

Assisted-by: Grok:grok-4.7
2026-09-30 20:10:19 +09:00
1ndahous3 74019cf627 Update concurrent.futures for Python 3.14 (#8904)
* Update concurrent.futures for Python 3.14

Assisted-by: Codex:GPT-6

* Enable relative import completion test

Assisted-by: Codex:GPT-6
2026-09-30 02:16:15 +09:00
1ndahous3 ea76251294 Cache parsed struct formats per interpreter (#8903)
* Cache parsed struct formats per interpreter

Assisted-by: Codex:GPT-6

* Preserve struct cache entries on concurrent misses

Assisted-by: Codex:GPT-6
2026-09-30 02:13:21 +09:00
1ndahous3 b09402a782 Keep automatic GC requests local to each interpreter (#8902)
Assisted-by: Codex:GPT-6
2026-09-30 02:12:17 +09:00
1ndahous3 55a5f1c600 Validate native subclass object layouts (#8905)
Assisted-by: Codex:GPT-6
2026-09-30 02:11:38 +09:00
Jeong, YunWon a7d75d25a7 Move vm #[pymethod] and #[pygetset] items to impl Py<T> (#8901)
In crates/vm, #[pyclass] impl blocks whose Python methods or getset
descriptors took &self are moved to impl Py<T>, so those items receive the
object instead of the payload. Non-Python helpers stay in impl T. Where Rust
code calls one of these methods on a payload value, the logic stays in a
payload method and the Python method forwards to it.

#[pymethod] and #[pygetset] items in the vm #[pyclass] traits (the dict view
traits and the _io buffered mixins) take zelf: &Py<Self> instead of &self.

Assisted-by: Claude Code:claude-opus-5-5
Assisted-by: Grok:grok-4.7
2026-09-29 22:29:48 +09:00
Jeong, YunWon d54a4c5932 Set builtin __doc__, __module__, and datetime signatures (#8897)
* Set builtin __doc__, __module__, and datetime signatures

Expose empty doc-db member entries as ''. Heap __dict__ and __weakref__
getsets carry their descriptor docs. Member repr uses the qualified
type name.

__module__ is collections for the deque iterators, _csv for Dialect,
ctypes for ArgumentError and CField, _lzma for LZMAError, operator for
the getter types, pickle for PickleBuffer, _ssl for MemoryBIO,
SSLSession, _SSLContext and _SSLSocket, weakref for the proxy types,
and posix (nt on Windows) for times_result and uname_result.
unicodedata's copied UCD functions set __module__ to unicodedata.

_testinternalcapi function docs and weakproxy.__str__ use the stored
text. datetime replace defaults render as unchanged, __replace__ is
($self, /, **changes) and binds through replace, now defaults tz to
None, and fromisoformat and tzinfo methods name the parameter object.

Assisted-by: Grok:grok-4.7

* Move doctext helpers above the test module

db_slice and docs sat after mod tests.
Remove expectedFailure from test_html_doc, test_text_doc,
test_help_output_redirect, test_method_aliases, and test_long_signatures.

Assisted-by: Grok:grok-4.7
2026-09-29 21:11:06 +09:00
1ndahous3 a242ed1457 Select derive doc code by rustpython-vm's doc feature (#8900)
Assisted-by: Claude Code:claude-opus-5-5
2026-09-29 20:58:50 +09:00
Jiwoo Ahn 309d909fe1 fix: cross-interp buffer sharing drops shape without threading (#8899)
- Problem: without the threading feature, sending a non-contiguous or
  non-byte-format memoryview through _interpchannels loses its data or
  metadata on the receiving end
- Cause: the no-threading path copied only the contiguous bytes and
  discarded the buffer's descriptor, reconstructing everything as flat bytes
- Fix: copy the whole backing memory plus its BufferDescriptor, and
  reconstruct with that descriptor instead of a byte-only default

Assisted-by: Claude Code:claude-sonnet-5

Signed-off-by: Jiwoo Ahn <ikwydls1314@gmail.com>
2026-09-29 20:57:01 +09:00
Alessio Attilio d857eebdce fix(gc): traverse annotate name and qualname fields in PyFunction (#8839)
* fix(gc): traverse annotate name and qualname fields in PyFunction

* style: format extra_tests/snippets/stdlib_gc.py

Assisted-by: Antigravity:Gemini-3.8-Flash
2026-09-29 11:49:59 +09:00
1ndahous3 f909d2a30d Add native _datetime module (#8872)
* Add native _datetime module

Assisted-by: Claude Code:claude-opus-5-5
Assisted-by: Codex:GPT-6

* Use pymember and safe timezone storage in datetime

Assisted-by: Codex:GPT-6
2026-09-29 11:45:11 +09:00
Jeong, YunWon 04eb0ad94c Match remaining native docs and signatures to CPython 3.14 (#8877)
* Record native docstrings the doc DB generator skipped

- Stop ignoring `__dir__`; skip `__base__`, `__bases__` and `__module__` instead.
- Recurse into native classes whose metaclass is not `type`, and also key a
  class stored under another attribute name by its own `__name__`.
- Collect native objects exposed by pure-Python modules, such as the codec
  error handlers, under their own `__module__` and `__name__`.
- Key types found through `object.__subclasses__()` by their own module as
  well as `builtins`.
- Record getset/member descriptors named like class metadata
  (`coroutine.__name__`), and native types a module exposes only through an
  instance (`sys.flags`).
- Treat a class as native only when it owns its C descriptors, so an enum
  holding a copied `str.__format__` is skipped.
- Keep docstrings unstripped, and import every module in sorted order before
  walking them.
- Apply the resulting key delta, measured on macOS with CPython 3.14.7, to
  data.inc.rs.

Assisted-by: Claude:claude-opus-5-5
Assisted-by: Grok:grok-4.7

* Stop Rust comments on Python items from becoming __doc__

Use `//` instead of `///` on Python-exposed items so their comments are not
published as `__doc__`.

Assisted-by: Claude:claude-opus-5-5
Assisted-by: Grok:grok-4.7

* Match native names, signatures and doc lookup

- Rename the weakref type to `weakref.ReferenceType` and define
  `__class_getitem__` on `_ctypes.Array` instead of its metaclass.
- Give each name of a multi-name `#[pymodule(sub)]` function
  (`posix.remove`/`unlink`) its database doc.
- Fill methods from `with(...)` traits (dict view `__reversed__`,
  `isdisjoint`) with the implementing class's database doc.
- Resolve database docs for `#[pymodule]` classes at macro time, and for the
  codec error handlers and `_lzma.LZMAError` in const context.
- Add `Context::new_exception_type_with_doc`.
- Match `posix_spawn` parameter names and defaults, make `dict_keys` and
  `dict_items` unhashable, add `_CData.__ctypes_from_outparam__`, and omit
  the Windows-only lines of `sys.__doc__` on other platforms.

Assisted-by: Claude:claude-opus-5-5
Assisted-by: Grok:grok-4.7

* Give struct sequence fields their database docs

`new_readonly_tuple_member` takes the field doc, looked up in the class's
attribute doc table through the new `class_attr_item_doc`.

Assisted-by: Claude:claude-opus-5-5

* Look up pymodule(sub) docs by an explicit module name

posix and nt share one doc entry. A sub module sets name to the
Python module it is merged into, and doc lookup uses that name.

Assisted-by: Claude:claude-opus-5-5
Assisted-by: Grok:grok-4.7

* Record pending class doc bodies on PyMethodDef

The pymethod derive sets doc_body_pending when a method has no Rust
doc body and expansion did not resolve a database span.
concat_with_attr_docs copies that span from the class table and
clears the flag. Other constructors leave the flag false.

Assisted-by: Claude:claude-opus-5-5
Assisted-by: Grok:grok-4.7

* Name argument fields after their Python parameters

PosixSpawnArgs.args is renamed to argv, and nt's follow_symlinks field
drops a name override equal to the field name.

Assisted-by: Claude:claude-opus-5-5
2026-09-29 11:05:10 +09:00
Jeong, YunWon 63b80efc9f Read cached __slotnames__ and propagate slot name errors (#8895)
type_slot_names reads __slotnames__ from the type's own dict before
calling copyreg._slotnames, and rejects a cached value that is not a
list or None. object_getstate_default no longer replaces errors from
type_slot_names with TypeError("cannot pickle object"), so a
RecursionError raised there propagates.

Assisted-by: Claude:claude-opus-5-5
2026-09-29 11:04:41 +09:00
Jeong, YunWon 8f84fea377 Fix 32-bit member layout assertions and address member descriptor review (#8882)
* Address review findings on member descriptors

Member definitions carry an internal `PY_ATOMIC` flag taken from
`MemberLayout::ATOMIC`, and loads use an atomic access only for atomic
storage; extension members never set it. `XMLParser` publishes its new
state, `entity` and `target` only after every handler lookup succeeds.
`subs_parameters` reads type names only for the error. The unraisable
hook prints `<unknown>` when `__module__` fails or is not a str. The
builtin function `__reduce__` decides on the bound object alone.

Assisted-by: Grok:4.7
Assisted-by: Claude:claude-opus-5-5

* Fix member layout assertions on 32-bit targets

The tp_flags assertion compared AtomicU64 with u64 alignment, which differs
on i686. The kevent assertion required usize, isize, c_ulong and c_long to be
64-bit. Both now check only what the member cells need.

Assisted-by: Grok:4.7
Assisted-by: Claude:claude-opus-5-5

* Use bitflags for member definition flags

PY_READONLY, PY_AUDIT_READ, PY_RELATIVE_OFFSET and PY_ATOMIC become the
repr(transparent) PyMemberFlags. C extension flags are read with
from_bits_retain.

Assisted-by: Grok:4.7
Assisted-by: Claude:claude-opus-5-5
2026-09-29 01:59:05 +09:00
Jeong, YunWon 4344a34a3c Stop Rust comments on Python items from becoming __doc__ (#8893)
Use `//` instead of `///` on Python-exposed items so their comments are not
published as `__doc__`.

Assisted-by: Claude:claude-opus-5-5
Assisted-by: Grok:grok-4.7
2026-09-29 00:24:48 +09:00
1ndahous3 cc8b4d4f7e Compact sparse dictionary and set copies (#8879)
Assisted-by: Codex:GPT-6
2026-09-28 14:44:25 +00:00
1ndahous3 56eadb2997 Choose smaller difference scans and install completed intersections (#8881)
Assisted-by: Codex:GPT-6
2026-09-28 22:13:26 +09:00
1ndahous3 fd108237e4 Preallocate binary joins and preserve buffer lifetimes (#8880)
Assisted-by: Codex:GPT-6
2026-09-28 22:12:37 +09:00
1ndahous3 47eb9509b9 Reuse unchanged strings and fix mixed-representation stripping (#8878)
Assisted-by: Codex:GPT-6
2026-09-28 22:09:29 +09:00
1ndahous3 403a38cf49 Fix percent-format precision and integer conversion (#8876)
* Fix percent-format precision and integer conversion

Assisted-by: Codex:GPT-6

* Honor float subclass integer conversions in percent formatting

Assisted-by: Codex:GPT-6
2026-09-28 22:08:26 +09:00
1ndahous3 bb054f6e1b Release deque items outside locks during clear and assignment (#8875)
Assisted-by: Codex:GPT-6
2026-09-28 22:07:28 +09:00
1ndahous3 19e9b7c38a Preallocate dictionary storage in dict.fromkeys (#8874)
Assisted-by: Codex:GPT-6
2026-09-28 22:06:48 +09:00
1ndahous3 d1327131a5 Specialize module attribute loads (#8864)
Assisted-by: Claude Code:claude-opus-5-5
Assisted-by: Codex:GPT-6
2026-09-28 21:55:11 +09:00
Alessio Attilio 7329839fb5 fix(core): align hashability of dict views, namespace, and typing builtins (#8844) 2026-09-28 21:53:58 +09:00
1ndahous3 e2335f1ca9 Avoid redundant work in set difference operations (#8869)
Assisted-by: Codex:GPT-6
2026-09-28 06:53:13 +09:00
1ndahous3 456a8dc9db Match common encoding names before normalizing them (#8862)
Assisted-by: Claude Code:claude-opus-5-5
Assisted-by: Codex:GPT-6
2026-09-27 16:53:42 +00:00
1ndahous3 8c928c2444 Fix recursive sequence repetition fallback (#8865)
Assisted-by: Claude Code:claude-opus-5-5
Assisted-by: Codex:GPT-6
2026-09-27 16:44:38 +00:00
1ndahous3 15c99b32a6 Look up attributes without raising before calling __getattr__ (#8863)
Assisted-by: Claude Code:claude-opus-5-5
Assisted-by: Codex:GPT-6
2026-09-27 16:31:40 +00:00
Alessio Attilio b9fea9c7be fix(contextvars): isolate context stack per interpreter and thread on VirtualMachine (#8838)
* fix(contextvars): isolate context stack per interpreter and thread on VirtualMachine

* style: format and sort imports in extra_tests/snippets/stdlib_contextvars.py

Assisted-by: Antigravity:Gemini-3.8-Flash
2026-09-27 16:31:18 +00:00
1ndahous3 9fd6d76a39 Reuse stored hashes for frozenset hashing and in-place set union (#8859)
Assisted-by: Codex:GPT-6
2026-09-27 15:49:40 +00:00
1ndahous3 ad8bfdba4e Reuse stored hashes when merging dictionaries (#8858)
Assisted-by: Codex:GPT-6
2026-09-27 15:31:00 +00:00
1ndahous3 d31ccae04d Collect cycles through deque iterators and cmp_to_key wrappers (#8861)
Assisted-by: Codex:GPT-6
2026-09-27 15:14:17 +00:00
1ndahous3 f24b0d896b Stream deque inputs and release evicted items outside locks (#8860)
Assisted-by: Codex:GPT-6
2026-09-28 00:05:12 +09:00
Jeong, YunWon 31963fa767 Expose more attributes as member descriptors (#8857)
* Read type.__module__ from the dict only for heap types

A static type's `__module__` is the text before the last `.` of its name,
or `builtins`. A heap type reads its dict and raises `AttributeError`
when the entry is missing; `type` and `object` reprs ignore that error.
`type()` no longer stores `__module__ = None` when the caller's globals
have no `__name__`.

`#[pyexception(module = ...)]` now reaches the type name, and static
types that relied on a dict `__module__` (ast nodes, pyexpat, ssl,
_hashlib and _interpchannels errors, ExceptionGroup) carry their module.

Assisted-by: Grok:4.7
Assisted-by: Claude:claude-opus-5-5

* Expose more builtin attributes as member descriptors

`code.co_flags`, `co_nlocals`, `co_filename`, `super.__self__`,
`__self_class__`, `__thisclass__`, `traceback.tb_frame`, `tb_lasti`,
`type.__itemsize__` and `builtin_function_or_method.__module__` are now
`#[pymember]` fields instead of getsets. `super` keeps its three
references in atomic cells instead of a lock, `tb_lasti` is an `i32`,
`co_nlocals` is stored at construction, and the builtin function
`__module__` is a writable object cell.

Assisted-by: Grok:4.7
Assisted-by: Claude:claude-opus-5-5

* Expose stdlib attributes as member descriptors

Decompressor `eof`/`needs_input`/`unused_data` (bz2, lzma, zlib, plus
lzma `check`), `ContextVar.name`, `functools.partial` `func`/`args`/
`keywords`, socket `family`/`type`/`proto`, `select.kevent` fields,
`SemLock` `handle`/`kind`/`maxvalue`/`name`, `Pickler.bin`/`fast`,
`xmlparser.intern`, ElementTree `XMLParser.entity`/`target`,
`UCD.unidata_version`, `_ExceptHookArgs` fields, multibytecodec
`stream`, and `_io` `Buffered*.raw` and `TextIOWrapper`
`buffer`/`encoding`/`line_buffering`/`write_through` are now
`#[pymember]` fields instead of getsets.

Values that changed under a lock move to atomic cells on the payload.
New member kinds `Short`, `Long`, `String`, `UShort`, `ULong`,
`LongLong` and `ULongLong` access exactly the width of their C type.

Assisted-by: Grok:4.7
Assisted-by: Claude:claude-opus-5-5

* Match bare super and static type itemsize

`super.__new__(super)` leaves `__thisclass__`, `__self__` and
`__self_class__` empty, and a super with no type takes the
zero-argument path when bound.

Static types carry their `tp_itemsize` (`type`, `memoryview`, `code`,
`frame`, generators, `_sre` objects, `_ctypes` thunks), inherit it from
their direct base when they leave it at 0, and a variable-size base
rejects any nonempty `__slots__` and a weakref slot.
`_thread._ExceptHookArgs` is a struct sequence.

Assisted-by: Grok:4.7
Assisted-by: Claude:claude-opus-5-5

* Expose SimpleNamespace.__dict__, __basicsize__ and __flags__ as members

`SimpleNamespace.__dict__` addresses the instance dict cell the same way
`module.__dict__` does. `slots.basicsize` now holds the full
`tp_basicsize` (header plus payload), so `__basicsize__` reads it
directly. Type flags are one atomic word: `__abstractmethods__` and
`_abc` set their bits in it in place, the separate `abc_tpflags` word is
gone, and `__flags__` reads the word as an unsigned long member.

Assisted-by: Grok:4.7
Assisted-by: Claude:claude-opus-5-5
2026-09-28 00:03:27 +09:00
1ndahous3 1a259e494f Read live list elements during repr without copying the list (#8853)
Assisted-by: Codex:GPT-6
2026-09-27 20:50:22 +09:00
1ndahous3 360803ff47 Reuse set hashes and fix stored key comparison order (#8852)
Assisted-by: Codex:GPT-6
2026-09-27 20:49:26 +09:00
1ndahous3 afe106c89c Collect functools cycles and release replaced state outside locks (#8851)
Assisted-by: Codex:GPT-6
2026-09-27 20:47:33 +09:00
Jeong, YunWon be23b2c562 Store database docstrings xz-compressed behind an optional doc feature (#8836)
* Skip Python-defined classes when collecting builtin type docs

generate.py recorded every object.__subclasses__() entry as builtins.<name>
after importing the whole stdlib, so classes written in Python (tkinter,
unittest, tarfile, ...) ended up in the doc database. Classes that are
mutable heap types without C method descriptors and whose module is not a
C extension are now skipped, and their entries are removed from
data.inc.rs.

Assisted-by: Claude:claude-opus-5-5
Assisted-by: Grok:grok-4.7

* Store database docstrings xz-compressed behind an optional doc feature

- rustpython-doc's build script compresses all database docstrings into
  one xz blob with an offset index; identical texts share one offset.
- Macros emit blob references for database docs; vm decompresses the
  blob on first access.
- New `doc` cargo feature (default on) in rustpython, rustpython-vm and
  rustpython-stdlib. Without it the blob and xz are not linked and
  database-backed __doc__ is None; __text_signature__ is unaffected.
- Native types without a docstring get __doc__ = None in their dict.
- Workflows that build with --no-default-features enable `doc` where
  tests or releases need docstrings.

Assisted-by: Claude:claude-opus-5-5
Assisted-by: Grok:grok-4.7
2026-09-27 20:45:45 +09:00
1ndahous3 90afe00d6c Fix dict view comparisons and avoid temporary sets (#8849)
Assisted-by: Codex:GPT-6
2026-09-27 13:40:51 +09:00
1ndahous3andJeong, YunWon 04d5e12ead Raise MemoryError on impossible allocations and reuse list/tuple buffers (#8840)
* Raise MemoryError instead of aborting on impossible allocations

Assisted-by: Claude Code:claude-opus-5-5

* use CPython 3.14's integer digit limit for left shifts

* size the pow memory check by what malachite allocates

* Make sequence allocations fallible and reuse deallocation buffers

Assisted-by: Codex:GPT-6

---------

Co-authored-by: Jeong, YunWon <69878+youknowone@users.noreply.github.com>
2026-09-27 13:39:29 +09:00
Jeong, YunWon bdc9246b37 Address builtin members by field offset (#8816)
* Address builtin members by field offset

`#[pymember]` now declares a field of the payload and generates a
PyMemberDef whose offset is the payload offset plus the field offset.
The getter/setter function form and `MemberAccess::Func` are removed;
builtin, slot, and C-API members share one get/set path. The macro
checks the field type against the member type.

Converted members: property fget/fset/fdel/__doc__, function
__globals__/__closure__/__builtins__/__doc__/__module__, staticmethod
and classmethod __func__/__wrapped__, complex real/imag (T_DOUBLE),
frame f_trace_lines (T_BOOL), defaultdict default_factory, getset and
member descriptor __name__/__objclass__, module __dict__.
frame.f_trace_opcodes is a getset.

Writable member fields are atomic pointer cells. The instance dict
pointer is an atomic cell at a fixed offset in front of the object,
and dict presence follows the type's HAS_DICT flag. The pickle
basicsize check no longer counts dict, weakref, and slot storage,
which live outside the payload.

Assisted-by: Grok:4.7
Assisted-by: Claude:claude-opus-5-5

* Address member review findings

- `#[pymember]` keeps leading `#[cfg]` attributes on the generated
  registration, and malformed `field`/`offset` values are compile
  errors.
- The `descriptor` module and the items the expansion names are public.
- Writable double members are an `AtomicU64` cell, loaded and stored
  with relaxed atomics. Readonly double members stay plain `f64` reads.
- `LOAD_ATTR_SLOT` is not specialized for `Py_AUDIT_READ` members.
- `defaultdict.__reduce__` passes no factory argument when
  `default_factory` is None.

Assisted-by: Grok:4.7
Assisted-by: Claude:claude-opus-5-5

* Declare members on payload fields

`#[pymember]` goes on a payload struct field, or on the struct with an
explicit `offset` when the storage is not a payload field. The struct
`#[pyclass]` builds `PyClassDef::MEMBERS`, a static `PyMemberSpec`
table, and `extend_class` registers one member descriptor per entry.
`path` addresses a subfield, one field may carry several members, and a
`#[cfg]` on the field gates its entry. The impl-side const form is
removed and now a compile error.

Assisted-by: Grok:4.7
Assisted-by: Claude:claude-opus-5-5

* Split nullable and non-null atomic cells

The nullable cell API (`new_empty`, `load_owned`, `store`,
`compare_exchange_empty`) moves from `PyAtomicRef<PyObject>` to
`PyAtomicRef<Option<PyObject>>`, and the typed `PyAtomicRef<Option<T>>`
shares it. The non-null `PyAtomicRef<PyObject>` gets `load_owned` and
`store` that return the object directly.

Slot cells, property fget/fset/fdel/__doc__, and defaultdict
default_factory are nullable cells. staticmethod and classmethod
callable are non-null cells and are read without `unwrap`. The
instance dict cell is `PyAtomicRef<Option<PyDict>>`, so reading it no
longer downcasts.

Assisted-by: Grok:4.7
Assisted-by: Claude:claude-opus-5-5

* Drop the unused cold trace_lines flag

`FrameColdData::trace_lines` had no writer after `f_trace_lines`
became a frame object member. A frame without a frame object traces
lines by default.

Assisted-by: Claude:claude-opus-5-5

* Compute payload offsets from the merged Py header

`PyInner` is now `Py`, so the payload offset and the dict member
offset are measured from `Py<T>`.

Assisted-by: Claude:claude-opus-5-5

* Expose descriptor names and owner types as members

method_descriptor, classmethod_descriptor, and wrapper_descriptor
expose `__objclass__` and `__name__` as readonly members, and
method-wrapper exposes `__self__` as a readonly member. The separate
`objclass` field of method and classmethod descriptors always equaled
`common.typ` and is removed.

Assisted-by: Grok:4.7
Assisted-by: Claude:claude-opus-5-5

* Expose remaining builtin tp_members as members

Attributes stored in a payload field become readonly or writable
member descriptors: bound method __func__/__self__, code
co_argcount/co_posonlyargcount/co_kwonlyargcount/co_stacksize (Py_T_UINT),
generic alias, coroutine and async generator members, interpolation and
template, range and slice start/stop/step, type __base__, union, and the
typing TypeVar, ParamSpec, TypeVarTuple, and TypeAliasType members.
`MemberKind` gains `Int` and `Uint` with the `PyMember_GetOne`/`SetOne`
range checks.

Attributes computed on access or kept behind a lock stay getsets.

Assisted-by: Grok:4.7
Assisted-by: Claude:claude-opus-5-5

* Spell a docless member as doc = false

`#[pymember(doc = false)]` stores no docstring, matching the
`doc = "..."` form, and replaces the separate `no_doc` flag.

Assisted-by: Claude:claude-opus-5-5

* Make pymember readonly by default

`#[pymember]` declares a readonly member unless `writable` is given.
The `readonly` option is removed and every declaration is converted.

Assisted-by: Claude:claude-opus-5-5

* Check the receiver of method descriptors

Binding or calling a method_descriptor with an object that is not an
instance of the owner type raises TypeError before the native function
runs. Static and class methods are exempt.

Assisted-by: Grok:4.7
Assisted-by: Claude:claude-opus-5-5

* Expose exception attributes as members

BaseException.__suppress_context__, StopIteration.value, SystemExit.code,
ImportError, NameError, AttributeError, OSError, SyntaxError,
BaseExceptionGroup and the UnicodeEncode/Decode/TranslateError attributes
are member descriptors backed by payload fields. ImportError, NameError,
AttributeError and SyntaxError gain payloads for their fields, and the
methods that read those attributes read the fields. Add the Py_ssize_t
member kind for UnicodeError start and end.

Assisted-by: Grok:4.7
Assisted-by: Claude:claude-opus-5-5

* Avoid same-type casts in unsigned member conversion

On targets where c_ulong is u32, casting through c_ulong was a no-op
cast that clippy rejects.

Assisted-by: Claude:claude-opus-5-5

* Prefer an explicit pymember doc over the doc table

doc = "text" is used as the docstring without consulting the stored
attribute documentation; only an omitted doc looks it up. Drop the member
doc strings that repeated the stored documentation.

Assisted-by: Grok:4.7
Assisted-by: Claude:claude-opus-5-5

* Infer pymember kinds from field types

MemberLayout gives the member kind of each accepted field type, and the
macro reads it from the field or path subfield, so type = "bool" and
the other numeric spellings are gone. type only selects object_ex.
Writable doubles use a new AtomicF64 cell instead of AtomicU64 bits.
MemberLayout and MemberCell replace the per-kind marker traits.

Assisted-by: Grok:4.7
Assisted-by: Claude:claude-opus-5-5

* Read nullable atomic cells only through load_owned

PyAtomicRef<Option<T>> and PyAtomicRef<Option<PyObject>> lose to_owned,
to_owned_ordering and try_to_owned; load_owned, which retries the
incref until it applies to the pointer still in the cell, is the one
owned read. Callers that kept a deref borrow across VM calls now take a
strong reference.

Assisted-by: Grok:4.7
Assisted-by: Claude:claude-opus-5-5

* Inherit static type slots from the whole MRO

extend_class copied slots only from the direct bases, and tp_init is
copied only from a base that defines it, so TabError missed the
SyntaxError init that IndentationError inherits. Walk the MRO the same
way init_slots does.

Assisted-by: Grok:4.7
Assisted-by: Claude:claude-opus-5-5

* Let Rust doc comments win over stored docstrings

A /// comment on a Python-exposed item is its docstring; the
rustpython-doc entry is used only when there is none. #[pymember] reads
the field's /// comment instead of doc = "...". Doc comments that were
shadowed by a stored docstring become // comments, so no docstring
changes. AGENTS.md describes the rule.

Assisted-by: Grok:4.7
Assisted-by: Claude:claude-opus-5-5

* Adapt members to upstream changes

_collections._tuplegetter.__doc__ is a writable nullable member on the
payload field. The impl-side const form no longer compiles.

Assisted-by: Grok:4.7

* Drop docstring text left as comments and inline downcast_cell

Comments that only repeated an item's docstring are removed; developer
notes stay as // comments. The typed cell methods call
downcast_unchecked directly, each with its own safety note.

Assisted-by: Grok:4.7
Assisted-by: Claude:claude-opus-5-5

* Read sqlite row factories through load_owned

The nullable cell no longer has to_owned; the sqlite module is built only
with the sqlite feature, so the default build did not catch it.

Assisted-by: Claude:claude-opus-5-5
2026-09-27 13:38:11 +09:00