16 Commits
Author SHA1 Message Date
Jeong, YunWon 04eb0ad94c Match remaining native docs and signatures to CPython 3.14 (#8877)
* Record native docstrings the doc DB generator skipped

- Stop ignoring `__dir__`; skip `__base__`, `__bases__` and `__module__` instead.
- Recurse into native classes whose metaclass is not `type`, and also key a
  class stored under another attribute name by its own `__name__`.
- Collect native objects exposed by pure-Python modules, such as the codec
  error handlers, under their own `__module__` and `__name__`.
- Key types found through `object.__subclasses__()` by their own module as
  well as `builtins`.
- Record getset/member descriptors named like class metadata
  (`coroutine.__name__`), and native types a module exposes only through an
  instance (`sys.flags`).
- Treat a class as native only when it owns its C descriptors, so an enum
  holding a copied `str.__format__` is skipped.
- Keep docstrings unstripped, and import every module in sorted order before
  walking them.
- Apply the resulting key delta, measured on macOS with CPython 3.14.7, to
  data.inc.rs.

Assisted-by: Claude:claude-opus-5-5
Assisted-by: Grok:grok-4.7

* Stop Rust comments on Python items from becoming __doc__

Use `//` instead of `///` on Python-exposed items so their comments are not
published as `__doc__`.

Assisted-by: Claude:claude-opus-5-5
Assisted-by: Grok:grok-4.7

* Match native names, signatures and doc lookup

- Rename the weakref type to `weakref.ReferenceType` and define
  `__class_getitem__` on `_ctypes.Array` instead of its metaclass.
- Give each name of a multi-name `#[pymodule(sub)]` function
  (`posix.remove`/`unlink`) its database doc.
- Fill methods from `with(...)` traits (dict view `__reversed__`,
  `isdisjoint`) with the implementing class's database doc.
- Resolve database docs for `#[pymodule]` classes at macro time, and for the
  codec error handlers and `_lzma.LZMAError` in const context.
- Add `Context::new_exception_type_with_doc`.
- Match `posix_spawn` parameter names and defaults, make `dict_keys` and
  `dict_items` unhashable, add `_CData.__ctypes_from_outparam__`, and omit
  the Windows-only lines of `sys.__doc__` on other platforms.

Assisted-by: Claude:claude-opus-5-5
Assisted-by: Grok:grok-4.7

* Give struct sequence fields their database docs

`new_readonly_tuple_member` takes the field doc, looked up in the class's
attribute doc table through the new `class_attr_item_doc`.

Assisted-by: Claude:claude-opus-5-5

* Look up pymodule(sub) docs by an explicit module name

posix and nt share one doc entry. A sub module sets name to the
Python module it is merged into, and doc lookup uses that name.

Assisted-by: Claude:claude-opus-5-5
Assisted-by: Grok:grok-4.7

* Record pending class doc bodies on PyMethodDef

The pymethod derive sets doc_body_pending when a method has no Rust
doc body and expansion did not resolve a database span.
concat_with_attr_docs copies that span from the class table and
clears the flag. Other constructors leave the flag false.

Assisted-by: Claude:claude-opus-5-5
Assisted-by: Grok:grok-4.7

* Name argument fields after their Python parameters

PosixSpawnArgs.args is renamed to argv, and nt's follow_symlinks field
drops a name override equal to the field name.

Assisted-by: Claude:claude-opus-5-5
2026-09-29 11:05:10 +09:00
Jeong, YunWon be23b2c562 Store database docstrings xz-compressed behind an optional doc feature (#8836)
* Skip Python-defined classes when collecting builtin type docs

generate.py recorded every object.__subclasses__() entry as builtins.<name>
after importing the whole stdlib, so classes written in Python (tkinter,
unittest, tarfile, ...) ended up in the doc database. Classes that are
mutable heap types without C method descriptors and whose module is not a
C extension are now skipped, and their entries are removed from
data.inc.rs.

Assisted-by: Claude:claude-opus-5-5
Assisted-by: Grok:grok-4.7

* Store database docstrings xz-compressed behind an optional doc feature

- rustpython-doc's build script compresses all database docstrings into
  one xz blob with an offset index; identical texts share one offset.
- Macros emit blob references for database docs; vm decompresses the
  blob on first access.
- New `doc` cargo feature (default on) in rustpython, rustpython-vm and
  rustpython-stdlib. Without it the blob and xz are not linked and
  database-backed __doc__ is None; __text_signature__ is unaffected.
- Native types without a docstring get __doc__ = None in their dict.
- Workflows that build with --no-default-features enable `doc` where
  tests or releases need docstrings.

Assisted-by: Claude:claude-opus-5-5
Assisted-by: Grok:grok-4.7
2026-09-27 20:45:45 +09:00
Jeong, YunWon 3c654204f5 Match native text signatures to CPython names, kinds and defaults (#8815)
* Render CPython defaults in generated text signatures

Arguments whose default rendered as `<unrepresentable>`, while the
same callable has a representable default in CPython, now render that
default:

- Where the Rust value can be a literal, the argument becomes a plain
  field with `default = <literal>`.
- The remaining cases keep their Rust default and add `py_default`
  with a comment. These are wrappers such as ArgIntoFloat or
  ArgIntoBool, values that need the VM, enums, and Option where None
  means another value.

Byte string literals render as Python bytes. A function registered
under several names gets an internal doc per name, so its __doc__ no
longer carries the signature header.

Removed expectedFailure from test_pydoc tests that now pass.

Assisted-by: Grok:grok-4.7
Assisted-by: Claude:claude-opus-5-5

* Match native callable signatures to CPython names and kinds

Rename parameters and adjust positional-only / keyword-only kinds and
defaults of native functions and methods so their generated text
signatures match CPython. Remove expectedFailure markers from tests
that now pass.

Assisted-by: Claude:claude-opus-5-5
Assisted-by: Grok:grok-4.7

* Fix Windows console read size and keep type.__prepare__ permissive

Assisted-by: Claude:claude-opus-5-5

* Derive signature defaults from typed Rust values

Param.default is now a DefaultRepr rendered by the const writer.
Literals, integer and bool constants become typed defaults, and other
default expressions use the type's const fn py_default. A bare
`optional` renders as None, so `default = None` rewrites are reverted.

Address review comments: zlib max_length defaults, get_ca_certs
binary_form keyword, getpeercert der, WASM ssl certificate keywords,
fstatvfs fd-only, and _sre endpos shown as sys.maxsize.

Assisted-by: Claude:claude-opus-5-5
Assisted-by: Grok:grok-4.7

* Generate class constructor text signatures at compile time

A #[pyclass] impl composes the type's internal doc from the Args of
its Initializer or Constructor, so type.__text_signature__ and
inspect.signature work for native classes. __doc__ stays the plain
doc. Struct sequences get (iterable=(), /) and __reduce__ (self, /).
Hand-parsed constructors are converted to FromArgs structs, and
_weakref.proxy is split from ProxyType.

Assisted-by: Claude:claude-opus-5-5
Assisted-by: Grok:grok-4.7

* Remove expectedFailure from test_hmac tests that now pass

Assisted-by: Claude:claude-opus-5-5

* Match remaining native signatures and count missing CPython signatures as matches

Give lock.__enter__ and RLock.__enter__ their own no-argument methods,
add FromArgs for start_new_thread and start_joinable_thread keeping
their errors, take RTLD_GLOBAL/RTLD_LOCAL from libc, and align names,
kinds and <unrepresentable> defaults where CPython's signature is
invalid (bytes.hex, reduce, warn, link, setlocale, add_done_callback,
...). getaddrinfo accepts keyword arguments. whats_left treats a
callable CPython has no signature for as matching.

Assisted-by: Claude:claude-opus-5-5
Assisted-by: Grok:grok-4.7

* Drop unfulfilled dead_code expectation on os.link dir_fd args

Assisted-by: Claude:claude-opus-5-5

* Distinguish OptionalArg and Option in signature defaults

An `optional` field's signature default now comes from its type:
Option<T> renders None, while OptionalArg<T> (a missing argument that
is not any Python value) renders <unrepresentable>. Fields where a
missing argument behaves exactly like None become Option<T>; fields
whose missing state differs keep OptionalArg. Redundant
py_default = "<unrepresentable>" entries are removed.

Assisted-by: Claude:claude-opus-5-5
Assisted-by: Grok:grok-4.7

* Remove expectedFailure from test_rlcompleter.test_attr_matches

Assisted-by: Claude:claude-opus-5-5

* Use Option for optional arguments whose missing and None states match

OptionalOption and OptionalArg<Option<T>> fields that treat a missing
argument like None become Option<T> with #[pyarg(optional)], which renders
=None in the text signature. Plain optional parameters of that kind move
into small FromArgs structs. Uses where missing and None differ (reduce
initial, StringIO newline, TextIOWrapper.reconfigure, set_asyncgen_hooks)
keep OptionalOption.

Assisted-by: Claude:claude-opus-5-5
Assisted-by: Grok:grok-4.7

* Render ::NAME defaults by name and restore named constant defaults

- default = ::NAME writes NAME into the text signature and stores
  Into::into(NAME) as the Rust value.
- A one-field tuple-struct argument pattern such as Fildes(fd) names the
  parameter after its binding, so fcntl and termios keep destructuring.
- zlib, lzma, syslog, mmap and select defaults use their named constants
  again instead of literals or <unrepresentable>.

Assisted-by: Claude:claude-opus-5-5
Assisted-by: Grok:grok-4.7

* Name FromArgs fields after their Python parameters

Drop #[pyarg(name = ...)] where the Python name is a valid Rust field name
and rename the field instead. The option stays only for keywords (fn, loop,
final, type, self) and the non-snake_case memLevel.

Assisted-by: Claude:claude-opus-5-5
Assisted-by: Grok:grok-4.7

* Build literal defaults from Python values and type <unrepresentable>

- default = LIT on a non-primitive field converts the literal through
  ToPyObject and TryFromObject when the argument is missing, so object
  fields no longer need a vm expression plus a matching py_default.
- A bare default on a primitive integer, bool or float field renders its
  zero value.
- DefaultRepr::Unrepresentable replaces the <unrepresentable> text, and
  py_default = "<unrepresentable>" is a compile error.
- pyexpat.ParserCreate distinguishes a missing intern from intern=None,
  mmap AccessMode renders its value, and sqlite blobopen names rowid.

Assisted-by: Claude:claude-opus-5-5
Assisted-by: Grok:grok-4.7

* Resolve native docstrings from the doc database at compile time

- rustpython-doc stores its table as a sorted const slice with const fn
  lookups (get, get_attr, class_attr_doc) instead of a phf map, and the
  update-doc-db workflow emits that form.
- Class, method, getset, module function and struct sequence docs, and
  per-class slot wrapper docs, are chosen by const lookups in the derive
  output. The database text wins over Rust doc comments.
- Macros reach the table through rustpython_vm::__exports, so only the vm
  depends on rustpython-doc.
- Remove expectedFailure markers from doctest and pydoc tests that pass.

Assisted-by: Claude:claude-opus-5-5
Assisted-by: Grok:grok-4.7

* Keep time.gmtime, localtime and ctime without a text signature

Their [seconds] parameter has no text signature, and unittest.mock relies
on inspect.signature failing for them. Take OptionalOption again so the
default stays unrepresentable, and treat None like a missing argument.

Assisted-by: Claude:claude-opus-5-5
2026-09-26 10:39:04 +09:00
github-actions[bot]andgithub-actions[bot] ffd10d9094 Update doc DB for CPython 3.14.7 (#8579)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2026-08-23 00:12:30 +09:00
github-actions[bot]andgithub-actions[bot] cbfa330780 Update doc DB for CPython 3.14.6 (#8106)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2026-06-16 15:59:36 +09:00
Shahar Naveh c3e1f0fa37 Get all builtin types dynamiclly (#8095) 2026-06-15 11:11:32 +09:00
github-actions[bot]andgithub-actions[bot] f34cfca1f3 Update doc DB for CPython 3.14.6 (#8088)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2026-06-12 23:58:41 +09:00
Shahar Naveh 9a0291188c Clippy rules (test related) (#7968) 2026-05-25 22:03:18 +09:00
github-actions[bot]andgithub-actions[bot] 97c6419dde Update doc DB for CPython 3.14.5 (#7828)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2026-05-12 00:44:34 +09:00
github-actions[bot]andgithub-actions[bot] 4a8415d543 Update doc DB for CPython 3.14.4 (#7598)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2026-04-14 16:07:22 +09:00
Jeong, YunWon 9e3a554412 fix crates (#7549) 2026-03-31 21:05:08 +09:00
Jeong, YunWon 15cd2e3764 no_std for doc, replace HashSet in codegen (#7054) 2026-02-09 08:14:53 +09:00
github-actions[bot] b98e8cbf52 Update doc DB for CPython 3.14.3 2026-02-06 12:08:22 +09:00
Jeong, YunWon 48e4e7c661 Implement more ast features (#6986)
Assisted-by: Codex:gpt-6
2026-02-05 16:20:22 +09:00
Jeong, YunWonandgithub-actions[bot] 90717e5ef7 doc db to include types.* (#6391)
* Add types.* for doc/generate.py

* udpate-doc-db workflow

* Update doc DB for CPython 3.13.9

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2025-12-10 19:23:27 +09:00
Shahar Naveh db1adaa2c2 Move __doc__ crate to crates/doc (#6234)
* Add `__doc__` crate

* Base auto-generate ci

* Add dummy files

* Update docs

* Set codegen-units to 1 for doc db

* Mark `*.inc.rs` as auto generated

* Disable doctest

* Reset docs
2025-11-14 23:47:51 +09:00