Commit Graph
9 Commits
Author SHA1 Message Date
Jeong, YunWon be23b2c562 Store database docstrings xz-compressed behind an optional doc feature (#8836)
* Skip Python-defined classes when collecting builtin type docs

generate.py recorded every object.__subclasses__() entry as builtins.<name>
after importing the whole stdlib, so classes written in Python (tkinter,
unittest, tarfile, ...) ended up in the doc database. Classes that are
mutable heap types without C method descriptors and whose module is not a
C extension are now skipped, and their entries are removed from
data.inc.rs.

Assisted-by: Claude:claude-opus-5-5
Assisted-by: Grok:grok-4.7

* Store database docstrings xz-compressed behind an optional doc feature

- rustpython-doc's build script compresses all database docstrings into
  one xz blob with an offset index; identical texts share one offset.
- Macros emit blob references for database docs; vm decompresses the
  blob on first access.
- New `doc` cargo feature (default on) in rustpython, rustpython-vm and
  rustpython-stdlib. Without it the blob and xz are not linked and
  database-backed __doc__ is None; __text_signature__ is unaffected.
- Native types without a docstring get __doc__ = None in their dict.
- Workflows that build with --no-default-features enable `doc` where
  tests or releases need docstrings.

Assisted-by: Claude:claude-opus-5-5
Assisted-by: Grok:grok-4.7
2026-09-27 20:45:45 +09:00
Jeong, YunWon c382a607bb host_env: keep rustls off the rustls-free ssl surface (#8727)
* host_env: keep rustls off the rustls-free ssl surface

ssl is MemoryBIO, constants, OID, and ALPN. ssl-rustls is
the rustls engine. Drop the wasm32 rustls-pki-types web
dep; the browser crate already enables that feature.

Assisted-by: Claude

* host_env: drop rustls-pki-types from the lockfile

Assisted-by: Claude

* host_env: name the rustls engine feature rustls

ssl is the rustls-free surface. rustls is the engine.
stdlib's ssl-rustls product feature still selects this.

Assisted-by: Claude
2026-09-18 00:18:05 +09:00
Jeong, YunWon 3191ecaf38 host_env: share thread, locale, loader and BSTR (#8702)
* host_env: share thread, locale, loader and BSTR

Add Windows thread name, locale-info/wcscoll, CompareStringOrdinal,
LoadLibraryEx/GetProcAddress/FreeLibrary, BSTR and COM helpers, and
AF_HYPERV UUID conversion. Wire _thread, locale, ctypes LoadLibrary
and WMI BSTRs through those wrappers. Ensure =X: before spawnve/execve.

Assisted-by: Claude

* host_env: expose rustls-free ssl core on wasm

Compile MemoryBIO, protocol constants, OID tables, hostname
checks, and ALPN parsing without the rustls feature so wasm
and other hosts can implement _ssl on the shared types.

Assisted-by: Claude

* stdlib: add wasm _ssl on rustls-free host_env

Install _ssl on wasm32 against rustpython_host_env::ssl.
MemoryBIO, constants, OID, hostname, and ALPN work. Context
settings are stored; wrap/handshake and rustls-only loaders
raise SSLError until a wasm TLS engine is wired.

Assisted-by: Claude

* host_env: address #8702 review comments

Cache-miss Windows ctypes lookup uses GetProcAddress. FreeLibrary
propagates OSError. start_new_thread returns GetThreadId. Export
_thread._get_name and drop pyfunction doc comments. Reject
embedded NULs in Windows locale.strcoll/strxfrm. Terminate
wcscoll/wcsxfrm and SetThreadDescription inputs.

wasm _ssl accepts wrap_bio owner and get_ca_certs(binary_form),
and MemoryBIO.read(-1) returns all pending bytes.

Assisted-by: Claude

* host_env: gate the whole ssl module on feature ssl

Compile rustpython_host_env::ssl only with the ssl feature.
Rustls crates stay native-only, so enabling ssl on wasm does
not pull rustls. Inside the module, rustls files follow
not(wasm32) rather than a second feature flag.

Assisted-by: Claude

* host_env: fix Windows types and thread names

Cast SysAllocStringLen to *mut u16 and RPC status to u32 so
windows-gnu and MSVC agree. Export _NAME_MAXLEN, truncate names
at NUL and the OS byte cap, and decode get_name with fsdecode.

Assisted-by: Claude

* host_env: compile rustls engine on WASI

Keep rustls-free MemoryBIO/constants/OID/ALPN on all wasm
targets. Compile the rustls engine on WASI, where UnixTime::now
exists. Allow the rustpython ssl feature on wasm without
ssl-rustls or ssl-openssl.

Assisted-by: Claude

* compiler: pick the earliest CPython parse override

A later incompatible-prefix scan must not hide an earlier
number or bracket diagnostic. Parser-driven tokenization
stops at the first failure.

Assisted-by: Claude

* host_env: address remaining #8702 review comments

Map a missing Windows export to a symbol-load error, keep
LoadLibrary's Win32 code on the OSError, drop incomplete
UTF-16 pairs when truncating thread names, hide set_name
and _get_name off linux/macOS/Windows, and let ssl-rustls
enable host_env.

Assisted-by: Claude

* stdlib: add wasm _socket and read-all MemoryBIO

Register a rustls-free-style _socket shim on wasm32 so
Lib/ssl.py can import. Treat MemoryBIO.read(-1) as read-all,
matching _ssl_MemoryBIO_read_impl.

Assisted-by: Claude

* host_env: fix clippy, WASI ssl cfg, and wasm rustls

Defer unclosed-bracket overrides until no token diagnostic
exists, keep WASI rustls and wasm ssl modules exclusive,
preserve lone Windows surrogates, compile the rustls engine
on browser wasm, and clear clippy denials.

Assisted-by: Claude

* host_env: fix Windows clippy and wasm UnixTime

Gate thread-name byte truncation to linux/macOS, allow
GetProcAddress/FreeLibrary pointer deref, and enable
rustls-pki-types web time on browser wasm.

Assisted-by: Claude

* host_env: lock rustls-pki-types web for wasm

Assisted-by: Claude

* host_env: silence Windows unused thread id and shear

Assisted-by: Claude

* stdlib: drop Windows locale clippy needless wraps

Windows clippy failed on locale.strcoll/strxfrm needless return
and _getdefaultlocale always wrapping Ok.

Assisted-by: Claude

* compiler: keep lexer errors over print hints

print x; 0x must report an invalid hex literal. Ranking by source
offset let the earlier print hint win. Treat a finished radix
literal as valid so print x; 0x1 still gets the print message.

Assisted-by: Claude

* compiler: keep decode errors over unclosed openers

Ranking by offset made ( '\\N' report an unclosed paren and
wiped the unclosed-bracket flag on f'{, so compile snippets
raised SyntaxError instead of incomplete input. Export wasm
socket._accept for Lib/socket.py.

Assisted-by: Claude

* wasm: drop duplicate _socket; split locale cfgs

_socket lives in rustpython-stdlib's socket_wasm. The wasm crate
only re-exported that file when freeze-stdlib was off. Split
strcoll, strxfrm, and getencoding into cfg'd function defs.

Assisted-by: Claude

* compiler: keep decode blockers over unclosed openers

(print x; '\\N' must keep the print hint. A later malformed
escape still suppresses the EOF opener even when print wins.

Assisted-by: Claude

* compiler: keep earlier f-string over later lexer errors

f'{x'; ' and f'{x'; 0x must not report the trailing quote or
hex literal. Decode and f-string diagnostics now compete with
lexer failures by offset.

Assisted-by: Claude

* compiler: rank decode vs lexer vs print by class

Unterminated quotes no longer hide format-spec newlines or
empty f-string fields. Print is considered only after the
final winner is not a lexer failure, so print x; f'{x'; 0x
keeps the print hint. Line-continuation uses CPython wording.

Assisted-by: Claude

* compiler: keep unclosed paren over line-cont at EOF

test_exceptions wants '(' was never closed at the opener when
a final backslash follows. Skip escaped quotes in the bracket
scan so {\'a\'} still reports the continuation error. Un-xfail
test_join_daemon_thread_in_finalization.

Assisted-by: Claude

* test_threading: xfail daemon join finalization off Linux

The test passes on Linux. macOS and Windows leave an ignored
deallocator exception on stderr.

Assisted-by: Claude

* vm: hang daemon threads at interpreter finalize

Forced-off daemons must stay not-done so join() during
finalization GC raises PythonFinalizationError and
Thread.is_alive() stays true. SystemExit made the handle
look finished on macOS/Windows. Drop the Lib/test xfail.

Assisted-by: Claude

(cherry picked from commit d1c4925534)
Assisted-by: Codex:gpt-6
2026-09-17 01:33:20 +09:00
Jeong, YunWon 73e547a9c3 Honor generator close, throw context, and docs (#8701)
* Honor close unraisable and deallocator messages

A failed close lookup on a yield-from target is unraisable.
Ignored GeneratorExit from finalize gets a traceback and the
closing-generator err_msg. __del__ uses the deallocator message.

Assisted-by: Grok:grok-4.6

* address review: claim, ag_running, frame steal

Keep the running claim through close() cleanup. ag_running
reads running_async. aclose ignore-yield does not mark the
generator closed. Steal the frame slot before uniqueness
and expose it with try_to_owned.

Assisted-by: Grok:grok-4.6

* Honor wrapper docs, yield assign, throw context

method-wrapper exposes the slot __doc__. Parenthesized yield
assignment uses the invalid-target messages. throw() restores
without overwriting __context__, then chains only from the
generator's own exc_info slot.

Assisted-by: Grok:grok-4.6

* Remove passing expectedFailure markers

method-wrapper docs, deallocator messages, and help pager
output now match, so drop the leftover markers.

Assisted-by: Grok:grok-4.6

* Honor wrapper signatures and assignment diagnostics

Slot docs include text signatures. method-wrapper compares and
hashes by identity. Keyword genexp and if/while assignment use
the invalid-target messages.

Assisted-by: Grok:grok-4.6

* address review: aclose, unraisable, frame publish

aclose ignore-yield does not close the generator. Format
unraisable object is None. close() leaves traceback to the
caller; finalize attaches the generator frame. Publish the
iframe before the lock-free slot.

Assisted-by: Grok:grok-4.6

* Honor member descriptors and wrapper annotations

Assisted-by: Grok:grok-4.6

* address review: getitem, bool doc, if assign

Assisted-by: Grok:grok-4.6

* Honor eval-mode assignment invalid syntax

Assisted-by: Grok:grok-4.6

* Supply getset and member docs from rustpython-doc

Assisted-by: Grok:grok-4.6

* Enable rustls SSL on WASI targets

Compile the rustls _ssl module for WASI and stub
socket waits plus the native certificate store.

Assisted-by: Grok:grok-4.6

* Add browser TLS _ssl in rustpython-wasm

Register rustls _ssl and a MemoryBIO-oriented _socket
from rustpython-wasm. Use rustls-rustcrypto because
ring cannot compile for wasm32-unknown-unknown.

Assisted-by: Grok:grok-4.6

* Define SSL protocol and cert constants as enums

Keep OpenSSL bitmask defines as integer constants.

Assisted-by: Grok:grok-4.6

* Ignore rustls deps cargo-shear cannot see

The rustpython-wasm _ssl sources are compiled through
a symlink, so shear does not count their crate uses.

Assisted-by: Grok:grok-4.6

* Replace wasm ssl symlink with a BIO module

Drop the stdlib ssl.rs link and keep a rustpython-wasm
_ssl that only covers MemoryBIO wrap_bio TLS.

Assisted-by: Grok:grok-4.6
2026-09-14 22:52:15 +09:00
Jeong, YunWon c716a7d070 common: share the zlib stream engine (#8631)
* common: share the zlib stream engine

Move the VM-independent zlib-rs stream owner and state machine into a feature-gated rustpython-common module. Make rustpython-stdlib adapt that engine instead of carrying its own flate2 implementation, including native stream copy/deepcopy, Z_BLOCK flushes, and wbits=0 decompression.

Assisted-by: Codex:GPT-5

* common: share the lzma stream engine

Move the VM-independent liblzma stream owner, filter properties, and compression state into a target-gated rustpython-common module. Keep rustpython-stdlib as the Python object and exception adapter, and remove its direct xz dependencies and lzma-only generic compressor machinery.

Assisted-by: Codex:GPT-5

* common: address compression engine reviews
2026-09-01 15:29:56 +09:00
Jeong, YunWon a1ba6a662c Align compiler and AST behavior with CPython (#8138)
* Align compiler and AST behavior with CPython

Update the compiler pipeline, AST module support, and related VM compile/eval paths toward CPython 3.14 behavior.

Use upstream Ruff parser crates directly and keep RustPython-specific syntax preflight handling in the compile path.

Refresh opcode metadata, snapshots, and targeted tests for the aligned bytecode and AST behavior.

* fix

* Fix PR 8138 CI failures

* Fix match mapping wildcard scan complexity

* fixes

* Use Ruff runtime AST fields

* Minimize residual patch diff

* refactor error handling

* Remove _ast conversion context wrappers

* Clean up _ast conversion glue and compiler warning escalation

Rename the Node conversion parameter `ctx` to `vm` throughout the _ast
module.

Remove the unused compile_program, compile_program_single,
compile_block_expression, and compile_expression forwarders; compile_top
stays as the entry point.

Stash an escalated compiler SyntaxWarning so a non-SyntaxWarning category
propagates unchanged instead of being rewritten to SyntaxError, matching
PyErr_ExceptionMatches(SyntaxWarning) in compiler_warn. Drop the now-unused
CompileWarningError::into_codegen_error.

Replace the PositionalArguments two-variant enum with a struct holding a
shared range and a PositionalArgumentsKind enum.

Assisted-by: Claude

* Represent compiler flags as a CompilerFlags bitflags

Replace the loose PY_CF_*/CO_* i32 constants in compile_mode with a
CompilerFlags bitflags covering the PyCF_* flags and __future__
CO_FUTURE_* bits. The Python-visible ast.PyCF_* attribute values are
derived from it via bits(); parser input modes stay plain consts.

builtins::compile and the compile_string flag handling test flags
through CompilerFlags::contains instead of raw bit math.

Assisted-by: Claude

* Fix PR 8138 test regressions

* Fix PR 8138 CI failures

* Avoid shell interpolation in Linux deps action

* Fix review findings in compiler/AST changes

- frame.rs: compute the MATCH_CLASS class name lazily so a successful
  match no longer takes the type name lock or allocates; the
  __match_args__-not-a-tuple branch reuses it instead of recomputing.
- _ast/constant.rs: add the missing leading space to the with_recursion
  label ("maximum recursion depth exceeded during compilation").
- sys.set_int_max_str_digits: restore the "maxdigits must be 0 or larger
  than {n}" error text.
- codegen: set CO_MAXBLOCKS to 20 in both ir.rs and compile.rs.
- _ast/exception.rs: extract the duplicated excepthandler node guard
  into ensure_excepthandler_node.
- codegen: correct the validate_keywords doc comment.
- Drop redundant "CPython" qualifiers from comments added in this branch.

Assisted-by: Claude
2026-06-26 18:22:08 +09:00
fanninpm 986550e519 Move dependencies to workspace for wasm crate (#7794)
* Add dependencies from `wasm` crate to workspace

* Declare dependencies as workspace = true
2026-05-13 03:12:22 +00:00
Shahar Naveh 2e777a6fb3 Add cargo shear CI step (#7461)
* Add `cargo shear` CI step

* Remove some deps

* Move `ruff_python_parser` to dev deps

* Remove `thread_local` crate

* Cleanup `num-integer` usage

* Remove parser fro. wasm

* Remove windows-sys from venv launcher

* Update lock

* Resolve more

* Fix wasm

* Update lock
2026-03-20 12:42:51 +09:00
Shahar Naveh 916d3ba94b Move wasm/lib -> crates/wasm (#6280) 2025-11-17 18:47:05 +09:00