mirror of
https://github.com/RustPython/RustPython.git
synced 2026-10-28 18:59:04 +08:00
df653cea699a74f730e94a787ffb86853dc7f7cb
4399
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Hanjeong Lee
|
df653cea69 |
Name generated signature parameters the way CPython names them (#8725)
* Leave Rust-only name decoration out of a generated signature A leading underscore marks an argument Rust sees as unused and `r#` escapes a Rust keyword. Neither describes the parameter a Python caller passes, and `r#` is not valid Python, so `inspect.signature()` rejects any signature carrying one. Reporting the bare name fixes _sre.template, select.epoll.__exit__ and bytearray.__reduce_ex__ without touching them, and lets a method name a parameter after a Rust keyword. A parameter CPython itself names with a leading underscore, such as compile's `_feature_version`, reaches Python through a FromArgs field, which this never sees. Assisted-by: Claude Code:claude-opus-5 * Name builtin type parameters the way CPython names them A generated __text_signature__ reports the Rust parameter name, so a name chosen for Rust reaches anyone reading help() or inspect. Rename the ones CPython describes differently: list.pop's `i` becomes `index`, list.count's `needle` becomes `value`, __format__'s `spec` becomes `format_spec`, bytes.strip's `chars` becomes `bytes`, and object.__reduce_ex__'s `proto` becomes `protocol`. Left alone are the parameters CPython reports as `object` or `unused`. Those are synthesised from the METH_* flags rather than named: CPython reports slice.indices as ($self, object, /) while its own docstring calls the argument len. Assisted-by: Claude Code:claude-opus-5 * Name os parameters the way CPython names them os.read's `n` becomes `length`, os.lseek's `how` becomes `whence`, os.kill's `sig` becomes `signal`, os.stat's `file` becomes `path`, and os.putenv's `key` becomes `name`, among others. os.chmod is left alone. It reports (path, dir_fd, mode, follow_symlinks) while the call binds mode second and takes follow_symlinks by keyword, so the order and the kind are wrong rather than the names, which a rename cannot fix. os.fstatvfs is left alone too: it shares one Rust function with statvfs, which CPython names `path`, so the two aliases cannot report different names. Assisted-by: Claude Code:claude-opus-5 * Name sys parameters the way CPython names them sys.excepthook's (exc_type, exc_val, exc_tb) become (exctype, value, traceback), sys._getframe's `offset` becomes `depth`, sys.intern's `s` becomes `string`, and sys.settrace's `tracefunc` becomes `function`. sys._clear_type_descriptors takes the name CPython gives it, which Rust spells r#type. Assisted-by: Claude Code:claude-opus-5 * Name vm stdlib parameters the way CPython names them deque.rotate's `mid` becomes `n`, deque.extend's `iter` becomes `iterable`, _sre.ascii_tolower's `ch` becomes `character`, _operator.neg's `pos` becomes `a`, signal.alarm's `time` becomes `seconds`, and codecs.register_error's `name` becomes `errors`. gc.collect and charmap_build keep theirs. Their argument is a FromArgs struct or a PosArgs bundle, so the reported name is the binding rather than a parameter, and renaming the binding would only disguise that. Assisted-by: Claude Code:claude-opus-5 * Name stdlib crate parameters the way CPython names them unicodedata.category's `character` becomes `chr`, math.isqrt's `x` becomes `n`, cmath.polar's `x` becomes `z`, struct.calcsize's `fmt` becomes `format`, binascii.crc32's `init` becomes `crc`, zlib.adler32's `begin_state` becomes `value`, and array.append's `x` becomes `v`. array.fromunicode's parameter is `ustr`, which cspell has to be told about. array.__deepcopy__ keeps `_memo`, which the generator now reports as `memo`; CPython calls it `unused`, a name synthesised from METH_O rather than chosen. Assisted-by: Claude Code:claude-opus-5 * Name BaseExceptionGroup.split's parameter the way CPython names it CPython calls the argument of split and subgroup `matcher_value`. Assisted-by: Claude Code:claude-opus-5 * Strip a generated parameter name with unraw and one prefix check syn already removes r# through IdentExt::unraw, the way from_args.rs does for a field name. The empty-name guard never ran: a bare `_` is Pat::Wild, which func_sig has already refused. Assisted-by: Claude Code:claude-fable-5-1 * Take object and unused from CPython's Clinic signatures CPython also falls back to these names for METH_O, but list.append and insert, set.add, discard and remove, _weakref.getweakrefcount and getweakrefs, sys.displayhook and getrefcount, and array.__deepcopy__ get them from an Argument Clinic signature. The earlier commit that kept array.__deepcopy__'s `_memo` was wrong to call `unused` synthesised. test_pydoc's set.add summary-line tests now pass. Assisted-by: Claude Code:claude-opus-5 |
||
|
Jeong, YunWon
|
3191ecaf38 |
host_env: share thread, locale, loader and BSTR (#8702)
* host_env: share thread, locale, loader and BSTR
Add Windows thread name, locale-info/wcscoll, CompareStringOrdinal,
LoadLibraryEx/GetProcAddress/FreeLibrary, BSTR and COM helpers, and
AF_HYPERV UUID conversion. Wire _thread, locale, ctypes LoadLibrary
and WMI BSTRs through those wrappers. Ensure =X: before spawnve/execve.
Assisted-by: Claude
* host_env: expose rustls-free ssl core on wasm
Compile MemoryBIO, protocol constants, OID tables, hostname
checks, and ALPN parsing without the rustls feature so wasm
and other hosts can implement _ssl on the shared types.
Assisted-by: Claude
* stdlib: add wasm _ssl on rustls-free host_env
Install _ssl on wasm32 against rustpython_host_env::ssl.
MemoryBIO, constants, OID, hostname, and ALPN work. Context
settings are stored; wrap/handshake and rustls-only loaders
raise SSLError until a wasm TLS engine is wired.
Assisted-by: Claude
* host_env: address #8702 review comments
Cache-miss Windows ctypes lookup uses GetProcAddress. FreeLibrary
propagates OSError. start_new_thread returns GetThreadId. Export
_thread._get_name and drop pyfunction doc comments. Reject
embedded NULs in Windows locale.strcoll/strxfrm. Terminate
wcscoll/wcsxfrm and SetThreadDescription inputs.
wasm _ssl accepts wrap_bio owner and get_ca_certs(binary_form),
and MemoryBIO.read(-1) returns all pending bytes.
Assisted-by: Claude
* host_env: gate the whole ssl module on feature ssl
Compile rustpython_host_env::ssl only with the ssl feature.
Rustls crates stay native-only, so enabling ssl on wasm does
not pull rustls. Inside the module, rustls files follow
not(wasm32) rather than a second feature flag.
Assisted-by: Claude
* host_env: fix Windows types and thread names
Cast SysAllocStringLen to *mut u16 and RPC status to u32 so
windows-gnu and MSVC agree. Export _NAME_MAXLEN, truncate names
at NUL and the OS byte cap, and decode get_name with fsdecode.
Assisted-by: Claude
* host_env: compile rustls engine on WASI
Keep rustls-free MemoryBIO/constants/OID/ALPN on all wasm
targets. Compile the rustls engine on WASI, where UnixTime::now
exists. Allow the rustpython ssl feature on wasm without
ssl-rustls or ssl-openssl.
Assisted-by: Claude
* compiler: pick the earliest CPython parse override
A later incompatible-prefix scan must not hide an earlier
number or bracket diagnostic. Parser-driven tokenization
stops at the first failure.
Assisted-by: Claude
* host_env: address remaining #8702 review comments
Map a missing Windows export to a symbol-load error, keep
LoadLibrary's Win32 code on the OSError, drop incomplete
UTF-16 pairs when truncating thread names, hide set_name
and _get_name off linux/macOS/Windows, and let ssl-rustls
enable host_env.
Assisted-by: Claude
* stdlib: add wasm _socket and read-all MemoryBIO
Register a rustls-free-style _socket shim on wasm32 so
Lib/ssl.py can import. Treat MemoryBIO.read(-1) as read-all,
matching _ssl_MemoryBIO_read_impl.
Assisted-by: Claude
* host_env: fix clippy, WASI ssl cfg, and wasm rustls
Defer unclosed-bracket overrides until no token diagnostic
exists, keep WASI rustls and wasm ssl modules exclusive,
preserve lone Windows surrogates, compile the rustls engine
on browser wasm, and clear clippy denials.
Assisted-by: Claude
* host_env: fix Windows clippy and wasm UnixTime
Gate thread-name byte truncation to linux/macOS, allow
GetProcAddress/FreeLibrary pointer deref, and enable
rustls-pki-types web time on browser wasm.
Assisted-by: Claude
* host_env: lock rustls-pki-types web for wasm
Assisted-by: Claude
* host_env: silence Windows unused thread id and shear
Assisted-by: Claude
* stdlib: drop Windows locale clippy needless wraps
Windows clippy failed on locale.strcoll/strxfrm needless return
and _getdefaultlocale always wrapping Ok.
Assisted-by: Claude
* compiler: keep lexer errors over print hints
print x; 0x must report an invalid hex literal. Ranking by source
offset let the earlier print hint win. Treat a finished radix
literal as valid so print x; 0x1 still gets the print message.
Assisted-by: Claude
* compiler: keep decode errors over unclosed openers
Ranking by offset made ( '\\N' report an unclosed paren and
wiped the unclosed-bracket flag on f'{, so compile snippets
raised SyntaxError instead of incomplete input. Export wasm
socket._accept for Lib/socket.py.
Assisted-by: Claude
* wasm: drop duplicate _socket; split locale cfgs
_socket lives in rustpython-stdlib's socket_wasm. The wasm crate
only re-exported that file when freeze-stdlib was off. Split
strcoll, strxfrm, and getencoding into cfg'd function defs.
Assisted-by: Claude
* compiler: keep decode blockers over unclosed openers
(print x; '\\N' must keep the print hint. A later malformed
escape still suppresses the EOF opener even when print wins.
Assisted-by: Claude
* compiler: keep earlier f-string over later lexer errors
f'{x'; ' and f'{x'; 0x must not report the trailing quote or
hex literal. Decode and f-string diagnostics now compete with
lexer failures by offset.
Assisted-by: Claude
* compiler: rank decode vs lexer vs print by class
Unterminated quotes no longer hide format-spec newlines or
empty f-string fields. Print is considered only after the
final winner is not a lexer failure, so print x; f'{x'; 0x
keeps the print hint. Line-continuation uses CPython wording.
Assisted-by: Claude
* compiler: keep unclosed paren over line-cont at EOF
test_exceptions wants '(' was never closed at the opener when
a final backslash follows. Skip escaped quotes in the bracket
scan so {\'a\'} still reports the continuation error. Un-xfail
test_join_daemon_thread_in_finalization.
Assisted-by: Claude
* test_threading: xfail daemon join finalization off Linux
The test passes on Linux. macOS and Windows leave an ignored
deallocator exception on stderr.
Assisted-by: Claude
* vm: hang daemon threads at interpreter finalize
Forced-off daemons must stay not-done so join() during
finalization GC raises PythonFinalizationError and
Thread.is_alive() stays true. SystemExit made the handle
look finished on macOS/Windows. Drop the Lib/test xfail.
Assisted-by: Claude
(cherry picked from commit
|
||
|
Jeong, YunWon
|
7c4017b3d8 |
Retry select/poll/fcntl on EINTR and add kqueue (#8721)
* Retry select/poll/fcntl on EINTR
select() no longer shares one empty list for r/w/x and now
calls select(2) even with empty sets so the timeout elapses.
poll() rejects type instantiation, treats float timeout as
milliseconds, and raises RuntimeError on concurrent poll().
fcntl, ioctl, flock, and lockf retry after EINTR.
Assisted-by: Grok:grok-4.6
* Add select.kqueue and select.kevent
Expose kqueue(2)/kevent(2) on BSD and macOS, including
control() EINTR retry and marking open kqueue objects
closed after fork.
Assisted-by: Grok:grok-4.6
* Add login_tty and fix termios/syslog/mmap
termios.error is Exception, not OSError.
os.login_tty prepares a tty for a new session.
syslog.openlog encodes ident as UTF-8.
mmap.seek returns the new position.
Assisted-by: Grok:grok-4.6
* Honor mmap trackfd on Unix
When trackfd is false, do not keep a duplicated file
descriptor. size() then fails with EBADF and resize()
raises ValueError.
Assisted-by: Grok:grok-4.6
* address review: portable kevent and kqueue close
Build kevent values by field assignment so FreeBSD's
ext array compiles. Normalize timespec nanoseconds,
keep the kqueue fd live across control(), reinit the
open list lock after fork, make trackfd keyword-only,
and skip login_tty on Solaris/illumos.
Assisted-by: Grok:grok-4.6
(cherry picked from commit
|
||
|
sijun-yang
|
6cc068474f |
Return distinct empty lists from select() (#8720)
Assisted-by: Codex:gpt-6-astra
(cherry picked from commit
|
||
|
Jeong, YunWon
|
73e547a9c3 |
Honor generator close, throw context, and docs (#8701)
* Honor close unraisable and deallocator messages A failed close lookup on a yield-from target is unraisable. Ignored GeneratorExit from finalize gets a traceback and the closing-generator err_msg. __del__ uses the deallocator message. Assisted-by: Grok:grok-4.6 * address review: claim, ag_running, frame steal Keep the running claim through close() cleanup. ag_running reads running_async. aclose ignore-yield does not mark the generator closed. Steal the frame slot before uniqueness and expose it with try_to_owned. Assisted-by: Grok:grok-4.6 * Honor wrapper docs, yield assign, throw context method-wrapper exposes the slot __doc__. Parenthesized yield assignment uses the invalid-target messages. throw() restores without overwriting __context__, then chains only from the generator's own exc_info slot. Assisted-by: Grok:grok-4.6 * Remove passing expectedFailure markers method-wrapper docs, deallocator messages, and help pager output now match, so drop the leftover markers. Assisted-by: Grok:grok-4.6 * Honor wrapper signatures and assignment diagnostics Slot docs include text signatures. method-wrapper compares and hashes by identity. Keyword genexp and if/while assignment use the invalid-target messages. Assisted-by: Grok:grok-4.6 * address review: aclose, unraisable, frame publish aclose ignore-yield does not close the generator. Format unraisable object is None. close() leaves traceback to the caller; finalize attaches the generator frame. Publish the iframe before the lock-free slot. Assisted-by: Grok:grok-4.6 * Honor member descriptors and wrapper annotations Assisted-by: Grok:grok-4.6 * address review: getitem, bool doc, if assign Assisted-by: Grok:grok-4.6 * Honor eval-mode assignment invalid syntax Assisted-by: Grok:grok-4.6 * Supply getset and member docs from rustpython-doc Assisted-by: Grok:grok-4.6 * Enable rustls SSL on WASI targets Compile the rustls _ssl module for WASI and stub socket waits plus the native certificate store. Assisted-by: Grok:grok-4.6 * Add browser TLS _ssl in rustpython-wasm Register rustls _ssl and a MemoryBIO-oriented _socket from rustpython-wasm. Use rustls-rustcrypto because ring cannot compile for wasm32-unknown-unknown. Assisted-by: Grok:grok-4.6 * Define SSL protocol and cert constants as enums Keep OpenSSL bitmask defines as integer constants. Assisted-by: Grok:grok-4.6 * Ignore rustls deps cargo-shear cannot see The rustpython-wasm _ssl sources are compiled through a symlink, so shear does not count their crate uses. Assisted-by: Grok:grok-4.6 * Replace wasm ssl symlink with a BIO module Drop the stdlib ssl.rs link and keep a rustpython-wasm _ssl that only covers MemoryBIO wrap_bio TLS. Assisted-by: Grok:grok-4.6 |
||
|
Jeong, YunWon
|
5ae21bee60 |
Keep escaped generator frames and wrap asyncgen throw (#8697)
* Keep escaped generator frames and wrap asyncgen throw Mark gi_frame/cr_frame/ag_frame as escaped so locals survive deallocation, and let frame.clear() drop those locals after owner finalize. Route throw() through finalize_send_result so StopAsyncIteration is wrapped like send(). Add ag_suspended and drop never-started frame locals on close(). Assisted-by: Grok:grok-4.6 * Store escaped on iframe; skip unused cold alloc Keep the escaped flag on InterpreterFrame so mark_escaped and has_escaped do not allocate FrameColdData. Clear optional cold fields only when they exist. Transfer frame owner in close() the same way send/throw already do. Assisted-by: Grok:grok-4.6 * Replace escaped flag with take_ownership gi_frame returns the frame object. Close uses a uniquely-referenced check to clear locals or take_ownership onto a husk. frame.clear() finalizes a generator-owned frame and only clears FRAME_OBJECT frames. close() leaves the generator suspended when it yields on GeneratorExit. Assisted-by: Grok:grok-4.6 * Drop frame mutex; steal frame_obj atomically take_ownership stores None in an atomic slot instead of locking and swapping a husk. gi_code reads the kept executable. Resume exclusivity stays on the running claim. Assisted-by: Grok:grok-4.6 |
||
|
Jeong, YunWon
|
850fadd388 |
Classify continuable unclosed strings as incomplete (#8694)
* Classify continuable unclosed strings as incomplete Mark EOF-open strings for PyCF_ALLOW_INCOMPLETE_INPUT, treat blank eval input as incomplete, detect unparenthesized lambda after a comma in f-strings, and advance parameter scans past default values. Assisted-by: Grok:4.6 * Diagnose incompatible prefixes and nested f-strings Report incompatible string prefixes, malformed \\N escapes, and f/t-string nesting beyond 150. Omit the extra newline on unterminated triple-quoted f-string SyntaxError.text. Assisted-by: Grok:4.6 * Fix empty defaults and longer incompatible prefixes Report an empty parameter default before later parameters, collect the full prefix before a quote, and prefer the u/b pair. Assisted-by: Grok:4.6 |
||
|
Jiseok CHOI
|
08e6e697a1 |
sqlite3: fix iterdump import and filter support (#8692)
* sqlite3: fix iterdump import and filter support Fix Connection.iterdump() to import sqlite3.dump with a from-list and forward its keyword-only filter argument, matching CPython 3.14. Remove stale expected-failure markers from all passing iterdump tests. Assisted-by: GitHub Copilot:GPT-5.6 Luna (high) * sqlite3: fix clippy core iterator lint Use core::iter instead of std::iter in the iterdump filter kwargs path, as required by RustPython clippy. Add the sqlite3 expected-failure workflow and CI clippy feedback reports. Assisted-by: GitHub Copilot:GPT-5.6 Luna (high) |
||
|
Jeong, YunWon
|
ce45fc24c7 |
Honor coroutine reuse, origin, and unawaited warnings (#8695)
Raise RuntimeError on send/throw of a closed coroutine, track cr_origin, emit never-awaited warnings through the warnings helper, and add cr_suspended. Stop treating coroutines as iterable. Warn on unawaited asyncgen asend/athrow/aclose, match async for TypeError messages, and attach coroutine method and name getset docs. Assisted-by: Grok:grok-4.6 |
||
|
Jiseok CHOI
|
407ba437cd |
sqlite3: fix row_factory deletion error (#8693)
Match CPython 3.14 by distinguishing row_factory assignment from deletion and raising AttributeError with the compatible message. Unmark the passing row_factory deletion test. Assisted-by: GitHub Copilot:GPT-5.6 Luna (high) |
||
|
Jeong, YunWon
|
a2e2bd299e |
asyncio: honor pending-task, unawaited, and get_referrers (#8689)
* asyncio: honor pending-task, unawaited, and get_referrers Report Task destructor as 'Task was destroyed but it is pending!'. Warn when a never-started coroutine is collected. Preserve signal() errno. Attribute generator-owned frame locals to the owner in gc.get_referrers. Remove the expectedFailure markers that now pass. Assisted-by: Grok:grok-4.6 * tests: drop expectedFailure after unawaited warning The never-started coroutine RuntimeWarning unblocks test_bpo_45813_1 and the AsyncMock assertNeverAwaited cases. Assisted-by: Grok:grok-4.6 |
||
|
Jeong, YunWon
|
66a3d9daa7 |
common: implement remaining builtin text codecs (#8687)
* common: implement remaining builtin text codecs Move utf-16/32/7, unicode-escape, raw-unicode-escape and the bytes escape transform into rustpython-common and call them from _codecs instead of _pycodecs. Assisted-by: Claude:grok-4.6 * host_env: share locale grouping, uuid and pipe state Add localeconv_numeric so grouping keeps a CHAR_MAX terminator, UuidCreateSequential for _uuid, and GetNamedPipeHandleStateW. Assisted-by: Claude:grok-4.6 * Fix utf-7 clippy and the utf-16-be snippet length Assisted-by: Claude:grok-4.6 * codecs: treat escape decode as final and accept str unicode_escape_decode and raw_unicode_escape_decode default final to true and accept a str as UTF-8. Fix clippy on wide_order, escape encode, and encode closures. Remove expectedFailure markers that now pass. Assisted-by: Grok:grok-4.6 |
||
|
Jeong, YunWon
|
43ed0a80b6 |
Warn on invalid octal escapes and parse-fail strings (#8686)
* Warn on invalid octal escapes and parse-fail strings Treat three-digit octal values above 0o377 as invalid escapes and scan quoted literals even when parse fails, so SyntaxWarning is emitted before a later SyntaxError. Keep warning and SyntaxError text on InvalidEscape. Enable the remaining test_string_literals invalid-escape cases. Assisted-by: Grok:4.6 * Remove expectedFailure from passing test_codeop cases test_invalid_warning and test_raw_raises_error now emit SyntaxWarning on parse-fail invalid escapes. Assisted-by: Grok:4.6 * Scan unparsed strings with token-boundary prefixes Require a complete prefix at an identifier boundary, and walk f-string interpolations separately after a parse failure. Assisted-by: Grok:4.6 |
||
|
Jeong, YunWon
|
cfee8a7ffa |
Fix -m argv[0] and file SyntaxError display (#8684)
* Fix -m argv[0] and file SyntaxError display Set sys.argv[0] to -m for module execution. Convert an escalated invalid-escape SyntaxWarning into a SyntaxError that spans just the `\X` pair. Report null bytes in source files with filename, lineno, and the truncated line. Enable the four corresponding test_cmd_line_script cases. Assisted-by: Grok:4.6 * Pass SyntaxError text when escalating escape warnings Keep the warning and SyntaxError messages as separate strings at the invalid-escape site. CompileWarningError carries an optional replacement instead of rewriting the warning text later. Assisted-by: Grok:4.6 * Remove expectedFailure from bytes invalid-escape test The invalid-escape SyntaxError path now satisfies test_eval_bytes_invalid_escape. Assisted-by: Grok:4.6 |
||
|
Jeong, YunWon
|
5afe140abd |
Store ContextVar.name as the original str object (#8682)
* Store ContextVar.name as the original str object Keep the constructor argument instead of converting it to a Rust String so lone surrogates and str subclasses are preserved. Build __repr__ from the name and default reprs. Assisted-by: Grok:4.6 * Compare Context objects by their variable maps Implement Context ==/!= from the mapping contents and mark Context unhashable. Snapshot the map before value equality so re-entrant ContextVar.set during __eq__ cannot deadlock. Enable test_context.ContextTest.test_context_run_4. Assisted-by: Grok:4.6 |
||
|
Jeong, YunWon
|
3fe3f37dac |
unicodedata: resolve name aliases and named sequences (#8676)
* unicodedata: resolve name aliases and named sequences
Vendor Unicode 17 NameAliases.txt and NamedSequences.txt into
rustpython-unicode so unicodedata.lookup matches CPython: formal
aliases and named sequences on the modern view, none on ucd_3_2_0.
Tangut derived-name lookup is case-insensitive. \N{} still rejects
named sequences because they are not a single character.
Assisted-by: Claude
* test_ucn: drop expectedFailure on test_aliases
The name-alias tables make the test pass; leaving the marker
fails CI as an unexpected success.
Assisted-by: Claude
* unicode: reject whitespace in Tangut derived names
parse_hex_code does not skip spaces around the hex digits.
Assisted-by: Claude
|
||
|
Jeong, YunWon
|
3a8a196996 |
ssl: share rustls engine in host_env and fire msg callbacks (#8665)
* ssl: fire msg callbacks and compute tls-unique Observe TLS records to invoke _msg_callback, and derive TLS 1.2 tls-unique from the KeyLog master secret plus rustls PRF. Restore CHANNEL_BINDING_TYPES and drop the remaining test_ssl RUSTPYTHON skips. Assisted-by: Claude * lzma: build the xz-core engine on wasm32 Expose `_lzma` on wasm32 and Android by compiling the xz-core engine on those targets instead of gating it out. Assisted-by: Claude * ssl: move the rustls engine into rustpython-common Extract the VM-independent rustls helpers (MemoryBio, certificate decode, cipher strings, OID table, chain builder, providers, and TLS record observation) so stdlib _ssl wraps a shared engine. Assisted-by: Claude * ssl: keep the rustls engine in host_env TLS helpers share the host trust store and default verify paths, so they belong next to native_certs rather than in rustpython-common. Assisted-by: Claude * stdlib: drop unused jiff after moving cert decode Assisted-by: Claude |
||
|
Jeong, YunWon
|
ac5fcb9be3 |
ssl: drop rustls lock before socket I/O (#8662)
Hold the rustls connection mutex only around rustls operations in handshake, read, write, and shutdown. Socket and MemoryBIO waits run without that lock so a blocking recv cannot deadlock a concurrent send. Remove the hang skip on test_thread_recv_while_main_thread_sends. Assisted-by: Claude |
||
|
Jeong, YunWon
|
eee9ad6420 |
Align contextvars, CLI startup, and stat validation with CPython (#8657)
* Align contextvars, CLI startup, and stat validation with CPython Validate Context and ContextVar arguments and mapping keys, hash names through the Python protocol, and preserve string contents without invoking __str__. Set the main module loader for command and stdin execution, report missing scripts with exit status 2, and reject file descriptors combined with follow_symlinks=False. Enable the nine corresponding stdlib regression tests. Assisted-by: Codex:gpt-6 * Move set_main_builtin_importer off the compiler feature Define the method in vm_new.rs so builds without rustpython-compiler, such as example_projects/barebone, can compile. Also set BuiltinImporter on an existing __main__ when its loader is missing or None. Assisted-by: Grok:4.6 |
||
|
Jeong, YunWon
|
def52d06b1 |
ssl: cap rustls unsent output at 4 KiB (#8660)
Keep BIO writes from filling asyncio's socket transport past its high-water mark when SO_SNDBUF is small. Drop the Unix skip on test_sendfile_ssl_pre_and_post_data. Assisted-by: Claude |
||
|
Jeong, YunWon
|
96925f49f8 |
Update Ruff to 0.16.5 (#8617)
Pin the RustPython Ruff fork to the patch commit based on Ruff 0.16.5 and adapt compiler and AST integration to the updated APIs. Assisted-by: Codex:GPT-5 |
||
|
Jeong, YunWon
|
531cf2b378 |
ssl: unify handshake, transport and shutdown state (#8656)
* ssl: accept ClientHello before selecting the SNI context Replace the provisional server connection and replay flags with explicit handshake states. Retain fatal alerts across nonblocking retries and preserve fragmented input on timeout. Enable the three SNI alert regressions. Assisted-by: Codex:GPT-5 * ssl: share socket and MemoryBIO transport progress Queue drained TLS records before fallible output and use one handshake record loop for both transports. Assisted-by: Codex:GPT-5 * ssl: unify shutdown and fatal alert lifecycle Track close-notify progress in TlsState, preserve pending alerts across retries, and share record-boundary shutdown for sockets and MemoryBIO. Retain rustls errors until Python exception conversion. Assisted-by: Codex:GPT-5 * ssl: emit the full peer certificate dict Use the same certificate dict helper for getpeercert() and Certificate.get_info() so OCSP, caIssuers, and CRL URLs are present. Clear CHANNEL_BINDING_TYPES on the rustls backend, and add a MemoryBIO leftover-plaintext snippet. Assisted-by: Claude |
||
|
Jeong, YunWon
|
25b5dfe5ca |
Fix /dev/fd, rlimit, spawn scheduler, and context cache (#8651)
* Fix /dev/fd, rlimit, spawn scheduler, and context cache Honor PYTHONEXECUTABLE when computing sys.executable. Rewind the file offset after a pyc magic probe. Stop masking rlimit values with RLIM_INFINITY. Print the pyrepl-unavailable warning before the rustyline shell. Parse posix_spawn scheduler and apply it where the host supports it. Update ContextVar cache fields under one lock. Remove expectedFailure and skip markers for tests that now pass. Assisted-by: Grok:4.6 * address review: check rlim_t overflow on convert Replace the u64-to-rlim_t as-cast in py2rlim_limit with a checked try_from, matching py2rlim. Assisted-by: Grok:4.6 |
||
|
Jeong, YunWon
|
699913d8cd |
Align codec, call, exception, and Unix module behavior with CPython (#8647)
* Collect consecutive charmap encoding failures Pass one contiguous run of unmappable characters to the codec error handler, and use the full run when a text replacement is itself unmappable. Assisted-by: Codex:gpt-5 * Leave conversions out of standalone format specs A conversion belongs to a replacement field rather than the format specification consumed by format(), so stop accepting and storing it in FormatSpec. Assisted-by: Codex:gpt-5 * Decode empty byte strings without looking up a codec Match PyUnicode_FromEncodedObject by returning the shared empty string before resolving the requested encoding for empty bytes and bytearray inputs. Assisted-by: Codex:gpt-5 * Validate call shape before converting builtin arguments Reject unknown keywords and excess arguments before conversions can raise a different error. Align Python-function positional-only handling, use declared list arguments in heapq, keep delegated codecs positional-only, and preserve CPython parameter names and validation order. Assisted-by: Codex:gpt-5 * Align attribute and exception initialization errors Give __dict__ and type __qualname__ their CPython diagnostics, preserve OSError filename fields correctly, accept seven-item SyntaxError details, and report Unicode error constructor arity before converting values. Assisted-by: Codex:gpt-5 * Expose missing Unix and Darwin module APIs Add the platform constants exported by fcntl, mmap, select, socket, syslog, termios, stat, errno, time, and _sre. Implement posix.killpg and resource.getpagesize, and expose resource.error as OSError. Assisted-by: Codex:gpt-5 * Honor the configured Python CPU count Parse -X cpu_count and PYTHON_CPU_COUNT, expose the stored value through sys._get_cpu_count_config, and let os.cpu_count prefer it over the host count. Also publish getsizeof with its CPython parameter name. Assisted-by: Codex:gpt-5 * Fix CPU count and WASI CI coverage Remove the obsolete expected-failure marker now that CPU count overrides pass, and avoid exporting FASYNC where WASI libc has no O_ASYNC constant. Assisted-by: Codex:gpt-5 * self review |
||
|
Jeong, YunWonandIvan Mironov
|
4a267b2616 |
ssl: discover default certificate paths in host_env (#8649)
Keep host certificate path probing with the native certificate provider, select the first available OpenSSL-style file and directory, and leave environment overrides to ssl.py. This also gives Android string defaults instead of the previous None values. Co-authored-by: Ivan Mironov <mironov.ivan@gmail.com> Assisted-by: OpenAI Codex: GPT-5 |
||
|
Joshua Megnauth
|
3442228b59 |
unicodedata: Fix name, unmask test_name (#8599)
This commit fixes most of UCD's names. The two main fixes are: * Handling chars missing in 3.2.0 by checking for membership #8548 * Handling names not present in unicodenames AI disclosure: I used AI for the initial research of this patch to determine why the Tangut characters are missing. AI also assisted with a bug fix where I neglected to parse the Tangut u32 as hex. I wrote all of the code. Assisted-by: GPT 5.6 Luna |
||
|
Jeong, YunWon
|
6e6792ee5c |
common: share the hashlib engine, and publish blake2b/blake2s as types (#8643)
The digest, HMAC, PBKDF2 and scrypt state moves out of the `_hashlib` module into a feature-gated `rustpython_common::hashlib` that owns each state behind a `Mutex` and reports plain Rust results; `_hashlib` and `_blake2` keep the object glue. The engine writes its state through a raw pointer into a caller-owned buffer, so the buffer is boxed and only the pointer travels with the Python object, and it carries the alignment the engine asserts. The wrapper is built only after a successful init, so a rejected digest name leaves nothing to drop over storage the engine never wrote. blake2 moves from `blake2` to `blake2b_simd` / `blake2s_simd`, whose `Params` carry the whole parameter block. `_blake2.blake2b` and `blake2s` become immutable types with `SALT_SIZE`, `PERSON_SIZE`, `MAX_KEY_SIZE` and `MAX_DIGEST_SIZE` as class attributes, and their instances are instances of those types. They accept `key`, `salt`, `person`, `fanout`, `depth`, `leaf_size`, `node_offset`, `node_depth`, `inner_size` and `last_node`. `_hashlib.scrypt` is published on the same engine. Six `test_hashlib` expectedFailure markers drop with it. `blake2`, `digest`, `dyn-clone`, `hmac` and `pbkdf2` lose their last consumer. Assisted-by: Grok Assisted-by: Claude |
||
|
Jeong, YunWon
|
aa59f1ee53 |
common: share the bzip2 stream engine (#8638)
Move the VM-independent bzip2 stream owner into a `bz2`-gated `rustpython-common` module beside the zlib and lzma engines, and keep `rustpython-stdlib` as the Python object and exception adapter. The adapter no longer names the `bzip2` crate; the `bz2` feature carries it. `_bz2` was the last user of the generic `DecompressState` / `DecompressStatus` / `Decompressor` machinery in `stdlib/src/compression.rs`, so that goes with it; `DecompressArgs` stays for zlib and lzma. The decompressor feeds the stream through the existing `Chunker`, so buffered and freshly supplied input are no longer joined into a new allocation first. `test_decompress_after_data_error` passes on the ported engine, so its `expectedFailure` marker goes. Assisted-by: Grok |
||
|
OkJa
|
93f151691c |
interpolated strings: align f-string and t-string diagnostics with CPython (#8601)
* builtins: name the other operand with tp_name in str and Template __add__
`str.__add__` and `Template.__add__` built their TypeError with
`PyType::name()`, which drops the module, where CPython formats `tp_name`:
>>> t"a" + "b"
TypeError: can only concatenate Template (not 'str') to Template
CPython 3.14 reports the qualified name and quotes the operand with double
quotes, as `Objects/unicodeobject.c` and `Objects/templateobject.c` do:
TypeError: can only concatenate string.templatelib.Template
(not "str") to string.templatelib.Template
`Template.__add__` also spelled its own name literally rather than taking it
from `PyClassDef::TP_NAME`, so the two halves of the message could drift apart.
Known gap: `slot_name()` still does not qualify a class defined in a module, so
`"a" + collections.OrderedDict()` names `OrderedDict` where CPython names
`collections.OrderedDict`. This affects both methods.
Unblocks test_template_concatenation in test_tstring.
Assisted-by: Claude Code:claude-opus-5
* interpolated strings: align f-string and t-string diagnostics with CPython
RustPython re-derives CPython's syntax diagnostics by scanning the source rather
than by translating ruff's parse errors, and the scanner shared by f-strings and
t-strings misread several replacement field states:
- A field that ran off the end of the literal (`f'{'`) was reported as an
expression that failed to start, where CPython asks for the closing brace.
- A field whose expression cannot start (`f'{;'`) and one whose expression runs
into a stray character (`f'{a;'`) both fell through to the missing brace
message instead of CPython's two distinct ones.
- A conversion or format spec following `=` was never validated, and a format
spec following a valid conversion was skipped entirely.
- Format specs were scanned with the `{{` escape rule that only applies to
literal text, so nested replacement fields inside them went unchecked.
- Brackets opened inside a field were never matched, so `f'{a[4)}'` and
`f'{3)+(4}'` fell back to a bare "invalid syntax".
- Comments inside a field were not recognised, so a `#` that swallows the
closing brace was reported as an unterminated literal, and a comment in a
multi-line field hid the expression behind it.
- Unterminated literals were named "string" whatever their prefix, and an
interpolated literal left with a replacement field open reported the missing
quote rather than the brace.
A field is only "never closed" while the tokenizer is still reading its
expression. Past the field's own `:` it is emitting literal text again, so
`f'{a:>5` runs out of input as an ordinary unterminated literal while `f'{a`
reports the brace. The field's own `:` is also not the one in a slice, a display
or a lambda, and an unclosed bracket inside the expression is what CPython names
rather than the field around it, so this walk tracks the whole delimiter stack.
Two messages were worded from the wrong branch of CPython's tokenizer. The hint
"perhaps you escaped the end quote?" is raised only where lexer.c handles a
literal without an interpolation prefix; its `%c-string` branch has just the
triple-quoted and plain forms, so pairing the hint with a prefix produced
`unterminated f-string literal (...); perhaps you escaped the end quote?`, which
CPython never emits. And a bracket mismatch inside a field dropped the
` on line %d` clause that lexer.c adds whenever `parenlinenostack[level]` differs
from the current `lineno`; the general bracket scanner in this file already
computed that suffix, so `f"""{a[\n4)}"""` now names line 1 as CPython does.
The rules these now follow are the ones CPython spells out in
Grammar/python.gram (`invalid_fstring_replacement_field` and its t-string twin)
and in Parser/lexer/lexer.c, which likewise parameterises the literal's prefix
rather than duplicating the messages.
Only a field's expression is code. A format spec is text, so a `(` there opens
nothing and a `#` selects the alternate form instead of starting a comment;
inside the expression a `#` does start one. Scanning the whole field for either
therefore reported diagnostics against valid literals, and because these
scanners only run once the source has already failed to parse, that let a good
literal take the blame for an error further down the file. Expression-level
checks now stop at the field's top-level separator and skip comments, and the
literal is walked field by field rather than byte by byte so that a `{` inside a
comment no longer opens one. A pre-existing instance of the same bug in the
line-continuation check is fixed along with them.
Ruff reports a mixed literal concatenation only as a bytes/non-bytes mix, so
`t"x" b"y"` arrived as a bytes error where CPython names the t-string. CPython's
`invalid_string_tstring_concat` is an `invalid_` rule, reached only on the error
pass, and `strings` tries `(fstring|string)+` ahead of it: that alternative
consumes the concatenation's leading run of non-t-string literals, so a mix among
those raises from `_PyPegen_concatenate_strings` on the first pass and sets
`error_indicator`, which suppresses the t-string rule. The t-string message
therefore wins for `t"x" b"y"` and the bytes message keeps precedence for
`"a" b"b" t"c"`. Both are now settled here rather than left to whichever scanner
runs next, which had been answering the second case with an unrelated
"Is this intended to be part of the string?".
A leading doubled `=` or `!` was read as a marker with an empty expression before it,
so `f"{==a}"` reported `valid expression required before '='` where CPython has no
expression to report at all. The same lookahead now guards that branch, and a
doubled `=` or `!` counts as an expression that cannot start.
A top-level `=` or `!` was read as a debug or conversion marker without checking
whether it belonged to a longer operator, so `f"{a==b}"` and `f"{a!=b}"` were also
read as fields ending early. CPython tokenises `==`, `!=`, `<=` and the rest as
single tokens before it ever considers the debug marker, so the separator scan now
skips a `=` that follows one of `= ! < > + - * / % & | ^ @ :` or precedes another
`=`, and a `!` that precedes one.
An expression cannot end on an operator that still wants an operand, and CPython
points at that operator rather than at the brace. `f"{a==}"`, `f"{a and}"` and
`f"{a.b.}"` fell through to a plain `invalid syntax`; they now carry the same
message and column CPython gives, reusing the scan that already handled `;` and
`$` for exactly this shape. `is not` and `not in` are single operators, so the
first word is what gets pointed at, and `...` is consumed in whole triples so
that `f"{....}"` blames the fourth dot rather than the first. The region has to
be walked forwards: a comment's terminating newline is whitespace, so trimming
backwards from the end would step into the comment body and read a triple-quoted
field whose comment ends in `+` as an expression ending in `+`.
That check runs only once the field is known to close, which is where a stray
character and a dangling operator part ways. A stray character is a finished
token, so the parser rejects it on the lookahead in
`annotated_rhs !('='|'!'|':'|'}')` and `f"{a;"` gets the separator message even
with no closing brace. A dangling operator instead makes the parser ask for one
more token, and producing it runs into the literal's closing quote, where
lexer.c answers from its `INSIDE_FSTRING(tok)` branch with
`%c-string: expecting '}'` before any `invalid_` rule is reached. So `f"{a and"`
is a missing brace while `f"{a and}"` names the operator.
A leading `.` was read as a character that cannot start an expression, so the
Ellipsis literal `f"{...}"`, the float `f"{.5}"` and its signed form `f"{-.5}"`
were reported as broken whenever the file failed to parse somewhere else.
`pegen` joins the cpython spelling dictionary, for the `_PyPegen_*` names these
comments cite.
Known gaps, none covered by either test file. Needing the longest valid expression
prefix, which a character scanner cannot compute: a starred tuple (`f"{*a,}"`) and
a dict-unpacking display are read as unable to start an expression, and
`f"{a===b}"`, `f"{a b}"` and `f"{a,,}"` fall through to a bare `invalid syntax`.
The caret still differs from CPython's on `expecting a valid expression after '{'`,
`expecting '}'`, `expecting '}', or format specs` and `unterminated ... literal`,
and on the messages CPython points at a whole token for, such as
`f"{lambda x: x}"` and `f"{x! r}"`. A quote inside a format spec is still treated
as a string delimiter.
Two more are reachable only through a t-string concatenation that ruff does not
report as a bytes mix, so `mixed_tstring_literal_error` never runs: three or more
literals with no bytes literal among them (`t"a" t"b" "c"`, `"a" "b" t"c"`) answer
with `invalid syntax. Is this intended to be part of the string?`, and so does a
tokenizer-level nesting overflow where CPython has
`too many nested f-strings or t-strings`. Two-literal mixes are correct because
ruff's own error carries the wording.
Two pre-existing bugs outside this change are worth naming, since it touches
their neighbourhood. `unterminated triple-quoted ... literal` reports
`detected at line 2` for a one-line source where CPython reports line 1, for every
prefix including none, so it is in the shared line arithmetic rather than the
interpolated path. And `str.__add__`'s message is unreachable for an operand that
defines `__radd__`: `"a" + 1` falls through to the generic
`unsupported operand type(s)` from the binop dispatch, where CPython has
`can only concatenate str (not "int") to str`.
Unblocks test_syntax_errors and test_literal_concatenation in test_tstring, and
test_comments,
test_conversions, test_invalid_syntax_error_message, test_mismatched_braces,
test_mismatched_parens, test_parens_in_expressions and
test_syntax_error_after_debug in test_fstring.
Assisted-by: Claude Code:claude-opus-5
* diagnostics: give the source scanners a diagnostic type
Every scanner in `cpython_parse_diagnostic_override` returned a bare
`Option<(String, usize, usize)>` — 69 signatures and 114 construction sites of
message, start offset, end offset — which the consumer then reassembled into a
message and a range one call later. They return a `CpythonDiagnostic` now, and
the two things that consume one, `NormalizedParseDiagnostic::other` and
`CompileError::from_source_error`, take it whole.
The type earns its name. These scanners run *after* ruff's parse has failed, so
what they produce is not a parse error but a reconstruction of what CPython
would have said about the same source; nothing here ever hands one back to ruff.
Reusing `ruff_python_parser::ParseError` would have fit the shape — it is
`{ ParseErrorType, TextRange }` — but every one of the 114 sites would have
wrapped its message in the single `OtherError` variant of a ninety-variant enum,
widened `other` and `from_source_error` to accept variants no caller
constructs, and, being a foreign type, ruled out the constructor that now
carries the `u32` cast and its justification. The `OtherError` wrapping happens
once per consumer instead, at the boundary where this does cross into ruff's
vocabulary.
Two nameless tuples that carried more than a diagnostic get names, following
`CallArgFrame` and `AssignmentContext` in this file:
- `bracket_syntax_error` returned the message alongside a bare `bool` for
whether the bracket was left open, which the caller needs apart from the
message because ruff reports that case as an EOF error. It returns a
`BracketError` now.
- `unclosed_replacement_field_error` walked its delimiter stack as
`Vec<(usize, u8, bool)>`, where the third element decided whether the field
had reached its format spec — `.2 = true` and `Some((_, b'{', false))` at the
use sites. `OpenDelimiter` names all three.
The `source_error!` macro is untouched at the 34 call sites it already had and
drops to five lines. Two copies of its body had been written out by hand
(`invalid_number_literal_error`, `unterminated_string_error`) and fold into it,
bringing it to 36.
One thing does change beyond the packaging. `TextRange::new` asserts that its
start does not exceed its end, and it asserts unconditionally. Building the
range at the scanner rather than at the consumer therefore puts that assert in
front of the hundred-odd scanners that reach `NormalizedParseDiagnostic::other`,
which never built a range before — they handed their two offsets straight to
`source_locations`. A scanner that emitted a reversed span used to produce a
garbled location; now it aborts. Every site was read for this and each clamp is
anchored to a bound at or after its start, and neither two hundred thousand
generated sources nor eleven thousand mutations of stdlib files reached it, so
this is a latent invariant made loud rather than a new failure — but it is not
packaging, so it is written down here.
Otherwise a pure refactor: no diagnostic changes message, column or line.
Verified by building the parent commit and diffing its output against this one
over a malformed-source corpus and over every `Lib/**/*.py` that parses clean
(1728 files, each recompiled with a trailing `$` so a scanner that misreads
valid code shows up as a moved line) — byte-identical, and the count of files
where the reported line moves is 1149 on both. test_fstring, test_tstring,
test_string_literals, test_syntax, test_exceptions, test_grammar and
test_compile all pass.
Addresses the review on #8601.
Assisted-by: Claude Code:claude-opus-5
* diagnostics: name the operator-character set behind the debug marker
`is_replacement_field_marker` carried its list of operator characters inline, a
sixteen-line `matches!` wrapped around three lines of logic, and it looked very
much like `is_dangling_operator_byte` a few functions down — enough that a
reviewer asked whether they were the same set.
They are not, and neither contains the other. Written in the same order the
difference is three characters:
precedes_equals_in_one_operator ! % & * + - / : < = > @ ^ |
is_dangling_operator_byte ! % & * + - . / < = > @ ^ | ~
The two are derived from different things. One asks which bytes pair with a
following `=` to make a single token, so that the `=` is part of that operator
rather than the start of a debug specifier — read off Python's token table, and
`.=` and `~=` are not on it, while `:=` is. The other asks which bytes make up
an operator that still wants an operand, which is why `~a` and `a.b.` put `.`
and `~` there and why the separator `:` stays out.
So the list moves next to the one it resembles, under a name, both sorted the
same way, and each says what the other has that it does not. Neither is defined
in terms of the other: the thirteen characters they share are two answers that
happen to coincide, not a set with a meaning of its own, and a common list would
tie each to the other's reasons to change.
No behavior change: the extracted list is character for character what was
inline. Verified against a build of the parent over a malformed-source corpus
and every `Lib/**/*.py` that parses clean — byte-identical — and with
test_fstring, test_tstring, test_string_literals, test_syntax, test_exceptions,
test_grammar and test_compile.
Addresses the review on #8601.
Assisted-by: Claude Code:claude-opus-5
* diagnostics: give the unterminated-literal scan one exit
A single-quoted literal that meets a newline and one that runs off the end of
the source are the same failure, and CPython's lexer says so in one condition:
if (c == EOF || (quote_size == 1 && c == '\n')) {
Everything after that — the check for a replacement field left open, and the
choice of message — is written once there. This scan had grown a second exit
for the newline case with its own copy of both, so the guard added for
interpolated literals had to be added twice and the message assembled twice.
The newline case now breaks out of the scan and falls into the exit that was
already handling end of source. `line` cannot have moved from `start_line` on
that path, and `quote_size` is 1, so the surviving exit computes the same
`detected_line` and the same `triple` the deleted one did.
No behavior change: verified against a build of the parent over a
malformed-source corpus and every `Lib/**/*.py` that parses clean —
byte-identical — and with test_fstring, test_tstring, test_string_literals,
test_syntax, test_exceptions, test_grammar and test_compile.
Assisted-by: Claude Code:claude-opus-5
* diagnostics: classify an unterminated triple-quoted t-string as incomplete input
`analyze_compile_error` pairs the f-string and t-string error variants
everywhere it names a message, but the match that decides whether a failure is
a `SyntaxError` or an `IncompleteInputError` listed only the f-string one. So a
`t'''` typed at the prompt would not have asked for another line where an
`f'''` would.
Would not have, rather than does not: the arm is unreachable today. The
compiler's `unterminated_string_error` scanner claims these first and hands
back an `OtherError`, so `f'''` and a bare `'''` get a `SyntaxError` at the
prompt too, on `main` as much as here — which is why
`test_codeop.test_incomplete` carries an `expectedFailure`. Whoever untangles
that precedence should not then find that f-strings work and t-strings
silently do not.
CPython does not separate the two either: `lexer.c` emits one message
parameterised with `%c` for the prefix and sets `E_EOFS` the same way for both,
and `_is_end_of_source` (`Parser/pegen.c`) looks only at that code, never at
which kind of literal produced it.
No behavior change, and none is testable while the arm cannot be reached.
Assisted-by: Claude Code:claude-opus-5
|
||
|
Jeong, YunWon
|
c716a7d070 |
common: share the zlib stream engine (#8631)
* common: share the zlib stream engine Move the VM-independent zlib-rs stream owner and state machine into a feature-gated rustpython-common module. Make rustpython-stdlib adapt that engine instead of carrying its own flate2 implementation, including native stream copy/deepcopy, Z_BLOCK flushes, and wbits=0 decompression. Assisted-by: Codex:GPT-5 * common: share the lzma stream engine Move the VM-independent liblzma stream owner, filter properties, and compression state into a target-gated rustpython-common module. Keep rustpython-stdlib as the Python object and exception adapter, and remove its direct xz dependencies and lzma-only generic compressor machinery. Assisted-by: Codex:GPT-5 * common: address compression engine reviews |
||
|
Jeong, YunWon
|
516a116f9a |
Name the function a failed argument binding was binding for (#8630)
* Name the function a failed argument binding was binding for A built-in call that binds badly reported neither the function nor, for a method, the right numbers: `self` was prepended before binding, so it was counted as both an expected parameter and a given argument. `[].append()` read `expected at least 2 arguments, got 1`. `Callee` carries what a message says about the function: the name, and whether the leading argument fills the instance parameter. `PyNativeFn` takes one, `PyNativeFunction` and `PyMethodDescriptor` supply it, and the four method-descriptor call specializations in `frame.rs` do the same. `bind_for` subtracts the instance from both numbers. The messages now read as their counterparts do: - `_PyArg_CheckPositional`: `find expected at least 1 argument, got 0`, with the singular when the count is one. - `_PyArg_UnpackKeywords`: `split() got an unexpected keyword argument 'bogus'`, and `cast() missing required argument 'format' (pos 1)` for a parameter a call may pass by name. A positional-only parameter is only ever counted, so it keeps the first form. A binding that happens where the name isn't known leaves it off, the way `_PyArg_Parser.fname` is NULL. `test_find_etc_raise_correct_error_messages` now passes in both `test_bytes` and `string_tests`. Assisted-by: Claude * Name the type whose constructor or initializer bound the arguments `Constructor::slot_new` and `Initializer::slot_init` bound their arguments without a name, and so did the slots that override them, so a constructor that was called wrongly said only `expected 1 argument, got 0`. Both defaults and every override now name the type the slot was written for, not the subclass being constructed and not the module `tp_name` carries: `float expected at most 1 argument, got 2`, and `deque expected at most 2 arguments, got 3` for a `deque` subclass. `slice()` and `range()` counted their own arguments and raised messages of their own; they now raise the one `_PyArg_CheckPositional` raises, which makes `test_range_constructor_error_messages` pass. Assisted-by: Claude |
||
|
Jeong, YunWon
|
322f294a21 |
common: add VM-independent codec and text engines (#8622)
* common: add PyPy CJK codec engines
Assisted-by: Codex:GPT-5
* common: mark CJK mapping tables generated
Assisted-by: Codex:GPT-5
* common: add reusable byte and text engines
Assisted-by: Codex:GPT-5
* common: record opt-in engine dependencies
Assisted-by: Codex:GPT-5
* stdlib: add `_multibytecodec` and the six `_codecs_*` modules
Wire the CJK codec engines in `rustpython-common` up to the VM. `_multibytecodec`
holds `MultibyteCodec`, `MultibyteIncrementalEncoder`, `MultibyteIncrementalDecoder`,
`MultibyteStreamReader` and `MultibyteStreamWriter`, the encode/decode drivers and
the error-handler protocol; each `_codecs_*` module exposes `getcodec` for the
encodings of its region.
`rustpython-stdlib` now enables the `cjk-codecs` feature of `rustpython-common`.
Add `cjk::decode_reset`, which the decoder `reset()` methods need. It differs from
building a fresh decoder state: ISO 2022 keeps its G1..G3 designations and the
escape-throughout flag, and resets only G0 and the shifted flag.
Drop the markers from the CJK codec tests and from the `test_codecs`, `test_io`,
`test_email`, `test_zipfile` and `test_traceback` cases that needed a CJK codec.
`test_codecs.BasicUnicodeTest.{test_basics,test_decoder_state}` stay expected
failures for an unrelated reason: `codecs.charmap_encode` rejects a `None` mapping.
Assisted-by: Claude Code:claude-opus-5
* stdlib: use the byte and text engines from `rustpython-common`
`binascii`, `_json` and `_socket`'s address converters now call the engines in
`rustpython-common` instead of carrying their own copies, and `rustpython-stdlib`
enables the `binascii`, `inet` and `json` features for them. That leaves
`base64` and `crc32fast` with no caller in `rustpython-stdlib`, and
`crates/stdlib/src/json/machinery.rs`, the last of the json_in_type-derived code,
goes away with its last caller.
The address converters change behaviour. `inet_aton` reads the lenient forms
(`127.1`, `16909060`, `0x7f.1`, `0177.0.0.1`), `inet_pton` rejects an octet with a
redundant leading zero and a group of five hex digits, `inet_ntop` writes the
IPv4-compatible form as `::c000:201`, and an address family with no converter
raises `EAFNOSUPPORT` instead of a bare `OSError`.
Give the JSON engine the closing-quote scan and the allocation-free `\uXXXX`
escape it needs to carry `_json`'s workload. Against the code it replaces,
`json.dumps` runs 2.3x faster with `ensure_ascii=True` and 1.7x without it, and
`json.loads` 1.2x.
`binascii.a2b_base64` now reports the messages `binascii.c` does: no
`error decoding base64: ` prefix, and the data-character count in parentheses.
Assisted-by: Claude Code:claude-opus-5
* test: enable the Windows cp932 codec case
The shared CJK engine now satisfies the Windows code-page adapter, so keeping this test marked as an expected failure turns its success into a CI failure.
Assisted-by: Codex:GPT-5
|
||
|
Hanjeong Lee
|
d1bbf4bc1a |
Report every generated __text_signature__, and keep it out of __doc__ (#8614)
* Strip the signature prefix from builtin __doc__
A function's signature and its documentation share one string, separated
by `\n--\n\n`, as in CPython. __text_signature__ read the front of it,
but __doc__ returned the whole thing:
>>> len.__doc__
'len(obj, /)\n--\n\nReturn the number of items in a container.' # was
'Return the number of items in a container.' # now
44 of the 46 builtin functions leaked the prefix.
get_doc_from_internal_doc already stripped it but was reachable only from
PyType.__doc__. Split it the way CPython does, into doc_without_signature
(_PyType_DocWithoutSignature) and a wrapper mapping an empty result to
None (_PyType_GetDocFromInternalDoc), and call it from PyNativeFunction
and PyMethodDescriptor as well.
Assisted-by: Claude Code:claude-opus-5
* Terminate a generated __text_signature__ that carries no documentation
A signature and its documentation share one string, separated by
`\n--\n\n`, and readers find the signature by searching for that marker.
A #[pyfunction] with no doc comment stored the bare signature, which no
reader could parse:
>>> time.clock_getres.__doc__
'clock_getres(clk_id, /)'
>>> time.clock_getres.__text_signature__
None
Emit the marker in that case too, as Argument Clinic does for an
undocumented function. 136 of the 915 functions reachable from the
importable modules were affected.
__doc__ now reports None for them, matching CPython, because nothing
follows the marker.
Assisted-by: Claude Code:claude-opus-5
* Report a __text_signature__ for every method
A #[pymethod] built its doc only when the method carried a doc comment,
so one without a comment dropped the generated signature too:
>>> list.append.__text_signature__
None
Build it from either part, as #[pyfunction] does. All 1101 method
descriptors reachable from the builtin types now report a signature, up
from 1, and 713 of the 1030 that CPython also describes match it exactly.
A method that takes its receiver as an ordinary first argument instead of
`&self` reported that argument by name, so binding the method could not
drop it:
>>> inspect.signature([].__dir__)
(obj, /) # was
() # now
Mark it $self, or $type for a classmethod, as CPython does. 310 methods
were affected.
test_unbound_builtin_method_noargs and test_bound_builtin_method_noargs
pass now.
Assisted-by: Claude Code:claude-opus-5
* Drop the expectedFailure on test_simple_completion
rlcompleter closes the parenthesis for a callable whose signature takes
no parameters, and leaves it open when the signature cannot be read:
def _callable_postfix(self, val, word):
if callable(val):
word += "("
try:
if not inspect.signature(val).parameters:
word += ")"
except ValueError:
pass
os.getpid reports `()` since the generated signature carries the `--`
terminator, so completing `os.getpid` yields `os.getpid()` and the test
passes.
Assisted-by: Claude Code:claude-opus-5
* Name every #[pymethod] receiver zelf
A method that needs an owned handle cannot take `&self`, so it takes the
receiver as its first argument and names it `zelf`, `self` being a Rust
keyword. 24 of the 163 such methods named it something else: `instance`
(11), `obj` (4), `exc` (3), `_self` (3), `_instance` (2).
Rename them. The generated signature is unaffected either way, since the
macro marks the receiver by position rather than by name.
Assisted-by: Claude Code:claude-opus-5
* Drop the unreachable zelf name check in func_sig
The receiver is now marked by position, so the name it carries no longer
matters, and no path reaches this check with an argument named zelf:
- a #[pymethod] or #[pyclassmethod] without a `&self` receiver has its
first argument replaced by the marker before the name is read
- a `&self` receiver is handled as syn::FnArg::Receiver
- a #[pymethod(raw)] is passed to static_raw_func, whose PyNativeFn bound
fixes its signature to (&VirtualMachine, FuncArgs)
- no #[pystaticmethod] or #[pyfunction] names an argument zelf
- #[pymember], #[pygetset] and #[pyslot] never reach func_sig
Every generated signature is byte for byte identical without it.
Assisted-by: Claude Code:claude-opus-5
* Keep the argument bundle when marking the receiver
A #[pyclassmethod] that takes FuncArgs receives the class inside the
bundle along with every other argument, but the marker was spent on the
bundle and the arguments went unreported:
>>> object.__subclasshook__.__text_signature__
'($type, /)' # was, claims it takes nothing
'($type, *args, **kwargs)' # now
Report both, the shape CPython uses for __new__. Two methods take their
arguments this way, __subclasshook__ on object and on type.
Assisted-by: Claude Code:claude-opus-5
|
||
|
Jeong, YunWon
|
5b1a1c08d7 |
Leave a byte that starts no UTF-8 sequence at the end of a console write (#8621)
* Leave a byte that starts no UTF-8 sequence at the end of a console write `find_last_utf8_boundary` read every byte at or above 0xc0 as the start of a sequence, giving 0xf8..=0xff an expected length of 4 and cutting the tail back to before it. `_find_last_utf8_boundary` stops the four-byte range at 0xf8 and returns the full length for anything above it, leaving the byte for MultiByteToWideChar to answer with U+FFFD. `write_console_utf8` therefore reported fewer bytes written than it was given whenever one of those bytes fell in the last three, and for a buffer that began with one it cut the length to 0, converted nothing and returned Ok(0) -- a write that never advances. The scan also ran a fourth iteration the C loop does not; the arms happen to answer the same there, but the bound is now the same 3. * Drop the expectedFailure on test_winconsoleio's test_write The marker went stale with the boundary fix in the parent commit. The test writes `b'\xff'*10` and `data + b'\xff'` to CONOUT$ and asserts the returned count, which is the case `find_last_utf8_boundary` was answering wrong, so regrtest now reports it as an unexpected success. |
||
|
Lee Dogeon
|
578217e1f6 |
Fix tuple subclass preservation in type bases (#8420)
* Fix tuple subclass preservation in type bases Assisted-by: Codex:gpt-5.6-sol * Refine type bases representation Store validated base classes in the original typed tuple, while retaining a bootstrap vector only until a Python tuple can be materialized. Assisted-by: gpt-5.6-sol Assisted-by: Codex:gpt-5.6-sol * Bootstrap tuple type with type hierarchy Initialize tuple and the canonical empty tuple alongside type and object so every type stores an actual typed bases tuple from construction. Remove the bootstrap bases representation and keep raw initialization Miri-safe. Assisted-by: Codex:gpt-5.6-sol |
||
|
김은빈
|
8b65a5455f | feat: reuse weak callable proxy (#8618) | ||
|
Jeong, YunWon
|
2c8d3ce50b |
Implement PEP 667 frame locals semantics (#8602)
* Implement PEP 667 frame locals semantics Assisted-by: Codex:GPT-5 * Fix C API test CString import Assisted-by: Codex:GPT-5 * Address PEP 667 review findings Assisted-by: Codex:GPT-5 |
||
|
Jeong, YunWon
|
7c6c78f955 |
Implement CPython 3.14 subinterpreter APIs (#8605)
* Add _interpreters and _interpchannels modules
Apply the patch baseline for the low-level subinterpreter modules:
- crates/vm/src/stdlib/_interpreters.rs: create/destroy/list_all/get_main/
get_current/is_running/exec/call/run_string/run_func/set___main___attrs/
is_shareable/whence and the Interpreter*Error exceptions
- crates/vm/src/stdlib/_interpchannels.rs: channel create/destroy/send/recv/
list_all/list_interpreters/release/close and the Channel*Error exceptions
- crates/vm/src/vm/crossinterp.rs: cross-interpreter data protocol
- crates/vm/src/vm/{interpreter,mod,runtime,vm_new}.rs: interpreter registry
and lifecycle plumbing the modules need
- Lib/concurrent/interpreters/: high-level PEP 734 package
Assisted-by: Claude:claude-opus-5
* Correct _interpreters and _interpchannels against CPython 3.14
Argument matching:
- Add `function::ArgSpec`, a `PyArg_ParseTupleAndKeywords` equivalent that
applies the `|`, `$` and `:` markers of a format string to a `kwlist`, and
route every module function of both modules through it. This enforces
keyword-only parameters, rejects unexpected keywords, accepts keyword forms
that were previously positional-only (`whence(id=)`, `get_config(id=)`,
`set___main___attrs(id=, updates=)`, `capture_exception(exc=)`,
`_register_end_types(send=, recv=)`), and reproduces the arity messages.
- `O!` slots (`shared`, `updates`, `call`'s `args`/`kwargs`) now reject None,
and `_PyArg_BadArgument` renders None as "None" rather than "NoneType".
- `new_config` takes at most one positional `str`; `list_all` of
`_interpchannels` is argument-less.
- Convert arguments in `kwlist` order, so an argument's own error is raised
ahead of the checks that follow it.
_interpreters:
- Add the `CrossInterpreterBufferView` type and build the received memoryview
on it, keeping the sending interpreter's exporter out of the destination.
- Build `excinfo.errdisplay` from `traceback.TracebackException`.
Feature flags:
- `os.fork` checks finalization before `allow_fork`, matching `os_fork_impl`.
- `_thread.start_new_thread` and `start_joinable_thread` reject an interpreter
without `allow_threads`.
Lib/concurrent/interpreters/_crossinterp.py: restore the stripped docstrings
and comments.
Assisted-by: Claude:claude-opus-5
* Correct the stateless checks and error shapes of _interpreters against CPython 3.14
verify_stateless_function follows _PyFunction_VerifyStateless: it rejects
non-dict builtins, a non-empty __defaults__, __kwdefaults__ or __closure__,
and code whose LOAD_GLOBAL names are held by the function's globals or
absent from its builtins.
code_returns_only_none follows _PyCode_ReturnsOnlyNone: generator,
coroutine and async-generator code is rejected up front, the instruction
walk skips inline caches and maps specialized and instrumented opcodes
back, and the LOAD_CONST preceding each RETURN_VALUE is compared against
the index of None in co_consts.
ExcInfo::capture keeps an empty exception message instead of dropping it,
and builds `formatted` as `module.qualname: msg`, leaving out the builtins
and __main__ modules.
_interpreters.call packs the callable through _PyFunction_GetXIData before
pickle and re-raises the stateless-check failure when both fail. A func,
args or kwargs that cannot be rebuilt in the target, and a result without
cross-interpreter data, now raise NotShareableError with the snapshot as
the cause rather than being returned as excinfo.
_interpreters.capture_exception() without an argument returns None.
pickle_loads and _PyFunction_GetXIData attach the underlying failure as
__cause__.
channel_send converts the object to cross-interpreter data after the
channel's `closing` check.
ChannelID rich comparison returns the result of comparing its id with the
other number instead of coercing that result to bool.
PyMemoryView_FromObjectAndFlags raises "memoryview: a bytes-like object is
required, not 'X'" for an object that is not a buffer.
Assisted-by: Claude:claude-opus-5
* Correct cross-interpreter data conversion and channel argument parsing
Bound shareable ints by isize, raising OverflowError("try sending as
bytes") outside that range and keeping it as the NotShareableError cause.
Report an unshareable object by its repr, and attach a failing conversion
as __cause__ of the NotShareableError raised for the object it was called
for, so each level of a nested tuple appears in the chain.
Guard each tuple item conversion with with_recursion("while sharing a
tuple").
Gate the memoryview getdata function on `_interpreters` having been
imported, and route channel_send_buffer through a memoryview so it uses
that function and the resolved fallback.
Create the cross-interpreter exception types from the module_exec of
either module that raises them, instead of only `_interpreters`.
parse_cid raises OverflowError("int too big to convert"); int_arg
reproduces the `i` converter's three overflow messages and is also used
while parsing channel_send arguments.
Add BASETYPE to _queue.Empty, and gate nt.execv/nt.execve on allow_exec.
Assisted-by: Claude:claude-opus-5
* Correct interpreter ID parsing and the channel_send defaults lookup
_PyInterpreterState_ObjectToID accepts any object with __index__, raises
OverflowError("int too big to convert") outside the int64 range, and
reports a negative ID with the repr of the original object.
channelsmod_send reads the channel's default unboundop and fallback only
when one of the two arguments is negative, so an explicit pair is
validated before the channel is looked up.
Assisted-by: Claude:claude-opus-5
* Add the _interpqueues module and concurrent.interpreters._queues
_interpqueues holds a process-wide queue table and exposes create,
destroy, list_all, put, get, bind, release, get_maxsize,
get_queue_defaults, is_full, get_count and _register_heap_types, raising
QueueError and QueueNotFoundError. QueueEmpty and QueueFull, which
subclass queue.Empty and queue.Full, are registered per interpreter by
concurrent.interpreters._queues.
Cross-interpreter data carries a queue as a refcounted queue ID that
binds when the data is captured and releases when it is dropped, and
is_shareable reports a registered Queue as shareable.
Queue and channel items owned by an interpreter are now cleared from
Interpreter::finalize, after module finalization, rather than from
destroy_owned_interpreter, so values sent by an atexit callback also
become unbound.
is_shareable gates memoryview on the same registration flag as the
getdata lookup.
Assisted-by: Claude:claude-opus-5
* Add the high-level concurrent.interpreters API
Interpreter, ExecutionFailed and the queue aliases, copied verbatim from
CPython 3.14.
Assisted-by: Claude:claude-opus-5
* Destroy leftover subinterpreters when the main interpreter finalizes
Interpreter::finalize now runs finalize_subinterpreters between the atexit
handlers and the finalizing flag: when the main interpreter still owns
subinterpreters it emits the RuntimeWarning "remaining subinterpreters;
close them with Interpreter.close()" and then finalizes each of them.
runtime::owned_interpreter_ids lists the runtime-owned interpreters.
Assisted-by: Claude:claude-opus-5
* Absolutize the script path in run_file
_Py_abspath and _PyPathConfig_ComputeSysPath0 arrive as abs_path and
script_sys_path0. run_file hands the absolutized path to get_importer
and to the runner, so __file__ and co_filename no longer stay relative,
and inserts the symlink-resolved script directory as sys.path[0]
instead of the argument's parent.
test_cmd_line_script.test_script_abspath no longer expects a failure.
Assisted-by: Claude:claude-opus-5
* Name types in error messages the way tp_name does
The object, number, sequence and mapping protocols, the
unsupported-operand helpers, and the type-specific messages fed from
them now read PyType::slot_name() where they read PyType::name(),
so a type declared with a module keeps it.
Message texts corrected along the way:
- PySequence_Size, PyMapping_Size, PySequence_GetItem and
PySequence_SetItem/DelItem split their merged wording in two, and
PyObject_SetItem/DelItem say "object".
- PyObject_GenericSetAttr reports a read-only attribute and the
missing __dict__.
- check_class drops the type name it appended.
- __int__ and __index__ drop the class prefix; __float__ keeps it.
- float(), complex(), round(), next(), str subscripting and list
subscripting take their own wording.
- str.join and bytes.join number the item they turn down and name what
they wanted, and report a non-iterable as "can only join an
iterable".
test_class.testObjectAttributeAccessErrorMessages,
string_tests.test_subscript and the test_descrtut doctest no longer
expect a failure.
Assisted-by: Claude:claude-opus-5
* Gate script_code's source compilation on the compiler feature
crossinterp::script_code called vm.compile unconditionally, which only
exists with the rustpython-compiler feature. Builds without it, such as
example_projects/barebone, now raise a TypeError instead of failing to
compile.
Assisted-by: Claude:claude-opus-5
* Skip test_pyinit_function_raises_exception
The test imports _testsinglephase in its body without a decorator
guarding it, and now that _interpreters exists requires_subinterpreters
no longer skips it.
Assisted-by: Claude:claude-opus-5
|
||
|
Jeong, YunWon
|
6658ffa717 |
memoryview: traverse the view for the collector, and name types as tp_name does (#8608)
* memoryview: walk the view and the buffer wrapper for the collector
Neither `memoryview` nor the `_buffer_wrapper` a `__buffer__` produces was
traversed, so the object a view looks at was never reported and a cycle
running through one was classified as reachable:
class Src:
def __buffer__(self, flags): return memoryview(b'abcdef')
o = Src(); o.keep = memoryview(o) # o is never freed
Both now report what they hold, as `memory_traverse` and `mbuf_traverse` do.
Assisted-by: Claude
* Name the type in an unhashable-type message the way `tp_name` does
The message took the short name, so a type defined in a module lost the
module: `hash(array.array('B', b'a'))` said `unhashable type: 'array'`
where the name is `array.array`, and `collections.deque` said `'deque'`.
`slot_name()` is what `tp_name` holds, and it leaves a class defined in
Python reported by its own name as before.
Assisted-by: Claude
|
||
|
Jeong, YunWon
|
f9b177719b |
Fix formatting, numeric, unicode, codec, complex and marshal mismatches found by differential testing (#8600)
* Fix %-formatting of bytes precision, width and %c errors
`format_bytes` subtracted the byte length from the field width without
saturating, so `b'%1s' % (b'abcde',)` aborted with a capacity overflow.
A precision written as a bare dot now truncates to nothing, and `%r`/`%a`
on bytes now honour the precision and width like the other conversions.
`%c` with a bytes, bytearray or str argument that is not one unit long
now names the argument by its length instead of by its type.
Assisted-by: Claude Code:claude-opus-5
* Fix format spec handling of grouping, bool, 'c' and complex
A grouping option next to an unrecognised presentation type now reports
"Cannot specify ',' with 'a'." instead of falling through to the type
check. `format(True, spec)` now validates the spec the way an int does.
The 'c' type now pads and aligns its character, and an integer outside
the C long range raises OverflowError rather than reporting a code point
out of range.
A complex formatted with the 'n'/'g' types or with a precision and no
type now goes through the general format without forcing a fraction.
Assisted-by: Claude Code:claude-opus-5
* Round floats at a negative ndigits from the exact decimal digits
Dividing by a power of ten rounded twice: once when the quotient was
stored and again at the tie, so e.g. `round(2.5e2, -2)` came out as 300.0.
The digits now come from the exact decimal expansion of the value.
Assisted-by: Claude Code:claude-opus-5
* Report the requested base in int literal errors and reject a bare sign
An invalid literal parsed with base 0 named the base that the prefix
turned out to mean rather than 0. A string of nothing but a sign was
taken for the old octal form and returned 0 instead of raising.
Assisted-by: Claude Code:claude-opus-5
* Split, strip and expand tabs over the Py_UNICODE_ISSPACE set
`str.split`, `rsplit`, `strip`, `lstrip` and `rstrip` with no separator
used Rust's whitespace predicate, which differs from the 29 code points
`Py_UNICODE_ISSPACE` names: U+001C-U+001F and U+0085 were skipped and
U+200B was treated as a separator.
`expandtabs` went through `str`, so it dropped the lone surrogates of a
WTF-8 string; it now works on `Wtf8` throughout.
Indexing a str out of range now says "string index out of range".
Assisted-by: Claude Code:claude-opus-5
* Read Unicode numeric values as exact rationals
`DerivedNumericValues.txt` field 1 carries a rounded decimal, so
`unicodedata.numeric('⅐')` and its neighbours came back short of the
value CPython reports. The generator now reads the rational in field 3,
and takes the 3.2 values from field 8 of `UnicodeData-3.2.0.txt`.
Assisted-by: Claude Code:claude-opus-5
* Return a mappingproxy from Pattern.groupindex
`pattern_groupindex` wraps the group index in a dict proxy, and hands
back a plain dict only when the pattern has no named group.
Assisted-by: Claude Code:claude-opus-5
* Fix the utf-16/32/7 codecs' error handling
The utf-32 decoder dropped a code point above U+10FFFF instead of calling
the error handler, and both the utf-16 and utf-32 handlers were told the
encoding was "utf-16"/"utf-32" whichever byte order the data was in.
The encoders now take a bytes replacement from the handler as it stands,
rejecting one that does not fill whole code units, rather than iterating
over it as text; a str replacement from the utf-16 encoder now splits an
astral code point into a surrogate pair.
The utf-7 decoder names its encoding "utf7", as its codec does.
Assisted-by: Claude Code:claude-opus-5
* Rework complex multiplication, division and power
Division now divides by whichever part of the divisor is larger, the way
`_Py_c_quot` does, and both it and multiplication recover the infinities
that the plain formulas turn into nan, following C11 Annex G. An operand
that is a real number is now multiplied and divided component-wise, which
keeps the sign of a zero and the other part of an infinity; a real
numerator has its own quotient so that its zero keeps a sign too.
An integer exponent no greater than 100 in magnitude is now reached by
repeated multiplication rather than through the polar form, so e.g.
`(1+2j) ** 2` is exact. A zero exponent gives 1, and an exponent with no
imaginary part no longer reaches for a logarithm that turns an infinite
magnitude into a nan. A result that came out infinite raises
OverflowError.
`complex() arg is a malformed string` is replaced by `could not convert
string to complex: ...` when the string carries a misplaced underscore,
`complex division by zero` by `division by zero`, and the message for
zero raised to a negative or complex power no longer names the base.
Assisted-by: Claude Code:claude-opus-5
* Intern the string constants that read as names
A code object's string constants were interned by length, so a short
string that is not a name was interned and a long name was not. They are
now picked out the way `all_name_chars` does. The one-character latin-1
strings are interned as well, so that every route to one of them lands on
the same object.
Assisted-by: Claude Code:claude-opus-5
* Honour the marshal version and separate its EOF errors
Versions below 2 now write a float or complex as its digits under
TYPE_FLOAT/TYPE_COMPLEX, versions from 4 on write a tuple of fewer than
256 items under TYPE_SMALL_TUPLE and a long ascii string under
TYPE_ASCII, and a string is marked interned only when it is.
`marshal.loads` now tells apart the three ways its data can run out:
"EOF read where object expected" where an object was to start, "EOF read
where not expected" where a single byte was, and "marshal data too short"
where a run of bytes was cut short.
Assisted-by: Claude Code:claude-opus-5
|
||
|
Jeong, YunWon
|
0ae835a073 |
Give heap types a dict namespace (#8606)
Replace `PyType::attributes` (`PyRwLock<PyAttributes>`) with a `TypeNamespace` enum. A type created while an interpreter is running holds a `PyDict`; the types `Context::genesis` and `PyType::new_static` build keep the interned-key `IndexMap`, since hashing a string needs a VM. `type.__new__` binds `__classdictcell__` to the type's own namespace rather than the namespace dict passed to it, so an annotation scope reads attribute changes made after the class body ran. Removes the `expectedFailure` on `test_type_params.TypeParamsClassScopeTest.test_modified_later`. `PyGetSet` holds its class as a `PyRef<PyType>` instead of a non-owning `PointerSlot`, and traverses it, because a namespace can outlive the type it belongs to. `PointerSlot` and the `unsafe` on `Context::new_getset` are gone. `subs_parameters` raises `TypeError` when `__typing_subst__` returns a non-tuple in an unpack position, using the new `PyType::fully_qualified_name` for the message. Removes the `expectedFailure` on `test_typing.GenericTests.test_return_non_tuple_while_unpacking`. Assisted-by: Claude |
||
|
Shahar Naveh
|
fab3fdb704 |
Update some libs and tests to 3.14.7 (#8603)
* `typing.py` * tempfile * tarfile * imaplib |
||
|
Shahar Naveh
|
0fdd061ed4 |
Update some tests & libs to 3.14.7 (#8595)
* test_memoryio * test_ordered_dict * more * urllib * test_compileall * urllib test * more * test_descr * test_enumerate |
||
|
Joshua Megnauth
|
895be40542 |
Update test_unicodedata to 3.15 (#8598)
See: #8548 |
||
|
Shahar Naveh
|
a6d88b144e |
Update some libs and tests to 3.14.7 (#8593)
* test_weakref * test_inspect * socket * test_copy * pickle * test_ast * string * zstd * test_py_compile * test_os * mark failing test * mark failing * fix markers |
||
|
Luan Taraschi
|
782d5d2639 |
Warn instead of aborting when a class namespace holds a non-string key (#8581)
* Warn instead of aborting when a class namespace holds a non-string key
type() took the whole process down when the namespace it was handed had
a key that was not an exact str:
>>> type("MyClass", (), {1: 2})
thread 'main' panicked at crates/vm/src/builtins/dict.rs:781:65:
dict has non-string keys: [PyObject PyInt { value: 1 }]
to_attributes called expect() on downcast_exact, so a str subclass hit
the same line, and that one names an attribute as well as a str does.
PyAttributes is keyed by interned strings and can hold nothing else, so
a str subclass is now interned like any other name, and a key that is no
kind of string is skipped, with the caller deciding what the skip means.
type() warns once per class, the way CPython does, and
_thread.start_new_thread raises the TypeError an ordinary call already
raises for the same mapping.
The key itself still does not land in the class dict, which is what
test_type_lookup_mro_reference needs, so that one stays skipped with the
reason rewritten.
Assisted-by: Claude Code:claude-opus-5
* Mark test_type_lookup_mro_reference as an expected failure
It aborted the runner before, which is why it was skipped. Now that it
only fails, the run can carry it: five runs in a row report it as an
expected failure, and it cannot turn into a pass while the class dict
drops the key its lookup needs.
Assisted-by: Claude Code:claude-opus-5
|
||
|
Joshua Megnauth
|
9b6feb5132 |
unicodedata: Fix Bidirectional, improve parser (#8548)
We can save space by calculating the diffs between modern Unicode and 3.2.0 and storing membership information where valid. This avoids storing entire tables for 3.2.0, and is also more correct in the long run since it handles absence from 3.2.0 correctly. I switched over Bidi to this new method which partially fixed the test. Unfortunately, our Unicode data is more up to date than Python 3.14 so the test fails for modern Unicode. I updated the test to 3.15 as per review, and now our bidi passes the test. |
||
|
Jeong, YunWon
|
f87cfe7e7a |
Match CPython on code objects, ast, symtable and syntax error positions (#8580)
* Guard all/any/tuple(genexpr) calls when the generator is a coroutine
`maybe_optimize_function_call()` reserves a `skip_optimization` label for
every `name(genexpr)` call and, for `all`, `any` and `tuple`, emits an
identity guard against the builtin plus an inlined loop. The port skipped
both whenever the generator expression's symbol table was a coroutine, so
`all(await x for x in xs)` and `all(x async for x in xs)` compiled to a
plain call.
An `await` or an `async for` in the generator does not disqualify the
shape: the inlined `FOR_ITER` raises the same `TypeError` that calling the
builtin on an async generator raises, which is what
test_builtin test_builtin_call_async_genexpr_no_crash asserts.
This was the last code-level difference against CPython across `Lib/`:
comparing every file's code tree by structure, opcode and constant value
(qualnames aside, which moved in 3.14.3+, and set constants compared by
value rather than by hash order) now matches on 1720 of 1720 comparable
files, up from 1719.
Assisted-by: Claude Code:claude-opus-5
* Name the ellipsis type `ellipsis`
`type(...).__name__`, its `__qualname__`, and `types.EllipsisType.__name__`
read `EllipsisType`.
Assisted-by: Claude Code:claude-opus-5
* Align compiler front end and instruction positions with CPython
Reject a bare generator expression in a class header and a format spec
nested more than two deep. Both parse here; the reference grammar has no
rule for either, since a class header only takes `arguments` and the
tokenizer runs out of nesting levels at the third format spec.
Scan the interpolations a format spec carries inside another format spec,
so `f"{x:{y:{z}}}"` reaches `z` when building the symbol table.
Give every generic function's type params scope a `.defaults` slot,
whether or not the signature has any default, so `co_varnames` and
`co_nlocals` match. Classes and type aliases keep theirs empty.
Decide an unparenthesized sole generator argument by the parser's
`parenthesized` flag instead of scanning the source for the shape of its
element. `_sum((d := x - c) * d for x in data)` opens with a parenthesized
group the rest of the expression continues, which the scan read as the
generator's own parentheses and left the call's out of its range.
Drop empty f-string fragments unconditionally. A concatenation whose
fragments are all empty now loads its empty string at the whole
concatenation instead of at whichever fragments happened to be plain
string literals.
Keep a byte order mark in source handed over as text, and report it as an
invalid non-printable character. Reading a file to freeze strips it, as
the encoded-bytes path already did.
Carry the filename into the errors `compile(..., PyCF_ONLY_AST)` raises;
they reported `<unknown>` or an empty name.
Across `Lib/`, the 1729 files that compile on both sides now agree on
opcodes, constants, flags, variable names, names, and exception tables,
and on instruction positions in all but 7 files, where a compound
statement whose body ends in a trailing `;` ends one column short.
Assisted-by: Claude Code:claude-opus-5
* Align ast module output with CPython
- Keep lone surrogates in string, f-string and t-string literal values by
reading such a literal from its source; the literal-value helpers move
out of `Compiler` into `rustpython_codegen` and `ConstantLiteral::Str`
holds `Wtf8Buf`.
- Give `{expr=}` interpolations their leading text as a Constant and
default their conversion to `repr`, in both f-strings and t-strings.
- Span folded consecutive literals from the first of them to the last.
- Take the `:` into a format spec's range only when the parser left it
out, instead of always widening by one.
- Report a node end that lands on a line start as that line, column 0.
- Give a generator expression written straight into a call's parentheses
the range of those parentheses, in the preprocess pass, replacing
codegen's source scan and `_ast`'s one-column adjustment.
- Drop the expectedFailure on test_ast_line_numbers_with_parentheses.
Assisted-by: Claude Code:claude-opus-5
* Align symbol table output with CPython
- Register an import as `DEF_IMPORT` alone, and an annotated parameter as
a plain parameter, so `import x; global x` is accepted and annotated
parameters are not reported as annotated names.
- Record a `global` declaration in the module block as well, and resolve a
name a `global` in an enclosing scope covers as an implicit global, so a
class body reading such a name emits `LOAD_NAME`.
- Resolve a name bound nowhere as an implicit global instead of leaving
its scope unknown.
- Require a binding only for a name an explicit `nonlocal` declared;
a free variable that reached the scope any other way already resolved.
- Name a generator expression's block `genexpr`; the code object keeps
`<genexpr>`.
- Locate a `def`/`class` block at its keyword rather than at the first
decorator. `decorated_definition_range()` moves from `Compiler` to
`rustpython_codegen` for that.
- Carry a comprehension symbol's flags over when inlining it into a parent
entry that only holds a free variable propagated from a child.
- Leave `__conditional_annotations__` out of an annotation block's
symbols and have the compiler cook up the free variable, which places
it after the ones the symbol table supplies.
- Check `from __future__` placement while building the symbol table, and
give `symtable.symtable()` the same docstring stripping and future
feature validation the compiler does.
- Floor `co_stacksize` at 1.
Assisted-by: Claude Code:claude-opus-5
* Locate the eval return and the interactive print like CPython
Leave `eval` mode's `RETURN_VALUE` without a location, so the exit block
is duplicated per reaching path and each copy carries the location of the
branch that jumps to it. `a and b` now ends with two returns as it does
in CPython, and `a if b else c` gives the first one the `a` it returns.
Emit an interactive `CALL_INTRINSIC_1(Print)` at the expression statement
rather than wherever the expression left the location, which differ for
`(a := 10)` and `f'{0:fz}'`.
Assisted-by: Claude Code:claude-opus-5
* Give interactive mode the conditional annotations cell
`compile_program_single()` emitted the `__conditional_annotations__` set
and its name accesses but never added the cell that holds it, so a module
compiled in `single` mode was missing the `MAKE_CELL` and the cellvar that
the same source gets in `exec` mode.
Assisted-by: Claude Code:claude-opus-5
* Give compiler-raised SyntaxErrors an end position
`CodegenError` and `SymbolTableError` now carry an `end_location`
alongside `location`, and `CompileError::python_end_location()` returns
it, so errors such as "'await' outside function" set `end_lineno` and
`end_offset` instead of leaving them None.
`SymbolTableBuilder::error_ranged()` builds both positions from a range,
replacing the inline `source_location()` calls at the error sites.
Drop the `expectedFailure` on
`test_exceptions.SyntaxErrorTests.test_file_source`, which now passes.
Assisted-by: Claude Code:claude-opus-5
* Report parser diagnostics with CPython's wording
`InvalidAnnotatedAssignmentTarget` becomes "illegal target for
annotation", `UnexpectedExpressionToken` becomes plain "invalid syntax"
rather than prefixing the parser's text, and the list-recovery
diagnostics ("Expected an expression or a '}'" and its siblings) join the
two that already mapped to "invalid syntax".
Assisted-by: Claude Code:claude-opus-5
* Punctuate the code object repr like CPython
`repr(code)` was missing the comma after the address and escaped the
filename through Rust's Debug formatting.
Drop the `expectedFailure` on the six `test_dis` tests that compare
disassembly text, which now pass.
Assisted-by: Claude Code:claude-opus-5
* Match CPython's parser error positions
A syntax error the parser reports counts its columns in characters, so
`source_location()` converts with UTF-32 rather than UTF-8; an offset
inside a character walks back to where that character starts.
Spans the parser reported differently:
- an error between two tokens covered nothing, and now covers one
character, since the narrowest token still covers one
- a "was never closed" bracket ends at column 0
- an unterminated string literal ends where it starts
- "leading zeros in decimal integer literals" spans the run of zeros
- an exponent with no digits and no sign is blamed on the digit before
the exponent letter, which the tokenizer puts back
Assisted-by: Claude Code:claude-opus-5
* codegen: seek to the finally body's own scopes before copying it
An early exit from a try block emits an extra copy of the finally body.
Nested scopes are handed out by position, and the cursors still sit inside
the try block's own run of scopes there, so the copy took those instead of
the ones the finally body opens: a generator expression got a sibling's
symbol table and raised "the symbol 'k' must be present in the symbol
table", or, where the sibling was a named scope, built a code object with
no `.0` argument that raised UnboundLocalError when it ran.
Seek the cursors to the first scope beginning on or after the finally
body's first line before compiling the copy. The existing restore then
leaves the try statement's own copies starting from the same place.
Assisted-by: Claude
|
||
|
Luan Taraschi
|
97a142d809 |
Bound the recursion in exception group split and subgroup (#8584)
Both walk nested groups by calling themselves back through
vm.call_method, and neither pushes a Python frame, so the frame limit
never sees the nesting and the native stack runs out first. A group
nested past the recursion limit segfaults:
e = TypeError(1)
for _ in range(exceeds_recursion_limit()):
e = ExceptionGroup("eg", [e])
e.split(TypeError) # dumps core
vm.with_recursion is the guard for exactly this, the same one comparison
and repr already use, and it turns the crash into the RecursionError
CPython raises. test_deep_split and test_deep_subgroup were skipped on
the segfault and now pass.
Assisted-by: Claude Code:claude-opus-5
|
||
|
Luan Taraschi
|
0c9fa4219d |
Stop skipping six tests whose reason no longer reproduces (#8583)
Three skips in test_array say "Hangs", one in test_marshal carries no reason at all, and two in test_thread say "panic" on Linux. None of the six does any of that now. Each ran three to five times in a row and passed, and the array trio was also checked case by case against CPython 3.14, including the mutation-during-index-conversion shapes the tests were written for. The neighbouring markers that say "Flaky", "Flakey on CI" and "flaky EOFError" are left alone. One green run says nothing about those. Assisted-by: Claude Code:claude-opus-5 |