mirror of
https://github.com/RustPython/RustPython.git
synced 2026-10-28 18:59:04 +08:00
main
86
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 youknowdotandCodex
|
648b3b886b |
Support Python 3.15 array half and complex formats (#9419)
Add e, Zf and Zd storage, typecode routing, component byteswap and portable pickle decoding, preserving half float32 narrowing and complex sequence comparison semantics. Migrate test_array unchanged from CPython d220409 via update_lib. Replace byte-allocation ownership casts with aligned typed copies and correct signed pickle format selection. Assisted-by: OpenAI Codex: model version unavailable Co-authored-by: Codex <codex@openai.com> |
||
|
Joshua Megnauth
|
a2c11aa51d |
Optional Unicode 3.2.0 but enabled by default (#9011)
Useful for smaller builds, like WebAssembly. The UCD 3.2.0 path is only directly used in a handful of places in Python. Assisted-by: GPT-5.6-Luna |
||
|
Jeong YunWon
|
e1dba29fb2 |
Revert "Drop parking_lot_core git patch and tkinter deps"
Tag 0.6.0 keeps that removal. This restores the parking_lot_core git patch and the tkinter dependencies on main. Assisted-by: Grok:4.7 |
||
|
Jeong YunWon
|
b2626c56de |
Drop parking_lot_core git patch and tkinter deps
Comment out the parking_lot_core git patch and use the crates.io crate. Remove tcl-sys, tk-sys, and the tkinter feature. Assisted-by: Grok:4.7 |
||
|
Alessio Attilio
|
65647e99ee |
fix(csv): quote non-numeric objects in writer QUOTE_NONNUMERIC mode (#8843)
* fix(csv): quote non-numeric objects in writer QUOTE_NONNUMERIC mode * style: format extra_tests/snippets/stdlib_csv.py Assisted-by: Antigravity:Gemini-3.8-Flash |
||
|
Jeong, YunWon
|
be23b2c562 |
Store database docstrings xz-compressed behind an optional doc feature (#8836)
* Skip Python-defined classes when collecting builtin type docs generate.py recorded every object.__subclasses__() entry as builtins.<name> after importing the whole stdlib, so classes written in Python (tkinter, unittest, tarfile, ...) ended up in the doc database. Classes that are mutable heap types without C method descriptors and whose module is not a C extension are now skipped, and their entries are removed from data.inc.rs. Assisted-by: Claude:claude-opus-5-5 Assisted-by: Grok:grok-4.7 * Store database docstrings xz-compressed behind an optional doc feature - rustpython-doc's build script compresses all database docstrings into one xz blob with an offset index; identical texts share one offset. - Macros emit blob references for database docs; vm decompresses the blob on first access. - New `doc` cargo feature (default on) in rustpython, rustpython-vm and rustpython-stdlib. Without it the blob and xz are not linked and database-backed __doc__ is None; __text_signature__ is unaffected. - Native types without a docstring get __doc__ = None in their dict. - Workflows that build with --no-default-features enable `doc` where tests or releases need docstrings. Assisted-by: Claude:claude-opus-5-5 Assisted-by: Grok:grok-4.7 |
||
|
Jeong, YunWon
|
c382a607bb |
host_env: keep rustls off the rustls-free ssl surface (#8727)
* host_env: keep rustls off the rustls-free ssl surface ssl is MemoryBIO, constants, OID, and ALPN. ssl-rustls is the rustls engine. Drop the wasm32 rustls-pki-types web dep; the browser crate already enables that feature. Assisted-by: Claude * host_env: drop rustls-pki-types from the lockfile Assisted-by: Claude * host_env: name the rustls engine feature rustls ssl is the rustls-free surface. rustls is the engine. stdlib's ssl-rustls product feature still selects this. Assisted-by: Claude |
||
|
Jeong, YunWon
|
3191ecaf38 |
host_env: share thread, locale, loader and BSTR (#8702)
* host_env: share thread, locale, loader and BSTR
Add Windows thread name, locale-info/wcscoll, CompareStringOrdinal,
LoadLibraryEx/GetProcAddress/FreeLibrary, BSTR and COM helpers, and
AF_HYPERV UUID conversion. Wire _thread, locale, ctypes LoadLibrary
and WMI BSTRs through those wrappers. Ensure =X: before spawnve/execve.
Assisted-by: Claude
* host_env: expose rustls-free ssl core on wasm
Compile MemoryBIO, protocol constants, OID tables, hostname
checks, and ALPN parsing without the rustls feature so wasm
and other hosts can implement _ssl on the shared types.
Assisted-by: Claude
* stdlib: add wasm _ssl on rustls-free host_env
Install _ssl on wasm32 against rustpython_host_env::ssl.
MemoryBIO, constants, OID, hostname, and ALPN work. Context
settings are stored; wrap/handshake and rustls-only loaders
raise SSLError until a wasm TLS engine is wired.
Assisted-by: Claude
* host_env: address #8702 review comments
Cache-miss Windows ctypes lookup uses GetProcAddress. FreeLibrary
propagates OSError. start_new_thread returns GetThreadId. Export
_thread._get_name and drop pyfunction doc comments. Reject
embedded NULs in Windows locale.strcoll/strxfrm. Terminate
wcscoll/wcsxfrm and SetThreadDescription inputs.
wasm _ssl accepts wrap_bio owner and get_ca_certs(binary_form),
and MemoryBIO.read(-1) returns all pending bytes.
Assisted-by: Claude
* host_env: gate the whole ssl module on feature ssl
Compile rustpython_host_env::ssl only with the ssl feature.
Rustls crates stay native-only, so enabling ssl on wasm does
not pull rustls. Inside the module, rustls files follow
not(wasm32) rather than a second feature flag.
Assisted-by: Claude
* host_env: fix Windows types and thread names
Cast SysAllocStringLen to *mut u16 and RPC status to u32 so
windows-gnu and MSVC agree. Export _NAME_MAXLEN, truncate names
at NUL and the OS byte cap, and decode get_name with fsdecode.
Assisted-by: Claude
* host_env: compile rustls engine on WASI
Keep rustls-free MemoryBIO/constants/OID/ALPN on all wasm
targets. Compile the rustls engine on WASI, where UnixTime::now
exists. Allow the rustpython ssl feature on wasm without
ssl-rustls or ssl-openssl.
Assisted-by: Claude
* compiler: pick the earliest CPython parse override
A later incompatible-prefix scan must not hide an earlier
number or bracket diagnostic. Parser-driven tokenization
stops at the first failure.
Assisted-by: Claude
* host_env: address remaining #8702 review comments
Map a missing Windows export to a symbol-load error, keep
LoadLibrary's Win32 code on the OSError, drop incomplete
UTF-16 pairs when truncating thread names, hide set_name
and _get_name off linux/macOS/Windows, and let ssl-rustls
enable host_env.
Assisted-by: Claude
* stdlib: add wasm _socket and read-all MemoryBIO
Register a rustls-free-style _socket shim on wasm32 so
Lib/ssl.py can import. Treat MemoryBIO.read(-1) as read-all,
matching _ssl_MemoryBIO_read_impl.
Assisted-by: Claude
* host_env: fix clippy, WASI ssl cfg, and wasm rustls
Defer unclosed-bracket overrides until no token diagnostic
exists, keep WASI rustls and wasm ssl modules exclusive,
preserve lone Windows surrogates, compile the rustls engine
on browser wasm, and clear clippy denials.
Assisted-by: Claude
* host_env: fix Windows clippy and wasm UnixTime
Gate thread-name byte truncation to linux/macOS, allow
GetProcAddress/FreeLibrary pointer deref, and enable
rustls-pki-types web time on browser wasm.
Assisted-by: Claude
* host_env: lock rustls-pki-types web for wasm
Assisted-by: Claude
* host_env: silence Windows unused thread id and shear
Assisted-by: Claude
* stdlib: drop Windows locale clippy needless wraps
Windows clippy failed on locale.strcoll/strxfrm needless return
and _getdefaultlocale always wrapping Ok.
Assisted-by: Claude
* compiler: keep lexer errors over print hints
print x; 0x must report an invalid hex literal. Ranking by source
offset let the earlier print hint win. Treat a finished radix
literal as valid so print x; 0x1 still gets the print message.
Assisted-by: Claude
* compiler: keep decode errors over unclosed openers
Ranking by offset made ( '\\N' report an unclosed paren and
wiped the unclosed-bracket flag on f'{, so compile snippets
raised SyntaxError instead of incomplete input. Export wasm
socket._accept for Lib/socket.py.
Assisted-by: Claude
* wasm: drop duplicate _socket; split locale cfgs
_socket lives in rustpython-stdlib's socket_wasm. The wasm crate
only re-exported that file when freeze-stdlib was off. Split
strcoll, strxfrm, and getencoding into cfg'd function defs.
Assisted-by: Claude
* compiler: keep decode blockers over unclosed openers
(print x; '\\N' must keep the print hint. A later malformed
escape still suppresses the EOF opener even when print wins.
Assisted-by: Claude
* compiler: keep earlier f-string over later lexer errors
f'{x'; ' and f'{x'; 0x must not report the trailing quote or
hex literal. Decode and f-string diagnostics now compete with
lexer failures by offset.
Assisted-by: Claude
* compiler: rank decode vs lexer vs print by class
Unterminated quotes no longer hide format-spec newlines or
empty f-string fields. Print is considered only after the
final winner is not a lexer failure, so print x; f'{x'; 0x
keeps the print hint. Line-continuation uses CPython wording.
Assisted-by: Claude
* compiler: keep unclosed paren over line-cont at EOF
test_exceptions wants '(' was never closed at the opener when
a final backslash follows. Skip escaped quotes in the bracket
scan so {\'a\'} still reports the continuation error. Un-xfail
test_join_daemon_thread_in_finalization.
Assisted-by: Claude
* test_threading: xfail daemon join finalization off Linux
The test passes on Linux. macOS and Windows leave an ignored
deallocator exception on stderr.
Assisted-by: Claude
* vm: hang daemon threads at interpreter finalize
Forced-off daemons must stay not-done so join() during
finalization GC raises PythonFinalizationError and
Thread.is_alive() stays true. SystemExit made the handle
look finished on macOS/Windows. Drop the Lib/test xfail.
Assisted-by: Claude
(cherry picked from commit
|
||
|
Jeong, YunWon
|
73e547a9c3 |
Honor generator close, throw context, and docs (#8701)
* Honor close unraisable and deallocator messages A failed close lookup on a yield-from target is unraisable. Ignored GeneratorExit from finalize gets a traceback and the closing-generator err_msg. __del__ uses the deallocator message. Assisted-by: Grok:grok-4.6 * address review: claim, ag_running, frame steal Keep the running claim through close() cleanup. ag_running reads running_async. aclose ignore-yield does not mark the generator closed. Steal the frame slot before uniqueness and expose it with try_to_owned. Assisted-by: Grok:grok-4.6 * Honor wrapper docs, yield assign, throw context method-wrapper exposes the slot __doc__. Parenthesized yield assignment uses the invalid-target messages. throw() restores without overwriting __context__, then chains only from the generator's own exc_info slot. Assisted-by: Grok:grok-4.6 * Remove passing expectedFailure markers method-wrapper docs, deallocator messages, and help pager output now match, so drop the leftover markers. Assisted-by: Grok:grok-4.6 * Honor wrapper signatures and assignment diagnostics Slot docs include text signatures. method-wrapper compares and hashes by identity. Keyword genexp and if/while assignment use the invalid-target messages. Assisted-by: Grok:grok-4.6 * address review: aclose, unraisable, frame publish aclose ignore-yield does not close the generator. Format unraisable object is None. close() leaves traceback to the caller; finalize attaches the generator frame. Publish the iframe before the lock-free slot. Assisted-by: Grok:grok-4.6 * Honor member descriptors and wrapper annotations Assisted-by: Grok:grok-4.6 * address review: getitem, bool doc, if assign Assisted-by: Grok:grok-4.6 * Honor eval-mode assignment invalid syntax Assisted-by: Grok:grok-4.6 * Supply getset and member docs from rustpython-doc Assisted-by: Grok:grok-4.6 * Enable rustls SSL on WASI targets Compile the rustls _ssl module for WASI and stub socket waits plus the native certificate store. Assisted-by: Grok:grok-4.6 * Add browser TLS _ssl in rustpython-wasm Register rustls _ssl and a MemoryBIO-oriented _socket from rustpython-wasm. Use rustls-rustcrypto because ring cannot compile for wasm32-unknown-unknown. Assisted-by: Grok:grok-4.6 * Define SSL protocol and cert constants as enums Keep OpenSSL bitmask defines as integer constants. Assisted-by: Grok:grok-4.6 * Ignore rustls deps cargo-shear cannot see The rustpython-wasm _ssl sources are compiled through a symlink, so shear does not count their crate uses. Assisted-by: Grok:grok-4.6 * Replace wasm ssl symlink with a BIO module Drop the stdlib ssl.rs link and keep a rustpython-wasm _ssl that only covers MemoryBIO wrap_bio TLS. Assisted-by: Grok:grok-4.6 |
||
|
Jeong, YunWon
|
2fc2479f72 |
openssl: use host_env for CA paths and file checks (#8680)
Replace openssl-probe and local std::fs checks with host_env native_certs and fs. Vendored OpenSSL loads system CA paths from host_env; OpenSSL's own defaults stay for a system lib. Assisted-by: Grok:grok-4.6 |
||
|
Jeong, YunWon
|
979576e2e4 |
ssl: move TlsConnection and session store to host_env (#8678)
TlsConnection, RecordCursor, rustls error mapping, hostname/ALPN/version helpers, and the capturing client session store now live in host_env. stdlib _ssl keeps socket/MemoryBIO I/O and Python objects. Remove unused stdlib sha2 and leftover workspace TLS crates. Assisted-by: Grok:grok-4.6 |
||
|
Jeong, YunWon
|
bfd52eda61 |
ssl: move remaining rustls engine into host_env (#8675)
* ssl: move remaining rustls engine into host_env Cert loading, rustls verifiers, ClientConfig/ServerConfig construction, key logging, acceptor helpers, and X509 error mapping now live next to the trust store. stdlib _ssl keeps the Python object and socket adapters. Assisted-by: Claude * Remove unused workspace SSL dependencies Drop oid-registry, rustls-platform-verifier, and x509-cert from workspace.dependencies. Assisted-by: Grok 4.6 |
||
|
Jeong, YunWon
|
3a8a196996 |
ssl: share rustls engine in host_env and fire msg callbacks (#8665)
* ssl: fire msg callbacks and compute tls-unique Observe TLS records to invoke _msg_callback, and derive TLS 1.2 tls-unique from the KeyLog master secret plus rustls PRF. Restore CHANNEL_BINDING_TYPES and drop the remaining test_ssl RUSTPYTHON skips. Assisted-by: Claude * lzma: build the xz-core engine on wasm32 Expose `_lzma` on wasm32 and Android by compiling the xz-core engine on those targets instead of gating it out. Assisted-by: Claude * ssl: move the rustls engine into rustpython-common Extract the VM-independent rustls helpers (MemoryBio, certificate decode, cipher strings, OID table, chain builder, providers, and TLS record observation) so stdlib _ssl wraps a shared engine. Assisted-by: Claude * ssl: keep the rustls engine in host_env TLS helpers share the host trust store and default verify paths, so they belong next to native_certs rather than in rustpython-common. Assisted-by: Claude * stdlib: drop unused jiff after moving cert decode Assisted-by: Claude |
||
|
Jeong, YunWon
|
ad4532c0e0 |
ssl: retain WebPKI-selected certificate chains (#8653)
Port the connection-owned chain builder from Pyre. Preserve verification inputs with TLS configurations and sessions, report the WebPKI-selected path, and publish only its selected capath anchor. Match CPython chain reporting for CERT_NONE and resumed sessions. Assisted-by: Codex:GPT-6 |
||
|
Jeong, YunWonandIvan Mironov
|
de6053099a |
ssl: take the OID table and the OpenSSL cipher-string parser from #8007 (#8646)
* ssl: read the OID table out of OpenSSL's object database
`txt2obj` and `nid2obj` have to answer with OpenSSL's own numbers, and the
table behind them was 39 entries transcribed by hand, so every object
outside that set was reported unknown.
Read OpenSSL's own files instead: `obj_mac.num` carries each object's NID
and `objects.txt` carries its OID, short name and description, and the
identifier `objects.pl` derives per object -- the `!Cname` override, else
the description, else the short name, each qualified by the enclosing
`!module` with `-` spelled `_` -- joins the two. `OidEntry`, `OidTable`
and the `find_by_*` functions keep their shape, so `ssl.rs` only changes
where an object has no OID to report.
Three details the object database settles that a curated table did not
have to:
- an object written with one name carries it as both, so
`nid2obj(48)` is `('emailAddress', 'emailAddress', ...)`;
- an object need not have an OID at all -- `DES-EDE3` and 139 other
algorithm names have a NID and nothing to encode -- so the fourth
field of the tuple is None for those, as it is on OpenSSL;
- one arc does not encode, since DER packs the first two into a byte,
so the bare roots `iso` and `itu-t` are OID-less too, while two arcs
such as X500's `2.5` are a real object.
Checked against `_ssl.nid2obj` on CPython built with OpenSSL, over every
NID it answers: 1387 of its 1390 present, with no disagreement on any of
them, up from 39. The three absent are OpenSSL's unnamed placeholder
slots, which it reports as ('NULL', 'NULL', None).
The data files come from openssl/openssl `crypto/objects/`, via the
rustls integration rewrite in #8007.
Co-authored-by: Ivan Mironov <mironov.ivan@gmail.com>
Assisted-by: Claude
* ssl: parse OpenSSL cipher strings in set_ciphers
Replaces the pattern matcher `set_ciphers` used with the grammar
`man openssl-ciphers` documents, in a new `ssl/cipher.rs` taken from
#8007: `!`, `-`, `+`, `@STRENGTH`, `@SECLEVEL=n`, `+`-joined
conjunctions, the DEFAULT/ALL/COMPLEMENTOF* groups, protocol versions,
SUITEB*, and the aXXX/kXXX/eXXX prefixes.
`set_ciphers` raises "No cipher can be selected." for a string that
names nothing the provider offers, which is the one failure
SSL_CTX_set_cipher_list reports, and keeps the TLS 1.3 suites selected
in provider order because that setter does not reach them. A SUITEB*
string pins its own key exchange groups, which `prepare_kx_groups`
now reads.
Against CPython's OpenSSL over 78 cipher strings, the selections agree
except under `@SECLEVEL=n`, which OpenSSL applies at handshake time and
this applies to the selection.
`CryptoExt` gains `default_cipher_suites` so that `DEFAULT` and `ALL`
are distinguishable.
Co-authored-by: Ivan Mironov <mironov.ivan@gmail.com>
Assisted-by: Claude
* ssl: finish cipher reporting and provider handling
Use the provider-ordered cipher metadata table as the single source for parsing, reporting, OpenSSL names and descriptions. This makes get_ciphers reflect the context selection and validates curves and Suite B against the active provider.
Keep set_ciphers from changing TLS 1.3 suites, preserve the provider's order, fix empty conjunctions, and keep FIPS initialization within the provider's filtered cipher and key-exchange defaults.
Co-authored-by: Ivan Mironov <mironov.ivan@gmail.com>
Assisted-by: Claude
Assisted-by: CodeRabbit
Assisted-by: OpenAI Codex: GPT-5
* ssl: preserve OpenSSL OID lookup rules
Keep object-name lookup case-sensitive and canonicalize numeric OID arcs before consulting the generated OpenSSL object table.
Assisted-by: OpenAI Codex: GPT-5
* ssl: preserve cleartext after MemoryBIO shutdown
Read at most one TLS record while waiting for close_notify in BIO mode, leaving any post-TLS cleartext queued for the BIO owner.
Co-authored-by: Ivan Mironov <mironov.ivan@gmail.com>
Assisted-by: OpenAI Codex: GPT-5
* ssl: reject empty cipher conjunction operands
Keep empty operands visible to the parser so malformed OpenSSL cipher strings such as AES+ and ALL++AES are rejected instead of normalized.
Assisted-by: CodeRabbit
Assisted-by: OpenAI Codex: GPT-5
* ssl: make Suite B cipher ids unambiguous
Spell out the u16 conversion so Windows-only iterator implementations cannot make collection type inference ambiguous.
Assisted-by: OpenAI Codex: GPT-5
* ssl: move native certificate loading behind host_env
Keep platform trust-store loading and filesystem predicates on the host boundary while exposing backend-neutral certificate bytes to the rustls integration.
Assisted-by: OpenAI Codex: GPT-5
* ssl: invalidate server config after cipher changes
Rebuild cached server configuration after set_ciphers updates the selected suites or Suite B key-exchange groups.
Assisted-by: CodeRabbit
Assisted-by: OpenAI Codex: GPT-5
* ssl: load default certificate sources independently
Honor SSL_CERT_FILE and SSL_CERT_DIR independently. Keep directory certificates as lazy capath candidates, and fall back to native roots only when no environment certificate file was loaded.
Co-authored-by: Ivan Mironov <mironov.ivan@gmail.com>
Assisted-by: OpenAI Codex: GPT-5
---------
Co-authored-by: Ivan Mironov <mironov.ivan@gmail.com>
|
||
|
Jeong, YunWon
|
6e6792ee5c |
common: share the hashlib engine, and publish blake2b/blake2s as types (#8643)
The digest, HMAC, PBKDF2 and scrypt state moves out of the `_hashlib` module into a feature-gated `rustpython_common::hashlib` that owns each state behind a `Mutex` and reports plain Rust results; `_hashlib` and `_blake2` keep the object glue. The engine writes its state through a raw pointer into a caller-owned buffer, so the buffer is boxed and only the pointer travels with the Python object, and it carries the alignment the engine asserts. The wrapper is built only after a successful init, so a rejected digest name leaves nothing to drop over storage the engine never wrote. blake2 moves from `blake2` to `blake2b_simd` / `blake2s_simd`, whose `Params` carry the whole parameter block. `_blake2.blake2b` and `blake2s` become immutable types with `SALT_SIZE`, `PERSON_SIZE`, `MAX_KEY_SIZE` and `MAX_DIGEST_SIZE` as class attributes, and their instances are instances of those types. They accept `key`, `salt`, `person`, `fanout`, `depth`, `leaf_size`, `node_offset`, `node_depth`, `inner_size` and `last_node`. `_hashlib.scrypt` is published on the same engine. Six `test_hashlib` expectedFailure markers drop with it. `blake2`, `digest`, `dyn-clone`, `hmac` and `pbkdf2` lose their last consumer. Assisted-by: Grok Assisted-by: Claude |
||
|
Jeong, YunWon
|
aa59f1ee53 |
common: share the bzip2 stream engine (#8638)
Move the VM-independent bzip2 stream owner into a `bz2`-gated `rustpython-common` module beside the zlib and lzma engines, and keep `rustpython-stdlib` as the Python object and exception adapter. The adapter no longer names the `bzip2` crate; the `bz2` feature carries it. `_bz2` was the last user of the generic `DecompressState` / `DecompressStatus` / `Decompressor` machinery in `stdlib/src/compression.rs`, so that goes with it; `DecompressArgs` stays for zlib and lzma. The decompressor feeds the stream through the existing `Chunker`, so buffered and freshly supplied input are no longer joined into a new allocation first. `test_decompress_after_data_error` passes on the ported engine, so its `expectedFailure` marker goes. Assisted-by: Grok |
||
|
Jeong, YunWon
|
c716a7d070 |
common: share the zlib stream engine (#8631)
* common: share the zlib stream engine Move the VM-independent zlib-rs stream owner and state machine into a feature-gated rustpython-common module. Make rustpython-stdlib adapt that engine instead of carrying its own flate2 implementation, including native stream copy/deepcopy, Z_BLOCK flushes, and wbits=0 decompression. Assisted-by: Codex:GPT-5 * common: share the lzma stream engine Move the VM-independent liblzma stream owner, filter properties, and compression state into a target-gated rustpython-common module. Keep rustpython-stdlib as the Python object and exception adapter, and remove its direct xz dependencies and lzma-only generic compressor machinery. Assisted-by: Codex:GPT-5 * common: address compression engine reviews |
||
|
Jeong, YunWon
|
322f294a21 |
common: add VM-independent codec and text engines (#8622)
* common: add PyPy CJK codec engines
Assisted-by: Codex:GPT-5
* common: mark CJK mapping tables generated
Assisted-by: Codex:GPT-5
* common: add reusable byte and text engines
Assisted-by: Codex:GPT-5
* common: record opt-in engine dependencies
Assisted-by: Codex:GPT-5
* stdlib: add `_multibytecodec` and the six `_codecs_*` modules
Wire the CJK codec engines in `rustpython-common` up to the VM. `_multibytecodec`
holds `MultibyteCodec`, `MultibyteIncrementalEncoder`, `MultibyteIncrementalDecoder`,
`MultibyteStreamReader` and `MultibyteStreamWriter`, the encode/decode drivers and
the error-handler protocol; each `_codecs_*` module exposes `getcodec` for the
encodings of its region.
`rustpython-stdlib` now enables the `cjk-codecs` feature of `rustpython-common`.
Add `cjk::decode_reset`, which the decoder `reset()` methods need. It differs from
building a fresh decoder state: ISO 2022 keeps its G1..G3 designations and the
escape-throughout flag, and resets only G0 and the shifted flag.
Drop the markers from the CJK codec tests and from the `test_codecs`, `test_io`,
`test_email`, `test_zipfile` and `test_traceback` cases that needed a CJK codec.
`test_codecs.BasicUnicodeTest.{test_basics,test_decoder_state}` stay expected
failures for an unrelated reason: `codecs.charmap_encode` rejects a `None` mapping.
Assisted-by: Claude Code:claude-opus-5
* stdlib: use the byte and text engines from `rustpython-common`
`binascii`, `_json` and `_socket`'s address converters now call the engines in
`rustpython-common` instead of carrying their own copies, and `rustpython-stdlib`
enables the `binascii`, `inet` and `json` features for them. That leaves
`base64` and `crc32fast` with no caller in `rustpython-stdlib`, and
`crates/stdlib/src/json/machinery.rs`, the last of the json_in_type-derived code,
goes away with its last caller.
The address converters change behaviour. `inet_aton` reads the lenient forms
(`127.1`, `16909060`, `0x7f.1`, `0177.0.0.1`), `inet_pton` rejects an octet with a
redundant leading zero and a group of five hex digits, `inet_ntop` writes the
IPv4-compatible form as `::c000:201`, and an address family with no converter
raises `EAFNOSUPPORT` instead of a bare `OSError`.
Give the JSON engine the closing-quote scan and the allocation-free `\uXXXX`
escape it needs to carry `_json`'s workload. Against the code it replaces,
`json.dumps` runs 2.3x faster with `ensure_ascii=True` and 1.7x without it, and
`json.loads` 1.2x.
`binascii.a2b_base64` now reports the messages `binascii.c` does: no
`error decoding base64: ` prefix, and the data-character count in parentheses.
Assisted-by: Claude Code:claude-opus-5
* test: enable the Windows cp932 codec case
The shared CJK engine now satisfies the Windows code-page adapter, so keeping this test marked as an expected failure turns its success into a CI failure.
Assisted-by: Codex:GPT-5
|
||
|
Joshua Megnauth
|
3c70a1ba58 |
Replace deprecated chrono with jiff (#8436)
See: chronotope/chrono#1768 |
||
|
Jeong, YunWon
|
210ccc889c |
Thread-safe type operations: type lock, QSBR type-cache reclamation, GC stop-the-world, and interpreter optimizations (#7416)
* type lock * Drop old PyObjectRef outside type lock to prevent deadlock Dropping values inside with_type_lock can trigger weakref callbacks, which may access attributes (LOAD_ATTR specialization) and re-acquire the non-reentrant type mutex, causing deadlock. Return old values from lock closures so they drop after lock release. * Align type lock behavior with CPython * Add PUBLISHED flag bit to RefCount state word Assisted-by: Claude * Add QSBR module for deferred memory reclamation Assisted-by: Claude * Defer memory free of cache-published objects via QSBR Assisted-by: Claude * Wire QSBR checkpoints into thread lifecycle and eval breaker Assisted-by: Claude * Publish type-cache values to QSBR and bypass cache without a VM Mark cached type-method values as QSBR-published before storing their pointer in TYPE_CACHE, so racing readers never try-incref freed memory. Debug-assert that the current thread is ATTACHED whenever a lock-free cache read happens. Delete find_name_in_mro_without_vm and its unsound direct SeqLock read outside any VM/thread registration; find_name_in_mro now falls back to the locked, uncached MRO walk when no VM is current. This also removes assign_version_tag()'s last caller, so its unlocked fallback is deleted along with it (version_for_specialization keeps the locked path). VirtualMachine::initialize() runs Python bytecode (e.g. importing codecs/encodings) before any enter_vm scope exists, which left the bootstrap thread not ATTACHED for cache reads during startup. Add thread::VmBootstrapGuard, an RAII counterpart to enter_vm usable across statements that need &mut VirtualMachine, and wrap initialize() with it. Assisted-by: Claude * Use try-incref reads and QSBR-backed swaps in specialization cache Assisted-by: Claude * Reset QSBR thread registry after fork in the child Add Qsbr::reset_after_fork, which clears the registered thread slots before draining the retire queue. Call it in py_os_after_fork_child, right after type_cache_after_fork() and before _thread::after_fork_child() re-registers the surviving thread. Dead parent threads' Arc<QsbrSlot> handles live on in the child's copied memory with no destructor ever running, so without this the registry keeps them 'online' forever and future grace periods never complete. Assisted-by: Claude * Add threaded stress test for type cache mutation races Assisted-by: Claude * Restrict qsbr internals to pub(crate) and fix clippy lints Downgrade pub items in the private qsbr::threading module (QsbrSlot, Qsbr, QSBR, and its methods) to pub(crate), since they were never reachable outside the crate. Downgrade ThreadSlot::qsbr accordingly to avoid a private-interfaces warning. Import Arc/Weak from alloc instead of std to match clippy::std_instead_of_alloc. Rewrite the process() comment: goals are not strictly ordered by push order since advance() and the queue push aren't atomic together, so concurrent free_delayed calls can interleave; the drained prefix stays sound because each item individually passed poll, with reordering only affecting reclamation latency. Assisted-by: Claude * Gate per-instruction QSBR check behind a global pending flag eval_breaker_tripped() and check_signals() called thread::qsbr_break_requested() on every bytecode instruction, which does a thread-local lookup, RefCell borrow, and atomic load even when no QSBR retirement is pending. This caused a measured 10-18% slowdown on tight interpreter loops. Add Qsbr::pending, a global AtomicBool set under the queue lock while free_delayed() holds a retired allocation, and cleared under the same lock by process()/drain_all() once the queue empties. The hot path now checks Qsbr::break_pending() (a single relaxed static load) before touching thread-local state, and only pays the TLS cost while a retirement is actually in flight. Assisted-by: Claude * Merge signal and QSBR eval-breaker flags into one atomic word Replace the separate ANY_TRIGGERED AtomicBool and QSBR pending AtomicBool checks in the per-instruction eval-breaker path with a single AtomicU8 (EVAL_BREAKER) holding a bit per source. The global QSBR instance mirrors its pending state into the QSBR bit under the same queue-lock critical sections that already toggle pending, while local QSBR instances used by unit tests are left untouched. eval_breaker_tripped() and check_signals() now read/act on the merged word (eval_breaker_pending(), qsbr_bit_set()) instead of issuing two separate atomic loads per instruction. set_triggered() switches from store to fetch_or so a signal handler never clobbers the QSBR bit. Removed the now-unreferenced is_triggered() helper; break_pending() is kept for unit tests only and marked allow(dead_code) outside test builds. Assisted-by: Claude * Skip freelist reuse for published objects in default_dealloc Freelist-eligible payloads (tuple, int, list, dict, float, ...) were routed to T::freelist_push before ever reaching PyInner::dealloc, so a published object (e.g. a tuple cached as a type attribute) bypassed the is_published -> free_delayed QSBR hook entirely. That let PyRef::new_ref reuse the slot and rewrite the refcount word with a non-atomic core::ptr::write racing a reader's atomic try-incref, and let FreeList::drop call alloc::dealloc directly while a reader could still be mid-read. Gate the freelist branch in default_dealloc on RefCount::is_published(), checked before any teardown. A published object now always falls through to PyInner::dealloc, whose existing hook defers the memory free via QSBR. Document the non-unix (e.g. Windows) reclamation-latency limitation in the QSBR design doc's Lifecycle integration section (previously only noted in a code comment in vm/thread.rs). Extend the type-cache stress test so the mutator also churns a freelist-eligible published tuple class attribute (C.shape), read by readers each inner loop and deleted on the same cadence as C.m, to exercise the fixed bypass. Assisted-by: Claude * Use itoa for PyInt::to_str_radix_10 i64 fast path Format small integers with itoa::Buffer instead of i64::to_string() in PyInt::to_str_radix_10(), which backs both str(int)/repr(int) for exact int and PyObject::str()'s exact-int fast path. Adds the itoa crate as a workspace dependency. Assisted-by: Claude * Make object attribute dunders wrapper descriptors Remove the __getattribute__, __setattr__ and __delattr__ #[pymethod]s from PyBaseObject so add_operators installs PyWrapper slot wrappers for them from the #[pyslot] functions instead of method descriptors. lookup_slot_in_mro now classifies these entries as NativeSlot, so heap types without Python-level overrides keep the native getattro/setattro slot functions, which lets LOAD_ATTR/STORE_ATTR specialization apply to instances of Python-defined classes. Since __setattr__ and __delattr__ share the setattro slot, resolve both names together in update_one_slot: overriding only one of them no longer lets the other name's native resolution overwrite the dispatching wrapper. Assisted-by: Claude * Resolve TpSetattro pair on attribute deletion, not just addition update_one_slot's TpSetattro branch only ran the combined __setattr__/__delattr__ resolution (added in the previous commit) when ADD was true. On deletion (e.g. del C.__setattr__), it fell back to inherit_from_mro, which copies the base class's slot and discards the class's own remaining Python-level override of the other name. lookup_slot_in_mro reads the current attribute dicts, so the same resolution is correct for both addition and deletion; the ADD-only guard is removed and the logic now always runs. Assisted-by: Claude * Generate Opcode::cache_entries/deopt as table lookups generate_rs_opcode_metadata.py now emits a 256-entry const array for each function instead of a chained match over opcode names. Opcode::deopt() indexes an [Option<Opcode>; 256] table built from the existing specialization-to-base mapping, and Opcode::cache_entries() indexes a [u8; 256] table that bakes in the same deopt/to_base composition deoptimize() previously performed before its own match. Regenerated opcode_metadata.rs from the updated script; PseudoOpcode's bodies are unchanged since it has no cache/deopt entries to table. Added an exhaustive test in instruction.rs that checks every valid u8 opcode against a frozen copy of the old chained-match bodies. Assisted-by: Claude * Make Opcode/Instruction numeric conversions O(1) Opcode::as_numeric, Opcode::as_instruction and Instruction::as_opcode were each a per-variant match over ~230 arms in the define_opcodes! macro. Opcode now carries the same explicit $op_id discriminants and #[repr($typ)] as Instruction, which makes the conversions sound as: - Opcode::as_numeric: a plain `self as $typ` identity cast. - Opcode::as_instruction / Instruction::as_opcode: a mem::transmute, relying on both enums sharing one-$typ-wide layout (Instruction's payload fields are all the zero-sized Arg<T> marker, checked by a new per-instantiation size_of assertion). try_from_numeric is left as a match: $op_id values have gaps (specialized/instrumented opcode ranges), so a range check can't replace it. Adds #[inline] to the rewritten conversions plus deoptimize, and to the small generated wrapper/table-lookup functions from the previous opcode-metadata change (as_u8/as_u16, cache_entries, deopt, to_base) via generate_rs_opcode_metadata.py, then regenerates opcode_metadata.rs. Extends the instruction.rs equivalence-test pattern with two exhaustive tests (u8 and u16 instantiations) checking the new identity-cast/transmute conversions against the untouched try_from_numeric/TryFrom paths. Assisted-by: Claude * Pop CallIsinstance arguments directly instead of collecting into Vecs The specialized handler built two heap-allocated Vecs per call (pop_multiple().collect() plus a with_capacity(2) buffer) before invoking is_instance. Pop the class and instance straight off the stack for the effective two-argument case, removing the per-call allocations. Assisted-by: Claude * Inherit tp_new instead of always installing new_wrapper update_one_slot's TpNew branch stored new_wrapper for every ADD, so all heap types had slots.new == new_wrapper and the CallAllocAndEnterInit specialization never fired. Now __new__ is resolved through the MRO dicts: an own or Python-level definition installs new_wrapper, while a builtin __new__ entry (or none) inherits slots.new from the solid base. The now-reachable CallAllocAndEnterInit path gains the missing guards: the specializer requires co_argcount == oparg + 1 (via can_specialize_call) and rejects generator-like __init__, and the handler re-checks the argcount against the class-level init cache. Without the argcount check, __init__ defaults broke re._parser (UnboundLocalError in SubPattern.__init__). Remove the now-unused is_simple_for_call_specialization. Assisted-by: Claude * Skip locals dict allocation for the init-cleanup shim frame Mark the shim code object NEWLOCALS and create the shim frame with no locals mapping so FrameLocals::lazy() is used. The shim only executes ExitInitCheck/ReturnValue and never touches locals, so this removes one dict allocation per specialized instantiation. Assisted-by: Claude * Guard CALL_ALLOC_AND_ENTER_INIT with cached function version The init specialization cache stored only the type version, so swapping __init__.__code__ (which zeroes func_version) did not deopt the warmed call site and stale code assumptions (argcount shape, no varargs/kwonly, not a generator) were applied to the new code object. Store get_version_for_current_state() in the specialization cache when caching __init__ and re-check func_version() in the handler before using the cached function, deopting to the generic call path on mismatch, the same way the getitem specialization cache does. Assisted-by: Claude * Apply rustfmt and ruff-format fixes to files from earlier commits Assisted-by: Claude * Add frame object freelist Frame.iframe becomes FrameUnsafeCell<Option<InterpreterFrame>> so a dead frame can be left as an empty husk. Traverse::clear extracts all owned child references (including localsplus values via LocalsPlus::clear_into) and empties the cell. Dead frame allocations are cached in a thread-local FreeList<Frame> (up to 200 entries) and reused by PyRef::new_ref. clear_generator returns early when the frame was already cleared by cycle collection; generator drops can run after tp_clear of their frame. gc: skip tp_clear for objects saved to gc.garbage by DEBUG_SAVEALL, since they remain reachable from Python. Assisted-by: Claude * Drop frame children directly in tp_clear instead of extracting Pushing the frame's ~10 child references into the clear buffer grew a Vec on every frame deallocation. Take the interpreter frame out of the cell and drop it in place; LocalsPlus::clear_into is removed. Assisted-by: Claude * Untrack objects before tp_clear and guard cross-thread f_locals The cycle collector called tp_clear on unreachable objects while they were still linked into the GC generation lists, so another thread could obtain a strong reference to an already-cleared object (e.g. a frame husk with iframe == None) via gc.get_objects() and access the cleared payload. Untrack the dead set before the clear phase, mirroring the untrack-then-clear ordering of the refcount dealloc path. Objects that gained an external reference before untracking are found by comparing each object's strong count against the references coming from within the dead set; such late-resurrected objects and everything reachable from them are re-tracked and skipped, letting a later collection retry once the external references are released. Also reject f_locals access for a frame currently executing on another thread. Reading fastlocals of such a frame races with the executing thread overwriting the slots (torn reads of dropped values). Access from the executing thread itself (locals(), trace callbacks) is still allowed via the current-frame chain check. Assisted-by: Claude * Skip localsplus heap copy for uniquely referenced dying frames release_datastack_frame now checks the frame's strong count after untracking it from the GC generation lists. When the caller holds the only reference, localsplus values are dropped in place and the data stack storage is released without the heap copy; escaped frames (traceback, sys._getframe, trace callbacks) keep the copy path. The three inline materialize+pop call sites in function.rs are routed through release_datastack_frame. Assisted-by: Claude * Stage exact-args call arguments in stack slot buffers CallPyExactArgs, CallBoundMethodExactArgs and CallAllocAndEnterInit no longer collect popped arguments into a per-call Vec (plus a second Vec for self-prepending). Arguments are popped into a fixed-size Option<PyObjectRef> slot buffer (CallArgBuffer, 8 inline slots, heap fallback for larger arities) and moved into fastlocals via the new invoke_exact_args_slots / a take() iterator. - prepare_exact_args_frame now accepts an ExactSizeIterator of args - specialization_run_init_cleanup_shim takes the slot buffer and fills slot 0 with new_obj, dropping one redundant clone - LoadAttrGetattributeOverridden, LoadAttrProperty and BinaryOpSubscrGetitem pass inline slot arrays instead of vec![] Assisted-by: Claude * Reduce allocations in generic call paths - FuncArgs::prepend_arg: use reserve instead of reserve_exact so exact-capacity vectors do not realloc on every prepend. - IntoFuncArgs::into_method_args: build the final args vec once with capacity len + 1 instead of prepending into a full vector. - PyType::call: skip cloning FuncArgs when no init slot can run after slot_new (no init slot, not `type`, and slot_new is not new_wrapper). Assisted-by: Claude * Skip redundant exc_info restore on frame exit with_frame_impl saves the current exc_info slot value and restores it when the frame exits. When the slot still holds the same value, the restore rewrites the slot and recomputes the thread-exception mirror for no effect. Add restore_exception, which compares the slot against the saved value by pointer identity and skips the store when they match. The saved value is a strong reference held for the whole frame scope, so the pointed-to object cannot be freed and its address reused while the frame runs, making the pointer comparison free of ABA. Assisted-by: Claude * Apply rustfmt to gc_state.rs Assisted-by: Claude * Avoid empty FuncArgs clone in PyType::call Cloning args for the init call clones the kwargs map even when args is empty, which shows up in no-argument instantiation profiles. Prepare the init-call args before slot_new: a default FuncArgs when args is empty (indistinguishable from a clone of empty args), the existing clone otherwise. The empty check also keeps the init-slot load of the clone-elision path off the no-argument call path. Assisted-by: Claude * Push freelist husks only after tp_clear and child drops default_dealloc pushed the object onto the thread-local freelist before running clear_fn. Payloads that drop children inside clear (Frame) can run __del__ there, and a reentrant allocation could pop the husk and overwrite its payload while clear_fn still held a &mut borrow of it. Tuples had the same window through the extracted-edges drop, which ran after the push. Reorder default_dealloc to clear, drop the extracted children, and only then attempt the freelist push, so the husk becomes reusable only when no borrows into the payload remain. The tuple freelist bucketed husks by element count, which required reading the payload during push; after this reordering the elements are already cleared. PyInner<PyTuple> is a fixed-size allocation (the elements box is dropped and replaced on reuse), so replace the per-size buckets with the shared single-list FreeList. Remove the now-unused pyinner_layout helper. Assisted-by: Claude * Resolve __set__ and __delete__ together for the descr_set slot The descr_set slot serves both __set__ and __delete__, but the TpDescrSet accessor resolved only the single modified name and fell back to MRO inheritance on delete. Deleting one of the pair could disable the surviving operation: after `del D.__set__`, an instance delete stopped calling the remaining __delete__. Resolve both names together like the TpSetattro accessor does: any Python-level definition selects the dispatching wrapper, matching native functions are stored directly, a single native function with the other name absent is stored directly, and the slot is inherited from the MRO only when neither name resolves. Assisted-by: Claude * Update type base on __bases__ assignment set_bases replaced bases and mro but never updated the stored base, so __base__ kept reporting the old value after reassignment and slot resolution (tp_new inheritance, set_new/set_alloc during init_slots) read the stale solid base. Change the base field to PyAtomicRef<Option<PyType>> so it can be swapped under the type lock while remaining lock-free for readers, and recompute it in set_bases with the same best_base validation type creation uses (BASETYPE flag, instance layout conflict among bases). The swapped-out base is parked in the frame's temporary refs; other references released inside the critical section are dropped after the lock is released. Also in set_bases: - remove this class from the old bases' subclass lists (pruning dead entries), so repeated assignment no longer accumulates duplicates - roll back bases, base, and all updated mros when the recursive mro update fails, instead of leaving the type half-reparented - fix the misformatted empty-tuple error message Remove the expectedFailure marker from test_unsubclassable_types, which now passes. Assisted-by: Claude * Fix rollback order, dead weakrefs, and lock-held drops in set_bases - Restore recorded mros in reverse on rollback so a class visited multiple times through diamond inheritance ends with its original mro instead of an intermediate one. - Skip dead weakrefs in the subclass list during the recursive mro update instead of panicking on upgrade. - Retire the replaced mros on the success path instead of dropping them while the type lock is held. Assisted-by: Claude * Reify number sub-slot wrapper once in update_one_slot The update_sub_slot! macro expanded the Python-method wrapper closure at both the own and inherited store sites. Each expansion is a distinct fn item, so a base and a non-overriding subclass stored wrappers with distinct addresses in unmerged debug builds. binary_op1 compares slot fn addresses to detect whether a subclass overrides the operator, so the inherited slot was misread as an override and C() // E() dispatched to __rfloordiv__ instead of __floordiv__. Bind the wrapper store in a single closure and call it from both branches so the fn item is reified once and the slot value stays identical across a base and its subclasses. Document the conservative-on-mismatch nature of the new_wrapper address guard in call_wrapped. Assisted-by: Claude * Restrict type_cache_after_fork visibility and guard type-cache reads type_cache_after_fork is only called from within the crate, so mark it pub(crate) to resolve the unreachable-pub warning. Add debug_assert_current_thread_attached at the has_name_in_mro lock-free type-cache read site, matching lookup_ref_and_version_interned, and gate the function on debug_assertions so it is not compiled unused in release builds where the call sites are elided. Assisted-by: Claude * Guard BinaryOpSubscrGetitem with the cached type version The specialized BINARY_OP_SUBSCR_GETITEM handler used the cached __getitem__ after checking only the function version, unlike the sibling specialized handlers which revalidate the type version tag first. Store the type version in the inline cache at specialization time and revalidate owner.class().tp_version_tag against it before using the cached function, deopting to the generic subscript path on mismatch. Assisted-by: Claude * Call __abstractmethods__ __len__ once in object.__new__ The abstract-method check invoked the user-visible __len__ twice: once via length_opt for the count and again while materializing the method names. Derive the count from the materialized list instead, so __len__ runs once. Assisted-by: Claude * Use i64 fast paths for specialized int add/sub/mul Rewrite execute_binary_op_int to box results through new_int via i64 checked arithmetic instead of raw BigInt ops with new_bigint. Add an int_mul helper and a shared int_fast_op that computes the i64 result and falls back to the BigInt operation on to_i64 or checked-op failure. Wire int_mul into both the specialized BinaryOpMultiplyInt handler and the generic execute_bin_op Multiply/InplaceMultiply path, gated on exact int operands so subclasses keep dispatching through _mul/_imul. Assisted-by: Claude * Use i64 fast paths for exact int floordiv and remainder Add floordiv_i64/mod_i64 computing i64 floor-division and divisor-signed remainder, guarding zero divisor and i64::MIN overflow by returning None. Wire them through int_floordiv/int_mod (shared int_div_fast_op boxes via new_int) into the generic execute_bin_op FloorDivide/Remainder and their Inplace variants, gated on exact int operands so subclasses and zero divisors fall through to the existing _floordiv/_mod/_ifloordiv/_imod slow path. Assisted-by: Claude * Skip trashcan and untrack for non-GC-tracked objects in dealloc default_dealloc read is_gc_tracked() only to decide untracking, and entered the trashcan recursion guard unconditionally. Non-GC objects (int, float, str, ...) own no child references that recurse during deallocation, so they need neither the trashcan nor untracking. Read is_gc_tracked() once and gate both trashcan begin/end and untrack on it, removing three thread-local accesses per non-GC object deallocation. Assisted-by: Claude * Merge trashcan depth and defer queue into one thread-local struct trashcan::end accessed two separate thread-locals (DEALLOC_DEPTH and DEALLOC_QUEUE) at the outermost deallocation. Combine both into a single `Trashcan` thread-local holding Cell-based depth and queue fields, so begin and end each reach their state through one thread-local access. The queue is set back before each deferred dealloc call so reentrant begin/end during draining never holds an outstanding borrow. Assisted-by: Claude * Check object layout compatibility on __bases__ and __class__ assignment Add a shared compatible_for_assignment helper that walks each type to the base that fixes its instance layout and rejects the assignment when the old and new layouts differ (basicsize, itemsize, member count, __dict__, __weakref__, and __slots__ names). Wire it into set_bases, which previously performed no layout check, and replace the inline check in __class__ assignment, which compared the two types directly and over-rejected a subclass that adds no layout. Enable test_descr.test_builtin_bases, which the new set_bases check passes. Assisted-by: Claude * Stop the world around GC pointer-reading phases The cycle collector reads each tracked object's interpreter state during reference subtraction, the reachability walk and the strong-reference snapshot, including the localsplus of frames other threads are executing. Those slots are written without synchronization, so the reads are a data race under threading. collect_inner now stops the world before taking the generation read locks and restarts it after the strong-reference snapshot, before the finalizer/weakref/tp_clear phases. A debug assertion checks that no frame on any thread's call stack was classified unreachable. Automatic collections from maybe_collect are deferred to the next bytecode safepoint via a new eval-breaker GC bit instead of running synchronously: a synchronous collection can hold an internal lock (e.g. the lazy frame locals cell) that another thread is blocked on with no way to reach a safepoint, deadlocking the stop. Hardens the stop-the-world machinery for these frequent, concurrent GC stops: attach_thread now honors a pending stop after re-attaching so a thread doing rapid allow_threads calls cannot run past the requester; stop_the_world completion is level-triggered on all-threads-suspended rather than an edge-triggered countdown; and the thread-start started/ready handshakes detach while waiting so the waiter is parkable. Updates the Frame and PyRwLock/PyMutex traversal SAFETY comments to state the actual invariant. Assisted-by: Claude * Add threaded GC vs executing-frame stress snippet Workers churn frame state (recursion, generators, frame cycles) while a collector loops gc.collect() and an introspector walks live frame objects. Exercises the stop-the-world barrier around GC traversal. Assisted-by: Claude * Serialize fork and GC stop-the-world requesters fork() (posix before/after-fork) and the cycle collector both drive the single StopTheWorldState. With no mutual exclusion their requester word and suspension countdown could interleave and be clobbered, so the completion check never converged and a requester waited on itself forever (reproducible parent-side hang when forking with GC enabled). Add one exclusion held for the whole stop->start span of either requester: stop_the_world acquires it before any stop bookkeeping and start_the_world/reset_after_fork release it. The acquire is park-friendly (poll try-lock and honor a pending suspend between tries) so a requester blocked behind an active stop can still be force-parked instead of deadlocking the active requester. The acquirer holds no other lock while spinning. Also correct the PyRwLock traverse SAFETY note: a failed try_read may mean a force-parked thread holds the write lock; skipping is safe because under-traversal only over-approximates liveness. Document the residual gc.collect()-under-a-non-generation-lock exposure at the collect_inner barrier. Assisted-by: Claude * Add fork under concurrent GC stop-the-world snippet Worker threads allocate cyclic garbage with GC enabled while the main thread forks repeatedly and each child collects. Exercises the fork/GC stop-the-world exclusion; the allocation rate is throttled so a collection stays cheap in unoptimized builds. Assisted-by: Claude * Detach while acquiring the import lock The global import lock is held across bytecode by the importlib bootstrap, so its holder can be parked at a safepoint mid-hold. Acquiring it while attached let another thread block attached on the lock, so a stop-the-world requester could wait forever for that attached thread to suspend while the holder stayed parked. Wrap IMP_LOCK acquisition in allow_threads in both _imp.acquire_lock and the pre-fork acquire_imp_lock_for_fork so the wait honors stop-the-world requests. Correct the acquire_exclusion comment: a spinning requester may hold IMP_LOCK (fork) or the collecting mutex (GC); safety relies on those never being acquired attached-blocking by another thread. Assisted-by: Claude * Add concurrent-import vs GC deadlock snippet Two threads re-import modules (contending the import lock) while a third storms the cycle collector and a fourth allocates cyclic garbage. Regressed as a hang before the import lock acquisition was made park-friendly. Assisted-by: Claude * Create call frames untracked and track generators explicitly Add PyPayload::NEW_REF_UNTRACKED (default false, true for Frame) so PyRef::new_ref skips auto-tracking ordinary call frames in the GC. Generator/coroutine/async-generator frames are tracked explicitly in invoke_with_locals before their generator back-reference is installed. run_frame debug-asserts that a datastack frame is untracked on entry. Assisted-by: Claude * Track escaped call frames lazily at datastack release Rewrite release_datastack_frame around the invariant that a datastack frame is never GC-tracked while it runs: strong_count() == 1 means the frame never escaped, so drop localsplus in place without touching the GC; strong_count() > 1 means it escaped, so materialize localsplus onto the heap and then track the frame. This removes the untrack and the untrack-recheck dance from the common non-escaping path. Assisted-by: Claude * Guard the tracked-frame localsplus invariant Document at Frame::traverse that references to a frame are always recorded as graph edges, so the collector reads a frame's localsplus only when the frame is itself a tracked candidate. Add debug assertions at both frame track sites (escaped datastack frames and generator frames) that a frame is heap-backed before it is tracked, so a collector never reads data-stack-resident, still-mutating storage. Assisted-by: Claude * Reword frame/dealloc invariant comments and drop needless borrow - frame.rs: replace the task codename in the tracked-frame localsplus invariant comment with a description of the collector-vs-executing-frame behavior. - object.rs: remove needless borrow on current_cls in the __class__ assignment compatibility check. - core.rs: rewrite the default_dealloc trashcan/untrack-skip comment to state the actual invariant, which now covers untracked non-escaped frames releasing at interpreter depth with bounded recursion. Assisted-by: Claude * Move type SetAttr slot rewrite inside the type lock and tighten cache reads Run update_slot inside the same with_type_lock transaction as modified_inner and the attributes dict mutation, so the version invalidation, dict change, and slot-table rewrite are published together. Load init_version and getitem_version with Acquire in the specialization cache readers to pair with the Release stores in the writers. Assisted-by: Claude * Capture the type version before reading mutable slots in specializers specialize_load_attr, specialize_store_attr, specialize_to_bool, and the CallAllocAndEnterInit path of specialize_call read the version tag with version_for_specialization before inspecting getattro, setattro, the bool/len slots, and tp_new/tp_alloc, so a concurrent install of the corresponding dunder invalidates the version the specialization is cached against. In the BinaryOpSubscr __getitem__ path, check the HEAPTYPE and eval-frame gates before lookup_ref_and_version_interned, which takes the global type lock and may allocate a version tag. Assisted-by: Claude * Use _get_method_dict for the test_type check in the test runner Replace the direct __func__.__dict__ access with the _get_method_dict helper so plain functions without __func__ do not raise AttributeError. Assisted-by: Claude * Fix lint hooks: cspell dictionary entries, spelling fix, formatting Add qsbr to the rustpython dictionary and reborrows/reparenting to the top-level word list. Rename oldto/newto locals to old_to/new_to. Fix Stabilise -> Stabilize typo in a frame.rs comment. Apply cargo fmt to object.rs, type.rs, and frame.rs, and ruff format/check fixes to two extra_tests snippet files. Assisted-by: Claude * Rebuild all slots for type and descendants on __bases__ reassignment Add PyType::update_all_slots, which invalidates version tags and iterates the full SLOT_DEFS name table calling update_slot::<true> for each distinct name. Unlike init_slots, which is additive and driven only by dunder names present in the current MRO, this resets a slot whose method left the MRO instead of leaving a stale dispatcher pointer. Call it from set_bases in place of the previous modified_inner + init_slots pair, so reassigning __bases__ rebuilds slots for the type and every descendant. Add extra_tests/snippets/type_bases_slot_rebuild.py covering removed-method resets on zelf and deep descendants, the wrong-target switch, __getattr__, the added-method mirror, and a swap-away-and-back round trip. Assisted-by: Claude * Derive the thread-local frame stack from the frame chain Remove the per-VM `frames` Vec. current_frame, sys._getframe, sys._getframemodulename, frame.f_back, sys.monitoring re-instrumentation, gc.get_referrers, faulthandler stack dumps and the post-fork slot rebuild now walk the signal-safe CURRENT_FRAME/`previous` chain instead. Add `Py::from_payload_ptr` to recover a frame object from a chain pointer. The cross-thread `ThreadSlot::frames` registry (sys._current_frames, cross-thread f_back, GC stop-the-world assertion) is unchanged. Assisted-by: Claude * Inherit sub-slot fallback into the field being resolved update_one_slot's number/sequence/mapping fallback inherited via the accessor's default field. Left and right binary ops (add/right_add) share one accessor but occupy distinct fields, so resolving an absent right op or deleting it reset the left op's field, dropping a still-defined __add__ dispatcher. Inherit the exact field under resolution instead. Assisted-by: Claude * Gate type_cache_after_fork to its fork caller and use Self in downcast type_cache_after_fork is only called from the unix fork path, so gate it with all(feature = "host_env", unix) to match the caller and avoid a dead-code error on non-unix targets. Replace the explicit PyType with Self in the modified_inner downcast. Assisted-by: Claude * Unmark test_attr and test_method_call_error in test_monitoring Both TestLoadSuperAttr tests now pass; remove their expectedFailure markers. Assisted-by: Claude * Run tp_new specialization __init__ without a trampoline frame CallAllocAndEnterInit ran __init__ inside a synthetic init-cleanup shim frame whose code carried the __init__ name. Since the thread frame stack is derived from the frame chain, that shim frame was visible to sys._getframe, f_back walks, traceback construction and inspect.stack / inspect.trace. Its code object has empty co_positions, so a stack walk that reached it raised StopIteration inside inspect._get_code_position, cascading into unrelated failures (test_inspect trace/stack/frame, asyncio source traceback). Call __init__ directly via run_frame, enforce the __init__() should return None contract inline, and drop the shim: the init-cleanup code object, its builder, with_frame_untraced, monitoring_disabled_for_code, and the extra-frame datastack/recursion budget. Removing the second frame per construction also cuts the specialization's per-call cost. Assisted-by: Claude * Replace per-call thread-frame mutex with an atomic top-of-stack (unix) On unix threading builds, publish each thread's top Python frame in a single relaxed AtomicPtr store from set_current_frame instead of pushing onto a parking_lot::Mutex<Vec<FramePtr>> per call. Cross-thread readers (sys._current_frames, cross-thread f_back, faulthandler.dump_traceback, the GC unreachable debug-assert) run under stop-the-world and walk the published top frame down the Frame::previous chain; the owning thread is then parked at a safepoint, so the pointer and the frames it reaches are quiescent and alive. The faulthandler watchdog is a plain OS thread that cannot stop-the-world, so it walks the chain lock-free and best-effort. Non-unix threading builds have no stop-the-world and keep the existing mutex-guarded frame stack unchanged. Assisted-by: Claude * Skip exc_info save/restore for callees that never touch the slot with_frame saves and restores the shared exc_info slot around every Python call to contain frames that leave it unbalanced. Cache a has_exc_handling bit on PyCode at creation, set when the bytecode contains any opcode that calls vm.set_exception (PushExcInfo, PopExcept, CheckEgMatch, EndAsyncFor, InstrumentedEndAsyncFor). A callee whose code has none of these cannot mutate the slot, so the save and restore are skipped for it. Generators go through resume_gen_frame and are unaffected. Assisted-by: Claude * Return a write-through FrameLocalsProxy from frame.f_locals Optimized (function) frames now expose `f_locals` as a `FrameLocalsProxy` implementing PEP 667 semantics instead of a cached snapshot dict: - reads go live through the fast-local slots; each access mints a fresh proxy; keys that do not name a fast local are stored in a per-frame `f_extra_locals` side dict and folded into `locals()`. - writes to a fast-local key store into the slot (or its cell) in place; deleting a fast local raises ValueError; extra keys delete normally. - full mapping protocol: keys/values/items (lists), get/pop/setdefault, update (dict or FrameLocalsProxy only), __or__/__ior__/__ror__ (dict result), copy (plain dict), __reduce__ blocks pickling/copy, repr with recursion guard, mapping-pattern and Mapping ABC support. Class/module/exec frames keep returning their namespace mapping directly. Cross-thread access to a frame running on another thread still raises RuntimeError. A closed generator now keeps its frame locals when a durable frame reference escaped (f_locals proxy, sys._getframe, f_back), matching take_ownership; the escape is tracked with a per-frame flag. The snapshot-then-fold locals_to_fast/locals_dirty write-back is retired since proxy writes reach the slots directly. Assisted-by: Claude * Retain the caller frame so f_back resolves after it returns When a frame escapes its execution (referenced through a traceback, `sys._getframe`, `f_locals`, ...), capture a strong reference to its caller at release time. `f_back` consults it once the caller has left the live frame chain, so the Python-visible frame chain survives return. The retained reference is a GC-traversed edge and is cleared by `frame.clear()`, so ancestor chains stay collectable. Assisted-by: Claude * Fix debug-build native stack overflow on deep recursion Make check_c_stack_overflow one-sided (trip whenever the stack pointer is below the soft limit) so a single native frame larger than the margin cannot step past the danger band undetected. Raise the debug STACK_MARGIN_BYTES from 4096 to 16384 words so the margin exceeds a single debug interpreter frame, leaving headroom to raise RecursionError. Release margin unchanged. Clamp the soft-limit margin to half the stack so small explicit thread stacks do not get a soft limit above their stack top. Assisted-by: Claude * Allocate exception instance __dict__ lazily Exception construction went through into_ref_with_type, which eagerly allocated an empty instance dict for every HAS_DICT type. Add into_ref_with_type_lazy_dict, which builds the instance with an unallocated dict slot, and route the four exception construction sites (PyBaseException, PyOSError, OSErrorBuilder, PyBaseExceptionGroup) through it. The dict now materializes on first attribute write or __dict__ access via the existing get_or_insert path. add_note and PyImportError::slot_init now obtain the dict through object_get_dict so they materialize it instead of assuming it exists. A freshly constructed exception no longer reports an empty dict in gc.get_referents, matching the reference interpreter. Assisted-by: Claude * Allocate exception __dict__ lazily in vm.new_exception Route vm.new_exception() through into_ref_with_type_lazy_dict so internally raised exceptions (new_type_error, new_value_error, etc.) start without an instance dict, matching the slot_new path. The dict is materialized on the first attribute write or __dict__ access. Assisted-by: Claude * Validate FrameLocalsProxy.update() arguments Reject keyword arguments and require exactly one positional argument, raising TypeError with the "takes no keyword arguments" and "takes exactly one argument (N given)" messages. Assisted-by: Claude * Address review nits in frame and faulthandler - Drop stale vm.frames reference from the release_datastack_frame uniqueness argument. - Assert has_exc_handling when unwinding an Except-typed stack slot, documenting the invariant that guards the shared exc_info write. - Truncate the specialized __init__ return-type name to 200 chars, matching the unspecialized wrapper. - Reword two comments to describe behavior without prose references to CPython. Assisted-by: Claude * Use scopeguard for start_the_world in faulthandler dump_all_threads Wrap the unix stop-the-world registry walk in a scope with scopeguard::defer! so start_the_world runs on panic, matching the f_back and get_all_current_frames sites. Add scopeguard to the stdlib dependencies. Also correct the restore_exception doc comment to name with_frame after the rename. Assisted-by: Claude * Gitignore docs/superpowers Assisted-by: Claude * Rename type_bases_slot_rebuild.py to builtin_type_bases.py Match the builtin_* naming convention of extra_tests/snippets. Assisted-by: Claude * Apply rustfmt, ruff, and cspell lint fixes Reformat with rustfmt, fix import spacing in builtin_type_bases.py with ruff, and add "pointee" to the cspell word list. Assisted-by: Claude * Gate unix-only QSBR methods to their call sites `online`, `drain_all`, and `reset_after_fork` are called only from unix code (thread attach/detach, post-fork reset), and `offline` from unix code plus a unit test. Gate them with matching cfg so `-D dead_code` does not fire on non-unix targets. Assisted-by: Claude * Visit the function closure tuple as a GC edge PyFunction::traverse visited the cells inside the closure tuple instead of the tuple object itself, so the tuple's reference from the function was never subtracted during cycle collection. A closure tuple that reached back to its function (or, through a frame retained by f_back, to a Thread) was stranded as a false GC root and never collected, leaking the whole cycle. Visit the tuple itself, matching clear(). Assisted-by: Claude * Apply formatting hook fix to builtin_type_bases.py Assisted-by: Claude * Unmark asyncgen finalization-by-gc tests in test_base_events test_asyncgen_finalization_by_gc and test_asyncgen_finalization_by_gc_in_other_thread now pass; GC finalizes the async generators. Assisted-by: Claude * Unmark test_sni_callback_refcycle in test_ssl The servername-callback reference cycle is now collected by GC. Assisted-by: Claude * Widen GC stop-the-world gates from unix-only to all threading builds Change `cfg(all(unix, feature = "threading"))` to `cfg(feature = "threading")` on the stop-the-world machinery so it also compiles and runs on non-unix threading builds: - StopTheWorldState, its stats, stw_trace, and the stop_the_world field - ThreadSlot state/stop_requested/thread fields and their initializers - wait_while_suspended/attach_thread/detach_thread/suspend_if_needed/do_suspend, allow_threads, stop_requested_for_current_thread, and the enter_vm / VmBootstrapGuard / attach_current_thread / release_current_thread / cleanup attach-state wiring - eval_breaker_tripped, check_signals, run_scheduled_gc, signal GC_BIT / schedule_gc / take_gc_scheduled, and the frame.rs safepoint call - CollectStopTheWorld and its use in collect_inner - QSBR::online/offline, now called from attach/detach on all threading builds - debug_assert_current_thread_attached and its type-cache call sites maybe_collect defers auto-collection to the bytecode safepoint on every threading build instead of only unix; non-threading builds keep the inline collect. stw_trace writes to std stderr on non-unix. top_frame publishing (CURRENT_TOP_FRAME_SLOT, set_current_frame), the frame-walk debug assert in collect_inner, and the fork reinit helpers remain unix-only; non-unix keeps ThreadSlot::frames for introspection. Assisted-by: Claude * Detach current thread around blocking _winapi/_overlapped waits Wrap the blocking Windows wait calls in `vm.allow_threads` so the calling thread transitions ATTACHED -> DETACHED for the duration of the wait: - _winapi: WaitForSingleObject, WaitForMultipleObjects, BatchedWaitForMultipleObjects, ConnectNamedPipe, ReadFile, Overlapped.GetOverlappedResult - _overlapped: Overlapped.getresult These previously blocked while ATTACHED, so a stop-the-world requester could never suspend the thread and spun in its wait loop indefinitely. * Format WaitForMultipleObjects allow_threads closure * Treat concurrent sni_callback removal as no-op in invoke_sni_callback * Assert capi refcount on a fresh mortal list instead of the int type object The refcount test asserted exact incref/decref deltas on PyInt's shared type object. That object's reference count is perturbed by the other capi tests running in parallel, and is immortal under some interpreter configurations, so the deltas were not reliably +1/-1. Assert them on a freshly created, uniquely owned list whose reference count is private to the test and mortal. Assisted-by: Claude * Traverse and GC-track PyOSError instances The `#[pyexception]` struct macro now forwards a `traverse` option to the generated `#[pyclass]`, and `ExceptionItemMeta` accepts the `traverse` key. `PyOSError` is marked `traverse = "manual"`, so `HAS_TRAVERSE` is true and OSError-family instances are tracked at creation and traversed by the collector. `PyOSError::traverse` now visits the underlying `PyBaseException` (traceback, cause, context, args) instead of `PyException::try_traverse`, which was a no-op because `PyException` has `HAS_TRAVERSE = false`. * Unmark test_blockingioerror in test_io The BlockingIOError reference cycle is now collected by GC. Assisted-by: Claude |
||
|
Jeong, YunWon
|
fafd9c75df |
more hostenv isolation (#7932)
* Move which crate dependency from vm to host_env
Add host_env::fs::which wrapper and update getpath.rs call site.
* Move readline module from vm to host_env
Move crates/vm/src/readline.rs to crates/host_env/src/readline.rs and
re-export it from rustpython_vm::readline. Move rustyline dependency
from vm to host_env. Drop the host_env feature gates on history I/O
inside the moved module.
* Move gethostname and mac_address dependencies from stdlib to host_env
Add host_env::socket::hostname and host_env::socket::mac_address
wrappers. Update _socket.gethostname and _uuid.get_node_id call sites.
* Re-export libc constants and types through host_env
Move direct libc constant and type references in stdlib (faulthandler,
fcntl, multiprocessing, posixshmem, select, syslog) to use host_env
re-exports. Replace libc::c_int and libc::c_uint in stdlib with
core::ffi equivalents.
* Re-export mmap libc constants through host_env
Move direct libc MADV_*, MAP_*, PROT_*, EOVERFLOW references in
stdlib::mmap to host_env::mmap re-exports. Replace libc::c_int with
core::ffi::c_int.
* Re-export locale and termios libc constants through host_env
Move direct libc LC_*, ABDAY_*, ABMON_*, TIOC*, FIO* and related
constants in stdlib::locale and stdlib::termios to host_env
re-exports. Replace libc::c_char in stdlib::locale and
stdlib::openssl::cert with core::ffi::c_char.
* Replace libc c-types and FILE wrappers in openssl with host_env
Replace libc::c_int, c_long, c_uchar, c_uint, c_ulong, c_void,
c_char in stdlib::openssl with core::ffi equivalents. Replace
libc::ENOENT with host_env::errno::errors::ENOENT. Add
host_env::fileutils::{CFile, fclose} and route the load_dh_params
fclose call and PEM_read_DHparams FILE pointer type through them.
* Fix Windows build: expose EOVERFLOW, SIGSEGV, SIGFPE on Windows
host_mmap::EOVERFLOW and host_faulthandler::{SIGSEGV, SIGFPE}
re-exports were gated to cfg(unix) but stdlib uses them from Windows
and platform-agnostic code paths.
* Re-export remaining libc types and constants through host_env
Move direct libc references in stdlib (resource, posixsubprocess,
grp, socket) to host_env re-exports. Replace libc::c_long,
libc::c_longlong, libc::c_char in stdlib::socket with core::ffi
equivalents. Add resource::{RLIMIT_*, RUSAGE_*, c_long, rlim_t,
rlimit, timeval}, posix::{c_char, pid_t}, grp::gid_t, and
socket::{AF_*, SOCK_STREAM, sa_family_t, socklen_t, sockaddr_*}
re-exports.
* Move socket2 and dns-lookup dependencies from stdlib to host_env
Re-export socket2 as host_env::socket::raw and dns-lookup as
host_env::socket::dns. Update _socket call sites to route through
host_env aliases.
* Move system-configuration dependency from stdlib to host_env
Re-export the macOS system-configuration crate as
host_env::system_configuration and update _scproxy call sites.
* Restrict CAN/ALG sockaddr structs to target_os linux
libc on Android (bionic) does not define sockaddr_can, so importing it
for target_os = "android" fails to compile. Keep AF_ALG/AF_CAN for both
linux and android, and gate sockaddr_alg/sockaddr_can to linux only.
Assisted-by: Claude
|
||
|
Jeong, YunWon
|
55c012cffd |
unicode crate (#8211)
* Extract unicodedata core into rustpython-unicode crate
Move the vm-free Unicode character database access — the generated UCD 3.2.0 /
latest tables, their build-time generation, and the icu4x/unicode_names2 lookups
for category/bidirectional/combining/east_asian_width/mirrored/decomposition/
normalize/is_normalized/digit/decimal/numeric/name/lookup — into a new leaf crate
over char/CodePoint/&Wtf8. stdlib/unicodedata.rs keeps the UCD pyclass binding: it
extracts the code point, boxes results, and maps errors. The unicode data files,
build.rs table generation, and the icu_properties/icu_normalizer/unicode_names2
dependencies move out of stdlib.
Assisted-by: Claude
* Route str predicates, casefold, and printable through rustpython-unicode
Add classify/case/identifier modules to the shared crate over char/&str/&Wtf8:
isalpha/isalnum/isdecimal/isdigit/isnumeric/isspace/isprintable classification,
XID identifier predicates, and full-mapping casefold. vm/builtins/str.rs and
literal/char.rs now call these instead of icu_properties/icu_casemap directly.
String-level iteration with final-sigma handling (lower/upper/title/capitalize,
islower/isupper) stays in the runtime.
Assisted-by: Claude
* Route sre_engine char classes through rustpython-unicode
Add the regex module with the SRE character-class and case predicates
(is_word/is_space/is_digit, is_uni_* Unicode variants, ascii/locale/unicode case
folding). sre_engine/string.rs now forwards to it, keeping its public API and
behavior identical — including the ASCII-only is_uni_digit and the hardcoded
is_uni_space table.
Assisted-by: Claude
* Route \N{} name lookups through rustpython-unicode
codegen/string_parser.rs (\N{...} escapes) and common/encodings.rs (the
namereplace error handler) now resolve character names via rustpython-unicode
instead of depending on unicode_names2 directly, leaving the crate as the sole
owner of the name database.
Assisted-by: Claude
* Add differential Unicode sweep and shared-crate snippet test
tests/differential.rs sweeps the full 0..0x110000 range and compares each str
classification predicate against a committed CPython 3.14 reference dataset
(tests/data/cpython3.14_predicates.txt, produced by generate_reference.py). Code
points that differ only because the Rust std / icu4x build ships a later Unicode
release than CPython 3.14's 16.0.0 are recorded in
tests/data/version_skew_cpython3.14.txt, regenerable via
RUSTPYTHON_UNICODE_REGEN_SKEW=1; the regen refuses to record any
cpython=true/crate=false divergence, so only newly-assigned code points are
allowed. Any other divergence fails. Both data files use a run-length
`predicate start:end,...` encoding.
extra_tests/snippets/stdlib_unicode_shared.py exercises the routed surface end to
end (str predicates, casefold, identifiers, unicodedata, normalize, \N{}, and re
character classes) and passes identically on CPython 3.14 and RustPython.
Assisted-by: Claude
* Match regex \d to Unicode decimal digits
is_uni_digit previously matched only ASCII 0-9, so re's \d in Unicode mode
missed decimal digits like ٥ and ५. SRE_UNI_IS_DIGIT matches Py_UNICODE_ISDECIMAL
(category Nd), so route it through classify::is_decimal. Unmasks test_bug_6561 in
test_re and extends the shared-crate snippet with the Nd/Nl/No cases.
Assisted-by: Claude
* Replace shallow Unicode wrappers with re-exports
Address review feedback on the extraction.
Functions that only forwarded to the shared crate become `use ... as`
re-exports rather than hand-written wrappers:
- literal: is_printable is dropped; escape.rs calls
rustpython_unicode::classify::is_repr_printable directly.
- unicode::identifier::is_continue re-exports is_xid_continue.
- unicode::data::lookup_character re-exports unicode_names2::character.
- Drop the unused repr(u8) on DecompositionType.
The SRE character-class and case predicates move back into
sre_engine::string (their pre-extraction home) instead of living in a
unicode::regex module that only sre_engine and the vm _sre binding used;
is_uni_digit/is_uni_alnum still delegate to rustpython_unicode::classify.
engine.rs and _sre consume them from sre_engine::string again.
The unicode crate's tests move from a single root tests module in lib.rs
into per-module test submodules (case, classify, data, identifier,
normalize).
Assisted-by: Claude
* no-std
* Address review feedback: test module, dead variant, WTF-8 normalization
- differential.rs: move the sweep tests into a `mod tests` block and drop
the file-level allow of clippy::tests_outside_test_module and
std_instead_of_alloc; the test uses alloc collections instead.
- data.rs: remove the never-constructed DecompositionType::Canonical variant
and its allow(unused); compatibility decomposition never produces it and
canonical decomposition is handled through icu4x.
- normalize.rs: is_normalized now takes &Wtf8 and checks each UTF-8 run,
skipping lone surrogates, matching normalize's run-wise behavior.
- unicodedata.rs: pass as_wtf8() to is_normalized.
Assisted-by: Claude
|
||
|
Joshua Megnauth
|
5c1803e019 |
Fix most of unicodedata; unmask multiple tests (#7947)
|
||
|
Jeong, YunWon
|
cf4d076ad4 | xz (#7756) | ||
|
Joshua Megnauth
|
3d53342b61 |
Update random and cryptography crates (#8010)
Redo of #7800. Bumping the random and crypto crates is not trivial. Both sets of crates have had API changes which require manual updates. AI use: I wrote all of the code myself but used AI to figure out the new APIs. AI explained SHAKE's block size and why the block_size() function doesn't exist in the new crate. I confirmed the block size, which is actually represented in the new types for SHAKE. `Shake128` is actually `Shake<168>` - 168 is the block size. I cited Wikipedia as it lists the SHAKE block size in bits. I also used AI to figure out HMAC for SHA3. Assisted-by: Codex:gpt-5.4 |
||
|
Jeong, YunWon
|
024201c050 | rename ssl-vendor (#7989) | ||
|
Joshua MegnauthandJeong, YunWon
|
f347ae4fbb |
Support custom rustls providers (#7657)
* Modularize rustls as work towards providers `rustls`'s architecture is very clean and trait-driven. There are many providers for `rustls` including the built-in `aws-lc-rs` and `ring` as well as backends for `boringssl`, `graviola`, `openssl`, `mbedtls`, etc. This commit removes the hard dependency on `aws-lc-rs` and adds support for `ring`. It works towards #7059 as well. * Clean up rustls features * Remove ring as an explicit feature * ssl-rustls is the default and implies aws-lc * Support custom rustls crypto providers The new feature, `ssl-rustls-no-provider`, enables custom rustls providers. By default, `aws-lc-rs` is enabled which matches the old behavior and keeps backward compatibility. I wrote a new type that abstracts what we need from crypto providers. CryptoExt encapsulates the ticketer as well as cipher suites and KX groups. I wrote fallbacks to help select a reasonable default if a provider is missing features (they all seem to support the same things though). I also wrote an example to show how to actually use custom providers. * Fix duplicate VERIFY_X509 constants and unused imports in ssl module Remove duplicate VERIFY_X509_STRICT/VERIFY_X509_PARTIAL_CHAIN definitions from compat.rs (already defined in _ssl module with #[pyattr]). Remove unused imports: ClientConnection, ServerConnection. * no-provider as default * Fix CI failures: openssl build, wasm target, cargo-shear - Update openssl.rs to use renamed sock_wait/SockWaitKind and add vm parameter - Add skip_ssl for wasm32-wasip2 target (aws-lc-sys cannot build for wasm) - Remove unused workspace dependency aws-lc-rs - Fix foreign-types-shared version to match openssl's dependency (0.1) - Restore Cargo.lock from upstream/main --------- Co-authored-by: Jeong, YunWon <jeong@youknowone.org> |
||
|
fanninpm
|
c4968e3717 |
Replace ahash with rapidhash (#7954)
* Add `rapidhash` to list of dependencies * Use `rapidhash::quality::RandomState` in `codegen` crate * Use `rapidhash::quality::RandomState` in `stdlib` crate * Use `rapidhash::quality::RandomState` in `vm` crate * Remove `ahash` from lists of dependencies |
||
|
Joshua Megnauth
|
2c558fa749 |
rustix/windows-sys for page size; drop maplit (#7876)
The `page_size` crate is a simple libc wrapper for Unix and uses `winapi-rs` for Windows. `windows-sys` and `windows-rs` are the modern alternatives for the unmaintained `winapi-rs`. Both crates are maintained by Microsoft - they're official. Getting the page size is a simple call for both Unix and Windows. Besides Unix and Windows, I also added the page size for wasm32 which the `page_size` crate did not support. `wasm32`'s page size is a constant that is defined by the spec, so I hard coded it without adding additional dependencies. Finally, I dropped `maplit` which is seven years old and only used in one place. Calling `collect()` with a single item iterator is idiomatic as well as better in this case because Rust can optimize it. `maplit` called `HashMap::insert` which over allocates to amortize future allocs. |
||
|
Jeong, YunWon
|
bf588b3c4e | more hostenv impl (#7604) | ||
|
Shahar Naveh
|
4290fd8966 |
Fixed: unused manifest key: dependencies.libz-sys.package (#7792)
* Fixed: `unused manifest key: dependencies.libz-sys.package` * Use `libz-rs-sys` instead * Adjust rust code |
||
|
Shahar Naveh
|
76128d183f |
Remove unused rust impl for formatting dis output (#7660)
* Remove unused rust impl for formatting dis output
* remove `examples/dis.rs`
* Added tests
* Update lock
* Try to set snapshot dir
* Remove verbose flag
* Regenerate snapshots after #7711
* Revert "Bump insta from 1.46.3 to 1.47.2 (#7706)"
This reverts commit
|
||
|
fanninpm
|
3c8929fcfe |
Move dependencies to workspace for stdlib crate (#7747)
* Add dependencies from `stdlib` crate to workspace * Declare dependencies as workspace = true |
||
|
dependabot[bot]
|
86a1ee68ec |
Bump rustls from 0.23.38 to 0.23.39 (#7705)
Bumps [rustls](https://github.com/rustls/rustls) from 0.23.38 to 0.23.39. - [Release notes](https://github.com/rustls/rustls/releases) - [Changelog](https://github.com/rustls/rustls/blob/main/CHANGELOG.md) - [Commits](https://github.com/rustls/rustls/compare/v/0.23.38...v/0.23.39) --- updated-dependencies: - dependency-name: rustls dependency-version: 0.23.39 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Shahar Naveh
|
6f09043fd4 |
Upgrade der to 0.8 (#7695)
* Update `der` to 0.8, move to workspace dependencies * Add `pem` feature |
||
|
dependabot[bot]
|
0bef8ef5f7 |
Bump uuid from 1.23.0 to 1.23.1 (#7692)
Bumps [uuid](https://github.com/uuid-rs/uuid) from 1.23.0 to 1.23.1. - [Release notes](https://github.com/uuid-rs/uuid/releases) - [Commits](https://github.com/uuid-rs/uuid/compare/v1.23.0...v1.23.1) --- updated-dependencies: - dependency-name: uuid dependency-version: 1.23.1 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
797e03f16e |
Bump foreign-types-shared from 0.1.1 to 0.3.1 (#7691)
Bumps [foreign-types-shared](https://github.com/sfackler/foreign-types) from 0.1.1 to 0.3.1. - [Release notes](https://github.com/sfackler/foreign-types/releases) - [Commits](https://github.com/sfackler/foreign-types/commits/foreign-types-shared-v0.3.1) --- updated-dependencies: - dependency-name: foreign-types-shared dependency-version: 0.3.1 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
e3b94e032f |
Bump aws-lc-rs from 1.16.2 to 1.16.3 (#7684)
Bumps [aws-lc-rs](https://github.com/aws/aws-lc-rs) from 1.16.2 to 1.16.3. - [Release notes](https://github.com/aws/aws-lc-rs/releases) - [Commits](https://github.com/aws/aws-lc-rs/compare/v1.16.2...v1.16.3) --- updated-dependencies: - dependency-name: aws-lc-rs dependency-version: 1.16.3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
63aa18329e |
Bump rustls from 0.23.37 to 0.23.38 (#7678)
Bumps [rustls](https://github.com/rustls/rustls) from 0.23.37 to 0.23.38. - [Release notes](https://github.com/rustls/rustls/releases) - [Changelog](https://github.com/rustls/rustls/blob/main/CHANGELOG.md) - [Commits](https://github.com/rustls/rustls/compare/v/0.23.37...v/0.23.38) --- updated-dependencies: - dependency-name: rustls dependency-version: 0.23.38 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
4732351f98 |
Bump rustls-platform-verifier from 0.6.2 to 0.7.0 (#7664)
Bumps [rustls-platform-verifier](https://github.com/rustls/rustls-platform-verifier) from 0.6.2 to 0.7.0. - [Release notes](https://github.com/rustls/rustls-platform-verifier/releases) - [Changelog](https://github.com/rustls/rustls-platform-verifier/blob/main/CHANGELOG) - [Commits](https://github.com/rustls/rustls-platform-verifier/compare/v/0.6.2...v/0.7.0) --- updated-dependencies: - dependency-name: rustls-platform-verifier dependency-version: 0.7.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
b4e230103b |
Bump openssl from 0.10.77 to 0.10.78 (#7659)
Bumps [openssl](https://github.com/rust-openssl/rust-openssl) from 0.10.77 to 0.10.78. - [Release notes](https://github.com/rust-openssl/rust-openssl/releases) - [Commits](https://github.com/rust-openssl/rust-openssl/compare/openssl-v0.10.77...openssl-v0.10.78) --- updated-dependencies: - dependency-name: openssl dependency-version: 0.10.78 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
|
5bcb75bc0f |
Bump uuid from 1.22.0 to 1.23.0 (#7565)
* Bump uuid from 1.22.0 to 1.23.0 Bumps [uuid](https://github.com/uuid-rs/uuid) from 1.22.0 to 1.23.0. - [Release notes](https://github.com/uuid-rs/uuid/releases) - [Commits](https://github.com/uuid-rs/uuid/compare/v1.22.0...v1.23.0) --- updated-dependencies: - dependency-name: uuid dependency-version: 1.23.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Fix CI for uuid 1.23.0 Context deprecation Agent-Logs-Url: https://github.com/RustPython/RustPython/sessions/bec0d567-f41e-407f-bf99-0463ca555f74 Co-authored-by: youknowone <69878+youknowone@users.noreply.github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: youknowone <69878+youknowone@users.noreply.github.com> |
||
|
Shahar Naveh
|
9c729bc80c |
Use cfg_select! (#7636)
|
||
|
dependabot[bot]
|
e9ab572322 |
Bump the openssl group with 2 updates (#7643)
Bumps the openssl group with 2 updates: [openssl](https://github.com/rust-openssl/rust-openssl) and [openssl-sys](https://github.com/rust-openssl/rust-openssl). Updates `openssl` from 0.10.76 to 0.10.77 - [Release notes](https://github.com/rust-openssl/rust-openssl/releases) - [Commits](https://github.com/rust-openssl/rust-openssl/compare/openssl-v0.10.76...openssl-v0.10.77) Updates `openssl-sys` from 0.9.112 to 0.9.114 - [Release notes](https://github.com/rust-openssl/rust-openssl/releases) - [Commits](https://github.com/rust-openssl/rust-openssl/compare/openssl-sys-v0.9.112...openssl-sys-v0.9.114) --- updated-dependencies: - dependency-name: openssl dependency-version: 0.10.77 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: openssl - dependency-name: openssl-sys dependency-version: 0.9.114 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: openssl ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
|
f51f27c949 |
Create rustpython-host-env (#7582)
* Create rustpython-host-env crate; move host abstractions out of common
Move os, crt_fd, fileutils, windows, macros modules from
rustpython-common into the new rustpython-host-env crate.
This isolates host OS API wrappers behind a crate boundary
with zero Python runtime dependency.
- Add crates/host_env to workspace
- Drop nix, windows-sys, widestring deps from common
- Wire vm and stdlib to depend on rustpython-host-env
- Migrate all imports from common::{os,crt_fd,fileutils,windows}
to rustpython_host_env::
* refactor: extract host helpers
Agent-Logs-Url: https://github.com/RustPython/RustPython/sessions/48d1e64d-37ce-409f-b511-8e61a349665c
Co-authored-by: youknowone <69878+youknowone@users.noreply.github.com>
* lint: enforce direct host API boundaries
Agent-Logs-Url: https://github.com/RustPython/RustPython/sessions/97225fb7-7b3d-4197-a77c-eb44aead5b13
Co-authored-by: youknowone <69878+youknowone@users.noreply.github.com>
* refactor: extract remaining host env helpers
Agent-Logs-Url: https://github.com/RustPython/RustPython/sessions/d96f57e1-b196-4460-9983-97d5ff118835
Co-authored-by: youknowone <69878+youknowone@users.noreply.github.com>
* fix: clean extracted host env follow-up
Agent-Logs-Url: https://github.com/RustPython/RustPython/sessions/d96f57e1-b196-4460-9983-97d5ff118835
Co-authored-by: youknowone <69878+youknowone@users.noreply.github.com>
* fix: document env mutation safety
Agent-Logs-Url: https://github.com/RustPython/RustPython/sessions/d96f57e1-b196-4460-9983-97d5ff118835
Co-authored-by: youknowone <69878+youknowone@users.noreply.github.com>
* refactor: split host fs helpers from fileutils
Agent-Logs-Url: https://github.com/RustPython/RustPython/sessions/c57424c5-0e1d-490a-82b3-2d2f6c8cf2cd
Co-authored-by: youknowone <69878+youknowone@users.noreply.github.com>
* fix: resolve latest host env ci regressions
Agent-Logs-Url: https://github.com/RustPython/RustPython/sessions/899eb717-ebc6-4a4a-870c-2a15c5f33e02
Co-authored-by: youknowone <69878+youknowone@users.noreply.github.com>
* fix: resolve remaining windows clippy host fs calls
Agent-Logs-Url: https://github.com/RustPython/RustPython/sessions/12f32740-8173-4b10-a1d6-00b29e90a8ec
Co-authored-by: youknowone <69878+youknowone@users.noreply.github.com>
* host_env
---------
Co-authored-by: Jeong, YunWon <jeong@youknowone.org>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: youknowone <69878+youknowone@users.noreply.github.com>
|
||
|
Shahar Naveh
|
c857db6e9e |
Replace unmaintained unic crates (#7555)
|
||
|
dependabot[bot]
|
870d15d031 |
Bump aws-lc-rs from 1.16.0 to 1.16.2 (#7509)
Bumps [aws-lc-rs](https://github.com/aws/aws-lc-rs) from 1.16.0 to 1.16.2. - [Release notes](https://github.com/aws/aws-lc-rs/releases) - [Commits](https://github.com/aws/aws-lc-rs/compare/v1.16.0...v1.16.2) --- updated-dependencies: - dependency-name: aws-lc-rs dependency-version: 1.16.2 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
de3aa3471e |
Bump libsqlite3-sys from 0.36.0 to 0.37.0 (#7485)
Bumps [libsqlite3-sys](https://github.com/rusqlite/rusqlite) from 0.36.0 to 0.37.0. - [Release notes](https://github.com/rusqlite/rusqlite/releases) - [Changelog](https://github.com/rusqlite/rusqlite/blob/master/Changelog.md) - [Commits](https://github.com/rusqlite/rusqlite/compare/v0.36.0...v0.37.0) --- updated-dependencies: - dependency-name: libsqlite3-sys dependency-version: 0.37.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Shahar Naveh
|
2e777a6fb3 |
Add cargo shear CI step (#7461)
* Add `cargo shear` CI step * Remove some deps * Move `ruff_python_parser` to dev deps * Remove `thread_local` crate * Cleanup `num-integer` usage * Remove parser fro. wasm * Remove windows-sys from venv launcher * Update lock * Resolve more * Fix wasm * Update lock |