mirror of
https://github.com/RustPython/RustPython.git
synced 2026-10-28 18:59:04 +08:00
ea4e51d52da360ad96854a5e5feeccf07cb5ceff
426
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Jeong, YunWon
|
979ea4c4dc |
host_env: export vsock, wasm socket, Darwin flags (#8779)
* host_env: export vsock, wasm socket, Darwin flags Publish vsock/UDPLITE/SOL fallbacks, wasm socket and errno constants, remaining BSD chflags, and S_IFDOOR/PORT/WHT. Assisted-by: Claude * stdlib: take vsock, wasm socket, Darwin flags from host_env _socket, wasm _socket/_io/errno, _stat, and openssl alerts now re-export the host_env names. Assisted-by: Claude * host_env: keep wasm socket/errno off emscripten wasm32-unknown-emscripten is unix, so the stub modules must be unknown-unknown (and WASI) only. Assisted-by: Claude * stdlib: match socket module_def cfg to the module Assisted-by: Claude * host_env: drop duplicate Windows SOL_UDP The IANA fallback also compiled on Windows, colliding with the WinSock definition. Assisted-by: Claude |
||
|
Jeong, YunWon
|
eb1546ae08 |
posix: add os.forkpty so pty.spawn does not hang (#8782)
* posix: add os.forkpty pty.fork prefers forkpty so a nested spawn is a new session with its own controlling tty. The openpty+login_tty fallback hung the jsontests collector. Assisted-by: Claude * select: rebuild fd sets after EINTR select(2) updates the sets in place. Retrying an interrupted call with empty sets and a NULL timeout blocks forever. Assisted-by: Claude * http.server: keep CGI off the body-read timeout A millisecond handler timeout races the client thread writing the request and is inherited across CGI fork, which RSTs the client under load. Assisted-by: Claude * Revert "http.server: keep CGI off the body-read timeout" Lib edits outside a CPython copy are not allowed. Assisted-by: Claude * host_env: note why forkpty is libc not nix Assisted-by: Claude * benches: run deepcopy 10x more iterations Assisted-by: Claude |
||
|
Jeong, YunWon
|
f56bfc9a5c |
Show stdin traceback lines; omit StopIteration None args (#8778)
* Show stdin traceback lines; omit StopIteration None args Register interactive <stdin> source in linecache so REPL tracebacks include the source line and caret. Build StopIteration with empty args when the value is None, and wrap monitoring STOP_ITERATION the same way. Unskip REPL stderr flush tests and pdb Flaky doctests. Assisted-by: Grok:4.6 * Take &Py<PyCode> in register_code_in_linecache Assisted-by: Grok:4.6 * Count Unknown closing braces in tokenize EOF check `$` inside an f-string interpolation is TokenKind::Unknown, so the matching `}` is Unknown rather than Rbrace and generate_tokens raised TokenError. Also convert &Py<PyCode> to PyObjectRef explicitly when registering linecache. Assisted-by: Grok:4.6 * Unskip test_pdb.PdbTestCase.test_zipimport The test now passes, so the expectedFailure is an unexpected success under CI. Assisted-by: Grok:4.6 * Park STW threads; poll sockets after timeout deadline Check per-thread stop_requested in eval_breaker_tripped so workers still park when another interpreter clears STOP_BIT. Start socket timeout after the fd snapshot, and poll once with timeout 0 when the deadline has already passed, so a readable socket is not dropped as timed out. Assisted-by: Grok:4.6 |
||
|
Jeong, YunWon
|
7b018871a3 |
host_env: export remaining stdlib constants (#8760)
* host_env: export winsound, spawn, and remaining SSL_OP flags Assisted-by: Claude * stdlib: take winsound, nt spawn, and SSL_OP names from host_env Assisted-by: Claude * host_env: export ssl alerts, sysexits, socket extras Assisted-by: Claude * stdlib: take remaining host_env and common constants Assisted-by: Claude * Remove unused workspace exitcode dependency Assisted-by: Claude * host_env: export PROTO, access, reparse, wait flags Assisted-by: Claude * stdlib: take PROTO, access, reparse, wait from host_env Assisted-by: Claude |
||
|
Jeong, YunWon
|
54420c4471 |
Drop serial MP CI retries and killpg overlay (#8756)
Multiprocessing fork/spawn/forkserver now run in the main -j $CORES CPython test job. Remove expectedFailureIf for os.killpg now that killpg exists. Assisted-by: Grok:4.6 |
||
|
Jeong, YunWon
|
f759c18a0e |
Take &Py<T> instead of bare payload references (#8753)
* Drop gc_traversal from CodSpeed benches Remove benches/benchmarks/gc_traversal.py so CodSpeed no longer includes that execution benchmark. Assisted-by: Grok:4.6 * Take &Py<T> instead of bare payload references Change `&PyCode`, `&PyInt`, `&PyTuple`, `&PySlice`, `&PyStr`, `&PyFunction`, `&PyList` and `&PyDict` in function parameters, closure types and local type annotations to `&Py<T>` where every caller already holds a `Py<T>`. Assisted-by: Claude Fable 5.1 Assisted-by: grok-4.6 |
||
|
hyoinandout
|
c857f3acfd |
mmap: correct find/rfind for empty subsequences (#8749)
get_find_range clamped end up to start, so an inverted range such as (2, 1) became the empty-but-valid range (2, 2). A real subsequence then found nothing there, which looked right, but the empty subsequence matched and both methods reported a hit where CPython reports -1. rfind also returned the start of the range for an empty subsequence, because its body was copied from find and the returned edge was never adjusted. An empty subsequence matches at the edge the scan begins from, so a reverse scan must report the end. Dropping the clamp lets an inverted range stay inverted and be rejected explicitly. find and rfind now share find_inner, which keeps the two directions from drifting apart again, and it checks the mmap is open before inspecting the subsequence, so find(b"") on a closed mmap raises ValueError instead of returning a position. m = mmap.mmap(-1, 3); m[:] = b"abc" m.rfind(b"") # was 0, now 3 m.rfind(b"", 1, 2) # was 1, now 2 m.find(b"", 2, 1) # was 2, now -1 m.rfind(b"", 2, 1) # was 2, now -1 Verified on macOS arm64 against CPython 3.14.7 with a 19-case table covering both methods over empty and non-empty subsequences, clamped, negative and inverted ranges: 8 mismatches before, none after. The added snippet assertions fail on the unpatched build and pass on CPython 3.14.7 as well. test_mmap, cargo fmt --check and cargo clippy -p rustpython-stdlib -D warnings are clean. Fixes #8722 Assisted-by: Claude Code:claude-opus-5 |
||
|
Jeong, YunWon
|
7bc3ef6c0a |
ExceptionGroup fields, OSError use-init, and live f_lineno (#8738)
* Add ExceptionGroup fields and OSError use-init BaseExceptionGroup stores msg, excs, and excs_str; args keep the original sequence. OSError defers parse when a subclass overrides only __init__. characters_written can be deleted. Exception args/traceback/cause/context reject invalid setattr/delattr. UnicodeError.object is a deletable field. except* remainder projection copies traceback/context/cause. SyntaxError.__init__ clears omitted location attributes. Assisted-by: Grok:grok-4.6 * Fix OSError payload init and UnicodeError construction new_payload_exception now runs slot_new so BlockingIOError parses characters_written. new_exception falls back to invoke_exception for extra-payload types. Mixed tab/space UnexpectedIndentation is TabError. AttributeError pickle omits obj. Assisted-by: Grok:grok-4.6 * address review: classify TabError from error line Assisted-by: Grok:grok-4.6 * Add new_simple_exception; fill SyntaxError and f_lineno new_exception constructs extra-payload types; BaseException-sized types use new_simple_exception. Non-UTF-8 SyntaxError sets location and text. Codegen errors load text via ProgramText. Keep Missing parentheses capitalization. f-string adjacent atoms report a missing comma. f_lineno may be None; lasti is not restored on unwind. Assisted-by: Grok:grok-4.6 * address review: skip f-string continuation keywords Assisted-by: Grok:grok-4.6 * Fix clippy map_unwrap_or in Non-UTF-8 offset Assisted-by: Grok:grok-4.6 * Rename encode error ctor and exception setters new_unicode_encode_error_real is new_unicode_encode_error. set___traceback__/__cause__/__context__ are pygetset setters; Rust uses set_traceback/set_cause/set_context. Assisted-by: Grok:grok-4.6 * Use set_context in native pickle Assisted-by: Grok:grok-4.6 * Collapse nested if in asyncio traceback restore Assisted-by: Grok:grok-4.6 * Use current instruction for f_lineno addr2line lasti is stored as the next instruction index. addr2line of that byte offset is past RETURN_VALUE, so f_lineno was None and stdlib_threading_current_frames hung on leftover workers. Assisted-by: Grok:grok-4.6 * Look up format args, dir class, and frame builtins Consume printf args before rejecting unknown conversions. object.__dir__ uses getattr of __class__/__dict__. _PyEval_GetBuiltin looks up iter/__import__ on the frame. Assisted-by: Grok:grok-4.6 * Fix builtin lookups after rebase and unxfail passing tests Assisted-by: Grok:grok-4.6 * Restore RERAISE lasti and SyntaxError locations RERAISE with oparg restores lasti from the exception table before unwind, so leftover f_lineno matches the original raise. Handler lookup still uses the raising instruction. Indentation errors outrank print/exec missing-parentheses. BOM-with-non-utf8, bytes literals, and global+nonlocal now report the same offsets as the parser/symtable. object.__dir__ keeps string keys; type.__type_params__ skips the getset wrapper stored on builtin types. Assisted-by: Grok:grok-4.6 * Unxfail syntax/import tests and bind jump locals IMPORT_NAME passes fromlist None instead of an empty tuple. Jumping f_lineno binds None into unbound localsplus slots. str lstrip/rstrip copy subclasses to exact str when unchanged. Assisted-by: Grok:grok-4.6 * Fix import_from callers and rename frame lineno sqlite3.iterdump now passes fromlist as a PyObject. FrameObject::lineno is the i32 helper; f_lineno is the getset. Assisted-by: Grok:grok-4.6 * Pass sqlite fromlist without Into conversion from_list.into() could not infer Into<PyObjectRef> at the import_from call. The typed tuple already converts. Assisted-by: Grok:grok-4.6 |
||
|
Jeong, YunWon
|
f1b8b09492 |
Don't hold socket locks across blocking I/O (#8746)
accept/recv/send kept the socket RwLock during the syscall. After fork, the child copied that held lock, so _ResourceSharer._afterfork hung in listener.close(). Unskip fork connection pickling, pool task ref release, and spawn thread-pool terminate. Add extra_tests for fork+accept close and fork after DupFd. Assisted-by: Grok:4.6 |
||
|
Jeong, YunWonandLee Dogeon
|
d02d81e740 |
perf: VM and runtime hot paths from pyperformance (#8737)
* time: use CLOCK_UPTIME_RAW for monotonic/perf_counter on Apple
CLOCK_MONOTONIC keeps advancing during system sleep on macOS, unlike
CPython's mach_absolute_time()-backed perf_counter/monotonic. Benchmarks
that spanned a laptop sleep recorded thousands of seconds for a single
iteration (pyperformance float/fannkuch showed ~1000x slowdowns).
Use CLOCK_UPTIME_RAW (equivalent to mach_absolute_time) on Apple targets
and report it as such in time.get_clock_info().
Assisted-by: Claude Code:claude-fable-5-1
* vm: derive f_lineno lazily instead of tracking prev_line per instruction
The dispatch loop wrote prev_line (a locations lookup plus a Cell store)
on every instruction only so f_lineno could read it. f_lineno now derives
the line from lasti and code.locations, prev_line is only maintained on
the cold tracing path for 'line' event de-duplication, the two use_tracing
reads per instruction are merged into one, and the pre-dispatch lasti
reload is replaced by the value just stored.
When a trace function is installed on an already-running frame (bdb
stepping back into an untraced caller, frame.f_trace assignment, a local
trace returned from the 'call' event on a resumed generator) prev_line is
synced from lasti so no spurious 'line' event fires.
Assisted-by: Claude Code:claude-fable-5-1
* vm: coalesce per-call thread-locals into one FRAME_SLOT_CACHE
set_current_frame ran on every frame enter/exit and touched three separate
thread_local! statics (each a _tlv_get_addr call on macOS). Merge them into
a single thread-local struct so the hot path does one TLS access.
Assisted-by: Claude Code:claude-fable-5-1
* vm: fold stop/finalizing checks into EVAL_BREAKER
eval_breaker_tripped ran three checks per bytecode: the finalizing flag,
a thread-local stop_requested lookup, and the EVAL_BREAKER load. Add
STOP_BIT and FINALIZING_BIT so the per-instruction check is a single
relaxed atomic load; the slow path (check_signals) still re-derives the
exact per-thread answer. The bits are set alongside the per-thread flags
and cleared under the stop-the-world exclusion.
Assisted-by: Claude Code:claude-fable-5-1
* gc: set the TRACKED bit with a plain store for freshly allocated objects
gc_bits is zero for a fresh allocation (both the freelist reuse path and
PyInner::new zero it), so track_new_object can initialise the bit with a
relaxed store instead of a fetch_or. Re-tracking resurrected objects keeps
the read-modify-write variant.
Assisted-by: Claude Code:claude-fable-5-1
* vm: skip thread exception publish when pushing/popping a None exc slot
Generator/coroutine resume pushes and pops an exc_info slot on every
send. When that slot is None the value returned by topmost_exception
cannot change (it skips None entries), so the thread-local exception
publish in push_exception/pop_exception is unnecessary.
Assisted-by: Claude Code:claude-fable-5-1
* vm: allocation-free keyword-call fast path and super() MRO walk
Keyword calls to plain functions where every parameter is filled exactly
once are reordered in place into the fast-locals layout (no IndexMap, no
keyword name clones); any mismatch falls back to the existing slow path
so error messages are unchanged. super().attr now walks the type's MRO
by reference instead of cloning every class into two Vecs per lookup.
Assisted-by: Claude Code:claude-fable-5-1
* builtins: cheaper enumerate and zip iteration
enumerate kept its counter as a BigInt behind a write lock and did bignum
arithmetic on every item; use a machine integer that promotes to BigInt
only for huge/negative starts or on overflow. zip pre-sizes its per-item
Vec so the common two-iterable case allocates once.
Assisted-by: Claude Code:claude-fable-5-1
* _asyncio: cache module objects and use native set/dict operations
Task registration re-resolved _asyncio module attributes and re-imported
contextvars on every task; the eager path additionally went through generic
call_method for set add/discard. Cache the module objects per thread
(validated by identity), split the contextvars cache from the task
containers, and use PySet/PyDict methods directly.
Assisted-by: Claude Code:claude-fable-5-1
* perf: use mimalloc as global allocator for rustpython CLI
Adds mimalloc (0.1.52, default-features = false) as the #[global_allocator]
for the rustpython binary, gated behind a new `mimalloc` feature enabled
by default on non-wasm targets (dep declared under
target.'cfg(not(target_arch = "wasm32"))'.dependencies, so wasm builds
never pull it in regardless of default-features).
Profiling a `t = (a, b, c)` loop showed malloc/free at ~40% of samples on
macOS with the system allocator. Switching to mimalloc gives consistent
wins across small-object-heavy workloads, medians over 5-9 interleaved
A/B rounds (release build, this machine):
tuple (5M `(a,b,c)`) 1.020s -> 0.863s (1.18x)
4-thread tuple (5M total) 1.643s -> 1.382s (1.19x)
class instances (2M) 0.874s -> 0.797s (1.10x)
dict-heavy (200k x 20) 2.196s -> 1.992s (1.10x)
string-heavy (join x 20) 0.662s -> 0.592s (1.12x)
nqueens (pyperformance) 0.327s -> 0.286s (1.14x)
chaos (pyperformance) 0.204s -> 0.191s (1.06x)
richards (pyperformance) 0.154s -> 0.151s (1.02x)
go (pyperformance) 0.411s -> 0.405s (1.01x)
listcomp / fib ~neutral (<=1%)
Peak RSS (/usr/bin/time -l): instances workload roughly unchanged
(29.0MB -> 29.1MB); dict-heavy workload actually drops (155MB -> 106MB),
likely because mimalloc reclaims freed pages more aggressively than the
system allocator across repeated large-dict churn.
`cargo build` and `cargo build --release` are warning-free with the
default feature set, and `cargo build --release --no-default-features
--features stdlib,threading,importlib,stdio,host_env` (mimalloc
excluded) also builds cleanly. CI's wasm32-wasip1 and other
--no-default-features build matrices in .github/workflows already omit
`mimalloc` from their feature lists and target wasm via the
cfg(not(wasm32)) dependency gate, so no workflow changes needed.
`rustpython -m test -j4 test_gc test_weakref test_threading test_dict
test_list test_tuple test_str test_bytes test_memoryview test_ctypes
test_mmap test_array` passes identically (2541 tests, 241 skipped) on
both the baseline and mimalloc binaries.
Assisted-by: Claude Code:claude-fable-5-1
* vm: cap dict/set table shrink and compact holes on resize
asyncio's eager task path (and its per-task bookkeeping sets in general)
repeatedly adds and discards distinct task objects from a handful of
long-lived sets (_eager_tasks, _scheduled_tasks). Two dict_inner issues
turned that steady-state churn into the dominant cost of async_tree_eager:
1. resize() picked a target size of `used * 2` with no floor, so once a
churny set settled around 1-2 live entries it shrank to a table that
the very next insert would already overflow, forcing a resize on
almost every single add. Clamping to the original minimum table size
(8) restores the headroom dict/set start with.
2. `entries` only ever grows (holes from deletions are never reclaimed),
so resize()'s rebuild pass scanned every entry ever inserted into the
dict's lifetime, not just the live ones - a set with a stable working
set size but heavy churn grew this scan, and the dict's memory, without
bound. resize() now compacts holes out of `entries` first, matching
CPython's compacting hash table and keeping the shape-stamp fast path
(which already requires a hole-free layout) reachable more often.
Measured with a standalone async_tree micro-benchmark (6 levels x 6
branches, pyperformance's bm_async_tree "none"/"eager" shape) run inside
one process for 10 asyncio.run() iterations, comparing medians of
interleaved binary runs on an otherwise idle machine:
- eager: ~2.91s -> ~0.86s median (about 3.4x)
- none: ~1.98s -> ~1.23s median (about 1.6x)
A single fresh-process run (no in-process accumulation) still improves
by roughly 15-20% for eager, since the resize-storm from point 1 hits
even a single run's worth of task churn.
test_asyncio (34 files, all passing), test_dict, test_set, test_weakref,
test_weakset, test_ordered_dict, test_gc, test_descr, test_class, and
`cargo test -p rustpython-vm -p rustpython-stdlib` all pass.
Assisted-by: Claude Code:claude-fable-5-1
* _functools: add native _lru_cache_wrapper
RustPython had no native _functools._lru_cache_wrapper, so
functools.lru_cache always fell back to the pure-Python implementation
in Lib/functools.py (RLock, linked-list bookkeeping, closures). That
fallback dominates typing.Protocol's __instancecheck__ path: it calls
inspect.getattr_static, whose _shadowed_dict is itself lru_cache'd, so
every isinstance() check against a runtime-checkable protocol paid for
several pure-Python cache-wrapper calls plus lock acquisitions.
Add a native pyclass mirroring CPython's C _lru_cache_wrapper: a
reentrant lock scoped only around the cache lookup/insert steps (not
held across the wrapped call, matching CPython's threading behavior),
LRU ordering piggybacked on the cache dict's own insertion order
(move-to-end via del+reinsert, oldest evicted from the front) so no
separate key-comparison structure is needed, and __copy__/__deepcopy__
returning self to match CPython since __reduce__ (used only for
pickling) requires __qualname__, which wrapped non-function callables
such as functools.partial don't have.
Measured with a standalone script reproducing pyperformance's
bm_typing_runtime_protocols isinstance() loop (300 loops, medians of
5 interleaved runs): 0.260s -> 0.182s, about 1.43x faster. In
isolation, a cache-hit through lru_cache went from ~0.31s to ~0.07s
for 200k calls, now faster than calling the underlying function
uncached (matching CPython's C accelerator).
Verified with test_typing, test_abc, test_inspect, test_functools,
test_weakref, test_isinstance, and test_descr (all pass, matching the
pre-existing baseline, including test_functools's already-present
TestLRUC suite which now actually exercises the native implementation).
Assisted-by: Claude Code:claude-fable-5-1
* sre_engine: ASCII fast path for \d \s \w unicode category checks
is_uni_digit/is_uni_space/is_uni_alnum went through icu_properties'
GeneralCategory table lookup for every character, even the common
ASCII case. Add a `ch < 0x80` fast path that resolves directly via
std's ascii predicates, falling back to the full table only for
non-ASCII code points (Latin-1 supplement chars like 'A-umlaut' still
route through the table since they are alphanumeric but not ASCII).
Measured on a tight `re.findall(r"\w+", ...)` / `re.sub` loop over
mostly-ASCII text (interleaved A/B, 4 rounds, medians):
re.sub: 1.178s -> 1.134s (~3.7% faster)
re.findall: 1.110s -> 1.062s (~4.3% faster)
Assisted-by: Claude Code:claude-fable-5-1
* vm: skip VM attach for no-op generator/coroutine __del__
Generator, coroutine, and async-generator dealloc always went through
drop_slow_inner's call_slot_del, which unconditionally attaches to a VM
(with_vm -> begin_interpreter_section -> per-interpreter thread-slot
hashmap lookup) before running del(), even though del() itself is a
documented no-op once the object is already closed or running - the
common case for a generator/coroutine consumed to completion.
Add PyTypeSlots::del_needed, an optional VM-free fast predicate checked
before call_slot_del/with_vm; generator, coroutine, and async_generator
install a check mirroring their own del() no-op branches, letting
drop_slow_inner and try_call_finalizer skip the VM attach entirely for
finished objects.
Standalone microbenchmarks (median of 5 interleaved runs):
- create+drain a 3-yield generator, 2M times: 1.099s -> 0.997s (-9%)
- pyperformance bm_generators shape (yield-from tree traversal): -4%
- pyperformance bm_coroutines shape (recursive fibonacci coroutines): -13%
Assisted-by: Claude Code:claude-fable-5-1
* _asyncio: skip full __import__ dispatch on module-cache hot path
with_asyncio_cache/with_contextvars_cache called vm.import("_asyncio"/
"contextvars", 0) on every Task/Future registration just to validate
their module-identity cache, even on the (overwhelming) hit path. That
still pays the full __import__ builtin dispatch (attribute lookup on
builtins, FuncArgs construction, native slot_call, ImportArgs parsing,
import_module_level) instead of a cheap sys.modules lookup. Added
fast_import_cached(), which checks sys.modules directly and only falls
back to vm.import on a miss (module missing/reloaded).
Measured with a standalone asyncio_tree-style microbenchmark
(6 levels x 6 branches, several asyncio.run() iterations per process,
median of 5 interleaved A/B rounds):
none: 1.045s -> 0.895s (~14% faster)
eager: 0.716s -> 0.591s (~17% faster)
eager_tg: 1.023s -> 0.849s (~17% faster)
test_asyncio (all files), test_contextvars (module absent from this
checkout, pre-existing), test_weakref, test_weakset all pass/unchanged.
Assisted-by: Claude Code:claude-fable-5-1
* vm: skip VM-attach section in with_vm when already on the current interpreter
Object teardown (drop_slow_inner -> call_slot_del, try_call_finalizer,
weakref callbacks) calls with_vm(), which unconditionally went through
set_current_vm -> begin_interpreter_section: an INTERP_THREAD_SLOTS
HashMap lookup plus an Arc clone, even in the overwhelmingly common
case where the dropping thread is already attached to the exact
interpreter that owns the object (a __del__ object or weakref target
going out of scope mid-bytecode).
with_vm now checks whether the object's owning interpreter is already
the one on top of VM_STACK. Only the top of VM_STACK is ever ATTACHED
on a thread (every push is preceded by an attach, every pop restores
the enclosing attach), so when it matches we call f(vm) directly and
skip begin_interpreter_section/set_current_vm entirely. The rare
not-attached/different-interpreter cases (nested subinterpreters,
thread outside any VM, shutdown) fall through to the original
attach/detach path unchanged.
Interleaved release-build measurements (5 rounds, medians), plain-Rust
micro-scripts, no pyperf:
- 2M create+drop of a __del__ object: ~0.55s -> ~0.46s (~16% faster)
- 2M create+drop with a weakref callback: ~1.14s -> ~1.06s (~7% faster)
- 2M io.StringIO() create+drop: ~0.734s -> ~0.697s (~5% faster)
- fib(30) and a 5M-iteration tuple-unpack loop: unchanged (noise-level)
`sample` profiles of the __del__ benchmark confirm
begin_interpreter_section/ensure_thread_slot frames disappear after
the change.
Verified: `cargo test -p rustpython-vm --release`, and
`-m test -j4 test_gc test_weakref test_finalization test_threading
test_thread test_concurrent_futures test_os test_signal test_generators
test_io test_atexit test_sys` (test_interpreters module absent in this
tree, pre-existing). Also a 4-thread stress script hammering __del__
object creation against a concurrent gc.collect() loop for 10s, and a
threading+queue handoff dropping a __del__ object after cross-thread
transfer, both pass.
Assisted-by: Claude Code:claude-fable-5-1
* vm: general sys.modules fast path for vm.import
VirtualMachine::import always resolved and called builtins.__import__
(FuncArgs construction, native slot_call, ImportArgs::from_args, then
import_module_level) even when the module was already cached in
sys.modules. ~63 Rust stdlib call sites hit this on hot paths
(copyreg, pickle's reduce_newobj, warnings, asyncio, etc).
Add VirtualMachine::try_import_cached: for a plain absolute import
(level 0, empty from-list), if builtins.__import__ is still the
original import function (compared by identity against vm.import_func)
and the module -- or, for a dotted name, its top-level package -- is
already present and fully initialized in sys.modules, return it
directly, skipping the __import__ dispatch and import_module_level
entirely. Falls through to the existing slow path whenever the module
is uncached, still initializing, or __import__ has been overridden by
user code, so test_import/test_importlib/test_builtin overrides still
work.
Exposed import::is_module_initializing (refactored out of
import_ensure_initialized) so the fast path can check
__spec__._initializing the same way the slow path does.
Removes _asyncio's local fast_import_cached workaround (added in
cb2064f0e) now that vm.import itself has the same fast path generally.
Measured (interleaved A/B, 5 rounds, pickle.dumps of a plain object,
which hits copyreg import via reduce_newobj): ~2% wall-clock
improvement per call (baseline ~4.75s / 100k dumps vs ~4.66s
candidate), consistent across all rounds. Verified: overridden
__import__ still invoked, a replaced sys.modules is read through the
same accessor, and a self-importing/initializing module is not handed
out early -- all match baseline behavior.
test suite (test_import test_importlib test_builtin test_pkg
test_pkgutil test_runpy test_zipimport test_site test_warnings
test_copy test_pickle test_asyncio.test_tasks test_threaded_import
test_sys): 13/14 files pass; test_threaded_import fails identically on
baseline (ModuleNotFoundError, pre-existing, unrelated to this change).
Assisted-by: Claude Code:claude-fable-5-1
* _asyncio: bypass generic context.run dispatch in task_eager_start
task_eager_start ran the first step via context.run(coro.send, None),
which builds a bound `send` method object and then re-enters the
generic Callable/FuncArgs dispatch just to invoke context.run's own
generic call. Since eager tasks in async_tree_eager complete almost
entirely synchronously (no event-loop scheduling needed), this
per-task dispatch overhead dominated. Enter/exit the native PyContext
directly and call coro.send via a single call_method instead,
falling back to the generic path for non-native Context subclasses.
async_tree_eager median: ~705ms -> ~697ms across 5 interleaved A/B
rounds (~1% eager-only win); async_tree (none) unaffected (noise-level
delta). test_asyncio (all files), test_weakref, test_weakset pass;
pre-existing test_contextvars module-not-found failure unchanged from
baseline.
Assisted-by: Claude Code:claude-fable-5-1
* vm: drop scopeguard::defer! from generator/coroutine resume hot path
resume_gen_frame restored owner/exc-slot/frame-chain state via a
scopeguard::defer! closure that ran on every exit (Ok, Err, or panic).
Since the closure's body is the function's only tail expression, a
plain captured result plus explicit cleanup (the same shape with_frame
already uses for the analogous call path) covers both normal exit
paths identically; only the recursion-depth decrement keeps a
scopeguard, matching with_frame's precedent that this codebase does
not otherwise guarantee generator/frame state consistency across a
Rust panic (panic=unwind is not used to implement any Python-level
control flow here).
Delegation-chain benchmark (chain.py, forwarding N values through a
yield-from chain) measures a small but consistent win: ~2-5% less
wall time per forwarded value at depths 1/4/16 across interleaved A/B
rounds. Single-generator create+finish cost is a wash.
Assisted-by: Claude Code:claude-fable-5-1
* vm: run generator bodies through the call trampoline
`Py<FrameObject>::resume` built its `ExecutingFrame` with
`tailcall_enabled: false`, because a `TailCall` result had nowhere to go:
the trampoline was reachable only from `run_frame_fast`, whose entry frame
lives on the data stack and can never yield. So every ordinary Python call
made from inside a generator or coroutine body re-entered `run_frame_fast`
recursively instead of being flattened.
Generalize the trampoline to drive a heap-resident entry frame whose
enter/exit bookkeeping belongs to its caller, and whose `Yield` is a valid
terminal result rather than a panic. `FrameKind` now says, per frame, what
the trampoline owes it: `Entry` (exit it, caller frees the storage),
`GenEntry` (`resume_gen_frame` linked it and will unlink it; a `Yield` is
the trampoline's result) or `Callee` (exit it and free its storage). The
three near-identical dispatch arms collapse into one `Step::Finished` /
`Step::Deliver` loop.
The suspended-frame stack moves onto the VM and is reused across
invocations: a generator body enters the trampoline once per resume, so a
`Vec::with_capacity(8)` there cost a malloc/free per yield (+27 ns/resume
when first measured). Invocations nest strictly LIFO, so each owns the
region above the length it finds on entry.
Neutral on its own — a generator body's first call was the only one not
already flattened, since the callee's own frame runs under the trampoline
either way. It is what lets SEND/FOR_ITER hand a generator frame to the
same loop. Measured (chain of `yield from`, 1-yield generator create+finish,
bm_generators, bm_coroutines, async_tree none/eager): within noise, except
generator create+finish 625 ns -> 603 ns.
Assisted-by: Claude Code:claude-fable-5-1
* vm: run a `yield from` chain in one eval loop
`SEND` on a suspended generator recursed: `Coro::send` -> `resume_gen_frame`
-> a fresh `ExecutingFrame::run` per level, and the value came back the same
way. Measured at ~55 ns per level of delegation, against CPython's ~5.8 ns.
Only about 5 ns of that is the resume bookkeeping itself (measured by
stubbing out the claim, the exception slot, the owner swap and the recursion
checks). The rest is what a delegating frame does per value: leave and
re-enter the eval loop, and dispatch `RESUME`, `JUMP_BACKWARD_NO_INTERRUPT`,
`SEND` and `YIELD_VALUE` — four instructions that, in the canonical
`yield from` / `await` shape, only hand a value along.
So the trampoline now runs the chain itself. `SendGen` parks its frame and
returns `ExecutionResult::GenResume`; the trampoline claims and links the
sub-generator (`coroutine::flat_resume_enter`, the first half of what
`Coro::send` does around a resume) and runs its frame in the same loop. On
the way down, every frame it finds already suspended at a `yield from` whose
delegate can be resumed is parked without being run at all
(`yield_from_delegate`); on the way back up, a yielded value is re-yielded
for each such frame by parking its `lasti` past the `YIELD_VALUE`
(`park_after_yield_from`) instead of running it. A whole chain therefore
costs one claim/link/unlink per level and no instruction dispatch at all,
with `Py<FrameObject>::resume` extending the same treatment to the outermost
level, which is where `next()` and asyncio's task step enter a chain.
Each level is still claimed, linked and unlinked exactly as a recursive
`Coro::send` would, so `gi_running`, `f_back`, `gi_frame`, `gi_yieldfrom`,
`sys._getframe`, tracebacks, `throw`, `close`, PEP 479 and the
`RecursionError` depth are unchanged; `resume_gen_frame` and the flat path
share `gen_frame_link`/`gen_frame_unlink` and the invariants are documented
on `FlatResume`. Everything else keeps the recursive path: tracing or
monitoring on (both would see the skipped instructions), `throw`/`close`,
async generators, subclasses, non-generator iterators, `FOR_ITER`, a `SEND`
that is not a `yield from`, and a delegate that has not started or is not
itself delegating — the last two because one lone level costs more to hand
to the trampoline than to send into from the eval loop the caller is in.
Measured (median of 5, ns/value or seconds):
chain of `yield from`, depth 16 911 -> 479 -47%
chain of `yield from`, depth 4 212 -> 158 -26%
bm_generators 0.272 -> 0.187 -31%
richards 0.772 -> 0.771 ~0
bm_coroutines 0.0129 -> 0.0130 +0.8%
async_tree none ~neutral
chain depth 1 (no delegation) 81.1 -> 83.6 +3.1%
fib(24), calls through trampoline 0.0180 -> 0.0183 +1.7%
Per level of delegation: 55.3 ns -> 26.4 ns (CPython 3.14: 5.8 ns).
Assisted-by: Claude Code:claude-fable-5-1
* vm: reset the oparg prefix when a signal unwinds a frame
An `EXTENDED_ARG` leaves its prefix in `arg_state` for the instruction that
follows it. The signal path in the eval loop unwinds to a handler and
`continue`s straight back to the top, skipping the loop's own
`arg_state.reset()`, so the handler's first instruction was decoded with the
interrupted instruction's prefix still applied. Every other path that jumps
somewhere new — the `set_f_lineno` restart just above — resets first.
Reachable whenever a signal arrives on the instruction right after an
`EXTENDED_ARG` in a frame that has a handler: a `KeyboardInterrupt` into a
`yield from` chain panicked with `InvalidBytecode` in roughly half of the
runs of a 16-deep chain, and now does not in ten out of ten.
Assisted-by: Claude Code:claude-fable-5-1
* bytecode: fetch a code unit with one 16-bit acquire load
The dispatch loop read the opcode with an Acquire `AtomicU8` load and the
arg byte with a separate Relaxed one. The acquire load sits between the
two, so LLVM had to re-load `self.code` and `code.instructions` for the
second access: four extra instructions and two extra dependent loads on
the instruction-fetch path of every bytecode instruction.
`CodeUnits::read_unit` reads both bytes as a single aligned Acquire
`AtomicU16`, which pairs with `replace_op`'s Release store exactly as
`read_op` did. `CodeUnit` gains `align(2)` so the 16-bit access is
guaranteed aligned rather than relying on the allocator.
5.91 -> 5.76 ns/bytecode-instruction on a straight-line
LOAD_FAST/STORE_FAST loop (-2.5%); range loop -2%.
Assisted-by: Claude Code:claude-fable-5-1
* vm: compute the cache-entry count after dispatch, not before
`op.cache_entries()` is a byte-table lookup used only to skip inline
cache units after an instruction falls through. Computing it before the
handler kept it live across the whole (very large) inlined match, so
every bytecode instruction paid an address materialization, the table
load, a stack spill and a reload of the count.
Moving it next to its only use lets it be computed on the fall-through
path with the value in a register.
Measured together with the following commit (the eval-breaker outline):
5.76 -> 5.55 ns/bytecode-instruction on a straight-line
LOAD_FAST/STORE_FAST loop, range loop -1.5%, fib(27) -2.5%.
Assisted-by: Claude Code:claude-fable-5-1
* vm: move the eval-breaker body out of the dispatch loop
The eval-breaker check itself stays inline on the per-instruction path
(moving it off regresses badly), but everything it guards -- the
finalizing/main-thread check, stop-the-world suspend, the QSBR
checkpoint, signal delivery and the scheduled GC -- was inlined into the
loop body too. That code is never executed on the hot path yet its
address materializations and live values competed for registers with
every opcode handler, so the loop head spilled more than it needed to.
`eval_breaker_work` is `#[cold] #[inline(never)]`, leaving a single
`ldrb`/`cbz` pair inline for the common "nothing pending" case.
Measured with the preceding commit: 5.76 -> 5.55 ns/bytecode-instruction
on a straight-line LOAD_FAST/STORE_FAST loop.
Assisted-by: Claude Code:claude-fable-5-1
* vm: always inline the eval stack push/pop primitives
`pop_value`, `pop_value_opt`, `top_value` and friends are a handful of
instructions each, but `ExecutingFrame::run` is ~50 KB of inlined opcode
handlers, so LLVM's inline budget refused them and emitted real calls:
STORE_FAST and POP_TOP each began with `bl ...pop_value`, which also
forced spills around the call. `#[inline(always)]` puts the four or five
instructions they actually are back into the handlers.
5.55 -> 5.29 ns/bytecode-instruction on a straight-line
LOAD_FAST/STORE_FAST loop (-4.8%); nbody -3.5%, fib(27) -3.5%,
`s += i` loop -3.5%, `for _ in range(n)` -4%.
Assisted-by: Claude Code:claude-fable-5-1
* vm: carry the instruction fetch state in loop locals
Instruction fetch went through memory three times per bytecode
instruction: `self.lasti` (a pointer field) was re-loaded, the program
counter was stored to it and read back at the top of the next iteration,
and `self.code` -> `code.instructions` was re-loaded to address the code
unit. LLVM cannot keep any of them in registers because the handlers
write through `&mut self`, so the fetch chain started with a
store-to-load round trip followed by two dependent pointer loads.
`lasti_cell` and `instructions` are now copied into locals once (both are
plain shared references with the frame's lifetime), and the program
counter is carried in `idx`: after a handler that did not jump the next
index is already known from `lasti_before` plus the cache count. The
frame's `lasti` is still stored before every instruction, so tracebacks,
`f_lineno`, signals and `sys._getframe` see exactly what they did; only
the read-back is gone. The paths that unwind (signal, exception handler,
`set_f_lineno`) re-read it, since they jump to a new position.
Also collapses the eval-breaker `if`, which clippy flagged after the
previous commit moved its body out of line.
range loop -3.3%, richards -3.1%, nbody -2.9%, `s += i` loop -2.2%,
go -1.7%, raytrace -1.2%; mean -1.1% over the 13-benchmark set. The
straight-line LOAD_FAST/STORE_FAST micro is unchanged at 5.28
ns/bytecode-instruction (it has no cache entries and never jumps, so it
was already the best case for the old code).
Assisted-by: Claude Code:claude-fable-5-1
* bytecode: hoist the instruction array base out of the dispatch loop
Addressing a code unit still went `self.code` -> `code.instructions` ->
the `Box<[CodeUnit]>` data pointer on every instruction, two dependent
loads before the acquire load that fetches the unit itself, because the
handlers write through `&mut self` and LLVM has to assume the fields
could change.
The instruction array is allocated once when the code object is built
and never reallocated -- specialization rewrites units in place -- so
`CodeUnits::units_ptr` may be read once before the loop and reused.
`read_unit_from` is the same 16-bit acquire load taken from that base;
`read_unit` now delegates to it, and the index is unchecked exactly as
it was before.
nqueens -1.9%, raytrace -1.6%, range loop -1.3%, deltablue -1.1%,
nbody -0.8%, dispatch micro -0.7%; mean -0.7% over the 13-benchmark set.
Assisted-by: Claude Code:claude-fable-5-1
* common: an immortal bit in the refcount word
Atomic reference counting is about a third of what a bytecode costs now
that the dispatch loop is trimmed: built `--no-default-features`, where
`PyAtomic` is a `Cell`, the straight-line dispatch micro runs at
3.5 ns/instruction against 5.4. Every `TO_BOOL`/`POP_JUMP_IF_*` pair and
every `None` or small-int push pays a relaxed `ldadd` and a `Release`
`ldaddl` on an object that can never die.
PEP 683's answer is to say so in the word. A fourth flag bit, `IMMORTAL`,
takes a bit from the strong count (leaving 60 on a 64-bit target, 28 on a
32-bit one, both far past what a program reaches); `make_immortal` sets it
and parks the count at CPython's `_Py_IMMORTAL_REFCNT`, and `inc`,
`inc_by`, `dec` and `safe_inc` read the bit and return without touching
the counter. The parked count is what keeps the rest of the tree honest
without a special case: it is far too large for a `strong_count() == 1`
in-place-mutation fast path to fire on, and `start_gc_refs` already reads
a count that large as `GC_REACHABLE`, so an immortal object is a permanent
root that keeps its referents alive.
Two details pay for the branch the mortal path now runs, which cost 4% on
object-heavy benchmarks before they went in:
* `leak` sets `IMMORTAL` alongside `LEAKED`. An interned object could
already never reach zero, so `dec` no longer needs the test it made on
the result of the decrement: the immortal test at the top decides both.
* `inc`'s three exceptional cases — a destructed word, a count at the
ceiling, a count at zero — fold into one comparison against the masked
word, with the handling moved to a cold helper.
What is left is one instruction on `dec` and roughly nothing on `inc`, and
the mortal path measures at parity.
Assisted-by: Claude Code:claude-fable-5-1
* vm: make the singletons, small ints, interned strings and static types immortal
Everything the `Context` builds once and keeps — `None`, `True`, `False`,
`Ellipsis`, `NotImplemented`, the empty tuple, frozenset, bytes and str,
the -5..256 int cache and the interned one-character strings — is now
marked immortal, as is every static type object, which its `static_cell`
holds for the life of the process. So is every entry of the string pool:
`mark_intern` already stopped those refcounts from reaching zero, so
immortality frees no memory that would otherwise have been freed; it only
takes the atomic read-modify-write off every incref and decref of an
attribute name, a dict key or a docstring.
Straight-line dispatch micro (200 LOAD_FAST/STORE_FAST pairs over cached
ints, median of 5 interleaved runs): 5.39 -> 4.01 ns/instruction, -26%.
A four-thread `if flag:` loop, where the contention on the shared bool was
real: 2.36 s -> 2.14 s, -10%.
pyperformance bench functions run standalone, medians of 5 interleaved
rounds against a matched build: fannkuch -7.8%, nqueens -3.8%, chaos
-3.0%, raytrace +1.0%, go +1.2%, richards +4.2%, nine of them together
-2.9%. Per-benchmark deltas under about 4% sit inside this tree's
build-to-build ThinLTO layout noise, which reaches that size between two
builds of identical source.
Observable: `sys.getrefcount` of an immortal object reports 4294967295,
CPython's `_Py_IMMORTAL_REFCNT`, and stops moving — what
`test_sys.test_refcount` expects of `None`. `is_interned`, and the dict
pointer-equality key fast path behind it, keep answering from the separate
`LEAKED` bit.
Assisted-by: Claude Code:claude-fable-5-1
* codecs: fast-path utf-8 decode, skipping the codec registry round trip
bytes.decode('utf-8') / str(bytes, 'utf-8', ...) always went through the
generic codec registry: a per-call attribute lookup for
_is_text_encoding, then a full call into the pure-Python
encodings.utf_8.decode wrapper, which called back into the native
utf_8_decode and unpacked a 2-tuple result.
CPython's PyUnicode_Decode special-cases a handful of built-in encoding
names (utf-8 among them) and decodes them directly, bypassing the codec
registry entirely -- a codecs.register() override does not affect
str(b"...", "utf-8") there either. This adds the same fast path here:
recognized utf-8 aliases go straight to the native decoder.
Measured with a pure-Python `str(bytes, 'utf-8', 'surrogatepass')` loop:
~723ns -> ~401ns per call (1.8x). On the pyperformance unpickle
benchmark (which decodes short strings this way for every
SHORT_BINUNICODE opcode): ~14.4ms -> ~13.2ms per loop, an ~8.3% win.
Assisted-by: Claude Code:claude-fable-5-1
* codecs: fast-path utf-8/ascii/latin-1 encode and ascii/latin-1 decode
Extend the utf-8 decode fast path from 3fe684629 to the symmetric encode
direction (str.encode()/bytes(s, enc)), and add the same registry-skipping
fast path for ascii and latin-1 encode/decode, all of which are simple
native codecs.rs functions already. For utf-8 encode with no surrogates in
the string (the common case), encoding is just a copy of the string's
existing UTF-8 bytes, so that's the fastest path; strings containing
surrogates and non-strict error handlers still go through the native
encoder + ErrorsHandler, matching custom codecs.register_error handlers.
Measured (median of interleaved runs vs baseline, 2M iterations):
str.encode("utf-8"): 351ns -> 195ns (-44%)
str.encode(): 449ns -> 291ns (-35%)
bytes(s, "utf-8"): 542ns -> 382ns (-30%)
str.encode("ascii"): 436ns -> 302ns (-31%)
str.encode("latin-1"): 487ns -> 333ns (-32%)
bytes.decode("ascii"): 519ns -> 359ns (-31%)
bytes.decode("latin-1"):590ns -> 417ns (-29%)
bytes.decode("utf-8")/str(b,"utf-8"): unchanged (already fast-pathed)
Assisted-by: Claude Code:claude-fable-5-1
* vm: make the runtime safe for borrowed stack entries
`PyStackRef` has carried a borrow tag since the stack was unified with the
fastlocals, but only `LOAD_SMALL_INT` -- which borrows immortal `Context`
objects -- ever set it, so nothing in the runtime had to be right about
borrowed entries. Turning `LOAD_FAST_BORROW` into a real borrow makes three
places load-bearing, and this fixes them ahead of the switch.
`Traverse for PyStackRef` handed every entry to the tracer, borrowed ones
included. The cycle collector subtracts one from an object's `gc_refs` per
edge it is shown, so a borrowed entry -- which owns no strong count -- would
have pushed a live object's count to zero and freed it out from under the
frame. Borrowed entries are now skipped; nothing becomes unreachable, because
the fastlocals slot the entry borrows from is on the same traversal.
`park_after_yield_from` stands in for a `YIELD_VALUE` that never runs, so the
`promote_stack()` in that handler never ran either; it now promotes there too,
and the debug assertion after it stops being vacuous.
`STORE_FAST`, `DELETE_FAST` and the fused stores now call
`LocalsPlus::debug_audit_local_release`, which in debug builds panics if the
slot is about to drop the last count of an object a borrowed entry still
points at -- the one shape codegen's `optimize_load_fast` exists to prevent,
and the one that would be a use-after-free if the analysis or a modelled stack
effect were wrong.
`BORROW_LOCAL_LOADS` (off) is the switch for the borrow itself, and
`LOAD_FAST_BORROW`/`LOAD_FAST_BORROW_LOAD_FAST_BORROW` now go through
`push_local`, which is a plain clone while it is off. The invariant the switch
depends on is written out where `PyStackRef` is defined.
No measurable change: with the switch off the only behavioral difference is
the traversal fix, which no current bytecode can reach.
Assisted-by: Claude Code:claude-fable-5-1
* vm: read consumed stack entries in place instead of owning them
`pop_value()` turns whatever is on the stack into an owned `PyObjectRef`, so
an instruction that only reads its operand still pays a reference count round
trip -- and, once `LOAD_FAST_BORROW` starts pushing borrows, would pay an
atomic increment to promote a borrow it is about to throw away. This moves
every consumer that never needs ownership onto `pop_stackref()`, which hands
back the tagged entry and costs nothing for a borrow.
Discarding pops (62 of them, across the `LOAD_ATTR`, `TO_BOOL`, `BINARY_OP`,
`COMPARE_OP`, `CONTAINS_OP` and `CALL` specializations, which read their
operands through `top_value()`/`nth_value()` and then drop them) become
`pop_stackref()`. So do the read-only operands of `POP_TOP`, `POP_JUMP_IF_*`,
`TO_BOOL`, `BINARY_OP`, `COMPARE_OP`, `CONTAINS_OP`, `IS_OP`, `STORE_ATTR`,
`DELETE_ATTR`, `STORE_SUBSCR`, `DELETE_SUBSCR` and `UNPACK_SEQUENCE`.
Two protocol entry points had to move from an owned receiver to `&PyObject` to
make that reachable: `try_to_bool`, which never used the ownership it took,
and `rich_compare`, which forwards to the `&self` `_cmp` anyway. Both keep
working unchanged for `PyObjectRef` callers through deref.
`LOAD_ATTR`'s method specializations popped the receiver and pushed it
straight back below the bound function; they now move the entry across with
its tag intact, which is how CPython keeps a method call's receiver borrowed
from `LOAD_ATTR` all the way through `CALL`. The plain attribute read reads
through the receiver rather than owning it.
Neutral while `BORROW_LOCAL_LOADS` is off -- for an owned entry
`pop_stackref()` and `pop_value()` compile to the same thing.
Assisted-by: Claude Code:claude-fable-5-1
* vm: turn LOAD_FAST_BORROW into an actual borrow
`LOAD_FAST_BORROW` accounts for 21.5% of executed instructions on
nbody+richards+nqueens, and until now it did exactly what `LOAD_FAST` does:
an atomic increment on the way onto the stack and, from the instruction that
consumed it, an atomic release decrement on the way off. Neither is needed.
The fastlocals slot holds a count for as long as the entry is on the stack --
codegen's `optimize_load_fast` only rewrites a `LOAD_FAST` when it can prove
the entry is consumed before anything can release that slot -- so the entry
can be a borrow that owns nothing, which is what CPython 3.14 does with a
tagged `_PyStackRef`.
Flipping `BORROW_LOCAL_LOADS` on is the whole change; the two preceding
commits are what make it safe (cycle-collector traversal, yield-point
promotion, a debug audit of the invariant) and what make it pay (consumers
that read through the entry rather than promoting it back to owned).
Medians of 7 interleaved rounds, this machine, against the pre-series binary:
straight-line while loop 6.32 -> 6.01 ns/instruction (-5.0%)
x = a + b loop -6.3% if flag: loop -3.8%
attribute-read loop -5.2% method-call loop -3.3%
subscript loop -6.7% fib(27) -0.8%
nbody -2.6% fannkuch -1.7% nqueens +1.3% richards -3.0%
deltablue -2.3% go -3.0% raytrace -1.7% float -3.2%
chaos -2.6% spectral_norm -2.5% mean -2.1%
About half of that is the consumer rewrite alone (mean -1.0% with the switch
still off) and the rest is the borrow. nqueens is the one benchmark that goes
the other way, by less than the roughly 4% of build-to-build layout noise this
tree has, and it moves with the consumer rewrite rather than the switch.
Full suite green (418 files, 42,110 tests). A build with debug assertions on
runs the suite with no audit failure, and trips the audit exactly when
`LOAD_FAST` is forced to borrow in a shape codegen refuses to convert
(`f(x, (x := None))`). Adversarial scripts -- swap loops and reassignment
under `__del__` tracking, argument evaluation that deletes its sibling,
`a[a.pop()]`, closures over a rebound cell, generators dropped while
suspended, tracebacks whose `f_locals` are read after unwinding, `settrace`
writing `f_locals` mid-expression, a method receiver whose local is cleared
while the call is being built, `__add__`/`__lt__`/`__bool__` that clear the
local their operand came from, threads sharing objects through locals, and
`gc.collect()` from a finalizer -- all agree with CPython 3.13 under
`gc.set_threshold(1, 1, 1)`, in both builds.
Assisted-by: Claude Code:claude-fable-5-1
* vm: exact-dict fast path for DICT_MERGE and CALL_FUNCTION_EX kwargs
`g(**d)` on an exact dict compiled to BUILD_MAP+DICT_MERGE+CALL_FUNCTION_EX
and paid for two Python-level `keys()`/`__getitem__` round trips per key at
both the DICT_MERGE handler and collect_ex_args' iterate_mapping_keys, even
though the source is a plain dict. Add an exact-dict (downcast + type
check, so subclasses still go through the mapping protocol) fast path to
both sites that iterates entries natively via PyDict::items_vec, matching
CPython's PyDict_Check fast paths in PyDict_Merge / CALL_FUNCTION_EX.
Measured (1M calls, release build): g(**d) 1320ns -> 665ns, f(**d)
1470ns -> 900ns, c.m(**d) 1690ns -> 1135ns, f(x=1,**d) 1600ns -> 1035ns,
f(**d1,**d2) 1700ns -> 1013ns. bm_django_template ~7% faster,
bm_argparse (subparsers) ~7% faster.
Assisted-by: Claude Code:claude-fable-5-1
* vm: skip the recursion guard hashing exact ints inside a tuple
tuple.__hash__ hashed every element through PyObject::hash, which wraps
each call in with_recursion's native-stack-depth check. That guard exists
for elements whose hash can recurse into arbitrary Python code, but
int.__hash__ is a leaf computation - it can neither recurse nor call back
into Python - so the check is pure overhead for the overwhelmingly common
case of tuples of plain ints (e.g. namedtuple-shaped dict keys).
pyperformance's mdp benchmark builds huge tables keyed by tuples of deeply
nested namedtuples whose leaves are ints; profiling with `sample` showed
PyObject::hash/with_recursion/tuple's slot_hash dominating the CPU time.
Hashing exact-int elements directly via hash::hash_bigint cuts mdp's
wall time from ~6.79s to ~4.81s (median of 5 interleaved runs), about 29%
faster, with no change to hash values or recursion-guard behavior for any
other element type.
Assisted-by: Claude Code:claude-fable-5-1
* vm: wire slot descriptors from __slots__ into C-level type slots
Classes that put a dunder name (e.g. "__setitem__") into __slots__ get
a per-instance member descriptor for it, which is a legitimate way to
override a special method per-instance (chameleon's tal.RepeatDict does
exactly this). PyType::new_verbose ran init_slots() to compute
mp_ass_subscript/mp_subscript/etc. before these member descriptors were
added to the type's attribute namespace, so the slot lookup never saw
them and __setitem__/__getitem__ silently fell back to "does not
support item assignment".
Fix by calling update_slot for each dunder-shaped __slots__ member
right after it is installed on the type, mirroring what
type.__setattr__ already does for regular attribute updates.
Assisted-by: Claude Code:claude-fable-5-1
* frame: stop allocating cold data to close a generator
`clear_locals_and_stack`, which runs whenever a generator or coroutine
comes to an end, reached for the frame's lazily-allocated `FrameColdData`
through `cold()`. That allocates it on first access, so every finishing
generator paid a `Box<FrameColdData>` malloc and free just to find
`f_extra_locals`, `f_locals_cache` and `f_overwritten_fast_locals` all
empty -- cold data is only ever populated by tracing and frame
introspection. Ask for it with `cold_opt()` instead and skip the three
reads when it was never allocated.
Medians of interleaved A/B runs, ns per generator create+finish over a
plain call: `for _ in g(): pass` 265 -> 231 (-13%), a coroutine that
completes synchronously 642 -> 617 (-4%), `sum(x for x in range(3))`
1033 -> 997 (-3.5%).
Assisted-by: Claude Code:claude-fable-5-1
* function: bind exact args straight into a generator's frame
Calling a generator, coroutine or async generator function took the
general argument-binding path even when the call site was the simplest
possible one. `invoke_exact_args_slots` and the vectorcall fast path both
recognised the shape -- every parameter positional, filled exactly once
-- and then handed generator-like code back to `invoke()`, which built a
`FuncArgs` (a `Vec` plus an `IndexMap`) only for
`fill_locals_from_args_inner` to walk varargs packing, keyword matching
and defaults that this shape cannot reach.
Add `make_generator_exact_args`, the counterpart of
`prepare_exact_args_frame` for the one call shape it refuses: it builds
the heap frame the generator will own, moves the arguments straight into
its fastlocals and returns the generator. Both fast paths now use it. The
vectorcall one matters on its own for generator expressions, whose fresh
`MAKE_FUNCTION` on every evaluation keeps them out of the call-site
specialization that would otherwise catch them.
Medians of interleaved A/B runs, ns per create+finish over a plain call:
`for _ in g(): pass` 231 -> 218 (-6%), a synchronously-completing
coroutine 616 -> 598 (-3%), `sum(x for x in range(3))` 1007 -> 967 (-4%).
Assisted-by: Claude Code:claude-fable-5-1
* frame: drain a dead frame in place instead of moving it out
Three allocations and a large move on the path a finishing generator
takes to release its frame.
`Traverse::clear` opened with `self.iframe.get_mut().take()`, moving the
whole `InterpreterFrame` -- a couple of hundred bytes -- out of its slot
and onto the stack, only to drop it there. Drain it through `as_mut()`
and empty the slot afterwards.
`LocalsPlus::clear_into` collected the evaluation stack into a `Vec` that
existed only to be extended into the caller's; push into that one
directly. And `clear_stack_and_cells` built (and dropped) a `Vec` for a
stack it had not looked at -- a frame that reached its `return`, which is
every generator that simply finished, left nothing there. Look first.
Medians of 5 interleaved rounds, ns per create+finish: `for _ in g():
pass` 307 -> 297 (-3.4%), a synchronously-completing coroutine 600 -> 585
(-2.5%), `list(g())` 565 -> 558; a plain call is unchanged.
Assisted-by: Claude Code:claude-fable-5-1
* vm: don't raise a TypeError to ask an iterator for its length
`length_hint_opt` opened with `iter.length()`, whose whole job on a type
with no length slot is to build the `TypeError` that says so -- which
this caller then catches, recognises and discards. Every iterator and
every generator takes that path, and `list()` calls it on whatever it is
handed, so a `list(gen)` paid for an exception object per call.
`length_opt` answers the same question with `None`. CPython's
`PyObject_LengthHint` gates the call on the slots for the same reason.
Median of 5 interleaved rounds: `list(g())` x200k for a 1-yield
generator 556 -> 407 ns (-27%). pyperformance generators, coroutines,
comprehensions and async_tree are unchanged, as are nqueens, richards
and chaos.
Assisted-by: Claude Code:claude-fable-5-1
* gc: track a generator and its frame in one pass
Making a generator tracks two objects that are born together and die
together -- the generator and the frame it owns -- and each one took the
gen0 list lock and a thread-local read of the running interpreter for
itself. Give the pair one entry point: `PyGenerator`, `PyCoroutine` and
`PyAsyncGen` opt out of tracking at allocation, the way frames already
do, and `make_generator_or_coro` links both into gen0 under a single
lock, with one owner read and one collection check.
Medians of 5 interleaved rounds: pyperformance coroutines -2.5%,
generators -0.9%; `for _ in g(): pass` over a plain call 205 -> 200 ns.
Assisted-by: Claude Code:claude-fable-5-1
* dict_inner: fast-path exact-int equality in key_eq
Mirrors CPython's lookdict: two exact `int` keys now compare by BigInt
value directly instead of going through the generic rich-compare
dispatch (identical_or_equal -> bool_eq -> rich_compare_bool -> __eq__
slot lookup). Falls back to bool_eq whenever either side isn't exactly
`int` (subclasses may override __eq__), so behavior is unchanged there.
Measured (min-of-6 rounds, 2M-iteration micros, this tree's build has
~4% layout noise so min is used as the noise floor):
- dict_int_lookup (`d[i]`, int keys): 70.9ns -> 62.0ns (-12.5%)
- dict_str_lookup / dict_tuple_lookup / set_str_contains: unchanged
(within noise), confirming no regression on non-int keys.
pyperformance sanity checks (single round each, base vs candidate):
bm_mdp median ~5027ms/loop vs ~5003ms/loop (no regression, hashing
isn't the dominant cost there); bm_nqueens, bm_go, bm_deltablue,
bm_hexiom, bm_chaos all within ~1% (noise).
Hash/equality values cross-checked bit-identical against CPython 3.14
for 0, 1, -1, True, False, 2**61-1, 2**61, -(2**63), 2**100, -(2**100),
and int/float/bool/Decimal/Fraction(1) hash-equality invariants.
Also tried an exact-type fast dispatch in PyObject::hash (skipping the
slots.hash vtable call + with_recursion for exact int/str) - it only
moved hash_small_int/hash_large_int by ~2%, under the 3% bar, so it was
not committed.
Tests: `rustpython -m test -j4 test_hash test_int test_long test_tuple
test_dict test_set test_dictviews test_ordered_dict test_collections
test_fractions test_numeric_tower test_float test_bool test_str
test_userdict test_weakref` all pass (1796 run, 72 skipped). `cargo
test -p rustpython-vm --release -p rustpython-common` passes. clippy
clean on the touched lines.
Assisted-by: Claude Code:claude-fable-5-1
* vm: let the modulus type handle a three-argument pow
CPython's ternary dispatch offers the operation to the third operand's type
whenever its slot differs from the two already tried, including the case where
neither operand had one. The port required both of those slots to exist, so
`pow(10, 2, Decimal(7))` never reached `Decimal` and raised TypeError.
Assisted-by: Claude Code:claude-fable-5-1
* vm: add a feature-gated opcode-pair histogram
Counts `(prev_op, op)` in the dispatch loop and dumps the nonzero pairs at
exit, so superinstruction and specialization candidates can be ranked from
what pyperformance-shaped code actually dispatches rather than guessed at.
Entirely behind the non-default `opcode-histogram` feature: the module is
`#[cfg]`-ed out and the loop's two-line hook with it, so a default build is
byte-identical to one without the patch.
Assisted-by: Claude Code:claude-fable-5-1
* vm: let boolean producers run the branch that follows them
`TO_BOOL*`, `COMPARE_OP*`, `CONTAINS_OP*` and `IS_OP` materialised a bool
only for the `POP_JUMP_IF_*` one dispatch later to pop and test it, and the
compiler's `NOT_TAKEN` marker cost a third dispatch on the fall-through edge.
Generalise the `COMPARE_OP_INT`-only jump fusion so every boolean producer
resolves the jump itself, and step over `NOT_TAKEN` from there and from the
plain `POP_JUMP_IF_*` handlers. Both back off to ordinary dispatch when the
successor is an instrumented opcode or while tracing, so `sys.monitoring`
branch events and `sys.settrace` opcode events are unaffected.
Dispatches over 13 standalone pyperformance benchmarks: -5.0% total
(richards -11.7%, unpickle_pure_python -8.9%, sympy -8.4%, hexiom -8.1%,
django_template -7.1%, go -6.1%; nbody unchanged). Times, best of 7
interleaved rounds: deltablue -4.0%, hexiom -3.5%, go -3.4%,
unpickle_pure_python -3.3%, richards -3.2%, raytrace -3.0%, nqueens -1.9%,
geomean -1.0%; a branch-heavy microbenchmark -4.7% against a straight-line
control at 0%.
Assisted-by: Claude Code:claude-fable-5-1
* Adapt yield-from resume to optional generator frames
Coro.frame is now PyAtomicRef<Option<FrameObject>>, so the flattened
resume path uses frame_opt/clear_except_code instead of a direct frame.
Assisted-by: Grok:4.6
* Fix clippy and cr_origin after try_to_bool/f_lineno changes
try_to_bool now takes &self, so drop the leftover clones. Derive
cr_origin lines from lasti instead of prev_line, and satisfy
-Dwarnings on the immortal-refcount and del_needed types.
Assisted-by: Grok:4.6
* Convert fd filenames without the os ToPyObject impl
host_env-free builds (sandbox check, wasm) do not compile os.rs, so
OsPathOrFd::filename cannot use Borrowed's ToPyObject. Use as_raw()
instead, which is the same conversion.
Assisted-by: Grok:4.6
* Gate fd path helpers on the host_env feature
WASM clippy builds rustpython-vm without host_env, so OsPathOrFd and
the fspath/fd helpers are unused and fail -Dwarnings.
Assisted-by: Grok:4.6
* Drop redundant pickle try_to_bool clones after rebase
Assisted-by: Grok:4.6
* Document why enumerate keeps a Counter beside BigInt
Assisted-by: Grok:4.6
* Move fd path helpers into ospath_fd under host_env
Assisted-by: Grok:4.6
* Group threading eval-breaker bits in one module
Assisted-by: Grok:4.6
* Make InterpreterFrame construction unsafe
run_iframe stays safe: a live frame already has valid identity pointers.
Assisted-by: Grok:4.6
* Move OsPathOrFd re-export to the top of ospath.rs
Assisted-by: Grok:4.6
---------
Co-authored-by: Lee Dogeon <dev.moreal@gmail.com>
|
||
|
Jeong, YunWon
|
013333e5f9 |
host_env: keep Windows if_nameindex names as WTF-8 (#8742)
* host_env: keep Windows if_nameindex names as WTF-8 ConvertInterfaceLuidToNameW writes UTF-16; from_utf16_lossy replaced unpaired surrogates with U+FFFD. Return Wtf8Buf from those units instead. Assisted-by: Claude * host_env: put if_nameindex wide test under cfg(test) Windows clippy denies tests_outside_test_module for cfg(all(test, windows)) modules. Assisted-by: Claude |
||
|
Jeong, YunWonandLee Dogeon
|
358f6a875a |
native _json, _pickle, pyexpat, and _elementtree (#8741)
* _json: add native make_encoder (c_make_encoder)
json.encoder fell back to the pure-Python _make_iterencode because _json
had no make_encoder, making json.dumps ~25x slower than CPython. Implement
the C encoder semantics natively: str/None/bool/int/float/list/tuple/dict
ordering, key coercion and skipkeys, sort_keys via the VM's fallible sort
(comparison errors propagate instead of panicking), circular reference
detection, indent handling, allow_nan, default() fallback with recursion
guard, and CPython's add_note annotations. Removes the now-passing
expectedFailure markers in test_json.
Assisted-by: Claude Code:claude-fable-5-1
* pyexpat: add UseForeignDTD, fix silently swallowed handler exceptions
genshi's XML template loader (bm_genshi) called xmlparser.UseForeignDTD(),
which pyexpat.rs didn't implement, raising AttributeError. Add it as an
accepted no-op: the pyexpat module here wraps the pure-Rust xml-rs crate
(not real libexpat), which has no hook for splicing in a foreign DTD.
Root-causing the resulting bug also uncovered why genshi's template
generate()/render() produced empty output instead of erroring: do_parse's
invoke_handler swallowed every exception raised by a Python callback
(handler.call(...).ok()), and genshi's expat callbacks fail immediately
by reading `parser.CurrentLineNumber`/`CurrentColumnNumber`, which
pyexpat.rs never exposed. The AttributeError was silently discarded,
so no start/end/text events ever reached genshi's queue.
Fix both: propagate handler exceptions out of Parse()/ParseFile() (matching
libexpat/CPython semantics), and add CurrentLineNumber/CurrentColumnNumber/
CurrentByteIndex as read-only attributes backed by xml-rs's per-event
TextPosition (byte index is derived from it and is exact for ASCII
documents). This also fixes test_minidom's
testExceptionOnSpacesInXMLNSValue, previously an expected failure because
the ValueError from an invalid xmlns value was swallowed the same way.
PositionTest in test_pyexpat.py checks byte-exact expat semantics (token
*start* position) that xml-rs's *current parse cursor* position can't
reproduce exactly, so that one test is now marked an expected failure.
Assisted-by: Claude Code:claude-fable-5-1
* pyexpat: support multi-chunk Parse()/ParseFile() feeds
bm_xml_etree failed with AssertionError: missing end tags from
TreeBuilder.close(), because xml.etree.ElementTree reads and feeds large
documents to XMLParser.feed() in ~64KB chunks (and xml.sax's
IncrementalParser and ElementTree's XMLPullParser do the same). Each
Parse()/ParseFile() call built a brand-new xml-rs EventReader from only
the bytes passed in that call, discarding all parser state between calls.
When a chunk boundary fell in the middle of an element, the fragment
parsed by itself was invalid/incomplete XML, so xml-rs silently errored
out (matching this module's existing "ignore stricter xml-rs errors"
behavior) and the corresponding start/end events were dropped entirely
-- not just delayed.
xml-rs has no API to resume a parser across separate Read sources, so
instead each call now appends the new chunk to a persistent buffer and
re-parses the buffer as a whole, replaying only the events at or beyond
the count already dispatched in previous calls (dispatched_events). This
also makes position tracking (CurrentLineNumber/Column/ByteIndex) correct
regardless of how the input was chunked. Trade-off: each Parse() call is
now O(size fed so far), so a document delivered in many small chunks is
parsed several times over; acceptable for correctness but noticeably
slower than a true incremental parser for pathological chunk sizes.
This also fixes several previously-xfailed tests that depend on
incremental feeding: test_xml_etree.py's XMLPullParserTest (test_events,
test_feed_while_iterating, test_simple_xml, test_simple_xml_chunk_22,
test_simple_xml_with_ns) and ElementTreeTest.test_parseliteral, plus
test_sax.py's ExpatReaderTest (test_expat_entityresolver_default,
test_expat_incremental, test_expat_incremental_reset) -- their
expectedFailure markers are removed. test_simple_xml_chunk_1/_chunk_5
(1- and 5-byte chunks) still fail and remain xfail: at that granularity
xml-rs's stricter-than-expat tokenizer errors on some intermediate
prefixes that libexpat tolerates.
Assisted-by: Claude Code:claude-fable-5-1
* test: drop stale expectedFailure markers fixed by the pyexpat changes
test_plistlib's four invalid-document tests and test_xmlrpc's
test_loads_unsupported pass now that pyexpat no longer swallows
handler exceptions and keeps parser state across chunked feeds.
Assisted-by: Claude Code:claude-fable-5-1
* pyexpat: drive a single background EventReader for true incremental Parse()
8802451f3 fixed multi-chunk Parse()/ParseFile() feeds by accumulating every
fed byte and re-parsing the whole buffer from scratch on each call, which is
O(total bytes) per chunk -- quadratic in document size. xml-rs's EventReader
has no API to pause/resume across separate Read sources, so instead one
EventReader is now driven to completion on a dedicated background thread fed
through a condvar-backed queue (FeedBuffer/FeedReader); Parse()/ParseFile()
push their chunk, wake the thread, and drain (on the calling thread only)
whatever events that unblocks, stopping once the thread reports it is
blocked waiting for more input. Handlers always run on the calling thread.
A handler exception stops dispatch without tearing down the stream, so the
next Parse() resumes where it left off, mirroring libexpat. CurrentByteIndex
is now computed incrementally per line (with an amortized per-query cursor,
needed since real documents are often a single very long line) instead of
scanning the whole buffer. ParserCreate() never spawns a thread; one is
spawned lazily by the first Parse()/ParseFile() call, and dropping the
parser always stops and joins it.
Fixes test_xml_etree{,_c}'s test_events_comment, previously xfailed because
the old whole-buffer replay lost top-level comment events across feeds.
Measured (M4 Pro, release build):
- Scaling (ElementTree.parse, newline-separated doc): new backend 1/4/16 MB
= 0.15/0.61/2.35s (linear); CPython 3.14 same file = 0.015/0.055/0.23s.
- pyperformance's bm_xml_etree doc (156KB, effectively one line): old
backend single etree.parse() = 1.99s, single iterparse() = 3.53s; new
backend = 0.025s / 0.031s (~80-110x). Full bench_parse/iterparse/generate/
process (30/10/10/1 reps) now finish in ~1.5s/0.6s/0.4s/0.3s instead of
timing out past pyperformance's 400s limit.
- 10k ParserCreate() (never fed / partially fed / fully fed and dropped):
0.32s, no residual native threads, no RSS growth.
test -j4 test_pyexpat test_xml_etree test_xml_etree_c test_sax test_minidom
test_xml_dom_minicompat test_pulldom test_plistlib test_xmlrpc test_html
test_htmlparser test_threading: all pass (1350 tests, 89 skipped).
Assisted-by: Claude Code:claude-fable-5-1
* pyexpat: add a Sync fallback backend for targets without threads
2f58e9531 made pyexpat always drive its EventReader on a background
thread (ParserStream::spawn), panicking via .expect() if the thread
couldn't be spawned. pyexpat is registered unconditionally, including
on wasm32, where std::thread::spawn always fails at runtime, so the
first Parse() call panicked there.
Introduce a Backend enum: Backend::Threaded (the existing ParserStream)
on targets that can spawn threads, and a new Backend::Sync, the
accumulate-and-reparse implementation from 8802451f3, used on wasm32
and as a runtime fallback if spawning a thread ever fails. Both share
LineTracker for identical CurrentLineNumber/CurrentColumnNumber/
CurrentByteIndex and handler/error semantics.
Fixed two correctness bugs while porting the Sync backend forward:
marking the parser permanently "finished" as soon as a from-scratch
reparse happened to reach EndDocument (which can be a false positive
on a not-yet-complete buffer, e.g. right before epilog content
arrives) rather than only on the caller's real isfinal signal; and
counting that same EndDocument towards the dispatched-event index,
which could make a real event land on an index a later, longer reparse
had already skipped past. Verified by force-selecting Backend::Sync
under a test-only env var (removed) and diffing its output against
Backend::Threaded for the xml/sax/etree/plistlib/xmlrpc test suites
and a 2MB document fed in 1KB chunks.
Assisted-by: Claude Code:claude-fable-5-1
* _pickle: native Pickler, Unpickler and PickleBuffer
Adds `crates/stdlib/src/pickle.rs`, a native `_pickle` implementing
Pickler, Unpickler, PickleBuffer, PicklerMemoProxy/UnpicklerMemoProxy,
dump/dumps/load/loads and the PickleError family, so `Lib/pickle.py`
stops falling back to `_Pickler`/`_Unpickler`.
Protocols 0-5 are supported and produce byte-identical output to the
pure-Python pickler, including framing for protocol >= 4, memoization,
`fast` mode, dispatch_table, reducer_override, persistent_id /
persistent_load, find_class with `_compat_pickle` fix_imports, buffer
callbacks and out-of-band protocol 5 buffers. This also turns on
`pickle.PickleBuffer`, which RustPython did not have before.
Speedups over the pure-Python implementation (best of 5, us/op):
pickle 835 -> 21 (39x), pickle_list 525 -> 4.4 (119x),
pickle_dict 4018 -> 32 (125x), unpickle 630 -> 24 (26x),
unpickle_list 317 -> 5.0 (63x). CPython 3.14's C accelerator runs the
same workloads 1.5-2.9x faster than this.
test_pickle now runs 1008 tests (was 470) since the C-implementation
test classes are enabled; test_pickle, test_pickletools,
test_picklebuffer are green and 22 obsolete expectedFailure markers are
removed.
Assisted-by: Claude Code:claude-fable-5-1
* _pickle: raise the pickle.find_class audit event
Unpickler.find_class now raises `sys.audit('pickle.find_class', module,
name)` like CPython, so audit hooks can veto global lookups
(test_audit.test_pickle). Also drops four expectedFailure markers that
the native module makes pass: test_pydoc's builtin-method description
tests, test_memoryview.test_picklebuffer_reference_loop and
test_pyclbr.test_others.
Full suite: 456 test files / 42,648 tests green.
Assisted-by: Claude Code:claude-fable-5-1
* _pickle: avoid temporary buffers on the save fast paths
Opcode plus payload is now appended straight to the output buffer
(`write_pair`) instead of building a `Vec` per int, str, bytes and memo
opcode, `save_tuple` no longer copies the element slice, and the
`when serializing ...` note text is only formatted when an error is
actually raised.
bm_pickle "pickle" workload: 21.4 -> 17.0 us/op.
Assisted-by: Claude Code:claude-fable-5-1
* pyexpat: make the threaded backend fork-safe
fork() only keeps the calling thread. test_platform's
test_mac_ver_with_fork crashed the child with SIGTRAP: plistlib parses
a plist through pyexpat (spawning a background parser thread) before
os.fork(), then parses again in the child. Bisecting away every other
explanation (a plain threading.Thread survives the same fork fine,
since it goes through this VM's own fork-safe thread bookkeeping)
showed the crash is not from touching the *inherited* ParserStream
(already joined by refcounting before the fork in this repro) but from
spawning a *brand new* parser thread from the child, which reliably
crashed during the new thread's own startup.
Register a pthread_atfork child hook (lazily, the first time a parser
thread is spawned) that sets a permanent, process-wide flag the
instant this process is ever the child of a fork(). Once set:
- ParserStream::try_spawn refuses to spawn any further thread and
Backend::new falls back to Backend::Sync instead, exactly as on
targets that cannot spawn threads at all.
- Feeding an inherited Backend::Threaded raises a clear ExpatError
instead of touching its FeedBuffer/channel (whose mutex/condvar may
look held by a thread that no longer exists), and dropping it leaks
its JoinHandle/buffer/channel via ManuallyDrop rather than joining or
destroying them, since even destroying a POSIX mutex that looks held
is undefined behavior.
The parent is unaffected: the pthread_atfork hook only runs in the
child. Verified with a 200-iteration fork stress test (fresh parsers
and parsers forked mid-parse) and the full test_platform/xml/fork/
subprocess/multiprocessing_fork test modules.
Assisted-by: Claude Code:claude-fable-5-1
* pyexpat: implement name interning via self.intern, matching libexpat
self.intern (an always-present dict by default) was stored but never
consulted; every StartElement/EndElement/attribute name allocated a
fresh PyStr even when the same tag or attribute name repeated across
many sibling events. Wire it up the way libexpat does
(PyDict_SetDefault(self->intern, name, name)): element and attribute
*names* are now memoized so a repeated name reuses the same PyStr
object (and its cached hash) instead of reallocating and rehashing.
Attribute values are left out of the cache, same as libexpat.
This fixes test_pyexpat's previously-xfailed InterningTest.test, which
asserts object identity across repeated StartElementHandler/
EndElementHandler calls.
Measured on the pyperformance xml_etree parse/iterparse/generate/
process variants: no measurable wall-clock change (the dominant cost
there is Python-level TreeBuilder/Element bytecode execution and
thread hand-off, not name allocation), so this is a correctness/
feature-completion fix rather than a throughput win for that
benchmark; kept because it is a real, previously-inert compatibility
gap and is otherwise free.
Assisted-by: Claude Code:claude-fable-5-1
* _elementtree: native Element, SubElement and ParseError
Adds crates/stdlib/src/elementtree.rs providing the accelerator names
xml/etree/ElementTree.py picks up at import time. Element carries the
full API (tag/text/tail/attrib, indexing and slicing, find*/iter/
itertext, copy/deepcopy/pickle state) with CPython's lazily created
attrib dict and deferred text join, is subclassable, weak-referenceable
and GC-traversable, and drops every displaced object after releasing its
lock so a __del__ can re-enter safely.
xml_etree, best of 3 (s): parse 1.64 -> 1.14, iterparse 0.68 -> 0.54,
generate 0.44 -> 0.25, process 0.27 -> 0.16.
Assisted-by: Claude Code:claude-fable-5-1
* _elementtree: native TreeBuilder and XMLParser
TreeBuilder keeps CPython's element/data/tail bookkeeping in Rust,
including the pull-parser event hooks _setevents installs. XMLParser
drives a pyexpat parser whose handlers are this type's own native
methods, so a start or end tag reaches the builder without a Python
frame: the universal-name cache, attribute dict and TreeBuilder call all
happen in Rust when the target is our own builder, while foreign targets
still go through the documented start/end/data/comment/pi protocol.
ElementTree.parse() also picks up the _parse_whole fast path.
xml_etree, best of 3 (s): parse 1.14 -> 0.53, iterparse 0.54 -> 0.28,
generate 0.25 -> 0.24, process 0.16 -> 0.16. Against the pure-Python
baseline that is parse 3.1x, iterparse 2.4x, generate 1.8x, process
1.7x.
Assisted-by: Claude Code:claude-fable-5-1
* pyexpat: stop rescanning the whole line per position query
LineTracker converted a TextPosition column to a byte offset by
validating and decoding its entire buffered line tail on every query,
and appended fed bytes to that tail one at a time. On a document that is
one long line -- what xml.etree.tostring() writes, and what the
xml_etree benchmark parses -- the tail is the whole read-ahead, so a
per-event lookup became quadratic. Walk only the characters the query
actually asks for, over the deque's two halves in place, and record fed
bytes a line at a time. Byte indices are unchanged.
xml_etree, best of 3 (s): parse 0.52 -> 0.38, iterparse 0.28 -> 0.25.
Assisted-by: Claude Code:claude-fable-5-1
* _elementtree: charge Element.__deepcopy__ to the recursion budget
A chain of elements is copied by native recursion, so a deep enough tree
overflowed the real stack instead of raising. Wrap the copy in
with_recursion, as CPython does, and drop the skip on
test_deeply_nested_deepcopy for the accelerated run.
Assisted-by: Claude Code:claude-fable-5-1
* pyexpat: don't let a stale NeedMore end a Parse() early
The parser thread reports NeedMore when it finds the feed buffer empty
and parks. If it reached that point before the chunk Parse() was about
to push became visible -- which it does whenever the thread happens to
be scheduled first, so more often under load -- pump() consumed that
report as if it answered the new chunk and handed control back to Python
having dispatched nothing, silently losing the whole document.
Stamp each NeedMore with the push count observed when the buffer was
found empty and ignore reports older than the push being pumped. The
push notifies the condvar under the same mutex the report was sent
from, so the parser thread is guaranteed to wake and report again.
Fixes an intermittent failure of test_pyexpat's InterningTest under
parallel load (about 5 losses per 20000 parses before, none after).
Assisted-by: Claude Code:claude-fable-5-1
* Restore xml.etree test markers instead of skipTest in bodies
Assisted-by: Grok:4.6
* Clone state before try_to_bool in pickle load_build
try_to_bool takes the object by value, and load_build still needs the
state dict afterwards.
Assisted-by: Grok:4.6
* Drop decorative section-separator comments
Assisted-by: Grok:4.6
* Represent pickle opcodes as an enum
Assisted-by: Grok:4.6
* Use full paths for thread-only types in pyexpat
Assisted-by: Grok:4.6
---------
Co-authored-by: Lee Dogeon <dev.moreal@gmail.com>
|
||
|
Jeong, YunWon
|
6d39d132d8 |
Keep SemLock OS calls behind SemHandle methods (#8740)
Stop exporting libc semaphore pointers from host_env. Stdlib now waits, posts, and reads values through SemHandle instead of *mut sem_t / HANDLE. Assisted-by: Grok:4.6 |
||
|
Jeong, YunWon
|
0bdb61bfb6 |
Serialize SemLock bookkeeping for Condition notify (#8739)
* Serialize SemLock owner bookkeeping around waits Hold a mutex around SemLock acquire/release/_count so the per-object owner count cannot tear while threads and processes share a Condition. Drop the mutex only for the blocking wait. Mark Unix SemHandle Send/Sync. Add a spawn Condition.notify_all snippet. Assisted-by: Grok:4.6 * Repeat the Condition.notify_all snippet 20 times Give the snippets job more chances to see a lost wakeup. Assisted-by: Grok:4.6 * Cover one-by-one Condition.notify in the snippet Wake a process waiter and a thread waiter with two notify() calls, not only notify_all(), and keep the 20-repeat loop. Assisted-by: Grok:4.6 * address review: bound snippet waits with timeouts Assisted-by: Grok:4.6 |
||
|
Jeong, YunWon
|
a7c0500a42 |
host_env: export remaining Windows socket and fault constants (#8736)
* host_env: export remaining Windows socket and fault constants Publish AF_HYPERV/AF_BLUETOOTH, HV_* GUIDs, SIO_TCP_SET_ACK_FREQUENCY, EXCEPTION_CONTINUE_SEARCH, and winreg ERROR_FILE_NOT_FOUND/INVALID_DATA. os.socket pack/unpack Hyper-V and RFCOMM addresses through host_env. Assisted-by: Claude * socket: publish HV_GUID_* as string attributes Assisted-by: Claude * host_env: cast SIO_TCP_SET_ACK_FREQUENCY to i32 The WinSock constant is u32 and does not fit i32 without a wrapping cast. Socket module attributes are i32. Assisted-by: Claude * socket: use socklen_t width for Windows SockAddr::new Windows socklen_t is i32; a u32 cast does not match socket2. Assisted-by: Claude * socket: drop unused Windows AF_* imports from c AF_BTH, AF_IRDA, and AF_SNA are published via host_socket pyattr and are not read through the Windows c alias module. Assisted-by: Claude * host_env: export Windows faulthandler exception codes Route stdlib _EXCEPTION_* pyattrs and exception_description through named host_env constants instead of hex literals. Assisted-by: Claude |
||
|
Jeong, YunWon
|
c382a607bb |
host_env: keep rustls off the rustls-free ssl surface (#8727)
* host_env: keep rustls off the rustls-free ssl surface ssl is MemoryBIO, constants, OID, and ALPN. ssl-rustls is the rustls engine. Drop the wasm32 rustls-pki-types web dep; the browser crate already enables that feature. Assisted-by: Claude * host_env: drop rustls-pki-types from the lockfile Assisted-by: Claude * host_env: name the rustls engine feature rustls ssl is the rustls-free surface. rustls is the engine. stdlib's ssl-rustls product feature still selects this. Assisted-by: Claude |
||
|
Jeong, YunWon
|
7da001a7c9 |
address review: kqueue lock and host_env select (#8724)
* address review: kqueue lock and timeout overflow Hold the kqueue fd lock through kevent and drop it before changelist parsing and signal handlers. Reject timeouts that overflow Instant or Duration. Round poll millisecond timeouts upward. Cast kevent defaults for NetBSD field widths. Provide login_tty on Solaris/illumos via setsid, TIOCSCTTY, and dup2. Assisted-by: Grok:grok-4.6 * Move select timeout conversion into host_env poll millisecond ceiling, kevent timespec rounding, and default filter/flag widths now live in host_env. stdlib select.rs keeps Python conversion and signal retry only. Assisted-by: Grok:grok-4.6 * Move socket wait and rusage timeval into host_env socket.sock_wait_deadline now calls host_env wait_fd. resource and process-time conversion no longer read timeval fields in stdlib or vm. Assisted-by: Grok:grok-4.6 * Deny libc select syscalls in vm and stdlib clippy Block libc poll/select/kqueue/epoll entry points and non-reexported ABI types. host_env keeps empty lists. Assisted-by: Grok:grok-4.6 * address review: portable timeval and wait_fd Widen rusage timeval math to i64 so 32-bit time_t compiles. Saturate duration_to_timeval when tv_sec is narrower than u64. Round wait_fd poll timeouts up and accept Windows RawSocket. Ignore EPERM from setsid in the Solaris login_tty fallback. Deny libc::epoll_pwait2 in vm and stdlib clippy. Assisted-by: Grok:grok-4.6 |
||
|
Hanjeong Lee
|
df653cea69 |
Name generated signature parameters the way CPython names them (#8725)
* Leave Rust-only name decoration out of a generated signature A leading underscore marks an argument Rust sees as unused and `r#` escapes a Rust keyword. Neither describes the parameter a Python caller passes, and `r#` is not valid Python, so `inspect.signature()` rejects any signature carrying one. Reporting the bare name fixes _sre.template, select.epoll.__exit__ and bytearray.__reduce_ex__ without touching them, and lets a method name a parameter after a Rust keyword. A parameter CPython itself names with a leading underscore, such as compile's `_feature_version`, reaches Python through a FromArgs field, which this never sees. Assisted-by: Claude Code:claude-opus-5 * Name builtin type parameters the way CPython names them A generated __text_signature__ reports the Rust parameter name, so a name chosen for Rust reaches anyone reading help() or inspect. Rename the ones CPython describes differently: list.pop's `i` becomes `index`, list.count's `needle` becomes `value`, __format__'s `spec` becomes `format_spec`, bytes.strip's `chars` becomes `bytes`, and object.__reduce_ex__'s `proto` becomes `protocol`. Left alone are the parameters CPython reports as `object` or `unused`. Those are synthesised from the METH_* flags rather than named: CPython reports slice.indices as ($self, object, /) while its own docstring calls the argument len. Assisted-by: Claude Code:claude-opus-5 * Name os parameters the way CPython names them os.read's `n` becomes `length`, os.lseek's `how` becomes `whence`, os.kill's `sig` becomes `signal`, os.stat's `file` becomes `path`, and os.putenv's `key` becomes `name`, among others. os.chmod is left alone. It reports (path, dir_fd, mode, follow_symlinks) while the call binds mode second and takes follow_symlinks by keyword, so the order and the kind are wrong rather than the names, which a rename cannot fix. os.fstatvfs is left alone too: it shares one Rust function with statvfs, which CPython names `path`, so the two aliases cannot report different names. Assisted-by: Claude Code:claude-opus-5 * Name sys parameters the way CPython names them sys.excepthook's (exc_type, exc_val, exc_tb) become (exctype, value, traceback), sys._getframe's `offset` becomes `depth`, sys.intern's `s` becomes `string`, and sys.settrace's `tracefunc` becomes `function`. sys._clear_type_descriptors takes the name CPython gives it, which Rust spells r#type. Assisted-by: Claude Code:claude-opus-5 * Name vm stdlib parameters the way CPython names them deque.rotate's `mid` becomes `n`, deque.extend's `iter` becomes `iterable`, _sre.ascii_tolower's `ch` becomes `character`, _operator.neg's `pos` becomes `a`, signal.alarm's `time` becomes `seconds`, and codecs.register_error's `name` becomes `errors`. gc.collect and charmap_build keep theirs. Their argument is a FromArgs struct or a PosArgs bundle, so the reported name is the binding rather than a parameter, and renaming the binding would only disguise that. Assisted-by: Claude Code:claude-opus-5 * Name stdlib crate parameters the way CPython names them unicodedata.category's `character` becomes `chr`, math.isqrt's `x` becomes `n`, cmath.polar's `x` becomes `z`, struct.calcsize's `fmt` becomes `format`, binascii.crc32's `init` becomes `crc`, zlib.adler32's `begin_state` becomes `value`, and array.append's `x` becomes `v`. array.fromunicode's parameter is `ustr`, which cspell has to be told about. array.__deepcopy__ keeps `_memo`, which the generator now reports as `memo`; CPython calls it `unused`, a name synthesised from METH_O rather than chosen. Assisted-by: Claude Code:claude-opus-5 * Name BaseExceptionGroup.split's parameter the way CPython names it CPython calls the argument of split and subgroup `matcher_value`. Assisted-by: Claude Code:claude-opus-5 * Strip a generated parameter name with unraw and one prefix check syn already removes r# through IdentExt::unraw, the way from_args.rs does for a field name. The empty-name guard never ran: a bare `_` is Pat::Wild, which func_sig has already refused. Assisted-by: Claude Code:claude-fable-5-1 * Take object and unused from CPython's Clinic signatures CPython also falls back to these names for METH_O, but list.append and insert, set.add, discard and remove, _weakref.getweakrefcount and getweakrefs, sys.displayhook and getrefcount, and array.__deepcopy__ get them from an Argument Clinic signature. The earlier commit that kept array.__deepcopy__'s `_memo` was wrong to call `unused` synthesised. test_pydoc's set.add summary-line tests now pass. Assisted-by: Claude Code:claude-opus-5 |
||
|
Jeong, YunWon
|
3191ecaf38 |
host_env: share thread, locale, loader and BSTR (#8702)
* host_env: share thread, locale, loader and BSTR
Add Windows thread name, locale-info/wcscoll, CompareStringOrdinal,
LoadLibraryEx/GetProcAddress/FreeLibrary, BSTR and COM helpers, and
AF_HYPERV UUID conversion. Wire _thread, locale, ctypes LoadLibrary
and WMI BSTRs through those wrappers. Ensure =X: before spawnve/execve.
Assisted-by: Claude
* host_env: expose rustls-free ssl core on wasm
Compile MemoryBIO, protocol constants, OID tables, hostname
checks, and ALPN parsing without the rustls feature so wasm
and other hosts can implement _ssl on the shared types.
Assisted-by: Claude
* stdlib: add wasm _ssl on rustls-free host_env
Install _ssl on wasm32 against rustpython_host_env::ssl.
MemoryBIO, constants, OID, hostname, and ALPN work. Context
settings are stored; wrap/handshake and rustls-only loaders
raise SSLError until a wasm TLS engine is wired.
Assisted-by: Claude
* host_env: address #8702 review comments
Cache-miss Windows ctypes lookup uses GetProcAddress. FreeLibrary
propagates OSError. start_new_thread returns GetThreadId. Export
_thread._get_name and drop pyfunction doc comments. Reject
embedded NULs in Windows locale.strcoll/strxfrm. Terminate
wcscoll/wcsxfrm and SetThreadDescription inputs.
wasm _ssl accepts wrap_bio owner and get_ca_certs(binary_form),
and MemoryBIO.read(-1) returns all pending bytes.
Assisted-by: Claude
* host_env: gate the whole ssl module on feature ssl
Compile rustpython_host_env::ssl only with the ssl feature.
Rustls crates stay native-only, so enabling ssl on wasm does
not pull rustls. Inside the module, rustls files follow
not(wasm32) rather than a second feature flag.
Assisted-by: Claude
* host_env: fix Windows types and thread names
Cast SysAllocStringLen to *mut u16 and RPC status to u32 so
windows-gnu and MSVC agree. Export _NAME_MAXLEN, truncate names
at NUL and the OS byte cap, and decode get_name with fsdecode.
Assisted-by: Claude
* host_env: compile rustls engine on WASI
Keep rustls-free MemoryBIO/constants/OID/ALPN on all wasm
targets. Compile the rustls engine on WASI, where UnixTime::now
exists. Allow the rustpython ssl feature on wasm without
ssl-rustls or ssl-openssl.
Assisted-by: Claude
* compiler: pick the earliest CPython parse override
A later incompatible-prefix scan must not hide an earlier
number or bracket diagnostic. Parser-driven tokenization
stops at the first failure.
Assisted-by: Claude
* host_env: address remaining #8702 review comments
Map a missing Windows export to a symbol-load error, keep
LoadLibrary's Win32 code on the OSError, drop incomplete
UTF-16 pairs when truncating thread names, hide set_name
and _get_name off linux/macOS/Windows, and let ssl-rustls
enable host_env.
Assisted-by: Claude
* stdlib: add wasm _socket and read-all MemoryBIO
Register a rustls-free-style _socket shim on wasm32 so
Lib/ssl.py can import. Treat MemoryBIO.read(-1) as read-all,
matching _ssl_MemoryBIO_read_impl.
Assisted-by: Claude
* host_env: fix clippy, WASI ssl cfg, and wasm rustls
Defer unclosed-bracket overrides until no token diagnostic
exists, keep WASI rustls and wasm ssl modules exclusive,
preserve lone Windows surrogates, compile the rustls engine
on browser wasm, and clear clippy denials.
Assisted-by: Claude
* host_env: fix Windows clippy and wasm UnixTime
Gate thread-name byte truncation to linux/macOS, allow
GetProcAddress/FreeLibrary pointer deref, and enable
rustls-pki-types web time on browser wasm.
Assisted-by: Claude
* host_env: lock rustls-pki-types web for wasm
Assisted-by: Claude
* host_env: silence Windows unused thread id and shear
Assisted-by: Claude
* stdlib: drop Windows locale clippy needless wraps
Windows clippy failed on locale.strcoll/strxfrm needless return
and _getdefaultlocale always wrapping Ok.
Assisted-by: Claude
* compiler: keep lexer errors over print hints
print x; 0x must report an invalid hex literal. Ranking by source
offset let the earlier print hint win. Treat a finished radix
literal as valid so print x; 0x1 still gets the print message.
Assisted-by: Claude
* compiler: keep decode errors over unclosed openers
Ranking by offset made ( '\\N' report an unclosed paren and
wiped the unclosed-bracket flag on f'{, so compile snippets
raised SyntaxError instead of incomplete input. Export wasm
socket._accept for Lib/socket.py.
Assisted-by: Claude
* wasm: drop duplicate _socket; split locale cfgs
_socket lives in rustpython-stdlib's socket_wasm. The wasm crate
only re-exported that file when freeze-stdlib was off. Split
strcoll, strxfrm, and getencoding into cfg'd function defs.
Assisted-by: Claude
* compiler: keep decode blockers over unclosed openers
(print x; '\\N' must keep the print hint. A later malformed
escape still suppresses the EOF opener even when print wins.
Assisted-by: Claude
* compiler: keep earlier f-string over later lexer errors
f'{x'; ' and f'{x'; 0x must not report the trailing quote or
hex literal. Decode and f-string diagnostics now compete with
lexer failures by offset.
Assisted-by: Claude
* compiler: rank decode vs lexer vs print by class
Unterminated quotes no longer hide format-spec newlines or
empty f-string fields. Print is considered only after the
final winner is not a lexer failure, so print x; f'{x'; 0x
keeps the print hint. Line-continuation uses CPython wording.
Assisted-by: Claude
* compiler: keep unclosed paren over line-cont at EOF
test_exceptions wants '(' was never closed at the opener when
a final backslash follows. Skip escaped quotes in the bracket
scan so {\'a\'} still reports the continuation error. Un-xfail
test_join_daemon_thread_in_finalization.
Assisted-by: Claude
* test_threading: xfail daemon join finalization off Linux
The test passes on Linux. macOS and Windows leave an ignored
deallocator exception on stderr.
Assisted-by: Claude
* vm: hang daemon threads at interpreter finalize
Forced-off daemons must stay not-done so join() during
finalization GC raises PythonFinalizationError and
Thread.is_alive() stays true. SystemExit made the handle
look finished on macOS/Windows. Drop the Lib/test xfail.
Assisted-by: Claude
(cherry picked from commit
|
||
|
Jeong, YunWon
|
7c4017b3d8 |
Retry select/poll/fcntl on EINTR and add kqueue (#8721)
* Retry select/poll/fcntl on EINTR
select() no longer shares one empty list for r/w/x and now
calls select(2) even with empty sets so the timeout elapses.
poll() rejects type instantiation, treats float timeout as
milliseconds, and raises RuntimeError on concurrent poll().
fcntl, ioctl, flock, and lockf retry after EINTR.
Assisted-by: Grok:grok-4.6
* Add select.kqueue and select.kevent
Expose kqueue(2)/kevent(2) on BSD and macOS, including
control() EINTR retry and marking open kqueue objects
closed after fork.
Assisted-by: Grok:grok-4.6
* Add login_tty and fix termios/syslog/mmap
termios.error is Exception, not OSError.
os.login_tty prepares a tty for a new session.
syslog.openlog encodes ident as UTF-8.
mmap.seek returns the new position.
Assisted-by: Grok:grok-4.6
* Honor mmap trackfd on Unix
When trackfd is false, do not keep a duplicated file
descriptor. size() then fails with EBADF and resize()
raises ValueError.
Assisted-by: Grok:grok-4.6
* address review: portable kevent and kqueue close
Build kevent values by field assignment so FreeBSD's
ext array compiles. Normalize timespec nanoseconds,
keep the kqueue fd live across control(), reinit the
open list lock after fork, make trackfd keyword-only,
and skip login_tty on Solaris/illumos.
Assisted-by: Grok:grok-4.6
(cherry picked from commit
|
||
|
sijun-yang
|
6cc068474f |
Return distinct empty lists from select() (#8720)
Assisted-by: Codex:gpt-6-astra
(cherry picked from commit
|
||
|
Jeong, YunWon
|
73e547a9c3 |
Honor generator close, throw context, and docs (#8701)
* Honor close unraisable and deallocator messages A failed close lookup on a yield-from target is unraisable. Ignored GeneratorExit from finalize gets a traceback and the closing-generator err_msg. __del__ uses the deallocator message. Assisted-by: Grok:grok-4.6 * address review: claim, ag_running, frame steal Keep the running claim through close() cleanup. ag_running reads running_async. aclose ignore-yield does not mark the generator closed. Steal the frame slot before uniqueness and expose it with try_to_owned. Assisted-by: Grok:grok-4.6 * Honor wrapper docs, yield assign, throw context method-wrapper exposes the slot __doc__. Parenthesized yield assignment uses the invalid-target messages. throw() restores without overwriting __context__, then chains only from the generator's own exc_info slot. Assisted-by: Grok:grok-4.6 * Remove passing expectedFailure markers method-wrapper docs, deallocator messages, and help pager output now match, so drop the leftover markers. Assisted-by: Grok:grok-4.6 * Honor wrapper signatures and assignment diagnostics Slot docs include text signatures. method-wrapper compares and hashes by identity. Keyword genexp and if/while assignment use the invalid-target messages. Assisted-by: Grok:grok-4.6 * address review: aclose, unraisable, frame publish aclose ignore-yield does not close the generator. Format unraisable object is None. close() leaves traceback to the caller; finalize attaches the generator frame. Publish the iframe before the lock-free slot. Assisted-by: Grok:grok-4.6 * Honor member descriptors and wrapper annotations Assisted-by: Grok:grok-4.6 * address review: getitem, bool doc, if assign Assisted-by: Grok:grok-4.6 * Honor eval-mode assignment invalid syntax Assisted-by: Grok:grok-4.6 * Supply getset and member docs from rustpython-doc Assisted-by: Grok:grok-4.6 * Enable rustls SSL on WASI targets Compile the rustls _ssl module for WASI and stub socket waits plus the native certificate store. Assisted-by: Grok:grok-4.6 * Add browser TLS _ssl in rustpython-wasm Register rustls _ssl and a MemoryBIO-oriented _socket from rustpython-wasm. Use rustls-rustcrypto because ring cannot compile for wasm32-unknown-unknown. Assisted-by: Grok:grok-4.6 * Define SSL protocol and cert constants as enums Keep OpenSSL bitmask defines as integer constants. Assisted-by: Grok:grok-4.6 * Ignore rustls deps cargo-shear cannot see The rustpython-wasm _ssl sources are compiled through a symlink, so shear does not count their crate uses. Assisted-by: Grok:grok-4.6 * Replace wasm ssl symlink with a BIO module Drop the stdlib ssl.rs link and keep a rustpython-wasm _ssl that only covers MemoryBIO wrap_bio TLS. Assisted-by: Grok:grok-4.6 |
||
|
Jiseok CHOI
|
08e6e697a1 |
sqlite3: fix iterdump import and filter support (#8692)
* sqlite3: fix iterdump import and filter support Fix Connection.iterdump() to import sqlite3.dump with a from-list and forward its keyword-only filter argument, matching CPython 3.14. Remove stale expected-failure markers from all passing iterdump tests. Assisted-by: GitHub Copilot:GPT-5.6 Luna (high) * sqlite3: fix clippy core iterator lint Use core::iter instead of std::iter in the iterdump filter kwargs path, as required by RustPython clippy. Add the sqlite3 expected-failure workflow and CI clippy feedback reports. Assisted-by: GitHub Copilot:GPT-5.6 Luna (high) |
||
|
Jiseok CHOI
|
407ba437cd |
sqlite3: fix row_factory deletion error (#8693)
Match CPython 3.14 by distinguishing row_factory assignment from deletion and raising AttributeError with the compatible message. Unmark the passing row_factory deletion test. Assisted-by: GitHub Copilot:GPT-5.6 Luna (high) |
||
|
Jeong, YunWon
|
a2e2bd299e |
asyncio: honor pending-task, unawaited, and get_referrers (#8689)
* asyncio: honor pending-task, unawaited, and get_referrers Report Task destructor as 'Task was destroyed but it is pending!'. Warn when a never-started coroutine is collected. Preserve signal() errno. Attribute generator-owned frame locals to the owner in gc.get_referrers. Remove the expectedFailure markers that now pass. Assisted-by: Grok:grok-4.6 * tests: drop expectedFailure after unawaited warning The never-started coroutine RuntimeWarning unblocks test_bpo_45813_1 and the AsyncMock assertNeverAwaited cases. Assisted-by: Grok:grok-4.6 |
||
|
Jeong, YunWon
|
fb88ea834f |
ssl: honor hostname_checks_common_name and tls-unique (#8683)
Expose HOSTFLAG_NEVER_CHECK_SUBJECT and _host_flags so ssl.py can disable Common Name fallback. HAS_TLS_UNIQUE matches the TLS 1.2 channel binding we already implement. Assisted-by: Grok:grok-4.6 |
||
|
Jeong, YunWon
|
5afe140abd |
Store ContextVar.name as the original str object (#8682)
* Store ContextVar.name as the original str object Keep the constructor argument instead of converting it to a Rust String so lone surrogates and str subclasses are preserved. Build __repr__ from the name and default reprs. Assisted-by: Grok:4.6 * Compare Context objects by their variable maps Implement Context ==/!= from the mapping contents and mark Context unhashable. Snapshot the map before value equality so re-entrant ContextVar.set during __eq__ cannot deadlock. Enable test_context.ContextTest.test_context_run_4. Assisted-by: Grok:4.6 |
||
|
Jeong, YunWon
|
2fc2479f72 |
openssl: use host_env for CA paths and file checks (#8680)
Replace openssl-probe and local std::fs checks with host_env native_certs and fs. Vendored OpenSSL loads system CA paths from host_env; OpenSSL's own defaults stay for a system lib. Assisted-by: Grok:grok-4.6 |
||
|
Jeong, YunWon
|
3fe3f37dac |
unicodedata: resolve name aliases and named sequences (#8676)
* unicodedata: resolve name aliases and named sequences
Vendor Unicode 17 NameAliases.txt and NamedSequences.txt into
rustpython-unicode so unicodedata.lookup matches CPython: formal
aliases and named sequences on the modern view, none on ucd_3_2_0.
Tangut derived-name lookup is case-insensitive. \N{} still rejects
named sequences because they are not a single character.
Assisted-by: Claude
* test_ucn: drop expectedFailure on test_aliases
The name-alias tables make the test pass; leaving the marker
fails CI as an unexpected success.
Assisted-by: Claude
* unicode: reject whitespace in Tangut derived names
parse_hex_code does not skip spaces around the hex digits.
Assisted-by: Claude
|
||
|
Jeong, YunWon
|
979576e2e4 |
ssl: move TlsConnection and session store to host_env (#8678)
TlsConnection, RecordCursor, rustls error mapping, hostname/ALPN/version helpers, and the capturing client session store now live in host_env. stdlib _ssl keeps socket/MemoryBIO I/O and Python objects. Remove unused stdlib sha2 and leftover workspace TLS crates. Assisted-by: Grok:grok-4.6 |
||
|
Jeong, YunWon
|
bfd52eda61 |
ssl: move remaining rustls engine into host_env (#8675)
* ssl: move remaining rustls engine into host_env Cert loading, rustls verifiers, ClientConfig/ServerConfig construction, key logging, acceptor helpers, and X509 error mapping now live next to the trust store. stdlib _ssl keeps the Python object and socket adapters. Assisted-by: Claude * Remove unused workspace SSL dependencies Drop oid-registry, rustls-platform-verifier, and x509-cert from workspace.dependencies. Assisted-by: Grok 4.6 |
||
|
Jeong, YunWon
|
3a8a196996 |
ssl: share rustls engine in host_env and fire msg callbacks (#8665)
* ssl: fire msg callbacks and compute tls-unique Observe TLS records to invoke _msg_callback, and derive TLS 1.2 tls-unique from the KeyLog master secret plus rustls PRF. Restore CHANNEL_BINDING_TYPES and drop the remaining test_ssl RUSTPYTHON skips. Assisted-by: Claude * lzma: build the xz-core engine on wasm32 Expose `_lzma` on wasm32 and Android by compiling the xz-core engine on those targets instead of gating it out. Assisted-by: Claude * ssl: move the rustls engine into rustpython-common Extract the VM-independent rustls helpers (MemoryBio, certificate decode, cipher strings, OID table, chain builder, providers, and TLS record observation) so stdlib _ssl wraps a shared engine. Assisted-by: Claude * ssl: keep the rustls engine in host_env TLS helpers share the host trust store and default verify paths, so they belong next to native_certs rather than in rustpython-common. Assisted-by: Claude * stdlib: drop unused jiff after moving cert decode Assisted-by: Claude |
||
|
Jeong, YunWon
|
ae7bc5b619 |
lzma: build the xz-core engine on wasm32 (#8664)
Expose `_lzma` on wasm32 and Android by compiling the xz-core engine on those targets instead of gating it out. Assisted-by: Claude |
||
|
Jeong, YunWon
|
ac5fcb9be3 |
ssl: drop rustls lock before socket I/O (#8662)
Hold the rustls connection mutex only around rustls operations in handshake, read, write, and shutdown. Socket and MemoryBIO waits run without that lock so a blocking recv cannot deadlock a concurrent send. Remove the hang skip on test_thread_recv_while_main_thread_sends. Assisted-by: Claude |
||
|
Jeong, YunWon
|
eee9ad6420 |
Align contextvars, CLI startup, and stat validation with CPython (#8657)
* Align contextvars, CLI startup, and stat validation with CPython Validate Context and ContextVar arguments and mapping keys, hash names through the Python protocol, and preserve string contents without invoking __str__. Set the main module loader for command and stdin execution, report missing scripts with exit status 2, and reject file descriptors combined with follow_symlinks=False. Enable the nine corresponding stdlib regression tests. Assisted-by: Codex:gpt-6 * Move set_main_builtin_importer off the compiler feature Define the method in vm_new.rs so builds without rustpython-compiler, such as example_projects/barebone, can compile. Also set BuiltinImporter on an existing __main__ when its loader is missing or None. Assisted-by: Grok:4.6 |
||
|
Jeong, YunWon
|
def52d06b1 |
ssl: cap rustls unsent output at 4 KiB (#8660)
Keep BIO writes from filling asyncio's socket transport past its high-water mark when SO_SNDBUF is small. Drop the Unix skip on test_sendfile_ssl_pre_and_post_data. Assisted-by: Claude |
||
|
Jeong, YunWon
|
531cf2b378 |
ssl: unify handshake, transport and shutdown state (#8656)
* ssl: accept ClientHello before selecting the SNI context Replace the provisional server connection and replay flags with explicit handshake states. Retain fatal alerts across nonblocking retries and preserve fragmented input on timeout. Enable the three SNI alert regressions. Assisted-by: Codex:GPT-5 * ssl: share socket and MemoryBIO transport progress Queue drained TLS records before fallible output and use one handshake record loop for both transports. Assisted-by: Codex:GPT-5 * ssl: unify shutdown and fatal alert lifecycle Track close-notify progress in TlsState, preserve pending alerts across retries, and share record-boundary shutdown for sockets and MemoryBIO. Retain rustls errors until Python exception conversion. Assisted-by: Codex:GPT-5 * ssl: emit the full peer certificate dict Use the same certificate dict helper for getpeercert() and Certificate.get_info() so OCSP, caIssuers, and CRL URLs are present. Clear CHANNEL_BINDING_TYPES on the rustls backend, and add a MemoryBIO leftover-plaintext snippet. Assisted-by: Claude |
||
|
Jeong, YunWon
|
7f9fa32d6e |
ssl: add key logging with a shared host append sink (#8655)
* ssl: add key logging with a shared host append sink Port the context-owned keylog sink design from Pyre, retaining CPython 3.14 path and context-switch semantics. Keep host file lifecycle in host_env and TLS formatting in stdlib. Assisted-by: Codex:GPT-5 * ssl: use core and alloc imports for key logging Assisted-by: Codex:GPT-5 |
||
|
Jeong, YunWon
|
e3bf84d1e7 |
Fix resource index conversion and REPL warning (#8654)
Accept __index__ values when converting resource limits and only report PyREPL fallback reasons for terminal-backed interactive sessions. Assisted-by: Codex:gpt-5 |
||
|
Jeong, YunWon
|
ad4532c0e0 |
ssl: retain WebPKI-selected certificate chains (#8653)
Port the connection-owned chain builder from Pyre. Preserve verification inputs with TLS configurations and sessions, report the WebPKI-selected path, and publish only its selected capath anchor. Match CPython chain reporting for CERT_NONE and resumed sessions. Assisted-by: Codex:GPT-6 |
||
|
Jeong, YunWon
|
25b5dfe5ca |
Fix /dev/fd, rlimit, spawn scheduler, and context cache (#8651)
* Fix /dev/fd, rlimit, spawn scheduler, and context cache Honor PYTHONEXECUTABLE when computing sys.executable. Rewind the file offset after a pyc magic probe. Stop masking rlimit values with RLIM_INFINITY. Print the pyrepl-unavailable warning before the rustyline shell. Parse posix_spawn scheduler and apply it where the host supports it. Update ContextVar cache fields under one lock. Remove expectedFailure and skip markers for tests that now pass. Assisted-by: Grok:4.6 * address review: check rlim_t overflow on convert Replace the u64-to-rlim_t as-cast in py2rlim_limit with a checked try_from, matching py2rlim. Assisted-by: Grok:4.6 |
||
|
강동안
|
7376e96665 |
Raise MemoryError without allocating via a MemoryError freelist (#8645)
* Add no_memory_error for allocation failures Assisted-by: Claude Code:claude-fable-5-1 * Use no_memory_error at allocation failure sites Assisted-by: Claude Code:claude-fable-5-1 * Raise MemoryError instead of aborting on sequence repeat allocation failure Vec::with_capacity and Vec::reserve in SequenceExt::mul and SequenceMutExt::imul call handle_alloc_error on failure, which aborts the process under panic = "abort". `[1, 2, 3] * 10**17` passes the MAX_MEMORY_SIZE guard (2.4e18 bytes < isize::MAX) and aborted at the allocation instead of raising MemoryError. Use try_reserve_exact and map failures to vm.no_memory_error(). This also removes the n != 2 special case in imul: with the reserve done unconditionally, the copy loop runs zero times for n == 2 and the final append uses the reserved capacity. Assisted-by: Claude Code:claude-fable-5-1 * Recycle MemoryError instances through a dedicated payload freelist no_memory_error allocated the exception object on every raise, so under real memory exhaustion the raise itself could abort. Keep dead MemoryError husks in a process-wide reserve and revive them on the next raise without touching the allocator. The freelist is scoped to MemoryError at the type level. PyMemoryError already exists as a transparent wrapper payload, but instances never used it: the default exception slot_new delegates to the base class, so everything was constructed as PyBaseException. Route no_memory_error through a PyMemoryError-typed construction instead; the freelist hooks live on PyMemoryError's own PyPayload impl, the dealloc glue is monomorphized per payload type, and the existing exact-type push gate admits exactly MemoryError husks. The #[pyexception] macro always generates the PyPayload impl, which left no way to attach freelist configuration; a payload = "manual" attribute now skips that generation so a type can write its own impl, mirroring the generated PAYLOAD_TYPE_ID chain and class-based downcast validation that keep the payload flavors interchangeable. Exceptions are the first HAS_DICT freelist type, so their husks carry an ObjExt prefix that PyInner::dealloc would normally drop; clear its instance dict and member slots before pushing so no stale attributes survive into a revived instance. The reserve is a process-wide Mutex rather than a thread_local so a single prewarm at interpreter init covers every thread; MemoryError is a cold path, so the lock is uncontended. Python-level MemoryError() still constructs a PyBaseException payload through the delegated slot_new and simply does not participate, matching CPython, where only the PyErr_NoMemory path uses the preallocated reserve. Assisted-by: Claude Code:claude-fable-5-1 * Clear MemoryError husks before they enter the freelist Assisted-by: Claude Code:claude-fable-5-1 * Use no_memory_error for the bz2 allocation failure The shared bz2 engine maps BZ_MEM_ERROR to Bz2Error::Mem; report it through no_memory_error like the other allocation-failure sites, matching PyErr_NoMemory in CPython's _bz2module. Assisted-by: Claude Code:claude-fable-5-1 * Change MemoryError husks Vec to a fixed-size array Assisted-by: Claude Code:claude-fable-5-1 * Increase MemoryError freelist size from 4 to 16 * Address review: keep repeat and husk clearing allocation-safe Assisted-by: Claude Code:claude-fable-5-1 |
||
|
Jeong, YunWon
|
699913d8cd |
Align codec, call, exception, and Unix module behavior with CPython (#8647)
* Collect consecutive charmap encoding failures Pass one contiguous run of unmappable characters to the codec error handler, and use the full run when a text replacement is itself unmappable. Assisted-by: Codex:gpt-5 * Leave conversions out of standalone format specs A conversion belongs to a replacement field rather than the format specification consumed by format(), so stop accepting and storing it in FormatSpec. Assisted-by: Codex:gpt-5 * Decode empty byte strings without looking up a codec Match PyUnicode_FromEncodedObject by returning the shared empty string before resolving the requested encoding for empty bytes and bytearray inputs. Assisted-by: Codex:gpt-5 * Validate call shape before converting builtin arguments Reject unknown keywords and excess arguments before conversions can raise a different error. Align Python-function positional-only handling, use declared list arguments in heapq, keep delegated codecs positional-only, and preserve CPython parameter names and validation order. Assisted-by: Codex:gpt-5 * Align attribute and exception initialization errors Give __dict__ and type __qualname__ their CPython diagnostics, preserve OSError filename fields correctly, accept seven-item SyntaxError details, and report Unicode error constructor arity before converting values. Assisted-by: Codex:gpt-5 * Expose missing Unix and Darwin module APIs Add the platform constants exported by fcntl, mmap, select, socket, syslog, termios, stat, errno, time, and _sre. Implement posix.killpg and resource.getpagesize, and expose resource.error as OSError. Assisted-by: Codex:gpt-5 * Honor the configured Python CPU count Parse -X cpu_count and PYTHON_CPU_COUNT, expose the stored value through sys._get_cpu_count_config, and let os.cpu_count prefer it over the host count. Also publish getsizeof with its CPython parameter name. Assisted-by: Codex:gpt-5 * Fix CPU count and WASI CI coverage Remove the obsolete expected-failure marker now that CPU count overrides pass, and avoid exporting FASYNC where WASI libc has no O_ASYNC constant. Assisted-by: Codex:gpt-5 * self review |
||
|
Jeong, YunWonandIvan Mironov
|
4a267b2616 |
ssl: discover default certificate paths in host_env (#8649)
Keep host certificate path probing with the native certificate provider, select the first available OpenSSL-style file and directory, and leave environment overrides to ssl.py. This also gives Android string defaults instead of the previous None values. Co-authored-by: Ivan Mironov <mironov.ivan@gmail.com> Assisted-by: OpenAI Codex: GPT-5 |
||
|
Jeong, YunWonandIvan Mironov
|
d67526cb08 |
ssl: load hashed capath certificates lazily (#8648)
Retain CA directories on SSLContext, materialize their OpenSSL-style hashed entries when building rustls configurations, and cache parsed DER until directory mtimes change. Publish only the trust anchor used by a completed handshake. Co-authored-by: Ivan Mironov <mironov.ivan@gmail.com> Assisted-by: OpenAI Codex: GPT-5 |
||
|
Joshua Megnauth
|
3442228b59 |
unicodedata: Fix name, unmask test_name (#8599)
This commit fixes most of UCD's names. The two main fixes are: * Handling chars missing in 3.2.0 by checking for membership #8548 * Handling names not present in unicodenames AI disclosure: I used AI for the initial research of this patch to determine why the Tangut characters are missing. AI also assisted with a bug fix where I neglected to parse the Tangut u32 as hex. I wrote all of the code. Assisted-by: GPT 5.6 Luna |
||
|
Jeong, YunWonandIvan Mironov
|
de6053099a |
ssl: take the OID table and the OpenSSL cipher-string parser from #8007 (#8646)
* ssl: read the OID table out of OpenSSL's object database
`txt2obj` and `nid2obj` have to answer with OpenSSL's own numbers, and the
table behind them was 39 entries transcribed by hand, so every object
outside that set was reported unknown.
Read OpenSSL's own files instead: `obj_mac.num` carries each object's NID
and `objects.txt` carries its OID, short name and description, and the
identifier `objects.pl` derives per object -- the `!Cname` override, else
the description, else the short name, each qualified by the enclosing
`!module` with `-` spelled `_` -- joins the two. `OidEntry`, `OidTable`
and the `find_by_*` functions keep their shape, so `ssl.rs` only changes
where an object has no OID to report.
Three details the object database settles that a curated table did not
have to:
- an object written with one name carries it as both, so
`nid2obj(48)` is `('emailAddress', 'emailAddress', ...)`;
- an object need not have an OID at all -- `DES-EDE3` and 139 other
algorithm names have a NID and nothing to encode -- so the fourth
field of the tuple is None for those, as it is on OpenSSL;
- one arc does not encode, since DER packs the first two into a byte,
so the bare roots `iso` and `itu-t` are OID-less too, while two arcs
such as X500's `2.5` are a real object.
Checked against `_ssl.nid2obj` on CPython built with OpenSSL, over every
NID it answers: 1387 of its 1390 present, with no disagreement on any of
them, up from 39. The three absent are OpenSSL's unnamed placeholder
slots, which it reports as ('NULL', 'NULL', None).
The data files come from openssl/openssl `crypto/objects/`, via the
rustls integration rewrite in #8007.
Co-authored-by: Ivan Mironov <mironov.ivan@gmail.com>
Assisted-by: Claude
* ssl: parse OpenSSL cipher strings in set_ciphers
Replaces the pattern matcher `set_ciphers` used with the grammar
`man openssl-ciphers` documents, in a new `ssl/cipher.rs` taken from
#8007: `!`, `-`, `+`, `@STRENGTH`, `@SECLEVEL=n`, `+`-joined
conjunctions, the DEFAULT/ALL/COMPLEMENTOF* groups, protocol versions,
SUITEB*, and the aXXX/kXXX/eXXX prefixes.
`set_ciphers` raises "No cipher can be selected." for a string that
names nothing the provider offers, which is the one failure
SSL_CTX_set_cipher_list reports, and keeps the TLS 1.3 suites selected
in provider order because that setter does not reach them. A SUITEB*
string pins its own key exchange groups, which `prepare_kx_groups`
now reads.
Against CPython's OpenSSL over 78 cipher strings, the selections agree
except under `@SECLEVEL=n`, which OpenSSL applies at handshake time and
this applies to the selection.
`CryptoExt` gains `default_cipher_suites` so that `DEFAULT` and `ALL`
are distinguishable.
Co-authored-by: Ivan Mironov <mironov.ivan@gmail.com>
Assisted-by: Claude
* ssl: finish cipher reporting and provider handling
Use the provider-ordered cipher metadata table as the single source for parsing, reporting, OpenSSL names and descriptions. This makes get_ciphers reflect the context selection and validates curves and Suite B against the active provider.
Keep set_ciphers from changing TLS 1.3 suites, preserve the provider's order, fix empty conjunctions, and keep FIPS initialization within the provider's filtered cipher and key-exchange defaults.
Co-authored-by: Ivan Mironov <mironov.ivan@gmail.com>
Assisted-by: Claude
Assisted-by: CodeRabbit
Assisted-by: OpenAI Codex: GPT-5
* ssl: preserve OpenSSL OID lookup rules
Keep object-name lookup case-sensitive and canonicalize numeric OID arcs before consulting the generated OpenSSL object table.
Assisted-by: OpenAI Codex: GPT-5
* ssl: preserve cleartext after MemoryBIO shutdown
Read at most one TLS record while waiting for close_notify in BIO mode, leaving any post-TLS cleartext queued for the BIO owner.
Co-authored-by: Ivan Mironov <mironov.ivan@gmail.com>
Assisted-by: OpenAI Codex: GPT-5
* ssl: reject empty cipher conjunction operands
Keep empty operands visible to the parser so malformed OpenSSL cipher strings such as AES+ and ALL++AES are rejected instead of normalized.
Assisted-by: CodeRabbit
Assisted-by: OpenAI Codex: GPT-5
* ssl: make Suite B cipher ids unambiguous
Spell out the u16 conversion so Windows-only iterator implementations cannot make collection type inference ambiguous.
Assisted-by: OpenAI Codex: GPT-5
* ssl: move native certificate loading behind host_env
Keep platform trust-store loading and filesystem predicates on the host boundary while exposing backend-neutral certificate bytes to the rustls integration.
Assisted-by: OpenAI Codex: GPT-5
* ssl: invalidate server config after cipher changes
Rebuild cached server configuration after set_ciphers updates the selected suites or Suite B key-exchange groups.
Assisted-by: CodeRabbit
Assisted-by: OpenAI Codex: GPT-5
* ssl: load default certificate sources independently
Honor SSL_CERT_FILE and SSL_CERT_DIR independently. Keep directory certificates as lazy capath candidates, and fall back to native roots only when no environment certificate file was loaded.
Co-authored-by: Ivan Mironov <mironov.ivan@gmail.com>
Assisted-by: OpenAI Codex: GPT-5
---------
Co-authored-by: Ivan Mironov <mironov.ivan@gmail.com>
|
||
|
Jeong, YunWon
|
6e6792ee5c |
common: share the hashlib engine, and publish blake2b/blake2s as types (#8643)
The digest, HMAC, PBKDF2 and scrypt state moves out of the `_hashlib` module into a feature-gated `rustpython_common::hashlib` that owns each state behind a `Mutex` and reports plain Rust results; `_hashlib` and `_blake2` keep the object glue. The engine writes its state through a raw pointer into a caller-owned buffer, so the buffer is boxed and only the pointer travels with the Python object, and it carries the alignment the engine asserts. The wrapper is built only after a successful init, so a rejected digest name leaves nothing to drop over storage the engine never wrote. blake2 moves from `blake2` to `blake2b_simd` / `blake2s_simd`, whose `Params` carry the whole parameter block. `_blake2.blake2b` and `blake2s` become immutable types with `SALT_SIZE`, `PERSON_SIZE`, `MAX_KEY_SIZE` and `MAX_DIGEST_SIZE` as class attributes, and their instances are instances of those types. They accept `key`, `salt`, `person`, `fanout`, `depth`, `leaf_size`, `node_offset`, `node_depth`, `inner_size` and `last_node`. `_hashlib.scrypt` is published on the same engine. Six `test_hashlib` expectedFailure markers drop with it. `blake2`, `digest`, `dyn-clone`, `hmac` and `pbkdf2` lose their last consumer. Assisted-by: Grok Assisted-by: Claude |
||
|
Jeong, YunWon
|
aa59f1ee53 |
common: share the bzip2 stream engine (#8638)
Move the VM-independent bzip2 stream owner into a `bz2`-gated `rustpython-common` module beside the zlib and lzma engines, and keep `rustpython-stdlib` as the Python object and exception adapter. The adapter no longer names the `bzip2` crate; the `bz2` feature carries it. `_bz2` was the last user of the generic `DecompressState` / `DecompressStatus` / `Decompressor` machinery in `stdlib/src/compression.rs`, so that goes with it; `DecompressArgs` stays for zlib and lzma. The decompressor feeds the stream through the existing `Chunker`, so buffered and freshly supplied input are no longer joined into a new allocation first. `test_decompress_after_data_error` passes on the ported engine, so its `expectedFailure` marker goes. Assisted-by: Grok |
||
|
Jeong, YunWon
|
39682562b3 |
Raise the shared argument-binding errors from the slots that count for themselves (#8634)
* Raise the shared argument-binding errors from the slots that count for themselves
Sites all over the tree checked their own argument counts and keywords and
wrote the message by hand, so they drifted: `attrgetter expected 1 argument,
got 0.` carried a full stop, `TypeVar() got unexpected keyword argument(s):
bogus` named neither the keyword the way its counterpart does nor the
function the way `typevar()` is named, and `_csv`'s dialect parser handed a
whole formatted sentence to `InvalidKeywordArgument`, which then wrapped it
in another one.
`Callee::arity_error` and `Callee::unexpected_keyword` are now public, and
these sites raise through them, so there is one place the wording lives.
`FrameLocalsProxy` counts its arguments before reading its keywords, and
`min`/`max` do too, which is the order `framelocalsproxy_new` and `min_max`
check in. `weakref.ref`, `GenericAlias`, `frozenset`, `attrgetter`,
`itemgetter`, `islice`, `start_new_thread`, `TextIOWrapper`, `AttributeError`,
`NameError`, `TypeVar`, `ParamSpec`, `TypeVarTuple` and `TypeAliasType` all
follow.
Of 25 calls measured against CPython 3.14.6, 21 now produce the identical
string. The four left over need what `Callee` cannot carry: `weakref.proxy`
is a function of its own there rather than a type, `frozenset` names the
subclass being constructed, and `TextIOWrapper` names its first parameter
and counts before converting.
Assisted-by: Claude
* Give the argument-binding errors the constructors the rest of them have
Raising one of these meant writing `Callee::of::<Self>(vm).arity_error(1..=3,
0, vm)` — the vm twice, and a type to know about before you could say what
you meant. Every other error in the tree is a `vm.new_*_error`.
So the message texts move out of `Callee` into functions the vm can call, and
the vm gets `new_arity_type_error` and `new_unexpected_keyword_type_error`
alongside `new_unsupported_bin_op_error` and the rest. A slot names itself
with the `NAME` its class already carries:
return Err(vm.new_arity_type_error(Self::NAME, 1..=3, 0));
return Err(vm.new_unexpected_keyword_type_error(Some("typevar"), &key));
The keyword one takes the name as an `Option` because the parser sometimes
has none of its own, which is what `_PyArg_Parser.fname` being NULL means and
what `_csv`'s dialect parser wants.
`bind_for` and `check_kwargs_empty_for` take anything a `Callee` converts
from, so the slots pass `Self::NAME` there too. `Callee` itself is left to the
binding machinery, which is the only place that needs to carry a name around
rather than use one.
No message changes: 22 of 26 calls still match CPython 3.14.6 exactly, and the
43-call set still stands at 23.
Assisted-by: Claude
|