mirror of
https://github.com/RustPython/RustPython.git
synced 2026-10-28 10:44:36 +08:00
main
465
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 youknowdot
|
9f288b9f5a |
Enable rustyline read-ahead buffer preservation (#9382)
Enable buffer-redux on the workspace rustyline dependency and add its locked buffer-redux 1.1.0 dependency. Keep existing dependency versions and checksums, including nix, libc and rustix, unchanged. RustPython retains a rustyline Editor across interactive prompts. On Unix, this feature lets a successful readline submission save unread buffered input for the next read instead of dropping the old reader's buffer. This is based on the source call chain, not a runtime reproduction. Assisted-by: OpenAI Codex:model version unavailable |
||
|
dependabot[bot]
|
b7bebe638c |
build(deps): bump object from 0.37.3 to 0.40.0 (#9351)
Bumps [object](https://github.com/gimli-rs/object) from 0.37.3 to 0.40.0. - [Changelog](https://github.com/gimli-rs/object/blob/main/CHANGELOG.md) - [Commits](https://github.com/gimli-rs/object/compare/0.37.3...v0.40.0) --- updated-dependencies: - dependency-name: object dependency-version: 0.40.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
ef6e2dbaee |
build(deps): bump mach2 from 0.4.3 to 0.7.0 (#9349)
Bumps [mach2](https://github.com/JohnTitor/mach2) from 0.4.3 to 0.7.0. - [Release notes](https://github.com/JohnTitor/mach2/releases) - [Commits](https://github.com/JohnTitor/mach2/compare/0.4.3...0.7.0) --- updated-dependencies: - dependency-name: mach2 dependency-version: 0.7.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
youknowdotanddependabot[bot]
|
8c0c242dc8 |
Align profiler bindings and capability checks with Python 3.15 (#9340)
* Align offline profiler argument binding with Python 3.15 Accept the canonical keyword names for binary sample writes and native context exits. Resolve the argument layout before numeric callbacks, then preserve float-subclass payloads while accepting numeric protocols for required and optional profile statistics. AI-assisted source extraction, implementation, and source review with OpenAI Codex. Runtime validation remains pending. Assisted-by: OpenAI Codex:model version unavailable * Restore the offline profiler capability flag Preserve the original mega branch's zero-valued process_vm_readv capability export. The native live unwinder remains unsupported; this restores the public feature query without claiming live inspection support or changing canonical test helpers. AI-assisted source preservation and review with OpenAI Codex. Assisted-by: OpenAI Codex:model version unavailable * Restore finite-EOF table bounds for offline profiles Match CPython's EOF-wide table count and decoding windows while retaining checked offsets, allocation limits, cursor bounds, and replay checks. This restores the original mega's finite-file contract without changing canonical tests or the already validated argument-binding implementation. Assisted-by: OpenAI Codex:model version unavailable * Preserve symtable warning exceptions and original filenames (#9282) Keep warning-hook exceptions untouched instead of rewriting SyntaxError subclasses. Thread the original Python filename through parser warning emission and fresh warning-as-error SyntaxError construction, preserving fsdecoded bytes paths and lone surrogates. Assisted-by: OpenAI Codex:model-version-unavailable * build(deps): align the Cranelift 0.136 dependency family (#9341) * build(deps): bump the wasmtime group across 1 directory with 2 updates Bumps the wasmtime group with 2 updates in the / directory: [cranelift](https://github.com/bytecodealliance/wasmtime) and [cranelift-module](https://github.com/bytecodealliance/wasmtime). Updates `cranelift` from 0.135.2 to 0.136.3 - [Release notes](https://github.com/bytecodealliance/wasmtime/releases) - [Changelog](https://github.com/bytecodealliance/wasmtime/blob/main/RELEASES.md) - [Commits](https://github.com/bytecodealliance/wasmtime/commits) Updates `cranelift-module` from 0.135.3 to 0.136.3 - [Release notes](https://github.com/bytecodealliance/wasmtime/releases) - [Changelog](https://github.com/bytecodealliance/wasmtime/blob/main/RELEASES.md) - [Commits](https://github.com/bytecodealliance/wasmtime/commits) --- updated-dependencies: - dependency-name: cranelift dependency-version: 0.136.3 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: wasmtime - dependency-name: cranelift-module dependency-version: 0.136.3 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: wasmtime ... Signed-off-by: dependabot[bot] <support@github.com> * Align cranelift-jit with the 0.136 dependency family Update cranelift-jit alongside cranelift and cranelift-module so the JIT shares their LibCall, FuncId, and Module trait identities. Regenerate only the affected dependency lock entries, preserving the original dependency PR changes. AI assistance covered the dependency diagnosis, alignment, and static review. Assisted-by: OpenAI Codex:model-version-unavailable --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Support selective canonical function migration Add hash-pinned function selection to the official Lib migration tool, preserving canonical source bytes and rejecting ambiguous dependencies. Assisted-by: OpenAI Codex:model version unavailable * Detect CPython runtimes for canonical inspection feature checks Add native runtime-cookie detection and import the exact Python 3.15 inspection helpers and tests through the official Lib migration tool. Preserve the offline profiler argument and finite-EOF corrections. Assisted-by: OpenAI Codex:model version unavailable --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
youknowdotanddependabot[bot]
|
5bccf31156 |
build(deps): align the Cranelift 0.136 dependency family (#9341)
* build(deps): bump the wasmtime group across 1 directory with 2 updates Bumps the wasmtime group with 2 updates in the / directory: [cranelift](https://github.com/bytecodealliance/wasmtime) and [cranelift-module](https://github.com/bytecodealliance/wasmtime). Updates `cranelift` from 0.135.2 to 0.136.3 - [Release notes](https://github.com/bytecodealliance/wasmtime/releases) - [Changelog](https://github.com/bytecodealliance/wasmtime/blob/main/RELEASES.md) - [Commits](https://github.com/bytecodealliance/wasmtime/commits) Updates `cranelift-module` from 0.135.3 to 0.136.3 - [Release notes](https://github.com/bytecodealliance/wasmtime/releases) - [Changelog](https://github.com/bytecodealliance/wasmtime/blob/main/RELEASES.md) - [Commits](https://github.com/bytecodealliance/wasmtime/commits) --- updated-dependencies: - dependency-name: cranelift dependency-version: 0.136.3 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: wasmtime - dependency-name: cranelift-module dependency-version: 0.136.3 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: wasmtime ... Signed-off-by: dependabot[bot] <support@github.com> * Align cranelift-jit with the 0.136 dependency family Update cranelift-jit alongside cranelift and cranelift-module so the JIT shares their LibCall, FuncId, and Module trait identities. Regenerate only the affected dependency lock entries, preserving the original dependency PR changes. AI assistance covered the dependency diagnosis, alignment, and static review. Assisted-by: OpenAI Codex:model-version-unavailable --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
youknowdotandyouknowdot
|
01c65128e4 |
Replace syn-ext with native syn 3 macro helpers (#9333)
Migrate macro metadata and item handling to syn 3, preserving option diagnostics and doc trimming. Add focused parser and attribute-edit tests. Assisted-by: OpenAI Codex:model version unavailable Co-authored-by: youknowdot <336763669+youknowdot@users.noreply.github.com> |
||
|
Jeong, YunWonandyouknowdot
|
f0b1e82652 |
Add tracing profiler and Python 3.15 offline profiling tools (#9271)
* Add deterministic profiler and canonical Python 3.15 tools
Use the merged operation critical sections with interpreter-bound profiler
entry checks and short payload borrows. Trace owned code references for GC.
Import the canonical tracing and offline profiling output closure via the
repository updater, retaining existing test markers.
Assisted-by: dot:GPT-6.1
* Clarify standalone profiling license cross-reference
Preserve license terms and copyright notices; point to the reproduced
Apache text below instead of CPython's original OpenSSL section above.
Assisted-by: dot:GPT-6.1
* Defer unsupported sampling package from profiler extraction
Keep the deterministic tracing profiler and its canonical tools together.
Defer the entirely new sampling production, tests, assets and their license
as one closure until native record and binary-collector support is ready.
Retain canonical pstats for deterministic empty-profile handling; its
upstream sampled-file sorting issue is not patched here.
Assisted-by: dot:GPT-6.1
* Stage profiling root as a namespace until sampling is available
Omit the whole newly added canonical package initializer rather than
advertise an absent sampling module or hand-edit canonical __all__.
Explicit profiling.tracing and cProfile imports remain available.
The namespace root intentionally lacks canonical metadata/automatic exports.
This addresses test___all__ failures in the prior Linux, macOS and Windows
CI runs. No test assertions, skips or canonical file contents are changed.
Assisted-by: dot:GPT-6.1
* Match canonical static builtin profiler labels and metadata
Normalize the raw builtin receiver instead of its call-argument accessor.
Snapshot static method module metadata at construction, matching CPython,
so explicit module mutation and deletion remain observable. Avoid a
profiler-time fallback that would obscure deleted metadata.
Correct float.__getformat__ to CPython's classmethod calling convention,
including subclass binding, rather than special-casing its profiler label.
Assisted-by: dot:GPT-6.1
* Support collision-safe frozen package containers
Generate empty frozen packages only for source namespace directories with
real frozen descendants. Preserve genuine modules and packages across
frozen-library registrations while retaining real-real replacement order.
Record the container marker in FrozenModule and its library envelope;
legacy module/package encodings remain unchanged. This is closed-tree
frozen packaging, not full PEP 420 namespace merging.
Add structural codec, mock generator, and registration-priority checks.
* Revert frozen package containers before restoring canonical profiling root
The container implementation changed filesystem namespace precedence.
Restore the existing importer and frozen envelope unchanged; the following
offline sampling closure restores the canonical regular package root.
This reverses
|
||
|
youknowdotandJeong YunWon
|
97bd4af533 |
Add interpreter-bound operation critical sections (#9215)
* Factor shared interpreter stop-the-world guard Move the existing collector barrier into the VM without changing its stop, restart or ownership behavior, preparing reuse by fork. Assisted-by: dot:GPT-6.1 * Add interpreter-bound operation section primitive Prepare attached-only ownership acquisition and detached observer parking with private per-interpreter scope stacks. Lifecycle wiring follows separately; no profiler consumer is enabled by this commit. Assisted-by: dot:GPT-6.1 * Suspend operation sections at VM lifecycle boundaries Integrate detach, stop-the-world, shutdown and interpreter-switch boundaries. Reuse the all-interpreter barrier for fork repair and add focused native ownership invariants. Assisted-by: dot:GPT-6.1 * Replace operation sections with PyMutex critical sections Add PyMutex with the lock.c protocol: LOCKED/HAS_PARKED bits, a bounded spin, direct handoff to waiters parked longer than 1 ms, and detaching while parked. Rebuild critical sections on PyMutex with the stack kept as a tagged pointer in ThreadSlot: single and two-mutex sections, recursive top-most skipping, locking skipped while the world is stopped, SuspendAll on detach/suspend/shutdown and Resume of the top-most section on attach. Fork carries the stacks into the replacement slots. Assisted-by: Claude Code:claude-opus-5-5 * Check attachment around critical-section suspension Assert that a thread entering a critical section is attached, and suspend sections in do_suspend only while the slot is still ATTACHED. Assisted-by: Claude Code:claude-opus-5-5 --------- Co-authored-by: Jeong YunWon <jeong@youknowone.org> |
||
|
Jeong, YunWon
|
cdb5cca177 |
Use the Ruff fork 0.17.1 for CPython syntax error messages and the obsolete '<>' hint (#9212)
* Use the RustPython Ruff fork 0.17.1 from git and place EOF errors by kind - The ruff crates come from the RustPython/ruff `rustpython-0.17.1` branch at 93967836c8 instead of the 0.16.10 crates.io release. - `EofToken.known_location` is replaced by `EofErrorLocation` (Tokenizer, Token, StartingFromToken), chosen from the parse error kind. The keyword scan in `clause_keyword` is removed, since the parser now reports a forced `:` as `ExpectedForcedToken`. - An empty import at EOF ends at the end of the line read, and an empty import before a newline is an empty range at the newline. Assisted-by: Claude Code:claude-opus-5-5 * Report the obsolete `<>` from the parser outside Barry mode - Bump the RustPython/ruff fork to 9290a88495, where the parser reports an adjacent `<` and `>` in a comparison as "invalid syntax. Maybe you meant '!=' instead of '<>'?" over both tokens. - Outside Barry mode, `BarrySource` no longer searches the source text for `<>`, so the parser's error is reported unchanged. A `<>` that starts a statement is reported at the `<` only, and `<<>` at the `>`. Assisted-by: Claude Code:claude-opus-5-5 |
||
|
youknowdot
|
c3b8ce79c8 |
Preserve regression collector outcomes and canonical process context (#9234)
* Preserve regression collector outcomes and canonical process context Record complete test identities, subtest diagnostics and fixture outcomes while keeping unittest event counts separate from once-per-case JSON totals. Reuse canonical process setup and avoid an unnecessary library-path override. Assisted-by: OpenAI Codex: model version unavailable * Fix regression runner subtest formatting and qualified discovery Assisted-by: OpenAI Codex: model version unavailable * Make runner traceback assertions independent of forced colors Assisted-by: OpenAI Codex: model version unavailable * Keep dev VM frames compact for stdlib imports Carry the established integration VM optimization override into the regression runner split. Retain debug assertions, stack guards and Python recursion limits. Assisted-by: OpenAI Codex: model version unavailable |
||
|
youknowdotandJeong YunWon
|
3f55cf4af0 |
Implement basic REPL execution and fix piped stdin handling (#9163)
* Preserve original main VM through C API execution
Run and finalize the original main VM while keeping the registered attachment template available without holding its mutex across Python callbacks.
Assisted-by: OpenAI Codex:model version unavailable
* Make unraisable reporting robust at hook boundaries
Keep the existing message API and traceback formatting while handling missing, disabled, audited and failing hooks without panicking.
Assisted-by: OpenAI Codex:model version unavailable
* Separate interactive warning collection from compilation
Retain ordinary warning behavior and typed module APIs while exposing record-local parser warnings and post-warning code generation. Add a source-only lazy future-import guard.
Assisted-by: OpenAI Codex:model version unavailable
* Evaluate native stdin with shared VM record semantics
Use one evaluator for basic REPL and root transports, preserve effective inspect snapshots and future flags, and carry owned host exit status through original-main finalization.
Assisted-by: OpenAI Codex:model version unavailable
* Refresh canonical REPL tests and retire fixed failures
Run update_lib migrate from pinned CPython 3.15.0, preserving three remaining platform/runtime markers and retiring the two traceback/startup failures verified on the native evaluator.
Assisted-by: OpenAI Codex:model version unavailable
* Keep the root compiler dependency for development targets
The native evaluator now compiles through the VM. Only root benchmarks and the parse_folder example use rustpython-compiler directly, so declare it as a dev dependency while preserving VM compiler features.
Assisted-by: OpenAI Codex:model version unavailable
* Validate explicit conditional failure marker retirement
Require exact selector and original decorator evidence for each conditional expected-failure marker retired by migrate. Preserve undeclared guards and reject missing, duplicate, modified or unconsumed declarations.
Assisted-by: OpenAI Codex:model version unavailable
* Refresh canonical stdin and exception audit tests
Migrate the pinned CPython 3.15.0 modules and their canonical helper fixtures through update_lib. Retire exactly four failure markers proven obsolete by hosted CI, preserving all other existing guards.
Assisted-by: OpenAI Codex:model version unavailable
* Synchronize runpy warning attribution and document remaining capabilities
Migrate whole canonical runpy.py to pass module names to compile. Use update_lib to annotate only reproduced missing-capability cases: absent _testcapi import-event tests, unsupported startup/import-time features, and the deferred frozen-loader module-attribution caller.
Assisted-by: OpenAI Codex:model version unavailable
* Propagate interrupted native stdin reads to signal dispatch
Preserve the shared input buffer and partial bytes while returning EINTR to the VM before retrying. Exercise the transport deterministically and verify a returning SIGINT handler while pipe stdin stays open.
Assisted-by: OpenAI Codex:model version unavailable
* Reject statements after a compound statement in single mode
single_mode_multiple_statements_error now reports "invalid syntax" at the
statement that follows a compound first statement, and interactive_input
runs it right after parsing as compile() does.
The lazy `from __future__` import check moved from lazy_future_import_error
into unsupported_grammar_error, so it applies to every compile mode.
Assisted-by: Claude Code:claude-opus-5-5
* Remove passing expected failure from test_runpy
test_run_module_filter_syntax_warnings_by_module passes.
Assisted-by: Claude Code:claude-opus-5-5
* Restructure stdin execution after Modules/main.c and pythonrun.c
- src/lib.rs follows pymain_run_python: pymain_run_stdin, pymain_repl,
pymain_run_startup, pymain_run_interactive_hook and pymain_run_pyrepl
return exit codes instead of raising SystemExit.
- VirtualMachine gains run_any_stdin (_PyRun_AnyFile),
run_interactive_loop (_PyRun_InteractiveLoop) and run_simple_stdin;
run_shell feeds rustyline lines into run_interactive_loop, and
sys._baserepl calls run_any_stdin.
- print_uncaught_exception (PyErr_Print) replaces
report_uncaught_exception, and handle_system_exit_and_keyboard_interrupt
replaces handle_reported_exception.
- An unhandled KeyboardInterrupt is recorded and the process exits
through SIGINT after finalization.
- Interpreter::run_main replaces run_with_exit_status and InterpreterExit;
rustpython_capi::run_main_interpreter takes an exit-code callback.
- config_set_inspect audits cpython.PyConfig_Set and rebuilds sys.flags
through set_flag_obj (_PySys_SetFlagObj); the inspect state is an
AtomicBool and dict replace_by_identity is removed.
- add_main_module (PyImport_AddModuleRef("__main__")) is used by the
interactive and simple-file paths.
- The interactive parser warning helpers take their state directly.
Assisted-by: Claude Code:claude-opus-5-5
* Exit through Py_Exit from PyErr_Print and run the basic REPL via stdin
- print_uncaught_exception (PyErr_Print) calls exit_process (Py_Exit) for a
SystemExit when inspect mode is off, instead of returning the exception.
run_interactive_loop no longer checks SystemExit itself and flushes after
printing an error; sys._baserepl returns None.
- exit_process finalizes through VirtualMachine::finalize_ex, which holds
the Py_FinalizeEx steps split out of Interpreter::finalize, and exits
through host_env::os::exit_with_cleanup (C exit()).
- The basic REPL on a terminal (PYTHON_BASIC_REPL, or _pyrepl missing)
runs run_any_stdin instead of the rustyline shell.
- run_shell saves the history after every line.
Assisted-by: Claude Code:claude-opus-5-5
* Display exceptions through PyErr_Display and track the main thread running __main__
- display_exception (PyErr_Display) and display_exception_to (_PyErr_Display)
try traceback._print_exception_bltin before the native printer and flush
the file. PyErr_Print's excepthook fallback, print_exception and
sys.excepthook use them. A missing sys.stderr is reported with
dump_object (PyObject_Dump); a None sys.stderr prints nothing.
- Paths gains program_name (argv[0]) from getpath. The "can't open file"
message uses it with strerror(errno) and goes to sys.stderr.
- running_main holds the ident of the thread running __main__.
pymain_run_python sets it around running the main code, Py_Exit clears it,
the fork child reinitializes it, and _interpreters.is_running reads it
instead of treating the main interpreter as always running.
- Interpreter::run_main clears the unhandled KeyboardInterrupt flag before
running and applies exit_sigint after finalizing; finalize no longer
does.
- run_interactive_loop returns a SystemExit raised outside inspect mode
again and no longer flushes after printing an error; run_shell saves the
history once after the loop.
- pymain_set_inspect reports its error through print_uncaught_exception.
Assisted-by: Claude Code:claude-opus-5-5
* Treat a parse error at the end of the source as incomplete input
A parser error that starts at the end of the source is reported as
_IncompleteInputError when incomplete input is allowed, so the
interactive loop and codeop keep reading after `try:` and its block or
after a decorator line.
Assisted-by: Claude Code:claude-opus-5-5
* Place syntax errors on tokens made up at the end of the input
A parse error on the NEWLINE, DEDENT or ENDMARKER the tokenizer makes up
at the end of the input now carries an EofToken: the number of lines
read and the last line. SyntaxError takes its lineno and text from it,
and its offset is that line's length with end_offset -1, or 0 for both
when the error is raised at the token itself (the generic syntax error
and a missing forced `:` or `(`). Exec input that does not end with a
newline is checked with one appended.
Assisted-by: Claude Code:claude-opus-5-5
* Slice the source with get when checking an indented block error at EOF
Assisted-by: Claude Code:claude-opus-5-5
* Rename the host exits after C exit() and fix wasi and Windows builds
- host_env::os::exit now calls the C exit() on every platform; on Windows
std::process::exit ends in ExitProcess, so it flushes the Rust stdout
and calls libc::exit there. The former exit, which calls _exit(), is
now exit_immediately, used by os._exit, faulthandler and
posixsubprocess.
- errno::strerror_string is available on wasi.
- PyGlobalState::reinit_running_main is built only where its fork caller
is.
Assisted-by: Claude Code:claude-opus-5-5
---------
Co-authored-by: Jeong YunWon <jeong@youknowone.org>
|
||
|
Joshua Megnauth
|
cf6ca56789 |
Fix Unicode 3.2.0 aliases, bump unicode_names2 (#9171)
Replaces: #9166 unicode_names2 fixed a bug with its aliases. Ironically, the bug caused a test in test_snippets.py to pass. Bumping unicode_names2 to 4.0.0 fixes the bug and thus the extra 3.2.0 test fails. I fixed the logic for name lookup for 3.2.0. Now the test correctly passes. |
||
|
codspeed[bot]andCodSpeed Bot
|
008b8e1c89 |
Skip unit-test harnesses when building benchmarks (#9158)
cargo bench and cargo codspeed build select every target with bench = true. That includes the library and binary targets, so the build also compiled the library unit-test harness as an extra LTO-linked copy of the interpreter that no benchmark runs. Mark them bench = false so only the criterion benchmarks (and the binary they depend on) are built. Co-authored-by: CodSpeed Bot <no-reply@codspeed.io> |
||
|
Jeong, YunWon
|
2dbe992438 |
Remove ucd_3_2_0 feature from rustpython-capi (#9153)
Drop the capi [features] table added in #9011 along with the root ucd_3_2_0 forwarding to rustpython-capi and the default-features = false on the workspace rustpython-capi dependency. The root ucd_3_2_0 feature still enables rustpython-stdlib/ucd_3_2_0. Assisted-by: Claude Code:claude-opus-5-5 |
||
|
Bas Schoenmaeckers
|
88f5189a93 |
Update pyo3 to 0.30 (#9152)
* Update `pyo3` to `0.30` * Fix cfunction with no module as self |
||
|
Joshua Megnauth
|
a2c11aa51d |
Optional Unicode 3.2.0 but enabled by default (#9011)
Useful for smaller builds, like WebAssembly. The UCD 3.2.0 path is only directly used in a handful of places in Python. Assisted-by: GPT-5.6-Luna |
||
|
youknowdot
|
0f127b07e9 |
Add Python 3.15 math APIs using pymath (#9122)
* Add Python 3.15 math APIs using pymath Pin pymath to merged dd1e9a4cf26d68caf3b549c09476f52e356e16a3 and expose isnormal, issubnormal, signbit, fmax and fmin through thin wrappers. Share the existing integer functions through math.integer and _math_integer, matching CPython module identity and metadata. Migrate test_math.py and test_math_integer.py via scripts/update_lib from CPython d220409a13b77de7a6c30abe16ea90db01528edc, preserving the existing Windows remainder marker. Add bounded Python snippet coverage. AI assistance: OpenAI Codex implemented and source-reviewed this change, ran update_lib, and performed focused automated validation. No human manual testing is claimed. Assisted-by: OpenAI Codex:model version unavailable * Track integer math tests in library updates Assisted-by: OpenAI Codex:model version unavailable |
||
|
Jeong, YunWon
|
0816af6543 |
Update malachite crates to 0.14.0 (#9117)
pymath requires malachite-bigint ^0, so the example projects, which have no lock file, resolve it to 0.14.0 while the workspace pinned 0.13.0, and fail to build. Assisted-by: Claude Code:claude-opus-5-5 |
||
|
youknowdot
|
5f3daa5386 |
Implement Python 3.15 base64 APIs (#9085)
* Implement Python 3.15 base64 encoding and decoding Follow CPython 3.15.0 final d220409a13b77de7a6c30abe16ea90db01528edc for base64 options, alphabets, padding and canonical decoding. Keep reverse-table caches in traced module-native state and preserve module-bound callable semantics. Add bounded Python regressions without editing Lib. Assisted-by: OpenAI Codex:model identifier unavailable * Complete binascii codecs for canonical Python 3.15 tests Migrate Lib/test/test_binascii.py unchanged from CPython 3.15.0 final using scripts/update_lib and replace the superseded custom base64 snippet. Implement the base32, Base85, Ascii85 and hex ignorechars APIs required by the canonical suite. Share the existing module-bound original-key alphabet cache across base32/base64/base85 and preserve argument and buffer cleanup semantics. Keep the existing base64 kernels and VM state integration unchanged. Assisted-by: OpenAI Codex:model identifier unavailable |
||
|
Jeong, YunWon
|
1e72b22470 |
Update malachite crates to 0.13.0 (#9003)
Assisted-by: Claude Code:claude-opus-5-5 |
||
|
|
a403341791 |
Take SyntaxError messages and ranges from the parser (#8983)
* Match CPython's SyntaxError range for unparenthesized `except` types
CPython's `invalid_except_stmt` rule raises the error only once the whole
clause has matched, and reports a range that starts at the first exception
type and ends at the `:` closing the clause, so it covers the `as NAME`
part as well:
invalid_except_stmt:
| 'except' a=expression ',' expressions 'as' NAME ':' {
RAISE_SYNTAX_ERROR_STARTING_FROM(a, "multiple exception types
must be parenthesized when using 'as'") }
The parser reports the exception types alone, so look up that `:` in the
source and widen the range to match. Only `as NAME` may follow the
exception types, which is why the first `:` after them is the one closing
the clause.
try:
pass
except A, B, C as e:
pass
CPython 3.14: ('x.py', 3, 8, 'except A, B, C as e:\n', 3, 20)
before: ('x.py', 3, 8, 'except A, B, C as e:\n', 3, 15)
Fixes #8496
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Clarify that the `except` error range stops before the closing colon
`invalid_except_stmt_end` returns the exclusive end of the range, so the
column it yields is the one the `:` sits on even though the `:` is not part
of the range. The previous "ending at the `:`" wording read as if the colon
were included, which invites an off-by-one "fix".
Comments only; no behavior change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Fix the `except ... as` error column for non-ASCII names and line joins
`invalid_except_stmt_end` recovers the end of CPython's range by scanning the
source, which has to redo by hand what CPython gets from its tokenizer:
- The column was the byte distance from the line start, but `end_offset` is a
character column. `except Ä, B as e:` reported 18 instead of 17. CPython
converts explicitly, in `_PyPegen_byte_offset_to_character_offset`.
- The colon scan already spans explicit line joins, but the `as` check ran on
the raw slice, so a `\` before `as` left the range unextended.
`except A, B \` + `as exc:` reported (3, 12) instead of (4, 7).
Add regression cases to `syntax_invalid.py` covering both, plus the whitespace,
`except*` and line-join placements that already worked. All values are verified
against CPython 3.14.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Update the Ruff fork to 0.16.10
Point the rustpython-ruff_* dependencies at the 0.16.10-rustpython tag
of the RustPython Ruff fork and port to its AST changes:
`ExprCompare` stores all operands in `operands`, call arguments use
`ThinVec`, dict comprehension generators and f-string elements are boxed
slices, `FStringValue` iterates `FStringPartRef`, and `TokenKind::Name`
is now `TokenKind::Identifier`.
Assisted-by: Grok:grok-4.6
Assisted-by: Claude Code:claude-opus-5-5
* Drop the source scan for the `except ... as` error range
The parser now reports the unparenthesized exception types error with a
range ending at the `:` that closes the clause, so
`invalid_except_stmt_end` and the message match that widened the range
are removed.
Assisted-by: Claude Code:claude-opus-5-5
* Take parse error messages from the parser as they are
The Ruff fork now emits the final message texts, so `vm_new.rs` no
longer rewrites parse error messages by matching their text and no
longer lowercases the first character of every message.
`analyze_compile_error` keeps only the caret narrowing for starred
expressions and the line number in the unterminated string message.
The compiler matches the new lowercase texts for indented block and
mixed bytes literal errors.
The fork is referenced by revision until it is published.
Assisted-by: Claude Code:claude-opus-5-5
* Take starred and bytes literal error ranges from the parser
The Ruff fork now reports misplaced starred expressions at the `*` and
non-ASCII bytes literal errors over the whole literal. `vm_new.rs` drops
`narrow_caret` together with `SyntaxErrorInfo`, which only carried the
message after that, and the compiler drops `bytes_literal_span`.
Assisted-by: Claude Code:claude-opus-5-5
* Take indentation error kinds and positions from the parser
The Ruff fork now reports `TabError` and `TooDeepIndentation` as their
own lexer errors and places indentation and line continuation errors
itself. `vm_new.rs` picks TabError or IndentationError from the error
kind instead of scanning the source for mixed indentation, drops the
TabError message override, and sets the end offset of tokenizer errors
to 0 or -1 by kind. The compiler drops the line-end relocation of
unindent errors and the line continuation rewrite. `_tokenize` raises
TabError and depth errors for the token they replace, also for sources
containing tabs.
Remove expectedFailure from test_tokenize.test_max_indent and four
test_tabnanny tests.
Assisted-by: Claude Code:claude-opus-5-5
* Take unterminated string messages from the parser
The Ruff fork's unterminated string errors now include the detected
line and are reported at the start of the string. `vm_new.rs` drops its
message override for `UnclosedStringError`, and the shell continues a
line on an unterminated triple-quoted string by the error's
`triple_quoted` flag instead of reading the quotes from the source.
Assisted-by: Claude Code:claude-opus-5-5
* Take bracket errors and tokenizer error priority from the parser
Bump the ruff fork, which now reports unmatched, mismatched, unclosed
and too deeply nested brackets and decides whether a tokenizer error
replaces the parser's first error.
Remove the bracket, unterminated string and nesting depth source
scanners and `pre_parse_source_error`. A tokenizer error from the
parser now wins over source scanner diagnostics found after it. An
unclosed bracket is incomplete input only when the parser reports it
as such, and other tokenizer errors end where they start.
Remove the expected failure markers from test_unicode_identifiers
test_invalid and the `import ä £` doctest in test_syntax.
Assisted-by: Claude Code:claude-opus-5-5
* Take number literal, string prefix and non-printable errors from the parser
Bump the ruff fork, which now reports malformed number literals,
incompatible string prefixes and non-printable characters itself.
Remove the number literal, string prefix and non-printable character
source scanners and the override classes that ranked them. Leading zero
and string prefix errors keep their range instead of ending where they
start.
Assisted-by: Claude Code:claude-opus-5-5
* Take the missing indented block error from the parser
Bump the ruff fork, which now reports `ExpectedIndentedBlock` with the
clause and the line of its header keyword.
Remove the message rewrite that derived the clause and line from the
source. The VM chooses IndentationError or incomplete input by the
error variant instead of its message.
Assisted-by: Claude Code:claude-opus-5-5
* Take binding target errors from the parser
Bump the ruff fork, which now names the invalid expression in
assignment, augmented assignment, delete, `for`, `with`, import,
pattern and `except` target errors.
Remove the source scanners for those targets and the message rewrites
of `InvalidAssignmentTarget` and `InvalidNamedAssignmentTarget`.
Remove the expected failure markers from test_syntax doctests that now
pass.
Assisted-by: Claude Code:claude-opus-5-5
* Take parameter, argument, star, comma and dictionary errors from the parser
Bump the ruff fork to 36919b62fa. Remove the source scanners for type
parameters, parameter lists, star annotations, call arguments, def type
parameters, parenthesized groups, missing commas and dictionary items.
Set end_offset 0 for ExpectedColonAfterDictionaryKey. Remove 14
EXPECTED_FAILURE markers in test_syntax and test_unpack_ex.
Assisted-by: Claude Code:claude-opus-5-5
* Take expression errors from the parser
Bump the ruff fork to 4f922d2124. Remove the source scanners for
comprehensions, collection and condition assignments, expressions
between strings, named expression targets, missing `in` after for-loop
variables and statements in `if` expressions. Remove the now passing
expected failure markers in test_syntax and test_named_expressions.
Assisted-by: Claude Code:claude-opus-5-5
* Take statement errors from the parser
Bump the ruff fork to c7f6594ed6. Remove the source scanners for
standalone except, `import ... from`, `**_` mapping rests, `elif` after
`else` and mixed except handlers, and the post-parse scanners for call
arguments, match targets, yield after a comma and parenthesized star
imports.
Assisted-by: Claude Code:claude-opus-5-5
* Take string and f-string errors from the parser
Bump the ruff fork to 6b46d0dfc8. Remove the source scanners for
malformed `\N` escapes, f-string and t-string replacement fields, mixed
t-string literals and `print`/`exec` statements, and the override
plumbing they used. Drop the `f'{x'; '` case from the compiler test,
whose expected message differs from CPython 3.14.
Assisted-by: Claude Code:claude-opus-5-5
* Restore the bracket depth check and fix incomplete unclosed strings
Restore `pre_parse_source_error`, which rejects bracket nesting past 200
levels before parsing. The parser reports the same error but still
builds the full tree, which overflows the native stack when dropped.
Remove the VM's unclosed string scanner, which ignored the mode and
treated `'abc` in exec mode as incomplete input. The compiler's check
now skips escaped newlines.
Assisted-by: Claude Code:claude-opus-5-5
* Report multiple statements before later errors in single mode
In single mode, when the first statement is a simple statement, report
code after its line as multiple statements before any error found
later, at the newline token or the comment before it. This replaces the
multiple statements check on the parsed body and also applies to
`ast.parse(mode='single')`.
An unclosed string that follows an earlier syntax error is no longer
incomplete input. The bracket depth error range is now empty.
Assisted-by: Claude Code:claude-opus-5-5
* Update the Ruff fork for parser speed and starred annotations
Bump the ruff fork to 8b59b85344, which skips the tokenizer error pass
when parsing finds no errors, stops allocating for every identifier's
string prefix check, and reports `def f(*args: *b = ...)` as invalid
syntax.
Remove expected failure markers from test_code_module
test_indentation_error and test_sysexcepthook_indentation_error, and
from test_dictcomps test_illegal_assignment.
Assisted-by: Claude Code:claude-opus-5-5
* Use the published Ruff fork 0.16.10 crates
Switch the rustpython-ruff_* dependencies from the git revision to the
0.16.10 crates.io release, published from the same commit (8b59b85344).
Assisted-by: Claude Code:claude-opus-5-5
---------
Co-authored-by: zzarbttoo <qodbwls70@naver.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Bas Schoenmaeckers
|
d81b7c2cb1 |
Add basic c-variadic functions to the c-api (#8933)
* Add basic c-variadic functions to the c-api * Review |
||
|
Jeong, YunWon
|
1f83b1593f |
Fix CI failures and bump version to 0.6.1 (#8940)
* Use AtomicPyTypeFlags in PyType_FromSlots PyType_FromSlots still imported the old PyAtomicTypeFlags name. Assisted-by: Grok:4.7 * Pass CPython test args as a bash array Assisted-by: Grok:4.7 * Bump workspace version to 0.6.1 Assisted-by: Grok:4.7 * Restore the CPython test step in ci.yaml Assisted-by: Grok:4.7 |
||
|
Jeong YunWon
|
e1dba29fb2 |
Revert "Drop parking_lot_core git patch and tkinter deps"
Tag 0.6.0 keeps that removal. This restores the parking_lot_core git patch and the tkinter dependencies on main. Assisted-by: Grok:4.7 |
||
|
Jeong YunWon
|
b2626c56de |
Drop parking_lot_core git patch and tkinter deps
Comment out the parking_lot_core git patch and use the crates.io crate. Remove tcl-sys, tk-sys, and the tkinter feature. Assisted-by: Grok:4.7 |
||
|
Jeong, YunWon
|
1d7d704414 |
Use bitflagset for type flags (#8873)
* Use bitflagset for type flags PyTypeFlags is defined with bitflagset! and the atomic tp_flags word with atomic_bitflagset!, replacing the hand-written PyAtomicTypeFlags. Assisted-by: Grok:4.7 Assisted-by: Claude:claude-opus-5-5 * Use bitflagset 0.0.4 from its git main branch bitflagset 0.0.4 is not on crates.io yet. Patch it to the git main branch in the workspace and in the example projects that depend on rustpython-vm, since each of them is a separate workspace. Assisted-by: Claude:claude-opus-5-5 * Depend on crates.io bitflagset 0.0.4 bitflagset 0.0.4 is published. Drop the git patches from the workspace and the separate example-project workspaces. Assisted-by: Grok:4.7 * Keep empty patch tables in example workspaces The example workspaces already have an empty [patch.crates-io] table. Dropping the bitflagset git line leaves that table in place. Assisted-by: Grok:4.7 * Call bitflagset methods directly for type flags Drop the plain-set has_feature wrapper and the intersects, is_created_with_flags, DEFAULT, and heap_type_flags aliases. Single-bit updates use insert, and flag combinations use from_slice. Assisted-by: Grok:4.7 * OR paired type flags in one update HAS_WEAKREF with MANAGED_WEAKREF, and HAS_DICT with MANAGED_DICT (and INLINE_VALUES when present), are applied with one |=. Assisted-by: Grok:4.7 * Store heap type flag sets as constants HEAPTYPE | BASETYPE, and the dict and immutable variants built from it, are const values instead of from_slice calls at each use. Assisted-by: Grok:4.7 * Test SEQUENCE and MAPPING in one load PyTypeFlags::COLLECTION is the two-bit mask. Callers check it with one is_disjoint instead of two contains loads. Assisted-by: Grok:4.7 * Fix clippy 1.99 lints in the vm Pass closure arguments by value where clippy reports a needless borrow, drop the unfulfilled redundant_else expectation on sum, and return the HRESULT after todo!(). Assisted-by: Grok:4.7 |
||
|
Alessio Attilio
|
65647e99ee |
fix(csv): quote non-numeric objects in writer QUOTE_NONNUMERIC mode (#8843)
* fix(csv): quote non-numeric objects in writer QUOTE_NONNUMERIC mode * style: format extra_tests/snippets/stdlib_csv.py Assisted-by: Antigravity:Gemini-3.8-Flash |
||
|
Jeong, YunWon
|
be23b2c562 |
Store database docstrings xz-compressed behind an optional doc feature (#8836)
* Skip Python-defined classes when collecting builtin type docs generate.py recorded every object.__subclasses__() entry as builtins.<name> after importing the whole stdlib, so classes written in Python (tkinter, unittest, tarfile, ...) ended up in the doc database. Classes that are mutable heap types without C method descriptors and whose module is not a C extension are now skipped, and their entries are removed from data.inc.rs. Assisted-by: Claude:claude-opus-5-5 Assisted-by: Grok:grok-4.7 * Store database docstrings xz-compressed behind an optional doc feature - rustpython-doc's build script compresses all database docstrings into one xz blob with an offset index; identical texts share one offset. - Macros emit blob references for database docs; vm decompresses the blob on first access. - New `doc` cargo feature (default on) in rustpython, rustpython-vm and rustpython-stdlib. Without it the blob and xz are not linked and database-backed __doc__ is None; __text_signature__ is unaffected. - Native types without a docstring get __doc__ = None in their dict. - Workflows that build with --no-default-features enable `doc` where tests or releases need docstrings. Assisted-by: Claude:claude-opus-5-5 Assisted-by: Grok:grok-4.7 |
||
|
Joshua Megnauth
|
86c7fa20c4 |
Fix missing workspace lints & fix lints (#8781)
Some of the workspace packages were missing the `[lints]` table. I enabled the lints then fixed the resulting errors. |
||
|
dependabot[bot]
|
94298caaf2 |
build(deps): bump zlib-rs from 0.6.7 to 0.6.8 (#8772)
Bumps [zlib-rs](https://github.com/trifectatechfoundation/zlib-rs) from 0.6.7 to 0.6.8. - [Release notes](https://github.com/trifectatechfoundation/zlib-rs/releases) - [Changelog](https://github.com/trifectatechfoundation/zlib-rs/blob/main/docs/release.md) - [Commits](https://github.com/trifectatechfoundation/zlib-rs/compare/v0.6.7...v0.6.8) --- updated-dependencies: - dependency-name: zlib-rs dependency-version: 0.6.8 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Jeong, YunWon
|
7b018871a3 |
host_env: export remaining stdlib constants (#8760)
* host_env: export winsound, spawn, and remaining SSL_OP flags Assisted-by: Claude * stdlib: take winsound, nt spawn, and SSL_OP names from host_env Assisted-by: Claude * host_env: export ssl alerts, sysexits, socket extras Assisted-by: Claude * stdlib: take remaining host_env and common constants Assisted-by: Claude * Remove unused workspace exitcode dependency Assisted-by: Claude * host_env: export PROTO, access, reparse, wait flags Assisted-by: Claude * stdlib: take PROTO, access, reparse, wait from host_env Assisted-by: Claude |
||
|
Jeong, YunWon
|
e0df2a10f0 |
Update malachite to 0.12 (#8755)
malachite-bigint 0.12.0 was published while the workspace pinned 0.11.0. pymath requires malachite-bigint = "0", so a fresh resolve picks 0.12 for it and leaves the workspace on 0.11, putting two incompatible BigInt types in one graph. example_projects carry no lockfile and resolve fresh. malachite-q and malachite-base move with it. No source change is needed. Assisted-by: Grok:grok-4.6 |
||
|
Jeong, YunWonandLee Dogeon
|
d02d81e740 |
perf: VM and runtime hot paths from pyperformance (#8737)
* time: use CLOCK_UPTIME_RAW for monotonic/perf_counter on Apple
CLOCK_MONOTONIC keeps advancing during system sleep on macOS, unlike
CPython's mach_absolute_time()-backed perf_counter/monotonic. Benchmarks
that spanned a laptop sleep recorded thousands of seconds for a single
iteration (pyperformance float/fannkuch showed ~1000x slowdowns).
Use CLOCK_UPTIME_RAW (equivalent to mach_absolute_time) on Apple targets
and report it as such in time.get_clock_info().
Assisted-by: Claude Code:claude-fable-5-1
* vm: derive f_lineno lazily instead of tracking prev_line per instruction
The dispatch loop wrote prev_line (a locations lookup plus a Cell store)
on every instruction only so f_lineno could read it. f_lineno now derives
the line from lasti and code.locations, prev_line is only maintained on
the cold tracing path for 'line' event de-duplication, the two use_tracing
reads per instruction are merged into one, and the pre-dispatch lasti
reload is replaced by the value just stored.
When a trace function is installed on an already-running frame (bdb
stepping back into an untraced caller, frame.f_trace assignment, a local
trace returned from the 'call' event on a resumed generator) prev_line is
synced from lasti so no spurious 'line' event fires.
Assisted-by: Claude Code:claude-fable-5-1
* vm: coalesce per-call thread-locals into one FRAME_SLOT_CACHE
set_current_frame ran on every frame enter/exit and touched three separate
thread_local! statics (each a _tlv_get_addr call on macOS). Merge them into
a single thread-local struct so the hot path does one TLS access.
Assisted-by: Claude Code:claude-fable-5-1
* vm: fold stop/finalizing checks into EVAL_BREAKER
eval_breaker_tripped ran three checks per bytecode: the finalizing flag,
a thread-local stop_requested lookup, and the EVAL_BREAKER load. Add
STOP_BIT and FINALIZING_BIT so the per-instruction check is a single
relaxed atomic load; the slow path (check_signals) still re-derives the
exact per-thread answer. The bits are set alongside the per-thread flags
and cleared under the stop-the-world exclusion.
Assisted-by: Claude Code:claude-fable-5-1
* gc: set the TRACKED bit with a plain store for freshly allocated objects
gc_bits is zero for a fresh allocation (both the freelist reuse path and
PyInner::new zero it), so track_new_object can initialise the bit with a
relaxed store instead of a fetch_or. Re-tracking resurrected objects keeps
the read-modify-write variant.
Assisted-by: Claude Code:claude-fable-5-1
* vm: skip thread exception publish when pushing/popping a None exc slot
Generator/coroutine resume pushes and pops an exc_info slot on every
send. When that slot is None the value returned by topmost_exception
cannot change (it skips None entries), so the thread-local exception
publish in push_exception/pop_exception is unnecessary.
Assisted-by: Claude Code:claude-fable-5-1
* vm: allocation-free keyword-call fast path and super() MRO walk
Keyword calls to plain functions where every parameter is filled exactly
once are reordered in place into the fast-locals layout (no IndexMap, no
keyword name clones); any mismatch falls back to the existing slow path
so error messages are unchanged. super().attr now walks the type's MRO
by reference instead of cloning every class into two Vecs per lookup.
Assisted-by: Claude Code:claude-fable-5-1
* builtins: cheaper enumerate and zip iteration
enumerate kept its counter as a BigInt behind a write lock and did bignum
arithmetic on every item; use a machine integer that promotes to BigInt
only for huge/negative starts or on overflow. zip pre-sizes its per-item
Vec so the common two-iterable case allocates once.
Assisted-by: Claude Code:claude-fable-5-1
* _asyncio: cache module objects and use native set/dict operations
Task registration re-resolved _asyncio module attributes and re-imported
contextvars on every task; the eager path additionally went through generic
call_method for set add/discard. Cache the module objects per thread
(validated by identity), split the contextvars cache from the task
containers, and use PySet/PyDict methods directly.
Assisted-by: Claude Code:claude-fable-5-1
* perf: use mimalloc as global allocator for rustpython CLI
Adds mimalloc (0.1.52, default-features = false) as the #[global_allocator]
for the rustpython binary, gated behind a new `mimalloc` feature enabled
by default on non-wasm targets (dep declared under
target.'cfg(not(target_arch = "wasm32"))'.dependencies, so wasm builds
never pull it in regardless of default-features).
Profiling a `t = (a, b, c)` loop showed malloc/free at ~40% of samples on
macOS with the system allocator. Switching to mimalloc gives consistent
wins across small-object-heavy workloads, medians over 5-9 interleaved
A/B rounds (release build, this machine):
tuple (5M `(a,b,c)`) 1.020s -> 0.863s (1.18x)
4-thread tuple (5M total) 1.643s -> 1.382s (1.19x)
class instances (2M) 0.874s -> 0.797s (1.10x)
dict-heavy (200k x 20) 2.196s -> 1.992s (1.10x)
string-heavy (join x 20) 0.662s -> 0.592s (1.12x)
nqueens (pyperformance) 0.327s -> 0.286s (1.14x)
chaos (pyperformance) 0.204s -> 0.191s (1.06x)
richards (pyperformance) 0.154s -> 0.151s (1.02x)
go (pyperformance) 0.411s -> 0.405s (1.01x)
listcomp / fib ~neutral (<=1%)
Peak RSS (/usr/bin/time -l): instances workload roughly unchanged
(29.0MB -> 29.1MB); dict-heavy workload actually drops (155MB -> 106MB),
likely because mimalloc reclaims freed pages more aggressively than the
system allocator across repeated large-dict churn.
`cargo build` and `cargo build --release` are warning-free with the
default feature set, and `cargo build --release --no-default-features
--features stdlib,threading,importlib,stdio,host_env` (mimalloc
excluded) also builds cleanly. CI's wasm32-wasip1 and other
--no-default-features build matrices in .github/workflows already omit
`mimalloc` from their feature lists and target wasm via the
cfg(not(wasm32)) dependency gate, so no workflow changes needed.
`rustpython -m test -j4 test_gc test_weakref test_threading test_dict
test_list test_tuple test_str test_bytes test_memoryview test_ctypes
test_mmap test_array` passes identically (2541 tests, 241 skipped) on
both the baseline and mimalloc binaries.
Assisted-by: Claude Code:claude-fable-5-1
* vm: cap dict/set table shrink and compact holes on resize
asyncio's eager task path (and its per-task bookkeeping sets in general)
repeatedly adds and discards distinct task objects from a handful of
long-lived sets (_eager_tasks, _scheduled_tasks). Two dict_inner issues
turned that steady-state churn into the dominant cost of async_tree_eager:
1. resize() picked a target size of `used * 2` with no floor, so once a
churny set settled around 1-2 live entries it shrank to a table that
the very next insert would already overflow, forcing a resize on
almost every single add. Clamping to the original minimum table size
(8) restores the headroom dict/set start with.
2. `entries` only ever grows (holes from deletions are never reclaimed),
so resize()'s rebuild pass scanned every entry ever inserted into the
dict's lifetime, not just the live ones - a set with a stable working
set size but heavy churn grew this scan, and the dict's memory, without
bound. resize() now compacts holes out of `entries` first, matching
CPython's compacting hash table and keeping the shape-stamp fast path
(which already requires a hole-free layout) reachable more often.
Measured with a standalone async_tree micro-benchmark (6 levels x 6
branches, pyperformance's bm_async_tree "none"/"eager" shape) run inside
one process for 10 asyncio.run() iterations, comparing medians of
interleaved binary runs on an otherwise idle machine:
- eager: ~2.91s -> ~0.86s median (about 3.4x)
- none: ~1.98s -> ~1.23s median (about 1.6x)
A single fresh-process run (no in-process accumulation) still improves
by roughly 15-20% for eager, since the resize-storm from point 1 hits
even a single run's worth of task churn.
test_asyncio (34 files, all passing), test_dict, test_set, test_weakref,
test_weakset, test_ordered_dict, test_gc, test_descr, test_class, and
`cargo test -p rustpython-vm -p rustpython-stdlib` all pass.
Assisted-by: Claude Code:claude-fable-5-1
* _functools: add native _lru_cache_wrapper
RustPython had no native _functools._lru_cache_wrapper, so
functools.lru_cache always fell back to the pure-Python implementation
in Lib/functools.py (RLock, linked-list bookkeeping, closures). That
fallback dominates typing.Protocol's __instancecheck__ path: it calls
inspect.getattr_static, whose _shadowed_dict is itself lru_cache'd, so
every isinstance() check against a runtime-checkable protocol paid for
several pure-Python cache-wrapper calls plus lock acquisitions.
Add a native pyclass mirroring CPython's C _lru_cache_wrapper: a
reentrant lock scoped only around the cache lookup/insert steps (not
held across the wrapped call, matching CPython's threading behavior),
LRU ordering piggybacked on the cache dict's own insertion order
(move-to-end via del+reinsert, oldest evicted from the front) so no
separate key-comparison structure is needed, and __copy__/__deepcopy__
returning self to match CPython since __reduce__ (used only for
pickling) requires __qualname__, which wrapped non-function callables
such as functools.partial don't have.
Measured with a standalone script reproducing pyperformance's
bm_typing_runtime_protocols isinstance() loop (300 loops, medians of
5 interleaved runs): 0.260s -> 0.182s, about 1.43x faster. In
isolation, a cache-hit through lru_cache went from ~0.31s to ~0.07s
for 200k calls, now faster than calling the underlying function
uncached (matching CPython's C accelerator).
Verified with test_typing, test_abc, test_inspect, test_functools,
test_weakref, test_isinstance, and test_descr (all pass, matching the
pre-existing baseline, including test_functools's already-present
TestLRUC suite which now actually exercises the native implementation).
Assisted-by: Claude Code:claude-fable-5-1
* sre_engine: ASCII fast path for \d \s \w unicode category checks
is_uni_digit/is_uni_space/is_uni_alnum went through icu_properties'
GeneralCategory table lookup for every character, even the common
ASCII case. Add a `ch < 0x80` fast path that resolves directly via
std's ascii predicates, falling back to the full table only for
non-ASCII code points (Latin-1 supplement chars like 'A-umlaut' still
route through the table since they are alphanumeric but not ASCII).
Measured on a tight `re.findall(r"\w+", ...)` / `re.sub` loop over
mostly-ASCII text (interleaved A/B, 4 rounds, medians):
re.sub: 1.178s -> 1.134s (~3.7% faster)
re.findall: 1.110s -> 1.062s (~4.3% faster)
Assisted-by: Claude Code:claude-fable-5-1
* vm: skip VM attach for no-op generator/coroutine __del__
Generator, coroutine, and async-generator dealloc always went through
drop_slow_inner's call_slot_del, which unconditionally attaches to a VM
(with_vm -> begin_interpreter_section -> per-interpreter thread-slot
hashmap lookup) before running del(), even though del() itself is a
documented no-op once the object is already closed or running - the
common case for a generator/coroutine consumed to completion.
Add PyTypeSlots::del_needed, an optional VM-free fast predicate checked
before call_slot_del/with_vm; generator, coroutine, and async_generator
install a check mirroring their own del() no-op branches, letting
drop_slow_inner and try_call_finalizer skip the VM attach entirely for
finished objects.
Standalone microbenchmarks (median of 5 interleaved runs):
- create+drain a 3-yield generator, 2M times: 1.099s -> 0.997s (-9%)
- pyperformance bm_generators shape (yield-from tree traversal): -4%
- pyperformance bm_coroutines shape (recursive fibonacci coroutines): -13%
Assisted-by: Claude Code:claude-fable-5-1
* _asyncio: skip full __import__ dispatch on module-cache hot path
with_asyncio_cache/with_contextvars_cache called vm.import("_asyncio"/
"contextvars", 0) on every Task/Future registration just to validate
their module-identity cache, even on the (overwhelming) hit path. That
still pays the full __import__ builtin dispatch (attribute lookup on
builtins, FuncArgs construction, native slot_call, ImportArgs parsing,
import_module_level) instead of a cheap sys.modules lookup. Added
fast_import_cached(), which checks sys.modules directly and only falls
back to vm.import on a miss (module missing/reloaded).
Measured with a standalone asyncio_tree-style microbenchmark
(6 levels x 6 branches, several asyncio.run() iterations per process,
median of 5 interleaved A/B rounds):
none: 1.045s -> 0.895s (~14% faster)
eager: 0.716s -> 0.591s (~17% faster)
eager_tg: 1.023s -> 0.849s (~17% faster)
test_asyncio (all files), test_contextvars (module absent from this
checkout, pre-existing), test_weakref, test_weakset all pass/unchanged.
Assisted-by: Claude Code:claude-fable-5-1
* vm: skip VM-attach section in with_vm when already on the current interpreter
Object teardown (drop_slow_inner -> call_slot_del, try_call_finalizer,
weakref callbacks) calls with_vm(), which unconditionally went through
set_current_vm -> begin_interpreter_section: an INTERP_THREAD_SLOTS
HashMap lookup plus an Arc clone, even in the overwhelmingly common
case where the dropping thread is already attached to the exact
interpreter that owns the object (a __del__ object or weakref target
going out of scope mid-bytecode).
with_vm now checks whether the object's owning interpreter is already
the one on top of VM_STACK. Only the top of VM_STACK is ever ATTACHED
on a thread (every push is preceded by an attach, every pop restores
the enclosing attach), so when it matches we call f(vm) directly and
skip begin_interpreter_section/set_current_vm entirely. The rare
not-attached/different-interpreter cases (nested subinterpreters,
thread outside any VM, shutdown) fall through to the original
attach/detach path unchanged.
Interleaved release-build measurements (5 rounds, medians), plain-Rust
micro-scripts, no pyperf:
- 2M create+drop of a __del__ object: ~0.55s -> ~0.46s (~16% faster)
- 2M create+drop with a weakref callback: ~1.14s -> ~1.06s (~7% faster)
- 2M io.StringIO() create+drop: ~0.734s -> ~0.697s (~5% faster)
- fib(30) and a 5M-iteration tuple-unpack loop: unchanged (noise-level)
`sample` profiles of the __del__ benchmark confirm
begin_interpreter_section/ensure_thread_slot frames disappear after
the change.
Verified: `cargo test -p rustpython-vm --release`, and
`-m test -j4 test_gc test_weakref test_finalization test_threading
test_thread test_concurrent_futures test_os test_signal test_generators
test_io test_atexit test_sys` (test_interpreters module absent in this
tree, pre-existing). Also a 4-thread stress script hammering __del__
object creation against a concurrent gc.collect() loop for 10s, and a
threading+queue handoff dropping a __del__ object after cross-thread
transfer, both pass.
Assisted-by: Claude Code:claude-fable-5-1
* vm: general sys.modules fast path for vm.import
VirtualMachine::import always resolved and called builtins.__import__
(FuncArgs construction, native slot_call, ImportArgs::from_args, then
import_module_level) even when the module was already cached in
sys.modules. ~63 Rust stdlib call sites hit this on hot paths
(copyreg, pickle's reduce_newobj, warnings, asyncio, etc).
Add VirtualMachine::try_import_cached: for a plain absolute import
(level 0, empty from-list), if builtins.__import__ is still the
original import function (compared by identity against vm.import_func)
and the module -- or, for a dotted name, its top-level package -- is
already present and fully initialized in sys.modules, return it
directly, skipping the __import__ dispatch and import_module_level
entirely. Falls through to the existing slow path whenever the module
is uncached, still initializing, or __import__ has been overridden by
user code, so test_import/test_importlib/test_builtin overrides still
work.
Exposed import::is_module_initializing (refactored out of
import_ensure_initialized) so the fast path can check
__spec__._initializing the same way the slow path does.
Removes _asyncio's local fast_import_cached workaround (added in
cb2064f0e) now that vm.import itself has the same fast path generally.
Measured (interleaved A/B, 5 rounds, pickle.dumps of a plain object,
which hits copyreg import via reduce_newobj): ~2% wall-clock
improvement per call (baseline ~4.75s / 100k dumps vs ~4.66s
candidate), consistent across all rounds. Verified: overridden
__import__ still invoked, a replaced sys.modules is read through the
same accessor, and a self-importing/initializing module is not handed
out early -- all match baseline behavior.
test suite (test_import test_importlib test_builtin test_pkg
test_pkgutil test_runpy test_zipimport test_site test_warnings
test_copy test_pickle test_asyncio.test_tasks test_threaded_import
test_sys): 13/14 files pass; test_threaded_import fails identically on
baseline (ModuleNotFoundError, pre-existing, unrelated to this change).
Assisted-by: Claude Code:claude-fable-5-1
* _asyncio: bypass generic context.run dispatch in task_eager_start
task_eager_start ran the first step via context.run(coro.send, None),
which builds a bound `send` method object and then re-enters the
generic Callable/FuncArgs dispatch just to invoke context.run's own
generic call. Since eager tasks in async_tree_eager complete almost
entirely synchronously (no event-loop scheduling needed), this
per-task dispatch overhead dominated. Enter/exit the native PyContext
directly and call coro.send via a single call_method instead,
falling back to the generic path for non-native Context subclasses.
async_tree_eager median: ~705ms -> ~697ms across 5 interleaved A/B
rounds (~1% eager-only win); async_tree (none) unaffected (noise-level
delta). test_asyncio (all files), test_weakref, test_weakset pass;
pre-existing test_contextvars module-not-found failure unchanged from
baseline.
Assisted-by: Claude Code:claude-fable-5-1
* vm: drop scopeguard::defer! from generator/coroutine resume hot path
resume_gen_frame restored owner/exc-slot/frame-chain state via a
scopeguard::defer! closure that ran on every exit (Ok, Err, or panic).
Since the closure's body is the function's only tail expression, a
plain captured result plus explicit cleanup (the same shape with_frame
already uses for the analogous call path) covers both normal exit
paths identically; only the recursion-depth decrement keeps a
scopeguard, matching with_frame's precedent that this codebase does
not otherwise guarantee generator/frame state consistency across a
Rust panic (panic=unwind is not used to implement any Python-level
control flow here).
Delegation-chain benchmark (chain.py, forwarding N values through a
yield-from chain) measures a small but consistent win: ~2-5% less
wall time per forwarded value at depths 1/4/16 across interleaved A/B
rounds. Single-generator create+finish cost is a wash.
Assisted-by: Claude Code:claude-fable-5-1
* vm: run generator bodies through the call trampoline
`Py<FrameObject>::resume` built its `ExecutingFrame` with
`tailcall_enabled: false`, because a `TailCall` result had nowhere to go:
the trampoline was reachable only from `run_frame_fast`, whose entry frame
lives on the data stack and can never yield. So every ordinary Python call
made from inside a generator or coroutine body re-entered `run_frame_fast`
recursively instead of being flattened.
Generalize the trampoline to drive a heap-resident entry frame whose
enter/exit bookkeeping belongs to its caller, and whose `Yield` is a valid
terminal result rather than a panic. `FrameKind` now says, per frame, what
the trampoline owes it: `Entry` (exit it, caller frees the storage),
`GenEntry` (`resume_gen_frame` linked it and will unlink it; a `Yield` is
the trampoline's result) or `Callee` (exit it and free its storage). The
three near-identical dispatch arms collapse into one `Step::Finished` /
`Step::Deliver` loop.
The suspended-frame stack moves onto the VM and is reused across
invocations: a generator body enters the trampoline once per resume, so a
`Vec::with_capacity(8)` there cost a malloc/free per yield (+27 ns/resume
when first measured). Invocations nest strictly LIFO, so each owns the
region above the length it finds on entry.
Neutral on its own — a generator body's first call was the only one not
already flattened, since the callee's own frame runs under the trampoline
either way. It is what lets SEND/FOR_ITER hand a generator frame to the
same loop. Measured (chain of `yield from`, 1-yield generator create+finish,
bm_generators, bm_coroutines, async_tree none/eager): within noise, except
generator create+finish 625 ns -> 603 ns.
Assisted-by: Claude Code:claude-fable-5-1
* vm: run a `yield from` chain in one eval loop
`SEND` on a suspended generator recursed: `Coro::send` -> `resume_gen_frame`
-> a fresh `ExecutingFrame::run` per level, and the value came back the same
way. Measured at ~55 ns per level of delegation, against CPython's ~5.8 ns.
Only about 5 ns of that is the resume bookkeeping itself (measured by
stubbing out the claim, the exception slot, the owner swap and the recursion
checks). The rest is what a delegating frame does per value: leave and
re-enter the eval loop, and dispatch `RESUME`, `JUMP_BACKWARD_NO_INTERRUPT`,
`SEND` and `YIELD_VALUE` — four instructions that, in the canonical
`yield from` / `await` shape, only hand a value along.
So the trampoline now runs the chain itself. `SendGen` parks its frame and
returns `ExecutionResult::GenResume`; the trampoline claims and links the
sub-generator (`coroutine::flat_resume_enter`, the first half of what
`Coro::send` does around a resume) and runs its frame in the same loop. On
the way down, every frame it finds already suspended at a `yield from` whose
delegate can be resumed is parked without being run at all
(`yield_from_delegate`); on the way back up, a yielded value is re-yielded
for each such frame by parking its `lasti` past the `YIELD_VALUE`
(`park_after_yield_from`) instead of running it. A whole chain therefore
costs one claim/link/unlink per level and no instruction dispatch at all,
with `Py<FrameObject>::resume` extending the same treatment to the outermost
level, which is where `next()` and asyncio's task step enter a chain.
Each level is still claimed, linked and unlinked exactly as a recursive
`Coro::send` would, so `gi_running`, `f_back`, `gi_frame`, `gi_yieldfrom`,
`sys._getframe`, tracebacks, `throw`, `close`, PEP 479 and the
`RecursionError` depth are unchanged; `resume_gen_frame` and the flat path
share `gen_frame_link`/`gen_frame_unlink` and the invariants are documented
on `FlatResume`. Everything else keeps the recursive path: tracing or
monitoring on (both would see the skipped instructions), `throw`/`close`,
async generators, subclasses, non-generator iterators, `FOR_ITER`, a `SEND`
that is not a `yield from`, and a delegate that has not started or is not
itself delegating — the last two because one lone level costs more to hand
to the trampoline than to send into from the eval loop the caller is in.
Measured (median of 5, ns/value or seconds):
chain of `yield from`, depth 16 911 -> 479 -47%
chain of `yield from`, depth 4 212 -> 158 -26%
bm_generators 0.272 -> 0.187 -31%
richards 0.772 -> 0.771 ~0
bm_coroutines 0.0129 -> 0.0130 +0.8%
async_tree none ~neutral
chain depth 1 (no delegation) 81.1 -> 83.6 +3.1%
fib(24), calls through trampoline 0.0180 -> 0.0183 +1.7%
Per level of delegation: 55.3 ns -> 26.4 ns (CPython 3.14: 5.8 ns).
Assisted-by: Claude Code:claude-fable-5-1
* vm: reset the oparg prefix when a signal unwinds a frame
An `EXTENDED_ARG` leaves its prefix in `arg_state` for the instruction that
follows it. The signal path in the eval loop unwinds to a handler and
`continue`s straight back to the top, skipping the loop's own
`arg_state.reset()`, so the handler's first instruction was decoded with the
interrupted instruction's prefix still applied. Every other path that jumps
somewhere new — the `set_f_lineno` restart just above — resets first.
Reachable whenever a signal arrives on the instruction right after an
`EXTENDED_ARG` in a frame that has a handler: a `KeyboardInterrupt` into a
`yield from` chain panicked with `InvalidBytecode` in roughly half of the
runs of a 16-deep chain, and now does not in ten out of ten.
Assisted-by: Claude Code:claude-fable-5-1
* bytecode: fetch a code unit with one 16-bit acquire load
The dispatch loop read the opcode with an Acquire `AtomicU8` load and the
arg byte with a separate Relaxed one. The acquire load sits between the
two, so LLVM had to re-load `self.code` and `code.instructions` for the
second access: four extra instructions and two extra dependent loads on
the instruction-fetch path of every bytecode instruction.
`CodeUnits::read_unit` reads both bytes as a single aligned Acquire
`AtomicU16`, which pairs with `replace_op`'s Release store exactly as
`read_op` did. `CodeUnit` gains `align(2)` so the 16-bit access is
guaranteed aligned rather than relying on the allocator.
5.91 -> 5.76 ns/bytecode-instruction on a straight-line
LOAD_FAST/STORE_FAST loop (-2.5%); range loop -2%.
Assisted-by: Claude Code:claude-fable-5-1
* vm: compute the cache-entry count after dispatch, not before
`op.cache_entries()` is a byte-table lookup used only to skip inline
cache units after an instruction falls through. Computing it before the
handler kept it live across the whole (very large) inlined match, so
every bytecode instruction paid an address materialization, the table
load, a stack spill and a reload of the count.
Moving it next to its only use lets it be computed on the fall-through
path with the value in a register.
Measured together with the following commit (the eval-breaker outline):
5.76 -> 5.55 ns/bytecode-instruction on a straight-line
LOAD_FAST/STORE_FAST loop, range loop -1.5%, fib(27) -2.5%.
Assisted-by: Claude Code:claude-fable-5-1
* vm: move the eval-breaker body out of the dispatch loop
The eval-breaker check itself stays inline on the per-instruction path
(moving it off regresses badly), but everything it guards -- the
finalizing/main-thread check, stop-the-world suspend, the QSBR
checkpoint, signal delivery and the scheduled GC -- was inlined into the
loop body too. That code is never executed on the hot path yet its
address materializations and live values competed for registers with
every opcode handler, so the loop head spilled more than it needed to.
`eval_breaker_work` is `#[cold] #[inline(never)]`, leaving a single
`ldrb`/`cbz` pair inline for the common "nothing pending" case.
Measured with the preceding commit: 5.76 -> 5.55 ns/bytecode-instruction
on a straight-line LOAD_FAST/STORE_FAST loop.
Assisted-by: Claude Code:claude-fable-5-1
* vm: always inline the eval stack push/pop primitives
`pop_value`, `pop_value_opt`, `top_value` and friends are a handful of
instructions each, but `ExecutingFrame::run` is ~50 KB of inlined opcode
handlers, so LLVM's inline budget refused them and emitted real calls:
STORE_FAST and POP_TOP each began with `bl ...pop_value`, which also
forced spills around the call. `#[inline(always)]` puts the four or five
instructions they actually are back into the handlers.
5.55 -> 5.29 ns/bytecode-instruction on a straight-line
LOAD_FAST/STORE_FAST loop (-4.8%); nbody -3.5%, fib(27) -3.5%,
`s += i` loop -3.5%, `for _ in range(n)` -4%.
Assisted-by: Claude Code:claude-fable-5-1
* vm: carry the instruction fetch state in loop locals
Instruction fetch went through memory three times per bytecode
instruction: `self.lasti` (a pointer field) was re-loaded, the program
counter was stored to it and read back at the top of the next iteration,
and `self.code` -> `code.instructions` was re-loaded to address the code
unit. LLVM cannot keep any of them in registers because the handlers
write through `&mut self`, so the fetch chain started with a
store-to-load round trip followed by two dependent pointer loads.
`lasti_cell` and `instructions` are now copied into locals once (both are
plain shared references with the frame's lifetime), and the program
counter is carried in `idx`: after a handler that did not jump the next
index is already known from `lasti_before` plus the cache count. The
frame's `lasti` is still stored before every instruction, so tracebacks,
`f_lineno`, signals and `sys._getframe` see exactly what they did; only
the read-back is gone. The paths that unwind (signal, exception handler,
`set_f_lineno`) re-read it, since they jump to a new position.
Also collapses the eval-breaker `if`, which clippy flagged after the
previous commit moved its body out of line.
range loop -3.3%, richards -3.1%, nbody -2.9%, `s += i` loop -2.2%,
go -1.7%, raytrace -1.2%; mean -1.1% over the 13-benchmark set. The
straight-line LOAD_FAST/STORE_FAST micro is unchanged at 5.28
ns/bytecode-instruction (it has no cache entries and never jumps, so it
was already the best case for the old code).
Assisted-by: Claude Code:claude-fable-5-1
* bytecode: hoist the instruction array base out of the dispatch loop
Addressing a code unit still went `self.code` -> `code.instructions` ->
the `Box<[CodeUnit]>` data pointer on every instruction, two dependent
loads before the acquire load that fetches the unit itself, because the
handlers write through `&mut self` and LLVM has to assume the fields
could change.
The instruction array is allocated once when the code object is built
and never reallocated -- specialization rewrites units in place -- so
`CodeUnits::units_ptr` may be read once before the loop and reused.
`read_unit_from` is the same 16-bit acquire load taken from that base;
`read_unit` now delegates to it, and the index is unchecked exactly as
it was before.
nqueens -1.9%, raytrace -1.6%, range loop -1.3%, deltablue -1.1%,
nbody -0.8%, dispatch micro -0.7%; mean -0.7% over the 13-benchmark set.
Assisted-by: Claude Code:claude-fable-5-1
* common: an immortal bit in the refcount word
Atomic reference counting is about a third of what a bytecode costs now
that the dispatch loop is trimmed: built `--no-default-features`, where
`PyAtomic` is a `Cell`, the straight-line dispatch micro runs at
3.5 ns/instruction against 5.4. Every `TO_BOOL`/`POP_JUMP_IF_*` pair and
every `None` or small-int push pays a relaxed `ldadd` and a `Release`
`ldaddl` on an object that can never die.
PEP 683's answer is to say so in the word. A fourth flag bit, `IMMORTAL`,
takes a bit from the strong count (leaving 60 on a 64-bit target, 28 on a
32-bit one, both far past what a program reaches); `make_immortal` sets it
and parks the count at CPython's `_Py_IMMORTAL_REFCNT`, and `inc`,
`inc_by`, `dec` and `safe_inc` read the bit and return without touching
the counter. The parked count is what keeps the rest of the tree honest
without a special case: it is far too large for a `strong_count() == 1`
in-place-mutation fast path to fire on, and `start_gc_refs` already reads
a count that large as `GC_REACHABLE`, so an immortal object is a permanent
root that keeps its referents alive.
Two details pay for the branch the mortal path now runs, which cost 4% on
object-heavy benchmarks before they went in:
* `leak` sets `IMMORTAL` alongside `LEAKED`. An interned object could
already never reach zero, so `dec` no longer needs the test it made on
the result of the decrement: the immortal test at the top decides both.
* `inc`'s three exceptional cases — a destructed word, a count at the
ceiling, a count at zero — fold into one comparison against the masked
word, with the handling moved to a cold helper.
What is left is one instruction on `dec` and roughly nothing on `inc`, and
the mortal path measures at parity.
Assisted-by: Claude Code:claude-fable-5-1
* vm: make the singletons, small ints, interned strings and static types immortal
Everything the `Context` builds once and keeps — `None`, `True`, `False`,
`Ellipsis`, `NotImplemented`, the empty tuple, frozenset, bytes and str,
the -5..256 int cache and the interned one-character strings — is now
marked immortal, as is every static type object, which its `static_cell`
holds for the life of the process. So is every entry of the string pool:
`mark_intern` already stopped those refcounts from reaching zero, so
immortality frees no memory that would otherwise have been freed; it only
takes the atomic read-modify-write off every incref and decref of an
attribute name, a dict key or a docstring.
Straight-line dispatch micro (200 LOAD_FAST/STORE_FAST pairs over cached
ints, median of 5 interleaved runs): 5.39 -> 4.01 ns/instruction, -26%.
A four-thread `if flag:` loop, where the contention on the shared bool was
real: 2.36 s -> 2.14 s, -10%.
pyperformance bench functions run standalone, medians of 5 interleaved
rounds against a matched build: fannkuch -7.8%, nqueens -3.8%, chaos
-3.0%, raytrace +1.0%, go +1.2%, richards +4.2%, nine of them together
-2.9%. Per-benchmark deltas under about 4% sit inside this tree's
build-to-build ThinLTO layout noise, which reaches that size between two
builds of identical source.
Observable: `sys.getrefcount` of an immortal object reports 4294967295,
CPython's `_Py_IMMORTAL_REFCNT`, and stops moving — what
`test_sys.test_refcount` expects of `None`. `is_interned`, and the dict
pointer-equality key fast path behind it, keep answering from the separate
`LEAKED` bit.
Assisted-by: Claude Code:claude-fable-5-1
* codecs: fast-path utf-8 decode, skipping the codec registry round trip
bytes.decode('utf-8') / str(bytes, 'utf-8', ...) always went through the
generic codec registry: a per-call attribute lookup for
_is_text_encoding, then a full call into the pure-Python
encodings.utf_8.decode wrapper, which called back into the native
utf_8_decode and unpacked a 2-tuple result.
CPython's PyUnicode_Decode special-cases a handful of built-in encoding
names (utf-8 among them) and decodes them directly, bypassing the codec
registry entirely -- a codecs.register() override does not affect
str(b"...", "utf-8") there either. This adds the same fast path here:
recognized utf-8 aliases go straight to the native decoder.
Measured with a pure-Python `str(bytes, 'utf-8', 'surrogatepass')` loop:
~723ns -> ~401ns per call (1.8x). On the pyperformance unpickle
benchmark (which decodes short strings this way for every
SHORT_BINUNICODE opcode): ~14.4ms -> ~13.2ms per loop, an ~8.3% win.
Assisted-by: Claude Code:claude-fable-5-1
* codecs: fast-path utf-8/ascii/latin-1 encode and ascii/latin-1 decode
Extend the utf-8 decode fast path from 3fe684629 to the symmetric encode
direction (str.encode()/bytes(s, enc)), and add the same registry-skipping
fast path for ascii and latin-1 encode/decode, all of which are simple
native codecs.rs functions already. For utf-8 encode with no surrogates in
the string (the common case), encoding is just a copy of the string's
existing UTF-8 bytes, so that's the fastest path; strings containing
surrogates and non-strict error handlers still go through the native
encoder + ErrorsHandler, matching custom codecs.register_error handlers.
Measured (median of interleaved runs vs baseline, 2M iterations):
str.encode("utf-8"): 351ns -> 195ns (-44%)
str.encode(): 449ns -> 291ns (-35%)
bytes(s, "utf-8"): 542ns -> 382ns (-30%)
str.encode("ascii"): 436ns -> 302ns (-31%)
str.encode("latin-1"): 487ns -> 333ns (-32%)
bytes.decode("ascii"): 519ns -> 359ns (-31%)
bytes.decode("latin-1"):590ns -> 417ns (-29%)
bytes.decode("utf-8")/str(b,"utf-8"): unchanged (already fast-pathed)
Assisted-by: Claude Code:claude-fable-5-1
* vm: make the runtime safe for borrowed stack entries
`PyStackRef` has carried a borrow tag since the stack was unified with the
fastlocals, but only `LOAD_SMALL_INT` -- which borrows immortal `Context`
objects -- ever set it, so nothing in the runtime had to be right about
borrowed entries. Turning `LOAD_FAST_BORROW` into a real borrow makes three
places load-bearing, and this fixes them ahead of the switch.
`Traverse for PyStackRef` handed every entry to the tracer, borrowed ones
included. The cycle collector subtracts one from an object's `gc_refs` per
edge it is shown, so a borrowed entry -- which owns no strong count -- would
have pushed a live object's count to zero and freed it out from under the
frame. Borrowed entries are now skipped; nothing becomes unreachable, because
the fastlocals slot the entry borrows from is on the same traversal.
`park_after_yield_from` stands in for a `YIELD_VALUE` that never runs, so the
`promote_stack()` in that handler never ran either; it now promotes there too,
and the debug assertion after it stops being vacuous.
`STORE_FAST`, `DELETE_FAST` and the fused stores now call
`LocalsPlus::debug_audit_local_release`, which in debug builds panics if the
slot is about to drop the last count of an object a borrowed entry still
points at -- the one shape codegen's `optimize_load_fast` exists to prevent,
and the one that would be a use-after-free if the analysis or a modelled stack
effect were wrong.
`BORROW_LOCAL_LOADS` (off) is the switch for the borrow itself, and
`LOAD_FAST_BORROW`/`LOAD_FAST_BORROW_LOAD_FAST_BORROW` now go through
`push_local`, which is a plain clone while it is off. The invariant the switch
depends on is written out where `PyStackRef` is defined.
No measurable change: with the switch off the only behavioral difference is
the traversal fix, which no current bytecode can reach.
Assisted-by: Claude Code:claude-fable-5-1
* vm: read consumed stack entries in place instead of owning them
`pop_value()` turns whatever is on the stack into an owned `PyObjectRef`, so
an instruction that only reads its operand still pays a reference count round
trip -- and, once `LOAD_FAST_BORROW` starts pushing borrows, would pay an
atomic increment to promote a borrow it is about to throw away. This moves
every consumer that never needs ownership onto `pop_stackref()`, which hands
back the tagged entry and costs nothing for a borrow.
Discarding pops (62 of them, across the `LOAD_ATTR`, `TO_BOOL`, `BINARY_OP`,
`COMPARE_OP`, `CONTAINS_OP` and `CALL` specializations, which read their
operands through `top_value()`/`nth_value()` and then drop them) become
`pop_stackref()`. So do the read-only operands of `POP_TOP`, `POP_JUMP_IF_*`,
`TO_BOOL`, `BINARY_OP`, `COMPARE_OP`, `CONTAINS_OP`, `IS_OP`, `STORE_ATTR`,
`DELETE_ATTR`, `STORE_SUBSCR`, `DELETE_SUBSCR` and `UNPACK_SEQUENCE`.
Two protocol entry points had to move from an owned receiver to `&PyObject` to
make that reachable: `try_to_bool`, which never used the ownership it took,
and `rich_compare`, which forwards to the `&self` `_cmp` anyway. Both keep
working unchanged for `PyObjectRef` callers through deref.
`LOAD_ATTR`'s method specializations popped the receiver and pushed it
straight back below the bound function; they now move the entry across with
its tag intact, which is how CPython keeps a method call's receiver borrowed
from `LOAD_ATTR` all the way through `CALL`. The plain attribute read reads
through the receiver rather than owning it.
Neutral while `BORROW_LOCAL_LOADS` is off -- for an owned entry
`pop_stackref()` and `pop_value()` compile to the same thing.
Assisted-by: Claude Code:claude-fable-5-1
* vm: turn LOAD_FAST_BORROW into an actual borrow
`LOAD_FAST_BORROW` accounts for 21.5% of executed instructions on
nbody+richards+nqueens, and until now it did exactly what `LOAD_FAST` does:
an atomic increment on the way onto the stack and, from the instruction that
consumed it, an atomic release decrement on the way off. Neither is needed.
The fastlocals slot holds a count for as long as the entry is on the stack --
codegen's `optimize_load_fast` only rewrites a `LOAD_FAST` when it can prove
the entry is consumed before anything can release that slot -- so the entry
can be a borrow that owns nothing, which is what CPython 3.14 does with a
tagged `_PyStackRef`.
Flipping `BORROW_LOCAL_LOADS` on is the whole change; the two preceding
commits are what make it safe (cycle-collector traversal, yield-point
promotion, a debug audit of the invariant) and what make it pay (consumers
that read through the entry rather than promoting it back to owned).
Medians of 7 interleaved rounds, this machine, against the pre-series binary:
straight-line while loop 6.32 -> 6.01 ns/instruction (-5.0%)
x = a + b loop -6.3% if flag: loop -3.8%
attribute-read loop -5.2% method-call loop -3.3%
subscript loop -6.7% fib(27) -0.8%
nbody -2.6% fannkuch -1.7% nqueens +1.3% richards -3.0%
deltablue -2.3% go -3.0% raytrace -1.7% float -3.2%
chaos -2.6% spectral_norm -2.5% mean -2.1%
About half of that is the consumer rewrite alone (mean -1.0% with the switch
still off) and the rest is the borrow. nqueens is the one benchmark that goes
the other way, by less than the roughly 4% of build-to-build layout noise this
tree has, and it moves with the consumer rewrite rather than the switch.
Full suite green (418 files, 42,110 tests). A build with debug assertions on
runs the suite with no audit failure, and trips the audit exactly when
`LOAD_FAST` is forced to borrow in a shape codegen refuses to convert
(`f(x, (x := None))`). Adversarial scripts -- swap loops and reassignment
under `__del__` tracking, argument evaluation that deletes its sibling,
`a[a.pop()]`, closures over a rebound cell, generators dropped while
suspended, tracebacks whose `f_locals` are read after unwinding, `settrace`
writing `f_locals` mid-expression, a method receiver whose local is cleared
while the call is being built, `__add__`/`__lt__`/`__bool__` that clear the
local their operand came from, threads sharing objects through locals, and
`gc.collect()` from a finalizer -- all agree with CPython 3.13 under
`gc.set_threshold(1, 1, 1)`, in both builds.
Assisted-by: Claude Code:claude-fable-5-1
* vm: exact-dict fast path for DICT_MERGE and CALL_FUNCTION_EX kwargs
`g(**d)` on an exact dict compiled to BUILD_MAP+DICT_MERGE+CALL_FUNCTION_EX
and paid for two Python-level `keys()`/`__getitem__` round trips per key at
both the DICT_MERGE handler and collect_ex_args' iterate_mapping_keys, even
though the source is a plain dict. Add an exact-dict (downcast + type
check, so subclasses still go through the mapping protocol) fast path to
both sites that iterates entries natively via PyDict::items_vec, matching
CPython's PyDict_Check fast paths in PyDict_Merge / CALL_FUNCTION_EX.
Measured (1M calls, release build): g(**d) 1320ns -> 665ns, f(**d)
1470ns -> 900ns, c.m(**d) 1690ns -> 1135ns, f(x=1,**d) 1600ns -> 1035ns,
f(**d1,**d2) 1700ns -> 1013ns. bm_django_template ~7% faster,
bm_argparse (subparsers) ~7% faster.
Assisted-by: Claude Code:claude-fable-5-1
* vm: skip the recursion guard hashing exact ints inside a tuple
tuple.__hash__ hashed every element through PyObject::hash, which wraps
each call in with_recursion's native-stack-depth check. That guard exists
for elements whose hash can recurse into arbitrary Python code, but
int.__hash__ is a leaf computation - it can neither recurse nor call back
into Python - so the check is pure overhead for the overwhelmingly common
case of tuples of plain ints (e.g. namedtuple-shaped dict keys).
pyperformance's mdp benchmark builds huge tables keyed by tuples of deeply
nested namedtuples whose leaves are ints; profiling with `sample` showed
PyObject::hash/with_recursion/tuple's slot_hash dominating the CPU time.
Hashing exact-int elements directly via hash::hash_bigint cuts mdp's
wall time from ~6.79s to ~4.81s (median of 5 interleaved runs), about 29%
faster, with no change to hash values or recursion-guard behavior for any
other element type.
Assisted-by: Claude Code:claude-fable-5-1
* vm: wire slot descriptors from __slots__ into C-level type slots
Classes that put a dunder name (e.g. "__setitem__") into __slots__ get
a per-instance member descriptor for it, which is a legitimate way to
override a special method per-instance (chameleon's tal.RepeatDict does
exactly this). PyType::new_verbose ran init_slots() to compute
mp_ass_subscript/mp_subscript/etc. before these member descriptors were
added to the type's attribute namespace, so the slot lookup never saw
them and __setitem__/__getitem__ silently fell back to "does not
support item assignment".
Fix by calling update_slot for each dunder-shaped __slots__ member
right after it is installed on the type, mirroring what
type.__setattr__ already does for regular attribute updates.
Assisted-by: Claude Code:claude-fable-5-1
* frame: stop allocating cold data to close a generator
`clear_locals_and_stack`, which runs whenever a generator or coroutine
comes to an end, reached for the frame's lazily-allocated `FrameColdData`
through `cold()`. That allocates it on first access, so every finishing
generator paid a `Box<FrameColdData>` malloc and free just to find
`f_extra_locals`, `f_locals_cache` and `f_overwritten_fast_locals` all
empty -- cold data is only ever populated by tracing and frame
introspection. Ask for it with `cold_opt()` instead and skip the three
reads when it was never allocated.
Medians of interleaved A/B runs, ns per generator create+finish over a
plain call: `for _ in g(): pass` 265 -> 231 (-13%), a coroutine that
completes synchronously 642 -> 617 (-4%), `sum(x for x in range(3))`
1033 -> 997 (-3.5%).
Assisted-by: Claude Code:claude-fable-5-1
* function: bind exact args straight into a generator's frame
Calling a generator, coroutine or async generator function took the
general argument-binding path even when the call site was the simplest
possible one. `invoke_exact_args_slots` and the vectorcall fast path both
recognised the shape -- every parameter positional, filled exactly once
-- and then handed generator-like code back to `invoke()`, which built a
`FuncArgs` (a `Vec` plus an `IndexMap`) only for
`fill_locals_from_args_inner` to walk varargs packing, keyword matching
and defaults that this shape cannot reach.
Add `make_generator_exact_args`, the counterpart of
`prepare_exact_args_frame` for the one call shape it refuses: it builds
the heap frame the generator will own, moves the arguments straight into
its fastlocals and returns the generator. Both fast paths now use it. The
vectorcall one matters on its own for generator expressions, whose fresh
`MAKE_FUNCTION` on every evaluation keeps them out of the call-site
specialization that would otherwise catch them.
Medians of interleaved A/B runs, ns per create+finish over a plain call:
`for _ in g(): pass` 231 -> 218 (-6%), a synchronously-completing
coroutine 616 -> 598 (-3%), `sum(x for x in range(3))` 1007 -> 967 (-4%).
Assisted-by: Claude Code:claude-fable-5-1
* frame: drain a dead frame in place instead of moving it out
Three allocations and a large move on the path a finishing generator
takes to release its frame.
`Traverse::clear` opened with `self.iframe.get_mut().take()`, moving the
whole `InterpreterFrame` -- a couple of hundred bytes -- out of its slot
and onto the stack, only to drop it there. Drain it through `as_mut()`
and empty the slot afterwards.
`LocalsPlus::clear_into` collected the evaluation stack into a `Vec` that
existed only to be extended into the caller's; push into that one
directly. And `clear_stack_and_cells` built (and dropped) a `Vec` for a
stack it had not looked at -- a frame that reached its `return`, which is
every generator that simply finished, left nothing there. Look first.
Medians of 5 interleaved rounds, ns per create+finish: `for _ in g():
pass` 307 -> 297 (-3.4%), a synchronously-completing coroutine 600 -> 585
(-2.5%), `list(g())` 565 -> 558; a plain call is unchanged.
Assisted-by: Claude Code:claude-fable-5-1
* vm: don't raise a TypeError to ask an iterator for its length
`length_hint_opt` opened with `iter.length()`, whose whole job on a type
with no length slot is to build the `TypeError` that says so -- which
this caller then catches, recognises and discards. Every iterator and
every generator takes that path, and `list()` calls it on whatever it is
handed, so a `list(gen)` paid for an exception object per call.
`length_opt` answers the same question with `None`. CPython's
`PyObject_LengthHint` gates the call on the slots for the same reason.
Median of 5 interleaved rounds: `list(g())` x200k for a 1-yield
generator 556 -> 407 ns (-27%). pyperformance generators, coroutines,
comprehensions and async_tree are unchanged, as are nqueens, richards
and chaos.
Assisted-by: Claude Code:claude-fable-5-1
* gc: track a generator and its frame in one pass
Making a generator tracks two objects that are born together and die
together -- the generator and the frame it owns -- and each one took the
gen0 list lock and a thread-local read of the running interpreter for
itself. Give the pair one entry point: `PyGenerator`, `PyCoroutine` and
`PyAsyncGen` opt out of tracking at allocation, the way frames already
do, and `make_generator_or_coro` links both into gen0 under a single
lock, with one owner read and one collection check.
Medians of 5 interleaved rounds: pyperformance coroutines -2.5%,
generators -0.9%; `for _ in g(): pass` over a plain call 205 -> 200 ns.
Assisted-by: Claude Code:claude-fable-5-1
* dict_inner: fast-path exact-int equality in key_eq
Mirrors CPython's lookdict: two exact `int` keys now compare by BigInt
value directly instead of going through the generic rich-compare
dispatch (identical_or_equal -> bool_eq -> rich_compare_bool -> __eq__
slot lookup). Falls back to bool_eq whenever either side isn't exactly
`int` (subclasses may override __eq__), so behavior is unchanged there.
Measured (min-of-6 rounds, 2M-iteration micros, this tree's build has
~4% layout noise so min is used as the noise floor):
- dict_int_lookup (`d[i]`, int keys): 70.9ns -> 62.0ns (-12.5%)
- dict_str_lookup / dict_tuple_lookup / set_str_contains: unchanged
(within noise), confirming no regression on non-int keys.
pyperformance sanity checks (single round each, base vs candidate):
bm_mdp median ~5027ms/loop vs ~5003ms/loop (no regression, hashing
isn't the dominant cost there); bm_nqueens, bm_go, bm_deltablue,
bm_hexiom, bm_chaos all within ~1% (noise).
Hash/equality values cross-checked bit-identical against CPython 3.14
for 0, 1, -1, True, False, 2**61-1, 2**61, -(2**63), 2**100, -(2**100),
and int/float/bool/Decimal/Fraction(1) hash-equality invariants.
Also tried an exact-type fast dispatch in PyObject::hash (skipping the
slots.hash vtable call + with_recursion for exact int/str) - it only
moved hash_small_int/hash_large_int by ~2%, under the 3% bar, so it was
not committed.
Tests: `rustpython -m test -j4 test_hash test_int test_long test_tuple
test_dict test_set test_dictviews test_ordered_dict test_collections
test_fractions test_numeric_tower test_float test_bool test_str
test_userdict test_weakref` all pass (1796 run, 72 skipped). `cargo
test -p rustpython-vm --release -p rustpython-common` passes. clippy
clean on the touched lines.
Assisted-by: Claude Code:claude-fable-5-1
* vm: let the modulus type handle a three-argument pow
CPython's ternary dispatch offers the operation to the third operand's type
whenever its slot differs from the two already tried, including the case where
neither operand had one. The port required both of those slots to exist, so
`pow(10, 2, Decimal(7))` never reached `Decimal` and raised TypeError.
Assisted-by: Claude Code:claude-fable-5-1
* vm: add a feature-gated opcode-pair histogram
Counts `(prev_op, op)` in the dispatch loop and dumps the nonzero pairs at
exit, so superinstruction and specialization candidates can be ranked from
what pyperformance-shaped code actually dispatches rather than guessed at.
Entirely behind the non-default `opcode-histogram` feature: the module is
`#[cfg]`-ed out and the loop's two-line hook with it, so a default build is
byte-identical to one without the patch.
Assisted-by: Claude Code:claude-fable-5-1
* vm: let boolean producers run the branch that follows them
`TO_BOOL*`, `COMPARE_OP*`, `CONTAINS_OP*` and `IS_OP` materialised a bool
only for the `POP_JUMP_IF_*` one dispatch later to pop and test it, and the
compiler's `NOT_TAKEN` marker cost a third dispatch on the fall-through edge.
Generalise the `COMPARE_OP_INT`-only jump fusion so every boolean producer
resolves the jump itself, and step over `NOT_TAKEN` from there and from the
plain `POP_JUMP_IF_*` handlers. Both back off to ordinary dispatch when the
successor is an instrumented opcode or while tracing, so `sys.monitoring`
branch events and `sys.settrace` opcode events are unaffected.
Dispatches over 13 standalone pyperformance benchmarks: -5.0% total
(richards -11.7%, unpickle_pure_python -8.9%, sympy -8.4%, hexiom -8.1%,
django_template -7.1%, go -6.1%; nbody unchanged). Times, best of 7
interleaved rounds: deltablue -4.0%, hexiom -3.5%, go -3.4%,
unpickle_pure_python -3.3%, richards -3.2%, raytrace -3.0%, nqueens -1.9%,
geomean -1.0%; a branch-heavy microbenchmark -4.7% against a straight-line
control at 0%.
Assisted-by: Claude Code:claude-fable-5-1
* Adapt yield-from resume to optional generator frames
Coro.frame is now PyAtomicRef<Option<FrameObject>>, so the flattened
resume path uses frame_opt/clear_except_code instead of a direct frame.
Assisted-by: Grok:4.6
* Fix clippy and cr_origin after try_to_bool/f_lineno changes
try_to_bool now takes &self, so drop the leftover clones. Derive
cr_origin lines from lasti instead of prev_line, and satisfy
-Dwarnings on the immortal-refcount and del_needed types.
Assisted-by: Grok:4.6
* Convert fd filenames without the os ToPyObject impl
host_env-free builds (sandbox check, wasm) do not compile os.rs, so
OsPathOrFd::filename cannot use Borrowed's ToPyObject. Use as_raw()
instead, which is the same conversion.
Assisted-by: Grok:4.6
* Gate fd path helpers on the host_env feature
WASM clippy builds rustpython-vm without host_env, so OsPathOrFd and
the fspath/fd helpers are unused and fail -Dwarnings.
Assisted-by: Grok:4.6
* Drop redundant pickle try_to_bool clones after rebase
Assisted-by: Grok:4.6
* Document why enumerate keeps a Counter beside BigInt
Assisted-by: Grok:4.6
* Move fd path helpers into ospath_fd under host_env
Assisted-by: Grok:4.6
* Group threading eval-breaker bits in one module
Assisted-by: Grok:4.6
* Make InterpreterFrame construction unsafe
run_iframe stays safe: a live frame already has valid identity pointers.
Assisted-by: Grok:4.6
* Move OsPathOrFd re-export to the top of ospath.rs
Assisted-by: Grok:4.6
---------
Co-authored-by: Lee Dogeon <dev.moreal@gmail.com>
|
||
|
Jeong, YunWon
|
3191ecaf38 |
host_env: share thread, locale, loader and BSTR (#8702)
* host_env: share thread, locale, loader and BSTR
Add Windows thread name, locale-info/wcscoll, CompareStringOrdinal,
LoadLibraryEx/GetProcAddress/FreeLibrary, BSTR and COM helpers, and
AF_HYPERV UUID conversion. Wire _thread, locale, ctypes LoadLibrary
and WMI BSTRs through those wrappers. Ensure =X: before spawnve/execve.
Assisted-by: Claude
* host_env: expose rustls-free ssl core on wasm
Compile MemoryBIO, protocol constants, OID tables, hostname
checks, and ALPN parsing without the rustls feature so wasm
and other hosts can implement _ssl on the shared types.
Assisted-by: Claude
* stdlib: add wasm _ssl on rustls-free host_env
Install _ssl on wasm32 against rustpython_host_env::ssl.
MemoryBIO, constants, OID, hostname, and ALPN work. Context
settings are stored; wrap/handshake and rustls-only loaders
raise SSLError until a wasm TLS engine is wired.
Assisted-by: Claude
* host_env: address #8702 review comments
Cache-miss Windows ctypes lookup uses GetProcAddress. FreeLibrary
propagates OSError. start_new_thread returns GetThreadId. Export
_thread._get_name and drop pyfunction doc comments. Reject
embedded NULs in Windows locale.strcoll/strxfrm. Terminate
wcscoll/wcsxfrm and SetThreadDescription inputs.
wasm _ssl accepts wrap_bio owner and get_ca_certs(binary_form),
and MemoryBIO.read(-1) returns all pending bytes.
Assisted-by: Claude
* host_env: gate the whole ssl module on feature ssl
Compile rustpython_host_env::ssl only with the ssl feature.
Rustls crates stay native-only, so enabling ssl on wasm does
not pull rustls. Inside the module, rustls files follow
not(wasm32) rather than a second feature flag.
Assisted-by: Claude
* host_env: fix Windows types and thread names
Cast SysAllocStringLen to *mut u16 and RPC status to u32 so
windows-gnu and MSVC agree. Export _NAME_MAXLEN, truncate names
at NUL and the OS byte cap, and decode get_name with fsdecode.
Assisted-by: Claude
* host_env: compile rustls engine on WASI
Keep rustls-free MemoryBIO/constants/OID/ALPN on all wasm
targets. Compile the rustls engine on WASI, where UnixTime::now
exists. Allow the rustpython ssl feature on wasm without
ssl-rustls or ssl-openssl.
Assisted-by: Claude
* compiler: pick the earliest CPython parse override
A later incompatible-prefix scan must not hide an earlier
number or bracket diagnostic. Parser-driven tokenization
stops at the first failure.
Assisted-by: Claude
* host_env: address remaining #8702 review comments
Map a missing Windows export to a symbol-load error, keep
LoadLibrary's Win32 code on the OSError, drop incomplete
UTF-16 pairs when truncating thread names, hide set_name
and _get_name off linux/macOS/Windows, and let ssl-rustls
enable host_env.
Assisted-by: Claude
* stdlib: add wasm _socket and read-all MemoryBIO
Register a rustls-free-style _socket shim on wasm32 so
Lib/ssl.py can import. Treat MemoryBIO.read(-1) as read-all,
matching _ssl_MemoryBIO_read_impl.
Assisted-by: Claude
* host_env: fix clippy, WASI ssl cfg, and wasm rustls
Defer unclosed-bracket overrides until no token diagnostic
exists, keep WASI rustls and wasm ssl modules exclusive,
preserve lone Windows surrogates, compile the rustls engine
on browser wasm, and clear clippy denials.
Assisted-by: Claude
* host_env: fix Windows clippy and wasm UnixTime
Gate thread-name byte truncation to linux/macOS, allow
GetProcAddress/FreeLibrary pointer deref, and enable
rustls-pki-types web time on browser wasm.
Assisted-by: Claude
* host_env: lock rustls-pki-types web for wasm
Assisted-by: Claude
* host_env: silence Windows unused thread id and shear
Assisted-by: Claude
* stdlib: drop Windows locale clippy needless wraps
Windows clippy failed on locale.strcoll/strxfrm needless return
and _getdefaultlocale always wrapping Ok.
Assisted-by: Claude
* compiler: keep lexer errors over print hints
print x; 0x must report an invalid hex literal. Ranking by source
offset let the earlier print hint win. Treat a finished radix
literal as valid so print x; 0x1 still gets the print message.
Assisted-by: Claude
* compiler: keep decode errors over unclosed openers
Ranking by offset made ( '\\N' report an unclosed paren and
wiped the unclosed-bracket flag on f'{, so compile snippets
raised SyntaxError instead of incomplete input. Export wasm
socket._accept for Lib/socket.py.
Assisted-by: Claude
* wasm: drop duplicate _socket; split locale cfgs
_socket lives in rustpython-stdlib's socket_wasm. The wasm crate
only re-exported that file when freeze-stdlib was off. Split
strcoll, strxfrm, and getencoding into cfg'd function defs.
Assisted-by: Claude
* compiler: keep decode blockers over unclosed openers
(print x; '\\N' must keep the print hint. A later malformed
escape still suppresses the EOF opener even when print wins.
Assisted-by: Claude
* compiler: keep earlier f-string over later lexer errors
f'{x'; ' and f'{x'; 0x must not report the trailing quote or
hex literal. Decode and f-string diagnostics now compete with
lexer failures by offset.
Assisted-by: Claude
* compiler: rank decode vs lexer vs print by class
Unterminated quotes no longer hide format-spec newlines or
empty f-string fields. Print is considered only after the
final winner is not a lexer failure, so print x; f'{x'; 0x
keeps the print hint. Line-continuation uses CPython wording.
Assisted-by: Claude
* compiler: keep unclosed paren over line-cont at EOF
test_exceptions wants '(' was never closed at the opener when
a final backslash follows. Skip escaped quotes in the bracket
scan so {\'a\'} still reports the continuation error. Un-xfail
test_join_daemon_thread_in_finalization.
Assisted-by: Claude
* test_threading: xfail daemon join finalization off Linux
The test passes on Linux. macOS and Windows leave an ignored
deallocator exception on stderr.
Assisted-by: Claude
* vm: hang daemon threads at interpreter finalize
Forced-off daemons must stay not-done so join() during
finalization GC raises PythonFinalizationError and
Thread.is_alive() stays true. SystemExit made the handle
look finished on macOS/Windows. Drop the Lib/test xfail.
Assisted-by: Claude
(cherry picked from commit
|
||
|
Jeong, YunWon
|
8de8f4bdb0 |
Bump version to 0.6.0 and use the crates.io ruff crates (#8700)
Point ruff_python_parser, ruff_python_ast, ruff_text_size and ruff_source_file at the published rustpython-ruff_* 0.16.5 crates instead of the git tag. Update chacha20 to 0.10.2 because 0.10.1 is yanked. Assisted-by: Claude Code:claude-fable-5-1 |
||
|
Jeong, YunWon
|
2fc2479f72 |
openssl: use host_env for CA paths and file checks (#8680)
Replace openssl-probe and local std::fs checks with host_env native_certs and fs. Vendored OpenSSL loads system CA paths from host_env; OpenSSL's own defaults stay for a system lib. Assisted-by: Grok:grok-4.6 |
||
|
Jeong, YunWon
|
bfd52eda61 |
ssl: move remaining rustls engine into host_env (#8675)
* ssl: move remaining rustls engine into host_env Cert loading, rustls verifiers, ClientConfig/ServerConfig construction, key logging, acceptor helpers, and X509 error mapping now live next to the trust store. stdlib _ssl keeps the Python object and socket adapters. Assisted-by: Claude * Remove unused workspace SSL dependencies Drop oid-registry, rustls-platform-verifier, and x509-cert from workspace.dependencies. Assisted-by: Grok 4.6 |
||
|
dependabot[bot]
|
5f0bcf6ca8 |
build(deps): bump lz4_flex from 0.13.1 to 0.14.0 (#8672)
Bumps [lz4_flex](https://github.com/pseitz/lz4_flex) from 0.13.1 to 0.14.0. - [Release notes](https://github.com/pseitz/lz4_flex/releases) - [Changelog](https://github.com/PSeitz/lz4_flex/blob/main/CHANGELOG.md) - [Commits](https://github.com/pseitz/lz4_flex/compare/0.13.1...0.14.0) --- updated-dependencies: - dependency-name: lz4_flex dependency-version: 0.14.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Jeong, YunWon
|
96925f49f8 |
Update Ruff to 0.16.5 (#8617)
Pin the RustPython Ruff fork to the patch commit based on Ruff 0.16.5 and adapt compiler and AST integration to the updated APIs. Assisted-by: Codex:GPT-5 |
||
|
Jeong, YunWon
|
ad4532c0e0 |
ssl: retain WebPKI-selected certificate chains (#8653)
Port the connection-owned chain builder from Pyre. Preserve verification inputs with TLS configurations and sessions, report the WebPKI-selected path, and publish only its selected capath anchor. Match CPython chain reporting for CERT_NONE and resumed sessions. Assisted-by: Codex:GPT-6 |
||
|
Joshua Megnauth
|
9b5e550f51 |
Bump deps & fix deprecated Atomic::fetch_update (#8650)
AI disclosure: I linted all of the JIT changes with AI since I wanted to be sure I followed cranelift's API properly. Assisted-by: GPT Luna-5.6 |
||
|
Jeong, YunWon
|
6e6792ee5c |
common: share the hashlib engine, and publish blake2b/blake2s as types (#8643)
The digest, HMAC, PBKDF2 and scrypt state moves out of the `_hashlib` module into a feature-gated `rustpython_common::hashlib` that owns each state behind a `Mutex` and reports plain Rust results; `_hashlib` and `_blake2` keep the object glue. The engine writes its state through a raw pointer into a caller-owned buffer, so the buffer is boxed and only the pointer travels with the Python object, and it carries the alignment the engine asserts. The wrapper is built only after a successful init, so a rejected digest name leaves nothing to drop over storage the engine never wrote. blake2 moves from `blake2` to `blake2b_simd` / `blake2s_simd`, whose `Params` carry the whole parameter block. `_blake2.blake2b` and `blake2s` become immutable types with `SALT_SIZE`, `PERSON_SIZE`, `MAX_KEY_SIZE` and `MAX_DIGEST_SIZE` as class attributes, and their instances are instances of those types. They accept `key`, `salt`, `person`, `fanout`, `depth`, `leaf_size`, `node_offset`, `node_depth`, `inner_size` and `last_node`. `_hashlib.scrypt` is published on the same engine. Six `test_hashlib` expectedFailure markers drop with it. `blake2`, `digest`, `dyn-clone`, `hmac` and `pbkdf2` lose their last consumer. Assisted-by: Grok Assisted-by: Claude |
||
|
dependabot[bot]
|
00ef37e346 |
build(deps): bump base64 from 0.22.1 to 0.23.1 (#8545)
Bumps [base64](https://github.com/marshallpierce/rust-base64) from 0.22.1 to 0.23.1. - [Changelog](https://github.com/marshallpierce/rust-base64/blob/master/RELEASE-NOTES.md) - [Commits](https://github.com/marshallpierce/rust-base64/compare/v0.22.1...v0.23.1) --- updated-dependencies: - dependency-name: base64 dependency-version: 0.23.1 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Jeong, YunWon
|
c716a7d070 |
common: share the zlib stream engine (#8631)
* common: share the zlib stream engine Move the VM-independent zlib-rs stream owner and state machine into a feature-gated rustpython-common module. Make rustpython-stdlib adapt that engine instead of carrying its own flate2 implementation, including native stream copy/deepcopy, Z_BLOCK flushes, and wbits=0 decompression. Assisted-by: Codex:GPT-5 * common: share the lzma stream engine Move the VM-independent liblzma stream owner, filter properties, and compression state into a target-gated rustpython-common module. Keep rustpython-stdlib as the Python object and exception adapter, and remove its direct xz dependencies and lzma-only generic compressor machinery. Assisted-by: Codex:GPT-5 * common: address compression engine reviews |
||
|
dependabot[bot]
|
7841d5a5a1 |
build(deps): bump system-configuration (#8627)
Bumps the system-configuration group with 1 update in the / directory: [system-configuration](https://github.com/mullvad/system-configuration-rs). Updates `system-configuration` from 0.7.0 to 0.8.0 - [Changelog](https://github.com/mullvad/system-configuration-rs/blob/main/CHANGELOG.md) - [Commits](https://github.com/mullvad/system-configuration-rs/compare/v0.7.0...v0.8.0) --- updated-dependencies: - dependency-name: system-configuration dependency-version: 0.8.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: system-configuration ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
codspeed-hq[bot]andCodSpeed Bot
|
062bd72c60 |
Add CodSpeed continuous benchmarking (#8613)
Alias criterion to codspeed-criterion-compat so the existing benchmark suites can be measured by CodSpeed with CPU simulation on every pull request, and add the workflow running them. Co-authored-by: CodSpeed Bot <no-reply@codspeed.io> |
||
|
Jeong, YunWon
|
47dc38cb21 |
Update malachite to 0.11 (#8604)
`malachite-bigint` 0.11.0 released while the workspace pinned 0.10.0. `pymath` requires `malachite-bigint = "0"`, so a fresh resolve picks 0.11 for it and leaves the workspace on 0.10, putting two incompatible copies of `BigInt` in one graph. `crates/stdlib/src/math.rs` then fails to compile, which is what the example projects do -- they carry no lockfile and resolve fresh on every run. `malachite-q` and `malachite-base` move with it; bumping `malachite-bigint` alone splits `malachite-nz` the same way. No source change is needed. Assisted-by: Claude |
||
|
Jeong, YunWon
|
17deb6af5b |
lock: add an opt-in rwlock that detaches before it blocks (#8556)
* lock: add an opt-in rwlock that detaches before it blocks A thread blocked acquiring a lock reaches no safepoint, so stop-the-world cannot stop it, and the lock it waits for is routinely one a thread the requester already suspended is holding. `RawDetachingRwLock` wraps the raw rwlock and hands the wait for a contended acquire to a hook that leaves the interpreter first; an acquire that takes the lock on its first try does not reach the hook. The vm installs the hook during interpreter init and implements it with `allow_threads`. The wait ends with the lock acquired while detached, so re-attaching can park the thread holding it. That is only safe where nothing reachable from a stop-the-world section takes the same lock, so it is opt-in per lock: `PyDetachingRwLock` is a separate type from `PyRwLock`, and `Traverse` is not implemented for it, so a payload holding one cannot derive `Traverse`. `PyByteArray::inner` takes it. `BorrowedValue`/`BorrowedValueMut` gain the matching mapped-guard variants. `a_thread_blocked_on_a_lock_does_not_stall_stop_the_world` blocks an interpreter thread on a `PyDetachingRwLock` and asserts stop-the-world still completes, running the stop on its own thread with a timeout so a stop that never completes fails rather than hangs. Without the hook installed it fails on the 10 s timeout; with it, it passes in 0.07 s. Assisted-by: Claude * lock: stop detaching where the lock may already be held `upgrade` runs with the upgradable lock held, and `lock_shared_recursive` may be the re-entrant take of a lock the calling thread holds; detaching there parks a thread holding the lock, which is what this type documents it must not do. They forward to the wrapped lock instead. `lock_upgradable` starts from holding nothing, but nothing takes an upgradable read of one of these, so it forwards too. `lock_shared` and `lock_exclusive` still detach. Also narrow two claims the comments overstated. Not implementing `Traverse` enforces the opt-in rule only against collections, not against the other things that stop the world. And the requester exemption the hook relies on is wider than `_PyEval_StopTheWorld` gives, so it is a local invariant. Assisted-by: Claude * os.readinto: read aside when the fd can wait `readinto` held the destination's write lock for the whole call, including the `read(2)` inside `allow_threads`. On a pipe, socket or terminal that read returns only when the other end writes, so a thread reaching the same object waited on a lock for an unbounded time, reaching no safepoint while it did. Take the fd that answers without waiting directly, as before, and otherwise read into scratch and take the lock only for the copy. This is what `FileIO.readinto`, `socket.recv_into` and `socket.recvfrom_into` already do; `os.readinto` was the site left over. The EINTR retry moves to `read_into_slice`, unchanged. Assisted-by: Claude * fcntl: detach around the calls that wait Every call in this module ran with the thread attached, so a thread inside one reached no safepoint until it returned. `flock(LOCK_EX)` and `lockf(F_LOCK)` return when whoever holds the lock gives it up, which may be never, and an ioctl on a terminal or socket answers when the device is ready to; the world could not be stopped for that long. `fcntl_fcntl_impl`, `fcntl_ioctl_impl`, `fcntl_flock_impl` and `fcntl_lockf_impl` all release around the call. `ioctl` with `mutate_flag` additionally held the target's write lock for the whole call, so a thread reaching the same object waited on a lock for as long as the device took. Its bytes now go in and come back through a buffer of our own, as `fcntl_ioctl_impl` copies through one of its own for anything up to IOCTL_BUFSZ. The export the argument holds is what keeps the length from changing in between. test_fcntl and test_ioctl pass. Assisted-by: Claude * openssl: read aside instead of holding the caller's buffer locked `SSLSocket.read` wrote straight into the destination buffer, holding the lock that reaching its bytes takes for the whole call. That read returns when the peer writes, which may be never, so a thread touching the same object waited on that lock for as long as the peer took, reaching no safepoint while it did. Read into a buffer of our own and take the destination's lock only for the copy. The rustls backend already reads this way, and `_ssl__SSLSocket_read_impl` works from a `Py_buffer` whose critical section ended before the read. `test_ssl` on this backend fails the same 16 tests before and after. Assisted-by: Claude * vm: add the inverse of allow_threads for callbacks A call that detaches hands the thread to stop-the-world, which counts it as parked. A callback that reaches Python from inside such a call would then run on a thread the requester believes is stopped, and nothing in the vm could stop it: `attach_thread` and `detach_thread` are private to `thread.rs`, and `allow_threads` only goes the one way. `attach_for_callback` attaches for the duration of the closure and returns the thread to where it was, the way `PyGILState_Ensure` and `PyGILState_Release` bracket `_servername_callback`. It tests for "not ATTACHED" rather than for DETACHED, so a thread a stop-the-world has already moved to SUSPENDED routes through `attach_thread` and parks there until the world starts again. `a_callback_inside_a_detached_call_waits_for_the_world` stops the world with a thread detached, then turns that thread loose at a callback and asserts it does not run until the world starts. With the transition disabled it fails. Assisted-by: Claude * openssl: attach around the callbacks that run Python `_servername_callback` and `_msg_callback` reach the interpreter from inside an SSL call -- they take a reference to the Python callback, build arguments and call it. That call is about to detach, and running Python from a detached thread runs it on a thread a stop-the-world requester counts as parked. `_servername_callback` opens with `PyGILState_Ensure()` for the same reason. Both now rejoin the interpreter for the duration of the callback and give the thread back afterwards. The reference to the callback moves inside that section, since taking it is itself an interpreter operation; the check for whether a callback is set at all stays outside, so a socket with none set never reaches the interpreter. No behavior change yet: nothing detaches around these calls, so `attach_for_callback` finds the thread already attached and just runs. Assisted-by: Claude * openssl: detach around the SSL calls that wait On a socket with a timeout the wait lands in `select` -> `sock_wait`, which already detaches. On a blocking socket there is no such return: `SSL_read` blocks in `recv(2)` through `impl Read for &PySocket`, with the thread attached and the connection's write lock held, so the world could not be stopped for as long as the peer stayed silent. `SSL_do_handshake`, `SSL_read_ex`, `SSL_write_ex` and `SSL_shutdown` all run between `Py_BEGIN_ALLOW_THREADS` and `Py_END_ALLOW_THREADS`; these now do too, in both socket and BIO mode as there. The connection lock stays held across the call and so becomes a detaching lock: a thread reaching the same socket gives up its interpreter rather than wait for it attached. `connection` is `#[pytraverse(skip)]`, so a collection does not walk into it and never takes that lock -- which is the rule for opting in, though the skip is what supplies it here rather than the missing `Traverse`. A server that completes a handshake and then says nothing used to deadlock the whole process: the collector suspended the main thread at a safepoint and then waited forever for the reader, so even the test's own timeout could not fire. It now collects in 3 ms. Verified separately that a Python `_msg_callback` still runs from inside the handshake -- 920 invocations across 40 handshakes with a collector looping, five runs clean. test_ssl on this backend fails the same 8 tests before and after, by name, and openssl.rs draws no clippy warning it did not draw before. Assisted-by: Claude * lock: assert the opt-in rule against every stop-the-world section Not implementing `Traverse` for `PyDetachingRwLock` states the rule to a collection: a payload holding one cannot derive `Traverse`, so a collection cannot walk into it. It says nothing to the other sections that stop the world -- fork, traceback dumps, frame enumeration -- and `#[pytraverse(skip)]` steps around it besides, which is how `_SSLSocket.connection` holds one. For those the rule was a comment. `set_world_stopped` records, on the one thread still running inside a stopped world, that it is that thread; `lock_shared` and `lock_exclusive` assert it is not set. A section that took one of these could block on a lock a parked thread holds and only that section can release, which is the deadlock the rule exists to prevent. Debug builds only; release builds track nothing. Nothing in the tree trips it: 180 rounds of collect, `sys._current_frames`, `faulthandler.dump_traceback` and 18 forks with four threads churning bytearrays, plus test_gc/test_bytes/test_threading/test_memoryview/test_buffer on a debug build, all clean. `taking_one_while_stopping_the_world_is_caught` takes one with the flag set and asserts the panic, so the guard is not dead code. Assisted-by: Claude * clippy: drop std_instead_of_core suppressions 1.98 no longer reports `std::io` items for the lint, so the six `expect` attributes for it are unfulfilled. Also drops `from_iter_instead_of_collect` from the workspace lint table, which 1.98 removed. Assisted-by: Claude * lock: spell out why the detaching rwlock exists States the motivation as the three-thread cycle it avoids: how a stop reaches a DETACHED thread but not an ATTACHED one, why a thread blocked on a lock reaches no safepoint, and why the waiter rather than the holder gives way. Puts it on `RawDetachingRwLock`, which is public and so rendered; the module doc is private and keeps only the hook description. Assisted-by: Claude * fcntl: raise the error the call returned The six detached calls discarded the `io::Error` and read `errno` again after `allow_threads`, which re-attaches in between and can park on the way. `lockf` already converts the returned error; these now do too. Assisted-by: Claude * openssl: allocate the read buffer through the vm Without a `buffer` argument `read_len` is whatever non-negative size the caller passed, so `vec![0u8; read_len]` aborts on allocation failure. `new_zeroed_bytes` raises `MemoryError` instead, as the two other reads in this file already do. Assisted-by: Claude * lock: leave no attached blocking acquire on a detaching lock `lock_upgradable` starts from holding nothing, so it detaches like `lock_shared`. A recursive read cannot: it may be the re-entrant take of a lock this thread holds, so detaching there parks a thread holding it, and staying attached stalls stop-the-world. `RawRwLockRecursive` is no longer implemented, which removes `read_recursive` from these locks; nothing took one. The assertion test no longer replaces the panic hook, which is process-wide and was suppressing panic output from whatever else ran beside it. Assisted-by: Claude * vm: wait for the actual block in the stop-the-world test The worker signalled before `read()` and the test slept 50ms, so a stop could complete with nothing blocked on the lock and the test would pass having checked nothing. It now publishes its thread id from inside the interpreter and the test waits for that slot to reach DETACHED, bounded so an acquire that never detaches fails rather than hangs. Assisted-by: Claude |
||
|
Joshua Megnauth
|
3c70a1ba58 |
Replace deprecated chrono with jiff (#8436)
See: chronotope/chrono#1768 |
||
|
강동안
|
e13bd1d9a6 |
Replace rust-timsort with an in-tree powersort (#8421)
* Add powersort implementation for list sorting Add crates/vm/src/sorting.rs implementing Tim Peters' timsort with powersort's merge-ordering policy (CPython 3.11+): run detection, binary insertion for short runs, galloping merge (merge_lo/merge_hi), and power-based merge ordering (powerloop). Comparison is passed in as a fallible `is_lt` closure, so the algorithm stays generic over the element type and free of interpreter details. Not yet wired into list.sort(); replaces rust-timsort in a follow-up. * Use powersort for list.sort() and drop rust-timsort Wire list sorting through crate::sorting::timsort and remove the rust-timsort dependency. Fixes the O(N^2) behavior on random input (1M random floats: ~16min -> ~0.8s), now within ~4x of CPython. * Fix usize underflow in merge_hi Succeed path * Fix clippy warnings and formatting in sorting.rs * Drop redundant test_ prefixes in sorting tests * Restore buffered elements when a comparison fails mid-merge * Copy one run-A element per gallop round in merge_lo |