Clippy doesn't run on everything which is why I keep accidentally
uncovering new lints to fix. I updated CI to run clippy on all features,
except tk, as well as all binaries, examples, and tests. That should
cover everything.
Tk currently doesn't build (at least for me), so that feature is still
disabled in CI.
While here, fix some of the lints that are now raised.
A Cargo.lock change no longer forces the cross-target cargo check by
itself. The check runs when the lock diff adds lines, or when a removed
package is still referenced from Rust sources.
Assisted-by: Claude Code:claude-opus-5-5
* Skip Python-defined classes when collecting builtin type docs
generate.py recorded every object.__subclasses__() entry as builtins.<name>
after importing the whole stdlib, so classes written in Python (tkinter,
unittest, tarfile, ...) ended up in the doc database. Classes that are
mutable heap types without C method descriptors and whose module is not a
C extension are now skipped, and their entries are removed from
data.inc.rs.
Assisted-by: Claude:claude-opus-5-5
Assisted-by: Grok:grok-4.7
* Store database docstrings xz-compressed behind an optional doc feature
- rustpython-doc's build script compresses all database docstrings into
one xz blob with an offset index; identical texts share one offset.
- Macros emit blob references for database docs; vm decompresses the
blob on first access.
- New `doc` cargo feature (default on) in rustpython, rustpython-vm and
rustpython-stdlib. Without it the blob and xz are not linked and
database-backed __doc__ is None; __text_signature__ is unaffected.
- Native types without a docstring get __doc__ = None in their dict.
- Workflows that build with --no-default-features enable `doc` where
tests or releases need docstrings.
Assisted-by: Claude:claude-opus-5-5
Assisted-by: Grok:grok-4.7
- Add test_fstring to the wasm-wasi test list.
- Expect test_global to fail when unittest is frozen
Assisted-by: Claude Code:claude-sonnet-5
Signed-off-by: Jiwoo Ahn <ikwydls1314@gmail.com>
* Render CPython defaults in generated text signatures
Arguments whose default rendered as `<unrepresentable>`, while the
same callable has a representable default in CPython, now render that
default:
- Where the Rust value can be a literal, the argument becomes a plain
field with `default = <literal>`.
- The remaining cases keep their Rust default and add `py_default`
with a comment. These are wrappers such as ArgIntoFloat or
ArgIntoBool, values that need the VM, enums, and Option where None
means another value.
Byte string literals render as Python bytes. A function registered
under several names gets an internal doc per name, so its __doc__ no
longer carries the signature header.
Removed expectedFailure from test_pydoc tests that now pass.
Assisted-by: Grok:grok-4.7
Assisted-by: Claude:claude-opus-5-5
* Match native callable signatures to CPython names and kinds
Rename parameters and adjust positional-only / keyword-only kinds and
defaults of native functions and methods so their generated text
signatures match CPython. Remove expectedFailure markers from tests
that now pass.
Assisted-by: Claude:claude-opus-5-5
Assisted-by: Grok:grok-4.7
* Fix Windows console read size and keep type.__prepare__ permissive
Assisted-by: Claude:claude-opus-5-5
* Derive signature defaults from typed Rust values
Param.default is now a DefaultRepr rendered by the const writer.
Literals, integer and bool constants become typed defaults, and other
default expressions use the type's const fn py_default. A bare
`optional` renders as None, so `default = None` rewrites are reverted.
Address review comments: zlib max_length defaults, get_ca_certs
binary_form keyword, getpeercert der, WASM ssl certificate keywords,
fstatvfs fd-only, and _sre endpos shown as sys.maxsize.
Assisted-by: Claude:claude-opus-5-5
Assisted-by: Grok:grok-4.7
* Generate class constructor text signatures at compile time
A #[pyclass] impl composes the type's internal doc from the Args of
its Initializer or Constructor, so type.__text_signature__ and
inspect.signature work for native classes. __doc__ stays the plain
doc. Struct sequences get (iterable=(), /) and __reduce__ (self, /).
Hand-parsed constructors are converted to FromArgs structs, and
_weakref.proxy is split from ProxyType.
Assisted-by: Claude:claude-opus-5-5
Assisted-by: Grok:grok-4.7
* Remove expectedFailure from test_hmac tests that now pass
Assisted-by: Claude:claude-opus-5-5
* Match remaining native signatures and count missing CPython signatures as matches
Give lock.__enter__ and RLock.__enter__ their own no-argument methods,
add FromArgs for start_new_thread and start_joinable_thread keeping
their errors, take RTLD_GLOBAL/RTLD_LOCAL from libc, and align names,
kinds and <unrepresentable> defaults where CPython's signature is
invalid (bytes.hex, reduce, warn, link, setlocale, add_done_callback,
...). getaddrinfo accepts keyword arguments. whats_left treats a
callable CPython has no signature for as matching.
Assisted-by: Claude:claude-opus-5-5
Assisted-by: Grok:grok-4.7
* Drop unfulfilled dead_code expectation on os.link dir_fd args
Assisted-by: Claude:claude-opus-5-5
* Distinguish OptionalArg and Option in signature defaults
An `optional` field's signature default now comes from its type:
Option<T> renders None, while OptionalArg<T> (a missing argument that
is not any Python value) renders <unrepresentable>. Fields where a
missing argument behaves exactly like None become Option<T>; fields
whose missing state differs keep OptionalArg. Redundant
py_default = "<unrepresentable>" entries are removed.
Assisted-by: Claude:claude-opus-5-5
Assisted-by: Grok:grok-4.7
* Remove expectedFailure from test_rlcompleter.test_attr_matches
Assisted-by: Claude:claude-opus-5-5
* Use Option for optional arguments whose missing and None states match
OptionalOption and OptionalArg<Option<T>> fields that treat a missing
argument like None become Option<T> with #[pyarg(optional)], which renders
=None in the text signature. Plain optional parameters of that kind move
into small FromArgs structs. Uses where missing and None differ (reduce
initial, StringIO newline, TextIOWrapper.reconfigure, set_asyncgen_hooks)
keep OptionalOption.
Assisted-by: Claude:claude-opus-5-5
Assisted-by: Grok:grok-4.7
* Render ::NAME defaults by name and restore named constant defaults
- default = ::NAME writes NAME into the text signature and stores
Into::into(NAME) as the Rust value.
- A one-field tuple-struct argument pattern such as Fildes(fd) names the
parameter after its binding, so fcntl and termios keep destructuring.
- zlib, lzma, syslog, mmap and select defaults use their named constants
again instead of literals or <unrepresentable>.
Assisted-by: Claude:claude-opus-5-5
Assisted-by: Grok:grok-4.7
* Name FromArgs fields after their Python parameters
Drop #[pyarg(name = ...)] where the Python name is a valid Rust field name
and rename the field instead. The option stays only for keywords (fn, loop,
final, type, self) and the non-snake_case memLevel.
Assisted-by: Claude:claude-opus-5-5
Assisted-by: Grok:grok-4.7
* Build literal defaults from Python values and type <unrepresentable>
- default = LIT on a non-primitive field converts the literal through
ToPyObject and TryFromObject when the argument is missing, so object
fields no longer need a vm expression plus a matching py_default.
- A bare default on a primitive integer, bool or float field renders its
zero value.
- DefaultRepr::Unrepresentable replaces the <unrepresentable> text, and
py_default = "<unrepresentable>" is a compile error.
- pyexpat.ParserCreate distinguishes a missing intern from intern=None,
mmap AccessMode renders its value, and sqlite blobopen names rowid.
Assisted-by: Claude:claude-opus-5-5
Assisted-by: Grok:grok-4.7
* Resolve native docstrings from the doc database at compile time
- rustpython-doc stores its table as a sorted const slice with const fn
lookups (get, get_attr, class_attr_doc) instead of a phf map, and the
update-doc-db workflow emits that form.
- Class, method, getset, module function and struct sequence docs, and
per-class slot wrapper docs, are chosen by const lookups in the derive
output. The database text wins over Rust doc comments.
- Macros reach the table through rustpython_vm::__exports, so only the vm
depends on rustpython-doc.
- Remove expectedFailure markers from doctest and pydoc tests that pass.
Assisted-by: Claude:claude-opus-5-5
Assisted-by: Grok:grok-4.7
* Keep time.gmtime, localtime and ctime without a text signature
Their [seconds] parameter has no text signature, and unittest.mock relies
on inspect.signature failing for them. Take OptionalOption again so the
default stays unrepresentable, and treat None like a missing argument.
Assisted-by: Claude:claude-opus-5-5
Multiprocessing fork/spawn/forkserver now run in the main -j $CORES
CPython test job. Remove expectedFailureIf for os.killpg now that
killpg exists.
Assisted-by: Grok:4.6
- Declare two lists (for wasm-wasi snippets and List/test files) and loop over them instead of one wasmer run line per file.
- Add ten passing test scripts under wasm32-wasip1
Assisted-By: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: Jiwoo Ahn <ikwydls1314@gmail.com>
* Unskip multiprocessing RLock repr test
Assisted-by: Grok:4.6
* Drop stable modules from the flaky MP CI list
Assisted-by: Grok:4.6
* Keep only multiprocessing fork on the flaky MP list
Assisted-by: Grok:4.6
* Drop gc_collect bench and unbreak two CI jobs
Remove benches/benchmarks/gc_collect.py so CodSpeed no longer
includes that execution benchmark.
On Windows, run test_multiprocessing_spawn serially; the
parallel snippets job hit a regrtest worker temp-dir error
while that suite was still running.
Install cargo-shear from a pinned prebuilt binary so the job
does not fall back to compiling 1.13.4 from source.
Assisted-by: Grok:4.6
* Unskip hang tests that now pass under -m test
Drop the RustPython hang skips on interpreter shutdown,
submit-after-shutdown, and connection pickling after they
passed with rustpython -m test.
Remove the unused expectedFailureIf around killpg on the
resource tracker deadlock test; os.killpg is present.
Pass matrix flaky-MP extras through the job environment
instead of interpolating them into the run script.
Assisted-by: Grok:4.6
* Restore hang skips and isolate spawn on CI
Put test_multiprocessing_spawn and
test_multiprocessing_forkserver back on the serial flaky MP
list. Ubuntu timed out in spawn test_notify_all under -j, and
forkserver.test_processes needed a rerun.
Restore the hang skips on interpreter shutdown, submit-after
shutdown, and connection pickling.
Assisted-by: Grok:4.6
* Skip rust CI jobs when sources did not change
Keep required rust-test and wasm-wasi checks green
without rebuilding. Run macOS/Windows clippy only
when the OS surface changed.
Assisted-by: Grok:4.6
* Cover root rust targets and honor cross-check clippy
Assisted-by: Grok:4.6
(cherry picked from commit 9de2baf539)
Assisted-by: Codex:gpt-6
* Split cron-ci cargo run and skip long jobs on PRs
Assisted-by: Grok:4.6
* Run cross-target cargo check only on platform diffs
Assisted-by: Grok:4.6
* Trigger cross-target checks on OS FFI and posix files
Assisted-by: Grok:4.6
* Watch every OS-specific module for cross-target checks
Assisted-by: Grok:4.6
* Restore cron-ci coverage and benchmarks on workflow PRs
Assisted-by: Grok:4.6
* Run gc_collect ten times per benchmark iteration
Assisted-by: Grok:4.6
* Reuse the prebuilt rustpython for CI whats_left
whats_left was invoking cargo build --release without threading,
so the snippets job rebuilt the interpreter after the earlier
release build.
Assisted-by: Grok:4.6
* Split CI cargo run and test into build then execute
Assisted-by: Grok:4.6
* Compile doctests in the CI build step
cargo test --no-run skips doctests, so rustdoc compilation
was still happening in the run step.
Assisted-by: Grok:4.6
* Drop cron-ci.yaml from this PR
A pull_request path filter matches the whole PR, so every
push was starting the periodic benchmark and coverage jobs.
Assisted-by: Grok:4.6
* Run doctests in their own CI step
cargo test --doc --no-run is rejected, so doctests cannot
be compiled without also executing them.
Assisted-by: Grok:4.6
* Do not run benchmark targets in rust tests
--all-targets includes Criterion benches, which cargo test
does not run by default.
Assisted-by: Grok:4.6
* Remove the deprecated rustpython-compiler-source crate
Nothing in the workspace depends on it, and its version was pinned to
0.4.1+deprecated, which is already on crates.io. Drop the crate, its
Cargo.lock entry, and the mentions in host_env_proposal.md.
Assisted-by: Claude Code:claude-fable-5-1
* ci: drop the rustpython-compiler-source exclusions
The crate no longer exists in the workspace.
Assisted-by: Claude Code:claude-fable-5-1
* venvlauncher: mark the crate as not publishable
It is a Windows-only binary that venv copies out of the installation,
not a crates.io dependency.
Assisted-by: Claude Code:claude-fable-5-1
* Add script to run upstream pyperformance against RustPython
pyperformance can't be installed as-is on RustPython since pyperf hard-depends
on psutil, a C extension RustPython cannot build or load (no CPython C-API /
extension-module support). Since pyperf already disables psutil usage on
interpreters reporting Py_GIL_DISABLED=1 (which RustPython does, having no
GIL), a functionality-free pure-Python psutil stub is enough to satisfy pip's
dependency resolution and unblock any pure-Python benchmark.
scripts/pyperformance/run_all.py drives pyperformance (under a host CPython
venv) with RustPython as the --python target, installs the stub via
PIP_FIND_LINKS + --inherit-environ, and catalogs pass/fail/timeout per
benchmark with an automatic failure classifier (C-extension build blocked vs.
a genuine exception raised by the benchmark itself).
Assisted-by: Claude Code:claude-sonnet-5
* pyperformance: support targeting any Python and comparing catalogs
Generalize run_all.py from a RustPython-only runner to a --python/--label
pair, so the same script can catalog a real CPython (with --no-psutil-stub,
since it has a genuine psutil) alongside RustPython. Each label gets its own
results/<label>/ subdirectory.
Add compare.py, which loads two labels' catalogs and writes a per-benchmark
Markdown table with both targets' status/mean and the candidate/baseline
time ratio, plus the median slowdown across benchmarks both targets passed.
Assisted-by: Claude Code:claude-sonnet-5
* ci: compare base, head and CPython with pyperformance on one runner
A ratio between two pyperformance runs measured on different machines can
move further than the change under review did, so a PR's numbers are only
worth reading when base, head and CPython were measured back to back on the
same runner. This adds a job that does exactly that -- two release builds
kept side by side with the stdlib of their own commit, then three runs with
nothing in between -- and posts the table as a PR comment.
`pr_diff.py` renders the three catalogs `run_all.py` writes: per benchmark
the CPython time, both RustPython times, both ratios against CPython, and
the head/base change, with anything under 3% shown as `~` because a shared
runner is not quiet enough to say more.
The comment is handed to a `workflow_run` companion the same way
`codspeed-comment.yaml` does it, since a `pull_request` run from a fork gets
a read-only token however the workflow is configured.
The job is opt-in behind a `run:pyperformance` label: it runs 28 benchmarks
three times over and costs well over an hour, which most PRs do not need.
`run_all.py` now forwards RUSTPYTHONPATH through pyperf, which a binary
copied away from its checkout needs in order to find the stdlib at all; the
two names go in one comma-separated `--inherit-environ`, because repeating
the flag keeps only the last.
Assisted-by: Claude Code:claude-fable-5-1
* ci: comment from the pyperformance run itself when the PR allows it
The `workflow_run` companion only starts firing once it is on the default
branch, so a pull request that introduces it -- this one -- would upload the
table and never show it. A pull request opened from this repo carries a
token that may write comments, so post it from the run that measured it and
leave the companion to the forks that genuinely cannot.
Both paths upsert on the same marker, so a fork PR that reaches the
companion after this step was skipped still lands one comment.
Assisted-by: Claude Code:claude-fable-5-1
* ci: address zizmor findings and fall back to head vs. CPython
- Pin `actions/github-script` to the commit the rest of the repo pins it to;
the hash carried over from an in-flight branch pointed elsewhere than the
`v9.0.0` comment claimed (ref-version-mismatch).
- Read the PR number, the CPython path and version, and the two commit shas
from `env` rather than expanding them into the shell (template-injection).
- Record the `workflow_run` trigger in `.github/zizmor.yml` next to the
existing `pull_request_target` entry, with the same justification: the
workflow reads one artifact and writes a comment, and never checks out or
runs pull request code (dangerous-triggers).
Building or measuring the base commit is now non-fatal, and `pr_diff.py`
renders head against CPython alone when the base column has nothing in it --
a base that will not build should still leave a usable table rather than
failing the job. The time and ratio columns are also named apart, since
`| base | head | base | head |` gave no clue which pair was which.
Assisted-by: Claude Code:claude-fable-5-1
* ci: derive pyperformance comment target from trusted workflow_run metadata
pr_number.txt came from the artifact a `pull_request` run uploads, which a
fork could tamper with to make this privileged `workflow_run` comment on an
arbitrary PR/issue. Resolve the PR from context.payload.workflow_run instead
(falling back to a commit-associated open PR lookup for workflow_dispatch
runs), and escape any HTML comments already present in diff.md so they can't
be mistaken for the marker.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wy8J7LpmdYjesTDonNoYkv
* ci: grant actions:read to the pyperformance comment workflow
download-artifact needs actions:read to fetch another run's artifact by
run-id. Specifying a permissions block sets every unlisted scope to none, so
without this the cross-run download silently failed for fork PRs -- the
scenario this workflow exists for -- continue-on-error masked it, and the
comment step found no diff.md and returned without posting.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wy8J7LpmdYjesTDonNoYkv
* pyperformance: kill the whole process group on a benchmark timeout
subprocess.run's own timeout only terminates the direct pyperformance
process; pyperf re-execs the target interpreter as a child, which can
survive the timeout and keep burning CPU into the next benchmark's
measurement. Run each benchmark in its own session/process group instead,
and on timeout SIGKILL the whole group via os.killpg.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wy8J7LpmdYjesTDonNoYkv
* pyperformance: compute the head-vs-base cell from head_vs_base directly
The cell was derived from base_vs_cpython and head_vs_cpython, both of which
are None when a benchmark has no CPython time -- so it rendered "-" even
though base and head both had valid times and head_vs_base was already
computed. Use that precomputed ratio instead.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wy8J7LpmdYjesTDonNoYkv
* pyperformance: invalidate the resumed catalog when the target executable changes
catalog.json records results without any fingerprint of target_python, and
resume skips benchmarks by name alone, so a local rebuild (or swapping the
target) under the same --label could silently reuse stale measurements.
Store a sha256 of the executable's contents alongside the catalog and, when
it no longer matches, ignore the existing catalog and re-run everything.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wy8J7LpmdYjesTDonNoYkv
* ci: fall back to github.sha for the head checkout on workflow_dispatch
github.event.pull_request.head.sha is empty on a workflow_dispatch run (no
pull_request payload), so `git checkout --quiet "" --` failed with "fatal:
invalid reference" on this non-optional step, aborting every manual run
before it could measure or render results.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wy8J7LpmdYjesTDonNoYkv
* pyperformance: resolve --out/--cache-dir to absolute paths
Every benchmark subprocess runs with cwd=work_dir (under --cache-dir), so a
relative --out or --cache-dir passed on the command line resolved against
that directory instead of the one run_all.py was invoked from, breaking
catalog output, the pyperformance binary path, and PIP_FIND_LINKS. Both
default to absolute paths already; resolve them up front so a relative
override behaves the same way.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wy8J7LpmdYjesTDonNoYkv
* pyperformance: persist catalog invalidation before the new fingerprint
The fingerprint file was rewritten before the invalidated catalog was ever
written back to disk. An interruption between the two (e.g. a crash during
the first benchmark of the re-run) left a fingerprint matching the new
executable pointing at a catalog.json still full of results measured
against the old one, so the next run would trust it as fresh. Write the
reset (empty) catalog first, then record the fingerprint.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wy8J7LpmdYjesTDonNoYkv
* pyperformance: show benchmarks that fail on both base and head
Rows were dropped whenever neither side had a mean, which also hid a
benchmark that failed or timed out identically on base and head -- e.g. a
C-extension gap unrelated to the PR under review. Only skip a benchmark
never attempted on either commit; keep rows with a fail/timeout status so
they still show up in the table.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wy8J7LpmdYjesTDonNoYkv
* pyperformance: write catalog.json atomically
write_catalog runs after every single benchmark, and Path.write_text
truncates the file before writing the replacement. An interruption
mid-write left catalog.json both missing every earlier result and invalid
JSON, so the next run's resume logic failed outright at json.loads(). Write
to a temporary sibling file and rename it into place instead.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wy8J7LpmdYjesTDonNoYkv
* pyperformance: fix passed count and median population in the summary table
"Benchmarks passed" counted len(base_ratios)/len(head_ratios), which require
a *CPython* ratio too -- a CPython-only failure hid an otherwise-passing
benchmark from both sides' counts. Count actual "ok" status per side
instead.
The two medians were also computed over independently-filtered
populations: whichever benchmarks had a valid ratio on that side alone. A
benchmark that regressed to failure on only one side dropped out of only
that side's median, which could shift it well beyond what any surviving
benchmark's timing actually changed by. Restrict both medians to
benchmarks with a valid ratio on both sides, and label the population size.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wy8J7LpmdYjesTDonNoYkv
* pyperformance: include RUSTPYTHONPATH's stdlib in the executable fingerprint
The catalog-invalidation fingerprint hashed only the executable's bytes, but
a RustPython binary copied away from its checkout finds its stdlib through
RUSTPYTHONPATH -- pyperf re-execs the target interpreter into a venv and
forwards that variable so it can still find Lib/. Changing that path, or
editing the files under it, without rebuilding the binary left resume
trusting results measured against a now-different stdlib. Fold its contents
into the fingerprint too.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wy8J7LpmdYjesTDonNoYkv
* pyperformance: require an explicit ok/fail status for pass/fail transitions
only_head/only_base flagged a benchmark as newly passing or no longer
passing based only on whether the other side had a mean, so a benchmark
never attempted on one side (e.g. a partial --benchmarks run) was reported
as a regression or fix it never was. Require the other side's status to be
an actual fail/timeout, not just missing.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wy8J7LpmdYjesTDonNoYkv
* pyperformance: don't gate the head/base geometric mean on a CPython ratio
The summary panel -- pass counts and the head/base geometric mean, neither
of which need a CPython comparison -- only rendered when at least one
benchmark also had a valid ratio against CPython on both sides. A build
with a perfectly good head-vs-base comparison but no common CPython result
lost the whole panel, including the one aggregate that didn't depend on
CPython at all.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wy8J7LpmdYjesTDonNoYkv
* pyperformance: normalize RUSTPYTHONPATH and extract main() into named steps
Fix: a relative RUSTPYTHONPATH was fingerprinted relative to this script's
invocation directory but resolved by pyperf's subprocess relative to
work_dir (its cwd), so a relative value could be hashed against the wrong
directory entirely, or fail to resolve at runtime. Resolve it to an
absolute path up front, alongside --out and --cache-dir.
Refactor: main() had accumulated argparse setup, executable resolution, and
the fingerprint/resume block inline, each wrapped in an explanatory comment
-- a sign the function was doing too much rather than that it needed more
narration. Extracted parse_args(), resolve_target_python(), and
load_resumable_catalog() so main() reads as a short sequence of named
steps and each comment sits next to the one function it actually explains.
No behavior change; verified with unit tests against the extracted
functions (fingerprint reacting to RUSTPYTHONPATH, resume picking up a
prior catalog, invalidation on an executable change).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wy8J7LpmdYjesTDonNoYkv
* pyperformance: split pr_diff.py's main() and fix a status-transition gap
Fix: only_head was still gated on have_base, which only asks whether *any*
base benchmark produced a timing -- so a benchmark that flipped from
failing to passing went unreported whenever every other base benchmark
also happened to fail. Status transitions don't need that gate; they
already require an explicit ok/fail/timeout status on both sides.
Refactor: main() had grown into one long function mixing argument parsing,
catalog loading, row computation, and three separate rendering concerns.
Split it into parse_args(), load_base_catalog(), build_rows(),
render_intro(), render_summary_panel(), render_table(), and
find_transitions(), each documenting the one thing it decides. No behavior
change beyond the fix above; re-verified against every scenario exercised
in the last several fixes (dual-fail rows, geo mean without a CPython
ratio, unattempted benchmarks, and the have_base gap) plus the existing
manual smoke tests.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wy8J7LpmdYjesTDonNoYkv
* pyperformance: don't resolve an empty RUSTPYTHONPATH to the cwd
The normalization only checked "RUSTPYTHONPATH" in os.environ, so a set-but-
empty value (RUSTPYTHONPATH="") resolved to the current directory via
Path("").resolve(). executable_fingerprint() would then recursively hash
every file under wherever run_all.py was invoked from -- including its own
results directory while writing to it -- causing spurious catalog
invalidation and a lot of pointless hashing. Check for a nonempty value
instead, matching the truthiness check already used everywhere else this
variable is read.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wy8J7LpmdYjesTDonNoYkv
* docs(pyperformance): distinguish the host runner from the benchmarked target
A reviewer read "pyperformance cannot be installed as-is on RustPython" and
reasonably asked why RustPython would need to support pyperformance's CLI at
all, since it's normally run as a command under any Python (see
https://github.com/RustPython/RustPython/pull/8691#discussion_r3997957942).
It doesn't: the pyperformance CLI (the host process) always runs under a
real CPython regardless of --python. What actually needs psutil/pyperf
importable is the *benchmark* subprocess -- pyperf re-execs the --python
target and that subprocess itself does `import pyperf`. Spell out that
host/worker split up front instead of leading with "cannot be installed".
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wy8J7LpmdYjesTDonNoYkv
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Pin the RustPython Ruff fork to the patch commit based on Ruff 0.16.5 and adapt compiler and AST integration to the updated APIs.
Assisted-by: Codex:GPT-5
* ci: hold a CodSpeed run to the machine main's baseline was measured on
CodSpeed compares a run against a base run and refuses to call the
comparison meaningful when the two ran on different machines: the runner
uploads a `SystemInfo` record beside the profile carrying the OS, the
architecture, the CPU's vendor id, brand and flags, the physical core
count and the installed memory, and a hosted pool hands out whatever
machine is free. A simulated measurement is fixed against how fast the
machine is, but not against what it makes the process execute -- glibc
selects `memcpy` and the string routines through IFUNC on CPU flags, a
new stable rustc emits different code, and these benchmarks run each
program under CPython through pyo3 as well as under RustPython, so the
runner's default interpreter is an input too.
`scripts/codspeed-environment.py` records those fields plus the rustc,
libc and CPython versions, and compares a run against a recording. The
benchmark job now restores the recording main last wrote, records over it
when the ref is main, and compares against it otherwise; the install,
build and measure steps carry `if: steps.environment.outputs.match !=
'false'`, so a run that drew a different machine finishes green having
uploaded nothing. A missing or unfetchable recording is read as "nothing
to disagree with" rather than as a mismatch.
Assisted-by: Claude
* ci: read an unreadable environment recording as not a match
A cache miss, an eviction or a cache-service outage leaves the reference file
absent, and the comparison then reported `match=true`: every guarded step ran
and the run uploaded a measurement without having identified the machine it was
taken on.
An absent recording now reports `match=false`, and says so as a `::warning`
rather than the `::notice` a mismatch emits, so a job that measures nothing
because it has nothing to compare against is distinguishable from one that
measures nothing because the machine differs.
Assisted-by: Claude
Alias criterion to codspeed-criterion-compat so the existing benchmark suites can be measured by CodSpeed with CPU simulation on every pull request, and add the workflow running them.
Co-authored-by: CodSpeed Bot <no-reply@codspeed.io>
* Correct what the non-unix frame stack is for
The comment said non-unix threading builds have no stop-the-world.
CollectStopTheWorld is gated on `feature = "threading"` alone, and
sys._current_frames stops the world on both paths; the field is the
fallback a reader uses when there is no `top_iframe` to materialize from.
Assisted-by: Claude
* Seek by the whole file position on Windows
`SetFilePointer` answers with the low half of the new position and signals
failure with INVALID_SET_FILE_POINTER, which is also that half of a
position four gigabytes in; telling them apart takes the error code, which
this did not read, so such a seek was reported as an error. Deciding
seekability from it also called such a file unseekable.
`SetFilePointerEx` returns the whole position and a success flag of its
own, which also removes the transmute of the position into halves.
Assisted-by: Claude
* memoryview: tell a value of the wrong kind from one that does not fit
Assigning an item reported every packing failure as a TypeError, so
m[0] = 300 on a 'B' view said the value was the wrong type rather than out
of range. Packing now says which of the two it was, and whether the value's
own code raised, in which case that error is the answer as it is:
m[0] = 300 ValueError: invalid value for format 'B'
m[0] = "x" TypeError: invalid type for format 'B'
m[0] = <__index__ that raises> the raised error
`struct` reports both as `struct.error` and is unchanged; the kind travels
beside the exception for the caller that tells them apart.
Also: None was read as a deletion, so m[0] = None answered "cannot delete
memory" instead of packing it; and deleting through the mapping protocol
never reached the read-only check, which comes first.
Assisted-by: Claude
* Raise MemoryError for allocations sized by Python input
os.read, _RawIOBase.read, int.to_bytes, struct.pack, ctypes array
creation and the _ssl RAND functions sized a Vec from a Python-supplied
length with vec![], which calls handle_alloc_error and, under
panic = "abort", ends the process. They now allocate through
vm.new_zeroed_bytes and raise MemoryError.
itertools.product built its pools without checking that
len(iterables) * repeat is representable; it now raises OverflowError
"repeat argument too large" and reserves the pool and index vectors
fallibly. repeat is read as isize, so a negative one raises ValueError
"repeat argument cannot be negative" instead of the conversion's message.
_ssl.RAND_bytes and RAND_pseudo_bytes read n as i32, matching the int
they are declared with.
Assisted-by: Claude
* Collect with the count in the object header
A collection kept the count it was working with in a table keyed by the
object's address, and the objects it had proved reachable in a second one.
Between them they were hashed once per candidate and twice per edge in the
heap, which is where most of a collection over a live heap went.
The count now lives in `PyInner::gc_refs`, with `GcBits::COLLECTING` saying
it is meaningful, and reachability is `gc_refs == GC_REACHABLE` rather than
membership in a set. Step 5 splits the candidates and clears the bit in one
pass. gcbench, five interleaved pairs, median: a live heap of 423k objects
goes from 0.101s to 0.055s and a dead one from 0.402s to 0.383s.
The bits, generation, owner and count take eight bytes between them. A
64-bit header had those eight as the padding its alignment forces, so it is
unchanged at 48 bytes; a 32-bit header grows from 24 to 28.
Assisted-by: Claude
* Take an explicit thread stack size as a floor in debug builds
A debug build already started Python threads on 8 MiB rather than Rust's
2 MiB default, but an explicit threading.stack_size(N) went through
verbatim. test_threading asks for 256 KiB, and starting a thread on that
walked off the end of the stack: the guard page fault landed in the
prologue of ExecutingFrame::run.
Unoptimized, that prologue reserves 80,848 bytes where the optimized one
reserves 656 -- execute_instruction is #[inline(always)] and LLVM only
colors stack slots from opt-level 1, so the frame is the sum of all 200
instruction arms' temporaries rather than the largest. A Python call costs
88,672 bytes of native stack there, and threading's bootstrap is six
frames deep, so 256 KiB holds less than half of what starting a thread
takes.
The floor reaches thread::Builder only; threading.stack_size() still
answers with what was asked for, and release builds are unchanged.
Assisted-by: Claude
* Check the native stack on every frame entry
The C-stack guard ran on one frame entry in eight. That asks the margin to
cover eight frames rather than one, and it does not: an unoptimized frame
entered through native code takes 88,672 bytes against a debug margin of
262,144. A recursion whose steps re-enter that way -- `__add__` calling
itself, a sort key that sorts -- ran off the end of the stack instead of
raising RecursionError. On a debug build `class Add: __add__ = lambda s, o:
s + o; Add() + 1` segfaulted on the main thread; it now raises, as it does
under CPython and in release builds.
enter_iframe checked and then called enter_iframe_unchecked, which checked
again; it now leaves the check to the one call.
Measured on a call-dominated benchmark, five interleaved pairs: instructions
retired go up 0.17%, about four per call, which is the stack pointer read
and the compare.
Assisted-by: Claude
* Take an iterable's length hint when a list is filled from it
`map_py_iter` read `__length_hint__` only to pass it to `PyIterIter`, and
returned an empty vector when the hint was `isize::MAX` or more. Collecting
through `PyResult` dropped the iterator's lower bound, so nothing reserved
the room the hint asked for.
`list()`, `list.extend()` and `list.__iadd__()` now reserve it and report a
hint they cannot honour as `MemoryError`; a hint that leaves no room for the
elements the list already holds is passed over, as `list_extend()` does.
`tuple()` and the other callers keep filling up without reserving.
`length_hint_opt` errors other than the `TypeError` it already turns into
`None` now reach the caller instead of being dropped.
`__iadd__` and `inplace_concat` went through `extract_cloned`, which reads
`__len__` and not `__length_hint__`; both call `PyList::extend` now, the way
`list_inplace_concat()` calls `list_extend()`.
The tuple, list and dict fast paths of `extract_elements_inner` reserve their
known length, which the `collect()` they used dropped.
Assisted-by: Claude
* Drop map.__length_hint__
`map_methods` has no `__length_hint__`, so `operator.length_hint()` on a map
answers 0, not the length of what it draws from.
The method walked into the length hint of every iterator it holds, and a
chain of maps 10000 long overflowed the native stack answering for the
outermost one. It also took the longest of its iterators, where a map stops
at the shortest.
Assisted-by: Claude
* Ask for a length hint where each caller asks for it
`map_py_iter` asked the iterable it was handed, for every caller, and reported
what asking raised. Only some callers ask it: `list_extend()` and
`_PyBytes_FromIterator()` ask the iterable, `PySequence_Tuple()` asks the
iterator, and the bytearray constructor asks nothing. `tuple()`, `min()`,
`max()`, `collections.deque()` and `f(*x)` raised for an iterable whose
`__len__` or `__length_hint__` does, where they answer.
Which object is asked is now the caller's to say. `sorted()` asks the iterable,
being `PySequence_List()`.
`bytes_from_object()` stood in for `PyBytes_FromObject()`, for
`bytearray_extend()` and for the bytearray constructor, which do not agree on
this: the first two ask, the last does not. It is split, and assigning to a
bytearray slice takes the constructor's side with `PyByteArray_FromObject()`.
`list.extend()` counted what it held before the iterable had been asked, where
`list_extend()` reads `Py_SIZE(self)` after. A `__length_hint__` that adds to
the list made the overflow guard read a count too small and raise `MemoryError`
where nothing is wrong; one that empties it made the guard skip a reservation
that cannot be served.
Assisted-by: Claude
* Settle product's pool count before it reads its arguments
`product_new()` checks `repeat` and works out `npools` before it calls
`PySequence_Tuple()` on any argument, and fills the pools `npools` times.
The pools were filled by repeating the arguments `repeat` times instead, which
walks that many steps even with no arguments to repeat: `product(repeat=2**62)`
counted up to it rather than answering `[()]`. The count was also worked out
after the arguments had been read, so a repeat too large to serve ran their
code first.
Assisted-by: Claude
* Let the bool format answer with the error its value raised
`pack_single()` leaves `'?'` to `PyObject_IsTrue()` and returns what that
raised. Packing classified the error instead, so a `ValueError` from a
`__bool__` came back as "memoryview: invalid value for format '?'".
Assisted-by: Claude
* Release a cell's old value after the lock
`PyCell::set` dropped what it replaced while still holding the mutex guarding
the cell contents. A `__del__` running from that drop and reading the same cell
waited on a lock its own caller held, so `del it` on a closure variable whose
value has such a `__del__` deadlocked.
The replaced value is now released once the guard is gone, as `Py_XSETREF`
stores before it decrefs.
Assisted-by: Claude
* Have a set iterator hold the set it iterates
The iterator kept only a reference to the inner hash table, so in
`it = iter(A(*args))` the `A()` temporary was the last owner and died as the
call returned, before `it` was bound. A `__del__` reading `it` there saw an
unbound name and its error was printed and ignored.
The iterator now holds the set object, as `si_set` does, and releases it once
exhausted. That release happens after the lock is dropped, where
`setiter_iternext()` puts its `Py_DECREF(so)` past `Py_END_CRITICAL_SECTION()`,
so a `__del__` that iterates again does not wait on a lock the call holds.
Assisted-by: Claude
* Release an exhausted iterator's container after the lock
`PositionIterInternal::_next` overwrote its `IterStatus::Active` while the
caller still held the mutex around it. Dropping the container there ran any
`__del__` under that lock, and a `__del__` that iterated the same object again
blocked on it.
`exhaust()` now hands the container back instead of dropping it, and
`locked_step()` releases it after the guard. list, list_reverseiterator, tuple,
str, dict and its views and reverse views, bytes, bytearray, memoryview,
array, deque, and the enumerate and sequence iterators all go through it.
Assisted-by: Claude
* Unskip test_free_after_iterating
Assisted-by: Claude
* Stop listing test_set as an environment polluter
`check_free_after_iterating` no longer leaves an ignored exception behind, and
the job that reruns the listed tests ten times fails once one of them stops
polluting.
Assisted-by: Claude
* Drop winsound imports left unused
`TryFromBorrowedObject`, `exceptions`, and `ToWideString` have no reference in
the module, which fails the Windows clippy line under `-Dwarnings`.
Assisted-by: Claude
* Ask nothing where the caller takes no room
`map_py_iter` asked the iterator for a length hint on behalf of every caller
that does not reserve, and reported what asking raised. Those callers ask
nothing at all: `tuple()`, `f(*x)`, `bytearray(x)`, `min()`, `max()` and
`collections.deque()` answer for an iterator whose `__length_hint__` raises,
where they had been raising it.
The answer was also never spent. It reached `PyIterIter` for a `size_hint()`
the push loop does not read, so the lookup and any call it made were work
thrown away: `tuple()` over a generator drops 22% of its instructions, and 17%
over an iterator with a `__length_hint__` written in Python.
Assisted-by: Claude
* Stop asking an iterator how long it is to walk it
`PyIter::iter` and `PyIter::into_iter` asked for a length hint and reported
what asking raised. Nothing spent the answer: it reached `PyIterIter` for a
`size_hint()` that every caller either loops past or drops, since collecting
into a `Result` reports no lower bound.
23 operations answered for an iterator whose `__length_hint__` raises, where
they had been raising it: `set`, `frozenset` and the nine `set` methods that
take an iterable, `dict.fromkeys` and the dict view operators, `array` and
`array.extend`, `all`, `any`, `sum`, `io.writelines`, `math.fsum`,
`math.prod`, and `csv.writerow` and `writerows`.
Over a generator, `set()` drops 16% of its instructions and `all()` 23%.
`str.join` and `bytes.join` do ask, reaching their elements through
`PySequence_Fast()`, which fills a list from the iterator. They take
`iter_sized()`, which is now the only way to ask. `iter_without_hint` is gone,
its callers being what `iter` already does.
Assisted-by: Claude
* Name the matrix jobs rather than let the matrix name them
A generated job name lists every value in the matrix entry, so `cargo check`
carried the booleans its dependencies and `skip_ssl` keys expand to, and the
snippets job carried its test arguments and timeout. Adding or removing a key
renames the check, which drops it from the required list until that list is
edited to match. Emptying `env_polluting_tests` renamed three checks this way.
`Run rust tests` and `clippy` keep the names they had. `cargo check` drops the
booleans from six of its nine, and the snippets job drops its arguments and
timeout from all three.
Assisted-by: Claude
* Keep a sequence iterator active when an element raises
`PositionIterInternal::_next` exhausted the iterator for any non-`Return`
result, so an error from `__getitem__` ended the walk. `iter_iternext()`
lets go of its sequence for `IndexError` and `StopIteration` alone, and
`PyIterReturn::from_getitem_result` has already turned the first of those
into the second, so only `StopIteration` exhausts now.
Both deque iterators keep exhausting on their own mutation guard, which
`deque_iternext()` does by zeroing the counter before it raises; they
share one step function for it, and the message it raises is lowercased
to match the three other sites in the module.
Assisted-by: Claude
* Name the caller in the bytes conversion TypeError
An object with no iteration protocol reached `PyObject_GetIter`'s "not
iterable" message, and `bytearray.extend()` reported bytes. Each entry
point now checks for the protocol first and names itself:
bytes(object()) cannot convert 'object' object to bytes
bytearray(object()) cannot convert 'object' object to bytearray
bytearray().extend(object()) can't extend bytearray with object
Assisted-by: Claude
* Take a strong count too large to hold as reachable
`start_gc_refs` clipped a count at `GC_REACHABLE - 1`, a number the
per-reference subtraction could still walk down to zero and collect a
live object. It now stores `GC_REACHABLE`, and `subtract_gc_ref` leaves
that value alone.
Assisted-by: Claude
* Raise AssertionError where a snippet asserts False
`assert False` is removed under `-O`, which the snippet suite may run.
Assisted-by: Claude
* Keep raising for a collection that moved under its iterator
A deque, set or dict iterator raised once and then read as spent. The
guard is sticky: `deque_iternext()` looks at the deque's state before the
count it keeps, and `dictiter_iternextkey()` and `setiter_iternext()`
write a size no collection can have, so every later call finds the same
thing and raises again. `dequereviter_next()` is the one exception,
looking at its count first, so it runs out after the first raise.
Both deque iterators now carry `dequeiterobject.counter` rather than
reading a length back from the deque, and the dict and set iterators
compare the size they captured against the collection's own every time
they are asked how much is left, which is what makes that answer nothing
from the moment the collection changes rather than only once the
iterator has raised.
The set's message is capitalized to match `setiter_iternext()`.
Measured against 3.14.6, for each of deque, reversed deque, set, dict
and a reversed dict view: the hint after the change, the error, the hint
after the error, and what a later call answers.
Assisted-by: Claude
* Give apt-get update a deadline to fail on
The step waits on `apt-get update`, which has no deadline of its own, so
a source that takes the connection and then stops answering holds the
job until the workflow's own timeout. The retry that disables the
Microsoft and azure-cli sources runs only when the update exits
non-zero, which a held connection never does; three jobs on this branch
sat on this step for 20 minutes to five and a half hours.
Each attempt is now bounded, and the transports are given a timeout, so
a source that stops answering reaches the retry.
Assisted-by: Claude
Joshua Megnauth