mirror of
https://github.com/RustPython/RustPython.git
synced 2026-10-28 18:59:04 +08:00
stack/python-3.15-8950
4
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Lee DogeonandClaude Sonnet 5
|
44dcd1de2d |
ci: add pyperformance benchmarking scripts and workflow (#8691)
* Add script to run upstream pyperformance against RustPython pyperformance can't be installed as-is on RustPython since pyperf hard-depends on psutil, a C extension RustPython cannot build or load (no CPython C-API / extension-module support). Since pyperf already disables psutil usage on interpreters reporting Py_GIL_DISABLED=1 (which RustPython does, having no GIL), a functionality-free pure-Python psutil stub is enough to satisfy pip's dependency resolution and unblock any pure-Python benchmark. scripts/pyperformance/run_all.py drives pyperformance (under a host CPython venv) with RustPython as the --python target, installs the stub via PIP_FIND_LINKS + --inherit-environ, and catalogs pass/fail/timeout per benchmark with an automatic failure classifier (C-extension build blocked vs. a genuine exception raised by the benchmark itself). Assisted-by: Claude Code:claude-sonnet-5 * pyperformance: support targeting any Python and comparing catalogs Generalize run_all.py from a RustPython-only runner to a --python/--label pair, so the same script can catalog a real CPython (with --no-psutil-stub, since it has a genuine psutil) alongside RustPython. Each label gets its own results/<label>/ subdirectory. Add compare.py, which loads two labels' catalogs and writes a per-benchmark Markdown table with both targets' status/mean and the candidate/baseline time ratio, plus the median slowdown across benchmarks both targets passed. Assisted-by: Claude Code:claude-sonnet-5 * ci: compare base, head and CPython with pyperformance on one runner A ratio between two pyperformance runs measured on different machines can move further than the change under review did, so a PR's numbers are only worth reading when base, head and CPython were measured back to back on the same runner. This adds a job that does exactly that -- two release builds kept side by side with the stdlib of their own commit, then three runs with nothing in between -- and posts the table as a PR comment. `pr_diff.py` renders the three catalogs `run_all.py` writes: per benchmark the CPython time, both RustPython times, both ratios against CPython, and the head/base change, with anything under 3% shown as `~` because a shared runner is not quiet enough to say more. The comment is handed to a `workflow_run` companion the same way `codspeed-comment.yaml` does it, since a `pull_request` run from a fork gets a read-only token however the workflow is configured. The job is opt-in behind a `run:pyperformance` label: it runs 28 benchmarks three times over and costs well over an hour, which most PRs do not need. `run_all.py` now forwards RUSTPYTHONPATH through pyperf, which a binary copied away from its checkout needs in order to find the stdlib at all; the two names go in one comma-separated `--inherit-environ`, because repeating the flag keeps only the last. Assisted-by: Claude Code:claude-fable-5-1 * ci: comment from the pyperformance run itself when the PR allows it The `workflow_run` companion only starts firing once it is on the default branch, so a pull request that introduces it -- this one -- would upload the table and never show it. A pull request opened from this repo carries a token that may write comments, so post it from the run that measured it and leave the companion to the forks that genuinely cannot. Both paths upsert on the same marker, so a fork PR that reaches the companion after this step was skipped still lands one comment. Assisted-by: Claude Code:claude-fable-5-1 * ci: address zizmor findings and fall back to head vs. CPython - Pin `actions/github-script` to the commit the rest of the repo pins it to; the hash carried over from an in-flight branch pointed elsewhere than the `v9.0.0` comment claimed (ref-version-mismatch). - Read the PR number, the CPython path and version, and the two commit shas from `env` rather than expanding them into the shell (template-injection). - Record the `workflow_run` trigger in `.github/zizmor.yml` next to the existing `pull_request_target` entry, with the same justification: the workflow reads one artifact and writes a comment, and never checks out or runs pull request code (dangerous-triggers). Building or measuring the base commit is now non-fatal, and `pr_diff.py` renders head against CPython alone when the base column has nothing in it -- a base that will not build should still leave a usable table rather than failing the job. The time and ratio columns are also named apart, since `| base | head | base | head |` gave no clue which pair was which. Assisted-by: Claude Code:claude-fable-5-1 * ci: derive pyperformance comment target from trusted workflow_run metadata pr_number.txt came from the artifact a `pull_request` run uploads, which a fork could tamper with to make this privileged `workflow_run` comment on an arbitrary PR/issue. Resolve the PR from context.payload.workflow_run instead (falling back to a commit-associated open PR lookup for workflow_dispatch runs), and escape any HTML comments already present in diff.md so they can't be mistaken for the marker. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wy8J7LpmdYjesTDonNoYkv * ci: grant actions:read to the pyperformance comment workflow download-artifact needs actions:read to fetch another run's artifact by run-id. Specifying a permissions block sets every unlisted scope to none, so without this the cross-run download silently failed for fork PRs -- the scenario this workflow exists for -- continue-on-error masked it, and the comment step found no diff.md and returned without posting. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wy8J7LpmdYjesTDonNoYkv * pyperformance: kill the whole process group on a benchmark timeout subprocess.run's own timeout only terminates the direct pyperformance process; pyperf re-execs the target interpreter as a child, which can survive the timeout and keep burning CPU into the next benchmark's measurement. Run each benchmark in its own session/process group instead, and on timeout SIGKILL the whole group via os.killpg. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wy8J7LpmdYjesTDonNoYkv * pyperformance: compute the head-vs-base cell from head_vs_base directly The cell was derived from base_vs_cpython and head_vs_cpython, both of which are None when a benchmark has no CPython time -- so it rendered "-" even though base and head both had valid times and head_vs_base was already computed. Use that precomputed ratio instead. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wy8J7LpmdYjesTDonNoYkv * pyperformance: invalidate the resumed catalog when the target executable changes catalog.json records results without any fingerprint of target_python, and resume skips benchmarks by name alone, so a local rebuild (or swapping the target) under the same --label could silently reuse stale measurements. Store a sha256 of the executable's contents alongside the catalog and, when it no longer matches, ignore the existing catalog and re-run everything. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wy8J7LpmdYjesTDonNoYkv * ci: fall back to github.sha for the head checkout on workflow_dispatch github.event.pull_request.head.sha is empty on a workflow_dispatch run (no pull_request payload), so `git checkout --quiet "" --` failed with "fatal: invalid reference" on this non-optional step, aborting every manual run before it could measure or render results. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wy8J7LpmdYjesTDonNoYkv * pyperformance: resolve --out/--cache-dir to absolute paths Every benchmark subprocess runs with cwd=work_dir (under --cache-dir), so a relative --out or --cache-dir passed on the command line resolved against that directory instead of the one run_all.py was invoked from, breaking catalog output, the pyperformance binary path, and PIP_FIND_LINKS. Both default to absolute paths already; resolve them up front so a relative override behaves the same way. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wy8J7LpmdYjesTDonNoYkv * pyperformance: persist catalog invalidation before the new fingerprint The fingerprint file was rewritten before the invalidated catalog was ever written back to disk. An interruption between the two (e.g. a crash during the first benchmark of the re-run) left a fingerprint matching the new executable pointing at a catalog.json still full of results measured against the old one, so the next run would trust it as fresh. Write the reset (empty) catalog first, then record the fingerprint. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wy8J7LpmdYjesTDonNoYkv * pyperformance: show benchmarks that fail on both base and head Rows were dropped whenever neither side had a mean, which also hid a benchmark that failed or timed out identically on base and head -- e.g. a C-extension gap unrelated to the PR under review. Only skip a benchmark never attempted on either commit; keep rows with a fail/timeout status so they still show up in the table. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wy8J7LpmdYjesTDonNoYkv * pyperformance: write catalog.json atomically write_catalog runs after every single benchmark, and Path.write_text truncates the file before writing the replacement. An interruption mid-write left catalog.json both missing every earlier result and invalid JSON, so the next run's resume logic failed outright at json.loads(). Write to a temporary sibling file and rename it into place instead. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wy8J7LpmdYjesTDonNoYkv * pyperformance: fix passed count and median population in the summary table "Benchmarks passed" counted len(base_ratios)/len(head_ratios), which require a *CPython* ratio too -- a CPython-only failure hid an otherwise-passing benchmark from both sides' counts. Count actual "ok" status per side instead. The two medians were also computed over independently-filtered populations: whichever benchmarks had a valid ratio on that side alone. A benchmark that regressed to failure on only one side dropped out of only that side's median, which could shift it well beyond what any surviving benchmark's timing actually changed by. Restrict both medians to benchmarks with a valid ratio on both sides, and label the population size. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wy8J7LpmdYjesTDonNoYkv * pyperformance: include RUSTPYTHONPATH's stdlib in the executable fingerprint The catalog-invalidation fingerprint hashed only the executable's bytes, but a RustPython binary copied away from its checkout finds its stdlib through RUSTPYTHONPATH -- pyperf re-execs the target interpreter into a venv and forwards that variable so it can still find Lib/. Changing that path, or editing the files under it, without rebuilding the binary left resume trusting results measured against a now-different stdlib. Fold its contents into the fingerprint too. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wy8J7LpmdYjesTDonNoYkv * pyperformance: require an explicit ok/fail status for pass/fail transitions only_head/only_base flagged a benchmark as newly passing or no longer passing based only on whether the other side had a mean, so a benchmark never attempted on one side (e.g. a partial --benchmarks run) was reported as a regression or fix it never was. Require the other side's status to be an actual fail/timeout, not just missing. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wy8J7LpmdYjesTDonNoYkv * pyperformance: don't gate the head/base geometric mean on a CPython ratio The summary panel -- pass counts and the head/base geometric mean, neither of which need a CPython comparison -- only rendered when at least one benchmark also had a valid ratio against CPython on both sides. A build with a perfectly good head-vs-base comparison but no common CPython result lost the whole panel, including the one aggregate that didn't depend on CPython at all. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wy8J7LpmdYjesTDonNoYkv * pyperformance: normalize RUSTPYTHONPATH and extract main() into named steps Fix: a relative RUSTPYTHONPATH was fingerprinted relative to this script's invocation directory but resolved by pyperf's subprocess relative to work_dir (its cwd), so a relative value could be hashed against the wrong directory entirely, or fail to resolve at runtime. Resolve it to an absolute path up front, alongside --out and --cache-dir. Refactor: main() had accumulated argparse setup, executable resolution, and the fingerprint/resume block inline, each wrapped in an explanatory comment -- a sign the function was doing too much rather than that it needed more narration. Extracted parse_args(), resolve_target_python(), and load_resumable_catalog() so main() reads as a short sequence of named steps and each comment sits next to the one function it actually explains. No behavior change; verified with unit tests against the extracted functions (fingerprint reacting to RUSTPYTHONPATH, resume picking up a prior catalog, invalidation on an executable change). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wy8J7LpmdYjesTDonNoYkv * pyperformance: split pr_diff.py's main() and fix a status-transition gap Fix: only_head was still gated on have_base, which only asks whether *any* base benchmark produced a timing -- so a benchmark that flipped from failing to passing went unreported whenever every other base benchmark also happened to fail. Status transitions don't need that gate; they already require an explicit ok/fail/timeout status on both sides. Refactor: main() had grown into one long function mixing argument parsing, catalog loading, row computation, and three separate rendering concerns. Split it into parse_args(), load_base_catalog(), build_rows(), render_intro(), render_summary_panel(), render_table(), and find_transitions(), each documenting the one thing it decides. No behavior change beyond the fix above; re-verified against every scenario exercised in the last several fixes (dual-fail rows, geo mean without a CPython ratio, unattempted benchmarks, and the have_base gap) plus the existing manual smoke tests. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wy8J7LpmdYjesTDonNoYkv * pyperformance: don't resolve an empty RUSTPYTHONPATH to the cwd The normalization only checked "RUSTPYTHONPATH" in os.environ, so a set-but- empty value (RUSTPYTHONPATH="") resolved to the current directory via Path("").resolve(). executable_fingerprint() would then recursively hash every file under wherever run_all.py was invoked from -- including its own results directory while writing to it -- causing spurious catalog invalidation and a lot of pointless hashing. Check for a nonempty value instead, matching the truthiness check already used everywhere else this variable is read. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wy8J7LpmdYjesTDonNoYkv * docs(pyperformance): distinguish the host runner from the benchmarked target A reviewer read "pyperformance cannot be installed as-is on RustPython" and reasonably asked why RustPython would need to support pyperformance's CLI at all, since it's normally run as a command under any Python (see https://github.com/RustPython/RustPython/pull/8691#discussion_r3997957942). It doesn't: the pyperformance CLI (the host process) always runs under a real CPython regardless of --python. What actually needs psutil/pyperf importable is the *benchmark* subprocess -- pyperf re-execs the --python target and that subprocess itself does `import pyperf`. Spell out that host/worker split up front instead of leading with "cannot be installed". Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wy8J7LpmdYjesTDonNoYkv --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
Lee Dogeon
|
8cfa667182 |
ci: automate OSCCA pull request triage (#8505)
Assisted-by: Codex:gpt-5.6-sol |
||
|
Jeong, YunWon
|
cdf006cfa1 |
Split InterpreterFrame from FrameObject for stack-allocated execution (#8354)
* Add LightFrame for unobserved Python-to-Python calls
Allocate a lightweight LightFrame header on the DataStack instead of a
full Frame PyObject for specialized exact-args call paths. The frame is
materialized lazily only when observed (sys._getframe, traceback, tracing).
Key changes:
- LightFrame struct with borrowed pointers and no refcount overhead
- FrameSource enum (Heavy/Light) replacing the object field on ExecutingFrame
- CURRENT_LIGHT_FRAME TLS for the light frame chain
- Interleaved heavy/light frame walking in frame_at_offset_vm and
current_thread_frame_vm
- invoke_light_slots on PyFunction for the fast call path
Call overhead reduced from ~92ns to ~58ns (~37% reduction).
fib(28) improved from ~152ms to ~111ms (~27% reduction).
Assisted-by: Claude
* Streamline with_frame: inline recursion check, skip tracing dispatch
- Inline check_recursive_call instead of wrapping in with_recursion closure
- Amortize C stack overflow check to every 64th call depth
- Skip dispatch_traced_frame when use_tracing is false (hot path)
- Add recursion_depth_increment/decrement helpers
Assisted-by: Claude
* Address review findings for LightFrame correctness
- Fix materialize_light_frame previous pointer: use payload pointer
(&**frame) instead of Py<Frame> pointer, matching the chain convention
- Fix predecessor ref management: use retained_back instead of
mem::forget to keep materialized predecessors alive
- Fix materialized ref reclaim: use FrameRef::from_raw to drop the
leaked keeper ref without double-incrementing
- Use PyAtomic<u32> for LightFrame.lasti instead of AtomicU32, removing
the unsound cast on non-threading targets
- Add amortized C stack overflow check to light frame path
- Add scopeguard to light frame execution for panic safety
- Remove unnecessary Send/Sync impls from LightFrame
- Make check_c_stack_overflow pub(crate) for light frame path access
Assisted-by: Claude
* Fix CI failures: cspell, stale materialized state, f_back chain gaps
- Fix cspell: 'amortised' → 'amortized'
- Sync lasti, prev_line, and fastlocals when re-observing a materialized
light frame, fixing stale f_locals/f_lineno in test_inspect and others
- In with_frame, materialize any active light frame as the heavy frame's
predecessor and store it in retained_back, so f_back and inspect.stack
see the correct interleaved order (H_new → L_mat → H_old)
- Guard invoke_light_slots on NEWLOCALS | OPTIMIZED code flags
Assisted-by: Claude
* Unify frame chain: replace dual-chain with single FrameChainPtr
Replace the dual-chain frame management (CURRENT_FRAME AtomicPtr<Frame>
+ CURRENT_LIGHT_FRAME Cell<*const LightFrame>) with a single unified
chain using FrameChainPtr, a tagged pointer where bit 0 distinguishes
heavy (*const Frame) from light (*const LightFrame) entries.
Changes:
- Add FrameChainPtr type with tagged pointer encoding
- Change LightFrame: replace previous_light + saved_current_frame with
single previous: FrameChainPtr field
- Change InterpreterFrame.previous from AtomicPtr<Frame> to PyAtomic<usize>
- Change CURRENT_FRAME TLS from AtomicPtr<Frame> to AtomicUsize
- Remove CURRENT_LIGHT_FRAME TLS entirely
- Simplify all chain walkers (frame_at_offset, frame_at_offset_vm,
find_owned_chain_frame, for_each_current_frame, current_thread_frame)
- Remove light-frame materialization block from with_frame
- Update f_back to dispatch on FrameChainPtr tags
- Update faulthandler and gc_state to walk unified chain
- Adaptive C stack check: every call in debug, every 16th in release
Assisted-by: Claude
* Fix light frame materialization: current_frame, f_back, and GC tracking
- current_frame() now uses current_thread_frame_vm() to materialize
light frames, fixing super()/compile()/exec() when called from light
frame context
- materialize_light_frame stores retained_back for heavy predecessors
too, so f_back resolves after the caller returns
- Escaped materialized frames (traceback, sys._getframe) are GC-tracked
at cleanup so reference cycles are collectible
- Restore pre-existing @expectedFailure markers for test_frame proxy tests
Assisted-by: Claude
* Limit light frame materialization to super() only
current_frame() returns heavy-only again to avoid unnecessary
materialization that extends object lifetimes. super() init uses
current_thread_frame_vm() directly since it must inspect the
light frame's code and freevars to find __class__.
Assisted-by: Claude
* Fix remaining CI failures: C stack check, GC tracking threshold, expectedFailure
- Check C stack overflow every call in light frame path (not every
16th) to prevent stack overflow on CI workers with smaller stacks
- Fix GC tracking threshold: escaped materialized frame detection
uses strong_count > 1 (was > 2, missed frames referenced only by
traceback in the result)
- current_frame() materializes light frames so warnings, compile(),
and other callers see the correct frame
- Mark test_futures2 PyFutureTests.test_task_exc_handler_correct_context
as expectedFailure (pre-existing on main: PyTask __del__ timing)
Assisted-by: Claude
* Avoid materialization in hot paths: add light-frame-aware accessors
current_frame() stays heavy-only to avoid costly materialization on
every call. Instead, add current_globals(), current_code(), and
current_builtins() that read directly from light frame raw pointers
without creating frame PyObjects.
Callers migrated:
- PyFunction:🆕 current_builtins() for fallback builtins lookup
- compile()/eval(): current_code() for future feature flags
- type.__new__: current_globals() for __module__ detection
- typevar/typing: current_globals() for caller module name
- _io: current_code() for source path check
- super(): current_thread_frame_vm() for __class__ cell access
- warnings.warn: current_thread_frame_vm() for stack level walk
- import: current_globals() for module resolution
Assisted-by: Claude
* Fix current_locals() to include light frames
current_locals() must materialize light frames so locals() builtin
returns the correct scope. This fixes test_zipimport doctest and
other tests that rely on locals() in light frame context.
Assisted-by: Claude
* Fix clippy, profile tracing, resume_gen_frame, and faulthandler issues
- Fix clippy: collapsible_if in materialize_light_frame, map_unwrap_or in import
- Fix profile breakage: use current_thread_frame_vm in trace_event_inner to
materialize light frames, preventing frame identity mismatch when profiling
is enabled inside a light frame (e.g. profile.Profile.runctx)
- Disable light frames inside trace/profile callbacks (tracing_is_suppressed)
- Fix resume_gen_frame: skip light frames in previous chain to avoid dangling
pointers after generator suspend
- Fix faulthandler: emit '<no Python frame>' when chain has only light frames
Assisted-by: Claude
* Fix materialized frame refcount: transfer localsplus ownership on exit
Remove @expectedFailure from test_futures2: the underlying issue was that
materialized light frames cloned localsplus values (refcount +1 each),
keeping objects alive until the next GC cycle collection.
Fix: after run_light_frame returns, transfer ownership of localsplus
values from the light frame to the materialized frame, replacing the
clones. This eliminates the extra refcount, allowing objects to be
GC'd immediately when their last reference is dropped.
Also stabilize materialized frame's localsplus on the heap in
sync_materialized_on_exit before GC tracking — the data-stack backing
will be popped imminently.
Assisted-by: Claude
* Fix rustfmt, increase stack margin, address review comments
- Fix rustfmt formatting issues (lint CI failure)
- Increase STACK_MARGIN_BYTES from 2048 to 3072 words to prevent C stack
overflow in deep recursion tests on macOS CI (test_functools SIGSEGV)
- Skip dispatch_traced_frame when use_tracing is false (review comment)
- Fix post-fork top_frame initialization: walk past light frames to find
nearest heavy frame (review comment)
Assisted-by: Claude
* Remove redundant test override in test_futures2
The override only called super() without any marker, making it a
no-op that prek flags as a redundant test patch.
Assisted-by: Claude
* Fix test_pdb and test_functools CI failures
- Revert with_frame dispatch_traced_frame skip: tracing can be enabled
mid-execution (pdb.set_trace), so Return events must always be checked.
This fixes 3 test_pdb doctest failures across all platforms.
- Check C stack overflow on every with_frame call instead of every 64th:
the previous sampling approach missed overflows between checks when
native stack frames are large (invoke_light_slots + lru_cache recursion).
- Add #[inline(never)] to invoke_light_slots to prevent the optimizer
from merging its stack frame into callers.
- Add early C stack check before DataStack allocation in invoke_light_slots.
Assisted-by: Claude
* Rename Frame to FrameObject, FrameRef to FrameObjectRef
Prepare for _PyInterpreterFrame/PyFrameObject separation by
renaming the PyObject-backed frame type to FrameObject.
Python-visible name ("frame") is unchanged.
Assisted-by: Claude
* Remove Deref<Target = InterpreterFrame> from FrameObject
Replace implicit field access through Deref with explicit
iframe()/iframe_ref()/iframe_mut() calls. This decouples
FrameObject from InterpreterFrame field layout, enabling
independent modification of InterpreterFrame in future commits.
Assisted-by: Claude
* Move InterpreterFrame identity fields to raw pointers
InterpreterFrame's code/globals/builtins/func_obj fields are now
borrowed raw pointers instead of owned PyRef/PyObjectRef. Ownership
of these references is anchored in FrameObject's new owned_code,
owned_globals, owned_builtins, owned_func_obj fields.
This aligns InterpreterFrame's layout with LightFrame's existing
raw-pointer pattern, enabling future DataStack-based allocation
without PyObject heap allocation.
- Add code()/globals()/builtins()/func_obj() accessor methods
- Add FrameObject::new_ref() for combined create+allocate+patch
- Update ExecutingFrame to use &Py<PyCode> / &Py<PyDict> / &PyObject
- Update monitoring.rs functions to take &Py<PyCode> instead of &PyRef<PyCode>
- Add Send+Sync impls for InterpreterFrame (raw pointers are !Send by default)
Assisted-by: Claude
* Remove LightFrame, FrameChainPtr, and dual-frame-type infrastructure
Delete LightFrame struct, FrameChainPtr tagged pointer, FrameSource enum,
and all supporting functions: materialize_light_frame, sync_light_to_materialized,
transfer_localsplus_to_materialized, sync_materialized_on_exit,
run_light_frame, invoke_light_slots body, LocalsPlus::from_datastack_raw.
All call paths now use a single frame type (FrameObject wrapping
InterpreterFrame). The frame chain is a simple *const FrameObject linked
list through InterpreterFrame.previous.
invoke_light_slots delegates to invoke_exact_args_slots.
ExecutingFrame.frame_source replaced with ExecutingFrame.frame.
Frame chain walking simplified throughout (no more light/heavy dispatch).
Net: -724 lines of dual-frame-type complexity.
Assisted-by: Claude
* Clean up invoke_light_slots and _vm function variants
- Replace invoke_light_slots calls with invoke_exact_args_slots
- Delete the invoke_light_slots delegate method
- Replace current_thread_frame_vm/frame_at_offset_vm with their
non-_vm equivalents (no more light frame materialization needed)
- Remove stale light-frame comment in callable.rs
Assisted-by: Claude
* Fix CI lint issues
- Move github context expressions from run blocks to env blocks
(zizmor/template-injection)
- Add workflow-level permissions: {} to lib-deps-check.yaml
- Suppress zizmor excessive-permissions for lib-deps-check.yaml
(pull_request_target is required for PR comments)
- Apply rustfmt formatting
Assisted-by: Claude
* Stack-allocate InterpreterFrame for non-generator function calls
- Extract InterpreterFrame::new() from FrameObject::new()
- Change frame chain from *const FrameObject to *const InterpreterFrame
- Add InterpreterFrame.materialized field for lazy FrameObject creation
- Add vm.with_iframe()/run_frame_fast() for stack-allocated frame execution
- Move ExecutingFrame.frame to iframe: *const InterpreterFrame
- Implement lazy materialize() for on-demand FrameObject creation
- Update invoke_with_locals/invoke_prepared_exact_args to skip heap allocation
- Update frame chain walking in faulthandler, gc_state, builtins/frame
- super() reads InterpreterFrame directly without materialization
Assisted-by: Claude
* Remove unnecessary refcount and atomic ops from fast call path
- Use raw pointer for func_obj instead of cloning PyObjectRef (saves 2 atomic RMW)
- Skip owner AcqRel swap for stack-allocated frames (always Thread-owned)
Assisted-by: Claude
* Change trace field to Option<PyObjectRef>, remove vm from InterpreterFrame::new
- trace: PyMutex<PyObjectRef> -> PyMutex<Option<PyObjectRef>>
- Eliminates vm.ctx.none() refcount inc/dec per frame init
- Remove vm parameter from InterpreterFrame::new() (no longer needed)
- f_trace getter returns None when trace is unset
- f_trace setter stores None instead of the None singleton
Assisted-by: Claude
* Store current frame pointer on VirtualMachine for fast lookup
- Add current_frame_ptr Cell<usize> field to VirtualMachine
- Hot path (with_iframe) reads/writes vm.current_frame_ptr instead of TLS
- TLS CURRENT_FRAME still maintained for signal-safe traceback walking
- Use set_current_frame_nosave() to skip cross-thread top_frame update
- super(), frame_at_offset, current_thread_frame_materialize use vm Cell
Assisted-by: Claude
* Apply rustfmt
Assisted-by: Claude
* Fix clippy use_self warnings, add inline hints for hot path
Assisted-by: Claude
* Remove code refcount clone from fast call path
Defer code.to_owned() to the generator/coroutine path only. On the fast
path, use &Py<PyCode> directly — the code object is alive via the
PyFunction on the caller's stack.
Assisted-by: Claude
* Remove VM current_frame_ptr Cell (fix SIGSEGV in threading builds)
The Cell<usize> on VirtualMachine could desynchronize from the TLS
CURRENT_FRAME in nested VM scenarios (enter_vm called with different
VMs on the same thread). Revert to TLS-only frame chain management.
Keep set_current_frame_nosave() for the stack-frame fast path: it
skips cross-thread top_frame publication but still does the TLS swap.
Fix clippy not_unsafe_ptr_arg_deref in set_current_frame.
Fix unnecessary Result wrapper in make_generator_or_coro.
Assisted-by: Claude
* Fix CI failures: GC frame assertion, faulthandler Radium import, dead code warning
- Fix SIGSEGV in threading builds: GC debug assertion walked frame
chain via top_frame, which is not updated by set_current_frame_nosave
used by stack-allocated iframes. Walk CURRENT_FRAME TLS chain instead.
- Add InterpreterFrame::get_lasti() accessor, remove direct Radium
import from faulthandler.rs (fixes unused import warning).
- Add cfg_attr allow(dead_code) for from_payload_ptr without threading
(fixes WASM build warning).
Assisted-by: Claude
* Fix materialized frame: copy localsplus, clear stale previous pointers
- materialize_slow: copy fastlocals snapshot to heap instead of empty
localsplus (fixes index-out-of-bounds panic in locals()/f_locals)
- materialize_slow: set previous to null (stack iframes become dangling)
- with_frame/resume_gen_frame: clear iframe.previous on pop (prevents
dangling pointers to freed stack iframes)
- f_back: walk TLS CURRENT_FRAME chain instead of top_frame for
cross-thread lookup (top_frame is stale for stack-allocated iframes)
- check_locals_access: match materialized frames by comparing the
Py<FrameObject> address stored in the stack iframe's materialized field
Assisted-by: Claude
* Fix Windows clippy: gate unix-only imports and variables
FrameObject, Py imports and top_iframe variable are only used in
unix+threading cfg blocks. Gate them properly for non-unix builds.
Assisted-by: Claude
* Fall back to FrameObject when tracing; fix cross-thread frame access
- invoke_with_locals: fall back to heap-allocated FrameObject path
when use_tracing is active, so trace/profile callbacks fire correctly.
This fixes test_trace, test_bdb, test_monitoring regressions.
- set_current_frame: publish top_iframe alongside top_frame in ThreadSlot
so cross-thread readers (sys._current_frames) can materialize
stack-allocated frames from other threads under stop-the-world.
- get_all_current_frames: use TLS CURRENT_FRAME for current thread,
fall back to top_iframe materialization for other threads.
- Use set_current_frame (not nosave) in with_iframe for proper
cross-thread visibility.
Assisted-by: Claude
* Apply rustfmt
Assisted-by: Claude
* Fix Windows: gate unix-only top_iframe_ptr variable
Assisted-by: Claude
* Fix frame chain: f_back, retained_back, owner, faulthandler
- Set materialized frame owner to FrameObject (not Thread) so
frame.clear() works on traceback frames
- Fix f_back for materialized frames: walk TLS chain to find source
iframe and materialize its previous frame
- Fix faulthandler dump_traceback to use dump_live_frames (reads
live lasti from stack iframes instead of stale materialized copies)
- Add retained_back propagation in with_frame cleanup: when a frame
escapes, materialize caller and mark it escaped for chain propagation
- Fix release_datastack_frame: don't overwrite retained_back if
already set by with_frame cleanup
Assisted-by: Claude
* Apply rustfmt
Assisted-by: Claude
* Refine with_frame retained_back: use frame_obj instead of materialize
Avoid materializing caller frames in with_frame cleanup to prevent
creating localsplus snapshots that keep extra refcounts. Only use
existing FrameObjects (frame_obj) for retained_back. Also keep
previous pointer alive in release_datastack_frame since the caller
is still executing at that point.
Assisted-by: Claude
* Fix f_lineno: use Cell<u32> for prev_line, update on every instruction
prev_line was only updated when tracing was active, causing f_lineno
to return stale values when observed mid-CALL (e.g. from warnings.warn
or sys._getframe). Changed prev_line from u32 to Cell<u32> for
interior mutability so it can be read safely while ExecutingFrame
holds a shared reference, and update it on every instruction so
f_lineno always returns the correct line.
Assisted-by: Claude
* Fix f_lineno for materialized frames: read live prev_line from TLS chain
Materialized FrameObjects have a snapshot of prev_line from materialize
time. When f_lineno is called on a materialized frame that is still
executing (its source iframe is on the TLS chain), walk the TLS chain
to find the source iframe and read its live prev_line. This fixes
lineno tracking in warnings, gettext, and other modules that read
frame line numbers during function calls.
Also fixed the pointer comparison: use from_payload_ptr to convert
FrameObject payload address to Py<FrameObject> address for matching
against the materialized field.
Assisted-by: Claude
* Sync live frame state for materialized frames
- Add find_live_source_iframe() to walk TLS chain and find the live
source InterpreterFrame for a materialized FrameObject
- f_lasti: read live lasti from source iframe when available
- f_lineno: use read_volatile for live prev_line from source iframe
- sync_visible_locals_to_mapping: read live localsplus from source iframe
- framelocalsproxy_getval: read live localsplus from source iframe
- with_iframe cleanup: sync localsplus, prev_line, lasti to materialized
FrameObject when the frame returns
- sync_fastlocals_from: new LocalsPlus method to update fastlocals
Fixes test_inspect (stale f_locals, stale lineno in positions),
test_listcomps (stale iteration variable in locals()).
Assisted-by: Claude
* Fix retained_back: use materialize_chain and strong_count check
- Add materialize_chain() that creates a lightweight FrameObject with
empty localsplus for f_back chain building only. This avoids cloning
local variables which would create extra refcounts (fixes test_memoryview).
- with_frame cleanup: use strong_count > 1 instead of escaped flag to
determine if retained_back is needed, ensuring chain propagation.
- with_iframe cleanup: use materialize_chain for retained_back to avoid
extra refcounts from localsplus snapshots.
Fixes test_memoryview (refcount from retained_back chain) and
test_traceback.test_extract_stack (f_back chain depth).
Assisted-by: Claude
* Fix cross-thread f_back chain for sys._current_frames/exceptions
- get_all_current_frames: materialize entire frame chain and link
retained_back during stop-the-world for other threads
- f_back: use stop-the-world to safely materialize cross-thread frame
chains on unix when prev iframe is on another thread
- f_back: check frame_obj() before STW for fast path
Fixes test_sys.test_current_frames and test_sys.test_current_exceptions.
Assisted-by: Claude
* Fix monitoring LINE events and faulthandler thread dump
- Skip all instrumented opcodes in bytecode loop prev_line update to
avoid defeating InstrumentedLine de-duplication
- Add prev_line update in execute_instrumented for non-RESUME/non-LINE
instrumented opcodes
- update_events_mask: walk TLS iframe chain to re-instrument all frames
including stack-allocated ones (fixes missing LINE events for inline
frames already past RESUME)
- faulthandler: use top_iframe fallback for stack-allocated frames in
dump_all_threads and watchdog_thread
- faulthandler: use dump_live_frames for current thread dump
Fixes test_monitoring LINE/CALL tests, test_faulthandler thread dumps.
Assisted-by: Claude
* Fix clippy: remove duplicate Radium import, use previous() helper
Assisted-by: Claude
* Fix materialized frame issues: GC tracking, live locals, f_trace propagation
- Track materialized FrameObjects in GC so cycle collection works
(fixes __del__ not called at exit for exception traceback cycles)
- materialize_chain returns owned PyRef without temporary_refs to
avoid defeating GC cycle detection from non-GC-tracked storage
- framelocalsproxy_setval writes to live source iframe when available
(fixes f_locals proxy writes not affecting executing frame)
- f_lineno uses lasti-based line for non-executing frames instead of
prev_line (fixes PEP 626 wrong line after exception unwind)
- f_trace/f_trace_lines/f_trace_opcodes setters propagate to live
source iframe (fixes pdb set_trace not working through materialized
frames)
- frame.clear() checks find_live_source_iframe to reject clearing a
frame that is backed by a live stack-allocated iframe
- Fix clippy warnings and unused imports in faulthandler
- Remove unsafe top_iframe access from faulthandler watchdog thread
Assisted-by: Claude
* Fix faulthandler crashes, GC tracking timing, and test_generators
- Use top_iframe instead of top_frame for frame chain walking in
faulthandler (dump_all_threads and watchdog), avoiding stale or
GC-cleared FrameObject pointer dereference
- Defer GC tracking of materialized FrameObjects to with_iframe
cleanup (after set_current_frame restores old chain), preventing
premature collection while temporary_refs still holds the only ref
- Add FrameObject::try_iframe() for safe access to potentially
GC-cleared frames
- Remove expectedFailure for test_exhausted_generator_frame_cycle
(now passes with GC tracking fix)
- Add 'noalias' to cspell dictionary
Assisted-by: Claude
* Fix Windows frame dump and remove dead code
- Publish top_iframe on all platforms (not just unix) so Windows
faulthandler and sys._current_frames can see stack-allocated frames
- Use top_iframe instead of frames Mutex in Windows faulthandler
dump and watchdog paths
- Use top_iframe in non-unix get_all_current_frames for sys._current_frames
- Remove unused dump_traceback_thread_chain (replaced by top_iframe walk)
- Mark test_pdb_await_support as expected failure
Assisted-by: Claude
* Fix ThreadSlot init on all platforms, skip test_pdb_await_support
- Add top_iframe to ThreadSlot initializer in init_thread_slot_if_needed
(fixes wasm32, Windows compile errors)
- Skip test_pdb_await_support on RustPython (async pdb exception
callback receives None exc argument)
Assisted-by: Claude
* Remove dead faulthandler code (dump_frame_from_ref, dump_traceback_thread_frames, try_iframe)
These functions are no longer called after switching all frame chain
walking to use top_iframe directly.
Assisted-by: Claude
* Fix Windows sys._current_frames: materialize full frame chain
The non-unix get_all_current_frames path was only materializing the
top iframe without linking retained_back, so f_back chain walking
could not find deeper frames like f123() in test_sys tests.
Replicate the full chain materialization from the unix path.
Assisted-by: Claude
* Use STW-based cross-thread f_back on all platforms
Replace the non-unix frames-mutex fallback with the same
stop-the-world iframe chain materialization used on unix.
Fixes test_current_exceptions on Windows where stack-allocated
iframes were not found in the FrameObject-only frames list.
Assisted-by: Claude
* Fix set_f_lineno to write to live iframe, use STW for cross-thread access
- set_f_lineno: write lasti/pending_stack_pops to the live source
iframe instead of the materialized copy so debugger jumps take
effect on stack-allocated frames
- f_back cross-thread: enter STW before dereferencing the prev
pointer to prevent use-after-free if the owning thread returns
- get_all_current_frames (non-unix): add STW protection when
walking cross-thread iframe chains, matching the unix path
Assisted-by: Claude
* Guard current_location() against lasti==0, always capture retained_back
- current_location(): return first_line_number instead of
panicking when lasti is 0 (before first instruction executes)
- with_frame retained_back: materialize the caller iframe when
needed so f_back always resolves for escaped FrameObjects
Assisted-by: Claude
* Fix format, revert eager retained_back materialization
Remove double blank line in frame.rs (lint failure).
Revert with_frame retained_back to only use already-materialized
caller FrameObjects — eagerly materializing the caller added
refcounts on local variables, preventing timely deallocation and
causing test_io, test_memoryview, and test_futures failures.
Assisted-by: Claude
* Remove rustpython-unicode-isolation-issue.md from gitignore
Assisted-by: Claude
* Remove test_pdb_await_support skip: async pdb now works correctly
Assisted-by: Claude
* Fix STOP_ITERATION monitoring: wrap value in StopIteration instance
fire_stop_iteration was passing the raw iterator return value to
callbacks, but the STOP_ITERATION event callback signature expects
a StopIteration exception instance. Wrap non-StopIteration values
in a new StopIteration(value), matching PyMonitoring_FireStopIterationEvent.
This fixes test_pdb_await_support where bdb's exception_callback
received None instead of a StopIteration instance.
Assisted-by: Claude
|
||
|
Shahar Naveh
|
fb1b4734a7 |
[zizmor] ignore superfluous-actions (#7548)
|