* Fix marshal recursive reference loading
Create reference-tracked containers before reading their children so recursive list, dict, set, and tuple graphs can be unmarshaled. Preserve interned string markers through the runtime bag and add an initialization-only tuple construction path.
Assisted-by: Codex:gpt-5
* Preserve marshal container insertion errors
Keep Python exceptions raised while constructing unmarshaled sets, frozensets, and dictionaries instead of collapsing them into ValueError. This makes abnormal recursive hash-container streams report TypeError like CPython and removes the remaining test_marshal expected failure.
Assisted-by: Codex:gpt-5
* Enable full abnormal marshal reference loop test
* Test direct marshal tuple reference loop
Assisted-by: Codex:gpt-5
* Reuse stored hashes when building a set from a set/frozenset/dict
set and frozenset recomputed __hash__ for every element even when the
source object already stored a hash per entry. CPython's
set_update_internal branches on PyAnySet_Check / PyDict_CheckExact and
feeds set_add_entry the hash read from the source table; RustPython
always iterated generically.
Split the hash computation out of the Dict entry points so callers can
supply a hash they already hold, add keys_with_hashes() to hand out
(key, hash) pairs, and take the fast path in the set constructors and
in the set operations whose argument is a set/frozenset/exact dict.
ArgIterable::as_object() exposes the pre-__iter__ object so the set
operations can dispatch on the source type without changing any of
their signatures.
Closes#8489. dict.fromkeys() is the dict-target counterpart and is
tracked in #8490, so test_do_not_rehash_dict_keys keeps its
expectedFailure marker until that lands too.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Rename the hash-carrying variants to *_known_hash
`_with_hash` said nothing about which direction the hash travels, and the
same suffix was already used both ways in this file: keys_with_hashes()
hands hashes out, while insert_with_hash() takes one in. The pre-existing
insert_with_hint()/get_with_hint() pair has the same problem.
Follow CPython's "KnownHash" variants (_PyDict_SetItem_KnownHash,
_PyDict_Contains_KnownHash, _PyDict_DelItem_KnownHash) instead, so the
suffix marks the argument direction and matches the name a reader
familiar with CPython already expects. keys_with_hashes() keeps `with`
because it really does return the hashes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Narrow visibility of the new helpers and trim their docs
from_object() and ArgIterable::as_object() had no caller outside their own
file and crate respectively, so drop them to private and pub(crate). The
dict_inner helpers stay pub(crate) because builtins::set calls them.
Also shorten the doc comments to match the density of the surrounding
code; only insert_known_hash keeps a real note, since a wrong hash there
corrupts the table silently.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* map: raise TypeError when called with no iterables
Assisted-by: Claude Code:claude-fable-5
* add test for map with no iterables
Assisted-by: Claude Code:claude-fable-5
* removed expectedFailures
* removed test for map constructed with no iterables - covered by existing in test_itertools
* Add new_payload_exception helper for built-in payload exceptions
* Use new_payload_exception for SystemExit raise sites
* Use new_payload_exception for BlockingIoError raise sites
* Use new_payload_exception for OSError errno dispatch in slot_new
* Run rustfmt
* Run clippy
* Add type guard to new_payload_exception and extract new_system_exit
* Extract new_system_exit and fix stale expect messages
The four SystemExit raise sites repeated the same construction, so route them through a new_system_exit helper next to new_stop_iteration. The expect messages still described invoke_exception's downcast failure, which no longer applies after moving to new_payload_exception.
* Update crates/vm/src/vm/vm_new.rs
---------
Co-authored-by: Jeong, YunWon <69878+youknowone@users.noreply.github.com>
Rust supports C string literals which automatically create a CStr with a
trailing NUL. `windows-sys` provides an analogous macro for wide
strings. Both of these avoid allocations which is nice for constants.
code.replace() called as_interned_str().unwrap() on its string
arguments, which panics when the caller passes a string that has not
been interned. modulefinder's replace_paths_in_code() builds a fresh
co_filename, so any use of ModuleFinder(replace_paths=...) aborted the
interpreter.
Intern the incoming strings instead, and raise TypeError rather than
panicking when a non-string appears in co_names/co_varnames/
co_cellvars/co_freevars.
Unskips test_modulefinder.test_replace_paths.
Co-authored-by: Pablo Garcia <pablito@MacBook-Neo-de-Pablo.local>
`search_info_literal`'s len>1 arm reset the tail-match cursor by advancing one
character past the matched prefix. That position equals the INFO block's
prefix_skip boundary only when the prefix ends where the skip does; for a
pattern like `ab(cd)`, `_get_literal_prefix` reports prefix_len=4 with
prefix_skip=2, so the tail resumed two characters too far.
`state.cursor` already holds `req.start + skip`, which is what
`sre_lib.h` SRE(search) computes as `ptr - (prefix_len - prefix_skip - 1)`,
and which the len==1 arm above already uses. Take it unconditionally.
Searching "xabcdcd" for `ab(cd)` returned span (1, 7) instead of (1, 5);
over a 600-case corpus stamped with CPython's own answers, 94 cases disagreed
before this change and none after. The wrong resume also dropped matches
outright, not only widened them.
Assisted-by: Claude
According to POSIX, `time_t` should be 64 bits. `musl` changed its
`time_t` to an `i64` over five years ago. `glibc` provides compatibility
features that declare `time_t` as either `i32` or `i64`. Rustix uses the
raw Linux syscall for stat which returns an `i64`. For our purposes, an
`i64` makes sense because it's modern and avoids the year 2038 problem.
It also reduces our dependency on what `libc` defines.
Sources:
* https://www.man7.org/linux/man-pages/man3/time_t.3type.html
* https://pubs.opengroup.org/onlinepubs/9799919799/basedefs/sys_types.h.html
* Allow automated commits through pre-commit hooks
* Fix agent C-API test instructions
Exclude rustpython-capi from root workspace test commands and run its tests from the crate directory so its Cargo configuration applies. Document the project AI policy and required commit trailer.
Assisted-by: Codex:gpt-5
CPython's _PyFrame_GetLocals only syncs fastlocals/cells into locals()
for function scope; module/class scope just returns the namespace dict
as-is. sync_visible_locals_to_mapping did this for every scope, so the
implicit __conditional_annotations__ cell (used for PEP 649/749
deferred annotations) broke two ways: at module scope its cell is
always empty, so syncing overwrote the dict's real value with None,
causing NameError on the next annotated statement; at class scope its
cell is the only real value, so syncing leaked it into locals()/dir(),
unlike CPython.
Skip cell/free slots outside function scope to match CPython, and add
a regression snippet.
Closes#8379
Assisted-by: Claude Code:claude-sonnet-5
* Use type-specialized comparators for homogeneous list sorts
Scan the sort keys once before sorting; when every element is exactly str, int, or float, compare wtf8 bytes / BigInt / f64 directly instead of going through rich_compare_bool dispatch, mirroring CPython's pre-sort check in listsort.c (unsafe_latin_compare and friends). Subclasses and mixed-type lists keep the generic __lt__ path.
* Add edge-case tests for specialized list sorts
* Cache the richcompare slot for homogeneous list sorts
* Specialize tuple sorts on their first elements
* Apply ruff formatting to the new list sort tests
* Deduplicate the reverse swap across sort dispatch arms
The cell type filled neither the richcompare nor the repr slot, so
object's address-based defaults showed through: cell(1) == cell(1) was
False, ordering raised TypeError, and repr rendered
<cell object at 0x...> rather than <cell at 0x...: int object at 0x...>.
Compare cells by contents, with empty cells ordering before everything
else, and render CPython's repr for both the filled and empty cases.
The comparison fills the richcompare slot directly rather than going
through Comparable, whose cmp() can only answer with a bool. CPython
returns whatever PyObject_RichCompare produced, so a contained __eq__
that yields a non-bool must pass through untouched; coercing it would
also call __bool__ and surface exceptions CPython never raises. The
empty-cell branch still answers with a bool, so both arms are needed and
the slot returns Either.
The repr truncates the contained type name the way "%.80s" does: at most
80 bytes, dropping a character the cut would leave incomplete.
Mark the type unhashable. Content-based equality combined with the
inherited identity hash would break the hash/eq contract, and hashing the
contents is not possible either because cell_contents is writable.
CPython gets this implicitly, because defining tp_richcompare suppresses
tp_hash inheritance.
Reference: CPython Objects/cellobject.c, cell_richcompare and cell_repr.
Assisted-by: Claude Code:claude-opus-5
* Factor with_iframe into enter_iframe/exit_iframe helpers
Extract the frame entry (recursion check, TLS link, exception save)
and exit (materialization sync, TLS restore, GC tracking) logic
from with_iframe into standalone enter_iframe/exit_iframe methods.
with_iframe now calls them, with no behavioral change.
This prepares for the trampoline loop where enter/exit are called
individually rather than wrapped around a closure.
Assisted-by: Claude
* Allocate InterpreterFrame and LocalsPlus together on the datastack
Add InterpreterFrame::new_on_datastack() that bump-allocates both the
InterpreterFrame struct and its LocalsPlus data array in a single
datastack push, eliminating one allocation per function call.
Update datastack_frame_size_bytes_for_code() to include InterpreterFrame
size. Convert invoke_prepared_exact_args() and the invoke() fast path to
use the combined allocation.
Add release_datastack_frame() method on InterpreterFrame that drops all
localsplus values, runs field destructors (trace, temporary_refs,
retained_back, etc.), and returns the datastack base pointer for pop.
Assisted-by: Claude
* Implement trampoline loop for Python-to-Python calls
Add ExecutionResult::TailCall variant and a trampoline in
run_frame_fast that flattens Python-to-Python calls into a single
Rust stack frame instead of recursing through the eval loop.
CallPyExactArgs now prepares the callee frame on the datastack and
returns TailCall when tailcall_enabled is set (run_iframe path only).
The trampoline dispatches via a state machine (EnterCallee / ReturnValue
/ Unwind) in a single loop, avoiding mutual recursion between helper
functions that would exhaust the C stack.
Exception propagation through suspended frames uses
trampoline_handle_exception which adds traceback entries and calls
unwind_blocks on each caller.
Assisted-by: Claude
* Optimize trampoline: skip recursion check, avoid temporary_refs mutex, add bound method TailCall
- Add enter_iframe_unchecked for trampoline callee entry (recursion
already checked by specialization_call_recursion_guard)
- Move callable ownership from per-frame temporary_refs mutex to
trampoline-local SuspendedFrame.owned_refs via VM side channel
- Add TailCall support for CallBoundMethodExactArgs
- Move args directly from caller stack to callee fastlocals
- Read materialized pointer once in exit_iframe
Incremental call overhead: ~55 ns -> ~35 ns
Assisted-by: Claude
* Use UnsafeCell for pending_tailcall_refs
The VM is per-thread so RefCell's runtime borrow checking is
unnecessary overhead. Replace with UnsafeCell for direct access.
Assisted-by: Claude
* Remove .claude/settings.json from tracking
Assisted-by: Claude
* Replace SendPtr with Option<SendNonNull> for type-safe pending tailcall
Use NonNull + Option instead of raw *mut T with manual null checks.
The compiler enforces non-null via the type system, and Option<NonNull>
has the same size as a raw pointer thanks to niche optimization.
Also extract take_pending_tailcall helper to deduplicate the pattern.
Assisted-by: Claude
* Restrict PendingFrame to private, expose only set/take methods
Rename SendNonNull to PendingFrame and make it fully private: the
struct, its field, and the pending_tailcall_frame Cell are all
non-pub. External code accesses the side channel only through
set_pending_tailcall (pub(crate)) and take_pending_tailcall (private).
This ensures the unsafe Send+Sync impl cannot be reused elsewhere
without justifying a new safety argument.
Assisted-by: Claude
* Restore C-stack overflow check in trampoline enter_iframe_unchecked
enter_iframe_unchecked was skipping C-stack checks under the
assumption that the trampoline stays in one Rust stack frame. But
each run_iframe call still consumes Rust stack, so deep Python
recursion through the trampoline can exhaust the C stack (observed
as STATUS_STACK_OVERFLOW on Windows CI).
Keep the C-stack check (every 8th call) while still skipping the
Python recursion depth check (already done by
specialization_call_recursion_guard).
Assisted-by: Claude
* Address code review: entry frame double-free, panic safety, cleanup
- Fix double-free: mark entry frame with is_entry flag in
SuspendedFrame so the trampoline skips its datastack release
(the caller owns that cleanup)
- Clear iframe.previous in exit_iframe before unlinking the chain,
matching with_frame and resume_gen_frame behavior
- Add scopeguard in with_iframe for panic safety
- Use saturating_sub(1) for lasti in trampoline_handle_exception
- Extract datastack_iframe_localsplus_offset helper to avoid
duplicated alignment computation
Assisted-by: Claude
Validate decimal integer literals from the parser token stream before
lowering the parsed AST. Reuse the validator for normal compilation and
the `_ast` parsing path so `PyCF_ONLY_AST`, `ast.parse()`, and
`ast.literal_eval()` honor `int_max_str_digits` consistently.
Remove the now-passing `test_literal_eval_str_int_limit` expected-failure
marker. The token-based validator excludes non-decimal integers, floats,
complex literals, strings, and comments without duplicating lexer logic.
Assisted-by: Codex:gpt-5.6-sol
specialization_run_init used prepare_exact_args_frame which heap-allocates
a FrameObject with 4-5 atomic refcount bumps per call. Switch to
invoke_prepared_exact_args which uses InterpreterFrame::new_on_datastack
with zero refcount bumps.
CallAllocAndEnterInit already guards against tracing (eval_frame_active)
and recursion limits before reaching this path.
Assisted-by: Claude
Lee Dogeon