mirror of
https://github.com/RustPython/RustPython.git
synced 2026-10-28 18:59:04 +08:00
copilot/fix-static-build-core-dump
4344
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Luan Taraschi
|
377c9dbc78 |
Follow CPython's padding rules in a2b_base64 (#8570)
With strict_mode=True the decoder accepted padding CPython rejects:
>>> binascii.a2b_base64(b'YWJj=', strict_mode=True)
b'abc'
It also dropped data in the default mode, because it returned at the
first pad that completed a quad:
>>> binascii.a2b_base64(b'abc=a')
b'i\xb7' # CPython: b'i\xb7\x1a'
The loop now counts pads and decides at the end, the way CPython does,
so a stray pad is ignored outside strict mode and named as leading,
excess or discontinuous inside it. DecodeError::InvalidPadding was dead
in this file and now carries the excess padding message.
test_base64_strict_mode and test_base64_excess_data were marked as
expected failures and pass now.
Assisted-by: Claude Code:claude-opus-5
|
||
|
Luan Taraschi
|
3d2ee6466d |
Drop the narrow-build surrogate pairing from unicode_escape (#8569)
unicodeescape_string is a port of CPython 2.7's
PyUnicode_EncodeUnicodeEscape, where the branch mapping a character at or
above 0x10000 to \U00xxxxxx sits under #ifdef Py_UNICODE_WIDE and the
branch folding a surrogate pair sits under the #else. The port kept the
"# ifdef Py_UNICODE_WIDE" and "# endif" comments around the first one but
dropped the #else, so both branches run.
The pairing branch reads the next character without checking that there
is one, so a high surrogate at the end of the string walks off it:
>>> "\ud800".encode("unicode_escape")
IndexError: index out of range # CPython: b'\ud800'
And when there is a next character it folds the two, which no wide build
does:
>>> "𐀀".encode("unicode_escape")
b'\U00010000' # CPython: b'\ud800\udc00'
With the branch gone a high surrogate falls through to the 16-bit case
and comes out as \udXXX, which is where a low surrogate already went.
test_incremental_surrogatepass in UnicodeEscapeTest carried an
expectedFailure for the IndexError. It passes now, so the override goes
with it.
Assisted-by: Claude Code:claude-opus-5
|
||
|
hyoinandout
|
728a61e982 |
csv: validate dialect options (#8402)
* csv: validate dialect options Resolve each dialect once and validate the merged options before constructing readers and writers. Handle Unicode character parsing consistently and enable the corresponding CPython CSV tests. Assisted-by: Tau:gpt-5.6-luna * cargo fmt * Validate csv dialect conflicts with line terminators - Reject dialect characters that overlap any lineterminator character - Validate Dialect construction consistently and allow non-ASCII terminators - Unskip csv invalid character coverage now that validation matches behavior * Remove stale CSV rejection snippet Removes a RustPython-only assertion that contradicts the CPython-compatible dialect validation added by this branch. Scope: test-only AI-Assisted-By: Codex * Revert "Remove stale CSV rejection snippet" This reverts commit |
||
|
Jeong, YunWon
|
dd2cc4d77a |
Align compiler futures, annotations, and symtable with CPython (#8550)
* Align compiler futures and annotations with CPython Handle Barry parsing consistently across AST, type-comment, REPL, and WASM paths. Preserve deferred annotation source and scope metadata, and make _symtable conversion cached and linear-time. Assisted-by: Codex:gpt-5.4 * Fix f-string debug ranges and update Barry/annotate test expectations Compute the debug f-string source range from the raw leading and trailing text instead of the comment-stripped text, so a comment inside a multi-line replacement field no longer shifts the emitted LOAD_CONST location. Restrict the obsolete-operator location shift to `<>` by also requiring a `>` at the reported offset; without it any `ExpectedExpression` that followed a `<` (`2 <;`) was reported one column to the left. Add `CodeFlags::FUTURE_MASK` and use it in builtins, the shell, and the WASM VM instead of repeating the eight-flag union, and collapse the duplicated Barry diagnostic sequences into `BarrySource::diagnostic`. Update the `__annotate__` disassembly snapshot to LOAD_FAST_BORROW and replace the RustPython-only `barry_as_FLUFL` co_flags assertion in extra_tests with the CPython-matching one. Assisted-by: Claude Code:claude-opus-5 * Restore the annotated function in __annotate__ qualnames Signature annotation scopes are compiled in the scope enclosing the annotated function, so the function name has to be folded back into the qualname: `f.__annotate__`, `C.m.__annotate__`, `outer.<locals>.inner.__annotate__`. test_type_annotations test_annotate_qualname covers this. test_pyrepl test_future_barry_as_flufl now passes, so drop its expectedFailure marker. Assisted-by: Claude Code:claude-opus-5 * Pass safepoints in the attached-thread interpreter tests `main_and_subinterpreter_run_sections_overlap` parked its workers on a condvar while attached, and `busy_main_interpreter_does_not_block_subinterpreter` looped over protocol calls that never reach a safepoint. An attached thread that never reaches a safepoint cannot be suspended, so a concurrent process-wide stop-the-world (from the gc test running in parallel) never completes and the already-stopped sibling interpreter stays stopped; the second worker then waits in `wait_while_suspended` until the 30s deadline. Both loops now call `vm.check_signals()` each iteration, and the condvar wait uses `wait_timeout` so the lock is released between safepoints. Reproduced on Linux (4 CPUs): the two tests with `--test-threads=2` failed 2/40 before and 0/40 after; the full `rustpython-vm` lib binary failed 1/30 before and 0/30 after. Assisted-by: Claude Code:claude-opus-5 * Report the obsolete `<>` operator from one place `<>` was located twice: `BarrySource` recorded the token ranges it rewrote to `!=`, and `cpython_parse_diagnostic_override` separately peeked at the bytes around an `ExpectedExpression` location. Both produced "invalid syntax" spanning the whole operator. `prepare_barry_as_flufl_source` now records the operator ranges in the non-Barry case too -- by plain text search, since nothing is rewritten there -- so `BarrySource::invalid_legacy_operator` covers both, and the byte-peeking helper is gone. It also defers to an unclosed bracket earlier in the source, which the diagnostic chain it used to sit in did for it. A `<>` that starts a statement now spans both characters: the parser stops at the `<` rather than one character in, which the byte peek did not handle. Assisted-by: Claude Code:claude-opus-5 * Drop a py object's payload before its class `PyInner` declares `typ` before `payload`, so `drop_in_place` dropped the class first and `PyAtomicRef::drop` left it null. A `PyWeak` payload is still linked into the weakref list of the object it points at until its own `Drop` unlinks it under the stripe lock, which happens after that. `WeakRefList::add` walks the same list looking for a proxy to reuse and reads `node.class()` off every node it passes, so it could dereference the null class of a node on its way out. `PyInner::dealloc` now runs the two destructors in the other order at all three of its exit paths, with a compile-time check that no other field has one. Reproduced by the new test in about 0.1s, as the panic CI hit in the c-api `weakrefobject::tests::new_proxy` run: panicked at crates/vm/src/object/ext.rs:312: unsafe precondition(s) violated: hint::unreachable_unchecked must never be reached Assisted-by: Claude Code:claude-opus-5 |
||
|
Jeong, YunWon
|
86d9407b3d |
Raise where the fuzzer found aborts and stack overflows, and collect with the count in the object header (#8551)
* Correct what the non-unix frame stack is for
The comment said non-unix threading builds have no stop-the-world.
CollectStopTheWorld is gated on `feature = "threading"` alone, and
sys._current_frames stops the world on both paths; the field is the
fallback a reader uses when there is no `top_iframe` to materialize from.
Assisted-by: Claude
* Seek by the whole file position on Windows
`SetFilePointer` answers with the low half of the new position and signals
failure with INVALID_SET_FILE_POINTER, which is also that half of a
position four gigabytes in; telling them apart takes the error code, which
this did not read, so such a seek was reported as an error. Deciding
seekability from it also called such a file unseekable.
`SetFilePointerEx` returns the whole position and a success flag of its
own, which also removes the transmute of the position into halves.
Assisted-by: Claude
* memoryview: tell a value of the wrong kind from one that does not fit
Assigning an item reported every packing failure as a TypeError, so
m[0] = 300 on a 'B' view said the value was the wrong type rather than out
of range. Packing now says which of the two it was, and whether the value's
own code raised, in which case that error is the answer as it is:
m[0] = 300 ValueError: invalid value for format 'B'
m[0] = "x" TypeError: invalid type for format 'B'
m[0] = <__index__ that raises> the raised error
`struct` reports both as `struct.error` and is unchanged; the kind travels
beside the exception for the caller that tells them apart.
Also: None was read as a deletion, so m[0] = None answered "cannot delete
memory" instead of packing it; and deleting through the mapping protocol
never reached the read-only check, which comes first.
Assisted-by: Claude
* Raise MemoryError for allocations sized by Python input
os.read, _RawIOBase.read, int.to_bytes, struct.pack, ctypes array
creation and the _ssl RAND functions sized a Vec from a Python-supplied
length with vec![], which calls handle_alloc_error and, under
panic = "abort", ends the process. They now allocate through
vm.new_zeroed_bytes and raise MemoryError.
itertools.product built its pools without checking that
len(iterables) * repeat is representable; it now raises OverflowError
"repeat argument too large" and reserves the pool and index vectors
fallibly. repeat is read as isize, so a negative one raises ValueError
"repeat argument cannot be negative" instead of the conversion's message.
_ssl.RAND_bytes and RAND_pseudo_bytes read n as i32, matching the int
they are declared with.
Assisted-by: Claude
* Collect with the count in the object header
A collection kept the count it was working with in a table keyed by the
object's address, and the objects it had proved reachable in a second one.
Between them they were hashed once per candidate and twice per edge in the
heap, which is where most of a collection over a live heap went.
The count now lives in `PyInner::gc_refs`, with `GcBits::COLLECTING` saying
it is meaningful, and reachability is `gc_refs == GC_REACHABLE` rather than
membership in a set. Step 5 splits the candidates and clears the bit in one
pass. gcbench, five interleaved pairs, median: a live heap of 423k objects
goes from 0.101s to 0.055s and a dead one from 0.402s to 0.383s.
The bits, generation, owner and count take eight bytes between them. A
64-bit header had those eight as the padding its alignment forces, so it is
unchanged at 48 bytes; a 32-bit header grows from 24 to 28.
Assisted-by: Claude
* Take an explicit thread stack size as a floor in debug builds
A debug build already started Python threads on 8 MiB rather than Rust's
2 MiB default, but an explicit threading.stack_size(N) went through
verbatim. test_threading asks for 256 KiB, and starting a thread on that
walked off the end of the stack: the guard page fault landed in the
prologue of ExecutingFrame::run.
Unoptimized, that prologue reserves 80,848 bytes where the optimized one
reserves 656 -- execute_instruction is #[inline(always)] and LLVM only
colors stack slots from opt-level 1, so the frame is the sum of all 200
instruction arms' temporaries rather than the largest. A Python call costs
88,672 bytes of native stack there, and threading's bootstrap is six
frames deep, so 256 KiB holds less than half of what starting a thread
takes.
The floor reaches thread::Builder only; threading.stack_size() still
answers with what was asked for, and release builds are unchanged.
Assisted-by: Claude
* Check the native stack on every frame entry
The C-stack guard ran on one frame entry in eight. That asks the margin to
cover eight frames rather than one, and it does not: an unoptimized frame
entered through native code takes 88,672 bytes against a debug margin of
262,144. A recursion whose steps re-enter that way -- `__add__` calling
itself, a sort key that sorts -- ran off the end of the stack instead of
raising RecursionError. On a debug build `class Add: __add__ = lambda s, o:
s + o; Add() + 1` segfaulted on the main thread; it now raises, as it does
under CPython and in release builds.
enter_iframe checked and then called enter_iframe_unchecked, which checked
again; it now leaves the check to the one call.
Measured on a call-dominated benchmark, five interleaved pairs: instructions
retired go up 0.17%, about four per call, which is the stack pointer read
and the compare.
Assisted-by: Claude
* Take an iterable's length hint when a list is filled from it
`map_py_iter` read `__length_hint__` only to pass it to `PyIterIter`, and
returned an empty vector when the hint was `isize::MAX` or more. Collecting
through `PyResult` dropped the iterator's lower bound, so nothing reserved
the room the hint asked for.
`list()`, `list.extend()` and `list.__iadd__()` now reserve it and report a
hint they cannot honour as `MemoryError`; a hint that leaves no room for the
elements the list already holds is passed over, as `list_extend()` does.
`tuple()` and the other callers keep filling up without reserving.
`length_hint_opt` errors other than the `TypeError` it already turns into
`None` now reach the caller instead of being dropped.
`__iadd__` and `inplace_concat` went through `extract_cloned`, which reads
`__len__` and not `__length_hint__`; both call `PyList::extend` now, the way
`list_inplace_concat()` calls `list_extend()`.
The tuple, list and dict fast paths of `extract_elements_inner` reserve their
known length, which the `collect()` they used dropped.
Assisted-by: Claude
* Drop map.__length_hint__
`map_methods` has no `__length_hint__`, so `operator.length_hint()` on a map
answers 0, not the length of what it draws from.
The method walked into the length hint of every iterator it holds, and a
chain of maps 10000 long overflowed the native stack answering for the
outermost one. It also took the longest of its iterators, where a map stops
at the shortest.
Assisted-by: Claude
* Ask for a length hint where each caller asks for it
`map_py_iter` asked the iterable it was handed, for every caller, and reported
what asking raised. Only some callers ask it: `list_extend()` and
`_PyBytes_FromIterator()` ask the iterable, `PySequence_Tuple()` asks the
iterator, and the bytearray constructor asks nothing. `tuple()`, `min()`,
`max()`, `collections.deque()` and `f(*x)` raised for an iterable whose
`__len__` or `__length_hint__` does, where they answer.
Which object is asked is now the caller's to say. `sorted()` asks the iterable,
being `PySequence_List()`.
`bytes_from_object()` stood in for `PyBytes_FromObject()`, for
`bytearray_extend()` and for the bytearray constructor, which do not agree on
this: the first two ask, the last does not. It is split, and assigning to a
bytearray slice takes the constructor's side with `PyByteArray_FromObject()`.
`list.extend()` counted what it held before the iterable had been asked, where
`list_extend()` reads `Py_SIZE(self)` after. A `__length_hint__` that adds to
the list made the overflow guard read a count too small and raise `MemoryError`
where nothing is wrong; one that empties it made the guard skip a reservation
that cannot be served.
Assisted-by: Claude
* Settle product's pool count before it reads its arguments
`product_new()` checks `repeat` and works out `npools` before it calls
`PySequence_Tuple()` on any argument, and fills the pools `npools` times.
The pools were filled by repeating the arguments `repeat` times instead, which
walks that many steps even with no arguments to repeat: `product(repeat=2**62)`
counted up to it rather than answering `[()]`. The count was also worked out
after the arguments had been read, so a repeat too large to serve ran their
code first.
Assisted-by: Claude
* Let the bool format answer with the error its value raised
`pack_single()` leaves `'?'` to `PyObject_IsTrue()` and returns what that
raised. Packing classified the error instead, so a `ValueError` from a
`__bool__` came back as "memoryview: invalid value for format '?'".
Assisted-by: Claude
* Release a cell's old value after the lock
`PyCell::set` dropped what it replaced while still holding the mutex guarding
the cell contents. A `__del__` running from that drop and reading the same cell
waited on a lock its own caller held, so `del it` on a closure variable whose
value has such a `__del__` deadlocked.
The replaced value is now released once the guard is gone, as `Py_XSETREF`
stores before it decrefs.
Assisted-by: Claude
* Have a set iterator hold the set it iterates
The iterator kept only a reference to the inner hash table, so in
`it = iter(A(*args))` the `A()` temporary was the last owner and died as the
call returned, before `it` was bound. A `__del__` reading `it` there saw an
unbound name and its error was printed and ignored.
The iterator now holds the set object, as `si_set` does, and releases it once
exhausted. That release happens after the lock is dropped, where
`setiter_iternext()` puts its `Py_DECREF(so)` past `Py_END_CRITICAL_SECTION()`,
so a `__del__` that iterates again does not wait on a lock the call holds.
Assisted-by: Claude
* Release an exhausted iterator's container after the lock
`PositionIterInternal::_next` overwrote its `IterStatus::Active` while the
caller still held the mutex around it. Dropping the container there ran any
`__del__` under that lock, and a `__del__` that iterated the same object again
blocked on it.
`exhaust()` now hands the container back instead of dropping it, and
`locked_step()` releases it after the guard. list, list_reverseiterator, tuple,
str, dict and its views and reverse views, bytes, bytearray, memoryview,
array, deque, and the enumerate and sequence iterators all go through it.
Assisted-by: Claude
* Unskip test_free_after_iterating
Assisted-by: Claude
* Stop listing test_set as an environment polluter
`check_free_after_iterating` no longer leaves an ignored exception behind, and
the job that reruns the listed tests ten times fails once one of them stops
polluting.
Assisted-by: Claude
* Drop winsound imports left unused
`TryFromBorrowedObject`, `exceptions`, and `ToWideString` have no reference in
the module, which fails the Windows clippy line under `-Dwarnings`.
Assisted-by: Claude
* Ask nothing where the caller takes no room
`map_py_iter` asked the iterator for a length hint on behalf of every caller
that does not reserve, and reported what asking raised. Those callers ask
nothing at all: `tuple()`, `f(*x)`, `bytearray(x)`, `min()`, `max()` and
`collections.deque()` answer for an iterator whose `__length_hint__` raises,
where they had been raising it.
The answer was also never spent. It reached `PyIterIter` for a `size_hint()`
the push loop does not read, so the lookup and any call it made were work
thrown away: `tuple()` over a generator drops 22% of its instructions, and 17%
over an iterator with a `__length_hint__` written in Python.
Assisted-by: Claude
* Stop asking an iterator how long it is to walk it
`PyIter::iter` and `PyIter::into_iter` asked for a length hint and reported
what asking raised. Nothing spent the answer: it reached `PyIterIter` for a
`size_hint()` that every caller either loops past or drops, since collecting
into a `Result` reports no lower bound.
23 operations answered for an iterator whose `__length_hint__` raises, where
they had been raising it: `set`, `frozenset` and the nine `set` methods that
take an iterable, `dict.fromkeys` and the dict view operators, `array` and
`array.extend`, `all`, `any`, `sum`, `io.writelines`, `math.fsum`,
`math.prod`, and `csv.writerow` and `writerows`.
Over a generator, `set()` drops 16% of its instructions and `all()` 23%.
`str.join` and `bytes.join` do ask, reaching their elements through
`PySequence_Fast()`, which fills a list from the iterator. They take
`iter_sized()`, which is now the only way to ask. `iter_without_hint` is gone,
its callers being what `iter` already does.
Assisted-by: Claude
* Name the matrix jobs rather than let the matrix name them
A generated job name lists every value in the matrix entry, so `cargo check`
carried the booleans its dependencies and `skip_ssl` keys expand to, and the
snippets job carried its test arguments and timeout. Adding or removing a key
renames the check, which drops it from the required list until that list is
edited to match. Emptying `env_polluting_tests` renamed three checks this way.
`Run rust tests` and `clippy` keep the names they had. `cargo check` drops the
booleans from six of its nine, and the snippets job drops its arguments and
timeout from all three.
Assisted-by: Claude
* Keep a sequence iterator active when an element raises
`PositionIterInternal::_next` exhausted the iterator for any non-`Return`
result, so an error from `__getitem__` ended the walk. `iter_iternext()`
lets go of its sequence for `IndexError` and `StopIteration` alone, and
`PyIterReturn::from_getitem_result` has already turned the first of those
into the second, so only `StopIteration` exhausts now.
Both deque iterators keep exhausting on their own mutation guard, which
`deque_iternext()` does by zeroing the counter before it raises; they
share one step function for it, and the message it raises is lowercased
to match the three other sites in the module.
Assisted-by: Claude
* Name the caller in the bytes conversion TypeError
An object with no iteration protocol reached `PyObject_GetIter`'s "not
iterable" message, and `bytearray.extend()` reported bytes. Each entry
point now checks for the protocol first and names itself:
bytes(object()) cannot convert 'object' object to bytes
bytearray(object()) cannot convert 'object' object to bytearray
bytearray().extend(object()) can't extend bytearray with object
Assisted-by: Claude
* Take a strong count too large to hold as reachable
`start_gc_refs` clipped a count at `GC_REACHABLE - 1`, a number the
per-reference subtraction could still walk down to zero and collect a
live object. It now stores `GC_REACHABLE`, and `subtract_gc_ref` leaves
that value alone.
Assisted-by: Claude
* Raise AssertionError where a snippet asserts False
`assert False` is removed under `-O`, which the snippet suite may run.
Assisted-by: Claude
* Keep raising for a collection that moved under its iterator
A deque, set or dict iterator raised once and then read as spent. The
guard is sticky: `deque_iternext()` looks at the deque's state before the
count it keeps, and `dictiter_iternextkey()` and `setiter_iternext()`
write a size no collection can have, so every later call finds the same
thing and raises again. `dequereviter_next()` is the one exception,
looking at its count first, so it runs out after the first raise.
Both deque iterators now carry `dequeiterobject.counter` rather than
reading a length back from the deque, and the dict and set iterators
compare the size they captured against the collection's own every time
they are asked how much is left, which is what makes that answer nothing
from the moment the collection changes rather than only once the
iterator has raised.
The set's message is capitalized to match `setiter_iternext()`.
Measured against 3.14.6, for each of deque, reversed deque, set, dict
and a reversed dict view: the hint after the change, the error, the hint
after the error, and what a later call answers.
Assisted-by: Claude
* Give apt-get update a deadline to fail on
The step waits on `apt-get update`, which has no deadline of its own, so
a source that takes the connection and then stops answering holds the
job until the workflow's own timeout. The retry that disables the
Microsoft and azure-cli sources runs only when the update exits
non-zero, which a held connection never does; three jobs on this branch
sat on this step for 20 minutes to five and a half hours.
Each attempt is now bounded, and the transports are given a timeout, so
a source that stops answering reaches the retry.
Assisted-by: Claude
|
||
|
mumallaeng
|
ca907d1e62 |
Collapse bare ExpectedExpression to 'invalid syntax'; fix '<>' diagnostic offset (#8540)
* Collapse bare ExpectedExpression to 'invalid syntax'; fix '<>' diagnostic offset `ParseErrorType::ExpectedExpression` currently surfaces as the raw ruff parser message (e.g. "Expected an expression") to callers that only depend on `rustpython-compiler` (no `rustpython-vm`). `rustpython-vm`'s `vm_new.rs` already collapses this to CPython's generic "invalid syntax" for its own callers; mirror that same collapse inside `cpython_parse_diagnostic_override` so non-vm consumers get the same CPython-compatible message. A bare `<>` outside Barry-as-BDFL mode (`2 <> 3`) lexes as `Less` then an unexpected `Greater`, so the resulting `ExpectedExpression` location points at the `>` -- one character past where CPython's tokenizer (which treats `<>` as a single obsolete token) reports the error. Detect the `<` immediately preceding the location and shift the reported range back over it. Assisted-by: Claude Code:claude-sonnet-5 * Require an adjacent '>' before treating '<' as the obsolete <> operator CodeRabbit review on #8540: the previous check only looked at the byte before the ExpectedExpression location for '<', without confirming an adjacent '>' really follows it. For inputs like a trailing '<' at EOF this could misclassify an unrelated ExpectedExpression as the bare '<>' case and report a bogus range. Require both bytes are present before constructing the diagnostic. Also remove the now-stale `@unittest.expectedFailure # TODO: RUSTPYTHON` markers on test_guido_as_bdfl and test_barry_as_bdfl_relative_import, which pass with this fix (test_barry_as_bdfl and test_barry_as_bdfl_look_ma_with_no_compiler_flags still need real Barry-as-BDFL tokenizer support and stay marked). Assisted-by: Claude Code:claude-sonnet-5 |
||
|
Hanjeong LeeandJeong, YunWon
|
d8bb7bbd6f |
builtins: generate accurate __text_signature__ (#8512)
* Drop the $module marker from generated __text_signature__
CPython's C functions receive the module as their first argument, so
PyCFunction.__self__ is the module and inspect strips the $module
parameter when building a Signature. A #[pyfunction] takes no such
argument, PyNativeFunction::zelf is None, and inspect has nothing to
strip, so the marker surfaced as a parameter that does not exist:
inspect.signature(len)
(module, /, obj) # was
(obj) # now
All 45 builtins shared with CPython carried it. Methods are unaffected;
their $self marker comes from func_sig and both branches now produce the
same string.
Assisted-by: Claude Code:claude-opus-5
* Mark generated __text_signature__ parameters positional-only
Arguments bind through `FuncArgs::take_positional`, which pops from the
positional list and never consults the keyword map, so a #[pyfunction]
argument cannot be passed by name:
>>> len(obj=[1, 2])
TypeError
The generated signature omitted the `/` marker, so inspect reported those
parameters as POSITIONAL_OR_KEYWORD, contradicting the call above. Emit
the marker, except for `*args`/`**kwargs`, which cannot be followed by
`/`, and for empty parameter lists.
14 of the 45 builtins shared with CPython now report an identical
signature, up from 0.
Assisted-by: Claude Code:claude-opus-5
* Emit no __text_signature__ when an argument has no name
Arguments bound by a destructuring pattern, e.g.
fn round(RoundArgs { number, ndigits }: RoundArgs, ..)
have no name to report, and func_sig stringified the pattern verbatim:
>>> round.__text_signature__
'($module, RoundArgs { number, ndigits })'
That is not valid Python, so inspect.signature() raised "builtin has
invalid signature". Return None instead, which leaves
__text_signature__ unset and makes inspect raise "no signature found",
the same as for a CPython builtin that has no signature.
Affects round, sum, os.pathconf, binascii.b2a_base64 and
binascii.b2a_uu. Their docstrings are unchanged; only the signature
prefix is dropped.
Assisted-by: Claude Code:claude-opus-5
* Name builtin parameters after CPython
These parameters are positional-only, so their names only ever appear in
__text_signature__ and cannot be used at a call site. Naming them after
CPython makes the generated signatures directly comparable:
bin x -> number
ord string -> character
divmod a, b -> x, y
setattr attr -> name
delattr attr -> name
hasattr attr -> name
isinstance typ -> class_or_tuple
issubclass subclass,typ -> cls, class_or_tuple
aiter iter_target -> async_iterable
23 of the 45 builtins shared with CPython now report an identical
signature, up from 0 before this branch. The remainder need FromArgs to
report the parameters of its own structs, which is left for a follow-up.
Add extra_tests/snippets/builtin_signature.py covering the phantom
module parameter, the positional-only marker, the names above, and the
signature-less builtins.
Assisted-by: Claude Code:claude-opus-5
* Drop expectedFailure from test_module_level_callable_noargs
pydoc's summary line for time.time was "time(module)" because the
generated signature carried a $module parameter that inspect could not
strip. It now reads "time()", as the test expects.
Assisted-by: Claude Code:claude-opus-5
* Guard the signature-less assertions to RustPython
test_snippets runs every snippet under CPython as well, and CPython does
have Argument Clinic signatures for round and sum, so that block only
holds for RustPython.
Assisted-by: Claude Code:claude-opus-5
* Update crates/derive-impl/src/util.rs
* Fix ord's parameter reference after the merge
The merge of main took ord's signature from this branch, which renamed
the parameter to character, and its body from main, which rewrote ord to
accept bytes and bytearray through a parameter named c. The body then
referenced a name that no longer existed and the build failed.
Assisted-by: Claude Code:claude-opus-5
---------
Co-authored-by: Jeong, YunWon <69878+youknowone@users.noreply.github.com>
|
||
|
|
fdd101b896 |
Fix int unicode decimal digits (#8521)
* Accept Unicode decimal digits in int(), Decimal() and complex()
CPython runs a string argument through
_PyUnicode_TransformDecimalAndSpaceToASCII before parsing it, so decimal
digits from any script are accepted:
int('١٢٣') # 123
int('0x١f', 16) # 31
Decimal('١٢٣') # Decimal('123')
complex('１+２j') # (1+2j)
RustPython only did this for float(), which had the transform inlined.
int() handed the raw UTF-8 bytes to bytes_to_int(), whose digit check is
is_ascii_alphanumeric(), so every non-ASCII digit was rejected — even
though float() accepted the same string.
Lift the inlined transform out of float_from_string() into
common::str::transform_decimal_and_space_to_ascii() and apply it to the
str paths of int() and complex() too. The result is always ASCII: as in
CPython, a character that is neither ASCII, whitespace nor a decimal
digit becomes '?' and truncates the string, which no parser accepts at
any base, leaving the caller to raise the error from the original string.
Bytes-like input keeps going straight to the parser, matching CPython's
split between PyLong_FromUnicodeObject and PyLong_FromString.
This unmarks two expectedFailure tests: test_int.test_unicode and
test_decimal.test_unicode_digits.
* Share one PyStr-to-numeric-literal step across int, float and complex
All three constructors need the same thing from a str argument: trim it,
fold Unicode decimal digits and whitespace to ASCII, and give up on a
string holding surrogates. Each expressed that last part differently —
float matched PyKindStr and returned b"", complex leaned on to_str()
returning None, int returned an empty Cow — so the rule lived in three
places at once.
Move it into protocol::numeric_literal_from_str() and have all three call
it. CPython repeats this per type because its wrapper is three lines over
a single PyUnicode representation; ours has to match over Ascii/Utf8/Wtf8,
which is worth writing once.
Only the shared step moves. int keeps its base handling, int and float
keep accepting bytes-like input, complex keeps rejecting it, and each
keeps raising its own error, because none of that is shared.
No behavior change: the CPython differential suite is byte-identical
before and after.
* Drop the now-empty test_unicode_digits override in test_decimal
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Jeong, YunWon <69878+youknowone@users.noreply.github.com>
|
||
|
Jeong, YunWon
|
182175b0c9 |
memoryview: count exports, hash the exporter, and index through __index__ (#8553)
* memoryview: index through `__index__` and name the dimension
A tuple key was taken apart only when every item was already an `int`, so
`m[I(), 2]` with an `__index__` on `I` was reported as an invalid slice key
rather than indexed. What kind of key it is now follows from the types in
it, the way `is_multiindex` decides, and each item is converted where it
sits; an item that answers `__index__` but raises on the way reports that
rather than making the whole key invalid.
That conversion runs Python, which can release the view, so the
multi-dimensional read goes through `unpack_single` and its re-check
rather than reading the bytes itself.
`IndexError` named the index that was out of bounds where
`lookup_dimension` names the dimension it was out of bounds on, counted
from one, and the one-dimensional path had a message of its own.
Assisted-by: Claude
* memoryview: count what a view has exported, and hash the exporter
The view kept no count of the buffers taken from it, so `release()` always
succeeded, even while something was still reading through an export. It now
answers `BufferError` the way `_memory_release` does, and `__exit__` reports
the same refusal.
`memory_hash` asks the exporter for its hash and throws the answer away, so
a view is no more hashable than what it looks at; a read-only view over a
`bytearray` hashed here where CPython refuses it. Asking runs Python, so the
view counts as exported for the duration and a release attempted from inside
that hash is refused rather than obeyed.
Assisted-by: Claude
* bytes, bytearray, memoryview: measure the hex separator as `len()` does
`hex()` took the separator's length from its bytes rather than from the
object, so a `bytes` subclass that answers `__len__` was measured by what it
holds instead of what it says: `b'abcd'.hex(S(b'::'))` with `__len__`
returning 1 was refused where `_Py_strhex_impl` accepts it, and a two-byte
answer was accepted where it refuses. The check also came after the
`bytes_per_sep == 0` and empty-data shortcuts, so `b'abcd'.hex('::', 0)`
never reached it at all.
Measuring runs Python, and the bytes to be written out must not be borrowed
while it does, so the separator is now resolved from the arguments before
the buffer is reached. For a memoryview that also means the view counts as
exported for the duration, so a release attempted from inside `__len__` is
refused.
Assisted-by: Claude
* array: refuse to hash an array
`array.array` is mutable but inherited `object.__hash__`, so an array could
be put in a set or used as a dict key and then changed underneath it. It is
unhashable, as `tp_hash = PyObject_HashNotImplemented` makes it.
A read-only memoryview over an array now reports the exporter as unhashable
too, which is what `memory_hash` asks it for.
Assisted-by: Claude
* Remove expectedFailure from three memoryview tests
test_hash_use_after_free, test_hex_use_after_free and
test_use_released_memory pass.
Assisted-by: Claude
* memoryview: bound the dimensions a cast can name
`cast()` accepted a shape of any length, so `mv.cast('B', (1,)*64 + (8,))`
built a 65-dimensional view where `memoryview_cast_impl` refuses anything
past `PyBUF_MAX_NDIM`. The limit is answered before the shape is looked at
any further, as it is there.
Assisted-by: Claude
|
||
|
Kwak Byoung Min
|
62d04c6d3d |
Add StringIO newline tracking (#8539)
* Report observed StringIO newline types Constraint: Match CPython StringIO newline reporting without changing configured newline translation. Rejected: Separate newline-tracking representation | reuse the existing SeenNewline bitflags. Confidence: high Scope-risk: narrow Directive: Keep observed newline state separate from the configured newline mode. Tested: prek run --all-files; test_memoryio; cargo clippy -p rustpython-vm --lib -- -D warnings; workspace tests excluding the macOS C-API baseline SIGSEGV. Not-tested: Full macOS workspace suite is blocked by the existing rustpython-capi SIGSEGV; the Linux suite is running. Assisted-by: Codex:gpt-5.6-sol * Preserve StringIO state boundaries Reset observed newline state when StringIO is reinitialized and keep the property subject to closed-stream checks. Constraint: Match CPython StringIO reinitialization and closed-stream behavior. Rejected: Returning stale or cached newline state | violates StringIO lifecycle semantics. Confidence: high Scope-risk: narrow Directive: Keep observed newline state tied to the current open StringIO contents. Tested: cargo test --workspace --exclude rustpython_wasm --exclude rustpython-venvlauncher; cargo run --release -- extra_tests/snippets/stdlib_io_stringio.py; cargo run --release -- -m test test_memoryio; prek run --all-files; cargo clippy -p rustpython-vm --lib -- -D warnings. Assisted-by: Codex:gpt-5.6-sol |
||
|
sijun-yang
|
25e76af1ed |
Expose symbol scopes through _symtable (#8538)
Pack symbol scopes with definition flags in _symtable.symbols for Lib/symtable.py. Assisted-by: Codex:gpt-5 |
||
|
Jeong, YunWon
|
aaeab4f754 |
Fix malformed Unicode error constructors (#8497)
* Fix malformed Unicode error constructors Replace message-only Unicode decode and encode errors with fully initialized exceptions, preserve source objects and failure ranges, and remove the obsolete constructors. Assisted-by: Codex:gpt-5 * Fix Windows Unicode CI regressions Assisted-by: Codex:gpt-5 |
||
|
Jeong, YunWon
|
c85b83f97b |
Fix crashes found hunting the last open fuzzing record (#8524)
* specialize: check the member descriptor's type before caching its slot offset
The LOAD_ATTR/STORE_ATTR specializations cached the slot offset of any member
descriptor found on the owner's type and then guarded the specialized
instruction on the type version alone, while descr_get()/descr_set() check on
every access that the instance belongs to the type the descriptor was defined
for. A descriptor taken from a wider class and bound to a narrower one read
past the instance's slot array once the cache warmed up:
class Big:
__slots__ = ("a0", "a1", "a2", "a3", "a4", "a5", "a6", "a7")
class Narrow:
__slots__ = ("z",)
Narrow.x = Big.__dict__["a7"]
o = Narrow()
for _ in range(1000):
try: o.x
except TypeError: pass
# index out of bounds: the len is 1 but the index is 7 (object/core.rs)
A class with no slots at all reached the ext_ref().unwrap() on the same line.
Assisted-by: Claude
* socket: reserve recv()'s buffer fallibly
recv() and recvfrom() handed the caller's bufsize straight to
Vec::with_capacity, so an unreachable size aborted the process through
handle_alloc_error before any syscall was made:
socket.socket().recv(2**62)
# memory allocation of 4611686018427387904 bytes failed -> SIGABRT
try_reserve_exact reports MemoryError instead, which is what CPython raises.
Assisted-by: Claude
* types: count the __call__ and __get__ slot dispatches as recursion
Both wrappers re-enter Python without pushing a frame, so nothing counted the
nesting when the special method named the object it was looked up on:
class C: pass
c = C(); C.__call__ = c
c() # native stack overflow, SIGSEGV
class D: pass
d = D(); D.__get__ = d; D.x = d
d.x # the same, through descr_get
with_recursion around the two dispatches raises RecursionError instead, the
way Py_EnterRecursiveCall bounds a tp_call dispatch. It costs about 5% on a
__call__ dispatch and 3% on a __get__ dispatch through these wrappers.
Assisted-by: Claude
* typevar: show a ParamSpecArgs origin by its repr
ParamSpecArgs and ParamSpecKwargs fell back to a Rust `{:?}` of __origin__ when
it had no __name__. That walks the object graph natively through Debug for
PyInner, where no recursion guard sits, so a single repr() of a deeply nested
chain overflowed the native stack:
a = object()
for _ in range(30000):
a = typing.ParamSpecArgs(a)
repr(a) # SIGSEGV
The origin is formatted with its repr now, which is guarded, and a ParamSpec
origin is recognized by its type rather than by carrying a __name__.
Assisted-by: Claude
* Do not hold a lock across a call back into Python
Three places kept a lock while running code that can reach the same object, so
a callback that touched it wedged the process:
_asyncio.future_add_to_awaited_by(fut, waiter) # waiter.__hash__ adds again
select.select(elements, [], [], 0) # fileno() clears `elements`
select.poll().poll(1000) # SIGALRM handler registers
The future's awaited-by field is read and written under its lock but the set is
built outside it, the list extraction re-reads the list on each step the way
map_iterable_object() does, and poll() waits on a copy of its descriptors. All
three ran forever before and now finish the way they do on CPython.
Assisted-by: Claude
* Validate memoryview.cast() arguments and export negative strides correctly
cast() accepted any struct format and any shape element. A zero-size
format ('0s') and a 0 in the shape both reached a division by zero;
cast() now takes only a native single character format, optionally
'@'-prefixed, and shape elements that are ints greater than zero.
A view with a negative stride starts at its last item, so the bytes it
exported began there and its own offsets walked off the front of them.
Such a view now exports the whole underlying buffer with `start` folded
into the descriptor's offsets, and zip_eq() hands over a whole run only
when both sides are contiguous in the last dimension.
Assisted-by: Claude
Assisted-by: Codex:GPT-5
* Charge native recursion to the stack, not to the frame limit
with_recursion() checked the limit sys.setrecursionlimit() sets and
incremented the same counter that pushing a frame does, so a guard on a
native dispatch spent what Python code had left to call with, and did so
where sys._getframe() cannot see it: test.support.get_recursion_available()
reported frames that were no longer there. Py_EnterRecursiveCall bounds
the native stack instead, which is a separate budget, and the C stack
check with_recursion already performs is that bound.
The snippets pinning the guarded paths nest deep enough to reach the
stack rather than the frame limit.
Assisted-by: Claude
* Report a size that cannot be allocated instead of aborting on it
A size taken from Python went straight into an infallible allocation in
several places, so the process aborted through handle_alloc_error before
any exception could be raised:
- str/bytes/bytearray center(), ljust(), rjust() and zfill() reserved
the padded result for the caller's width
- expandtabs() built its runs of spaces from a tabsize of any width;
the argument is a C int, and a wider one does not fit
- Buffered{Reader,Writer,Random} allocated buffer_size, and read(),
read1() and FileIO.read() their read size
- bytes(n) and bytearray(n) allocated n
- pbkdf2_hmac() allocated the derived key length, which is a C int
Each of these now reports MemoryError, or OverflowError where the
argument does not fit the type it is declared with.
new_zeroed_bytes() leaves the zeroing to the allocator, so a large
request costs the pages that are written to rather than all of them.
Assisted-by: Claude
* marshal: answer allow_code where a code object is written or read
allow_code was answered by walking the whole result a second time, with
no depth counter and no record of what it had already seen, so a value
that referred back to itself or nested deeply enough ran off the native
stack. w_object() and r_object() answer it where the code object is,
inside the walk that already bounds its depth and resolves references.
A container length is read the way r_long() reads one: it is signed, so
a length with the top bit set is out of range rather than four billion
items to reserve room for.
load() no longer holds a borrow of the buffer read() returned across the
seek() it makes afterwards.
Assisted-by: Claude
* Do not lock an object while running code that can reach it
Several places held a lock or a borrow of an object across a call back
into Python, so a callback that touched the same object waited on a lock
its own caller was holding:
- memoryview slice assignment read a source overlapping the destination,
and __setitem__ converted the value while holding the write borrow
- BytesIO.readinto() read into a buffer viewing the same BytesIO
- array.__setitem__ converted the value under the array's write lock,
and mmap.write() read a source viewing the same map
- bytearray.join() and bytearray.__mod__ drove Python with the
bytearray borrowed
- array and bytearray answered "is this resizable" after taking the
write lock, though an export is exactly a borrow someone else holds
A TextIOWrapper cookie now has to name a position inside what was
decoded in characters as well as in bytes; only the byte offset was
checked, and the character count is what read() and tell() index with.
Assisted-by: Claude
Assisted-by: Codex:GPT-5
* Stop asserting a pbkdf2 message that depends on the width of a C long
The snippet asserted "key length is too great.", which pbkdf2_hmac()
only reaches once the length has been converted; where a C long is
narrower than the length asked for, the conversion fails first and says
so instead. Both are OverflowError, which is what the case is about.
test_support.test_get_recursion_depth passes now that a native recursion
guard no longer spends frames get_recursion_depth() cannot see.
Assisted-by: Claude
* Publish and read the same pointer for a thread's top frame
set_current_frame() casts the `Py<FrameObject>` it publishes straight to
`*mut FrameObject`, so ThreadSlot::top_frame holds the object's base.
sys._current_frames() read it back through Py::from_payload_ptr(), which
subtracts the payload offset from what it is given. The reference it took
therefore incremented, and later decremented, a word 48 bytes ahead of
the frame -- inside the object allocated before it, whose OnceLock state
word sits exactly there for two frames adjacent in the size class. The
neighbour then read an initialized-looking cold pointer that had never
been written and locked whatever the uninitialized word addressed, so
the thread that owned it crashed rather than the one that read.
The slot now holds `*mut Py<FrameObject>`, which is what both sides mean.
A thread parked in a call has no FrameObject for its topmost frame, so
top_frame is null there and the reader takes the materialize path
instead: test_sys.test_current_frames never reaches the branch. The
snippet takes _current_frames() against threads that are running.
Assisted-by: Claude
Assisted-by: Codex:GPT-5
* Decide stop-the-world parking under the thread registry lock
do_suspend() published SUSPENDED first and only then re-read `requested`,
restoring itself to ATTACHED if the stop had ended in the meantime. That
made a thread the second writer able to leave SUSPENDED, so a stop whose
completion check had already observed the thread parked could be undone
behind the requester's back:
worker CAS ATTACHED -> SUSPENDED
requester all_non_requester_suspended() -> true, world_stopped = true
requester start_the_world(): requested = false, then walks the registry
worker reads requested == false, stores ATTACHED
With the store landing inside that walk the debug assertion in
start_the_world fires; with the walk already past the slot, a following
stop force-parks the thread DETACHED -> SUSPENDED, counts it as stopped,
and the store then puts it back to ATTACHED with the world declared
stopped and the thread running bytecode.
`requested` is set in init_thread_countdown() and cleared in
start_the_world() with the registry held, and start_the_world() keeps
holding it while releasing every SUSPENDED thread. Taking the registry
around the check and the transition therefore makes the two orders the
only ones possible: park before that release pass and be woken by it, or
find the request already withdrawn and stay ATTACHED. The requester is
left as the only writer that takes a thread out of SUSPENDED, and the
self-restore is gone.
suspend_if_needed() takes the VirtualMachine to reach the registry.
Assisted-by: Claude
Assisted-by: Codex:GPT-5
* Keep an atexit callback alive while it is being compared
atexit.unregister() releases the callback list around each __eq__ call and
identified the entry it had compared by the address of its Box. __eq__ can
call atexit._clear(), which drops that Box, and atexit.register(), whose
new Box lands on the freed allocation; the identity search then matched
the freshly registered callback and removed it.
atexit.register(a); atexit.register(b); atexit.register(c)
# __eq__ runs _clear() then register(d), returns True
atexit.unregister(probe)
left no callbacks registered where CPython leaves d.
Entries are Arc-shared now, so unregister() holds the one it is comparing
and matches it with Arc::ptr_eq: an address cannot be reused while the
comparison that named it is still running.
Assisted-by: Claude
* Hold atexit entries in PyRc rather than Arc
PyObjectRef is Send and Sync only under the threading feature, so an Arc
over a callback entry trips clippy::arc_with_non_send_sync in builds
without it, such as the wasm package. PyRc is Arc there and Rc otherwise.
Assisted-by: Claude
* Do not hold a buffer's storage while waiting for a peer
FileIO.readinto() and socket.recv_into()/recvfrom_into() took the target
buffer's write borrow and kept it for the whole call, including the wait
for data a pipe, socket or terminal may never deliver. What CPython holds
across that wait is the export, which only forbids resizing; the borrow is
a lock every other thread touching the same object waits on, so
threading.Thread(target=lambda: sock.recv_into(buf)).start()
len(buf)
did not answer until the peer sent. A thread parked on that lock is
ATTACHED and never reaches a safepoint, so gc.collect() in a third thread
waited for the peer as well: one incidental read of the buffer stopped the
world from being stopped at all.
The wait now runs against storage of its own and the bytes are copied over
once they arrive, with the export held throughout so the target still
cannot be resized meanwhile. A seekable file answers from itself rather
than from a peer, so FileIO.readinto() writes into the target directly
there and the buffered read path is unchanged.
Assisted-by: Claude
* Do not hold a buffer's storage while waiting to hand it over
socket.send()/sendall()/sendto()/sendmsg() and FileIO.write() kept the
source buffer's read borrow for the whole call, including the wait for a
peer that may never make room. That borrow is a lock every other thread
writing to the same object waits on, so
threading.Thread(target=lambda: sock.sendall(buf)).start()
buf[0] = 1
did not return until the peer read; and a thread parked there is ATTACHED
and never reaches a safepoint, so gc.collect() in a third thread waited
for the peer too -- the same wedge readinto() had on the receiving side.
ArgBytesLike::borrow_buf_unlocked() answers with bytes that survive the
borrow being dropped. An immutable object hands out a plain reference and
locks nothing, so those are sent where they lie and bytes and memoryviews
over them cost nothing; only bytes reached through a lock are copied out
first. The export is held throughout either way, so the source still
cannot be resized while it is being sent.
The regression snippet covers both directions now and is renamed for it.
Assisted-by: Claude
Assisted-by: Codex:GPT-5
* Check select()'s descriptor limit while the sequence is walked
seq2set() collected the whole sequence and compared the result's length
against FD_SETSIZE afterwards. Selectable::try_from_object() calls
fileno(), which runs Python and can append to the list being walked, and
the walk re-reads the list on every step, so the collection had no end to
reach and the comparison was never made. seq2set in Modules/selectmodule.c
checks the count per element instead.
stdlib_select.py gains a fileno() that appends to its own list, and
releases its sockets with close() rather than by dropping the name.
Assisted-by: Claude
* Bound native recursion where the C stack cannot be measured
check_c_stack_overflow() answers no unconditionally under miri and on
musl, where the stack pointer is not read. Since 9c9905aff that check is
all with_recursion() does, so every guard placed on native recursion --
__call__ and __get__ dispatch among them -- was a no-op on those targets
and the nesting ran until the stack ran out.
with_recursion() now counts its own depth on those targets and refuses
past NATIVE_RECURSION_LIMIT_UNMEASURED. The count is separate from the
frame limit sys.setrecursionlimit() sets, and compiles away where the
stack pointer can be read.
Assisted-by: Claude
* Report the failure to allocate pbkdf2's key and Take.readinto's scratch
Both buffers are sized from an argument -- pbkdf2_hmac's dklen accepts up
to i32::MAX, and readinto's from the length of the destination -- and were
built with vec![0u8; n], which aborts the process on allocation failure.
new_zeroed_bytes() raises MemoryError instead.
Assisted-by: Claude
* Read the frozen-code tuple length as a signed length
The '(' branches in read_marshal_str_vec() and read_marshal_const_tuple()
took the length with read_u32() as usize, so a value with the top bit set
read as four billion items rather than as out of range. read_len() is what
every other length in this file goes through, and it reinterprets as i32.
Both readers serve deserialize_code(), which reads only the frozen modules
baked in at build time, so this changes no reachable behavior;
marshal.loads() already went through read_len().
Assisted-by: Claude
* Make the snippets from the fuzzer sweep assert what they check
builtin_str.py expanded a tab to 2**31-1 columns, allocating 2 GiB to
observe that the width is accepted; a string with no tab observes the same
acceptance without laying anything out.
stdlib_array.py caught the refusal of frombytes() on its own exported
buffer and passed silently when nothing was raised. stdlib_hashlib.py
used a bare `assert False` as its failure branch. Both now say so through
the same shapes the other snippets use.
stdlib_socket.py also accepts OverflowError from recv() with a size that
does not fit the platform's C int.
stdlib_threading_current_frames.py indexed the frame chain for "f123"
without first asserting it is there.
Assisted-by: Claude
* Ask for a marshal container's room instead of assuming it
A flagged tuple or list is published in the reference table before its
children are read, and the placeholder was built with vec![none; len].
The length is the input's to choose and read_len() lets it reach
i32::MAX, so marshal.loads(b"\xa8\xff\xff\xff\x7f") -- five bytes -- asks
for 17 GB of element slots and aborts the process where the allocator
cannot serve it. r_object() allocates the container up front too, but
PyTuple_New() reports what it cannot get.
The elements are now reserved with try_reserve_exact() and a refusal is
raised as MemoryError through the decoder's pending-error channel.
PyTuple::new_marshal_placeholder() held nothing but that allocation and
is gone; the caller builds the elements and uses new_ref().
Assisted-by: Claude
* Compare the blocking-buffer snippet against something
measure() asserted len(buf) == len(buf), which holds whatever the length
is; it now takes the length the caller expects. The pipe case compared
the drained total against the whole source, while an unbuffered write()
reports only what it transferred and a signal can cut that short; it now
compares against what write() returned.
Assisted-by: Claude
* Refuse array.frombytes() a source whose items are not bytes
frombytes() reads its argument as bytes, but accepted any contiguous
buffer, so array("i").frombytes(memoryview(array("d", [1.0]))) appended a
double's bytes read as ints instead of raising. array_array_frombytes_impl
requires an itemsize of 1; ArgBytesLike now reports the itemsize so the
same check can be made here.
The BufferError that a resize meets while the array is exported also names
the array rather than repeating bytearray's wording.
stdlib_array.py's frombytes-of-itself case used typecode "i", where the
new check answers before the resize guard is reached; it uses "b" so the
guard is what refuses, and asserts the wider case separately.
Assisted-by: Claude
* Tell apart the ways marshal data can be bad
A type byte no reader knows, a back reference that names nothing, and
TYPE_NULL all came out as a bare "bad marshal data" ValueError. r_object()
answers the first two with "bad marshal data (unknown type code)" and
"bad marshal data (invalid reference)", and read_object() answers the
third with a TypeError, "NULL object in marshal data for object", since
TYPE_NULL stands for no object rather than for a value.
MarshalError gains the three cases and deserialize_value() maps them.
The container-specific wording r_object() uses for a NULL read inside a
tuple or list is not reproduced; the exception type is.
Assisted-by: Claude
* Hash the collector's tables by address rather than by SipHash
A collection keys three sets and two maps by object address, and it visits
every tracked object and every edge between them, so the hashing is a
per-edge cost. Those tables used the default RandomState, whose SipHash
buys resistance against a caller choosing colliding keys -- and nothing
chooses these keys: they are addresses this process handed out into tables
that live and die inside one collection. A profile of gc.collect() over a
423k-object heap spent 45% of its samples in SipHash.
They now hash with a splitmix64 finalizer. The shifts matter: a table
picks its bucket from the low bits and an address arrives with those bits
zeroed by alignment, so a plain multiply leaves every object in a handful
of buckets and is slower than SipHash was.
The reachability walk also copied each object's referent vector out of the
map it was cached in, a second pass over every edge; it reads them in
place, and reference subtraction hands its vector to the map instead of
cloning it.
Measured over 423k live objects: 0.93s to 0.15s. Over 843k dead ones:
3.00s to 0.79s. extra_tests/snippets/stdlib_threading_gc_import.py, whose
collector thread calls gc.collect() in a loop, ran anywhere from 2.7s to
28s and now runs in 3.1-3.5s: a collection that takes longer leaves more
garbage for the next one to walk, so the cost fed back on itself.
Assisted-by: Claude
Assisted-by: Codex:GPT-5
* Keep the collection's candidates and their counts in one table
A collection built a set of candidates and, beside it, a map from the same
addresses to their reference counts. Both were probed for every edge in
the heap -- membership from the set, the count from the map -- so each
edge paid to hash the same address twice, and each candidate paid to be
inserted twice. The map alone answers both questions.
The candidates also keep a walkable order now, which the reference
subtraction pass needs since it writes the counts while reading the
candidates, and which the unreachable set is built from instead of a
set difference.
Over the 423k-object heap measured in the previous commit: 0.15s to
0.13s live, and 0.79s to 0.49s dead.
Assisted-by: Claude
Assisted-by: Codex:GPT-5
* gc: collect referents into one buffer instead of a vector per object
Step 3 allocated a `Vec` for every tracked object to hold its referents
and kept them all in a map until step 4 read them back. The referents now
go into a single growing buffer, with the map holding each object's range
into it.
Adds `PyObject::gc_extend_referent_ptrs`, which appends to a caller's
buffer; `gc_get_referent_ptrs` calls it with a fresh one.
Assisted-by: Claude
* memoryview and struct: match the checks and errors of the reference
memoryview:
- `cast()` accepted a source and destination that are both item types,
which reinterprets the items rather than re-dividing the bytes; one side
now has to be a byte format.
- A cast to `shape=()` returned without checking that the buffer holds
exactly the one item that shape describes.
- `hash()` hashes the bytes, so it now raises ValueError for a view whose
items are not bytes, rather than returning a hash that disagrees with the
value the view compares equal to.
- `tobytes()` takes the `order` argument, with 'F' walking a
multidimensional view down its columns; `BufferDescriptor` gained
`for_each_segment_fortran` for that walk.
struct:
- A value the format has no room for reported "argument out of range"
instead of naming the format and its range. The format character is now
passed to the packing functions to report it.
- `Struct.__new__` no longer reads the format; `__init__` does, so
`__init__` can be called again and a subclass can pass the format up.
Methods raise RuntimeError until it has run, and `Struct` is a base type.
Removes the expectedFailure from test_Struct_reinitialization and
test_struct_subclass_instantiation.
Assisted-by: Claude
Assisted-by: Codex:GPT-5
* io: decide the readinto path by file type, not by seekability
FileIO.readinto wrote straight into the caller's buffer, holding its write
borrow, when the fd was seekable; otherwise it read aside into scratch and
copied. Seekability stood in for "this read answers without waiting on a
peer", which a pipe on Windows breaks: lseek on one succeeds, so the pipe
took the borrow-holding path and every other thread touching that bytearray
waited for the peer.
host_io::reads_without_waiting answers it directly -- seekability elsewhere,
GetFileType() == FILE_TYPE_DISK on Windows.
The regression snippet times each operation separately, so a failure names
the one that waited; it asserts the transfer is still in flight before
checking the export; and the socket case fills the connection until it
refuses rather than assuming a size that outruns it, which SO_SNDBUF on an
already-connected pair does not settle.
Assisted-by: Claude
|
||
|
Jeong, YunWon
|
f24b2570a9 |
Implement PEP 688 and rework the buffer protocol around managed exports (#8523)
* Implement PEP 688 __buffer__ and __release_buffer__ A Python class could not export a buffer: the slot machinery had no bf_getbuffer or bf_releasebuffer, and every consumer acquired buffers as PyBUF_FULL_RO through a module of PyBUF_* constants. Add both slots. PyBuffer::release now runs a Python __release_buffer__ before the exporter's own release, once per acquisition, which PyBuffer tracks with an `acquired` flag that clones do not inherit. An export made by a Python __buffer__ is held by a _buffer_wrapper payload that counts its exports and drops the returned memoryview with the last one, and the view handed to __release_buffer__ is a _buffer_window that owns no export, so releasing it inside the hook is inert instead of re-entering it. Replace the PyBUF_* constants with a BufferFlags bitflags type whose composite requests are supersets of the simpler ones, so `contains` answers the REQ_* questions, and pass the request to PyBuffer::from_object. Each consumer now asks for what its counterpart asks for: y* arguments for SIMPLE, w* for WRITABLE, BytesIO.write for CONTIG_RO, bytes(), bytearray() and memoryview() for FULL_RO. memoryview checks the request in memory_getbuf, and array.array and mmap.mmap expose __release_buffer__. Test buffer support with PyObject::check_buffer (PyObject_CheckBuffer) instead of attempting an acquisition, so an exception raised by __buffer__ is no longer reported as the object not being bytes-like, and a __buffer__ with side effects runs once. PyBytesInner becomes a y* conversion as a result: bytes and bytearray methods no longer accept iterables of ints, and find, index, count and __contains__ take the arguments parse_args_finds_byte and bytes_contains describe. A view exports its start offset in the descriptor rather than in its window, which fixes a panic when collecting from a negative-stride view. BytesIO.write rechecks closed after acquiring its buffer, which __buffer__ can close in between. Assisted-by: Claude Code:claude-opus-5 * Rework the buffer protocol around managed exports and view offsets Give `PyBuffer` the `_PyManagedBufferObject` shape: one `bf_getbuffer` acquisition is shared by every handle taken from it, cloning takes another share instead of re-acquiring, and the exporter's release runs once when the last share goes away. Remove `retain`, the unsafe `drop_without_release`, the three `impl Drop`s and the `ManuallyDrop` that stood in for this. Add `abort_acquisition` so a failed request does not run `bf_releasebuffer`. Move the view start into `BufferDescriptor::offset`, the `Py_buffer.buf` analogue, and drop the separate `start` fields on `PyMemoryView` and `PyBufferWrapper`. Slicing goes through `SaturatedSlice::adjust_indices_start`, which reproduces `PySlice_AdjustIndices` and keeps the adjusted start. Fix `zip_eq` to take its contiguous fast path only when both last dimensions are contiguous, and make `for_each_segment` and `zip_eq` handle zero-length and zero-dimensional views. Add `BufferDescriptor::projected` so a request without `PyBUF_ND`, `PyBUF_STRIDES` or `PyBUF_FORMAT` receives a correspondingly reduced descriptor, and reject a request without `PyBUF_INDIRECT` against an exporter that has suboffsets. Copy the source first in `memoryview` slice assignment when both sides reach the same root exporter. Hold the export across the resize in `bytearray.extend`, take `y*` in `marshal.loads`, stop probing the buffer protocol in `FsPath`, rewrite `ord` over the concrete string types, fold `array`'s buffer slot into one `slot_as_buffer`, take `w*`/`y*` in `_overlapped`, and thread the new `offset` field through the `_ctypes` descriptors. Assisted-by: Claude * Re-check a memoryview after a conversion and tighten cast and release `pack_single` and `unpack_single` addressed the buffer with a position taken before `__index__` ran, so releasing the view from that conversion read or wrote outside the exporter's storage, panicking when it had also shrunk. Check the released flag again once the conversion is done, as `CHECK_RELEASED_AGAIN` does. Reject a cast to a format that is not a single native format character with an optional `@` in front of it. An empty format reached a division by its item size of zero. get_native_fmtchar Report a second `__release_buffer__` on the same view as a `ValueError` rather than accepting it, and check that the view belongs to the object first; the silent case is a view that exports nothing. wrap_releasebuffer Compare against another memoryview by reading its view where it lies instead of acquiring a buffer from it, so the restricted view handed to `__release_buffer__` compares equal rather than unequal in one direction only. memory_richcompare Name the type in the unraisable an exception from `__release_buffer__` reports, as `releasebuffer_call_python` does, instead of reporting the exporter object. Assisted-by: Claude * Re-check the destination of a memoryview slice assignment Acquiring the source runs `__buffer__`, which can release the destination view, so check the released flag again once the source is in hand and before the structures are compared, as `copy_single` does. Build the sliced destination as a view that counts as no export, the way a `Py_buffer dest = *view` copy does. Holding one kept the exporter unresizable for the length of the assignment, so a source that released the view and then resized the exporter met a `BufferError` instead of the assignment reporting the released view. Assisted-by: Claude * Answer the memoryview contiguity getsets with the layout `f_contiguous` reported the row-major answer for one dimension and False for any more, so a view laid out both ways, such as one of shape (1, 8), reported False while `check_buffer_request` accepted a `F_CONTIGUOUS` request on it. `contiguous` answered row-major order alone. Move the Fortran-order test from `PyMemoryView` to `BufferDescriptor`, next to the row-major one it differs from only in iteration order, and answer all three getsets with it. Assisted-by: Claude |
||
|
sijun-yang
|
aa3c8b5a0d |
Remove private comprehension flags (#8537)
* Remove the private assignment-expression marker Why: extend_namedexpr_scope() already propagates assignment-expression targets and records DEF_GLOBAL or DEF_NONLOCAL in comprehension scopes. ASSIGNED_IN_COMPREHENSION remained only for conflict checks. Its private bit overlaps CPython's packed LOCAL scope bit. Changes: - Register assignment-expression targets as ordinary assignments. - Detect later target conflicts from existing declaration flags. Assisted-by: Codex:gpt-5.6-sol * Remove the private comprehension iterator flag Why: ITER duplicates the DEF_LOCAL and DEF_COMP_ITER facts already recorded for comprehension targets. Its private bit also overlaps CPython's packed scope field. Changes: - Use those flags for rebinding checks and local restoration around inlined comprehensions. Assisted-by: Codex:gpt-5.6-sol * Preserve source names in named-expression diagnostics Why: Assignment-expression diagnostics exposed the mangled keys used for symbol-table lookup. CPython reports identifiers as written in source. Changes: - Use source names in both comprehension conflict diagnostics. - Enable test_named_expression_invalid_mangled_class_variables. Assisted-by: Codex:gpt-5.6-sol |
||
|
Jiseok CHOI
|
7eb6d38c88 |
sqlite3: add SQLITE_DBCONFIG constants, setconfig() and getconfig() (#8535)
* sqlite3: add SQLITE_DBCONFIG constants, setconfig() and getconfig() Implement Connection.setconfig() and Connection.getconfig() using sqlite3_db_config(), and export all SQLITE_DBCONFIG_* integer constants to the module. Fixes test_connection_config in test_dbapi.py. Assisted-by: GitHub Copilot:claude-sonnet-4-6 * sqlite3: fix setconfig to propagate actual SQLite error When sqlite3_db_config() returns a non-OK result, propagate the actual SQLite error instead of replacing it with a generic 'Unable to set config' message, to match CPython's set_error_from_db() behavior. Assisted-by: GitHub Copilot:claude-sonnet-4-6 |
||
|
Jiseok CHOI
|
70b47dd7c7 |
sqlite3: pass None for NULL authorizer args instead of crashing (#8534)
* sqlite3: pass None for NULL authorizer args instead of crashing CPython's authorizer callback receives NULL for arg1/arg2/db_name/access when not applicable (e.g. SQLITE_READ on a table gives NULL for the database name in some versions). Previously RustPython passed these pointers to ptr_to_str which would crash or produce an error. Now ptr_to_str_or_none is used: NULL pointers become Python None, which matches CPython behavior and allows test_table_access and test_column_access to pass. Assisted-by: GitHub Copilot:claude-sonnet-4-6 * fix(sqlite3): avoid reentrant deadlock in Statement::new sqlite3_prepare_v2 can synchronously invoke the authorizer callback, which may call back into Connection methods (e.g. set_authorizer) that require the same db lock. Holding db_lock across the prepare() call caused a self-deadlock when a callback re-entered the connection. Release the lock after sql_limit check and copy the raw handle before calling prepare(), so FFI calls that can trigger Python re-entrancy happen outside the lock scope. Fixes hang in test_authorizer_concurrent_mutation_in_call |
||
|
Jiseok CHOI
|
32b1f21117 |
sqlite3: add Connection exception attributes and fix autocommit ValueError (#8533)
Add DB-API 2.0 optional extension: expose exception classes as attributes on Connection objects (Warning, Error, InterfaceError, DatabaseError, DataError, OperationalError, IntegrityError, InternalError, ProgrammingError, NotSupportedError). Also fix autocommit validation to raise ValueError (not TypeError) when an unsupported type is passed. Assisted-by: GitHub Copilot:claude-sonnet-4-6 |
||
|
Jiseok CHOI
|
36a59f5f7b |
sqlite3: validate narg before creating functions/aggregates (#8532)
CPython 3.14 added check_num_params() which raises ProgrammingError when narg/n_arg/num_params is out of range (-1..=SQLITE_LIMIT_FUNCTION_ARG). RustPython passed invalid values directly to SQLite, resulting in an OperationalError instead. Add check_num_params() helper and call it in create_function(), create_aggregate(), and create_window_function(). Assisted-by: GitHub Copilot:claude-sonnet-4-6 |
||
|
Jeong, YunWon
|
b109723242 |
Fix the itertools.tee leak and race, the contextvars and generator races, _asyncio's InvalidStateError lookup, and _imp's frozen data (#8518)
* _imp: read get_frozen_object's data as a whole marshal value
The data argument went through deserialize_code(), which reads a code body
without the type byte in front of it, so nothing marshal.dumps() produces was
accepted. Read it with marshal.loads() and require a code object back.
Assisted-by: Claude
* _imp: bind find_frozen's arguments with FromArgs
The arity check was done by hand on a FuncArgs. Take the arguments through a
FromArgs struct instead, which makes withdata keyword-only, and fill in the
data it asks for: the frozen encoding is not marshal, so the code is
re-serialized into what get_frozen_object() reads back.
Assisted-by: Claude
* itertools: give tee's shared buffer a Python type
The buffer was a PyRc<PyItertoolsTeeData>, which is not a Python object, so the
collector could not walk into it and any cycle running through a tee was
uncollectable. Make it the _tee_dataobject type with a traverse, held by
PyRef, and split the rest along the same lines: tee() is a function, _tee is
the iterator type it builds, and _tee takes a single iterable rather than
returning a tuple from __new__. _tee is weak-referenceable, tee() rejects a
negative n with ValueError and reserves its tuple fallibly.
test_itertools.test_tee passes now; its expectedFailure marker is removed.
Assisted-by: Claude
* contextvars: hold the shared context state under locks
The variable map was a RefCell, the enter flag, the context index, the token's
used flag and the variable hash were Cells, and each carried an unsafe impl
Sync. A Context or ContextVar shared between threads overlapped their borrows
and panicked. The map and the per-variable cache are now PyMutex, the flags and
the index are atomics, entering a context is a compare_exchange, and the three
unsafe impl Sync are gone.
The cache also stopped being read through AtomicCell::as_ptr, which raced a
concurrent store on a value holding a PyObjectRef.
Values displaced from the map or the cache are dropped after the lock is
released: __del__ can come straight back into the same context, and the locks
are not reentrant.
Assisted-by: Claude
* generator: read the frame state under the running claim
send(), send_none(), throw() and close() read `closed` and `frame.lasti()`
before `running` was compare_exchanged, so the frame they went on to resume
could be one another thread had already advanced. A resume that decided from
`lasti() == 0` that the generator had not started pushes no value onto the
value stack, and the code after the yield pops one, which underflows the stack.
The compare_exchange now hands back a guard, taken before those reads and
released after maybe_close(), so the generator is retired while it is still
claimed.
Assisted-by: Claude
* itertools: claim a tee's position and its buffer as one step
`_tee::next` read `index` and moved it on afterwards, and
`_tee_dataobject::get_item` released `running` before the value it fetched
from the source was cached.
Two callers on one `_tee` then read the same index, hand out the same value
twice and advance past a value that was never cached, which leaves `index`
past `values.len()` and indexes the buffer out of bounds. Two callers at the
same index on separate tees each fetch a value from the source, and one of the
two is dropped without reaching a caller.
Both claims now cover the read and the update.
Assisted-by: Claude
* _asyncio: require InvalidStateError to be a type
new_invalid_state_error() called whatever `asyncio.exceptions.InvalidStateError`
names and unwrapped the downcast of the result, so a future asked for a result
it does not have panicked once that attribute was rebound to something that is
not an exception:
asyncio.InvalidStateError = lambda *args: 42
_asyncio.Future(loop=object()).result()
The type is looked up the way get_cancelled_error_type() looks its own up, and
raised with new_exception_msg; a lookup that does not produce an exception type
falls back to RuntimeError, as the other arms already did.
Assisted-by: Claude
|
||
|
Jeong, YunWon
|
2690c16fdc |
marshal: refuse a back-reference to an object still being written (#8519)
`w_ref` marks a code or slice entry incomplete until `w_complete`, because the reader rebuilds both from their fields and a `TYPE_REF` issued while those fields are still on the wire names an object that does not exist yet. `WriterRefTable` carries that marker and `write_object_depth` raises `cannot marshal recursion <type> objects` instead of emitting the reference. `test_reference_loop_code`, `test_unmarshallable` and `test_reference_loop_slice` lose their RustPython markers; `test_marshal` is 75 run, 16 skipped. Assisted-by: Claude |
||
|
Jeong, YunWon
|
fd7d107c75 |
marshal: round-trip code constants through the runtime bag (#8516)
* Keep abnormal marshal loop test aligned with CPython Assisted-by: OpenAI Codex:gpt-5.5 * marshal: retain runtime code constants while loading Decode code fields through the runtime MarshalBag so co_consts values that do not fit the compiler constant enum remain available to VM code wrappers while the compiler table receives shape placeholders. Assisted-by: OpenAI Codex:GPT-5.4 * marshal: write and read co_consts through the runtime bag serialize_code gains a serialize_code_with variant that writes each co_consts entry through a caller-supplied writer; serialize_code keeps the BorrowedConstant writer as its default. The VM writer passes its own write_object_depth, so a code constant that Literal holds but BorrowedConstant cannot describe reaches the stream instead of panicking in borrow_obj_constant, and a constant shared with the enclosing object takes an entry in the writer's reference table. PyMarshalBag implements constant_ref_from_value, bytes_from_value, str_from_value and tuple_elements_from_value, which deserialize_code_value_inner requires to decode code fields through the runtime bag. Assisted-by: Claude |
||
|
Jeong, YunWon
|
2ed082a51e |
Fix 33 reproduced fuzzing and static-review defects (#8514)
* mmap: guard move() against dest/src past the mapping size
An offset larger than the mapping made `size - dest` underflow, producing
an out-of-range slice index panic. Check the high side first, as write() does.
Assisted-by: Claude
* exceptions: keep ImportError name/path/name_from out of an empty instance dict
ImportError.__reduce__ unwrapped get_arg(0), which is None when the exception
was constructed with no positional argument, aborting on pickle.dumps(ImportError()).
Fall back to the full args tuple there, and expose name/path/name_from as class
attributes defaulting to None so a bare ImportError() reduces to (cls, ()).
Assisted-by: Claude
* asyncio: stop unwrapping the _current_tasks downcast in current_task()
_asyncio._current_tasks is a reassignable module attribute; the three sibling
functions already degrade gracefully when it is not a dict, current_task() did not.
Assisted-by: Claude
* asyncio: raise TypeError when FutureIter.throw()'s exception class returns a non-exception
exc_type.call() goes through a Python-controlled __new__, so its result can be
any object; the downcast was unwrapped. Report it as a TypeError instead.
Assisted-by: Claude
* sequence: use a fallible reservation for sequence repetition
The guard only rejects a repeat whose per-element size crosses MAX_MEMORY_SIZE,
so (1,) * (10**12) still reached Vec::with_capacity and aborted in the allocator.
Reserve fallibly and surface a MemoryError.
Assisted-by: Claude
* collections: reject an oversized deque repetition with MemoryError
deque * sys.maxsize reached the allocator and aborted with a capacity overflow.
Apply the MAX_MEMORY_SIZE guard the other sequences already carry.
Assisted-by: Claude
* itertools: raise OverflowError for an out-of-range r argument
combinations/combinations_with_replacement/permutations narrowed r with
to_usize().unwrap(), so r=2**64 panicked instead of raising.
Assisted-by: Claude
* math: stream the generic sumprod path instead of collecting both iterables
The big-int path collected each argument into a Vec before multiplying, so an
unbounded iterable exhausted memory. Advance both iterators in lockstep and
accumulate, reporting a length mismatch when only one is exhausted.
Assisted-by: Claude
* _imp: raise TypeError for a second positional argument to find_frozen
withdata is keyword-only and unimplemented; passing it positionally hit an
unimplemented!() and aborted.
Assisted-by: Claude
* _typing: check _idfunc arity before indexing args
_typing._idfunc() with no argument indexed args[0] out of bounds.
Assisted-by: Claude
* exceptions: require a sequence for the ExceptionGroup excs argument
The argument was collected before being validated, so an unbounded iterable
such as itertools.count() exhausted memory. Reject a non-sequence up front.
Assisted-by: Claude
* mmap: treat an inverted find/rfind range as empty
find(b"x", 5, 2) built a slice whose start exceeded its end and panicked.
Assisted-by: Claude
* collections: give deque and defaultdict a GC traverse
Neither type opted into traversal, so a reference cycle through a deque or a
defaultdict was never collected.
Assisted-by: Claude
* itertools: opt the iterator types into GC traverse
cycle and its siblings hold Python references but declared no traverse, so a
cycle built through one of them leaked.
Assisted-by: Claude
* _ctypes: mask an out-of-range int instead of panicking
c_char_p(2**64) and pointer item assignment called .expect() on the narrowing
conversion. Wrap the value to the target width, which is what the C
implementation stores.
Assisted-by: Claude
* hashlib: import _hashlib when a hash module is loaded
The hash object type is a static type owned by _hashlib; calling _md5.md5()
without _hashlib imported hit an uninitialized static type and panicked.
Assisted-by: Claude
* _csv: fall back to the built-in dialect defaults when none is registered
The dialect table is empty until csv.py registers 'excel', so _csv.reader([])
unwrapped a missing entry and panicked.
Assisted-by: Claude
* builtins: reject surrogates in compile()/eval() source instead of panicking
expect_str() panics on a str containing surrogates; convert with try_as_utf8
so eval(chr(0xd800)) raises.
Assisted-by: Claude
* _suggestions: require a list for _generate_suggestions candidates
The argument was collected before validation, so an unbounded iterable
exhausted memory.
Assisted-by: Claude
* lzma: size the filter chain before consuming it
filters= was collected into a Vec before the length check, so an unbounded
iterable exhausted memory. Take the length through the sequence protocol first.
Assisted-by: Claude
* classmethod: opt into GC traverse
staticmethod already declares traverse; classmethod did not, so a cycle through
the wrapped callable leaked.
Assisted-by: Claude
* posix: reject unbounded iterables in posix_spawn and setgroups
argv, setsigdef, setsigmask and setgroups bound an ArgIterable and collected it
before validating, so an infinite generator exhausted memory. Require a
list/tuple for argv, validate signals while streaming, and take setgroups
through the sequence protocol.
Assisted-by: Claude
* _ctypes: size Array slice assignment and _argtypes_ before collecting
Both eagerly collected their argument, so an unbounded iterable exhausted
memory before the length check ran.
Assisted-by: Claude
* sys: propagate the breakpointhook warning failure
warn() was unwrapped, so an unimportable $PYTHONBREAKPOINT under
-W error panicked instead of raising.
Assisted-by: Claude
* structseq: require the sequence argument when constructing a struct sequence
A no-argument construction produced an empty backing tuple, and reading any
named field then indexed out of bounds.
Assisted-by: Claude
* Guard the hash slot dispatch against unbounded recursion
`PyObject::hash` invoked the type's hash slot directly, so an element-wise
`__hash__` following a deeply nested object graph recursed one native frame
per level and overflowed the stack. Wrap the dispatch in `with_recursion`,
matching the repr and rich-compare dispatches in the same file, so
`hash(x)` on a nested tuple/GenericAlias/slice raises `RecursionError`.
Assisted-by: Claude
* genericalias: guard the __parameters__ walk against unbounded recursion
`make_parameters_from_slice` recursed into every list/tuple argument with
nothing counting the frames, so `list[L]` for a self-referential or deeply
nested `L` overflowed the native stack at subscript time. Wrap the descent
in `with_recursion`, which makes the walk fallible; `PyGenericAlias::new`,
`from_args` and `make_parameters` now return `PyResult` and every caller
propagates.
Assisted-by: Claude
* Fix the type confusion in PyAtomicRef's Debug impl
`PyAtomicRef<T>` stores a pointer to a `Py<T>`, as `Deref`, `load_raw`,
`swap` and `Drop` all read it, but `Debug` cast it to a bare `T` and
formatted the object header as payload bytes. For `PyFunction`, whose
`code: PyAtomicRef<PyCode>` has a pointer-chasing `Debug`, that
dereferenced header words and segfaulted.
Cast to `PyObject` instead, which is what `Drop` already does and which
also covers the `PyAtomicRef<PyObject>` and `PyAtomicRef<Option<T>>`
instantiations that have no `Py<T>`.
`_asyncio._enter_task` reached this through `{:?}` in its "Cannot enter
into task" message; format the two tasks with their Python repr, which is
what the message is meant to show.
Assisted-by: Claude
* _sre: disallow instantiating Match
`Match` inherited `object.__new__`, so `M.__new__(M)` produced an instance
whose `regs`/`string`/`pattern` were never filled in by a match run; the
mapping subscript path read them and dereferenced garbage. The type has no
public constructor, so mark it `DISALLOW_INSTANTIATION`, which makes
`M.__new__(M)` raise `TypeError: cannot create 're.Match' instances`.
Assisted-by: Claude
* utils: return the empty repr instead of asserting a non-empty collection
`collection_repr` took the first element with an `.expect()` justified by the
caller's preceding non-empty check. Another thread clearing the collection
between that check and the iteration made the iterator yield nothing and
panicked the worker. Fall back to the caller-supplied empty form, which is
the text those callers already produce for an empty collection.
Assisted-by: Claude
* itertools: advance cycle's index atomically
`cycle.__next__` did `fetch_add(1)` and then reset the index to 0 in a
separate store, so two threads replaying the saved items could both read an
index past the end of `saved` and panic on the slice access. Do the advance
and the wrap in one `fetch_update`.
Assisted-by: Claude
* _ctypes: reject a float argument to a foreign call without argtypes
`conv_param`, the conversion used when `argtypes` is not set, converted its
argument with `try_int`, which goes through `__int__` and so accepts a
float. `libc.strlen(1.5)` therefore passed 1 where the callee expects a
`char *` and the callee dereferenced it. Match `ConvParam`, which does a
`PyLong_Check` and converts nothing: take the branch only for an `int` (or
a subclass, so `True` still converts), and let a float fall through to
"Don't know how to convert parameter".
The branch below it converted a float to a C double, but `try_int` claimed
every float before it could run, so it was dead; `CArgValue::Double`
existed only for that branch and both go. Typed doubles are unaffected —
they travel as `CArgValue::Typed` with code 'd'.
Assisted-by: Claude
* Report the iterator itself from PyIter's traverse
`Traverse for PyIter<O>` delegated to the inherent `PyObject::traverse` of
the object it wraps, so it reported that iterator's referents instead of
the iterator. The iterator's own reference to those referents was then
never subtracted during the collector's reference-subtraction pass, the
referents kept a non-zero gc_refs, and every object reachable from them
was classified as a root.
Any cycle running through a type with a `PyIter` field therefore survived
collection: `map`, `filter`, `zip`, `enumerate`, `reversed` and the
`itertools` iterators all leaked, while the same cycle through a `list`,
`tuple` or `list_iterator` collected. Report the wrapped object, as the
`PyObjectRef`, `PyRef<T>` and `PyStackRef` impls do.
`itertools.tee` still leaks: its shared buffer is a `PyRc<PyItertoolsTeeData>`
rather than a Python object, so the collector cannot see through it.
Assisted-by: Claude
* Remove the obsolete expectedFailure on test_code_module.test_unicode_error
Compiling a source string containing a lone surrogate now raises
UnicodeEncodeError, so the test passes.
Assisted-by: Claude
* itertools: reserve the combination indices fallibly
`combinations` and `combinations_with_replacement` built their index vector
with an infallible allocation, so an `r` that passes the ssize_t check but
does not fit in memory aborted the process instead of raising MemoryError.
Assisted-by: Claude
* Apply the struct sequence constructor's dict argument
`structseq(sequence, dict)` discarded its second argument, so the hidden
fields past `n_sequence_fields` — `tm_zone`, `st_atime` and friends — were
always None when constructed directly or restored from a `(sequence, dict)`
pickle, and a non-dict second argument was accepted silently.
Take the dict, require it to be a dict, and fill the hidden slots the
sequence did not cover from it. A key that names a field the sequence
already supplied, or no field at all, is now a
"got duplicate or unexpected field name(s)" TypeError instead of being
dropped. Both arguments are bindable by name, as `sequence` and `dict`.
`os.stat_result` and `os.statvfs_result` did not accept a second argument
at all; they and `time.struct_time` now share the parsing.
Assisted-by: Claude
* _imp: report the argument count in find_frozen's arity error
Assisted-by: Claude
* Add regression tests for the reproduced crashers
One case per catalog entry, each asserting the behavior the fix produces:
recursion guards, the memory-unsafety sites, the overflow and unbounded
allocation guards, the eager-collection rejections, the unwrap sites, and
the cycles the collector now breaks. Every expected value was checked
against CPython 3.14.
Assisted-by: Claude
* Tolerate a changed-size RuntimeError in the set repr stress test
The mutator and reader threads race on purpose; a "changed size during
iteration" RuntimeError is a valid outcome of that race and should not fail
the test. The panic it guards against is not.
Assisted-by: Claude
* Move the crash regression tests into per-module snippets
crash_regressions.py collected every reproduced crasher in one file. Split it
into the snippet for the module each case exercises, and add stdlib_gc.py,
stdlib_asyncio.py, stdlib_lzma.py, stdlib_threading_set_repr.py and
stdlib_threading_itertools_cycle.py for the cases with no existing home.
The suite runs every snippet under the host CPython too, so the checks only
RustPython raises are guarded by sys.implementation.name: the hash and
__parameters__ recursion depth, and the deque repeat overflow. The float
ctypes argument accepts either TypeError or ctypes.ArgumentError.
Assisted-by: Claude
|
||
|
이광효
|
670152068c |
itertools: defer dropwhile predicate validation (#8504)
Store the `dropwhile` predicate as a Python object and call it while advancing the iterator. This defers callable validation until the predicate is first needed, matching CPython for empty input while preserving exception propagation during iteration. Remove the now-passing `test_dropwhile` expected-failure marker. Assisted-by: Codex:gpt-5.6-sol |
||
|
Sanghun LeeandClaude Opus 5
|
db5de5e5e2 |
Reuse stored hashes in dict.fromkeys() (#8503)
* Reuse stored hashes in dict.fromkeys() Closes #8490. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Make fromkeys_known_hashes a PyDict associated fn --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Jeong, YunWon
|
d64cc2cff5 |
Fix marshal recursive reference loading (#8501)
* Fix marshal recursive reference loading Create reference-tracked containers before reading their children so recursive list, dict, set, and tuple graphs can be unmarshaled. Preserve interned string markers through the runtime bag and add an initialization-only tuple construction path. Assisted-by: Codex:gpt-5 * Preserve marshal container insertion errors Keep Python exceptions raised while constructing unmarshaled sets, frozensets, and dictionaries instead of collapsing them into ValueError. This makes abnormal recursive hash-container streams report TypeError like CPython and removes the remaining test_marshal expected failure. Assisted-by: Codex:gpt-5 * Enable full abnormal marshal reference loop test * Test direct marshal tuple reference loop Assisted-by: Codex:gpt-5 |
||
|
Jeaeun Kim
|
643039de94 |
Fix __qualname__ of compiler-generated __annotate__ functions (#8498)
Assisted-by: Codex:5.6-sol |
||
|
William Goode
|
e8431490e7 |
Fix map no iterables (#8478)
* map: raise TypeError when called with no iterables Assisted-by: Claude Code:claude-fable-5 * add test for map with no iterables Assisted-by: Claude Code:claude-fable-5 * removed expectedFailures * removed test for map constructed with no iterables - covered by existing in test_itertools |
||
|
Jeaeun Kim
|
bcf5c4b193 |
ssl: support explicit session reuse (#8466)
* ssl: support explicit session reuse Assisted-by: Codex-5.6-sol * ssl: support explicit TLS 1.3 session reuse Assisted-by: Codex-5.6-sol |
||
|
MsfPabloandPablo Garcia
|
5b3eb41b49 |
Fix panic in code.replace() with non-interned strings (#8471)
code.replace() called as_interned_str().unwrap() on its string arguments, which panics when the caller passes a string that has not been interned. modulefinder's replace_paths_in_code() builds a fresh co_filename, so any use of ModuleFinder(replace_paths=...) aborted the interpreter. Intern the incoming strings instead, and raise TypeError rather than panicking when a non-string appears in co_names/co_varnames/ co_cellvars/co_freevars. Unskips test_modulefinder.test_replace_paths. Co-authored-by: Pablo Garcia <pablito@MacBook-Neo-de-Pablo.local> |
||
|
chestnut1717
|
6ec43604a2 |
Fix EOF SyntaxError diagnostics (#8429)
* Fix EOF SyntaxError diagnostics * Fix Scope EOF SyntaxError location conversion * Fix Additional Unexpected Test * Docs Add Annotation source_location_in_code_points() |
||
|
Seonghun An
|
12a0a1ed8b |
Implement negative zero coercion format option (#8475)
Assisted-by: Codex:gpt-5 |
||
|
OkJa
|
bb75624a59 |
Implement cell comparison and repr (#8458)
The cell type filled neither the richcompare nor the repr slot, so object's address-based defaults showed through: cell(1) == cell(1) was False, ordering raised TypeError, and repr rendered <cell object at 0x...> rather than <cell at 0x...: int object at 0x...>. Compare cells by contents, with empty cells ordering before everything else, and render CPython's repr for both the filled and empty cases. The comparison fills the richcompare slot directly rather than going through Comparable, whose cmp() can only answer with a bool. CPython returns whatever PyObject_RichCompare produced, so a contained __eq__ that yields a non-bool must pass through untouched; coercing it would also call __bool__ and surface exceptions CPython never raises. The empty-cell branch still answers with a bool, so both arms are needed and the slot returns Either. The repr truncates the contained type name the way "%.80s" does: at most 80 bytes, dropping a character the cut would leave incomplete. Mark the type unhashable. Content-based equality combined with the inherited identity hash would break the hash/eq contract, and hashing the contents is not possible either because cell_contents is writable. CPython gets this implicitly, because defining tp_richcompare suppresses tp_hash inheritance. Reference: CPython Objects/cellobject.c, cell_richcompare and cell_repr. Assisted-by: Claude Code:claude-opus-5 |
||
|
이광효
|
1fa76a3ae0 |
ast: enforce integer string limits in AST-only parsing (#8454)
Validate decimal integer literals from the parser token stream before lowering the parsed AST. Reuse the validator for normal compilation and the `_ast` parsing path so `PyCF_ONLY_AST`, `ast.parse()`, and `ast.literal_eval()` honor `int_max_str_digits` consistently. Remove the now-passing `test_literal_eval_str_int_limit` expected-failure marker. The token-based validator excludes non-decimal integers, floats, complex literals, strings, and comments without duplicating lexer logic. Assisted-by: Codex:gpt-5.6-sol |
||
|
Hanif AriffinandJeong YunWon
|
74ae2d4711 |
Fix bytes.isspace (#5655)
* Fix implementation of bytes.isspace to match the implementation in CPython. * Fix implementation of bytes.istitle so that test_bigmem now passes. Also added some extra_test to capture this. * Fix some clippy issues Signed-off-by: Hanif Ariffin <hanif.ariffin.4326@gmail.com> * bytes.istitle returns false when the bytes is empty. * Fix some edges of istitle with non-alphanumeric characters * Disable a line because python 3.12 errors on bad escape sequence * Match CPython's istitle a little closer * Fix clippy warning * Rewrite bytes.istitle along _Py_bytes_istitle The lookahead-based check accepted b"Not--a Titlecase String", which test_bytes covers. Track whether the previous byte was cased instead. isspace compares bytes directly rather than converting to char. Restore the b"omkmok\Xaa" assert: 3.14 reports an invalid escape sequence as a SyntaxWarning, not an error. Assisted-by: Claude --------- Signed-off-by: Hanif Ariffin <hanif.ariffin.4326@gmail.com> Co-authored-by: Jeong YunWon <jeong@youknowone.org> |
||
|
Jeaeun Kim
|
12f3646e7b |
ssl: store SSL socket owners as weak references (#8423)
Assisted-by: Codex:gpt-5.6-sol |
||
|
Jeaeun Kim
|
aa4f98a765 |
fix(socket): use object repr in deallocation warnings (#8422)
Assisted-by: Codex:gpt-5.6-sol |
||
|
Yubin Kim
|
dc1cae490b |
Bind method descriptor when __get__ owner is omitted (#8404)
`method_descriptor.__get__(obj)` raised a TypeError when the owner (the optional second argument) was omitted, because the METHOD-flag branch required the owner to be a type. Match CPython: a missing owner binds to `obj`, while a non-type owner still raises "needs a type, not ...". Assisted-by: Claude Code:claude-opus-4-8 |
||
|
이광효
|
16bb018d31 |
csv: port CPython reader state machine (#8381)
* csv: port CPython reader state machine Replace the csv-core reader and per-item quote scanner with one Rust implementation of CPython's nine-state reader parser. Keep parser state across iterator items and distinguish virtual item boundaries from true iterator exhaustion. Centralize field completion so quote provenance, empty-field None conversion, float conversion, strict parsing, field limits, blank rows, and escaped or quoted newlines share one path. Apply the existing reentrant-iterator generation check to every iterator item consumed by a record. Also accept empty ASCII line terminators like CPython, report a missing csv-core writer sentinel as _csv.Error instead of panicking, and name the FSM entry point process_parser_input. Remove the expected-failure markers from the eleven reader tests that now pass. Assisted-by: Codex:gpt-5.6-sol * csv: test CRLF boundaries with custom lineterminator |
||
|
Chanho Lee
|
c9e48baebe |
Fix sqlite autocommit cursor transactions (#8419)
Assisted-by: Codex:gpt-5.6-sol |
||
|
Jeaeun Kim
|
249ae08911 |
fix(ssl): emit missing deprecation warnings (#8418)
Assisted-by: Codex:gpt-5.6-sol |
||
|
Jeaeun Kim
|
05f7f74248 |
fix(ssl): make Certificate immutable and non-instantiable (#8417)
Assisted-by: Codex:gpt-5.6-sol |
||
|
Jeong, YunWon
|
f08933be87 |
Split InterpreterFrame from FrameObject for stack-allocated execution (#8354)
* Add LightFrame for unobserved Python-to-Python calls
Allocate a lightweight LightFrame header on the DataStack instead of a
full Frame PyObject for specialized exact-args call paths. The frame is
materialized lazily only when observed (sys._getframe, traceback, tracing).
Key changes:
- LightFrame struct with borrowed pointers and no refcount overhead
- FrameSource enum (Heavy/Light) replacing the object field on ExecutingFrame
- CURRENT_LIGHT_FRAME TLS for the light frame chain
- Interleaved heavy/light frame walking in frame_at_offset_vm and
current_thread_frame_vm
- invoke_light_slots on PyFunction for the fast call path
Call overhead reduced from ~92ns to ~58ns (~37% reduction).
fib(28) improved from ~152ms to ~111ms (~27% reduction).
Assisted-by: Claude
* Streamline with_frame: inline recursion check, skip tracing dispatch
- Inline check_recursive_call instead of wrapping in with_recursion closure
- Amortize C stack overflow check to every 64th call depth
- Skip dispatch_traced_frame when use_tracing is false (hot path)
- Add recursion_depth_increment/decrement helpers
Assisted-by: Claude
* Address review findings for LightFrame correctness
- Fix materialize_light_frame previous pointer: use payload pointer
(&**frame) instead of Py<Frame> pointer, matching the chain convention
- Fix predecessor ref management: use retained_back instead of
mem::forget to keep materialized predecessors alive
- Fix materialized ref reclaim: use FrameRef::from_raw to drop the
leaked keeper ref without double-incrementing
- Use PyAtomic<u32> for LightFrame.lasti instead of AtomicU32, removing
the unsound cast on non-threading targets
- Add amortized C stack overflow check to light frame path
- Add scopeguard to light frame execution for panic safety
- Remove unnecessary Send/Sync impls from LightFrame
- Make check_c_stack_overflow pub(crate) for light frame path access
Assisted-by: Claude
* Fix CI failures: cspell, stale materialized state, f_back chain gaps
- Fix cspell: 'amortised' → 'amortized'
- Sync lasti, prev_line, and fastlocals when re-observing a materialized
light frame, fixing stale f_locals/f_lineno in test_inspect and others
- In with_frame, materialize any active light frame as the heavy frame's
predecessor and store it in retained_back, so f_back and inspect.stack
see the correct interleaved order (H_new → L_mat → H_old)
- Guard invoke_light_slots on NEWLOCALS | OPTIMIZED code flags
Assisted-by: Claude
* Unify frame chain: replace dual-chain with single FrameChainPtr
Replace the dual-chain frame management (CURRENT_FRAME AtomicPtr<Frame>
+ CURRENT_LIGHT_FRAME Cell<*const LightFrame>) with a single unified
chain using FrameChainPtr, a tagged pointer where bit 0 distinguishes
heavy (*const Frame) from light (*const LightFrame) entries.
Changes:
- Add FrameChainPtr type with tagged pointer encoding
- Change LightFrame: replace previous_light + saved_current_frame with
single previous: FrameChainPtr field
- Change InterpreterFrame.previous from AtomicPtr<Frame> to PyAtomic<usize>
- Change CURRENT_FRAME TLS from AtomicPtr<Frame> to AtomicUsize
- Remove CURRENT_LIGHT_FRAME TLS entirely
- Simplify all chain walkers (frame_at_offset, frame_at_offset_vm,
find_owned_chain_frame, for_each_current_frame, current_thread_frame)
- Remove light-frame materialization block from with_frame
- Update f_back to dispatch on FrameChainPtr tags
- Update faulthandler and gc_state to walk unified chain
- Adaptive C stack check: every call in debug, every 16th in release
Assisted-by: Claude
* Fix light frame materialization: current_frame, f_back, and GC tracking
- current_frame() now uses current_thread_frame_vm() to materialize
light frames, fixing super()/compile()/exec() when called from light
frame context
- materialize_light_frame stores retained_back for heavy predecessors
too, so f_back resolves after the caller returns
- Escaped materialized frames (traceback, sys._getframe) are GC-tracked
at cleanup so reference cycles are collectible
- Restore pre-existing @expectedFailure markers for test_frame proxy tests
Assisted-by: Claude
* Limit light frame materialization to super() only
current_frame() returns heavy-only again to avoid unnecessary
materialization that extends object lifetimes. super() init uses
current_thread_frame_vm() directly since it must inspect the
light frame's code and freevars to find __class__.
Assisted-by: Claude
* Fix remaining CI failures: C stack check, GC tracking threshold, expectedFailure
- Check C stack overflow every call in light frame path (not every
16th) to prevent stack overflow on CI workers with smaller stacks
- Fix GC tracking threshold: escaped materialized frame detection
uses strong_count > 1 (was > 2, missed frames referenced only by
traceback in the result)
- current_frame() materializes light frames so warnings, compile(),
and other callers see the correct frame
- Mark test_futures2 PyFutureTests.test_task_exc_handler_correct_context
as expectedFailure (pre-existing on main: PyTask __del__ timing)
Assisted-by: Claude
* Avoid materialization in hot paths: add light-frame-aware accessors
current_frame() stays heavy-only to avoid costly materialization on
every call. Instead, add current_globals(), current_code(), and
current_builtins() that read directly from light frame raw pointers
without creating frame PyObjects.
Callers migrated:
- PyFunction:🆕 current_builtins() for fallback builtins lookup
- compile()/eval(): current_code() for future feature flags
- type.__new__: current_globals() for __module__ detection
- typevar/typing: current_globals() for caller module name
- _io: current_code() for source path check
- super(): current_thread_frame_vm() for __class__ cell access
- warnings.warn: current_thread_frame_vm() for stack level walk
- import: current_globals() for module resolution
Assisted-by: Claude
* Fix current_locals() to include light frames
current_locals() must materialize light frames so locals() builtin
returns the correct scope. This fixes test_zipimport doctest and
other tests that rely on locals() in light frame context.
Assisted-by: Claude
* Fix clippy, profile tracing, resume_gen_frame, and faulthandler issues
- Fix clippy: collapsible_if in materialize_light_frame, map_unwrap_or in import
- Fix profile breakage: use current_thread_frame_vm in trace_event_inner to
materialize light frames, preventing frame identity mismatch when profiling
is enabled inside a light frame (e.g. profile.Profile.runctx)
- Disable light frames inside trace/profile callbacks (tracing_is_suppressed)
- Fix resume_gen_frame: skip light frames in previous chain to avoid dangling
pointers after generator suspend
- Fix faulthandler: emit '<no Python frame>' when chain has only light frames
Assisted-by: Claude
* Fix materialized frame refcount: transfer localsplus ownership on exit
Remove @expectedFailure from test_futures2: the underlying issue was that
materialized light frames cloned localsplus values (refcount +1 each),
keeping objects alive until the next GC cycle collection.
Fix: after run_light_frame returns, transfer ownership of localsplus
values from the light frame to the materialized frame, replacing the
clones. This eliminates the extra refcount, allowing objects to be
GC'd immediately when their last reference is dropped.
Also stabilize materialized frame's localsplus on the heap in
sync_materialized_on_exit before GC tracking — the data-stack backing
will be popped imminently.
Assisted-by: Claude
* Fix rustfmt, increase stack margin, address review comments
- Fix rustfmt formatting issues (lint CI failure)
- Increase STACK_MARGIN_BYTES from 2048 to 3072 words to prevent C stack
overflow in deep recursion tests on macOS CI (test_functools SIGSEGV)
- Skip dispatch_traced_frame when use_tracing is false (review comment)
- Fix post-fork top_frame initialization: walk past light frames to find
nearest heavy frame (review comment)
Assisted-by: Claude
* Remove redundant test override in test_futures2
The override only called super() without any marker, making it a
no-op that prek flags as a redundant test patch.
Assisted-by: Claude
* Fix test_pdb and test_functools CI failures
- Revert with_frame dispatch_traced_frame skip: tracing can be enabled
mid-execution (pdb.set_trace), so Return events must always be checked.
This fixes 3 test_pdb doctest failures across all platforms.
- Check C stack overflow on every with_frame call instead of every 64th:
the previous sampling approach missed overflows between checks when
native stack frames are large (invoke_light_slots + lru_cache recursion).
- Add #[inline(never)] to invoke_light_slots to prevent the optimizer
from merging its stack frame into callers.
- Add early C stack check before DataStack allocation in invoke_light_slots.
Assisted-by: Claude
* Rename Frame to FrameObject, FrameRef to FrameObjectRef
Prepare for _PyInterpreterFrame/PyFrameObject separation by
renaming the PyObject-backed frame type to FrameObject.
Python-visible name ("frame") is unchanged.
Assisted-by: Claude
* Remove Deref<Target = InterpreterFrame> from FrameObject
Replace implicit field access through Deref with explicit
iframe()/iframe_ref()/iframe_mut() calls. This decouples
FrameObject from InterpreterFrame field layout, enabling
independent modification of InterpreterFrame in future commits.
Assisted-by: Claude
* Move InterpreterFrame identity fields to raw pointers
InterpreterFrame's code/globals/builtins/func_obj fields are now
borrowed raw pointers instead of owned PyRef/PyObjectRef. Ownership
of these references is anchored in FrameObject's new owned_code,
owned_globals, owned_builtins, owned_func_obj fields.
This aligns InterpreterFrame's layout with LightFrame's existing
raw-pointer pattern, enabling future DataStack-based allocation
without PyObject heap allocation.
- Add code()/globals()/builtins()/func_obj() accessor methods
- Add FrameObject::new_ref() for combined create+allocate+patch
- Update ExecutingFrame to use &Py<PyCode> / &Py<PyDict> / &PyObject
- Update monitoring.rs functions to take &Py<PyCode> instead of &PyRef<PyCode>
- Add Send+Sync impls for InterpreterFrame (raw pointers are !Send by default)
Assisted-by: Claude
* Remove LightFrame, FrameChainPtr, and dual-frame-type infrastructure
Delete LightFrame struct, FrameChainPtr tagged pointer, FrameSource enum,
and all supporting functions: materialize_light_frame, sync_light_to_materialized,
transfer_localsplus_to_materialized, sync_materialized_on_exit,
run_light_frame, invoke_light_slots body, LocalsPlus::from_datastack_raw.
All call paths now use a single frame type (FrameObject wrapping
InterpreterFrame). The frame chain is a simple *const FrameObject linked
list through InterpreterFrame.previous.
invoke_light_slots delegates to invoke_exact_args_slots.
ExecutingFrame.frame_source replaced with ExecutingFrame.frame.
Frame chain walking simplified throughout (no more light/heavy dispatch).
Net: -724 lines of dual-frame-type complexity.
Assisted-by: Claude
* Clean up invoke_light_slots and _vm function variants
- Replace invoke_light_slots calls with invoke_exact_args_slots
- Delete the invoke_light_slots delegate method
- Replace current_thread_frame_vm/frame_at_offset_vm with their
non-_vm equivalents (no more light frame materialization needed)
- Remove stale light-frame comment in callable.rs
Assisted-by: Claude
* Fix CI lint issues
- Move github context expressions from run blocks to env blocks
(zizmor/template-injection)
- Add workflow-level permissions: {} to lib-deps-check.yaml
- Suppress zizmor excessive-permissions for lib-deps-check.yaml
(pull_request_target is required for PR comments)
- Apply rustfmt formatting
Assisted-by: Claude
* Stack-allocate InterpreterFrame for non-generator function calls
- Extract InterpreterFrame::new() from FrameObject::new()
- Change frame chain from *const FrameObject to *const InterpreterFrame
- Add InterpreterFrame.materialized field for lazy FrameObject creation
- Add vm.with_iframe()/run_frame_fast() for stack-allocated frame execution
- Move ExecutingFrame.frame to iframe: *const InterpreterFrame
- Implement lazy materialize() for on-demand FrameObject creation
- Update invoke_with_locals/invoke_prepared_exact_args to skip heap allocation
- Update frame chain walking in faulthandler, gc_state, builtins/frame
- super() reads InterpreterFrame directly without materialization
Assisted-by: Claude
* Remove unnecessary refcount and atomic ops from fast call path
- Use raw pointer for func_obj instead of cloning PyObjectRef (saves 2 atomic RMW)
- Skip owner AcqRel swap for stack-allocated frames (always Thread-owned)
Assisted-by: Claude
* Change trace field to Option<PyObjectRef>, remove vm from InterpreterFrame::new
- trace: PyMutex<PyObjectRef> -> PyMutex<Option<PyObjectRef>>
- Eliminates vm.ctx.none() refcount inc/dec per frame init
- Remove vm parameter from InterpreterFrame::new() (no longer needed)
- f_trace getter returns None when trace is unset
- f_trace setter stores None instead of the None singleton
Assisted-by: Claude
* Store current frame pointer on VirtualMachine for fast lookup
- Add current_frame_ptr Cell<usize> field to VirtualMachine
- Hot path (with_iframe) reads/writes vm.current_frame_ptr instead of TLS
- TLS CURRENT_FRAME still maintained for signal-safe traceback walking
- Use set_current_frame_nosave() to skip cross-thread top_frame update
- super(), frame_at_offset, current_thread_frame_materialize use vm Cell
Assisted-by: Claude
* Apply rustfmt
Assisted-by: Claude
* Fix clippy use_self warnings, add inline hints for hot path
Assisted-by: Claude
* Remove code refcount clone from fast call path
Defer code.to_owned() to the generator/coroutine path only. On the fast
path, use &Py<PyCode> directly — the code object is alive via the
PyFunction on the caller's stack.
Assisted-by: Claude
* Remove VM current_frame_ptr Cell (fix SIGSEGV in threading builds)
The Cell<usize> on VirtualMachine could desynchronize from the TLS
CURRENT_FRAME in nested VM scenarios (enter_vm called with different
VMs on the same thread). Revert to TLS-only frame chain management.
Keep set_current_frame_nosave() for the stack-frame fast path: it
skips cross-thread top_frame publication but still does the TLS swap.
Fix clippy not_unsafe_ptr_arg_deref in set_current_frame.
Fix unnecessary Result wrapper in make_generator_or_coro.
Assisted-by: Claude
* Fix CI failures: GC frame assertion, faulthandler Radium import, dead code warning
- Fix SIGSEGV in threading builds: GC debug assertion walked frame
chain via top_frame, which is not updated by set_current_frame_nosave
used by stack-allocated iframes. Walk CURRENT_FRAME TLS chain instead.
- Add InterpreterFrame::get_lasti() accessor, remove direct Radium
import from faulthandler.rs (fixes unused import warning).
- Add cfg_attr allow(dead_code) for from_payload_ptr without threading
(fixes WASM build warning).
Assisted-by: Claude
* Fix materialized frame: copy localsplus, clear stale previous pointers
- materialize_slow: copy fastlocals snapshot to heap instead of empty
localsplus (fixes index-out-of-bounds panic in locals()/f_locals)
- materialize_slow: set previous to null (stack iframes become dangling)
- with_frame/resume_gen_frame: clear iframe.previous on pop (prevents
dangling pointers to freed stack iframes)
- f_back: walk TLS CURRENT_FRAME chain instead of top_frame for
cross-thread lookup (top_frame is stale for stack-allocated iframes)
- check_locals_access: match materialized frames by comparing the
Py<FrameObject> address stored in the stack iframe's materialized field
Assisted-by: Claude
* Fix Windows clippy: gate unix-only imports and variables
FrameObject, Py imports and top_iframe variable are only used in
unix+threading cfg blocks. Gate them properly for non-unix builds.
Assisted-by: Claude
* Fall back to FrameObject when tracing; fix cross-thread frame access
- invoke_with_locals: fall back to heap-allocated FrameObject path
when use_tracing is active, so trace/profile callbacks fire correctly.
This fixes test_trace, test_bdb, test_monitoring regressions.
- set_current_frame: publish top_iframe alongside top_frame in ThreadSlot
so cross-thread readers (sys._current_frames) can materialize
stack-allocated frames from other threads under stop-the-world.
- get_all_current_frames: use TLS CURRENT_FRAME for current thread,
fall back to top_iframe materialization for other threads.
- Use set_current_frame (not nosave) in with_iframe for proper
cross-thread visibility.
Assisted-by: Claude
* Apply rustfmt
Assisted-by: Claude
* Fix Windows: gate unix-only top_iframe_ptr variable
Assisted-by: Claude
* Fix frame chain: f_back, retained_back, owner, faulthandler
- Set materialized frame owner to FrameObject (not Thread) so
frame.clear() works on traceback frames
- Fix f_back for materialized frames: walk TLS chain to find source
iframe and materialize its previous frame
- Fix faulthandler dump_traceback to use dump_live_frames (reads
live lasti from stack iframes instead of stale materialized copies)
- Add retained_back propagation in with_frame cleanup: when a frame
escapes, materialize caller and mark it escaped for chain propagation
- Fix release_datastack_frame: don't overwrite retained_back if
already set by with_frame cleanup
Assisted-by: Claude
* Apply rustfmt
Assisted-by: Claude
* Refine with_frame retained_back: use frame_obj instead of materialize
Avoid materializing caller frames in with_frame cleanup to prevent
creating localsplus snapshots that keep extra refcounts. Only use
existing FrameObjects (frame_obj) for retained_back. Also keep
previous pointer alive in release_datastack_frame since the caller
is still executing at that point.
Assisted-by: Claude
* Fix f_lineno: use Cell<u32> for prev_line, update on every instruction
prev_line was only updated when tracing was active, causing f_lineno
to return stale values when observed mid-CALL (e.g. from warnings.warn
or sys._getframe). Changed prev_line from u32 to Cell<u32> for
interior mutability so it can be read safely while ExecutingFrame
holds a shared reference, and update it on every instruction so
f_lineno always returns the correct line.
Assisted-by: Claude
* Fix f_lineno for materialized frames: read live prev_line from TLS chain
Materialized FrameObjects have a snapshot of prev_line from materialize
time. When f_lineno is called on a materialized frame that is still
executing (its source iframe is on the TLS chain), walk the TLS chain
to find the source iframe and read its live prev_line. This fixes
lineno tracking in warnings, gettext, and other modules that read
frame line numbers during function calls.
Also fixed the pointer comparison: use from_payload_ptr to convert
FrameObject payload address to Py<FrameObject> address for matching
against the materialized field.
Assisted-by: Claude
* Sync live frame state for materialized frames
- Add find_live_source_iframe() to walk TLS chain and find the live
source InterpreterFrame for a materialized FrameObject
- f_lasti: read live lasti from source iframe when available
- f_lineno: use read_volatile for live prev_line from source iframe
- sync_visible_locals_to_mapping: read live localsplus from source iframe
- framelocalsproxy_getval: read live localsplus from source iframe
- with_iframe cleanup: sync localsplus, prev_line, lasti to materialized
FrameObject when the frame returns
- sync_fastlocals_from: new LocalsPlus method to update fastlocals
Fixes test_inspect (stale f_locals, stale lineno in positions),
test_listcomps (stale iteration variable in locals()).
Assisted-by: Claude
* Fix retained_back: use materialize_chain and strong_count check
- Add materialize_chain() that creates a lightweight FrameObject with
empty localsplus for f_back chain building only. This avoids cloning
local variables which would create extra refcounts (fixes test_memoryview).
- with_frame cleanup: use strong_count > 1 instead of escaped flag to
determine if retained_back is needed, ensuring chain propagation.
- with_iframe cleanup: use materialize_chain for retained_back to avoid
extra refcounts from localsplus snapshots.
Fixes test_memoryview (refcount from retained_back chain) and
test_traceback.test_extract_stack (f_back chain depth).
Assisted-by: Claude
* Fix cross-thread f_back chain for sys._current_frames/exceptions
- get_all_current_frames: materialize entire frame chain and link
retained_back during stop-the-world for other threads
- f_back: use stop-the-world to safely materialize cross-thread frame
chains on unix when prev iframe is on another thread
- f_back: check frame_obj() before STW for fast path
Fixes test_sys.test_current_frames and test_sys.test_current_exceptions.
Assisted-by: Claude
* Fix monitoring LINE events and faulthandler thread dump
- Skip all instrumented opcodes in bytecode loop prev_line update to
avoid defeating InstrumentedLine de-duplication
- Add prev_line update in execute_instrumented for non-RESUME/non-LINE
instrumented opcodes
- update_events_mask: walk TLS iframe chain to re-instrument all frames
including stack-allocated ones (fixes missing LINE events for inline
frames already past RESUME)
- faulthandler: use top_iframe fallback for stack-allocated frames in
dump_all_threads and watchdog_thread
- faulthandler: use dump_live_frames for current thread dump
Fixes test_monitoring LINE/CALL tests, test_faulthandler thread dumps.
Assisted-by: Claude
* Fix clippy: remove duplicate Radium import, use previous() helper
Assisted-by: Claude
* Fix materialized frame issues: GC tracking, live locals, f_trace propagation
- Track materialized FrameObjects in GC so cycle collection works
(fixes __del__ not called at exit for exception traceback cycles)
- materialize_chain returns owned PyRef without temporary_refs to
avoid defeating GC cycle detection from non-GC-tracked storage
- framelocalsproxy_setval writes to live source iframe when available
(fixes f_locals proxy writes not affecting executing frame)
- f_lineno uses lasti-based line for non-executing frames instead of
prev_line (fixes PEP 626 wrong line after exception unwind)
- f_trace/f_trace_lines/f_trace_opcodes setters propagate to live
source iframe (fixes pdb set_trace not working through materialized
frames)
- frame.clear() checks find_live_source_iframe to reject clearing a
frame that is backed by a live stack-allocated iframe
- Fix clippy warnings and unused imports in faulthandler
- Remove unsafe top_iframe access from faulthandler watchdog thread
Assisted-by: Claude
* Fix faulthandler crashes, GC tracking timing, and test_generators
- Use top_iframe instead of top_frame for frame chain walking in
faulthandler (dump_all_threads and watchdog), avoiding stale or
GC-cleared FrameObject pointer dereference
- Defer GC tracking of materialized FrameObjects to with_iframe
cleanup (after set_current_frame restores old chain), preventing
premature collection while temporary_refs still holds the only ref
- Add FrameObject::try_iframe() for safe access to potentially
GC-cleared frames
- Remove expectedFailure for test_exhausted_generator_frame_cycle
(now passes with GC tracking fix)
- Add 'noalias' to cspell dictionary
Assisted-by: Claude
* Fix Windows frame dump and remove dead code
- Publish top_iframe on all platforms (not just unix) so Windows
faulthandler and sys._current_frames can see stack-allocated frames
- Use top_iframe instead of frames Mutex in Windows faulthandler
dump and watchdog paths
- Use top_iframe in non-unix get_all_current_frames for sys._current_frames
- Remove unused dump_traceback_thread_chain (replaced by top_iframe walk)
- Mark test_pdb_await_support as expected failure
Assisted-by: Claude
* Fix ThreadSlot init on all platforms, skip test_pdb_await_support
- Add top_iframe to ThreadSlot initializer in init_thread_slot_if_needed
(fixes wasm32, Windows compile errors)
- Skip test_pdb_await_support on RustPython (async pdb exception
callback receives None exc argument)
Assisted-by: Claude
* Remove dead faulthandler code (dump_frame_from_ref, dump_traceback_thread_frames, try_iframe)
These functions are no longer called after switching all frame chain
walking to use top_iframe directly.
Assisted-by: Claude
* Fix Windows sys._current_frames: materialize full frame chain
The non-unix get_all_current_frames path was only materializing the
top iframe without linking retained_back, so f_back chain walking
could not find deeper frames like f123() in test_sys tests.
Replicate the full chain materialization from the unix path.
Assisted-by: Claude
* Use STW-based cross-thread f_back on all platforms
Replace the non-unix frames-mutex fallback with the same
stop-the-world iframe chain materialization used on unix.
Fixes test_current_exceptions on Windows where stack-allocated
iframes were not found in the FrameObject-only frames list.
Assisted-by: Claude
* Fix set_f_lineno to write to live iframe, use STW for cross-thread access
- set_f_lineno: write lasti/pending_stack_pops to the live source
iframe instead of the materialized copy so debugger jumps take
effect on stack-allocated frames
- f_back cross-thread: enter STW before dereferencing the prev
pointer to prevent use-after-free if the owning thread returns
- get_all_current_frames (non-unix): add STW protection when
walking cross-thread iframe chains, matching the unix path
Assisted-by: Claude
* Guard current_location() against lasti==0, always capture retained_back
- current_location(): return first_line_number instead of
panicking when lasti is 0 (before first instruction executes)
- with_frame retained_back: materialize the caller iframe when
needed so f_back always resolves for escaped FrameObjects
Assisted-by: Claude
* Fix format, revert eager retained_back materialization
Remove double blank line in frame.rs (lint failure).
Revert with_frame retained_back to only use already-materialized
caller FrameObjects — eagerly materializing the caller added
refcounts on local variables, preventing timely deallocation and
causing test_io, test_memoryview, and test_futures failures.
Assisted-by: Claude
* Remove rustpython-unicode-isolation-issue.md from gitignore
Assisted-by: Claude
* Remove test_pdb_await_support skip: async pdb now works correctly
Assisted-by: Claude
* Fix STOP_ITERATION monitoring: wrap value in StopIteration instance
fire_stop_iteration was passing the raw iterator return value to
callbacks, but the STOP_ITERATION event callback signature expects
a StopIteration exception instance. Wrap non-StopIteration values
in a new StopIteration(value), matching PyMonitoring_FireStopIterationEvent.
This fixes test_pdb_await_support where bdb's exception_callback
received None instead of a StopIteration instance.
Assisted-by: Claude
|
||
|
이광효
|
55b3d681ef |
ast: match CPython argument validation order (#8414)
Validate parameter annotations while converting ast.arguments, before merging positional and keyword-only defaults. This gives invalid annotation contexts precedence over default-list shape errors, as in CPython. Merge positional defaults before keyword-only defaults so an excess positional-default error also takes precedence over a keyword-only default length mismatch. Remove the expected-failure markers from the FunctionDef and Lambda AST validator tests that now pass. Assisted-by: Codex:gpt-5.6-sol |
||
|
Dino
|
ede18e44cb |
dict: match CPython errors for invalid update sequence elements (#8338)
* fix dict update type error * Use tuple for dictionary update sequence pairs * Propagate errors while adding dict update sequence notes * Match BaseException.add_note error message with CPython |
||
|
조진명
|
6906aa1747 |
csv: support multi-character lineterminator in writer (#8328)
* csv: support multi-character lineterminator in writer Store the dialect line terminator as an owned String instead of the single-byte csv_core::Terminator, dropping PyDialect's Copy derive. The manual writer paths emit the full terminator; the csv-core-backed QUOTE_ALL/QUOTE_NONNUMERIC paths emit a sentinel byte (preserving csv-core's quote/empty-record bookkeeping) and append the real terminator. field_needs_quotes/escape now quote a field containing any terminator byte. The reader ignores lineterminator and always uses CRLF, matching CPython and avoiding mid-UTF-8 record splits. * csv: unmark now-passing test_write_lineterminator * csv: address review feedback - Remove the redundant per-branch sentinel terminator setup in to_writer; the unconditional terminator call after the match is the single source. - Reword the empty-lineterminator errors to "must not be empty" (the constraint is non-empty, not single-character) on both entry points. - Promote the sentinel invariant check in writerow from debug_assert_eq! to assert_eq! so it also guards release builds. - Add a snippet case for a field containing a line-break byte to ensure the csv-core path drops only the trailing terminator. * csv: reject non-ASCII lineterminator The writer decides what to quote and escape by comparing raw bytes, so a non-ASCII terminator quoted a field that merely shared a UTF-8 lead byte, and QUOTE_NONE escaped individual bytes of the terminator and then failed to decode the record back to a string. Reject non-ASCII terminators, including lone surrogates, as csv.Error when the dialect is parsed, and leave code-point-wise handling to a follow-up (#8310). The non-string error message now matches CPython as well. |
||
|
Seonghun An
|
bf4a2b110c |
Fix zero padding for string format specs (#8407)
* Fix zero padding for string format specs Assisted-by: Codex:gpt-5 * Add string alignment format regression test Assisted-by: Codex:gpt-5 |
||
|
Jeong, YunWon
|
1613741f86 |
Fix SymbolUsge::Iter, os.rename, sre_engine (#8390)
* Fix SymbolUsge::Iter * fix os.rename * fix sre * Re-export rename from crate::posix on Windows posix_windows.rs is mounted as the `posix` module on Windows, so `crate::posix_windows` does not resolve. Assisted-by: Claude * Apply rustfmt to sre_engine tests Assisted-by: Claude * Update test expectations for the sre and tokenize fixes - test_re: drop expectedFailure from test_word_boundaries, test_possessive_quantifiers and test_bug_gh101955 - test_inspect, test_pydoc: drop expectedFailure from the signature tests that now pass - test_pdb: drop expectedFailure from the two file-modification tests and +EXPECTED_FAILURE from the test_post_mortem_chained and test_pdb_asynctask doctests - test_unittest: mark test_autospec_on_bound_builtin_function as expectedFailure; inspect.signature() now succeeds on time.ctime because of its auto-generated __text_signature__ Assisted-by: Claude |
||
|
Kwak Byoung Min
|
59e903d807 |
Reject empty UU input before decoding (#8389)
Constraint: Match CPython's a2b_uu empty-input behavior. Confidence: high Scope-risk: narrow Directive: Preserve decoding behavior for non-empty UU buffers. Tested: prek run --all-files; test_binascii; cargo fmt --check; cargo clippy -p rustpython-stdlib -- -D warnings; cargo test -p rustpython-stdlib; manual full-suite verification Assisted-by: Codex:gpt-5.6-sol |
||
|
Chanho Lee
|
d62a3916f5 |
Fix sqlite autocommit lifecycle (#8387)
* fix: open disabled sqlite autocommit transactions Assisted-by: Codex:gpt-5.6-sol * fix: reopen disabled sqlite transactions after commit Assisted-by: Codex:gpt-5.6-sol * fix: reopen disabled sqlite transactions after rollback Assisted-by: Codex:gpt-5.6-sol * fix: roll back disabled sqlite connections on close Assisted-by: Codex:gpt-5.6-sol |