mirror of
https://github.com/RustPython/RustPython.git
synced 2026-10-28 10:44:36 +08:00
copilot/fix-static-build-core-dump
448
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Luan Taraschi
|
30be390943 |
Reject the string format flags CPython refuses (#8571)
A sign, a space or an alternate form parsed fine on a string spec and
then had no effect:
>>> format('ab', '+')
'ab'
>>> f"{'ab':#5}"
'ab '
CPython raises ValueError for all three. format_string already refused z
and an explicit '=' alignment with the same family of messages, so the
three checks go next to those, in the order CPython reports them: sign,
then z, then the alternate form, then the alignment.
Assisted-by: Claude Code:claude-opus-5
|
||
|
Luan Taraschi
|
f9e65c0995 |
Drop what a deque with maxlen of zero is handed (#8567)
append and appendleft trimmed before pushing, and the test they used,
maxlen == len, holds for an empty deque whose bound is zero. The pop then
had nothing to remove and the item stayed:
>>> d = deque(maxlen=0)
>>> d.append(1)
>>> list(d)
[1]
Both now push first and trim after, which is the order CPython uses, so a
bound of zero drops what just arrived. extend, extendleft, insert, rotate,
the operators and the constructor were already right.
Assisted-by: Claude Code:claude-opus-5
|
||
|
Luan Taraschi
|
4c448c4704 |
Apply the format spec to a bool instead of dropping it (#8566)
format_bool answered "True" or "False" for every spec that carries no
presentation type, so width, fill, alignment and sign were all discarded:
>>> f"{True:>5}"
'True'
CPython has no bool.__format__ of its own. It uses int's, where an empty
spec on a subclass gives str(self) and everything else formats the integer.
The empty spec keeps the spelled out answer, the rest now goes to
format_int, which also brings back the errors an integer spec raises.
Assisted-by: Claude Code:claude-opus-5
|
||
|
Luan Taraschi
|
f8de240feb |
Insert the item rather than its key in bisect.insort (#8565)
insort_left and insort_right rebound `x` to `key(x)` and then handed that
same value to both the search and `a.insert`, so the object the caller
passed in never reached the list:
>>> words = ["a", "ccc"]
>>> bisect.insort(words, "bb", key=len)
>>> words
['a', 2, 'ccc']
The key now feeds the search only, and the insert keeps the original.
Assisted-by: Claude Code:claude-opus-5
|
||
|
Luan Taraschi
|
3d2ee6466d |
Drop the narrow-build surrogate pairing from unicode_escape (#8569)
unicodeescape_string is a port of CPython 2.7's
PyUnicode_EncodeUnicodeEscape, where the branch mapping a character at or
above 0x10000 to \U00xxxxxx sits under #ifdef Py_UNICODE_WIDE and the
branch folding a surrogate pair sits under the #else. The port kept the
"# ifdef Py_UNICODE_WIDE" and "# endif" comments around the first one but
dropped the #else, so both branches run.
The pairing branch reads the next character without checking that there
is one, so a high surrogate at the end of the string walks off it:
>>> "\ud800".encode("unicode_escape")
IndexError: index out of range # CPython: b'\ud800'
And when there is a next character it folds the two, which no wide build
does:
>>> "𐀀".encode("unicode_escape")
b'\U00010000' # CPython: b'\ud800\udc00'
With the branch gone a high surrogate falls through to the 16-bit case
and comes out as \udXXX, which is where a low surrogate already went.
test_incremental_surrogatepass in UnicodeEscapeTest carried an
expectedFailure for the IndexError. It passes now, so the override goes
with it.
Assisted-by: Claude Code:claude-opus-5
|
||
|
Luan Taraschi
|
6f542bf285 |
Fix str.expandtabs aborting on a tab size of zero (#8562)
`"a\tb".expandtabs(0)` panicked with a capacity overflow. CPython returns `'ab'`: with no width to advance to, the tabs come out and nothing else moves. `expandtabs(-1)` is the same call, since `ExpandTabsArgs::tabsize` sends every negative value to 0. `expandtabs` keeps the tab stop in `tab_size` and the current column in `col_count`, and on a tab it does `tab_size - col_count`. With a tab size of zero both start at 0, the first character makes `col_count` 1 while `tab_size` stays 0, and the subtraction underflows. The run of spaces asked for next is `usize::MAX`, and the allocation aborts the process. A tab has to follow something on the line to reach it: `"\ta".expandtabs(0)` subtracts 0 from 0 and comes out right by accident. `BytesInner::expandtabs` already returns early for this and filters the tabs out. The string version now does the same. Assisted-by: Claude Code:claude-opus-5 |
||
|
hyoinandout
|
728a61e982 |
csv: validate dialect options (#8402)
* csv: validate dialect options Resolve each dialect once and validate the merged options before constructing readers and writers. Handle Unicode character parsing consistently and enable the corresponding CPython CSV tests. Assisted-by: Tau:gpt-5.6-luna * cargo fmt * Validate csv dialect conflicts with line terminators - Reject dialect characters that overlap any lineterminator character - Validate Dialect construction consistently and allow non-ASCII terminators - Unskip csv invalid character coverage now that validation matches behavior * Remove stale CSV rejection snippet Removes a RustPython-only assertion that contradicts the CPython-compatible dialect validation added by this branch. Scope: test-only AI-Assisted-By: Codex * Revert "Remove stale CSV rejection snippet" This reverts commit |
||
|
Yubin KimandJeong, YunWon
|
c420cc564c |
Rebase range iterator __reduce__ to match CPython (#8424)
* Rebase range iterator __reduce__ to match CPython range_iterator.__reduce__ (and longrange_iterator) returned the original range plus the current index as pickle state; CPython returns the range rebased to the current position with a None state. Rebase start by index * step (clamped to the length) and emit None. __setstate__ is kept so pickles carrying an integer state still load. Assisted-by: Claude Code:claude-opus-4-8 * Add snippet test for range iterator __reduce__ Assert the rebased range and None state added in the previous commit. The snippet runner executes it under both CPython and RustPython, so it also cross-checks the representation against CPython. Assisted-by: Claude Code:claude-opus-4-8 * Update extra_tests/snippets/builtin_range.py --------- Co-authored-by: Jeong, YunWon <69878+youknowone@users.noreply.github.com> |
||
|
Luan Taraschi
|
6079ad06e5 |
Fix re.findall reporting a group that did not match (#8563)
findall returns the matched text rather than a match object, so a group that
took no part in the match is reported as an empty value. With one group it
was reported as `None`:
>>> re.findall(r"(a)?b", "b ab")
[None, 'a'] # CPython: ['', 'a']
The branch for two or more groups was already right, since it passes `""` to
`Match.groups` as the default, so the two halves of the same function
disagreed with each other.
That default is built as a `str` whatever the pattern is, so a `bytes`
pattern came back with `str` mixed into it:
>>> re.findall(rb"(a)|(b)", b"ab")
[(b'a', ''), ('', b'b')] # CPython: [(b'a', b''), (b'', b'b')]
The empty value is now built once from `isbytes` and both branches use it.
`Match.groups` still reports `None`, which is what CPython does.
Assisted-by: Claude Code:claude-opus-5
|
||
|
Luan Taraschi
|
dbf6e6d048 |
Fix str.replace with an empty pattern splitting characters (#8561)
Wtf8::replace looks for the pattern with a byte search. An empty pattern
matches at every byte, so the replacement was inserted between the bytes of
a multi-byte character and the result was no longer WTF-8:
>>> "á".replace("", "-")
'-ím' # CPython: '-á-'
from_bytes_unchecked then took that as valid without looking, so every later
read of the string returned something else. An empty pattern now walks code
points instead. A pattern that is not empty stays on the byte search, which
is safe because a WTF-8 sequence never starts inside another one.
Assisted-by: Claude Code:claude-opus-5
|
||
|
Andrej730
|
02a80ea3ad |
str.format: fix not raising for unknown conversions (#8560)
* `str.format`: fix not raising for unknown conversions * Move test to Python, test error msg too |
||
|
Jeong, YunWon
|
dd2cc4d77a |
Align compiler futures, annotations, and symtable with CPython (#8550)
* Align compiler futures and annotations with CPython Handle Barry parsing consistently across AST, type-comment, REPL, and WASM paths. Preserve deferred annotation source and scope metadata, and make _symtable conversion cached and linear-time. Assisted-by: Codex:gpt-5.4 * Fix f-string debug ranges and update Barry/annotate test expectations Compute the debug f-string source range from the raw leading and trailing text instead of the comment-stripped text, so a comment inside a multi-line replacement field no longer shifts the emitted LOAD_CONST location. Restrict the obsolete-operator location shift to `<>` by also requiring a `>` at the reported offset; without it any `ExpectedExpression` that followed a `<` (`2 <;`) was reported one column to the left. Add `CodeFlags::FUTURE_MASK` and use it in builtins, the shell, and the WASM VM instead of repeating the eight-flag union, and collapse the duplicated Barry diagnostic sequences into `BarrySource::diagnostic`. Update the `__annotate__` disassembly snapshot to LOAD_FAST_BORROW and replace the RustPython-only `barry_as_FLUFL` co_flags assertion in extra_tests with the CPython-matching one. Assisted-by: Claude Code:claude-opus-5 * Restore the annotated function in __annotate__ qualnames Signature annotation scopes are compiled in the scope enclosing the annotated function, so the function name has to be folded back into the qualname: `f.__annotate__`, `C.m.__annotate__`, `outer.<locals>.inner.__annotate__`. test_type_annotations test_annotate_qualname covers this. test_pyrepl test_future_barry_as_flufl now passes, so drop its expectedFailure marker. Assisted-by: Claude Code:claude-opus-5 * Pass safepoints in the attached-thread interpreter tests `main_and_subinterpreter_run_sections_overlap` parked its workers on a condvar while attached, and `busy_main_interpreter_does_not_block_subinterpreter` looped over protocol calls that never reach a safepoint. An attached thread that never reaches a safepoint cannot be suspended, so a concurrent process-wide stop-the-world (from the gc test running in parallel) never completes and the already-stopped sibling interpreter stays stopped; the second worker then waits in `wait_while_suspended` until the 30s deadline. Both loops now call `vm.check_signals()` each iteration, and the condvar wait uses `wait_timeout` so the lock is released between safepoints. Reproduced on Linux (4 CPUs): the two tests with `--test-threads=2` failed 2/40 before and 0/40 after; the full `rustpython-vm` lib binary failed 1/30 before and 0/30 after. Assisted-by: Claude Code:claude-opus-5 * Report the obsolete `<>` operator from one place `<>` was located twice: `BarrySource` recorded the token ranges it rewrote to `!=`, and `cpython_parse_diagnostic_override` separately peeked at the bytes around an `ExpectedExpression` location. Both produced "invalid syntax" spanning the whole operator. `prepare_barry_as_flufl_source` now records the operator ranges in the non-Barry case too -- by plain text search, since nothing is rewritten there -- so `BarrySource::invalid_legacy_operator` covers both, and the byte-peeking helper is gone. It also defers to an unclosed bracket earlier in the source, which the diagnostic chain it used to sit in did for it. A `<>` that starts a statement now spans both characters: the parser stops at the `<` rather than one character in, which the byte peek did not handle. Assisted-by: Claude Code:claude-opus-5 * Drop a py object's payload before its class `PyInner` declares `typ` before `payload`, so `drop_in_place` dropped the class first and `PyAtomicRef::drop` left it null. A `PyWeak` payload is still linked into the weakref list of the object it points at until its own `Drop` unlinks it under the stripe lock, which happens after that. `WeakRefList::add` walks the same list looking for a proxy to reuse and reads `node.class()` off every node it passes, so it could dereference the null class of a node on its way out. `PyInner::dealloc` now runs the two destructors in the other order at all three of its exit paths, with a compile-time check that no other field has one. Reproduced by the new test in about 0.1s, as the panic CI hit in the c-api `weakrefobject::tests::new_proxy` run: panicked at crates/vm/src/object/ext.rs:312: unsafe precondition(s) violated: hint::unreachable_unchecked must never be reached Assisted-by: Claude Code:claude-opus-5 |
||
|
Jeong, YunWon
|
86d9407b3d |
Raise where the fuzzer found aborts and stack overflows, and collect with the count in the object header (#8551)
* Correct what the non-unix frame stack is for
The comment said non-unix threading builds have no stop-the-world.
CollectStopTheWorld is gated on `feature = "threading"` alone, and
sys._current_frames stops the world on both paths; the field is the
fallback a reader uses when there is no `top_iframe` to materialize from.
Assisted-by: Claude
* Seek by the whole file position on Windows
`SetFilePointer` answers with the low half of the new position and signals
failure with INVALID_SET_FILE_POINTER, which is also that half of a
position four gigabytes in; telling them apart takes the error code, which
this did not read, so such a seek was reported as an error. Deciding
seekability from it also called such a file unseekable.
`SetFilePointerEx` returns the whole position and a success flag of its
own, which also removes the transmute of the position into halves.
Assisted-by: Claude
* memoryview: tell a value of the wrong kind from one that does not fit
Assigning an item reported every packing failure as a TypeError, so
m[0] = 300 on a 'B' view said the value was the wrong type rather than out
of range. Packing now says which of the two it was, and whether the value's
own code raised, in which case that error is the answer as it is:
m[0] = 300 ValueError: invalid value for format 'B'
m[0] = "x" TypeError: invalid type for format 'B'
m[0] = <__index__ that raises> the raised error
`struct` reports both as `struct.error` and is unchanged; the kind travels
beside the exception for the caller that tells them apart.
Also: None was read as a deletion, so m[0] = None answered "cannot delete
memory" instead of packing it; and deleting through the mapping protocol
never reached the read-only check, which comes first.
Assisted-by: Claude
* Raise MemoryError for allocations sized by Python input
os.read, _RawIOBase.read, int.to_bytes, struct.pack, ctypes array
creation and the _ssl RAND functions sized a Vec from a Python-supplied
length with vec![], which calls handle_alloc_error and, under
panic = "abort", ends the process. They now allocate through
vm.new_zeroed_bytes and raise MemoryError.
itertools.product built its pools without checking that
len(iterables) * repeat is representable; it now raises OverflowError
"repeat argument too large" and reserves the pool and index vectors
fallibly. repeat is read as isize, so a negative one raises ValueError
"repeat argument cannot be negative" instead of the conversion's message.
_ssl.RAND_bytes and RAND_pseudo_bytes read n as i32, matching the int
they are declared with.
Assisted-by: Claude
* Collect with the count in the object header
A collection kept the count it was working with in a table keyed by the
object's address, and the objects it had proved reachable in a second one.
Between them they were hashed once per candidate and twice per edge in the
heap, which is where most of a collection over a live heap went.
The count now lives in `PyInner::gc_refs`, with `GcBits::COLLECTING` saying
it is meaningful, and reachability is `gc_refs == GC_REACHABLE` rather than
membership in a set. Step 5 splits the candidates and clears the bit in one
pass. gcbench, five interleaved pairs, median: a live heap of 423k objects
goes from 0.101s to 0.055s and a dead one from 0.402s to 0.383s.
The bits, generation, owner and count take eight bytes between them. A
64-bit header had those eight as the padding its alignment forces, so it is
unchanged at 48 bytes; a 32-bit header grows from 24 to 28.
Assisted-by: Claude
* Take an explicit thread stack size as a floor in debug builds
A debug build already started Python threads on 8 MiB rather than Rust's
2 MiB default, but an explicit threading.stack_size(N) went through
verbatim. test_threading asks for 256 KiB, and starting a thread on that
walked off the end of the stack: the guard page fault landed in the
prologue of ExecutingFrame::run.
Unoptimized, that prologue reserves 80,848 bytes where the optimized one
reserves 656 -- execute_instruction is #[inline(always)] and LLVM only
colors stack slots from opt-level 1, so the frame is the sum of all 200
instruction arms' temporaries rather than the largest. A Python call costs
88,672 bytes of native stack there, and threading's bootstrap is six
frames deep, so 256 KiB holds less than half of what starting a thread
takes.
The floor reaches thread::Builder only; threading.stack_size() still
answers with what was asked for, and release builds are unchanged.
Assisted-by: Claude
* Check the native stack on every frame entry
The C-stack guard ran on one frame entry in eight. That asks the margin to
cover eight frames rather than one, and it does not: an unoptimized frame
entered through native code takes 88,672 bytes against a debug margin of
262,144. A recursion whose steps re-enter that way -- `__add__` calling
itself, a sort key that sorts -- ran off the end of the stack instead of
raising RecursionError. On a debug build `class Add: __add__ = lambda s, o:
s + o; Add() + 1` segfaulted on the main thread; it now raises, as it does
under CPython and in release builds.
enter_iframe checked and then called enter_iframe_unchecked, which checked
again; it now leaves the check to the one call.
Measured on a call-dominated benchmark, five interleaved pairs: instructions
retired go up 0.17%, about four per call, which is the stack pointer read
and the compare.
Assisted-by: Claude
* Take an iterable's length hint when a list is filled from it
`map_py_iter` read `__length_hint__` only to pass it to `PyIterIter`, and
returned an empty vector when the hint was `isize::MAX` or more. Collecting
through `PyResult` dropped the iterator's lower bound, so nothing reserved
the room the hint asked for.
`list()`, `list.extend()` and `list.__iadd__()` now reserve it and report a
hint they cannot honour as `MemoryError`; a hint that leaves no room for the
elements the list already holds is passed over, as `list_extend()` does.
`tuple()` and the other callers keep filling up without reserving.
`length_hint_opt` errors other than the `TypeError` it already turns into
`None` now reach the caller instead of being dropped.
`__iadd__` and `inplace_concat` went through `extract_cloned`, which reads
`__len__` and not `__length_hint__`; both call `PyList::extend` now, the way
`list_inplace_concat()` calls `list_extend()`.
The tuple, list and dict fast paths of `extract_elements_inner` reserve their
known length, which the `collect()` they used dropped.
Assisted-by: Claude
* Drop map.__length_hint__
`map_methods` has no `__length_hint__`, so `operator.length_hint()` on a map
answers 0, not the length of what it draws from.
The method walked into the length hint of every iterator it holds, and a
chain of maps 10000 long overflowed the native stack answering for the
outermost one. It also took the longest of its iterators, where a map stops
at the shortest.
Assisted-by: Claude
* Ask for a length hint where each caller asks for it
`map_py_iter` asked the iterable it was handed, for every caller, and reported
what asking raised. Only some callers ask it: `list_extend()` and
`_PyBytes_FromIterator()` ask the iterable, `PySequence_Tuple()` asks the
iterator, and the bytearray constructor asks nothing. `tuple()`, `min()`,
`max()`, `collections.deque()` and `f(*x)` raised for an iterable whose
`__len__` or `__length_hint__` does, where they answer.
Which object is asked is now the caller's to say. `sorted()` asks the iterable,
being `PySequence_List()`.
`bytes_from_object()` stood in for `PyBytes_FromObject()`, for
`bytearray_extend()` and for the bytearray constructor, which do not agree on
this: the first two ask, the last does not. It is split, and assigning to a
bytearray slice takes the constructor's side with `PyByteArray_FromObject()`.
`list.extend()` counted what it held before the iterable had been asked, where
`list_extend()` reads `Py_SIZE(self)` after. A `__length_hint__` that adds to
the list made the overflow guard read a count too small and raise `MemoryError`
where nothing is wrong; one that empties it made the guard skip a reservation
that cannot be served.
Assisted-by: Claude
* Settle product's pool count before it reads its arguments
`product_new()` checks `repeat` and works out `npools` before it calls
`PySequence_Tuple()` on any argument, and fills the pools `npools` times.
The pools were filled by repeating the arguments `repeat` times instead, which
walks that many steps even with no arguments to repeat: `product(repeat=2**62)`
counted up to it rather than answering `[()]`. The count was also worked out
after the arguments had been read, so a repeat too large to serve ran their
code first.
Assisted-by: Claude
* Let the bool format answer with the error its value raised
`pack_single()` leaves `'?'` to `PyObject_IsTrue()` and returns what that
raised. Packing classified the error instead, so a `ValueError` from a
`__bool__` came back as "memoryview: invalid value for format '?'".
Assisted-by: Claude
* Release a cell's old value after the lock
`PyCell::set` dropped what it replaced while still holding the mutex guarding
the cell contents. A `__del__` running from that drop and reading the same cell
waited on a lock its own caller held, so `del it` on a closure variable whose
value has such a `__del__` deadlocked.
The replaced value is now released once the guard is gone, as `Py_XSETREF`
stores before it decrefs.
Assisted-by: Claude
* Have a set iterator hold the set it iterates
The iterator kept only a reference to the inner hash table, so in
`it = iter(A(*args))` the `A()` temporary was the last owner and died as the
call returned, before `it` was bound. A `__del__` reading `it` there saw an
unbound name and its error was printed and ignored.
The iterator now holds the set object, as `si_set` does, and releases it once
exhausted. That release happens after the lock is dropped, where
`setiter_iternext()` puts its `Py_DECREF(so)` past `Py_END_CRITICAL_SECTION()`,
so a `__del__` that iterates again does not wait on a lock the call holds.
Assisted-by: Claude
* Release an exhausted iterator's container after the lock
`PositionIterInternal::_next` overwrote its `IterStatus::Active` while the
caller still held the mutex around it. Dropping the container there ran any
`__del__` under that lock, and a `__del__` that iterated the same object again
blocked on it.
`exhaust()` now hands the container back instead of dropping it, and
`locked_step()` releases it after the guard. list, list_reverseiterator, tuple,
str, dict and its views and reverse views, bytes, bytearray, memoryview,
array, deque, and the enumerate and sequence iterators all go through it.
Assisted-by: Claude
* Unskip test_free_after_iterating
Assisted-by: Claude
* Stop listing test_set as an environment polluter
`check_free_after_iterating` no longer leaves an ignored exception behind, and
the job that reruns the listed tests ten times fails once one of them stops
polluting.
Assisted-by: Claude
* Drop winsound imports left unused
`TryFromBorrowedObject`, `exceptions`, and `ToWideString` have no reference in
the module, which fails the Windows clippy line under `-Dwarnings`.
Assisted-by: Claude
* Ask nothing where the caller takes no room
`map_py_iter` asked the iterator for a length hint on behalf of every caller
that does not reserve, and reported what asking raised. Those callers ask
nothing at all: `tuple()`, `f(*x)`, `bytearray(x)`, `min()`, `max()` and
`collections.deque()` answer for an iterator whose `__length_hint__` raises,
where they had been raising it.
The answer was also never spent. It reached `PyIterIter` for a `size_hint()`
the push loop does not read, so the lookup and any call it made were work
thrown away: `tuple()` over a generator drops 22% of its instructions, and 17%
over an iterator with a `__length_hint__` written in Python.
Assisted-by: Claude
* Stop asking an iterator how long it is to walk it
`PyIter::iter` and `PyIter::into_iter` asked for a length hint and reported
what asking raised. Nothing spent the answer: it reached `PyIterIter` for a
`size_hint()` that every caller either loops past or drops, since collecting
into a `Result` reports no lower bound.
23 operations answered for an iterator whose `__length_hint__` raises, where
they had been raising it: `set`, `frozenset` and the nine `set` methods that
take an iterable, `dict.fromkeys` and the dict view operators, `array` and
`array.extend`, `all`, `any`, `sum`, `io.writelines`, `math.fsum`,
`math.prod`, and `csv.writerow` and `writerows`.
Over a generator, `set()` drops 16% of its instructions and `all()` 23%.
`str.join` and `bytes.join` do ask, reaching their elements through
`PySequence_Fast()`, which fills a list from the iterator. They take
`iter_sized()`, which is now the only way to ask. `iter_without_hint` is gone,
its callers being what `iter` already does.
Assisted-by: Claude
* Name the matrix jobs rather than let the matrix name them
A generated job name lists every value in the matrix entry, so `cargo check`
carried the booleans its dependencies and `skip_ssl` keys expand to, and the
snippets job carried its test arguments and timeout. Adding or removing a key
renames the check, which drops it from the required list until that list is
edited to match. Emptying `env_polluting_tests` renamed three checks this way.
`Run rust tests` and `clippy` keep the names they had. `cargo check` drops the
booleans from six of its nine, and the snippets job drops its arguments and
timeout from all three.
Assisted-by: Claude
* Keep a sequence iterator active when an element raises
`PositionIterInternal::_next` exhausted the iterator for any non-`Return`
result, so an error from `__getitem__` ended the walk. `iter_iternext()`
lets go of its sequence for `IndexError` and `StopIteration` alone, and
`PyIterReturn::from_getitem_result` has already turned the first of those
into the second, so only `StopIteration` exhausts now.
Both deque iterators keep exhausting on their own mutation guard, which
`deque_iternext()` does by zeroing the counter before it raises; they
share one step function for it, and the message it raises is lowercased
to match the three other sites in the module.
Assisted-by: Claude
* Name the caller in the bytes conversion TypeError
An object with no iteration protocol reached `PyObject_GetIter`'s "not
iterable" message, and `bytearray.extend()` reported bytes. Each entry
point now checks for the protocol first and names itself:
bytes(object()) cannot convert 'object' object to bytes
bytearray(object()) cannot convert 'object' object to bytearray
bytearray().extend(object()) can't extend bytearray with object
Assisted-by: Claude
* Take a strong count too large to hold as reachable
`start_gc_refs` clipped a count at `GC_REACHABLE - 1`, a number the
per-reference subtraction could still walk down to zero and collect a
live object. It now stores `GC_REACHABLE`, and `subtract_gc_ref` leaves
that value alone.
Assisted-by: Claude
* Raise AssertionError where a snippet asserts False
`assert False` is removed under `-O`, which the snippet suite may run.
Assisted-by: Claude
* Keep raising for a collection that moved under its iterator
A deque, set or dict iterator raised once and then read as spent. The
guard is sticky: `deque_iternext()` looks at the deque's state before the
count it keeps, and `dictiter_iternextkey()` and `setiter_iternext()`
write a size no collection can have, so every later call finds the same
thing and raises again. `dequereviter_next()` is the one exception,
looking at its count first, so it runs out after the first raise.
Both deque iterators now carry `dequeiterobject.counter` rather than
reading a length back from the deque, and the dict and set iterators
compare the size they captured against the collection's own every time
they are asked how much is left, which is what makes that answer nothing
from the moment the collection changes rather than only once the
iterator has raised.
The set's message is capitalized to match `setiter_iternext()`.
Measured against 3.14.6, for each of deque, reversed deque, set, dict
and a reversed dict view: the hint after the change, the error, the hint
after the error, and what a later call answers.
Assisted-by: Claude
* Give apt-get update a deadline to fail on
The step waits on `apt-get update`, which has no deadline of its own, so
a source that takes the connection and then stops answering holds the
job until the workflow's own timeout. The retry that disables the
Microsoft and azure-cli sources runs only when the update exits
non-zero, which a held connection never does; three jobs on this branch
sat on this step for 20 minutes to five and a half hours.
Each attempt is now bounded, and the transports are given a timeout, so
a source that stops answering reaches the retry.
Assisted-by: Claude
|
||
|
Hanjeong LeeandJeong, YunWon
|
d8bb7bbd6f |
builtins: generate accurate __text_signature__ (#8512)
* Drop the $module marker from generated __text_signature__
CPython's C functions receive the module as their first argument, so
PyCFunction.__self__ is the module and inspect strips the $module
parameter when building a Signature. A #[pyfunction] takes no such
argument, PyNativeFunction::zelf is None, and inspect has nothing to
strip, so the marker surfaced as a parameter that does not exist:
inspect.signature(len)
(module, /, obj) # was
(obj) # now
All 45 builtins shared with CPython carried it. Methods are unaffected;
their $self marker comes from func_sig and both branches now produce the
same string.
Assisted-by: Claude Code:claude-opus-5
* Mark generated __text_signature__ parameters positional-only
Arguments bind through `FuncArgs::take_positional`, which pops from the
positional list and never consults the keyword map, so a #[pyfunction]
argument cannot be passed by name:
>>> len(obj=[1, 2])
TypeError
The generated signature omitted the `/` marker, so inspect reported those
parameters as POSITIONAL_OR_KEYWORD, contradicting the call above. Emit
the marker, except for `*args`/`**kwargs`, which cannot be followed by
`/`, and for empty parameter lists.
14 of the 45 builtins shared with CPython now report an identical
signature, up from 0.
Assisted-by: Claude Code:claude-opus-5
* Emit no __text_signature__ when an argument has no name
Arguments bound by a destructuring pattern, e.g.
fn round(RoundArgs { number, ndigits }: RoundArgs, ..)
have no name to report, and func_sig stringified the pattern verbatim:
>>> round.__text_signature__
'($module, RoundArgs { number, ndigits })'
That is not valid Python, so inspect.signature() raised "builtin has
invalid signature". Return None instead, which leaves
__text_signature__ unset and makes inspect raise "no signature found",
the same as for a CPython builtin that has no signature.
Affects round, sum, os.pathconf, binascii.b2a_base64 and
binascii.b2a_uu. Their docstrings are unchanged; only the signature
prefix is dropped.
Assisted-by: Claude Code:claude-opus-5
* Name builtin parameters after CPython
These parameters are positional-only, so their names only ever appear in
__text_signature__ and cannot be used at a call site. Naming them after
CPython makes the generated signatures directly comparable:
bin x -> number
ord string -> character
divmod a, b -> x, y
setattr attr -> name
delattr attr -> name
hasattr attr -> name
isinstance typ -> class_or_tuple
issubclass subclass,typ -> cls, class_or_tuple
aiter iter_target -> async_iterable
23 of the 45 builtins shared with CPython now report an identical
signature, up from 0 before this branch. The remainder need FromArgs to
report the parameters of its own structs, which is left for a follow-up.
Add extra_tests/snippets/builtin_signature.py covering the phantom
module parameter, the positional-only marker, the names above, and the
signature-less builtins.
Assisted-by: Claude Code:claude-opus-5
* Drop expectedFailure from test_module_level_callable_noargs
pydoc's summary line for time.time was "time(module)" because the
generated signature carried a $module parameter that inspect could not
strip. It now reads "time()", as the test expects.
Assisted-by: Claude Code:claude-opus-5
* Guard the signature-less assertions to RustPython
test_snippets runs every snippet under CPython as well, and CPython does
have Argument Clinic signatures for round and sum, so that block only
holds for RustPython.
Assisted-by: Claude Code:claude-opus-5
* Update crates/derive-impl/src/util.rs
* Fix ord's parameter reference after the merge
The merge of main took ord's signature from this branch, which renamed
the parameter to character, and its body from main, which rewrote ord to
accept bytes and bytearray through a parameter named c. The body then
referenced a name that no longer existed and the build failed.
Assisted-by: Claude Code:claude-opus-5
---------
Co-authored-by: Jeong, YunWon <69878+youknowone@users.noreply.github.com>
|
||
|
Jeong, YunWon
|
182175b0c9 |
memoryview: count exports, hash the exporter, and index through __index__ (#8553)
* memoryview: index through `__index__` and name the dimension
A tuple key was taken apart only when every item was already an `int`, so
`m[I(), 2]` with an `__index__` on `I` was reported as an invalid slice key
rather than indexed. What kind of key it is now follows from the types in
it, the way `is_multiindex` decides, and each item is converted where it
sits; an item that answers `__index__` but raises on the way reports that
rather than making the whole key invalid.
That conversion runs Python, which can release the view, so the
multi-dimensional read goes through `unpack_single` and its re-check
rather than reading the bytes itself.
`IndexError` named the index that was out of bounds where
`lookup_dimension` names the dimension it was out of bounds on, counted
from one, and the one-dimensional path had a message of its own.
Assisted-by: Claude
* memoryview: count what a view has exported, and hash the exporter
The view kept no count of the buffers taken from it, so `release()` always
succeeded, even while something was still reading through an export. It now
answers `BufferError` the way `_memory_release` does, and `__exit__` reports
the same refusal.
`memory_hash` asks the exporter for its hash and throws the answer away, so
a view is no more hashable than what it looks at; a read-only view over a
`bytearray` hashed here where CPython refuses it. Asking runs Python, so the
view counts as exported for the duration and a release attempted from inside
that hash is refused rather than obeyed.
Assisted-by: Claude
* bytes, bytearray, memoryview: measure the hex separator as `len()` does
`hex()` took the separator's length from its bytes rather than from the
object, so a `bytes` subclass that answers `__len__` was measured by what it
holds instead of what it says: `b'abcd'.hex(S(b'::'))` with `__len__`
returning 1 was refused where `_Py_strhex_impl` accepts it, and a two-byte
answer was accepted where it refuses. The check also came after the
`bytes_per_sep == 0` and empty-data shortcuts, so `b'abcd'.hex('::', 0)`
never reached it at all.
Measuring runs Python, and the bytes to be written out must not be borrowed
while it does, so the separator is now resolved from the arguments before
the buffer is reached. For a memoryview that also means the view counts as
exported for the duration, so a release attempted from inside `__len__` is
refused.
Assisted-by: Claude
* array: refuse to hash an array
`array.array` is mutable but inherited `object.__hash__`, so an array could
be put in a set or used as a dict key and then changed underneath it. It is
unhashable, as `tp_hash = PyObject_HashNotImplemented` makes it.
A read-only memoryview over an array now reports the exporter as unhashable
too, which is what `memory_hash` asks it for.
Assisted-by: Claude
* Remove expectedFailure from three memoryview tests
test_hash_use_after_free, test_hex_use_after_free and
test_use_released_memory pass.
Assisted-by: Claude
* memoryview: bound the dimensions a cast can name
`cast()` accepted a shape of any length, so `mv.cast('B', (1,)*64 + (8,))`
built a 65-dimensional view where `memoryview_cast_impl` refuses anything
past `PyBUF_MAX_NDIM`. The limit is answered before the shape is looked at
any further, as it is there.
Assisted-by: Claude
|
||
|
Kwak Byoung Min
|
62d04c6d3d |
Add StringIO newline tracking (#8539)
* Report observed StringIO newline types Constraint: Match CPython StringIO newline reporting without changing configured newline translation. Rejected: Separate newline-tracking representation | reuse the existing SeenNewline bitflags. Confidence: high Scope-risk: narrow Directive: Keep observed newline state separate from the configured newline mode. Tested: prek run --all-files; test_memoryio; cargo clippy -p rustpython-vm --lib -- -D warnings; workspace tests excluding the macOS C-API baseline SIGSEGV. Not-tested: Full macOS workspace suite is blocked by the existing rustpython-capi SIGSEGV; the Linux suite is running. Assisted-by: Codex:gpt-5.6-sol * Preserve StringIO state boundaries Reset observed newline state when StringIO is reinitialized and keep the property subject to closed-stream checks. Constraint: Match CPython StringIO reinitialization and closed-stream behavior. Rejected: Returning stale or cached newline state | violates StringIO lifecycle semantics. Confidence: high Scope-risk: narrow Directive: Keep observed newline state tied to the current open StringIO contents. Tested: cargo test --workspace --exclude rustpython_wasm --exclude rustpython-venvlauncher; cargo run --release -- extra_tests/snippets/stdlib_io_stringio.py; cargo run --release -- -m test test_memoryio; prek run --all-files; cargo clippy -p rustpython-vm --lib -- -D warnings. Assisted-by: Codex:gpt-5.6-sol |
||
|
Jeong, YunWon
|
c85b83f97b |
Fix crashes found hunting the last open fuzzing record (#8524)
* specialize: check the member descriptor's type before caching its slot offset
The LOAD_ATTR/STORE_ATTR specializations cached the slot offset of any member
descriptor found on the owner's type and then guarded the specialized
instruction on the type version alone, while descr_get()/descr_set() check on
every access that the instance belongs to the type the descriptor was defined
for. A descriptor taken from a wider class and bound to a narrower one read
past the instance's slot array once the cache warmed up:
class Big:
__slots__ = ("a0", "a1", "a2", "a3", "a4", "a5", "a6", "a7")
class Narrow:
__slots__ = ("z",)
Narrow.x = Big.__dict__["a7"]
o = Narrow()
for _ in range(1000):
try: o.x
except TypeError: pass
# index out of bounds: the len is 1 but the index is 7 (object/core.rs)
A class with no slots at all reached the ext_ref().unwrap() on the same line.
Assisted-by: Claude
* socket: reserve recv()'s buffer fallibly
recv() and recvfrom() handed the caller's bufsize straight to
Vec::with_capacity, so an unreachable size aborted the process through
handle_alloc_error before any syscall was made:
socket.socket().recv(2**62)
# memory allocation of 4611686018427387904 bytes failed -> SIGABRT
try_reserve_exact reports MemoryError instead, which is what CPython raises.
Assisted-by: Claude
* types: count the __call__ and __get__ slot dispatches as recursion
Both wrappers re-enter Python without pushing a frame, so nothing counted the
nesting when the special method named the object it was looked up on:
class C: pass
c = C(); C.__call__ = c
c() # native stack overflow, SIGSEGV
class D: pass
d = D(); D.__get__ = d; D.x = d
d.x # the same, through descr_get
with_recursion around the two dispatches raises RecursionError instead, the
way Py_EnterRecursiveCall bounds a tp_call dispatch. It costs about 5% on a
__call__ dispatch and 3% on a __get__ dispatch through these wrappers.
Assisted-by: Claude
* typevar: show a ParamSpecArgs origin by its repr
ParamSpecArgs and ParamSpecKwargs fell back to a Rust `{:?}` of __origin__ when
it had no __name__. That walks the object graph natively through Debug for
PyInner, where no recursion guard sits, so a single repr() of a deeply nested
chain overflowed the native stack:
a = object()
for _ in range(30000):
a = typing.ParamSpecArgs(a)
repr(a) # SIGSEGV
The origin is formatted with its repr now, which is guarded, and a ParamSpec
origin is recognized by its type rather than by carrying a __name__.
Assisted-by: Claude
* Do not hold a lock across a call back into Python
Three places kept a lock while running code that can reach the same object, so
a callback that touched it wedged the process:
_asyncio.future_add_to_awaited_by(fut, waiter) # waiter.__hash__ adds again
select.select(elements, [], [], 0) # fileno() clears `elements`
select.poll().poll(1000) # SIGALRM handler registers
The future's awaited-by field is read and written under its lock but the set is
built outside it, the list extraction re-reads the list on each step the way
map_iterable_object() does, and poll() waits on a copy of its descriptors. All
three ran forever before and now finish the way they do on CPython.
Assisted-by: Claude
* Validate memoryview.cast() arguments and export negative strides correctly
cast() accepted any struct format and any shape element. A zero-size
format ('0s') and a 0 in the shape both reached a division by zero;
cast() now takes only a native single character format, optionally
'@'-prefixed, and shape elements that are ints greater than zero.
A view with a negative stride starts at its last item, so the bytes it
exported began there and its own offsets walked off the front of them.
Such a view now exports the whole underlying buffer with `start` folded
into the descriptor's offsets, and zip_eq() hands over a whole run only
when both sides are contiguous in the last dimension.
Assisted-by: Claude
Assisted-by: Codex:GPT-5
* Charge native recursion to the stack, not to the frame limit
with_recursion() checked the limit sys.setrecursionlimit() sets and
incremented the same counter that pushing a frame does, so a guard on a
native dispatch spent what Python code had left to call with, and did so
where sys._getframe() cannot see it: test.support.get_recursion_available()
reported frames that were no longer there. Py_EnterRecursiveCall bounds
the native stack instead, which is a separate budget, and the C stack
check with_recursion already performs is that bound.
The snippets pinning the guarded paths nest deep enough to reach the
stack rather than the frame limit.
Assisted-by: Claude
* Report a size that cannot be allocated instead of aborting on it
A size taken from Python went straight into an infallible allocation in
several places, so the process aborted through handle_alloc_error before
any exception could be raised:
- str/bytes/bytearray center(), ljust(), rjust() and zfill() reserved
the padded result for the caller's width
- expandtabs() built its runs of spaces from a tabsize of any width;
the argument is a C int, and a wider one does not fit
- Buffered{Reader,Writer,Random} allocated buffer_size, and read(),
read1() and FileIO.read() their read size
- bytes(n) and bytearray(n) allocated n
- pbkdf2_hmac() allocated the derived key length, which is a C int
Each of these now reports MemoryError, or OverflowError where the
argument does not fit the type it is declared with.
new_zeroed_bytes() leaves the zeroing to the allocator, so a large
request costs the pages that are written to rather than all of them.
Assisted-by: Claude
* marshal: answer allow_code where a code object is written or read
allow_code was answered by walking the whole result a second time, with
no depth counter and no record of what it had already seen, so a value
that referred back to itself or nested deeply enough ran off the native
stack. w_object() and r_object() answer it where the code object is,
inside the walk that already bounds its depth and resolves references.
A container length is read the way r_long() reads one: it is signed, so
a length with the top bit set is out of range rather than four billion
items to reserve room for.
load() no longer holds a borrow of the buffer read() returned across the
seek() it makes afterwards.
Assisted-by: Claude
* Do not lock an object while running code that can reach it
Several places held a lock or a borrow of an object across a call back
into Python, so a callback that touched the same object waited on a lock
its own caller was holding:
- memoryview slice assignment read a source overlapping the destination,
and __setitem__ converted the value while holding the write borrow
- BytesIO.readinto() read into a buffer viewing the same BytesIO
- array.__setitem__ converted the value under the array's write lock,
and mmap.write() read a source viewing the same map
- bytearray.join() and bytearray.__mod__ drove Python with the
bytearray borrowed
- array and bytearray answered "is this resizable" after taking the
write lock, though an export is exactly a borrow someone else holds
A TextIOWrapper cookie now has to name a position inside what was
decoded in characters as well as in bytes; only the byte offset was
checked, and the character count is what read() and tell() index with.
Assisted-by: Claude
Assisted-by: Codex:GPT-5
* Stop asserting a pbkdf2 message that depends on the width of a C long
The snippet asserted "key length is too great.", which pbkdf2_hmac()
only reaches once the length has been converted; where a C long is
narrower than the length asked for, the conversion fails first and says
so instead. Both are OverflowError, which is what the case is about.
test_support.test_get_recursion_depth passes now that a native recursion
guard no longer spends frames get_recursion_depth() cannot see.
Assisted-by: Claude
* Publish and read the same pointer for a thread's top frame
set_current_frame() casts the `Py<FrameObject>` it publishes straight to
`*mut FrameObject`, so ThreadSlot::top_frame holds the object's base.
sys._current_frames() read it back through Py::from_payload_ptr(), which
subtracts the payload offset from what it is given. The reference it took
therefore incremented, and later decremented, a word 48 bytes ahead of
the frame -- inside the object allocated before it, whose OnceLock state
word sits exactly there for two frames adjacent in the size class. The
neighbour then read an initialized-looking cold pointer that had never
been written and locked whatever the uninitialized word addressed, so
the thread that owned it crashed rather than the one that read.
The slot now holds `*mut Py<FrameObject>`, which is what both sides mean.
A thread parked in a call has no FrameObject for its topmost frame, so
top_frame is null there and the reader takes the materialize path
instead: test_sys.test_current_frames never reaches the branch. The
snippet takes _current_frames() against threads that are running.
Assisted-by: Claude
Assisted-by: Codex:GPT-5
* Decide stop-the-world parking under the thread registry lock
do_suspend() published SUSPENDED first and only then re-read `requested`,
restoring itself to ATTACHED if the stop had ended in the meantime. That
made a thread the second writer able to leave SUSPENDED, so a stop whose
completion check had already observed the thread parked could be undone
behind the requester's back:
worker CAS ATTACHED -> SUSPENDED
requester all_non_requester_suspended() -> true, world_stopped = true
requester start_the_world(): requested = false, then walks the registry
worker reads requested == false, stores ATTACHED
With the store landing inside that walk the debug assertion in
start_the_world fires; with the walk already past the slot, a following
stop force-parks the thread DETACHED -> SUSPENDED, counts it as stopped,
and the store then puts it back to ATTACHED with the world declared
stopped and the thread running bytecode.
`requested` is set in init_thread_countdown() and cleared in
start_the_world() with the registry held, and start_the_world() keeps
holding it while releasing every SUSPENDED thread. Taking the registry
around the check and the transition therefore makes the two orders the
only ones possible: park before that release pass and be woken by it, or
find the request already withdrawn and stay ATTACHED. The requester is
left as the only writer that takes a thread out of SUSPENDED, and the
self-restore is gone.
suspend_if_needed() takes the VirtualMachine to reach the registry.
Assisted-by: Claude
Assisted-by: Codex:GPT-5
* Keep an atexit callback alive while it is being compared
atexit.unregister() releases the callback list around each __eq__ call and
identified the entry it had compared by the address of its Box. __eq__ can
call atexit._clear(), which drops that Box, and atexit.register(), whose
new Box lands on the freed allocation; the identity search then matched
the freshly registered callback and removed it.
atexit.register(a); atexit.register(b); atexit.register(c)
# __eq__ runs _clear() then register(d), returns True
atexit.unregister(probe)
left no callbacks registered where CPython leaves d.
Entries are Arc-shared now, so unregister() holds the one it is comparing
and matches it with Arc::ptr_eq: an address cannot be reused while the
comparison that named it is still running.
Assisted-by: Claude
* Hold atexit entries in PyRc rather than Arc
PyObjectRef is Send and Sync only under the threading feature, so an Arc
over a callback entry trips clippy::arc_with_non_send_sync in builds
without it, such as the wasm package. PyRc is Arc there and Rc otherwise.
Assisted-by: Claude
* Do not hold a buffer's storage while waiting for a peer
FileIO.readinto() and socket.recv_into()/recvfrom_into() took the target
buffer's write borrow and kept it for the whole call, including the wait
for data a pipe, socket or terminal may never deliver. What CPython holds
across that wait is the export, which only forbids resizing; the borrow is
a lock every other thread touching the same object waits on, so
threading.Thread(target=lambda: sock.recv_into(buf)).start()
len(buf)
did not answer until the peer sent. A thread parked on that lock is
ATTACHED and never reaches a safepoint, so gc.collect() in a third thread
waited for the peer as well: one incidental read of the buffer stopped the
world from being stopped at all.
The wait now runs against storage of its own and the bytes are copied over
once they arrive, with the export held throughout so the target still
cannot be resized meanwhile. A seekable file answers from itself rather
than from a peer, so FileIO.readinto() writes into the target directly
there and the buffered read path is unchanged.
Assisted-by: Claude
* Do not hold a buffer's storage while waiting to hand it over
socket.send()/sendall()/sendto()/sendmsg() and FileIO.write() kept the
source buffer's read borrow for the whole call, including the wait for a
peer that may never make room. That borrow is a lock every other thread
writing to the same object waits on, so
threading.Thread(target=lambda: sock.sendall(buf)).start()
buf[0] = 1
did not return until the peer read; and a thread parked there is ATTACHED
and never reaches a safepoint, so gc.collect() in a third thread waited
for the peer too -- the same wedge readinto() had on the receiving side.
ArgBytesLike::borrow_buf_unlocked() answers with bytes that survive the
borrow being dropped. An immutable object hands out a plain reference and
locks nothing, so those are sent where they lie and bytes and memoryviews
over them cost nothing; only bytes reached through a lock are copied out
first. The export is held throughout either way, so the source still
cannot be resized while it is being sent.
The regression snippet covers both directions now and is renamed for it.
Assisted-by: Claude
Assisted-by: Codex:GPT-5
* Check select()'s descriptor limit while the sequence is walked
seq2set() collected the whole sequence and compared the result's length
against FD_SETSIZE afterwards. Selectable::try_from_object() calls
fileno(), which runs Python and can append to the list being walked, and
the walk re-reads the list on every step, so the collection had no end to
reach and the comparison was never made. seq2set in Modules/selectmodule.c
checks the count per element instead.
stdlib_select.py gains a fileno() that appends to its own list, and
releases its sockets with close() rather than by dropping the name.
Assisted-by: Claude
* Bound native recursion where the C stack cannot be measured
check_c_stack_overflow() answers no unconditionally under miri and on
musl, where the stack pointer is not read. Since 9c9905aff that check is
all with_recursion() does, so every guard placed on native recursion --
__call__ and __get__ dispatch among them -- was a no-op on those targets
and the nesting ran until the stack ran out.
with_recursion() now counts its own depth on those targets and refuses
past NATIVE_RECURSION_LIMIT_UNMEASURED. The count is separate from the
frame limit sys.setrecursionlimit() sets, and compiles away where the
stack pointer can be read.
Assisted-by: Claude
* Report the failure to allocate pbkdf2's key and Take.readinto's scratch
Both buffers are sized from an argument -- pbkdf2_hmac's dklen accepts up
to i32::MAX, and readinto's from the length of the destination -- and were
built with vec![0u8; n], which aborts the process on allocation failure.
new_zeroed_bytes() raises MemoryError instead.
Assisted-by: Claude
* Read the frozen-code tuple length as a signed length
The '(' branches in read_marshal_str_vec() and read_marshal_const_tuple()
took the length with read_u32() as usize, so a value with the top bit set
read as four billion items rather than as out of range. read_len() is what
every other length in this file goes through, and it reinterprets as i32.
Both readers serve deserialize_code(), which reads only the frozen modules
baked in at build time, so this changes no reachable behavior;
marshal.loads() already went through read_len().
Assisted-by: Claude
* Make the snippets from the fuzzer sweep assert what they check
builtin_str.py expanded a tab to 2**31-1 columns, allocating 2 GiB to
observe that the width is accepted; a string with no tab observes the same
acceptance without laying anything out.
stdlib_array.py caught the refusal of frombytes() on its own exported
buffer and passed silently when nothing was raised. stdlib_hashlib.py
used a bare `assert False` as its failure branch. Both now say so through
the same shapes the other snippets use.
stdlib_socket.py also accepts OverflowError from recv() with a size that
does not fit the platform's C int.
stdlib_threading_current_frames.py indexed the frame chain for "f123"
without first asserting it is there.
Assisted-by: Claude
* Ask for a marshal container's room instead of assuming it
A flagged tuple or list is published in the reference table before its
children are read, and the placeholder was built with vec![none; len].
The length is the input's to choose and read_len() lets it reach
i32::MAX, so marshal.loads(b"\xa8\xff\xff\xff\x7f") -- five bytes -- asks
for 17 GB of element slots and aborts the process where the allocator
cannot serve it. r_object() allocates the container up front too, but
PyTuple_New() reports what it cannot get.
The elements are now reserved with try_reserve_exact() and a refusal is
raised as MemoryError through the decoder's pending-error channel.
PyTuple::new_marshal_placeholder() held nothing but that allocation and
is gone; the caller builds the elements and uses new_ref().
Assisted-by: Claude
* Compare the blocking-buffer snippet against something
measure() asserted len(buf) == len(buf), which holds whatever the length
is; it now takes the length the caller expects. The pipe case compared
the drained total against the whole source, while an unbuffered write()
reports only what it transferred and a signal can cut that short; it now
compares against what write() returned.
Assisted-by: Claude
* Refuse array.frombytes() a source whose items are not bytes
frombytes() reads its argument as bytes, but accepted any contiguous
buffer, so array("i").frombytes(memoryview(array("d", [1.0]))) appended a
double's bytes read as ints instead of raising. array_array_frombytes_impl
requires an itemsize of 1; ArgBytesLike now reports the itemsize so the
same check can be made here.
The BufferError that a resize meets while the array is exported also names
the array rather than repeating bytearray's wording.
stdlib_array.py's frombytes-of-itself case used typecode "i", where the
new check answers before the resize guard is reached; it uses "b" so the
guard is what refuses, and asserts the wider case separately.
Assisted-by: Claude
* Tell apart the ways marshal data can be bad
A type byte no reader knows, a back reference that names nothing, and
TYPE_NULL all came out as a bare "bad marshal data" ValueError. r_object()
answers the first two with "bad marshal data (unknown type code)" and
"bad marshal data (invalid reference)", and read_object() answers the
third with a TypeError, "NULL object in marshal data for object", since
TYPE_NULL stands for no object rather than for a value.
MarshalError gains the three cases and deserialize_value() maps them.
The container-specific wording r_object() uses for a NULL read inside a
tuple or list is not reproduced; the exception type is.
Assisted-by: Claude
* Hash the collector's tables by address rather than by SipHash
A collection keys three sets and two maps by object address, and it visits
every tracked object and every edge between them, so the hashing is a
per-edge cost. Those tables used the default RandomState, whose SipHash
buys resistance against a caller choosing colliding keys -- and nothing
chooses these keys: they are addresses this process handed out into tables
that live and die inside one collection. A profile of gc.collect() over a
423k-object heap spent 45% of its samples in SipHash.
They now hash with a splitmix64 finalizer. The shifts matter: a table
picks its bucket from the low bits and an address arrives with those bits
zeroed by alignment, so a plain multiply leaves every object in a handful
of buckets and is slower than SipHash was.
The reachability walk also copied each object's referent vector out of the
map it was cached in, a second pass over every edge; it reads them in
place, and reference subtraction hands its vector to the map instead of
cloning it.
Measured over 423k live objects: 0.93s to 0.15s. Over 843k dead ones:
3.00s to 0.79s. extra_tests/snippets/stdlib_threading_gc_import.py, whose
collector thread calls gc.collect() in a loop, ran anywhere from 2.7s to
28s and now runs in 3.1-3.5s: a collection that takes longer leaves more
garbage for the next one to walk, so the cost fed back on itself.
Assisted-by: Claude
Assisted-by: Codex:GPT-5
* Keep the collection's candidates and their counts in one table
A collection built a set of candidates and, beside it, a map from the same
addresses to their reference counts. Both were probed for every edge in
the heap -- membership from the set, the count from the map -- so each
edge paid to hash the same address twice, and each candidate paid to be
inserted twice. The map alone answers both questions.
The candidates also keep a walkable order now, which the reference
subtraction pass needs since it writes the counts while reading the
candidates, and which the unreachable set is built from instead of a
set difference.
Over the 423k-object heap measured in the previous commit: 0.15s to
0.13s live, and 0.79s to 0.49s dead.
Assisted-by: Claude
Assisted-by: Codex:GPT-5
* gc: collect referents into one buffer instead of a vector per object
Step 3 allocated a `Vec` for every tracked object to hold its referents
and kept them all in a map until step 4 read them back. The referents now
go into a single growing buffer, with the map holding each object's range
into it.
Adds `PyObject::gc_extend_referent_ptrs`, which appends to a caller's
buffer; `gc_get_referent_ptrs` calls it with a fresh one.
Assisted-by: Claude
* memoryview and struct: match the checks and errors of the reference
memoryview:
- `cast()` accepted a source and destination that are both item types,
which reinterprets the items rather than re-dividing the bytes; one side
now has to be a byte format.
- A cast to `shape=()` returned without checking that the buffer holds
exactly the one item that shape describes.
- `hash()` hashes the bytes, so it now raises ValueError for a view whose
items are not bytes, rather than returning a hash that disagrees with the
value the view compares equal to.
- `tobytes()` takes the `order` argument, with 'F' walking a
multidimensional view down its columns; `BufferDescriptor` gained
`for_each_segment_fortran` for that walk.
struct:
- A value the format has no room for reported "argument out of range"
instead of naming the format and its range. The format character is now
passed to the packing functions to report it.
- `Struct.__new__` no longer reads the format; `__init__` does, so
`__init__` can be called again and a subclass can pass the format up.
Methods raise RuntimeError until it has run, and `Struct` is a base type.
Removes the expectedFailure from test_Struct_reinitialization and
test_struct_subclass_instantiation.
Assisted-by: Claude
Assisted-by: Codex:GPT-5
* io: decide the readinto path by file type, not by seekability
FileIO.readinto wrote straight into the caller's buffer, holding its write
borrow, when the fd was seekable; otherwise it read aside into scratch and
copied. Seekability stood in for "this read answers without waiting on a
peer", which a pipe on Windows breaks: lseek on one succeeds, so the pipe
took the borrow-holding path and every other thread touching that bytearray
waited for the peer.
host_io::reads_without_waiting answers it directly -- seekability elsewhere,
GetFileType() == FILE_TYPE_DISK on Windows.
The regression snippet times each operation separately, so a failure names
the one that waited; it asserts the transfer is still in flight before
checking the export; and the socket case fills the connection until it
refuses rather than assuming a size that outruns it, which SO_SNDBUF on an
already-connected pair does not settle.
Assisted-by: Claude
|
||
|
Jeong, YunWon
|
f24b2570a9 |
Implement PEP 688 and rework the buffer protocol around managed exports (#8523)
* Implement PEP 688 __buffer__ and __release_buffer__ A Python class could not export a buffer: the slot machinery had no bf_getbuffer or bf_releasebuffer, and every consumer acquired buffers as PyBUF_FULL_RO through a module of PyBUF_* constants. Add both slots. PyBuffer::release now runs a Python __release_buffer__ before the exporter's own release, once per acquisition, which PyBuffer tracks with an `acquired` flag that clones do not inherit. An export made by a Python __buffer__ is held by a _buffer_wrapper payload that counts its exports and drops the returned memoryview with the last one, and the view handed to __release_buffer__ is a _buffer_window that owns no export, so releasing it inside the hook is inert instead of re-entering it. Replace the PyBUF_* constants with a BufferFlags bitflags type whose composite requests are supersets of the simpler ones, so `contains` answers the REQ_* questions, and pass the request to PyBuffer::from_object. Each consumer now asks for what its counterpart asks for: y* arguments for SIMPLE, w* for WRITABLE, BytesIO.write for CONTIG_RO, bytes(), bytearray() and memoryview() for FULL_RO. memoryview checks the request in memory_getbuf, and array.array and mmap.mmap expose __release_buffer__. Test buffer support with PyObject::check_buffer (PyObject_CheckBuffer) instead of attempting an acquisition, so an exception raised by __buffer__ is no longer reported as the object not being bytes-like, and a __buffer__ with side effects runs once. PyBytesInner becomes a y* conversion as a result: bytes and bytearray methods no longer accept iterables of ints, and find, index, count and __contains__ take the arguments parse_args_finds_byte and bytes_contains describe. A view exports its start offset in the descriptor rather than in its window, which fixes a panic when collecting from a negative-stride view. BytesIO.write rechecks closed after acquiring its buffer, which __buffer__ can close in between. Assisted-by: Claude Code:claude-opus-5 * Rework the buffer protocol around managed exports and view offsets Give `PyBuffer` the `_PyManagedBufferObject` shape: one `bf_getbuffer` acquisition is shared by every handle taken from it, cloning takes another share instead of re-acquiring, and the exporter's release runs once when the last share goes away. Remove `retain`, the unsafe `drop_without_release`, the three `impl Drop`s and the `ManuallyDrop` that stood in for this. Add `abort_acquisition` so a failed request does not run `bf_releasebuffer`. Move the view start into `BufferDescriptor::offset`, the `Py_buffer.buf` analogue, and drop the separate `start` fields on `PyMemoryView` and `PyBufferWrapper`. Slicing goes through `SaturatedSlice::adjust_indices_start`, which reproduces `PySlice_AdjustIndices` and keeps the adjusted start. Fix `zip_eq` to take its contiguous fast path only when both last dimensions are contiguous, and make `for_each_segment` and `zip_eq` handle zero-length and zero-dimensional views. Add `BufferDescriptor::projected` so a request without `PyBUF_ND`, `PyBUF_STRIDES` or `PyBUF_FORMAT` receives a correspondingly reduced descriptor, and reject a request without `PyBUF_INDIRECT` against an exporter that has suboffsets. Copy the source first in `memoryview` slice assignment when both sides reach the same root exporter. Hold the export across the resize in `bytearray.extend`, take `y*` in `marshal.loads`, stop probing the buffer protocol in `FsPath`, rewrite `ord` over the concrete string types, fold `array`'s buffer slot into one `slot_as_buffer`, take `w*`/`y*` in `_overlapped`, and thread the new `offset` field through the `_ctypes` descriptors. Assisted-by: Claude * Re-check a memoryview after a conversion and tighten cast and release `pack_single` and `unpack_single` addressed the buffer with a position taken before `__index__` ran, so releasing the view from that conversion read or wrote outside the exporter's storage, panicking when it had also shrunk. Check the released flag again once the conversion is done, as `CHECK_RELEASED_AGAIN` does. Reject a cast to a format that is not a single native format character with an optional `@` in front of it. An empty format reached a division by its item size of zero. get_native_fmtchar Report a second `__release_buffer__` on the same view as a `ValueError` rather than accepting it, and check that the view belongs to the object first; the silent case is a view that exports nothing. wrap_releasebuffer Compare against another memoryview by reading its view where it lies instead of acquiring a buffer from it, so the restricted view handed to `__release_buffer__` compares equal rather than unequal in one direction only. memory_richcompare Name the type in the unraisable an exception from `__release_buffer__` reports, as `releasebuffer_call_python` does, instead of reporting the exporter object. Assisted-by: Claude * Re-check the destination of a memoryview slice assignment Acquiring the source runs `__buffer__`, which can release the destination view, so check the released flag again once the source is in hand and before the structures are compared, as `copy_single` does. Build the sliced destination as a view that counts as no export, the way a `Py_buffer dest = *view` copy does. Holding one kept the exporter unresizable for the length of the assignment, so a source that released the view and then resized the exporter met a `BufferError` instead of the assignment reporting the released view. Assisted-by: Claude * Answer the memoryview contiguity getsets with the layout `f_contiguous` reported the row-major answer for one dimension and False for any more, so a view laid out both ways, such as one of shape (1, 8), reported False while `check_buffer_request` accepted a `F_CONTIGUOUS` request on it. `contiguous` answered row-major order alone. Move the Fortran-order test from `PyMemoryView` to `BufferDescriptor`, next to the row-major one it differs from only in iteration order, and answer all three getsets with it. Assisted-by: Claude |
||
|
Jeong, YunWon
|
833a2ba63d |
str, bytes: answer equality with equality rather than with an ordering (#8531)
PyStr's comparison, PyBytesInner's, and the specialized CompareOpStr
instruction all answered == and != by taking Ord::cmp of the two buffers
and asking whether the result was Equal. An ordering has to read the
bytes: it memcmps the common prefix even where the lengths already settle
the question. CompareOpStr bypasses the Comparable slot, so it had also
lost the identity shortcut that slot takes, and a string compared with
itself was read end to end.
Add PyComparisonOp::eval_eq, which settles Eq and Ne from an equality test
and leaves an ordering operator to the caller, and answer through it in the
three places: slice equality checks the length first, and CompareOpStr
answers an object compared with itself the way the slot it specializes does.
n=1,000,000, per comparison:
before after
s == s (the very same object) 23.21us 0.16us
s == a string one shorter 24.63us 0.17us
b == bytes one shorter 25.28us 0.20us
ba == bytearray one shorter 27.68us 0.23us
s == an equal, distinct string 23.78us 24.50us
s < an equal string 29.87us 25.07us
Assisted-by: Claude
|
||
|
Jeong, YunWon
|
5919be9c4c |
str: take search bounds through the character index (#8530)
* str: count an empty needle in characters
str.count with an empty needle ran find_iter over the range's bytes, which
reports one position per encoded byte boundary rather than one per
character: '가나다'.count('') answered 10 instead of 4. Count the range's
code points instead; bytes.count keeps the byte-position answer.
Assisted-by: Claude
* str: take search bounds through the character index
find, rfind, index, rindex, count, startswith and endswith resolved their
start/stop bounds with AnyStr::get_chars, which walks the payload to both
bounds, and find reported its hit by counting the code points in front of
it -- so a sweep over a subject's own indices walked it once per call.
Resolve the bounds with StrData::char_range_to_bytes instead, which is the
converter the subscript and slice paths already use, and map the hit back
with Wtf8Index::char_index_at_byte, added here as the table's inverse: a
bracketed search over the groups, then at most an entry's worth of steps.
The payload alone does not say whether it is ASCII, so get_chars walked
there too; that path is now off it as well.
n=8000, sweeping the bound over the subject, best of 3, interleaved:
'가나다라'*n/4 'abcd'*n/4
find 18.00ms -> 0.15ms 3.48ms -> 0.14ms
startswith 16.75ms -> 0.12ms
count 22.78ms -> 4.02ms
count stays linear in the range it is given, as it is in CPython.
Assisted-by: Claude
|
||
|
Jeong, YunWon
|
9f90967b0a |
str: count a stepped slice's characters with a ceiling division (#8525)
`do_stepped_slice` and `do_stepped_slice_reverse` took the character count of
a non-ASCII result as `(range.len() / step) + 1`, which overshoots by one
whenever the span is an exact multiple of the step: `"aéc"[::3]` is one
character and reported two, `"가나다라"[::2]` two and reported three. ASCII
subjects were unaffected, since that arm collects into an `AsciiString` whose
length comes from the data.
The count is stored as the string's character length, so the result then
claimed a character its buffer does not hold, and `reversed()` on it indexed
past the end and panicked:
s = "".join(["a", "é", "c"])
list(reversed(s[::3])) # index out of bounds: the len is 1 but the index is 1
Use `div_ceil`, which is the number of elements the underlying range yields.
A differential over 44331 subscript and slice shapes -- every combination of
start, stop and step over strings straddling the ASCII split, the surrogate
range and the astral plane -- now matches CPython 3.14 exactly, where it
differed on 3100 lines before.
Assisted-by: Claude
|
||
|
Jeong, YunWon
|
b109723242 |
Fix the itertools.tee leak and race, the contextvars and generator races, _asyncio's InvalidStateError lookup, and _imp's frozen data (#8518)
* _imp: read get_frozen_object's data as a whole marshal value
The data argument went through deserialize_code(), which reads a code body
without the type byte in front of it, so nothing marshal.dumps() produces was
accepted. Read it with marshal.loads() and require a code object back.
Assisted-by: Claude
* _imp: bind find_frozen's arguments with FromArgs
The arity check was done by hand on a FuncArgs. Take the arguments through a
FromArgs struct instead, which makes withdata keyword-only, and fill in the
data it asks for: the frozen encoding is not marshal, so the code is
re-serialized into what get_frozen_object() reads back.
Assisted-by: Claude
* itertools: give tee's shared buffer a Python type
The buffer was a PyRc<PyItertoolsTeeData>, which is not a Python object, so the
collector could not walk into it and any cycle running through a tee was
uncollectable. Make it the _tee_dataobject type with a traverse, held by
PyRef, and split the rest along the same lines: tee() is a function, _tee is
the iterator type it builds, and _tee takes a single iterable rather than
returning a tuple from __new__. _tee is weak-referenceable, tee() rejects a
negative n with ValueError and reserves its tuple fallibly.
test_itertools.test_tee passes now; its expectedFailure marker is removed.
Assisted-by: Claude
* contextvars: hold the shared context state under locks
The variable map was a RefCell, the enter flag, the context index, the token's
used flag and the variable hash were Cells, and each carried an unsafe impl
Sync. A Context or ContextVar shared between threads overlapped their borrows
and panicked. The map and the per-variable cache are now PyMutex, the flags and
the index are atomics, entering a context is a compare_exchange, and the three
unsafe impl Sync are gone.
The cache also stopped being read through AtomicCell::as_ptr, which raced a
concurrent store on a value holding a PyObjectRef.
Values displaced from the map or the cache are dropped after the lock is
released: __del__ can come straight back into the same context, and the locks
are not reentrant.
Assisted-by: Claude
* generator: read the frame state under the running claim
send(), send_none(), throw() and close() read `closed` and `frame.lasti()`
before `running` was compare_exchanged, so the frame they went on to resume
could be one another thread had already advanced. A resume that decided from
`lasti() == 0` that the generator had not started pushes no value onto the
value stack, and the code after the yield pops one, which underflows the stack.
The compare_exchange now hands back a guard, taken before those reads and
released after maybe_close(), so the generator is retired while it is still
claimed.
Assisted-by: Claude
* itertools: claim a tee's position and its buffer as one step
`_tee::next` read `index` and moved it on afterwards, and
`_tee_dataobject::get_item` released `running` before the value it fetched
from the source was cached.
Two callers on one `_tee` then read the same index, hand out the same value
twice and advance past a value that was never cached, which leaves `index`
past `values.len()` and indexes the buffer out of bounds. Two callers at the
same index on separate tees each fetch a value from the source, and one of the
two is dropped without reaching a caller.
Both claims now cover the read and the update.
Assisted-by: Claude
* _asyncio: require InvalidStateError to be a type
new_invalid_state_error() called whatever `asyncio.exceptions.InvalidStateError`
names and unwrapped the downcast of the result, so a future asked for a result
it does not have panicked once that attribute was rebound to something that is
not an exception:
asyncio.InvalidStateError = lambda *args: 42
_asyncio.Future(loop=object()).result()
The type is looked up the way get_cancelled_error_type() looks its own up, and
raised with new_exception_msg; a lookup that does not produce an exception type
falls back to RuntimeError, as the other arms already did.
Assisted-by: Claude
|
||
|
Jeong, YunWon
|
fd7d107c75 |
marshal: round-trip code constants through the runtime bag (#8516)
* Keep abnormal marshal loop test aligned with CPython Assisted-by: OpenAI Codex:gpt-5.5 * marshal: retain runtime code constants while loading Decode code fields through the runtime MarshalBag so co_consts values that do not fit the compiler constant enum remain available to VM code wrappers while the compiler table receives shape placeholders. Assisted-by: OpenAI Codex:GPT-5.4 * marshal: write and read co_consts through the runtime bag serialize_code gains a serialize_code_with variant that writes each co_consts entry through a caller-supplied writer; serialize_code keeps the BorrowedConstant writer as its default. The VM writer passes its own write_object_depth, so a code constant that Literal holds but BorrowedConstant cannot describe reaches the stream instead of panicking in borrow_obj_constant, and a constant shared with the enclosing object takes an entry in the writer's reference table. PyMarshalBag implements constant_ref_from_value, bytes_from_value, str_from_value and tuple_elements_from_value, which deserialize_code_value_inner requires to decode code fields through the runtime bag. Assisted-by: Claude |
||
|
Jeong, YunWon
|
2ed082a51e |
Fix 33 reproduced fuzzing and static-review defects (#8514)
* mmap: guard move() against dest/src past the mapping size
An offset larger than the mapping made `size - dest` underflow, producing
an out-of-range slice index panic. Check the high side first, as write() does.
Assisted-by: Claude
* exceptions: keep ImportError name/path/name_from out of an empty instance dict
ImportError.__reduce__ unwrapped get_arg(0), which is None when the exception
was constructed with no positional argument, aborting on pickle.dumps(ImportError()).
Fall back to the full args tuple there, and expose name/path/name_from as class
attributes defaulting to None so a bare ImportError() reduces to (cls, ()).
Assisted-by: Claude
* asyncio: stop unwrapping the _current_tasks downcast in current_task()
_asyncio._current_tasks is a reassignable module attribute; the three sibling
functions already degrade gracefully when it is not a dict, current_task() did not.
Assisted-by: Claude
* asyncio: raise TypeError when FutureIter.throw()'s exception class returns a non-exception
exc_type.call() goes through a Python-controlled __new__, so its result can be
any object; the downcast was unwrapped. Report it as a TypeError instead.
Assisted-by: Claude
* sequence: use a fallible reservation for sequence repetition
The guard only rejects a repeat whose per-element size crosses MAX_MEMORY_SIZE,
so (1,) * (10**12) still reached Vec::with_capacity and aborted in the allocator.
Reserve fallibly and surface a MemoryError.
Assisted-by: Claude
* collections: reject an oversized deque repetition with MemoryError
deque * sys.maxsize reached the allocator and aborted with a capacity overflow.
Apply the MAX_MEMORY_SIZE guard the other sequences already carry.
Assisted-by: Claude
* itertools: raise OverflowError for an out-of-range r argument
combinations/combinations_with_replacement/permutations narrowed r with
to_usize().unwrap(), so r=2**64 panicked instead of raising.
Assisted-by: Claude
* math: stream the generic sumprod path instead of collecting both iterables
The big-int path collected each argument into a Vec before multiplying, so an
unbounded iterable exhausted memory. Advance both iterators in lockstep and
accumulate, reporting a length mismatch when only one is exhausted.
Assisted-by: Claude
* _imp: raise TypeError for a second positional argument to find_frozen
withdata is keyword-only and unimplemented; passing it positionally hit an
unimplemented!() and aborted.
Assisted-by: Claude
* _typing: check _idfunc arity before indexing args
_typing._idfunc() with no argument indexed args[0] out of bounds.
Assisted-by: Claude
* exceptions: require a sequence for the ExceptionGroup excs argument
The argument was collected before being validated, so an unbounded iterable
such as itertools.count() exhausted memory. Reject a non-sequence up front.
Assisted-by: Claude
* mmap: treat an inverted find/rfind range as empty
find(b"x", 5, 2) built a slice whose start exceeded its end and panicked.
Assisted-by: Claude
* collections: give deque and defaultdict a GC traverse
Neither type opted into traversal, so a reference cycle through a deque or a
defaultdict was never collected.
Assisted-by: Claude
* itertools: opt the iterator types into GC traverse
cycle and its siblings hold Python references but declared no traverse, so a
cycle built through one of them leaked.
Assisted-by: Claude
* _ctypes: mask an out-of-range int instead of panicking
c_char_p(2**64) and pointer item assignment called .expect() on the narrowing
conversion. Wrap the value to the target width, which is what the C
implementation stores.
Assisted-by: Claude
* hashlib: import _hashlib when a hash module is loaded
The hash object type is a static type owned by _hashlib; calling _md5.md5()
without _hashlib imported hit an uninitialized static type and panicked.
Assisted-by: Claude
* _csv: fall back to the built-in dialect defaults when none is registered
The dialect table is empty until csv.py registers 'excel', so _csv.reader([])
unwrapped a missing entry and panicked.
Assisted-by: Claude
* builtins: reject surrogates in compile()/eval() source instead of panicking
expect_str() panics on a str containing surrogates; convert with try_as_utf8
so eval(chr(0xd800)) raises.
Assisted-by: Claude
* _suggestions: require a list for _generate_suggestions candidates
The argument was collected before validation, so an unbounded iterable
exhausted memory.
Assisted-by: Claude
* lzma: size the filter chain before consuming it
filters= was collected into a Vec before the length check, so an unbounded
iterable exhausted memory. Take the length through the sequence protocol first.
Assisted-by: Claude
* classmethod: opt into GC traverse
staticmethod already declares traverse; classmethod did not, so a cycle through
the wrapped callable leaked.
Assisted-by: Claude
* posix: reject unbounded iterables in posix_spawn and setgroups
argv, setsigdef, setsigmask and setgroups bound an ArgIterable and collected it
before validating, so an infinite generator exhausted memory. Require a
list/tuple for argv, validate signals while streaming, and take setgroups
through the sequence protocol.
Assisted-by: Claude
* _ctypes: size Array slice assignment and _argtypes_ before collecting
Both eagerly collected their argument, so an unbounded iterable exhausted
memory before the length check ran.
Assisted-by: Claude
* sys: propagate the breakpointhook warning failure
warn() was unwrapped, so an unimportable $PYTHONBREAKPOINT under
-W error panicked instead of raising.
Assisted-by: Claude
* structseq: require the sequence argument when constructing a struct sequence
A no-argument construction produced an empty backing tuple, and reading any
named field then indexed out of bounds.
Assisted-by: Claude
* Guard the hash slot dispatch against unbounded recursion
`PyObject::hash` invoked the type's hash slot directly, so an element-wise
`__hash__` following a deeply nested object graph recursed one native frame
per level and overflowed the stack. Wrap the dispatch in `with_recursion`,
matching the repr and rich-compare dispatches in the same file, so
`hash(x)` on a nested tuple/GenericAlias/slice raises `RecursionError`.
Assisted-by: Claude
* genericalias: guard the __parameters__ walk against unbounded recursion
`make_parameters_from_slice` recursed into every list/tuple argument with
nothing counting the frames, so `list[L]` for a self-referential or deeply
nested `L` overflowed the native stack at subscript time. Wrap the descent
in `with_recursion`, which makes the walk fallible; `PyGenericAlias::new`,
`from_args` and `make_parameters` now return `PyResult` and every caller
propagates.
Assisted-by: Claude
* Fix the type confusion in PyAtomicRef's Debug impl
`PyAtomicRef<T>` stores a pointer to a `Py<T>`, as `Deref`, `load_raw`,
`swap` and `Drop` all read it, but `Debug` cast it to a bare `T` and
formatted the object header as payload bytes. For `PyFunction`, whose
`code: PyAtomicRef<PyCode>` has a pointer-chasing `Debug`, that
dereferenced header words and segfaulted.
Cast to `PyObject` instead, which is what `Drop` already does and which
also covers the `PyAtomicRef<PyObject>` and `PyAtomicRef<Option<T>>`
instantiations that have no `Py<T>`.
`_asyncio._enter_task` reached this through `{:?}` in its "Cannot enter
into task" message; format the two tasks with their Python repr, which is
what the message is meant to show.
Assisted-by: Claude
* _sre: disallow instantiating Match
`Match` inherited `object.__new__`, so `M.__new__(M)` produced an instance
whose `regs`/`string`/`pattern` were never filled in by a match run; the
mapping subscript path read them and dereferenced garbage. The type has no
public constructor, so mark it `DISALLOW_INSTANTIATION`, which makes
`M.__new__(M)` raise `TypeError: cannot create 're.Match' instances`.
Assisted-by: Claude
* utils: return the empty repr instead of asserting a non-empty collection
`collection_repr` took the first element with an `.expect()` justified by the
caller's preceding non-empty check. Another thread clearing the collection
between that check and the iteration made the iterator yield nothing and
panicked the worker. Fall back to the caller-supplied empty form, which is
the text those callers already produce for an empty collection.
Assisted-by: Claude
* itertools: advance cycle's index atomically
`cycle.__next__` did `fetch_add(1)` and then reset the index to 0 in a
separate store, so two threads replaying the saved items could both read an
index past the end of `saved` and panic on the slice access. Do the advance
and the wrap in one `fetch_update`.
Assisted-by: Claude
* _ctypes: reject a float argument to a foreign call without argtypes
`conv_param`, the conversion used when `argtypes` is not set, converted its
argument with `try_int`, which goes through `__int__` and so accepts a
float. `libc.strlen(1.5)` therefore passed 1 where the callee expects a
`char *` and the callee dereferenced it. Match `ConvParam`, which does a
`PyLong_Check` and converts nothing: take the branch only for an `int` (or
a subclass, so `True` still converts), and let a float fall through to
"Don't know how to convert parameter".
The branch below it converted a float to a C double, but `try_int` claimed
every float before it could run, so it was dead; `CArgValue::Double`
existed only for that branch and both go. Typed doubles are unaffected —
they travel as `CArgValue::Typed` with code 'd'.
Assisted-by: Claude
* Report the iterator itself from PyIter's traverse
`Traverse for PyIter<O>` delegated to the inherent `PyObject::traverse` of
the object it wraps, so it reported that iterator's referents instead of
the iterator. The iterator's own reference to those referents was then
never subtracted during the collector's reference-subtraction pass, the
referents kept a non-zero gc_refs, and every object reachable from them
was classified as a root.
Any cycle running through a type with a `PyIter` field therefore survived
collection: `map`, `filter`, `zip`, `enumerate`, `reversed` and the
`itertools` iterators all leaked, while the same cycle through a `list`,
`tuple` or `list_iterator` collected. Report the wrapped object, as the
`PyObjectRef`, `PyRef<T>` and `PyStackRef` impls do.
`itertools.tee` still leaks: its shared buffer is a `PyRc<PyItertoolsTeeData>`
rather than a Python object, so the collector cannot see through it.
Assisted-by: Claude
* Remove the obsolete expectedFailure on test_code_module.test_unicode_error
Compiling a source string containing a lone surrogate now raises
UnicodeEncodeError, so the test passes.
Assisted-by: Claude
* itertools: reserve the combination indices fallibly
`combinations` and `combinations_with_replacement` built their index vector
with an infallible allocation, so an `r` that passes the ssize_t check but
does not fit in memory aborted the process instead of raising MemoryError.
Assisted-by: Claude
* Apply the struct sequence constructor's dict argument
`structseq(sequence, dict)` discarded its second argument, so the hidden
fields past `n_sequence_fields` — `tm_zone`, `st_atime` and friends — were
always None when constructed directly or restored from a `(sequence, dict)`
pickle, and a non-dict second argument was accepted silently.
Take the dict, require it to be a dict, and fill the hidden slots the
sequence did not cover from it. A key that names a field the sequence
already supplied, or no field at all, is now a
"got duplicate or unexpected field name(s)" TypeError instead of being
dropped. Both arguments are bindable by name, as `sequence` and `dict`.
`os.stat_result` and `os.statvfs_result` did not accept a second argument
at all; they and `time.struct_time` now share the parsing.
Assisted-by: Claude
* _imp: report the argument count in find_frozen's arity error
Assisted-by: Claude
* Add regression tests for the reproduced crashers
One case per catalog entry, each asserting the behavior the fix produces:
recursion guards, the memory-unsafety sites, the overflow and unbounded
allocation guards, the eager-collection rejections, the unwrap sites, and
the cycles the collector now breaks. Every expected value was checked
against CPython 3.14.
Assisted-by: Claude
* Tolerate a changed-size RuntimeError in the set repr stress test
The mutator and reader threads race on purpose; a "changed size during
iteration" RuntimeError is a valid outcome of that race and should not fail
the test. The panic it guards against is not.
Assisted-by: Claude
* Move the crash regression tests into per-module snippets
crash_regressions.py collected every reproduced crasher in one file. Split it
into the snippet for the module each case exercises, and add stdlib_gc.py,
stdlib_asyncio.py, stdlib_lzma.py, stdlib_threading_set_repr.py and
stdlib_threading_itertools_cycle.py for the cases with no existing home.
The suite runs every snippet under the host CPython too, so the checks only
RustPython raises are guarded by sys.implementation.name: the hash and
__parameters__ recursion depth, and the deque repeat overflow. The float
ctypes argument accepts either TypeError or ctypes.ArgumentError.
Assisted-by: Claude
|
||
|
Jeongseop Lim
|
c9a62449f0 |
Reject extra positional arguments to float() (#8509)
* Reject extra positional arguments to float()
PyFloat::slot_new's exact-float fast path tested args.args.first(), which
succeeds for any number of positional arguments, so float(1.5, True)
returned the first one and never reached args.bind(vm). CPython's clinic
generated float_new runs _PyArg_CheckPositional("float", nargs, 0, 1)
before it fetches the first argument.
Add args.args.len() == 1 to the condition, as PyInt::slot_new,
PyStr::slot_new and PyComplex::slot_new already do; the extra argument
then falls through to args.bind(vm) and raises TypeError.
Assisted-by: Claude Code:claude-opus-5
* Bind float() arguments before the fast path
The exact-float fast path repeated float()'s argument count inline, which
duplicates what FromArgs::arity already knows. Bind first and match on the
resulting OptionalArg instead, so the positional and keyword rules stay in
one place and the bound value is reused by py_new.
Assisted-by: Claude Code:claude-opus-5
|
||
|
Jeongseop Lim
|
0b150ca569 |
Reject a fifth argument to property() (#8510)
PropertyArgs declared a fifth positional-or-keyword field, name, so the
derived arity was 0..=5 and property(None, None, None, None, None) built
a property object with the fifth argument stored in the __name__ slot.
CPython's property.__init__ is Argument Clinic generated with maxpos = 4
and the keyword list {fget, fset, fdel, doc}, so it rejects both that call
and property(name='x').
Drop the field. Nothing passed it: clone_property_with always supplied
None and copies the name separately, and the name slot is still filled by
__set_name__ and the __name__ setter.
Assisted-by: Claude Code:claude-opus-5
|
||
|
Jeongseop Lim
|
901d8e186e |
Normalize round() result to an exact int (#8511)
int.__round__ returned the receiver unchanged on the no-rounding path, so round() on an int subclass handed back an instance of that subclass: round(True) was True, not 1. CPython routes both no-op paths (ndigits is None, and ndigits >= 0) through long_long(), which reuses the reference only when the value is an exact int and otherwise copies it into a fresh one. Return zelf.__int__(vm).into_pyref(), which goes through into_exact_or() and is the same normalization the neighbouring __trunc__, __floor__ and __ceil__ already perform. Assisted-by: Claude Code:claude-opus-5 |
||
|
Jeong, YunWon
|
525ba8cf0a |
codegen: preserve symbol tables across copied finally bodies (#8507)
Assisted-by: OpenAI Codex:gpt-5 |
||
|
Sumi Jeong
|
cf688760ca |
Fix locals()/vars() corrupting/leaking __conditional_annotations__ outside function scope (#8467)
CPython's _PyFrame_GetLocals only syncs fastlocals/cells into locals() for function scope; module/class scope just returns the namespace dict as-is. sync_visible_locals_to_mapping did this for every scope, so the implicit __conditional_annotations__ cell (used for PEP 649/749 deferred annotations) broke two ways: at module scope its cell is always empty, so syncing overwrote the dict's real value with None, causing NameError on the next annotated statement; at class scope its cell is the only real value, so syncing leaked it into locals()/dir(), unlike CPython. Skip cell/free slots outside function scope to match CPython, and add a regression snippet. Closes #8379 Assisted-by: Claude Code:claude-sonnet-5 |
||
|
강동안
|
4530ecd49c |
Specialize list.sort() comparisons for homogeneous lists (#8464)
* Use type-specialized comparators for homogeneous list sorts Scan the sort keys once before sorting; when every element is exactly str, int, or float, compare wtf8 bytes / BigInt / f64 directly instead of going through rich_compare_bool dispatch, mirroring CPython's pre-sort check in listsort.c (unsafe_latin_compare and friends). Subclasses and mixed-type lists keep the generic __lt__ path. * Add edge-case tests for specialized list sorts * Cache the richcompare slot for homogeneous list sorts * Specialize tuple sorts on their first elements * Apply ruff formatting to the new list sort tests * Deduplicate the reverse swap across sort dispatch arms |
||
|
Hanif AriffinandJeong YunWon
|
74ae2d4711 |
Fix bytes.isspace (#5655)
* Fix implementation of bytes.isspace to match the implementation in CPython. * Fix implementation of bytes.istitle so that test_bigmem now passes. Also added some extra_test to capture this. * Fix some clippy issues Signed-off-by: Hanif Ariffin <hanif.ariffin.4326@gmail.com> * bytes.istitle returns false when the bytes is empty. * Fix some edges of istitle with non-alphanumeric characters * Disable a line because python 3.12 errors on bad escape sequence * Match CPython's istitle a little closer * Fix clippy warning * Rewrite bytes.istitle along _Py_bytes_istitle The lookahead-based check accepted b"Not--a Titlecase String", which test_bytes covers. Track whether the previous byte was cased instead. isspace compares bytes directly rather than converting to char. Restore the b"omkmok\Xaa" assert: 3.14 reports an invalid escape sequence as a SyntaxWarning, not an error. Assisted-by: Claude --------- Signed-off-by: Hanif Ariffin <hanif.ariffin.4326@gmail.com> Co-authored-by: Jeong YunWon <jeong@youknowone.org> |
||
|
|
1306b71a65 |
__length_hint__ (#6636)
* Initial plan * fix: prevent iterator length_hint deadlock Co-authored-by: youknowone <69878+youknowone@users.noreply.github.com> * fix * Auto-format: cargo fmt --all --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> |
||
|
이광효
|
16bb018d31 |
csv: port CPython reader state machine (#8381)
* csv: port CPython reader state machine Replace the csv-core reader and per-item quote scanner with one Rust implementation of CPython's nine-state reader parser. Keep parser state across iterator items and distinguish virtual item boundaries from true iterator exhaustion. Centralize field completion so quote provenance, empty-field None conversion, float conversion, strict parsing, field limits, blank rows, and escaped or quoted newlines share one path. Apply the existing reentrant-iterator generation check to every iterator item consumed by a record. Also accept empty ASCII line terminators like CPython, report a missing csv-core writer sentinel as _csv.Error instead of panicking, and name the FSM entry point process_parser_input. Remove the expected-failure markers from the eleven reader tests that now pass. Assisted-by: Codex:gpt-5.6-sol * csv: test CRLF boundaries with custom lineterminator |
||
|
조진명
|
6906aa1747 |
csv: support multi-character lineterminator in writer (#8328)
* csv: support multi-character lineterminator in writer Store the dialect line terminator as an owned String instead of the single-byte csv_core::Terminator, dropping PyDialect's Copy derive. The manual writer paths emit the full terminator; the csv-core-backed QUOTE_ALL/QUOTE_NONNUMERIC paths emit a sentinel byte (preserving csv-core's quote/empty-record bookkeeping) and append the real terminator. field_needs_quotes/escape now quote a field containing any terminator byte. The reader ignores lineterminator and always uses CRLF, matching CPython and avoiding mid-UTF-8 record splits. * csv: unmark now-passing test_write_lineterminator * csv: address review feedback - Remove the redundant per-branch sentinel terminator setup in to_writer; the unconditional terminator call after the match is the single source. - Reword the empty-lineterminator errors to "must not be empty" (the constraint is non-empty, not single-character) on both entry points. - Promote the sentinel invariant check in writerow from debug_assert_eq! to assert_eq! so it also guards release builds. - Add a snippet case for a field containing a line-break byte to ensure the csv-core path drops only the trailing terminator. * csv: reject non-ASCII lineterminator The writer decides what to quote and escape by comparing raw bytes, so a non-ASCII terminator quoted a field that merely shared a UTF-8 lead byte, and QUOTE_NONE escaped individual bytes of the terminator and then failed to decode the record back to a string. Reject non-ASCII terminators, including lone surrogates, as csv.Error when the dialect is parsed, and leave code-point-wise handling to a follow-up (#8310). The non-string error message now matches CPython as well. |
||
|
Jiwoo Ahn
|
bf8bca8039 |
wasi: support monotonic / perf time (#8385)
Assisted-by: Codex:GPT-5.6 Sol Signed-off-by: Jiwoo Ahn <ikwydls1314@gmail.com> |
||
|
Kwak Byoung Min
|
942ddae1c5 |
Add seekable() to mmap.mmap (#8405)
* Allow mmap-backed readers to identify seek support mmap already supports seek() and tell(); expose seekable() to match CPython. Confidence: high Scope-risk: narrow Tested: test_mmap, stdlib_mmap snippet, clippy, prek, extra_tests Not-tested: full workspace test blocked by local rustpython-capi SIGSEGV Assisted-by: Codex:gpt-5.6-sol * Keep mmap regression test lint-clean CI ruff check removes the redundant blank line and treats the auto-fix as a failure. Constraint: CI runs prek with ruff check Confidence: high Scope-risk: narrow Tested: prek run --all-files; cargo run -- extra_tests/snippets/stdlib_mmap.py Assisted-by: Codex:gpt-5.6-sol * Keep mmap seekability method conventional The method is invoked at runtime through the Python wrapper, so const qualification provides no benefit. Constraint: Python methods execute at runtime Confidence: high Scope-risk: narrow Tested: prek run --all-files; cargo run -- extra_tests/snippets/stdlib_mmap.py Assisted-by: Codex:gpt-5.6-sol |
||
|
Seonghun An
|
bf4a2b110c |
Fix zero padding for string format specs (#8407)
* Fix zero padding for string format specs Assisted-by: Codex:gpt-5 * Add string alignment format regression test Assisted-by: Codex:gpt-5 |
||
|
sijun-yang
|
92fe3fd668 |
Propagate method wrapper repr errors (#8371)
Assisted-by: Codex:gpt-5.6-sol |
||
|
이광효
|
4a80f2e5b9 |
ast: fix identifier interning (#8361)
* Fix AST identifier interning Ensure parsed name and function identifier fields use interned strings, matching CPython behavior. Assisted-by: Codex:gpt-5.6-sol * Apply PEP 8 import ordering |
||
|
Jeong, YunWon
|
8d7d9d931c |
Use dict keys-version stamps and entry-index hints in LOAD_ATTR/STORE_ATTR specializations (#8350)
* Use dict keys-version stamps and entry-index hints in attr specializations Add a keys-version stamp to Dict: a globally unique u32 assigned lazily and reset on any key-set change (new key, deletion, clear). Value-only updates keep the stamp. - LoadAttrMethodWithValues / LoadAttrNondescriptorWithValues: cache the instance dict's stamp in the pointer cache and skip the shadow probe while the stamp matches. - LoadAttrWithHint: cache the entry index at specialization time; a hit is an identity check on the entry key instead of a hash probe, with self-refresh on miss. - StoreAttrWithHint / StoreAttrInstanceValue: replace values through the cached entry index via set_item_with_hint. Fixes youknowone#42 Assisted-by: Claude * Share keys-version stamps between dicts with identical layouts Derive the keys-version stamp from a shape — the ordered interned-key sequence of a hole-free dict — instead of always allocating a unique stamp. Dicts with identical layouts (instances of the same class built by the same __init__) now carry equal stamps, so a LOAD_ATTR cache entry populated by one instance skips the shadow probe for every instance sharing the layout. Shapes are held in a fixed-size lock-free table keyed by interned key addresses; dicts with holes, non-interned keys, or more than 32 keys fall back to dict-unique stamps. Assisted-by: Claude * Address clippy and review feedback for keys-version stamps - Use BuildHasher::hash_one for shape hashing (clippy manual_hash_one) - Fix set_item_with_hint doc: a refreshed hint is returned on any hint miss, not only when the hinted slot was vacant - Route both holey-dict instances through a single LOAD_ATTR cache site in the snippet test Assisted-by: Claude * Restore try_read_cached_descriptor doc comment to its function The doc block and #[inline] were left attached to store_attr_dict_hinted when it was inserted above try_read_cached_descriptor. Assisted-by: Claude * Specialize LoadAttrWithHint even when the entry index exceeds u16 hint_for_key returns None both for an absent key and for a present key whose entry index does not fit in u16, so very large instance dicts stopped specializing entirely. Use get_item_opt_refresh_hint to decide presence, degrading an unrepresentable hint to 0: the handler then keeps taking its full-probe fallback path. Assisted-by: Claude |
||
|
이광효
|
d16fd12bc7 |
csv: handle empty fields with skipinitialspace (#8304)
* csv: handle empty fields with skipinitialspace skipinitialspace preprocessed records by trimming both ends of every split field. Fields containing only spaces could therefore produce an invalid slice, while trailing spaces were removed even though CPython preserves them. Trim only field prefixes in the csv-core path and make the remaining all-space trimming helper return an empty slice safely. QUOTE_NOTNULL and QUOTE_STRINGS add another distinction: an unquoted empty field becomes None, but a quoted empty field remains an empty string. Both forms have identical decoded bytes, so extend the existing QUOTE_NONE custom reader into one shared path that records whether each field started quoted. Use that metadata only for the null conversion while retaining QUOTE_NONE escape behavior. Unmark Test_Csv.test_read_skipinitialspace now that its standard, QUOTE_NOTNULL, and QUOTE_STRINGS cases pass. This keeps the existing one-item reader lifecycle and does not add the larger strict or multiline parser state machine. Assisted-by: Codex:gpt-5-sol * csv: make skipinitialspace quote-aware The csv-core preprocessing path split each raw iterator item on every delimiter before trimming field prefixes. Delimiters inside quoted fields were therefore treated as separators, so spaces after them could be removed before csv-core parsed the record. Extract the quote, escape, delimiter, and field-start transitions from read_quote_record() into a small per-item scanner. Reuse its events in the skipinitialspace preprocessor and discard only InitialSpace events while copying all other source bytes unchanged. This keeps the custom quote-mode reader and preprocessing logic aligned without introducing persistent multiline or strict parser state. Remove the delimiter cache that was only needed by the old split-and-join path and add a regression for spaces following a delimiter inside a quoted field. Assisted-by: Codex:gpt-5-sol |
||
|
Jeong, YunWon
|
4c9b5d8854 |
ctypes: fix Windows pip truststore (#8314)
* ctypes: preserve result when errcheck returns args * ctypes: cover Windows pip truststore * capi: implement PyEval_SaveThread * capi: merge saved thread state into PyThreadState |
||
|
조진명
|
875400413f |
csv: quote embedded CR/LF under QUOTE_MINIMAL (#8315)
* csv: quote embedded CR/LF under QUOTE_MINIMAL with custom lineterminator QUOTE_MINIMAL rows were serialized by csv-core, which only quotes bytes registered for the configured terminator, so fields containing '\r' or '\n' were left unquoted when the lineterminator was not CR/LF (e.g. '!' or '\0'). Route QUOTE_MINIMAL through a hand-written path that uses field_needs_quotes, which always treats '\r'/'\n' as needing quotes. A single empty field is quoted and an empty row emits only the terminator, matching CPython. QUOTE_ALL / QUOTE_NONNUMERIC still use csv-core. Assisted-by: Claude Code:claude-opus-4-8 * csv: unmark now-passing writer tests test_write_iterable and test_write_empty_fields pass with the QUOTE_MINIMAL writer fix (empty rows no longer emit an empty field). Assisted-by: Claude Code:claude-opus-4-8 |
||
|
Seonghun An
|
d509daf5e1 |
Validate TextIOWrapper chunk size (#8265)
* Validate TextIOWrapper chunk size Match CPython for non-positive, oversized, and index-compatible values. Assisted-by: Codex:GPT-5 * Use explicit AssertionError in io snippet Assisted-by: Codex:GPT-5 * Report original type in chunk size overflow Assisted-by: Codex:GPT-5 * Chain io snippet assertion error Assisted-by: Codex:GPT-5 * Match CPython chunk size setter behavior Assisted-by: Codex:GPT-5 * Format io snippet assertion Assisted-by: Codex:GPT-5.6 * Match CPython type name truncation Assisted-by: Codex:GPT-5 * Format non-ASCII chunk size test Assisted-by: Codex:GPT-5 |
||
|
Sumi JeongandJeong, YunWon
|
e548114f5b |
_queue: check for pending signals during blocking get() (#8252)
* _queue: check for pending signals during blocking get() Semaphore::acquire() waited on the condvar in a single uninterrupted call, so a Python-level signal handler never ran until the wait itself returned. check_signals() is only invoked from the bytecode dispatch loop, and this native wait never passed through it -- CPython chunks its equivalent blocking wait for the same reason. Cap each wait at a short interval and call check_signals() between chunks, only giving up once the caller's real deadline has actually elapsed. The mutex guard must be dropped before check_signals() runs: a signal handler that calls back into the same queue on the same thread would otherwise try to relock a mutex it's still holding and deadlock against itself. Add a regression snippet that arms a SIGALRM well before a helper thread unblocks the queue, and asserts the handler ran near the alarm instead of only once get() returned. Refs #8250 Assisted-by: Claude * _queue: correct SIGNAL_CHECK_INTERVAL doc comment The old comment claimed this mirrors CPython's PyThread_acquire_lock_timed, but CPython doesn't chunk its wait either -- it makes one sem_timedwait call and relies on the OS reporting EINTR. parking_lot's Condvar has no equivalent signal to surface, so polling is a workaround for that gap, not a mirror of CPython's approach. Assisted-by: Claude * Apply suggestions from code review Co-authored-by: Jeong, YunWon <69878+youknowone@users.noreply.github.com> * _queue: fix comment indentation (cargo fmt) Assisted-by: Claude --------- Co-authored-by: Jeong, YunWon <69878+youknowone@users.noreply.github.com> |
||
|
Kwak Byoung Min
|
a9c2c529b1 |
Forward pyexpat lexical events (#8264)
* Restore pyexpat lexical callback compatibility Forward processing instructions, comments, and CDATA boundaries already emitted by xml-rs so registered Python handlers observe CPython-compatible events. Constraint: Reuse existing xml-rs events without changing the dependency Confidence: high Scope-risk: narrow Tested: prek; test_pyexpat; test_sax; 402 extra tests; workspace tests; clippy Assisted-by: Codex:gpt-5.6-sol * Recognize XML compatibility restored by lexical events Remove stale expected-failure markers for CPython XML tests now satisfied by pyexpat lexical-event forwarding, and simplify parser event error propagation. Constraint: CPython test edits are limited to removing stale expectedFailure markers Confidence: high Scope-risk: narrow Directive: Keep parser event dispatch aligned with pyexpat handler behavior Tested: user-verified full suite; prek run --all-files; cargo run --release -- -m test test_pyexpat test_xml_etree test_xml_etree_c test_minidom test_sax Assisted-by: Codex:gpt-5.6-sol |
||
|
이광효
|
67ddc1aa9e |
csv: fix csv escape fieldsep (#8260)
* csv: apply dialect escapechar when reading ReaderBuilder only received escapechar when it was passed directly as a keyword argument. A dialect object's or registered dialect's escapechar was visible through reader.dialect but not used by csv_core. Pass escapechar through for named, object, and default dialect paths. This makes escaped delimiters inside quoted fields follow the configured dialect, for example '"abc\,def"' reads as 'abc,def'. Unmark TestQuotedEscapedExcel.test_read_escape_fieldsep. Assisted-by: Codex * csv: handle escaped delimiters with QUOTE_NONE csv_core applies escapechar while reading quoted fields, but its unquoted field state treats an escaped delimiter as a delimiter. Consequently, QUOTE_NONE with escapechar split 'abc\,def' into 'abc\' and 'def'. Add a small QUOTE_NONE record parser for dialects with escapechar. It keeps the byte following an escape character as field data, so escaped delimiters do not end a field while ordinary delimiters and record terminators retain their usual meaning. Unmark TestEscapedExcel.test_read_escape_fieldsep. Assisted-by: Codex * csv: respect skipinitialspace with QUOTE_NONE The QUOTE_NONE parser returned before the normal reader's whitespace preprocessing, so it ignored skipinitialspace when escapechar was set. Track whether parsing has just followed a delimiter and skip only ordinary spaces in that position. Escaped spaces remain field data. Add a stdlib_csv snippet regression test for QUOTE_NONE with escapechar and skipinitialspace. Assisted-by: Codex:gpt-5.6-terra * csv: deduplicate reader dialect configuration to_reader configured ReaderBuilder separately for named, object, and default dialects even though the configuration was identical. Resolve the PyDialect once, then apply delimiter, double_quote, escapechar, and quotechar in one place. Preserve the existing unregistered-name fallback and required excel dialect lookup behavior. Assisted-by: Codex:gpt-5.6-terra |
||
|
Jeong, YunWon
|
238bed66fb |
literal: round-half-even tie for normal-range float/complex repr (#8243)
to_string's normal-magnitude branch and complex::component_to_string returned Rust's shortest formatting directly, which can pick the odd-digit neighbour of a rounding tie where repr() picks the even one (e.g. bits 0x42e26687db6b9b04 formats as 161852602146008.13 vs repr's ...08.12). Route both through prefer_cpython_tie_repr, generalized to fixed-notation strings. Add unit tests and float/complex repr snippet tests. Assisted-by: Claude |
||
|
Jeong, YunWon
|
150990e866 |
literal: reject internal whitespace in complex::parse_str (#8242)
parse_str split the token then parsed each part with float::parse_str, which tolerates surrounding whitespace, so "1 +2j" parsed as (1+2j). Reject whitespace inside the token after stripping optional parentheses. Add unit tests and complex() snippet tests. Assisted-by: Claude |
||
|
Kwak Byoung Min
|
5f9139a3f8 |
Support xml.sax pyexpat setup APIs (#8238)
* Let xml.sax initialize pyexpat-backed parsers RustPython already exposes an Expat-like XML parser, but xml.sax expects a few setup APIs before parsing. Add compatible parser setup methods and constants without pretending to implement full Expat external entity behavior. Constraint: Lib/test changes only remove expectedFailure markers for tests that now pass; test bodies and assertions are unchanged. Rejected: Implementing full Expat parameter entity parsing | xml-rs does not expose that configuration and the immediate failure is the missing setup API. Confidence: high Scope-risk: narrow Directive: Keep SetParamEntityParsing as a compatibility shim unless the backend grows real Expat-style entity parsing support. Tested: PATH=/tmp/pyshim:$PATH prek run --all-files Tested: cargo run --quiet -- extra_tests/snippets/stdlib_xml.py Tested: cargo run --release -- -m test test_sax Tested: cargo run --release -- -m test test_pyexpat Tested: cargo test --workspace --exclude rustpython-capi --exclude rustpython_wasm --exclude rustpython-compiler-source --exclude rustpython-venvlauncher --features threading --no-default-features --features stdlib,importlib,stdio,encodings,sqlite,ssl-rustls-aws-lc,host_env && (cd crates/capi && cargo test) Tested: PYO3_CONFIG_FILE=$PWD/crates/capi/pyo3-rustpython.config cargo test --workspace --exclude rustpython_wasm --exclude rustpython-venvlauncher Tested: cargo build --release --features sqlite && cd extra_tests && /tmp/rustpython-extra-tests-venv/bin/python -m pytest -v Tested: cargo clippy --workspace --exclude rustpython-capi --exclude rustpython_wasm --exclude rustpython-compiler-source --exclude rustpython-venvlauncher --features threading --no-default-features --features stdlib,importlib,stdio,encodings,sqlite,ssl-rustls-aws-lc,host_env Assisted-by: Codex:gpt-5.5 * Keep XML follow-up tests aligned with passing behavior The pyexpat setup API compatibility fix also makes pulldom's basic parse smoke test pass, so keep RustPython expected-failure markers aligned with current behavior and apply the ruff import-spacing fix. Constraint: Test body and assertions remain unchanged. Confidence: high Scope-risk: narrow Tested: cargo run --release -- -m test test_pulldom -v Tested: cargo run --release -- -m test test_sax Tested: cargo run --release -- -m test test_pyexpat Tested: cargo run --quiet -- extra_tests/snippets/stdlib_xml.py Tested: PATH=/tmp/pyshim:/Users/kbm/.codex/tmp/arg0/codex-arg0ejsdGi:/Users/kbm/.omx-runs/run-20260703105454-4fab/.omx/runtime/bin:/Users/kbm/.nvm/versions/node/v24.15.0/bin:/Users/kbm/Library/Java/JavaVirtualMachines/temurin-21.0.9/Contents/Home/bin:/Users/kbm/.antigravity/antigravity/bin:/opt/homebrew/opt/mysql-client/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Library/Apple/usr/bin:/var/folders/vq/1mhmtsbs2kj_03tsx74kd1mm0000gn/T//cmux-cli-shims/93BA47BD-630C-4ED3-8573-0FE6ABFAE824:/Applications/cmux.app/Contents/Resources/bin:/Users/kbm/.nvm/versions/node/v24.15.0/bin:/Users/kbm/Library/Java/JavaVirtualMachines/temurin-21.0.9/Contents/Home/bin:/Users/kbm/.antigravity/antigravity/bin:/opt/homebrew/opt/mysql-client/bin:/var/folders/vq/1mhmtsbs2kj_03tsx74kd1mm0000gn/T/cmux-cli-shims/93BA47BD-630C-4ED3-8573-0FE6ABFAE824:/Applications/Visual Studio Code.app/Contents/Resources/app/bin:/Applications/Visual Studio Code.app/Contents/Resources/app/bin:/Applications/Visual Studio Code.app/Contents/Resources/app/bin prek run --all-files Tested: cargo build --release --features sqlite && cd extra_tests && /tmp/rustpython-extra-tests-venv/bin/python -m pytest -v Tested: PYO3_CONFIG_FILE=/Users/kbm/IdeaProjects/RustPython/crates/capi/pyo3-rustpython.config cargo test --workspace --exclude rustpython_wasm --exclude rustpython-venvlauncher Tested: cargo clippy --workspace --exclude rustpython-capi --exclude rustpython_wasm --exclude rustpython-compiler-source --exclude rustpython-venvlauncher --features threading --no-default-features --features stdlib,importlib,stdio,encodings,sqlite,ssl-rustls-aws-lc,host_env Assisted-by: Codex:gpt-5.5 |