mirror of
https://github.com/RustPython/RustPython.git
synced 2026-10-28 18:59:04 +08:00
copilot/fix-isalnum-function-behavior
1184
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Jeong, YunWon
|
7aedaf919e | Merge branch 'main' into copilot/fix-isalnum-function-behavior | ||
|
Kwak Byoung Min
|
62f4c9895a |
Add StringIO newline tracking (#8539)
* Report observed StringIO newline types Constraint: Match CPython StringIO newline reporting without changing configured newline translation. Rejected: Separate newline-tracking representation | reuse the existing SeenNewline bitflags. Confidence: high Scope-risk: narrow Directive: Keep observed newline state separate from the configured newline mode. Tested: prek run --all-files; test_memoryio; cargo clippy -p rustpython-vm --lib -- -D warnings; workspace tests excluding the macOS C-API baseline SIGSEGV. Not-tested: Full macOS workspace suite is blocked by the existing rustpython-capi SIGSEGV; the Linux suite is running. Assisted-by: Codex:gpt-5.6-sol * Preserve StringIO state boundaries Reset observed newline state when StringIO is reinitialized and keep the property subject to closed-stream checks. Constraint: Match CPython StringIO reinitialization and closed-stream behavior. Rejected: Returning stale or cached newline state | violates StringIO lifecycle semantics. Confidence: high Scope-risk: narrow Directive: Keep observed newline state tied to the current open StringIO contents. Tested: cargo test --workspace --exclude rustpython_wasm --exclude rustpython-venvlauncher; cargo run --release -- extra_tests/snippets/stdlib_io_stringio.py; cargo run --release -- -m test test_memoryio; prek run --all-files; cargo clippy -p rustpython-vm --lib -- -D warnings. Assisted-by: Codex:gpt-5.6-sol |
||
|
sijun-yang
|
cd935b8ef6 |
Expose symbol scopes through _symtable (#8538)
Pack symbol scopes with definition flags in _symtable.symbols for Lib/symtable.py. Assisted-by: Codex:gpt-5 |
||
|
Jeong, YunWon
|
819b4b86fd |
Fix malformed Unicode error constructors (#8497)
* Fix malformed Unicode error constructors Replace message-only Unicode decode and encode errors with fully initialized exceptions, preserve source objects and failure ranges, and remove the obsolete constructors. Assisted-by: Codex:gpt-5 * Fix Windows Unicode CI regressions Assisted-by: Codex:gpt-5 |
||
|
Jeong, YunWon
|
a78d0eb577 |
Fix crashes found hunting the last open fuzzing record (#8524)
* specialize: check the member descriptor's type before caching its slot offset
The LOAD_ATTR/STORE_ATTR specializations cached the slot offset of any member
descriptor found on the owner's type and then guarded the specialized
instruction on the type version alone, while descr_get()/descr_set() check on
every access that the instance belongs to the type the descriptor was defined
for. A descriptor taken from a wider class and bound to a narrower one read
past the instance's slot array once the cache warmed up:
class Big:
__slots__ = ("a0", "a1", "a2", "a3", "a4", "a5", "a6", "a7")
class Narrow:
__slots__ = ("z",)
Narrow.x = Big.__dict__["a7"]
o = Narrow()
for _ in range(1000):
try: o.x
except TypeError: pass
# index out of bounds: the len is 1 but the index is 7 (object/core.rs)
A class with no slots at all reached the ext_ref().unwrap() on the same line.
Assisted-by: Claude
* socket: reserve recv()'s buffer fallibly
recv() and recvfrom() handed the caller's bufsize straight to
Vec::with_capacity, so an unreachable size aborted the process through
handle_alloc_error before any syscall was made:
socket.socket().recv(2**62)
# memory allocation of 4611686018427387904 bytes failed -> SIGABRT
try_reserve_exact reports MemoryError instead, which is what CPython raises.
Assisted-by: Claude
* types: count the __call__ and __get__ slot dispatches as recursion
Both wrappers re-enter Python without pushing a frame, so nothing counted the
nesting when the special method named the object it was looked up on:
class C: pass
c = C(); C.__call__ = c
c() # native stack overflow, SIGSEGV
class D: pass
d = D(); D.__get__ = d; D.x = d
d.x # the same, through descr_get
with_recursion around the two dispatches raises RecursionError instead, the
way Py_EnterRecursiveCall bounds a tp_call dispatch. It costs about 5% on a
__call__ dispatch and 3% on a __get__ dispatch through these wrappers.
Assisted-by: Claude
* typevar: show a ParamSpecArgs origin by its repr
ParamSpecArgs and ParamSpecKwargs fell back to a Rust `{:?}` of __origin__ when
it had no __name__. That walks the object graph natively through Debug for
PyInner, where no recursion guard sits, so a single repr() of a deeply nested
chain overflowed the native stack:
a = object()
for _ in range(30000):
a = typing.ParamSpecArgs(a)
repr(a) # SIGSEGV
The origin is formatted with its repr now, which is guarded, and a ParamSpec
origin is recognized by its type rather than by carrying a __name__.
Assisted-by: Claude
* Do not hold a lock across a call back into Python
Three places kept a lock while running code that can reach the same object, so
a callback that touched it wedged the process:
_asyncio.future_add_to_awaited_by(fut, waiter) # waiter.__hash__ adds again
select.select(elements, [], [], 0) # fileno() clears `elements`
select.poll().poll(1000) # SIGALRM handler registers
The future's awaited-by field is read and written under its lock but the set is
built outside it, the list extraction re-reads the list on each step the way
map_iterable_object() does, and poll() waits on a copy of its descriptors. All
three ran forever before and now finish the way they do on CPython.
Assisted-by: Claude
* Validate memoryview.cast() arguments and export negative strides correctly
cast() accepted any struct format and any shape element. A zero-size
format ('0s') and a 0 in the shape both reached a division by zero;
cast() now takes only a native single character format, optionally
'@'-prefixed, and shape elements that are ints greater than zero.
A view with a negative stride starts at its last item, so the bytes it
exported began there and its own offsets walked off the front of them.
Such a view now exports the whole underlying buffer with `start` folded
into the descriptor's offsets, and zip_eq() hands over a whole run only
when both sides are contiguous in the last dimension.
Assisted-by: Claude
Assisted-by: Codex:GPT-5
* Charge native recursion to the stack, not to the frame limit
with_recursion() checked the limit sys.setrecursionlimit() sets and
incremented the same counter that pushing a frame does, so a guard on a
native dispatch spent what Python code had left to call with, and did so
where sys._getframe() cannot see it: test.support.get_recursion_available()
reported frames that were no longer there. Py_EnterRecursiveCall bounds
the native stack instead, which is a separate budget, and the C stack
check with_recursion already performs is that bound.
The snippets pinning the guarded paths nest deep enough to reach the
stack rather than the frame limit.
Assisted-by: Claude
* Report a size that cannot be allocated instead of aborting on it
A size taken from Python went straight into an infallible allocation in
several places, so the process aborted through handle_alloc_error before
any exception could be raised:
- str/bytes/bytearray center(), ljust(), rjust() and zfill() reserved
the padded result for the caller's width
- expandtabs() built its runs of spaces from a tabsize of any width;
the argument is a C int, and a wider one does not fit
- Buffered{Reader,Writer,Random} allocated buffer_size, and read(),
read1() and FileIO.read() their read size
- bytes(n) and bytearray(n) allocated n
- pbkdf2_hmac() allocated the derived key length, which is a C int
Each of these now reports MemoryError, or OverflowError where the
argument does not fit the type it is declared with.
new_zeroed_bytes() leaves the zeroing to the allocator, so a large
request costs the pages that are written to rather than all of them.
Assisted-by: Claude
* marshal: answer allow_code where a code object is written or read
allow_code was answered by walking the whole result a second time, with
no depth counter and no record of what it had already seen, so a value
that referred back to itself or nested deeply enough ran off the native
stack. w_object() and r_object() answer it where the code object is,
inside the walk that already bounds its depth and resolves references.
A container length is read the way r_long() reads one: it is signed, so
a length with the top bit set is out of range rather than four billion
items to reserve room for.
load() no longer holds a borrow of the buffer read() returned across the
seek() it makes afterwards.
Assisted-by: Claude
* Do not lock an object while running code that can reach it
Several places held a lock or a borrow of an object across a call back
into Python, so a callback that touched the same object waited on a lock
its own caller was holding:
- memoryview slice assignment read a source overlapping the destination,
and __setitem__ converted the value while holding the write borrow
- BytesIO.readinto() read into a buffer viewing the same BytesIO
- array.__setitem__ converted the value under the array's write lock,
and mmap.write() read a source viewing the same map
- bytearray.join() and bytearray.__mod__ drove Python with the
bytearray borrowed
- array and bytearray answered "is this resizable" after taking the
write lock, though an export is exactly a borrow someone else holds
A TextIOWrapper cookie now has to name a position inside what was
decoded in characters as well as in bytes; only the byte offset was
checked, and the character count is what read() and tell() index with.
Assisted-by: Claude
Assisted-by: Codex:GPT-5
* Stop asserting a pbkdf2 message that depends on the width of a C long
The snippet asserted "key length is too great.", which pbkdf2_hmac()
only reaches once the length has been converted; where a C long is
narrower than the length asked for, the conversion fails first and says
so instead. Both are OverflowError, which is what the case is about.
test_support.test_get_recursion_depth passes now that a native recursion
guard no longer spends frames get_recursion_depth() cannot see.
Assisted-by: Claude
* Publish and read the same pointer for a thread's top frame
set_current_frame() casts the `Py<FrameObject>` it publishes straight to
`*mut FrameObject`, so ThreadSlot::top_frame holds the object's base.
sys._current_frames() read it back through Py::from_payload_ptr(), which
subtracts the payload offset from what it is given. The reference it took
therefore incremented, and later decremented, a word 48 bytes ahead of
the frame -- inside the object allocated before it, whose OnceLock state
word sits exactly there for two frames adjacent in the size class. The
neighbour then read an initialized-looking cold pointer that had never
been written and locked whatever the uninitialized word addressed, so
the thread that owned it crashed rather than the one that read.
The slot now holds `*mut Py<FrameObject>`, which is what both sides mean.
A thread parked in a call has no FrameObject for its topmost frame, so
top_frame is null there and the reader takes the materialize path
instead: test_sys.test_current_frames never reaches the branch. The
snippet takes _current_frames() against threads that are running.
Assisted-by: Claude
Assisted-by: Codex:GPT-5
* Decide stop-the-world parking under the thread registry lock
do_suspend() published SUSPENDED first and only then re-read `requested`,
restoring itself to ATTACHED if the stop had ended in the meantime. That
made a thread the second writer able to leave SUSPENDED, so a stop whose
completion check had already observed the thread parked could be undone
behind the requester's back:
worker CAS ATTACHED -> SUSPENDED
requester all_non_requester_suspended() -> true, world_stopped = true
requester start_the_world(): requested = false, then walks the registry
worker reads requested == false, stores ATTACHED
With the store landing inside that walk the debug assertion in
start_the_world fires; with the walk already past the slot, a following
stop force-parks the thread DETACHED -> SUSPENDED, counts it as stopped,
and the store then puts it back to ATTACHED with the world declared
stopped and the thread running bytecode.
`requested` is set in init_thread_countdown() and cleared in
start_the_world() with the registry held, and start_the_world() keeps
holding it while releasing every SUSPENDED thread. Taking the registry
around the check and the transition therefore makes the two orders the
only ones possible: park before that release pass and be woken by it, or
find the request already withdrawn and stay ATTACHED. The requester is
left as the only writer that takes a thread out of SUSPENDED, and the
self-restore is gone.
suspend_if_needed() takes the VirtualMachine to reach the registry.
Assisted-by: Claude
Assisted-by: Codex:GPT-5
* Keep an atexit callback alive while it is being compared
atexit.unregister() releases the callback list around each __eq__ call and
identified the entry it had compared by the address of its Box. __eq__ can
call atexit._clear(), which drops that Box, and atexit.register(), whose
new Box lands on the freed allocation; the identity search then matched
the freshly registered callback and removed it.
atexit.register(a); atexit.register(b); atexit.register(c)
# __eq__ runs _clear() then register(d), returns True
atexit.unregister(probe)
left no callbacks registered where CPython leaves d.
Entries are Arc-shared now, so unregister() holds the one it is comparing
and matches it with Arc::ptr_eq: an address cannot be reused while the
comparison that named it is still running.
Assisted-by: Claude
* Hold atexit entries in PyRc rather than Arc
PyObjectRef is Send and Sync only under the threading feature, so an Arc
over a callback entry trips clippy::arc_with_non_send_sync in builds
without it, such as the wasm package. PyRc is Arc there and Rc otherwise.
Assisted-by: Claude
* Do not hold a buffer's storage while waiting for a peer
FileIO.readinto() and socket.recv_into()/recvfrom_into() took the target
buffer's write borrow and kept it for the whole call, including the wait
for data a pipe, socket or terminal may never deliver. What CPython holds
across that wait is the export, which only forbids resizing; the borrow is
a lock every other thread touching the same object waits on, so
threading.Thread(target=lambda: sock.recv_into(buf)).start()
len(buf)
did not answer until the peer sent. A thread parked on that lock is
ATTACHED and never reaches a safepoint, so gc.collect() in a third thread
waited for the peer as well: one incidental read of the buffer stopped the
world from being stopped at all.
The wait now runs against storage of its own and the bytes are copied over
once they arrive, with the export held throughout so the target still
cannot be resized meanwhile. A seekable file answers from itself rather
than from a peer, so FileIO.readinto() writes into the target directly
there and the buffered read path is unchanged.
Assisted-by: Claude
* Do not hold a buffer's storage while waiting to hand it over
socket.send()/sendall()/sendto()/sendmsg() and FileIO.write() kept the
source buffer's read borrow for the whole call, including the wait for a
peer that may never make room. That borrow is a lock every other thread
writing to the same object waits on, so
threading.Thread(target=lambda: sock.sendall(buf)).start()
buf[0] = 1
did not return until the peer read; and a thread parked there is ATTACHED
and never reaches a safepoint, so gc.collect() in a third thread waited
for the peer too -- the same wedge readinto() had on the receiving side.
ArgBytesLike::borrow_buf_unlocked() answers with bytes that survive the
borrow being dropped. An immutable object hands out a plain reference and
locks nothing, so those are sent where they lie and bytes and memoryviews
over them cost nothing; only bytes reached through a lock are copied out
first. The export is held throughout either way, so the source still
cannot be resized while it is being sent.
The regression snippet covers both directions now and is renamed for it.
Assisted-by: Claude
Assisted-by: Codex:GPT-5
* Check select()'s descriptor limit while the sequence is walked
seq2set() collected the whole sequence and compared the result's length
against FD_SETSIZE afterwards. Selectable::try_from_object() calls
fileno(), which runs Python and can append to the list being walked, and
the walk re-reads the list on every step, so the collection had no end to
reach and the comparison was never made. seq2set in Modules/selectmodule.c
checks the count per element instead.
stdlib_select.py gains a fileno() that appends to its own list, and
releases its sockets with close() rather than by dropping the name.
Assisted-by: Claude
* Bound native recursion where the C stack cannot be measured
check_c_stack_overflow() answers no unconditionally under miri and on
musl, where the stack pointer is not read. Since 9c9905aff that check is
all with_recursion() does, so every guard placed on native recursion --
__call__ and __get__ dispatch among them -- was a no-op on those targets
and the nesting ran until the stack ran out.
with_recursion() now counts its own depth on those targets and refuses
past NATIVE_RECURSION_LIMIT_UNMEASURED. The count is separate from the
frame limit sys.setrecursionlimit() sets, and compiles away where the
stack pointer can be read.
Assisted-by: Claude
* Report the failure to allocate pbkdf2's key and Take.readinto's scratch
Both buffers are sized from an argument -- pbkdf2_hmac's dklen accepts up
to i32::MAX, and readinto's from the length of the destination -- and were
built with vec![0u8; n], which aborts the process on allocation failure.
new_zeroed_bytes() raises MemoryError instead.
Assisted-by: Claude
* Read the frozen-code tuple length as a signed length
The '(' branches in read_marshal_str_vec() and read_marshal_const_tuple()
took the length with read_u32() as usize, so a value with the top bit set
read as four billion items rather than as out of range. read_len() is what
every other length in this file goes through, and it reinterprets as i32.
Both readers serve deserialize_code(), which reads only the frozen modules
baked in at build time, so this changes no reachable behavior;
marshal.loads() already went through read_len().
Assisted-by: Claude
* Make the snippets from the fuzzer sweep assert what they check
builtin_str.py expanded a tab to 2**31-1 columns, allocating 2 GiB to
observe that the width is accepted; a string with no tab observes the same
acceptance without laying anything out.
stdlib_array.py caught the refusal of frombytes() on its own exported
buffer and passed silently when nothing was raised. stdlib_hashlib.py
used a bare `assert False` as its failure branch. Both now say so through
the same shapes the other snippets use.
stdlib_socket.py also accepts OverflowError from recv() with a size that
does not fit the platform's C int.
stdlib_threading_current_frames.py indexed the frame chain for "f123"
without first asserting it is there.
Assisted-by: Claude
* Ask for a marshal container's room instead of assuming it
A flagged tuple or list is published in the reference table before its
children are read, and the placeholder was built with vec![none; len].
The length is the input's to choose and read_len() lets it reach
i32::MAX, so marshal.loads(b"\xa8\xff\xff\xff\x7f") -- five bytes -- asks
for 17 GB of element slots and aborts the process where the allocator
cannot serve it. r_object() allocates the container up front too, but
PyTuple_New() reports what it cannot get.
The elements are now reserved with try_reserve_exact() and a refusal is
raised as MemoryError through the decoder's pending-error channel.
PyTuple::new_marshal_placeholder() held nothing but that allocation and
is gone; the caller builds the elements and uses new_ref().
Assisted-by: Claude
* Compare the blocking-buffer snippet against something
measure() asserted len(buf) == len(buf), which holds whatever the length
is; it now takes the length the caller expects. The pipe case compared
the drained total against the whole source, while an unbuffered write()
reports only what it transferred and a signal can cut that short; it now
compares against what write() returned.
Assisted-by: Claude
* Refuse array.frombytes() a source whose items are not bytes
frombytes() reads its argument as bytes, but accepted any contiguous
buffer, so array("i").frombytes(memoryview(array("d", [1.0]))) appended a
double's bytes read as ints instead of raising. array_array_frombytes_impl
requires an itemsize of 1; ArgBytesLike now reports the itemsize so the
same check can be made here.
The BufferError that a resize meets while the array is exported also names
the array rather than repeating bytearray's wording.
stdlib_array.py's frombytes-of-itself case used typecode "i", where the
new check answers before the resize guard is reached; it uses "b" so the
guard is what refuses, and asserts the wider case separately.
Assisted-by: Claude
* Tell apart the ways marshal data can be bad
A type byte no reader knows, a back reference that names nothing, and
TYPE_NULL all came out as a bare "bad marshal data" ValueError. r_object()
answers the first two with "bad marshal data (unknown type code)" and
"bad marshal data (invalid reference)", and read_object() answers the
third with a TypeError, "NULL object in marshal data for object", since
TYPE_NULL stands for no object rather than for a value.
MarshalError gains the three cases and deserialize_value() maps them.
The container-specific wording r_object() uses for a NULL read inside a
tuple or list is not reproduced; the exception type is.
Assisted-by: Claude
* Hash the collector's tables by address rather than by SipHash
A collection keys three sets and two maps by object address, and it visits
every tracked object and every edge between them, so the hashing is a
per-edge cost. Those tables used the default RandomState, whose SipHash
buys resistance against a caller choosing colliding keys -- and nothing
chooses these keys: they are addresses this process handed out into tables
that live and die inside one collection. A profile of gc.collect() over a
423k-object heap spent 45% of its samples in SipHash.
They now hash with a splitmix64 finalizer. The shifts matter: a table
picks its bucket from the low bits and an address arrives with those bits
zeroed by alignment, so a plain multiply leaves every object in a handful
of buckets and is slower than SipHash was.
The reachability walk also copied each object's referent vector out of the
map it was cached in, a second pass over every edge; it reads them in
place, and reference subtraction hands its vector to the map instead of
cloning it.
Measured over 423k live objects: 0.93s to 0.15s. Over 843k dead ones:
3.00s to 0.79s. extra_tests/snippets/stdlib_threading_gc_import.py, whose
collector thread calls gc.collect() in a loop, ran anywhere from 2.7s to
28s and now runs in 3.1-3.5s: a collection that takes longer leaves more
garbage for the next one to walk, so the cost fed back on itself.
Assisted-by: Claude
Assisted-by: Codex:GPT-5
* Keep the collection's candidates and their counts in one table
A collection built a set of candidates and, beside it, a map from the same
addresses to their reference counts. Both were probed for every edge in
the heap -- membership from the set, the count from the map -- so each
edge paid to hash the same address twice, and each candidate paid to be
inserted twice. The map alone answers both questions.
The candidates also keep a walkable order now, which the reference
subtraction pass needs since it writes the counts while reading the
candidates, and which the unreachable set is built from instead of a
set difference.
Over the 423k-object heap measured in the previous commit: 0.15s to
0.13s live, and 0.79s to 0.49s dead.
Assisted-by: Claude
Assisted-by: Codex:GPT-5
* gc: collect referents into one buffer instead of a vector per object
Step 3 allocated a `Vec` for every tracked object to hold its referents
and kept them all in a map until step 4 read them back. The referents now
go into a single growing buffer, with the map holding each object's range
into it.
Adds `PyObject::gc_extend_referent_ptrs`, which appends to a caller's
buffer; `gc_get_referent_ptrs` calls it with a fresh one.
Assisted-by: Claude
* memoryview and struct: match the checks and errors of the reference
memoryview:
- `cast()` accepted a source and destination that are both item types,
which reinterprets the items rather than re-dividing the bytes; one side
now has to be a byte format.
- A cast to `shape=()` returned without checking that the buffer holds
exactly the one item that shape describes.
- `hash()` hashes the bytes, so it now raises ValueError for a view whose
items are not bytes, rather than returning a hash that disagrees with the
value the view compares equal to.
- `tobytes()` takes the `order` argument, with 'F' walking a
multidimensional view down its columns; `BufferDescriptor` gained
`for_each_segment_fortran` for that walk.
struct:
- A value the format has no room for reported "argument out of range"
instead of naming the format and its range. The format character is now
passed to the packing functions to report it.
- `Struct.__new__` no longer reads the format; `__init__` does, so
`__init__` can be called again and a subclass can pass the format up.
Methods raise RuntimeError until it has run, and `Struct` is a base type.
Removes the expectedFailure from test_Struct_reinitialization and
test_struct_subclass_instantiation.
Assisted-by: Claude
Assisted-by: Codex:GPT-5
* io: decide the readinto path by file type, not by seekability
FileIO.readinto wrote straight into the caller's buffer, holding its write
borrow, when the fd was seekable; otherwise it read aside into scratch and
copied. Seekability stood in for "this read answers without waiting on a
peer", which a pipe on Windows breaks: lseek on one succeeds, so the pipe
took the borrow-holding path and every other thread touching that bytearray
waited for the peer.
host_io::reads_without_waiting answers it directly -- seekability elsewhere,
GetFileType() == FILE_TYPE_DISK on Windows.
The regression snippet times each operation separately, so a failure names
the one that waited; it asserts the transfer is still in flight before
checking the export; and the socket case fills the connection until it
refuses rather than assuming a size that outruns it, which SO_SNDBUF on an
already-connected pair does not settle.
Assisted-by: Claude
|
||
|
Jeong, YunWon
|
c603eb7c9a |
Implement PEP 688 and rework the buffer protocol around managed exports (#8523)
* Implement PEP 688 __buffer__ and __release_buffer__ A Python class could not export a buffer: the slot machinery had no bf_getbuffer or bf_releasebuffer, and every consumer acquired buffers as PyBUF_FULL_RO through a module of PyBUF_* constants. Add both slots. PyBuffer::release now runs a Python __release_buffer__ before the exporter's own release, once per acquisition, which PyBuffer tracks with an `acquired` flag that clones do not inherit. An export made by a Python __buffer__ is held by a _buffer_wrapper payload that counts its exports and drops the returned memoryview with the last one, and the view handed to __release_buffer__ is a _buffer_window that owns no export, so releasing it inside the hook is inert instead of re-entering it. Replace the PyBUF_* constants with a BufferFlags bitflags type whose composite requests are supersets of the simpler ones, so `contains` answers the REQ_* questions, and pass the request to PyBuffer::from_object. Each consumer now asks for what its counterpart asks for: y* arguments for SIMPLE, w* for WRITABLE, BytesIO.write for CONTIG_RO, bytes(), bytearray() and memoryview() for FULL_RO. memoryview checks the request in memory_getbuf, and array.array and mmap.mmap expose __release_buffer__. Test buffer support with PyObject::check_buffer (PyObject_CheckBuffer) instead of attempting an acquisition, so an exception raised by __buffer__ is no longer reported as the object not being bytes-like, and a __buffer__ with side effects runs once. PyBytesInner becomes a y* conversion as a result: bytes and bytearray methods no longer accept iterables of ints, and find, index, count and __contains__ take the arguments parse_args_finds_byte and bytes_contains describe. A view exports its start offset in the descriptor rather than in its window, which fixes a panic when collecting from a negative-stride view. BytesIO.write rechecks closed after acquiring its buffer, which __buffer__ can close in between. Assisted-by: Claude Code:claude-opus-5 * Rework the buffer protocol around managed exports and view offsets Give `PyBuffer` the `_PyManagedBufferObject` shape: one `bf_getbuffer` acquisition is shared by every handle taken from it, cloning takes another share instead of re-acquiring, and the exporter's release runs once when the last share goes away. Remove `retain`, the unsafe `drop_without_release`, the three `impl Drop`s and the `ManuallyDrop` that stood in for this. Add `abort_acquisition` so a failed request does not run `bf_releasebuffer`. Move the view start into `BufferDescriptor::offset`, the `Py_buffer.buf` analogue, and drop the separate `start` fields on `PyMemoryView` and `PyBufferWrapper`. Slicing goes through `SaturatedSlice::adjust_indices_start`, which reproduces `PySlice_AdjustIndices` and keeps the adjusted start. Fix `zip_eq` to take its contiguous fast path only when both last dimensions are contiguous, and make `for_each_segment` and `zip_eq` handle zero-length and zero-dimensional views. Add `BufferDescriptor::projected` so a request without `PyBUF_ND`, `PyBUF_STRIDES` or `PyBUF_FORMAT` receives a correspondingly reduced descriptor, and reject a request without `PyBUF_INDIRECT` against an exporter that has suboffsets. Copy the source first in `memoryview` slice assignment when both sides reach the same root exporter. Hold the export across the resize in `bytearray.extend`, take `y*` in `marshal.loads`, stop probing the buffer protocol in `FsPath`, rewrite `ord` over the concrete string types, fold `array`'s buffer slot into one `slot_as_buffer`, take `w*`/`y*` in `_overlapped`, and thread the new `offset` field through the `_ctypes` descriptors. Assisted-by: Claude * Re-check a memoryview after a conversion and tighten cast and release `pack_single` and `unpack_single` addressed the buffer with a position taken before `__index__` ran, so releasing the view from that conversion read or wrote outside the exporter's storage, panicking when it had also shrunk. Check the released flag again once the conversion is done, as `CHECK_RELEASED_AGAIN` does. Reject a cast to a format that is not a single native format character with an optional `@` in front of it. An empty format reached a division by its item size of zero. get_native_fmtchar Report a second `__release_buffer__` on the same view as a `ValueError` rather than accepting it, and check that the view belongs to the object first; the silent case is a view that exports nothing. wrap_releasebuffer Compare against another memoryview by reading its view where it lies instead of acquiring a buffer from it, so the restricted view handed to `__release_buffer__` compares equal rather than unequal in one direction only. memory_richcompare Name the type in the unraisable an exception from `__release_buffer__` reports, as `releasebuffer_call_python` does, instead of reporting the exporter object. Assisted-by: Claude * Re-check the destination of a memoryview slice assignment Acquiring the source runs `__buffer__`, which can release the destination view, so check the released flag again once the source is in hand and before the structures are compared, as `copy_single` does. Build the sliced destination as a view that counts as no export, the way a `Py_buffer dest = *view` copy does. Holding one kept the exporter unresizable for the length of the assignment, so a source that released the view and then resized the exporter met a `BufferError` instead of the assignment reporting the released view. Assisted-by: Claude * Answer the memoryview contiguity getsets with the layout `f_contiguous` reported the row-major answer for one dimension and False for any more, so a view laid out both ways, such as one of shape (1, 8), reported False while `check_buffer_request` accepted a `F_CONTIGUOUS` request on it. `contiguous` answered row-major order alone. Move the Fortran-order test from `PyMemoryView` to `BufferDescriptor`, next to the row-major one it differs from only in iteration order, and answer all three getsets with it. Assisted-by: Claude |
||
|
Jeong, YunWon
|
11de045ef8 |
Per-interpreter runtime state and a process-wide GC stop-the-world (#8517)
* vm: add per-interpreter runtime state and interpreter registry - vm/runtime.rs: process-global interpreter registry (monotonic ids, weak entries), InterpreterWhence/InterpreterInfo, process main id recording via main_interpreter_id(), a threading-gated owner map (store_owned_interpreter/ take_owned_interpreter/is_owned_interpreter/owned_interpreter_count), and the SUPPORTS_ISOLATED_INTERPRETERS constant. - PyGlobalState gains interpreter_id/whence/is_main and is_main_interpreter(); PyConfig/Settings derive Clone so a subinterpreter can clone parent config. - Interpreter: id()/whence()/is_main()/is_process_main(), create_subinterpreter() and create_owned_subinterpreter(); unregister on Drop. - thread.rs: per-interpreter thread slots (INTERP_THREAD_SLOTS), slot swap when switching interpreters on one OS thread, cleanup keyed by interpreter id. - Install signal handlers and init the main-thread ident only on the main interpreter; _thread._is_main_interpreter reflects the current interpreter. - sys.implementation.supports_isolated_interpreters reads the constant. - Guard the registry static for non-threading builds where rc::Weak is !Send. Assisted-by: Claude Code:claude-opus-4-8 * gc: stop every interpreter while collecting The generation lists are process-global, so a collection reads and frees objects owned by every interpreter. CollectStopTheWorld stopped only the collecting interpreter, leaving other interpreters' threads free to mutate the same object graph during the reference-subtraction, reachability and snapshot phases. Stop all live interpreters instead, in runtime id order, and restart them in reverse. The global `collecting` mutex serializes collectors process-wide, so no second collector takes these exclusions in another order; fork acquires a single interpreter's exclusion, so the orders cannot cycle. - runtime: add live_interpreter_states(), ordered by interpreter id. - StopTheWorldState methods take &PyGlobalState instead of &VirtualMachine, so an interpreter's world can be stopped without a VM for it; update the call sites in frame, _thread, posix, faulthandler and capi. - Document in gc_state() that the collector is process-wide: gc.disable(), thresholds, gc.garbage and gc.get_objects() observe process-wide state, and a per-interpreter collector additionally needs untrack_object (called from default_dealloc with no VM in scope) routed to the owning interpreter. Tests: stop_the_world_parks_threads_of_another_interpreter asserts a thread entered in one interpreter parks another interpreter's threads (it fails without this change), plus a collect-while-another-interpreter-churns test. Assisted-by: Claude Code:claude-opus-4-8 * vm: scope the interpreter registry per thread without threading The registry held `rc::Weak` in a process-global `OnceLock` and covered the resulting `!Send`/`!Sync` with an `unsafe impl` justifying it as "non-threading builds are single-threaded". That is not this codebase's model: `static_cell!` is thread-local without the `threading` feature precisely so each OS thread can own its own `Context::genesis()` and `GcState`, so two threads could reach the same `Rc` counts through the registry. Use `static_cell!` for the registry as well, matching `gc_state()`, and drop the `unsafe impl`. Ids are consequently unique per registry rather than per process in non-threading builds, which is documented on `alloc_interpreter_id`. Move the recorded main interpreter id into the registry so it follows the same scoping instead of living in a separate global `OnceLock`. Also make `CollectStopTheWorld::new` accumulate into a live guard: it built a bare `Vec` and only moved it into the restarting `Drop` type after stopping every interpreter, so an unwind partway through the loop left the already stopped interpreters parked forever with their exclusion held. Assisted-by: Claude Code:claude-opus-4-8 * vm: keep the interpreter registry usable across bootstrap, drop and fork Three gaps where the registry did not describe the interpreters that actually exist, each of which hides an interpreter from the collector's stop-the-world. Register before `initialize()`. Registration ran as the last step of `initialize_vm`, so the whole bootstrap — which executes Python bytecode and allocates GC-tracked objects — was invisible to `live_interpreter_states()`. It still cannot run any earlier than this: the init hooks take `PyRc::get_mut` on the state, which fails as soon as the registry holds a weak reference to it. Stop unregistering in `Interpreter::drop`. The handle does not decide the interpreter's lifetime — every `ThreadedVirtualMachine` from `new_thread()` holds its own `PyRc<PyGlobalState>` — so an interpreter with running workers disappeared from the registry while its threads kept mutating the object graph. The entries are weak, so lifetime is already correct without the removal; dead entries are now reaped when registering instead. Interpreters are consequently released rather than unregistered at a fixed point, so the two tests asserting disappearance now wait for it: a collection in progress legitimately holds a reference to every live interpreter. Repair other interpreters after fork. `py_os_after_fork_child` only fixed the forking interpreter, leaving every other one with slots for threads that did not survive (still ATTACHED if they were running bytecode) plus locks and stop-the-world flags held by them. Since a collection stops all interpreters, the child's first collection would wait for threads that no longer exist. Reset their locks, stop-the-world state and thread tables, drop this thread's cached slots for them, and reinit the registry's own locks first, since enumerating interpreters now takes them. Tests: test_gc, test_threading and test_fork1 pass, as do the vm tests in both the threading and default configurations. Assisted-by: Claude Code:claude-opus-4-8 * vm: attach and detach thread slots when switching interpreters `enter_vm` decided whether to attach from `was_outermost` (an empty VM_STACK), which held while a thread could only ever be in one interpreter. With a slot per (thread, interpreter) pair, entering interpreter B from a thread already inside interpreter A's section switched CURRENT_THREAD_SLOT to B's slot but attached nothing: the thread then ran B's bytecode with B's slot DETACHED while A's slot stayed ATTACHED. A collector stopping B force-parks the DETACHED slot and concludes B is stopped, and then walks the object graph this thread is still mutating. Pair the attach/detach with the slot switch instead (≈ `_PyThreadState_Swap`): `begin_interpreter_section` detaches the enclosing interpreter's slot, makes the target slot current and attaches it, and `end_interpreter_section` undoes that and re-attaches the enclosing interpreter. Both live in `set_current_vm`, which every path making a VM current already goes through, so `enter_vm` and `VmBootstrapGuard` no longer track outermost-ness themselves. `nested_enter_of_subinterpreter_is_stoppable` covers this: it runs a subinterpreter nested inside the parent's section and asserts the sub's threads park when the sub's world is stopped. It fails with the previous attach-at-outermost-only behavior. Assisted-by: Claude Code:claude-opus-4-8 * vm: list only the current interpreter's subclasses Interpreters share the context, so `class Foo(int)` in one of them pushes onto the same `int.subclasses` every other one reads, and `int.__subclasses__()` returned types no other interpreter can reach. Record the creating interpreter on `HeapTypeExt` and filter `__subclasses__` by it, the way `lookup_tp_subclasses` reads `tp_subclasses` out of per-interpreter state for static builtin types. Types built before any interpreter exists — the ones the shared context creates, including the exception hierarchy — carry no id and stay visible to every interpreter, which is what `_PyStaticType_InitBuiltin` produces by registering the builtin subclass links once per interpreter. The other walks over `subclasses` (version-tag invalidation, abc flag propagation, mro updates, slot propagation) are left as they are: each starts from a type being mutated, so from a heap type, whose subclasses all live in the interpreter that created it. `subinterpreter_subclasses_are_scoped_to_their_interpreter` covers this and fails without the filter. Assisted-by: Claude Code:claude-opus-5 * gc: give each interpreter its own collector state The generation lists stay process-wide, because an object is untracked from `default_dealloc`, where no interpreter is in scope to route to. What changes is that a collection no longer acts on every interpreter's objects, and the gc module no longer reports one interpreter's state to another. `track_object` stamps the running interpreter into a new `gc_owner` word on the object header, and a collection takes as candidates only the objects carrying its own tag plus the ones carrying none. `gc_owner` fits in the padding the header's alignment already forces, so objects do not grow; an assertion on the header size keeps it that way. Objects allocated with no interpreter running — everything the shared context builds — carry no owner and stay candidates for every interpreter, which is where they were before. Objects that outlive the interpreter that tracked them are adopted the same way by the next full collection, rather than being left to a collector that will never come. `enabled`, the thresholds, the debug flags, the statistics, `gc.garbage` and `gc.callbacks` move onto `PyGlobalState` — the last two off the shared `Context` — so `gc.disable()`, `gc.set_threshold()`, `gc.get_stats()`, `gc.get_objects()` and `gc.garbage` describe the interpreter that asks. The occupancy counts behind `gc.get_count()` and `gc.get_freeze_count()` stay process-wide: they measure how full the shared lists are. Their decrements are saturating now, since a collection zeroes the generations it emptied while another interpreter's objects are still sitting in them. Stop-the-world still stops every interpreter. Unowned objects are candidates and any interpreter can incref one, so the refcounts a collection reads are only stable while all of them are parked. This also fixes a deadlock it exposed: `CollectStopTheWorld` dropped its references to the stopped interpreters while the collection still held the generation read locks, so releasing the last reference to one — which frees its objects, and so untracks them — waited for a write lock behind that read lock. The references are now held until the guard itself drops. `collections_only_reach_the_collecting_interpreter` and `get_objects_only_reports_the_calling_interpreter` cover this and both fail without the owner check. Assisted-by: Claude Code:claude-opus-5 * _queue, _thread: detach before taking locks held across waits `_queue.Semaphore` holds its mutex across the `allow_threads` condvar wait, and `join_internal` holds a thread handle's completion mutex the same way. Stop-the-world can stop a thread while it holds either one. The remaining acquisitions ran attached, so a thread blocking on such a mutex had no safepoint left to reach: the stop never completed, and the holder was never resumed to release it. Route those acquisitions through helpers that detach first. The fork-child reinit paths keep their direct locks. Assisted-by: Claude * _io, _winapi: detach on the remaining stopped-holdable lock takes `TextIOWrapper.__repr__` took `data` directly while every other method takes it through `lock_opt`, which detaches. `Overlapped` holds `inner` across the `allow_threads` in `GetOverlappedResult`, and all four of its takes were direct. A thread stopped by stop-the-world can be holding either mutex, so taking one while attached left the blocked thread with no safepoint to reach. Assisted-by: Claude * gc: size the interpreter owner tag to the header padding `gc_owner` was a u32. With 4-byte pointers its alignment pushed it out of the padding that follows the gc bits and generation, growing every object by a word and tripping the `SIZEOF_PYOBJECT_HEAD` assertion on 32-bit targets. Introduce `GcOwner = u16`, which the `repr(C)` layout places at offset 10 on 32-bit and 18 on 64-bit, leaving the header at 6 words on both. Tags now run out after 65535 interpreters; `alloc_owner` already falls back to `GC_NO_OWNER`, so an interpreter past that collects as it did before tagging. Also widen three test deadlines that measure liveness, not speed. Assisted-by: Claude * vm: reuse cleared frame blocks and shorten interpreter hot paths - `datastack` remembers the most recently popped frame block. `push_frame` reports an exact LIFO reuse, and `setup_datastack_frame` then skips zero-filling localsplus. - Small-int loads push the context's cached int as a borrowed stack reference instead of taking a new one. - Calls to jitted functions go straight to `execute_call_vectorcall`. - Binary-op specialization reads ints through the new `PyInt::try_to_i64_fast` instead of the generic primitive conversion, and `try_to_bool` moves its non-bool path into a `#[cold]` helper. - Dict caches read an entry through a keys-version stamp (`get_index_if_keys_version`) rather than an entry-index hint. - Frame publishing caches a pointer to `ThreadSlot::top_iframe` in a thread-local `Cell` instead of borrowing `CURRENT_THREAD_SLOT`. * gc, vm: address review notes on owner tags and interpreter docs Look up retired owner tags with a sorted binary search instead of a linear scan, which every scanned object paid for once per dropped interpreter. Drop the claim that clearing a tag frees it for reuse; `alloc_owner` only ever hands out new tags. Also correct the tag space it mentions, which is 16-bit since the tag was sized to the header padding. Document `is_main` as "top-level interpreter" rather than "the process main": every top-level interpreter sets it, and only the first registered one becomes the main `main_interpreter_id` reports. Assert membership rather than an exact owned-interpreter count delta; the owned table is process-global and other tests store into it in parallel. Assisted-by: Claude * vm: gate interpreter registration on stop-the-world admission A collection snapshots the registry, stops the interpreters it found, and then reads tracked objects with their threads parked. An interpreter that registered after the snapshot was taken was absent from it, so nothing stopped it and its bootstrap ran Python — allocating and mutating the shared generation lists — underneath that scan. Registration and the stop now share a process-global gate: the collection holds it from before the snapshot until the restart, and registration takes it around the registry insert. The insert runs detached, since a thread that waited for the gate, or re-attached while holding it, would leave the stop it waits for no safepoint to complete at. Assisted-by: Claude * gc: keep the tracking counters off the barrier path Every tracked allocation and every free went through sequentially consistent counter updates, and each free through a `fetch_update` CAS loop. The counters drive only the gen0 threshold and `gc.get_count()`, and the generation locks — not the counters — order the list changes they describe, so they are relaxed now and the decrement is a load plus a conditional `fetch_sub`. `is_enabled` and `threshold`, both read once per allocation, are relaxed for the same reason: an allocation racing `gc.disable()` may use either value. Drop `alloc_count`, which nothing has ever read. Assisted-by: Claude * dict: settle lookups and iteration steps under one read guard A lookup probed under a read guard, dropped it with the matched entry in hand, and then took the lock again to re-find the entry. `lookup_extract` reads the entry while the probe still holds the guard when key identity settles the match, which is the case that cannot run Python. Dict and set iterators took one lock to compare the size and another to read the entry, then cloned both the key and the value even though a keys or values view keeps only one of them. `next_entry_checked` does the size check and the read under one guard, and clones through a per-view projection. A store that missed its inline-cache hint re-probed the dict afterwards only to recover the entry index the store had just computed; `unchecked_push` reports that index instead. Assisted-by: Claude * vm: build a call's argument vector once The specialized CALL handlers collected the positional arguments into one vector and then copied them into a second one to put `self` in front, so every specialized builtin, method-descriptor, class and non-Python call paid two allocations, a copy and two frees. The arguments are already laid out on the value stack in vectorcall order, so `take_call_args` fills one vector by index — the shape `execute_call_vectorcall` already used. `vectorcall_native_function` and the keyword path of `vectorcall_function` then cloned that vector again to build `FuncArgs`; both now move it in through `from_vectorcall_owned`, as the other vectorcall slots do. Assisted-by: Claude * vm: reach an instance dict without cloning it The specialized attribute instructions cloned the instance dict — a rwlock round-trip plus a refcount round-trip — for two things that only look at it: `LoadAttrMethodLazyDict` asking whether the dict exists, and the keys-version stamp check that is the whole of `shadowing_instance_attr`'s fast path. Both now read it borrowed, through `has_instance_dict` / `with_instance_dict`. `generic_getattr_opt` probed the dict with the name's `&Wtf8`, which hashes the name on every lookup and can never match a key by pointer. Passing the `Py<PyStr>` uses the string's cached hash and the interned-key identity check, and drops an allocation when a stored key is not an exact `str`. Assisted-by: Claude * vm: shorten the per-instruction safepoint and the call preamble The dispatch loop asks once per instruction whether stop-the-world wants this thread, and that read went through `CURRENT_THREAD_SLOT` — a `RefCell` borrow, so two stores to thread-local memory around an `Option` test. Cache the `stop_requested` pointer in a plain `Cell` at the same three places the frame pointers are cached, and the safepoint becomes one relaxed load. `lasti` is advanced from the index the loop just read, rather than reloaded to increment it, and `Resume` reads `quickened` before swapping it, so a call to an already-quickened code object costs a load instead of an atomic read-modify-write. An exact-args vectorcall to a Python function built a heap `FrameObject` where the equivalent `invoke` path uses a data stack frame; it now does the same when tracing is off. Assisted-by: Claude * vm: give KwArgs a zero-sized hasher `KwArgs::default()` is built for every call, keyword-less ones included, and it seeded a `RandomState` each time — a thread-local read and 16 bytes in every `FuncArgs`. Keyword names come from the program text, so the map now uses `BuildHasherDefault`, whose `Default` is a zero-init. Assisted-by: Claude * vm: promote borrowed stack refs before a yield `yield <small int>` tripped the "borrowed refs on stack at yield point" assertion: `LoadSmallInt` pushes a borrowed ref, and a yield saves the stack with the frame, which is exactly what that assertion forbids. Promote the stack first, so a suspended frame owns everything it holds. Assisted-by: Claude * vm: run the exact-args vectorcall on a heap frame again `vectorcall_function` took the data stack path for an exact-args call whenever tracing was off. A data stack frame that materializes — one traceback entry is enough — copies its localsplus into the `FrameObject`, so every value it holds gains a reference for as long as the frame runs, and a tail call holds the callee's function object until the caller's next call or return. Both are reachable without this hunk, through the `CallPyExactArgs` specialization, but routing the vectorcall entry point through the data stack made them apply from a call site's first execution: test_enumerate, test_memoryio, test_sys, test_tempfile and test_traceback all read a refcount or a finalization that moved. Assisted-by: Claude * vm: release a tail call's function at the callee's return The trampoline moved the callee's function object into the caller's `SuspendedFrame` and dropped it only once the caller had run again, so a function entered through a tail call kept one reference past its return: `sys.getrefcount` on it read one high for the caller's next stretch of bytecode, and anything the function reached stayed alive that much longer. The callee's frame is released before the `ReturnValue` or `Unwind` action is formed, and a materialized frame object holds its own references, so the owner is dropped where the suspended caller is popped instead. Assisted-by: Claude * vm: stop copying a running frame's locals when it materializes `materialize_slow` cloned the source frame's fast locals into the new `FrameObject`, so every value in a data stack frame gained a reference lasting as long as that frame object: one traceback entry through a frame kept all of its locals alive, and `sys.getrefcount` read one high for each of them. Nothing read that copy while the frame ran. `find_live_source_iframe` already routes reads to the live frame, and `exit_iframe` fills the slots from it at return, so the copy now starts empty as `materialize_slow_chain`'s does, and `snapshot_to_heap` goes with its last caller. Two readers had to be brought to the live frame the same way. `has_active_hidden_locals` read the frame object's own slots, which let `locals()` in a class body write an inlined comprehension's hidden variable into the class namespace; it now shares `live_fastlocals` with the other two. Another thread's frame has no live frame to reach from here, so the six cross-thread sites fill the copy under stop-the-world through `materialize_with_locals`. Assisted-by: Claude * vm: key data stack frame reuse on the exact frame size push_frame and pop_frame compared 16-byte-aligned sizes, so two frames whose sizes differ by less than the alignment matched as a reuse. The caller reads that as "every slot was cleared by the previous frame", but the larger frame's tail slots were never touched by it. Assisted-by: Claude * vm: store the frame object payload address in the cross-thread slot set_current_frame and reinit_frame_slot_after_fork wrote the address of the Py<FrameObject>, while the cross-thread readers hand that pointer to Py::from_payload_ptr, which subtracts the object header from it. Both writers now store the payload address. Assisted-by: Claude * vm: copy a foreign thread's frames instead of linking them sys._current_frames(), _thread._current_exceptions() and the cross-thread f_back path handed back frame objects still linked to the running frames. The owning thread wrote its locals into those same buffers at exit_iframe while the reader was cloning out of them. materialize_detached and materialize_detached_chain copy the values under stop-the-world and leave nothing for the owner to write into; materialize_with_locals is gone. FrameColdData gains attached_tid, the thread still running the frame a frame object was materialized from, which exit_iframe clears once it has written the values in. check_locals_access now rejects a read from any other thread while it is set; the FrameOwner test could not, because materialize_slow forces FrameObject ownership. Also removes with_iframe, which had no callers. Assisted-by: Claude * vm: resolve the two rustdoc links this branch added MAIN_INTERPRETER_ID is not in scope in interpreter.rs, and new_thread only exists in a threading build, so neither link resolved. Assisted-by: Claude |
||
|
sijun-yang
|
c3bfefaeb0 |
Remove private comprehension flags (#8537)
* Remove the private assignment-expression marker Why: extend_namedexpr_scope() already propagates assignment-expression targets and records DEF_GLOBAL or DEF_NONLOCAL in comprehension scopes. ASSIGNED_IN_COMPREHENSION remained only for conflict checks. Its private bit overlaps CPython's packed LOCAL scope bit. Changes: - Register assignment-expression targets as ordinary assignments. - Detect later target conflicts from existing declaration flags. Assisted-by: Codex:gpt-5.6-sol * Remove the private comprehension iterator flag Why: ITER duplicates the DEF_LOCAL and DEF_COMP_ITER facts already recorded for comprehension targets. Its private bit also overlaps CPython's packed scope field. Changes: - Use those flags for rebinding checks and local restoration around inlined comprehensions. Assisted-by: Codex:gpt-5.6-sol * Preserve source names in named-expression diagnostics Why: Assignment-expression diagnostics exposed the mangled keys used for symbol-table lookup. CPython reports identifiers as written in source. Changes: - Use source names in both comprehension conflict diagnostics. - Enable test_named_expression_invalid_mangled_class_variables. Assisted-by: Codex:gpt-5.6-sol |
||
|
Jiseok CHOI
|
dbd3a0a6b3 |
sqlite3: add SQLITE_DBCONFIG constants, setconfig() and getconfig() (#8535)
* sqlite3: add SQLITE_DBCONFIG constants, setconfig() and getconfig() Implement Connection.setconfig() and Connection.getconfig() using sqlite3_db_config(), and export all SQLITE_DBCONFIG_* integer constants to the module. Fixes test_connection_config in test_dbapi.py. Assisted-by: GitHub Copilot:claude-sonnet-4-6 * sqlite3: fix setconfig to propagate actual SQLite error When sqlite3_db_config() returns a non-OK result, propagate the actual SQLite error instead of replacing it with a generic 'Unable to set config' message, to match CPython's set_error_from_db() behavior. Assisted-by: GitHub Copilot:claude-sonnet-4-6 |
||
|
Jiseok CHOI
|
e19b622174 |
sqlite3: pass None for NULL authorizer args instead of crashing (#8534)
* sqlite3: pass None for NULL authorizer args instead of crashing CPython's authorizer callback receives NULL for arg1/arg2/db_name/access when not applicable (e.g. SQLITE_READ on a table gives NULL for the database name in some versions). Previously RustPython passed these pointers to ptr_to_str which would crash or produce an error. Now ptr_to_str_or_none is used: NULL pointers become Python None, which matches CPython behavior and allows test_table_access and test_column_access to pass. Assisted-by: GitHub Copilot:claude-sonnet-4-6 * fix(sqlite3): avoid reentrant deadlock in Statement::new sqlite3_prepare_v2 can synchronously invoke the authorizer callback, which may call back into Connection methods (e.g. set_authorizer) that require the same db lock. Holding db_lock across the prepare() call caused a self-deadlock when a callback re-entered the connection. Release the lock after sql_limit check and copy the raw handle before calling prepare(), so FFI calls that can trigger Python re-entrancy happen outside the lock scope. Fixes hang in test_authorizer_concurrent_mutation_in_call |
||
|
Jiseok CHOI
|
a142de9abc |
sqlite3: add Connection exception attributes and fix autocommit ValueError (#8533)
Add DB-API 2.0 optional extension: expose exception classes as attributes on Connection objects (Warning, Error, InterfaceError, DatabaseError, DataError, OperationalError, IntegrityError, InternalError, ProgrammingError, NotSupportedError). Also fix autocommit validation to raise ValueError (not TypeError) when an unsupported type is passed. Assisted-by: GitHub Copilot:claude-sonnet-4-6 |
||
|
Jiseok CHOI
|
7cda53b946 |
sqlite3: validate narg before creating functions/aggregates (#8532)
CPython 3.14 added check_num_params() which raises ProgrammingError when narg/n_arg/num_params is out of range (-1..=SQLITE_LIMIT_FUNCTION_ARG). RustPython passed invalid values directly to SQLite, resulting in an OperationalError instead. Add check_num_params() helper and call it in create_function(), create_aggregate(), and create_window_function(). Assisted-by: GitHub Copilot:claude-sonnet-4-6 |
||
|
Jeong, YunWon
|
9e4c5708c6 |
str, bytes: answer equality with equality rather than with an ordering (#8531)
PyStr's comparison, PyBytesInner's, and the specialized CompareOpStr
instruction all answered == and != by taking Ord::cmp of the two buffers
and asking whether the result was Equal. An ordering has to read the
bytes: it memcmps the common prefix even where the lengths already settle
the question. CompareOpStr bypasses the Comparable slot, so it had also
lost the identity shortcut that slot takes, and a string compared with
itself was read end to end.
Add PyComparisonOp::eval_eq, which settles Eq and Ne from an equality test
and leaves an ordering operator to the caller, and answer through it in the
three places: slice equality checks the length first, and CompareOpStr
answers an object compared with itself the way the slot it specializes does.
n=1,000,000, per comparison:
before after
s == s (the very same object) 23.21us 0.16us
s == a string one shorter 24.63us 0.17us
b == bytes one shorter 25.28us 0.20us
ba == bytearray one shorter 27.68us 0.23us
s == an equal, distinct string 23.78us 24.50us
s < an equal string 29.87us 25.07us
Assisted-by: Claude
|
||
|
Jeong, YunWon
|
a97e8f0746 |
str: take search bounds through the character index (#8530)
* str: count an empty needle in characters
str.count with an empty needle ran find_iter over the range's bytes, which
reports one position per encoded byte boundary rather than one per
character: '가나다'.count('') answered 10 instead of 4. Count the range's
code points instead; bytes.count keeps the byte-position answer.
Assisted-by: Claude
* str: take search bounds through the character index
find, rfind, index, rindex, count, startswith and endswith resolved their
start/stop bounds with AnyStr::get_chars, which walks the payload to both
bounds, and find reported its hit by counting the code points in front of
it -- so a sweep over a subject's own indices walked it once per call.
Resolve the bounds with StrData::char_range_to_bytes instead, which is the
converter the subscript and slice paths already use, and map the hit back
with Wtf8Index::char_index_at_byte, added here as the table's inverse: a
bracketed search over the groups, then at most an entry's worth of steps.
The payload alone does not say whether it is ASCII, so get_chars walked
there too; that path is now off it as well.
n=8000, sweeping the bound over the subject, best of 3, interleaved:
'가나다라'*n/4 'abcd'*n/4
find 18.00ms -> 0.15ms 3.48ms -> 0.14ms
startswith 16.75ms -> 0.12ms
count 22.78ms -> 4.02ms
count stays linear in the range it is given, as it is in CPython.
Assisted-by: Claude
|
||
|
Jeong, YunWon
|
9616b28d3d |
str: resolve subscripts and slices through the code point index (#8526)
`nth_char` and the four `SliceableSequenceOp` methods each walked the buffer to reach a character index, so `s[i]` and `s[a:b]` on a non-ASCII string were O(i) and O(b), and a loop over either was quadratic. They now resolve through `char_index_to_byte`, which is what the index table was added for: a plain slice becomes a byte reslice, and a stepped slice one lookup per collected character. An index within four code points of an end is still walked to, and so is a slice that reaches within four of both. PyPy draws the same line with `MAX_UNROLL_NEXT_CODEPOINT_POS`, in a guard that also asks the JIT whether the index is a constant; there is no JIT here, but the reason to skip the build survives it -- `s[0]` on a long string should not pay for a table. The stepped slices took their character count from `(range.len() / step) + 1`, which overshoots whenever the last step lands short: `"aéc"[::3]` reported a length of 2 for a one-character string, and `reversed()` on it read past the end of the buffer and panicked. The count is now the index iterator's own length, so it cannot drift from the characters actually collected. Assisted-by: Claude |
||
|
Jeong, YunWon
|
4a8c613b04 |
_sre: drive a non-ASCII str subject through a character index on the string (#8522)
* common: index a WTF-8 buffer's code points for random access `Wtf8`'s iterators are sequential, so resolving a code point index through them is O(n) and code that indexes the same string repeatedly walks it once per index. `Wtf8Index` is a side table -- one 24-byte group per 64 code points, 0.375 bytes per code point -- that answers the same question in constant time; the layout is PyPy's `UTF8_INDEX_STORAGE`. `StrData` builds one on the first call to the new `char_index_to_byte`, in a slot published by compare-exchange, and drops it with the string. ASCII strings answer from the index itself and never build a table. A clone gets an empty slot, since it indexes its own copy of the buffer. Assisted-by: Claude * _sre: drive a non-ASCII str subject through the string's character index The `&Wtf8` drive answers `count` and `create_cursor` by decoding from the start of the subject, so a scan that restarts at successive positions walks the subject once per position, and `slice` walks it again per extracted group. `Utf8Str` holds the `PyStr` and asks it instead: `count` is the cached character length, and `create_cursor` and `slice` resolve their positions through `char_index_to_byte`. Stepping is the `&Wtf8` drive's, unchanged. The table lives on the string, so a `Match` that outlives the scan shares it -- `group` has no cursor of its own to move relative to. `SreStr for &Wtf8` has no callers left; the `StrDrive` impl stays, since `Utf8Str` steps through it. The three subject helpers now share one downcast. Assisted-by: Claude |
||
|
Jeong, YunWon
|
ef51c5c164 |
str: count a stepped slice's characters with a ceiling division (#8525)
`do_stepped_slice` and `do_stepped_slice_reverse` took the character count of
a non-ASCII result as `(range.len() / step) + 1`, which overshoots by one
whenever the span is an exact multiple of the step: `"aéc"[::3]` is one
character and reported two, `"가나다라"[::2]` two and reported three. ASCII
subjects were unaffected, since that arm collects into an `AsciiString` whose
length comes from the data.
The count is stored as the string's character length, so the result then
claimed a character its buffer does not hold, and `reversed()` on it indexed
past the end and panicked:
s = "".join(["a", "é", "c"])
list(reversed(s[::3])) # index out of bounds: the len is 1 but the index is 1
Use `div_ceil`, which is the number of elements the underlying range yields.
A differential over 44331 subscript and slice shapes -- every combination of
start, stop and step over strings straddling the ASCII split, the surrogate
range and the astral plane -- now matches CPython 3.14 exactly, where it
differed on 3100 lines before.
Assisted-by: Claude
|
||
|
Jeong, YunWon
|
234ad2db31 |
_sre: drive an all-ASCII str subject over its bytes (#8520)
* sre_engine: document that StrDrive positions are character indices
StrDrive carried no documentation, so nothing recorded that a cursor's
position is a character index rather than a byte offset. Every implementation
satisfies it -- skip(n) advances position by exactly n in all three -- and the
engine depends on it: _count bounds a repeat with position + max_count and
reports the repeat length as a difference of positions, ASSERT compares
position against a lookbehind width, and search_info recovers a match start as
position - (len - 1). A drive over a variable-width encoding that stored byte
offsets would leave those type-correct and wrong, and would index a lookbehind
out of bounds.
Write the invariant down on StringCursor and on each trait method.
Also walk the group's own cursor in GROUPREF instead of counting to the
group's width, so the loop bound is the thing being stepped. Generated
machine code is unchanged.
Assisted-by: Claude
* _sre: drive an all-ASCII str subject over its bytes
with_sre_str handed every str subject to the &Wtf8 drive, which answers
count() by counting every code point and create_cursor(n) by stepping over the
first n. Both run once per Request, so a scan that restarts at successive
positions walked the subject again on every call: finditer over an ASCII
subject of n tokens was quadratic in n.
PyStr already records whether it is ASCII -- StrKind is decided when the
string is built -- and for ASCII a character index is a byte index, so the
&[u8] drive's cursor arithmetic already applies: count() is the byte length
and create_cursor() is a pointer offset. Add AsciiStr, which delegates every
StrDrive method to that impl and differs only in slice(), which reslices the
span and returns str rather than bytes.
Matching is unaffected: StrDrive carries no unicode semantics, because the
engine keys every unicode decision on the compiled pattern's opcode rather
than on the subject type.
Also bind the subject once in with_sre_str, so callers passing a temporary
(`&x.clone()`) build it once rather than per arm.
finditer over an ASCII subject, n tokens, this machine:
n before after
5000 41.60ms 1.37ms
10000 217.30ms 2.68ms
20000 1206.28ms 5.17ms
40000 5016.65ms 10.31ms
Per-doubling x5.22/x5.55/x4.16 becomes x1.95/x1.93/x1.99. Collecting
m.group(0) for every match goes 5061.02ms -> 17.80ms at n=40000.
test.test_re is unchanged at 166 tests, OK (skipped=14, expected failures=6),
and a 2731-line differential over the is_ascii() boundary -- findall,
finditer spans and groups, sub, split, match, search, fullmatch across
subjects that are empty, ASCII, non-ASCII, or mixed -- is byte-identical to
CPython 3.14.6. Introducing an off-by-one in AsciiStr::slice moves 1224 of
those lines, so the comparison reaches the new code.
Assisted-by: Claude
|
||
|
Jeong, YunWon
|
259ce8ce66 |
Fix the itertools.tee leak and race, the contextvars and generator races, _asyncio's InvalidStateError lookup, and _imp's frozen data (#8518)
* _imp: read get_frozen_object's data as a whole marshal value
The data argument went through deserialize_code(), which reads a code body
without the type byte in front of it, so nothing marshal.dumps() produces was
accepted. Read it with marshal.loads() and require a code object back.
Assisted-by: Claude
* _imp: bind find_frozen's arguments with FromArgs
The arity check was done by hand on a FuncArgs. Take the arguments through a
FromArgs struct instead, which makes withdata keyword-only, and fill in the
data it asks for: the frozen encoding is not marshal, so the code is
re-serialized into what get_frozen_object() reads back.
Assisted-by: Claude
* itertools: give tee's shared buffer a Python type
The buffer was a PyRc<PyItertoolsTeeData>, which is not a Python object, so the
collector could not walk into it and any cycle running through a tee was
uncollectable. Make it the _tee_dataobject type with a traverse, held by
PyRef, and split the rest along the same lines: tee() is a function, _tee is
the iterator type it builds, and _tee takes a single iterable rather than
returning a tuple from __new__. _tee is weak-referenceable, tee() rejects a
negative n with ValueError and reserves its tuple fallibly.
test_itertools.test_tee passes now; its expectedFailure marker is removed.
Assisted-by: Claude
* contextvars: hold the shared context state under locks
The variable map was a RefCell, the enter flag, the context index, the token's
used flag and the variable hash were Cells, and each carried an unsafe impl
Sync. A Context or ContextVar shared between threads overlapped their borrows
and panicked. The map and the per-variable cache are now PyMutex, the flags and
the index are atomics, entering a context is a compare_exchange, and the three
unsafe impl Sync are gone.
The cache also stopped being read through AtomicCell::as_ptr, which raced a
concurrent store on a value holding a PyObjectRef.
Values displaced from the map or the cache are dropped after the lock is
released: __del__ can come straight back into the same context, and the locks
are not reentrant.
Assisted-by: Claude
* generator: read the frame state under the running claim
send(), send_none(), throw() and close() read `closed` and `frame.lasti()`
before `running` was compare_exchanged, so the frame they went on to resume
could be one another thread had already advanced. A resume that decided from
`lasti() == 0` that the generator had not started pushes no value onto the
value stack, and the code after the yield pops one, which underflows the stack.
The compare_exchange now hands back a guard, taken before those reads and
released after maybe_close(), so the generator is retired while it is still
claimed.
Assisted-by: Claude
* itertools: claim a tee's position and its buffer as one step
`_tee::next` read `index` and moved it on afterwards, and
`_tee_dataobject::get_item` released `running` before the value it fetched
from the source was cached.
Two callers on one `_tee` then read the same index, hand out the same value
twice and advance past a value that was never cached, which leaves `index`
past `values.len()` and indexes the buffer out of bounds. Two callers at the
same index on separate tees each fetch a value from the source, and one of the
two is dropped without reaching a caller.
Both claims now cover the read and the update.
Assisted-by: Claude
* _asyncio: require InvalidStateError to be a type
new_invalid_state_error() called whatever `asyncio.exceptions.InvalidStateError`
names and unwrapped the downcast of the result, so a future asked for a result
it does not have panicked once that attribute was rebound to something that is
not an exception:
asyncio.InvalidStateError = lambda *args: 42
_asyncio.Future(loop=object()).result()
The type is looked up the way get_cancelled_error_type() looks its own up, and
raised with new_exception_msg; a lookup that does not produce an exception type
falls back to RuntimeError, as the other arms already did.
Assisted-by: Claude
|
||
|
박효종
|
00461a094c |
Preserve lone surrogates across WASM conversions (#8508)
Assisted-by: Codex:GPT-5 |
||
|
Jeong, YunWon
|
1d6c6aece0 |
marshal: refuse a back-reference to an object still being written (#8519)
`w_ref` marks a code or slice entry incomplete until `w_complete`, because the reader rebuilds both from their fields and a `TYPE_REF` issued while those fields are still on the wire names an object that does not exist yet. `WriterRefTable` carries that marker and `write_object_depth` raises `cannot marshal recursion <type> objects` instead of emitting the reference. `test_reference_loop_code`, `test_unmarshallable` and `test_reference_loop_slice` lose their RustPython markers; `test_marshal` is 75 run, 16 skipped. Assisted-by: Claude |
||
|
Jeong, YunWon
|
7108ff0f69 |
marshal: round-trip code constants through the runtime bag (#8516)
* Keep abnormal marshal loop test aligned with CPython Assisted-by: OpenAI Codex:gpt-5.5 * marshal: retain runtime code constants while loading Decode code fields through the runtime MarshalBag so co_consts values that do not fit the compiler constant enum remain available to VM code wrappers while the compiler table receives shape placeholders. Assisted-by: OpenAI Codex:GPT-5.4 * marshal: write and read co_consts through the runtime bag serialize_code gains a serialize_code_with variant that writes each co_consts entry through a caller-supplied writer; serialize_code keeps the BorrowedConstant writer as its default. The VM writer passes its own write_object_depth, so a code constant that Literal holds but BorrowedConstant cannot describe reaches the stream instead of panicking in borrow_obj_constant, and a constant shared with the enclosing object takes an entry in the writer's reference table. PyMarshalBag implements constant_ref_from_value, bytes_from_value, str_from_value and tuple_elements_from_value, which deserialize_code_value_inner requires to decode code fields through the runtime bag. Assisted-by: Claude |
||
|
Jeong, YunWon
|
d5d19e0611 |
Fix 33 reproduced fuzzing and static-review defects (#8514)
* mmap: guard move() against dest/src past the mapping size
An offset larger than the mapping made `size - dest` underflow, producing
an out-of-range slice index panic. Check the high side first, as write() does.
Assisted-by: Claude
* exceptions: keep ImportError name/path/name_from out of an empty instance dict
ImportError.__reduce__ unwrapped get_arg(0), which is None when the exception
was constructed with no positional argument, aborting on pickle.dumps(ImportError()).
Fall back to the full args tuple there, and expose name/path/name_from as class
attributes defaulting to None so a bare ImportError() reduces to (cls, ()).
Assisted-by: Claude
* asyncio: stop unwrapping the _current_tasks downcast in current_task()
_asyncio._current_tasks is a reassignable module attribute; the three sibling
functions already degrade gracefully when it is not a dict, current_task() did not.
Assisted-by: Claude
* asyncio: raise TypeError when FutureIter.throw()'s exception class returns a non-exception
exc_type.call() goes through a Python-controlled __new__, so its result can be
any object; the downcast was unwrapped. Report it as a TypeError instead.
Assisted-by: Claude
* sequence: use a fallible reservation for sequence repetition
The guard only rejects a repeat whose per-element size crosses MAX_MEMORY_SIZE,
so (1,) * (10**12) still reached Vec::with_capacity and aborted in the allocator.
Reserve fallibly and surface a MemoryError.
Assisted-by: Claude
* collections: reject an oversized deque repetition with MemoryError
deque * sys.maxsize reached the allocator and aborted with a capacity overflow.
Apply the MAX_MEMORY_SIZE guard the other sequences already carry.
Assisted-by: Claude
* itertools: raise OverflowError for an out-of-range r argument
combinations/combinations_with_replacement/permutations narrowed r with
to_usize().unwrap(), so r=2**64 panicked instead of raising.
Assisted-by: Claude
* math: stream the generic sumprod path instead of collecting both iterables
The big-int path collected each argument into a Vec before multiplying, so an
unbounded iterable exhausted memory. Advance both iterators in lockstep and
accumulate, reporting a length mismatch when only one is exhausted.
Assisted-by: Claude
* _imp: raise TypeError for a second positional argument to find_frozen
withdata is keyword-only and unimplemented; passing it positionally hit an
unimplemented!() and aborted.
Assisted-by: Claude
* _typing: check _idfunc arity before indexing args
_typing._idfunc() with no argument indexed args[0] out of bounds.
Assisted-by: Claude
* exceptions: require a sequence for the ExceptionGroup excs argument
The argument was collected before being validated, so an unbounded iterable
such as itertools.count() exhausted memory. Reject a non-sequence up front.
Assisted-by: Claude
* mmap: treat an inverted find/rfind range as empty
find(b"x", 5, 2) built a slice whose start exceeded its end and panicked.
Assisted-by: Claude
* collections: give deque and defaultdict a GC traverse
Neither type opted into traversal, so a reference cycle through a deque or a
defaultdict was never collected.
Assisted-by: Claude
* itertools: opt the iterator types into GC traverse
cycle and its siblings hold Python references but declared no traverse, so a
cycle built through one of them leaked.
Assisted-by: Claude
* _ctypes: mask an out-of-range int instead of panicking
c_char_p(2**64) and pointer item assignment called .expect() on the narrowing
conversion. Wrap the value to the target width, which is what the C
implementation stores.
Assisted-by: Claude
* hashlib: import _hashlib when a hash module is loaded
The hash object type is a static type owned by _hashlib; calling _md5.md5()
without _hashlib imported hit an uninitialized static type and panicked.
Assisted-by: Claude
* _csv: fall back to the built-in dialect defaults when none is registered
The dialect table is empty until csv.py registers 'excel', so _csv.reader([])
unwrapped a missing entry and panicked.
Assisted-by: Claude
* builtins: reject surrogates in compile()/eval() source instead of panicking
expect_str() panics on a str containing surrogates; convert with try_as_utf8
so eval(chr(0xd800)) raises.
Assisted-by: Claude
* _suggestions: require a list for _generate_suggestions candidates
The argument was collected before validation, so an unbounded iterable
exhausted memory.
Assisted-by: Claude
* lzma: size the filter chain before consuming it
filters= was collected into a Vec before the length check, so an unbounded
iterable exhausted memory. Take the length through the sequence protocol first.
Assisted-by: Claude
* classmethod: opt into GC traverse
staticmethod already declares traverse; classmethod did not, so a cycle through
the wrapped callable leaked.
Assisted-by: Claude
* posix: reject unbounded iterables in posix_spawn and setgroups
argv, setsigdef, setsigmask and setgroups bound an ArgIterable and collected it
before validating, so an infinite generator exhausted memory. Require a
list/tuple for argv, validate signals while streaming, and take setgroups
through the sequence protocol.
Assisted-by: Claude
* _ctypes: size Array slice assignment and _argtypes_ before collecting
Both eagerly collected their argument, so an unbounded iterable exhausted
memory before the length check ran.
Assisted-by: Claude
* sys: propagate the breakpointhook warning failure
warn() was unwrapped, so an unimportable $PYTHONBREAKPOINT under
-W error panicked instead of raising.
Assisted-by: Claude
* structseq: require the sequence argument when constructing a struct sequence
A no-argument construction produced an empty backing tuple, and reading any
named field then indexed out of bounds.
Assisted-by: Claude
* Guard the hash slot dispatch against unbounded recursion
`PyObject::hash` invoked the type's hash slot directly, so an element-wise
`__hash__` following a deeply nested object graph recursed one native frame
per level and overflowed the stack. Wrap the dispatch in `with_recursion`,
matching the repr and rich-compare dispatches in the same file, so
`hash(x)` on a nested tuple/GenericAlias/slice raises `RecursionError`.
Assisted-by: Claude
* genericalias: guard the __parameters__ walk against unbounded recursion
`make_parameters_from_slice` recursed into every list/tuple argument with
nothing counting the frames, so `list[L]` for a self-referential or deeply
nested `L` overflowed the native stack at subscript time. Wrap the descent
in `with_recursion`, which makes the walk fallible; `PyGenericAlias::new`,
`from_args` and `make_parameters` now return `PyResult` and every caller
propagates.
Assisted-by: Claude
* Fix the type confusion in PyAtomicRef's Debug impl
`PyAtomicRef<T>` stores a pointer to a `Py<T>`, as `Deref`, `load_raw`,
`swap` and `Drop` all read it, but `Debug` cast it to a bare `T` and
formatted the object header as payload bytes. For `PyFunction`, whose
`code: PyAtomicRef<PyCode>` has a pointer-chasing `Debug`, that
dereferenced header words and segfaulted.
Cast to `PyObject` instead, which is what `Drop` already does and which
also covers the `PyAtomicRef<PyObject>` and `PyAtomicRef<Option<T>>`
instantiations that have no `Py<T>`.
`_asyncio._enter_task` reached this through `{:?}` in its "Cannot enter
into task" message; format the two tasks with their Python repr, which is
what the message is meant to show.
Assisted-by: Claude
* _sre: disallow instantiating Match
`Match` inherited `object.__new__`, so `M.__new__(M)` produced an instance
whose `regs`/`string`/`pattern` were never filled in by a match run; the
mapping subscript path read them and dereferenced garbage. The type has no
public constructor, so mark it `DISALLOW_INSTANTIATION`, which makes
`M.__new__(M)` raise `TypeError: cannot create 're.Match' instances`.
Assisted-by: Claude
* utils: return the empty repr instead of asserting a non-empty collection
`collection_repr` took the first element with an `.expect()` justified by the
caller's preceding non-empty check. Another thread clearing the collection
between that check and the iteration made the iterator yield nothing and
panicked the worker. Fall back to the caller-supplied empty form, which is
the text those callers already produce for an empty collection.
Assisted-by: Claude
* itertools: advance cycle's index atomically
`cycle.__next__` did `fetch_add(1)` and then reset the index to 0 in a
separate store, so two threads replaying the saved items could both read an
index past the end of `saved` and panic on the slice access. Do the advance
and the wrap in one `fetch_update`.
Assisted-by: Claude
* _ctypes: reject a float argument to a foreign call without argtypes
`conv_param`, the conversion used when `argtypes` is not set, converted its
argument with `try_int`, which goes through `__int__` and so accepts a
float. `libc.strlen(1.5)` therefore passed 1 where the callee expects a
`char *` and the callee dereferenced it. Match `ConvParam`, which does a
`PyLong_Check` and converts nothing: take the branch only for an `int` (or
a subclass, so `True` still converts), and let a float fall through to
"Don't know how to convert parameter".
The branch below it converted a float to a C double, but `try_int` claimed
every float before it could run, so it was dead; `CArgValue::Double`
existed only for that branch and both go. Typed doubles are unaffected —
they travel as `CArgValue::Typed` with code 'd'.
Assisted-by: Claude
* Report the iterator itself from PyIter's traverse
`Traverse for PyIter<O>` delegated to the inherent `PyObject::traverse` of
the object it wraps, so it reported that iterator's referents instead of
the iterator. The iterator's own reference to those referents was then
never subtracted during the collector's reference-subtraction pass, the
referents kept a non-zero gc_refs, and every object reachable from them
was classified as a root.
Any cycle running through a type with a `PyIter` field therefore survived
collection: `map`, `filter`, `zip`, `enumerate`, `reversed` and the
`itertools` iterators all leaked, while the same cycle through a `list`,
`tuple` or `list_iterator` collected. Report the wrapped object, as the
`PyObjectRef`, `PyRef<T>` and `PyStackRef` impls do.
`itertools.tee` still leaks: its shared buffer is a `PyRc<PyItertoolsTeeData>`
rather than a Python object, so the collector cannot see through it.
Assisted-by: Claude
* Remove the obsolete expectedFailure on test_code_module.test_unicode_error
Compiling a source string containing a lone surrogate now raises
UnicodeEncodeError, so the test passes.
Assisted-by: Claude
* itertools: reserve the combination indices fallibly
`combinations` and `combinations_with_replacement` built their index vector
with an infallible allocation, so an `r` that passes the ssize_t check but
does not fit in memory aborted the process instead of raising MemoryError.
Assisted-by: Claude
* Apply the struct sequence constructor's dict argument
`structseq(sequence, dict)` discarded its second argument, so the hidden
fields past `n_sequence_fields` — `tm_zone`, `st_atime` and friends — were
always None when constructed directly or restored from a `(sequence, dict)`
pickle, and a non-dict second argument was accepted silently.
Take the dict, require it to be a dict, and fill the hidden slots the
sequence did not cover from it. A key that names a field the sequence
already supplied, or no field at all, is now a
"got duplicate or unexpected field name(s)" TypeError instead of being
dropped. Both arguments are bindable by name, as `sequence` and `dict`.
`os.stat_result` and `os.statvfs_result` did not accept a second argument
at all; they and `time.struct_time` now share the parsing.
Assisted-by: Claude
* _imp: report the argument count in find_frozen's arity error
Assisted-by: Claude
* Add regression tests for the reproduced crashers
One case per catalog entry, each asserting the behavior the fix produces:
recursion guards, the memory-unsafety sites, the overflow and unbounded
allocation guards, the eager-collection rejections, the unwrap sites, and
the cycles the collector now breaks. Every expected value was checked
against CPython 3.14.
Assisted-by: Claude
* Tolerate a changed-size RuntimeError in the set repr stress test
The mutator and reader threads race on purpose; a "changed size during
iteration" RuntimeError is a valid outcome of that race and should not fail
the test. The panic it guards against is not.
Assisted-by: Claude
* Move the crash regression tests into per-module snippets
crash_regressions.py collected every reproduced crasher in one file. Split it
into the snippet for the module each case exercises, and add stdlib_gc.py,
stdlib_asyncio.py, stdlib_lzma.py, stdlib_threading_set_repr.py and
stdlib_threading_itertools_cycle.py for the cases with no existing home.
The suite runs every snippet under the host CPython too, so the checks only
RustPython raises are guarded by sys.implementation.name: the hash and
__parameters__ recursion depth, and the deque repeat overflow. The float
ctypes argument accepts either TypeError or ctypes.ArgumentError.
Assisted-by: Claude
|
||
|
Jeongseop Lim
|
7b58a8980d |
Reject extra positional arguments to float() (#8509)
* Reject extra positional arguments to float()
PyFloat::slot_new's exact-float fast path tested args.args.first(), which
succeeds for any number of positional arguments, so float(1.5, True)
returned the first one and never reached args.bind(vm). CPython's clinic
generated float_new runs _PyArg_CheckPositional("float", nargs, 0, 1)
before it fetches the first argument.
Add args.args.len() == 1 to the condition, as PyInt::slot_new,
PyStr::slot_new and PyComplex::slot_new already do; the extra argument
then falls through to args.bind(vm) and raises TypeError.
Assisted-by: Claude Code:claude-opus-5
* Bind float() arguments before the fast path
The exact-float fast path repeated float()'s argument count inline, which
duplicates what FromArgs::arity already knows. Bind first and match on the
resulting OptionalArg instead, so the positional and keyword rules stay in
one place and the bound value is reused by py_new.
Assisted-by: Claude Code:claude-opus-5
|
||
|
Jeongseop Lim
|
6cfa079950 |
Reject a fifth argument to property() (#8510)
PropertyArgs declared a fifth positional-or-keyword field, name, so the
derived arity was 0..=5 and property(None, None, None, None, None) built
a property object with the fifth argument stored in the __name__ slot.
CPython's property.__init__ is Argument Clinic generated with maxpos = 4
and the keyword list {fget, fset, fdel, doc}, so it rejects both that call
and property(name='x').
Drop the field. Nothing passed it: clone_property_with always supplied
None and copies the name separately, and the name slot is still filled by
__set_name__ and the __name__ setter.
Assisted-by: Claude Code:claude-opus-5
|
||
|
Jeongseop Lim
|
2e05364117 |
Normalize round() result to an exact int (#8511)
int.__round__ returned the receiver unchanged on the no-rounding path, so round() on an int subclass handed back an instance of that subclass: round(True) was True, not 1. CPython routes both no-op paths (ndigits is None, and ndigits >= 0) through long_long(), which reuses the reference only when the value is an exact int and otherwise copies it into a fresh one. Return zelf.__int__(vm).into_pyref(), which goes through into_exact_or() and is the same normalization the neighbouring __trunc__, __floor__ and __ceil__ already perform. Assisted-by: Claude Code:claude-opus-5 |
||
|
Jeong, YunWon
|
2a793fcdd1 |
codegen: preserve symbol tables across copied finally bodies (#8507)
Assisted-by: OpenAI Codex:gpt-5 |
||
|
Jeong, YunWon
|
d5919ce2f4 | Fix future annotation block alignment (#8506) | ||
|
이광효
|
1178841eab |
itertools: defer dropwhile predicate validation (#8504)
Store the `dropwhile` predicate as a Python object and call it while advancing the iterator. This defers callable validation until the predicate is first needed, matching CPython for empty input while preserving exception propagation during iteration. Remove the now-passing `test_dropwhile` expected-failure marker. Assisted-by: Codex:gpt-5.6-sol |
||
|
Sanghun LeeandClaude Opus 5
|
7e6eaa8e3c |
Reuse stored hashes in dict.fromkeys() (#8503)
* Reuse stored hashes in dict.fromkeys() Closes #8490. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Make fromkeys_known_hashes a PyDict associated fn --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Joshua Megnauth
|
df44fa50df |
Allow and document clippy::drain_collect (#8500)
RustPython's tail call machinery pre-allocates and reuses a vector. The code drains the vector into a new vector which is stored elsewhere. Clippy warns that this pattern causes a spurious location. Clippy is usually right that this pattern is suspect, but in this case the initial vector is reused so we want to keep the initial location. |
||
|
Jeong, YunWon
|
161ba746fc |
ssl: pass test_ssl with the rustls backend (#8502)
* Fix rustls test_ssl compatibility Assisted-by: OpenAI Codex:GPT-5 * Keep urllib3 compatible SSL version prefix Assisted-by: OpenAI Codex:GPT-5 |
||
|
Jeong, YunWon
|
84a0ba7864 |
Fix marshal recursive reference loading (#8501)
* Fix marshal recursive reference loading Create reference-tracked containers before reading their children so recursive list, dict, set, and tuple graphs can be unmarshaled. Preserve interned string markers through the runtime bag and add an initialization-only tuple construction path. Assisted-by: Codex:gpt-5 * Preserve marshal container insertion errors Keep Python exceptions raised while constructing unmarshaled sets, frozensets, and dictionaries instead of collapsing them into ValueError. This makes abnormal recursive hash-container streams report TypeError like CPython and removes the remaining test_marshal expected failure. Assisted-by: Codex:gpt-5 * Enable full abnormal marshal reference loop test * Test direct marshal tuple reference loop Assisted-by: Codex:gpt-5 |
||
|
Jeaeun Kim
|
980be72e4f |
jit: preserve return types for recursive calls (#8499)
Assisted-by: Codex:5.6-sol |
||
|
Jeaeun Kim
|
dc19be06f2 |
Fix __qualname__ of compiler-generated __annotate__ functions (#8498)
Assisted-by: Codex:5.6-sol |
||
|
Sanghun LeeandClaude Opus 5
|
856c3da0be |
Reuse stored hashes when building a set from a set/frozenset/dict (#8491)
* Reuse stored hashes when building a set from a set/frozenset/dict set and frozenset recomputed __hash__ for every element even when the source object already stored a hash per entry. CPython's set_update_internal branches on PyAnySet_Check / PyDict_CheckExact and feeds set_add_entry the hash read from the source table; RustPython always iterated generically. Split the hash computation out of the Dict entry points so callers can supply a hash they already hold, add keys_with_hashes() to hand out (key, hash) pairs, and take the fast path in the set constructors and in the set operations whose argument is a set/frozenset/exact dict. ArgIterable::as_object() exposes the pre-__iter__ object so the set operations can dispatch on the source type without changing any of their signatures. Closes #8489. dict.fromkeys() is the dict-target counterpart and is tracked in #8490, so test_do_not_rehash_dict_keys keeps its expectedFailure marker until that lands too. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Rename the hash-carrying variants to *_known_hash `_with_hash` said nothing about which direction the hash travels, and the same suffix was already used both ways in this file: keys_with_hashes() hands hashes out, while insert_with_hash() takes one in. The pre-existing insert_with_hint()/get_with_hint() pair has the same problem. Follow CPython's "KnownHash" variants (_PyDict_SetItem_KnownHash, _PyDict_Contains_KnownHash, _PyDict_DelItem_KnownHash) instead, so the suffix marks the argument direction and matches the name a reader familiar with CPython already expects. keys_with_hashes() keeps `with` because it really does return the hashes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Narrow visibility of the new helpers and trim their docs from_object() and ArgIterable::as_object() had no caller outside their own file and crate respectively, so drop them to private and pub(crate). The dict_inner helpers stay pub(crate) because builtins::set calls them. Also shorten the doc comments to match the density of the surrounding code; only insert_known_hash keeps a real note, since a wrong hash there corrupts the table silently. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
William Goode
|
267c7916b1 |
Fix map no iterables (#8478)
* map: raise TypeError when called with no iterables Assisted-by: Claude Code:claude-fable-5 * add test for map with no iterables Assisted-by: Claude Code:claude-fable-5 * removed expectedFailures * removed test for map constructed with no iterables - covered by existing in test_itertools |
||
|
Jeaeun Kim
|
1c9daa0ef6 | Add JIT support for returning None (#8479) | ||
|
강동안andJeong, YunWon
|
862797d1ce |
Add new_payload_exception helper for constructing built-in payload exceptions (#8403)
* Add new_payload_exception helper for built-in payload exceptions * Use new_payload_exception for SystemExit raise sites * Use new_payload_exception for BlockingIoError raise sites * Use new_payload_exception for OSError errno dispatch in slot_new * Run rustfmt * Run clippy * Add type guard to new_payload_exception and extract new_system_exit * Extract new_system_exit and fix stale expect messages The four SystemExit raise sites repeated the same construction, so route them through a new_system_exit helper next to new_stop_iteration. The expect messages still described invoke_exception's downcast failure, which no longer applies after moving to new_payload_exception. * Update crates/vm/src/vm/vm_new.rs --------- Co-authored-by: Jeong, YunWon <69878+youknowone@users.noreply.github.com> |
||
|
Jeaeun Kim
|
6b747b5a76 |
ssl: support explicit session reuse (#8466)
* ssl: support explicit session reuse Assisted-by: Codex-5.6-sol * ssl: support explicit TLS 1.3 session reuse Assisted-by: Codex-5.6-sol |
||
|
Jiwoo Ahn
|
6104cb159f |
wasi: fix os.environb byte handling (#8476)
Part of: #4583 This will enable -m test for wasi preview 1 Signed-off-by: Jiwoo Ahn <ikwydls1314@gmail.com> |
||
|
Joshua Megnauth
|
f89dc115d5 |
Use C string literals and w! instead of allocs (#8474)
Rust supports C string literals which automatically create a CStr with a trailing NUL. `windows-sys` provides an analogous macro for wide strings. Both of these avoid allocations which is nice for constants. |
||
|
MsfPabloandPablo Garcia
|
ecfc5cd578 |
Fix panic in code.replace() with non-interned strings (#8471)
code.replace() called as_interned_str().unwrap() on its string arguments, which panics when the caller passes a string that has not been interned. modulefinder's replace_paths_in_code() builds a fresh co_filename, so any use of ModuleFinder(replace_paths=...) aborted the interpreter. Intern the incoming strings instead, and raise TypeError rather than panicking when a non-string appears in co_names/co_varnames/ co_cellvars/co_freevars. Unskips test_modulefinder.test_replace_paths. Co-authored-by: Pablo Garcia <pablito@MacBook-Neo-de-Pablo.local> |
||
|
chestnut1717
|
d6161ac321 |
Fix EOF SyntaxError diagnostics (#8429)
* Fix EOF SyntaxError diagnostics * Fix Scope EOF SyntaxError location conversion * Fix Additional Unexpected Test * Docs Add Annotation source_location_in_code_points() |
||
|
Seonghun An
|
33d98d122e |
Implement negative zero coercion format option (#8475)
Assisted-by: Codex:gpt-5 |
||
|
Jeong, YunWon
|
ee94b5b723 |
sre_engine: resume the tail match at prefix_skip, not past the whole prefix (#8473)
`search_info_literal`'s len>1 arm reset the tail-match cursor by advancing one character past the matched prefix. That position equals the INFO block's prefix_skip boundary only when the prefix ends where the skip does; for a pattern like `ab(cd)`, `_get_literal_prefix` reports prefix_len=4 with prefix_skip=2, so the tail resumed two characters too far. `state.cursor` already holds `req.start + skip`, which is what `sre_lib.h` SRE(search) computes as `ptr - (prefix_len - prefix_skip - 1)`, and which the len==1 arm above already uses. Take it unconditionally. Searching "xabcdcd" for `ab(cd)` returned span (1, 7) instead of (1, 5); over a 600-case corpus stamped with CPython's own answers, 94 cases disagreed before this change and none after. The wrong resume also dropped matches outright, not only widened them. Assisted-by: Claude |
||
|
Joshua Megnauth
|
15f8590f17 |
host_env: Simplify stat, fstatat with Rustix (#8455)
According to POSIX, `time_t` should be 64 bits. `musl` changed its `time_t` to an `i64` over five years ago. `glibc` provides compatibility features that declare `time_t` as either `i32` or `i64`. Rustix uses the raw Linux syscall for stat which returns an `i64`. For our purposes, an `i64` makes sense because it's modern and avoids the year 2038 problem. It also reduces our dependency on what `libc` defines. Sources: * https://www.man7.org/linux/man-pages/man3/time_t.3type.html * https://pubs.opengroup.org/onlinepubs/9799919799/basedefs/sys_types.h.html |
||
|
Sumi Jeong
|
7545bb1993 |
Fix locals()/vars() corrupting/leaking __conditional_annotations__ outside function scope (#8467)
CPython's _PyFrame_GetLocals only syncs fastlocals/cells into locals() for function scope; module/class scope just returns the namespace dict as-is. sync_visible_locals_to_mapping did this for every scope, so the implicit __conditional_annotations__ cell (used for PEP 649/749 deferred annotations) broke two ways: at module scope its cell is always empty, so syncing overwrote the dict's real value with None, causing NameError on the next annotated statement; at class scope its cell is the only real value, so syncing leaked it into locals()/dir(), unlike CPython. Skip cell/free slots outside function scope to match CPython, and add a regression snippet. Closes #8379 Assisted-by: Claude Code:claude-sonnet-5 |
||
|
lms0806
|
25e2df1731 | Implement TclObject support in tkinter varname_converter (#8465) | ||
|
강동안
|
85398d2f70 |
Specialize list.sort() comparisons for homogeneous lists (#8464)
* Use type-specialized comparators for homogeneous list sorts Scan the sort keys once before sorting; when every element is exactly str, int, or float, compare wtf8 bytes / BigInt / f64 directly instead of going through rich_compare_bool dispatch, mirroring CPython's pre-sort check in listsort.c (unsafe_latin_compare and friends). Subclasses and mixed-type lists keep the generic __lt__ path. * Add edge-case tests for specialized list sorts * Cache the richcompare slot for homogeneous list sorts * Specialize tuple sorts on their first elements * Apply ruff formatting to the new list sort tests * Deduplicate the reverse swap across sort dispatch arms |