mirror of
https://github.com/RustPython/RustPython.git
synced 2026-10-28 18:59:04 +08:00
322f294a2179bad4f353aaa42ea9592eef068ced
4894
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Jeong, YunWon
|
322f294a21 |
common: add VM-independent codec and text engines (#8622)
* common: add PyPy CJK codec engines
Assisted-by: Codex:GPT-5
* common: mark CJK mapping tables generated
Assisted-by: Codex:GPT-5
* common: add reusable byte and text engines
Assisted-by: Codex:GPT-5
* common: record opt-in engine dependencies
Assisted-by: Codex:GPT-5
* stdlib: add `_multibytecodec` and the six `_codecs_*` modules
Wire the CJK codec engines in `rustpython-common` up to the VM. `_multibytecodec`
holds `MultibyteCodec`, `MultibyteIncrementalEncoder`, `MultibyteIncrementalDecoder`,
`MultibyteStreamReader` and `MultibyteStreamWriter`, the encode/decode drivers and
the error-handler protocol; each `_codecs_*` module exposes `getcodec` for the
encodings of its region.
`rustpython-stdlib` now enables the `cjk-codecs` feature of `rustpython-common`.
Add `cjk::decode_reset`, which the decoder `reset()` methods need. It differs from
building a fresh decoder state: ISO 2022 keeps its G1..G3 designations and the
escape-throughout flag, and resets only G0 and the shifted flag.
Drop the markers from the CJK codec tests and from the `test_codecs`, `test_io`,
`test_email`, `test_zipfile` and `test_traceback` cases that needed a CJK codec.
`test_codecs.BasicUnicodeTest.{test_basics,test_decoder_state}` stay expected
failures for an unrelated reason: `codecs.charmap_encode` rejects a `None` mapping.
Assisted-by: Claude Code:claude-opus-5
* stdlib: use the byte and text engines from `rustpython-common`
`binascii`, `_json` and `_socket`'s address converters now call the engines in
`rustpython-common` instead of carrying their own copies, and `rustpython-stdlib`
enables the `binascii`, `inet` and `json` features for them. That leaves
`base64` and `crc32fast` with no caller in `rustpython-stdlib`, and
`crates/stdlib/src/json/machinery.rs`, the last of the json_in_type-derived code,
goes away with its last caller.
The address converters change behaviour. `inet_aton` reads the lenient forms
(`127.1`, `16909060`, `0x7f.1`, `0177.0.0.1`), `inet_pton` rejects an octet with a
redundant leading zero and a group of five hex digits, `inet_ntop` writes the
IPv4-compatible form as `::c000:201`, and an address family with no converter
raises `EAFNOSUPPORT` instead of a bare `OSError`.
Give the JSON engine the closing-quote scan and the allocation-free `\uXXXX`
escape it needs to carry `_json`'s workload. Against the code it replaces,
`json.dumps` runs 2.3x faster with `ensure_ascii=True` and 1.7x without it, and
`json.loads` 1.2x.
`binascii.a2b_base64` now reports the messages `binascii.c` does: no
`error decoding base64: ` prefix, and the data-character count in parentheses.
Assisted-by: Claude Code:claude-opus-5
* test: enable the Windows cp932 codec case
The shared CJK engine now satisfies the Windows code-page adapter, so keeping this test marked as an expected failure turns its success into a CI failure.
Assisted-by: Codex:GPT-5
|
||
|
Hanjeong Lee
|
d1bbf4bc1a |
Report every generated __text_signature__, and keep it out of __doc__ (#8614)
* Strip the signature prefix from builtin __doc__
A function's signature and its documentation share one string, separated
by `\n--\n\n`, as in CPython. __text_signature__ read the front of it,
but __doc__ returned the whole thing:
>>> len.__doc__
'len(obj, /)\n--\n\nReturn the number of items in a container.' # was
'Return the number of items in a container.' # now
44 of the 46 builtin functions leaked the prefix.
get_doc_from_internal_doc already stripped it but was reachable only from
PyType.__doc__. Split it the way CPython does, into doc_without_signature
(_PyType_DocWithoutSignature) and a wrapper mapping an empty result to
None (_PyType_GetDocFromInternalDoc), and call it from PyNativeFunction
and PyMethodDescriptor as well.
Assisted-by: Claude Code:claude-opus-5
* Terminate a generated __text_signature__ that carries no documentation
A signature and its documentation share one string, separated by
`\n--\n\n`, and readers find the signature by searching for that marker.
A #[pyfunction] with no doc comment stored the bare signature, which no
reader could parse:
>>> time.clock_getres.__doc__
'clock_getres(clk_id, /)'
>>> time.clock_getres.__text_signature__
None
Emit the marker in that case too, as Argument Clinic does for an
undocumented function. 136 of the 915 functions reachable from the
importable modules were affected.
__doc__ now reports None for them, matching CPython, because nothing
follows the marker.
Assisted-by: Claude Code:claude-opus-5
* Report a __text_signature__ for every method
A #[pymethod] built its doc only when the method carried a doc comment,
so one without a comment dropped the generated signature too:
>>> list.append.__text_signature__
None
Build it from either part, as #[pyfunction] does. All 1101 method
descriptors reachable from the builtin types now report a signature, up
from 1, and 713 of the 1030 that CPython also describes match it exactly.
A method that takes its receiver as an ordinary first argument instead of
`&self` reported that argument by name, so binding the method could not
drop it:
>>> inspect.signature([].__dir__)
(obj, /) # was
() # now
Mark it $self, or $type for a classmethod, as CPython does. 310 methods
were affected.
test_unbound_builtin_method_noargs and test_bound_builtin_method_noargs
pass now.
Assisted-by: Claude Code:claude-opus-5
* Drop the expectedFailure on test_simple_completion
rlcompleter closes the parenthesis for a callable whose signature takes
no parameters, and leaves it open when the signature cannot be read:
def _callable_postfix(self, val, word):
if callable(val):
word += "("
try:
if not inspect.signature(val).parameters:
word += ")"
except ValueError:
pass
os.getpid reports `()` since the generated signature carries the `--`
terminator, so completing `os.getpid` yields `os.getpid()` and the test
passes.
Assisted-by: Claude Code:claude-opus-5
* Name every #[pymethod] receiver zelf
A method that needs an owned handle cannot take `&self`, so it takes the
receiver as its first argument and names it `zelf`, `self` being a Rust
keyword. 24 of the 163 such methods named it something else: `instance`
(11), `obj` (4), `exc` (3), `_self` (3), `_instance` (2).
Rename them. The generated signature is unaffected either way, since the
macro marks the receiver by position rather than by name.
Assisted-by: Claude Code:claude-opus-5
* Drop the unreachable zelf name check in func_sig
The receiver is now marked by position, so the name it carries no longer
matters, and no path reaches this check with an argument named zelf:
- a #[pymethod] or #[pyclassmethod] without a `&self` receiver has its
first argument replaced by the marker before the name is read
- a `&self` receiver is handled as syn::FnArg::Receiver
- a #[pymethod(raw)] is passed to static_raw_func, whose PyNativeFn bound
fixes its signature to (&VirtualMachine, FuncArgs)
- no #[pystaticmethod] or #[pyfunction] names an argument zelf
- #[pymember], #[pygetset] and #[pyslot] never reach func_sig
Every generated signature is byte for byte identical without it.
Assisted-by: Claude Code:claude-opus-5
* Keep the argument bundle when marking the receiver
A #[pyclassmethod] that takes FuncArgs receives the class inside the
bundle along with every other argument, but the marker was spent on the
bundle and the arguments went unreported:
>>> object.__subclasshook__.__text_signature__
'($type, /)' # was, claims it takes nothing
'($type, *args, **kwargs)' # now
Report both, the shape CPython uses for __new__. Two methods take their
arguments this way, __subclasshook__ on object and on type.
Assisted-by: Claude Code:claude-opus-5
|
||
|
Jeong, YunWon
|
5b1a1c08d7 |
Leave a byte that starts no UTF-8 sequence at the end of a console write (#8621)
* Leave a byte that starts no UTF-8 sequence at the end of a console write `find_last_utf8_boundary` read every byte at or above 0xc0 as the start of a sequence, giving 0xf8..=0xff an expected length of 4 and cutting the tail back to before it. `_find_last_utf8_boundary` stops the four-byte range at 0xf8 and returns the full length for anything above it, leaving the byte for MultiByteToWideChar to answer with U+FFFD. `write_console_utf8` therefore reported fewer bytes written than it was given whenever one of those bytes fell in the last three, and for a buffer that began with one it cut the length to 0, converted nothing and returned Ok(0) -- a write that never advances. The scan also ran a fourth iteration the C loop does not; the arms happen to answer the same there, but the bound is now the same 3. * Drop the expectedFailure on test_winconsoleio's test_write The marker went stale with the boundary fix in the parent commit. The test writes `b'\xff'*10` and `data + b'\xff'` to CONOUT$ and asserts the returned count, which is the case `find_last_utf8_boundary` was answering wrong, so regrtest now reports it as an unexpected success. |
||
|
Lee Dogeon
|
578217e1f6 |
Fix tuple subclass preservation in type bases (#8420)
* Fix tuple subclass preservation in type bases Assisted-by: Codex:gpt-5.6-sol * Refine type bases representation Store validated base classes in the original typed tuple, while retaining a bootstrap vector only until a Python tuple can be materialized. Assisted-by: gpt-5.6-sol Assisted-by: Codex:gpt-5.6-sol * Bootstrap tuple type with type hierarchy Initialize tuple and the canonical empty tuple alongside type and object so every type stores an actual typed bases tuple from construction. Remove the bootstrap bases representation and keep raw initialization Miri-safe. Assisted-by: Codex:gpt-5.6-sol |
||
|
김은빈
|
8b65a5455f | feat: reuse weak callable proxy (#8618) | ||
|
Jeong, YunWon
|
2c8d3ce50b |
Implement PEP 667 frame locals semantics (#8602)
* Implement PEP 667 frame locals semantics Assisted-by: Codex:GPT-5 * Fix C API test CString import Assisted-by: Codex:GPT-5 * Address PEP 667 review findings Assisted-by: Codex:GPT-5 |
||
|
Jeong, YunWon
|
7c6c78f955 |
Implement CPython 3.14 subinterpreter APIs (#8605)
* Add _interpreters and _interpchannels modules
Apply the patch baseline for the low-level subinterpreter modules:
- crates/vm/src/stdlib/_interpreters.rs: create/destroy/list_all/get_main/
get_current/is_running/exec/call/run_string/run_func/set___main___attrs/
is_shareable/whence and the Interpreter*Error exceptions
- crates/vm/src/stdlib/_interpchannels.rs: channel create/destroy/send/recv/
list_all/list_interpreters/release/close and the Channel*Error exceptions
- crates/vm/src/vm/crossinterp.rs: cross-interpreter data protocol
- crates/vm/src/vm/{interpreter,mod,runtime,vm_new}.rs: interpreter registry
and lifecycle plumbing the modules need
- Lib/concurrent/interpreters/: high-level PEP 734 package
Assisted-by: Claude:claude-opus-5
* Correct _interpreters and _interpchannels against CPython 3.14
Argument matching:
- Add `function::ArgSpec`, a `PyArg_ParseTupleAndKeywords` equivalent that
applies the `|`, `$` and `:` markers of a format string to a `kwlist`, and
route every module function of both modules through it. This enforces
keyword-only parameters, rejects unexpected keywords, accepts keyword forms
that were previously positional-only (`whence(id=)`, `get_config(id=)`,
`set___main___attrs(id=, updates=)`, `capture_exception(exc=)`,
`_register_end_types(send=, recv=)`), and reproduces the arity messages.
- `O!` slots (`shared`, `updates`, `call`'s `args`/`kwargs`) now reject None,
and `_PyArg_BadArgument` renders None as "None" rather than "NoneType".
- `new_config` takes at most one positional `str`; `list_all` of
`_interpchannels` is argument-less.
- Convert arguments in `kwlist` order, so an argument's own error is raised
ahead of the checks that follow it.
_interpreters:
- Add the `CrossInterpreterBufferView` type and build the received memoryview
on it, keeping the sending interpreter's exporter out of the destination.
- Build `excinfo.errdisplay` from `traceback.TracebackException`.
Feature flags:
- `os.fork` checks finalization before `allow_fork`, matching `os_fork_impl`.
- `_thread.start_new_thread` and `start_joinable_thread` reject an interpreter
without `allow_threads`.
Lib/concurrent/interpreters/_crossinterp.py: restore the stripped docstrings
and comments.
Assisted-by: Claude:claude-opus-5
* Correct the stateless checks and error shapes of _interpreters against CPython 3.14
verify_stateless_function follows _PyFunction_VerifyStateless: it rejects
non-dict builtins, a non-empty __defaults__, __kwdefaults__ or __closure__,
and code whose LOAD_GLOBAL names are held by the function's globals or
absent from its builtins.
code_returns_only_none follows _PyCode_ReturnsOnlyNone: generator,
coroutine and async-generator code is rejected up front, the instruction
walk skips inline caches and maps specialized and instrumented opcodes
back, and the LOAD_CONST preceding each RETURN_VALUE is compared against
the index of None in co_consts.
ExcInfo::capture keeps an empty exception message instead of dropping it,
and builds `formatted` as `module.qualname: msg`, leaving out the builtins
and __main__ modules.
_interpreters.call packs the callable through _PyFunction_GetXIData before
pickle and re-raises the stateless-check failure when both fail. A func,
args or kwargs that cannot be rebuilt in the target, and a result without
cross-interpreter data, now raise NotShareableError with the snapshot as
the cause rather than being returned as excinfo.
_interpreters.capture_exception() without an argument returns None.
pickle_loads and _PyFunction_GetXIData attach the underlying failure as
__cause__.
channel_send converts the object to cross-interpreter data after the
channel's `closing` check.
ChannelID rich comparison returns the result of comparing its id with the
other number instead of coercing that result to bool.
PyMemoryView_FromObjectAndFlags raises "memoryview: a bytes-like object is
required, not 'X'" for an object that is not a buffer.
Assisted-by: Claude:claude-opus-5
* Correct cross-interpreter data conversion and channel argument parsing
Bound shareable ints by isize, raising OverflowError("try sending as
bytes") outside that range and keeping it as the NotShareableError cause.
Report an unshareable object by its repr, and attach a failing conversion
as __cause__ of the NotShareableError raised for the object it was called
for, so each level of a nested tuple appears in the chain.
Guard each tuple item conversion with with_recursion("while sharing a
tuple").
Gate the memoryview getdata function on `_interpreters` having been
imported, and route channel_send_buffer through a memoryview so it uses
that function and the resolved fallback.
Create the cross-interpreter exception types from the module_exec of
either module that raises them, instead of only `_interpreters`.
parse_cid raises OverflowError("int too big to convert"); int_arg
reproduces the `i` converter's three overflow messages and is also used
while parsing channel_send arguments.
Add BASETYPE to _queue.Empty, and gate nt.execv/nt.execve on allow_exec.
Assisted-by: Claude:claude-opus-5
* Correct interpreter ID parsing and the channel_send defaults lookup
_PyInterpreterState_ObjectToID accepts any object with __index__, raises
OverflowError("int too big to convert") outside the int64 range, and
reports a negative ID with the repr of the original object.
channelsmod_send reads the channel's default unboundop and fallback only
when one of the two arguments is negative, so an explicit pair is
validated before the channel is looked up.
Assisted-by: Claude:claude-opus-5
* Add the _interpqueues module and concurrent.interpreters._queues
_interpqueues holds a process-wide queue table and exposes create,
destroy, list_all, put, get, bind, release, get_maxsize,
get_queue_defaults, is_full, get_count and _register_heap_types, raising
QueueError and QueueNotFoundError. QueueEmpty and QueueFull, which
subclass queue.Empty and queue.Full, are registered per interpreter by
concurrent.interpreters._queues.
Cross-interpreter data carries a queue as a refcounted queue ID that
binds when the data is captured and releases when it is dropped, and
is_shareable reports a registered Queue as shareable.
Queue and channel items owned by an interpreter are now cleared from
Interpreter::finalize, after module finalization, rather than from
destroy_owned_interpreter, so values sent by an atexit callback also
become unbound.
is_shareable gates memoryview on the same registration flag as the
getdata lookup.
Assisted-by: Claude:claude-opus-5
* Add the high-level concurrent.interpreters API
Interpreter, ExecutionFailed and the queue aliases, copied verbatim from
CPython 3.14.
Assisted-by: Claude:claude-opus-5
* Destroy leftover subinterpreters when the main interpreter finalizes
Interpreter::finalize now runs finalize_subinterpreters between the atexit
handlers and the finalizing flag: when the main interpreter still owns
subinterpreters it emits the RuntimeWarning "remaining subinterpreters;
close them with Interpreter.close()" and then finalizes each of them.
runtime::owned_interpreter_ids lists the runtime-owned interpreters.
Assisted-by: Claude:claude-opus-5
* Absolutize the script path in run_file
_Py_abspath and _PyPathConfig_ComputeSysPath0 arrive as abs_path and
script_sys_path0. run_file hands the absolutized path to get_importer
and to the runner, so __file__ and co_filename no longer stay relative,
and inserts the symlink-resolved script directory as sys.path[0]
instead of the argument's parent.
test_cmd_line_script.test_script_abspath no longer expects a failure.
Assisted-by: Claude:claude-opus-5
* Name types in error messages the way tp_name does
The object, number, sequence and mapping protocols, the
unsupported-operand helpers, and the type-specific messages fed from
them now read PyType::slot_name() where they read PyType::name(),
so a type declared with a module keeps it.
Message texts corrected along the way:
- PySequence_Size, PyMapping_Size, PySequence_GetItem and
PySequence_SetItem/DelItem split their merged wording in two, and
PyObject_SetItem/DelItem say "object".
- PyObject_GenericSetAttr reports a read-only attribute and the
missing __dict__.
- check_class drops the type name it appended.
- __int__ and __index__ drop the class prefix; __float__ keeps it.
- float(), complex(), round(), next(), str subscripting and list
subscripting take their own wording.
- str.join and bytes.join number the item they turn down and name what
they wanted, and report a non-iterable as "can only join an
iterable".
test_class.testObjectAttributeAccessErrorMessages,
string_tests.test_subscript and the test_descrtut doctest no longer
expect a failure.
Assisted-by: Claude:claude-opus-5
* Gate script_code's source compilation on the compiler feature
crossinterp::script_code called vm.compile unconditionally, which only
exists with the rustpython-compiler feature. Builds without it, such as
example_projects/barebone, now raise a TypeError instead of failing to
compile.
Assisted-by: Claude:claude-opus-5
* Skip test_pyinit_function_raises_exception
The test imports _testsinglephase in its body without a decorator
guarding it, and now that _interpreters exists requires_subinterpreters
no longer skips it.
Assisted-by: Claude:claude-opus-5
|
||
|
Jeong, YunWon
|
6658ffa717 |
memoryview: traverse the view for the collector, and name types as tp_name does (#8608)
* memoryview: walk the view and the buffer wrapper for the collector
Neither `memoryview` nor the `_buffer_wrapper` a `__buffer__` produces was
traversed, so the object a view looks at was never reported and a cycle
running through one was classified as reachable:
class Src:
def __buffer__(self, flags): return memoryview(b'abcdef')
o = Src(); o.keep = memoryview(o) # o is never freed
Both now report what they hold, as `memory_traverse` and `mbuf_traverse` do.
Assisted-by: Claude
* Name the type in an unhashable-type message the way `tp_name` does
The message took the short name, so a type defined in a module lost the
module: `hash(array.array('B', b'a'))` said `unhashable type: 'array'`
where the name is `array.array`, and `collections.deque` said `'deque'`.
`slot_name()` is what `tp_name` holds, and it leaves a class defined in
Python reported by its own name as before.
Assisted-by: Claude
|
||
|
Jeong, YunWon
|
f9b177719b |
Fix formatting, numeric, unicode, codec, complex and marshal mismatches found by differential testing (#8600)
* Fix %-formatting of bytes precision, width and %c errors
`format_bytes` subtracted the byte length from the field width without
saturating, so `b'%1s' % (b'abcde',)` aborted with a capacity overflow.
A precision written as a bare dot now truncates to nothing, and `%r`/`%a`
on bytes now honour the precision and width like the other conversions.
`%c` with a bytes, bytearray or str argument that is not one unit long
now names the argument by its length instead of by its type.
Assisted-by: Claude Code:claude-opus-5
* Fix format spec handling of grouping, bool, 'c' and complex
A grouping option next to an unrecognised presentation type now reports
"Cannot specify ',' with 'a'." instead of falling through to the type
check. `format(True, spec)` now validates the spec the way an int does.
The 'c' type now pads and aligns its character, and an integer outside
the C long range raises OverflowError rather than reporting a code point
out of range.
A complex formatted with the 'n'/'g' types or with a precision and no
type now goes through the general format without forcing a fraction.
Assisted-by: Claude Code:claude-opus-5
* Round floats at a negative ndigits from the exact decimal digits
Dividing by a power of ten rounded twice: once when the quotient was
stored and again at the tie, so e.g. `round(2.5e2, -2)` came out as 300.0.
The digits now come from the exact decimal expansion of the value.
Assisted-by: Claude Code:claude-opus-5
* Report the requested base in int literal errors and reject a bare sign
An invalid literal parsed with base 0 named the base that the prefix
turned out to mean rather than 0. A string of nothing but a sign was
taken for the old octal form and returned 0 instead of raising.
Assisted-by: Claude Code:claude-opus-5
* Split, strip and expand tabs over the Py_UNICODE_ISSPACE set
`str.split`, `rsplit`, `strip`, `lstrip` and `rstrip` with no separator
used Rust's whitespace predicate, which differs from the 29 code points
`Py_UNICODE_ISSPACE` names: U+001C-U+001F and U+0085 were skipped and
U+200B was treated as a separator.
`expandtabs` went through `str`, so it dropped the lone surrogates of a
WTF-8 string; it now works on `Wtf8` throughout.
Indexing a str out of range now says "string index out of range".
Assisted-by: Claude Code:claude-opus-5
* Read Unicode numeric values as exact rationals
`DerivedNumericValues.txt` field 1 carries a rounded decimal, so
`unicodedata.numeric('⅐')` and its neighbours came back short of the
value CPython reports. The generator now reads the rational in field 3,
and takes the 3.2 values from field 8 of `UnicodeData-3.2.0.txt`.
Assisted-by: Claude Code:claude-opus-5
* Return a mappingproxy from Pattern.groupindex
`pattern_groupindex` wraps the group index in a dict proxy, and hands
back a plain dict only when the pattern has no named group.
Assisted-by: Claude Code:claude-opus-5
* Fix the utf-16/32/7 codecs' error handling
The utf-32 decoder dropped a code point above U+10FFFF instead of calling
the error handler, and both the utf-16 and utf-32 handlers were told the
encoding was "utf-16"/"utf-32" whichever byte order the data was in.
The encoders now take a bytes replacement from the handler as it stands,
rejecting one that does not fill whole code units, rather than iterating
over it as text; a str replacement from the utf-16 encoder now splits an
astral code point into a surrogate pair.
The utf-7 decoder names its encoding "utf7", as its codec does.
Assisted-by: Claude Code:claude-opus-5
* Rework complex multiplication, division and power
Division now divides by whichever part of the divisor is larger, the way
`_Py_c_quot` does, and both it and multiplication recover the infinities
that the plain formulas turn into nan, following C11 Annex G. An operand
that is a real number is now multiplied and divided component-wise, which
keeps the sign of a zero and the other part of an infinity; a real
numerator has its own quotient so that its zero keeps a sign too.
An integer exponent no greater than 100 in magnitude is now reached by
repeated multiplication rather than through the polar form, so e.g.
`(1+2j) ** 2` is exact. A zero exponent gives 1, and an exponent with no
imaginary part no longer reaches for a logarithm that turns an infinite
magnitude into a nan. A result that came out infinite raises
OverflowError.
`complex() arg is a malformed string` is replaced by `could not convert
string to complex: ...` when the string carries a misplaced underscore,
`complex division by zero` by `division by zero`, and the message for
zero raised to a negative or complex power no longer names the base.
Assisted-by: Claude Code:claude-opus-5
* Intern the string constants that read as names
A code object's string constants were interned by length, so a short
string that is not a name was interned and a long name was not. They are
now picked out the way `all_name_chars` does. The one-character latin-1
strings are interned as well, so that every route to one of them lands on
the same object.
Assisted-by: Claude Code:claude-opus-5
* Honour the marshal version and separate its EOF errors
Versions below 2 now write a float or complex as its digits under
TYPE_FLOAT/TYPE_COMPLEX, versions from 4 on write a tuple of fewer than
256 items under TYPE_SMALL_TUPLE and a long ascii string under
TYPE_ASCII, and a string is marked interned only when it is.
`marshal.loads` now tells apart the three ways its data can run out:
"EOF read where object expected" where an object was to start, "EOF read
where not expected" where a single byte was, and "marshal data too short"
where a run of bytes was cut short.
Assisted-by: Claude Code:claude-opus-5
|
||
|
Jeong, YunWon
|
0ae835a073 |
Give heap types a dict namespace (#8606)
Replace `PyType::attributes` (`PyRwLock<PyAttributes>`) with a `TypeNamespace` enum. A type created while an interpreter is running holds a `PyDict`; the types `Context::genesis` and `PyType::new_static` build keep the interned-key `IndexMap`, since hashing a string needs a VM. `type.__new__` binds `__classdictcell__` to the type's own namespace rather than the namespace dict passed to it, so an annotation scope reads attribute changes made after the class body ran. Removes the `expectedFailure` on `test_type_params.TypeParamsClassScopeTest.test_modified_later`. `PyGetSet` holds its class as a `PyRef<PyType>` instead of a non-owning `PointerSlot`, and traverses it, because a namespace can outlive the type it belongs to. `PointerSlot` and the `unsafe` on `Context::new_getset` are gone. `subs_parameters` raises `TypeError` when `__typing_subst__` returns a non-tuple in an unpack position, using the new `PyType::fully_qualified_name` for the message. Removes the `expectedFailure` on `test_typing.GenericTests.test_return_non_tuple_while_unpacking`. Assisted-by: Claude |
||
|
Shahar Naveh
|
fab3fdb704 |
Update some libs and tests to 3.14.7 (#8603)
* `typing.py` * tempfile * tarfile * imaplib |
||
|
Shahar Naveh
|
0fdd061ed4 |
Update some tests & libs to 3.14.7 (#8595)
* test_memoryio * test_ordered_dict * more * urllib * test_compileall * urllib test * more * test_descr * test_enumerate |
||
|
Joshua Megnauth
|
895be40542 |
Update test_unicodedata to 3.15 (#8598)
See: #8548 |
||
|
Shahar Naveh
|
a6d88b144e |
Update some libs and tests to 3.14.7 (#8593)
* test_weakref * test_inspect * socket * test_copy * pickle * test_ast * string * zstd * test_py_compile * test_os * mark failing test * mark failing * fix markers |
||
|
Luan Taraschi
|
782d5d2639 |
Warn instead of aborting when a class namespace holds a non-string key (#8581)
* Warn instead of aborting when a class namespace holds a non-string key
type() took the whole process down when the namespace it was handed had
a key that was not an exact str:
>>> type("MyClass", (), {1: 2})
thread 'main' panicked at crates/vm/src/builtins/dict.rs:781:65:
dict has non-string keys: [PyObject PyInt { value: 1 }]
to_attributes called expect() on downcast_exact, so a str subclass hit
the same line, and that one names an attribute as well as a str does.
PyAttributes is keyed by interned strings and can hold nothing else, so
a str subclass is now interned like any other name, and a key that is no
kind of string is skipped, with the caller deciding what the skip means.
type() warns once per class, the way CPython does, and
_thread.start_new_thread raises the TypeError an ordinary call already
raises for the same mapping.
The key itself still does not land in the class dict, which is what
test_type_lookup_mro_reference needs, so that one stays skipped with the
reason rewritten.
Assisted-by: Claude Code:claude-opus-5
* Mark test_type_lookup_mro_reference as an expected failure
It aborted the runner before, which is why it was skipped. Now that it
only fails, the run can carry it: five runs in a row report it as an
expected failure, and it cannot turn into a pass while the class dict
drops the key its lookup needs.
Assisted-by: Claude Code:claude-opus-5
|
||
|
Joshua Megnauth
|
9b6feb5132 |
unicodedata: Fix Bidirectional, improve parser (#8548)
We can save space by calculating the diffs between modern Unicode and 3.2.0 and storing membership information where valid. This avoids storing entire tables for 3.2.0, and is also more correct in the long run since it handles absence from 3.2.0 correctly. I switched over Bidi to this new method which partially fixed the test. Unfortunately, our Unicode data is more up to date than Python 3.14 so the test fails for modern Unicode. I updated the test to 3.15 as per review, and now our bidi passes the test. |
||
|
Jeong, YunWon
|
f87cfe7e7a |
Match CPython on code objects, ast, symtable and syntax error positions (#8580)
* Guard all/any/tuple(genexpr) calls when the generator is a coroutine
`maybe_optimize_function_call()` reserves a `skip_optimization` label for
every `name(genexpr)` call and, for `all`, `any` and `tuple`, emits an
identity guard against the builtin plus an inlined loop. The port skipped
both whenever the generator expression's symbol table was a coroutine, so
`all(await x for x in xs)` and `all(x async for x in xs)` compiled to a
plain call.
An `await` or an `async for` in the generator does not disqualify the
shape: the inlined `FOR_ITER` raises the same `TypeError` that calling the
builtin on an async generator raises, which is what
test_builtin test_builtin_call_async_genexpr_no_crash asserts.
This was the last code-level difference against CPython across `Lib/`:
comparing every file's code tree by structure, opcode and constant value
(qualnames aside, which moved in 3.14.3+, and set constants compared by
value rather than by hash order) now matches on 1720 of 1720 comparable
files, up from 1719.
Assisted-by: Claude Code:claude-opus-5
* Name the ellipsis type `ellipsis`
`type(...).__name__`, its `__qualname__`, and `types.EllipsisType.__name__`
read `EllipsisType`.
Assisted-by: Claude Code:claude-opus-5
* Align compiler front end and instruction positions with CPython
Reject a bare generator expression in a class header and a format spec
nested more than two deep. Both parse here; the reference grammar has no
rule for either, since a class header only takes `arguments` and the
tokenizer runs out of nesting levels at the third format spec.
Scan the interpolations a format spec carries inside another format spec,
so `f"{x:{y:{z}}}"` reaches `z` when building the symbol table.
Give every generic function's type params scope a `.defaults` slot,
whether or not the signature has any default, so `co_varnames` and
`co_nlocals` match. Classes and type aliases keep theirs empty.
Decide an unparenthesized sole generator argument by the parser's
`parenthesized` flag instead of scanning the source for the shape of its
element. `_sum((d := x - c) * d for x in data)` opens with a parenthesized
group the rest of the expression continues, which the scan read as the
generator's own parentheses and left the call's out of its range.
Drop empty f-string fragments unconditionally. A concatenation whose
fragments are all empty now loads its empty string at the whole
concatenation instead of at whichever fragments happened to be plain
string literals.
Keep a byte order mark in source handed over as text, and report it as an
invalid non-printable character. Reading a file to freeze strips it, as
the encoded-bytes path already did.
Carry the filename into the errors `compile(..., PyCF_ONLY_AST)` raises;
they reported `<unknown>` or an empty name.
Across `Lib/`, the 1729 files that compile on both sides now agree on
opcodes, constants, flags, variable names, names, and exception tables,
and on instruction positions in all but 7 files, where a compound
statement whose body ends in a trailing `;` ends one column short.
Assisted-by: Claude Code:claude-opus-5
* Align ast module output with CPython
- Keep lone surrogates in string, f-string and t-string literal values by
reading such a literal from its source; the literal-value helpers move
out of `Compiler` into `rustpython_codegen` and `ConstantLiteral::Str`
holds `Wtf8Buf`.
- Give `{expr=}` interpolations their leading text as a Constant and
default their conversion to `repr`, in both f-strings and t-strings.
- Span folded consecutive literals from the first of them to the last.
- Take the `:` into a format spec's range only when the parser left it
out, instead of always widening by one.
- Report a node end that lands on a line start as that line, column 0.
- Give a generator expression written straight into a call's parentheses
the range of those parentheses, in the preprocess pass, replacing
codegen's source scan and `_ast`'s one-column adjustment.
- Drop the expectedFailure on test_ast_line_numbers_with_parentheses.
Assisted-by: Claude Code:claude-opus-5
* Align symbol table output with CPython
- Register an import as `DEF_IMPORT` alone, and an annotated parameter as
a plain parameter, so `import x; global x` is accepted and annotated
parameters are not reported as annotated names.
- Record a `global` declaration in the module block as well, and resolve a
name a `global` in an enclosing scope covers as an implicit global, so a
class body reading such a name emits `LOAD_NAME`.
- Resolve a name bound nowhere as an implicit global instead of leaving
its scope unknown.
- Require a binding only for a name an explicit `nonlocal` declared;
a free variable that reached the scope any other way already resolved.
- Name a generator expression's block `genexpr`; the code object keeps
`<genexpr>`.
- Locate a `def`/`class` block at its keyword rather than at the first
decorator. `decorated_definition_range()` moves from `Compiler` to
`rustpython_codegen` for that.
- Carry a comprehension symbol's flags over when inlining it into a parent
entry that only holds a free variable propagated from a child.
- Leave `__conditional_annotations__` out of an annotation block's
symbols and have the compiler cook up the free variable, which places
it after the ones the symbol table supplies.
- Check `from __future__` placement while building the symbol table, and
give `symtable.symtable()` the same docstring stripping and future
feature validation the compiler does.
- Floor `co_stacksize` at 1.
Assisted-by: Claude Code:claude-opus-5
* Locate the eval return and the interactive print like CPython
Leave `eval` mode's `RETURN_VALUE` without a location, so the exit block
is duplicated per reaching path and each copy carries the location of the
branch that jumps to it. `a and b` now ends with two returns as it does
in CPython, and `a if b else c` gives the first one the `a` it returns.
Emit an interactive `CALL_INTRINSIC_1(Print)` at the expression statement
rather than wherever the expression left the location, which differ for
`(a := 10)` and `f'{0:fz}'`.
Assisted-by: Claude Code:claude-opus-5
* Give interactive mode the conditional annotations cell
`compile_program_single()` emitted the `__conditional_annotations__` set
and its name accesses but never added the cell that holds it, so a module
compiled in `single` mode was missing the `MAKE_CELL` and the cellvar that
the same source gets in `exec` mode.
Assisted-by: Claude Code:claude-opus-5
* Give compiler-raised SyntaxErrors an end position
`CodegenError` and `SymbolTableError` now carry an `end_location`
alongside `location`, and `CompileError::python_end_location()` returns
it, so errors such as "'await' outside function" set `end_lineno` and
`end_offset` instead of leaving them None.
`SymbolTableBuilder::error_ranged()` builds both positions from a range,
replacing the inline `source_location()` calls at the error sites.
Drop the `expectedFailure` on
`test_exceptions.SyntaxErrorTests.test_file_source`, which now passes.
Assisted-by: Claude Code:claude-opus-5
* Report parser diagnostics with CPython's wording
`InvalidAnnotatedAssignmentTarget` becomes "illegal target for
annotation", `UnexpectedExpressionToken` becomes plain "invalid syntax"
rather than prefixing the parser's text, and the list-recovery
diagnostics ("Expected an expression or a '}'" and its siblings) join the
two that already mapped to "invalid syntax".
Assisted-by: Claude Code:claude-opus-5
* Punctuate the code object repr like CPython
`repr(code)` was missing the comma after the address and escaped the
filename through Rust's Debug formatting.
Drop the `expectedFailure` on the six `test_dis` tests that compare
disassembly text, which now pass.
Assisted-by: Claude Code:claude-opus-5
* Match CPython's parser error positions
A syntax error the parser reports counts its columns in characters, so
`source_location()` converts with UTF-32 rather than UTF-8; an offset
inside a character walks back to where that character starts.
Spans the parser reported differently:
- an error between two tokens covered nothing, and now covers one
character, since the narrowest token still covers one
- a "was never closed" bracket ends at column 0
- an unterminated string literal ends where it starts
- "leading zeros in decimal integer literals" spans the run of zeros
- an exponent with no digits and no sign is blamed on the digit before
the exponent letter, which the tokenizer puts back
Assisted-by: Claude Code:claude-opus-5
* codegen: seek to the finally body's own scopes before copying it
An early exit from a try block emits an extra copy of the finally body.
Nested scopes are handed out by position, and the cursors still sit inside
the try block's own run of scopes there, so the copy took those instead of
the ones the finally body opens: a generator expression got a sibling's
symbol table and raised "the symbol 'k' must be present in the symbol
table", or, where the sibling was a named scope, built a code object with
no `.0` argument that raised UnboundLocalError when it ran.
Seek the cursors to the first scope beginning on or after the finally
body's first line before compiling the copy. The existing restore then
leaves the try statement's own copies starting from the same place.
Assisted-by: Claude
|
||
|
Luan Taraschi
|
97a142d809 |
Bound the recursion in exception group split and subgroup (#8584)
Both walk nested groups by calling themselves back through
vm.call_method, and neither pushes a Python frame, so the frame limit
never sees the nesting and the native stack runs out first. A group
nested past the recursion limit segfaults:
e = TypeError(1)
for _ in range(exceeds_recursion_limit()):
e = ExceptionGroup("eg", [e])
e.split(TypeError) # dumps core
vm.with_recursion is the guard for exactly this, the same one comparison
and repr already use, and it turns the crash into the RecursionError
CPython raises. test_deep_split and test_deep_subgroup were skipped on
the segfault and now pass.
Assisted-by: Claude Code:claude-opus-5
|
||
|
Luan Taraschi
|
0c9fa4219d |
Stop skipping six tests whose reason no longer reproduces (#8583)
Three skips in test_array say "Hangs", one in test_marshal carries no reason at all, and two in test_thread say "panic" on Linux. None of the six does any of that now. Each ran three to five times in a row and passed, and the array trio was also checked case by case against CPython 3.14, including the mutation-during-index-conversion shapes the tests were written for. The neighbouring markers that say "Flaky", "Flakey on CI" and "flaky EOFError" are left alone. One green run says nothing about those. Assisted-by: Claude Code:claude-opus-5 |
||
|
Luan Taraschi
|
d1e80365d6 |
Drop the handler-table borrow before running a signal handler (#8582)
trigger_signals borrowed the handler table for the whole dispatch loop
and the borrow was still alive while the Python handler ran. A handler
that disarms itself, which is the ordinary way to write one, reached for
the same cell:
>>> import signal
>>> def h(signum, frame): signal.signal(signal.SIGUSR1, signal.SIG_IGN)
...
>>> signal.signal(signal.SIGUSR1, h)
>>> signal.raise_signal(signal.SIGUSR1)
thread 'main' panicked at crates/vm/src/stdlib/_signal.rs:255:43:
RefCell already borrowed
The handler now comes out of the table one signal at a time, and the
borrow ends before it runs. Reading inside the loop rather than once up
front also keeps the table current, so a handler that arms or disarms
another signal is obeyed for the rest of the pass.
Assisted-by: Claude Code:claude-opus-5
|
||
|
Shahar Naveh
|
e71a955c99 |
Update some libs and tests to 3.14.7 (#8559)
* Update some libs * Update some tests * more tests * test_pydoc.py * test_mimetypes.py * Restore json decoder * reapply updaye |
||
|
Shahar Naveh
|
26ff16c762 |
Update Lib/support to 3.14.7 + some tests (#8578)
* Update `test/support` to 3.14.7 * `test_compile.py` * Update some tests |
||
|
Shahar Naveh
|
2dddc63d59 |
Update some libs & tests to 3.14.7 (#8577)
* Update some libs and tests to 3.14.7 * test_shlex.py * email |
||
|
Shahar Naveh
|
0a70069404 |
Update some tests to 3.14.7 (#8558)
* Update some tests to 3.14.7 * remove patch * restored `test_monitoring.py` |
||
|
Luan Taraschi
|
b9f591b665 |
Follow CPython's padding rules in a2b_base64 (#8570)
With strict_mode=True the decoder accepted padding CPython rejects:
>>> binascii.a2b_base64(b'YWJj=', strict_mode=True)
b'abc'
It also dropped data in the default mode, because it returned at the
first pad that completed a quad:
>>> binascii.a2b_base64(b'abc=a')
b'i\xb7' # CPython: b'i\xb7\x1a'
The loop now counts pads and decides at the end, the way CPython does,
so a stray pad is ignored outside strict mode and named as leading,
excess or discontinuous inside it. DecodeError::InvalidPadding was dead
in this file and now carries the excess padding message.
test_base64_strict_mode and test_base64_excess_data were marked as
expected failures and pass now.
Assisted-by: Claude Code:claude-opus-5
|
||
|
Luan Taraschi
|
79cf1d9dd0 |
Drop the narrow-build surrogate pairing from unicode_escape (#8569)
unicodeescape_string is a port of CPython 2.7's
PyUnicode_EncodeUnicodeEscape, where the branch mapping a character at or
above 0x10000 to \U00xxxxxx sits under #ifdef Py_UNICODE_WIDE and the
branch folding a surrogate pair sits under the #else. The port kept the
"# ifdef Py_UNICODE_WIDE" and "# endif" comments around the first one but
dropped the #else, so both branches run.
The pairing branch reads the next character without checking that there
is one, so a high surrogate at the end of the string walks off it:
>>> "\ud800".encode("unicode_escape")
IndexError: index out of range # CPython: b'\ud800'
And when there is a next character it folds the two, which no wide build
does:
>>> "𐀀".encode("unicode_escape")
b'\U00010000' # CPython: b'\ud800\udc00'
With the branch gone a high surrogate falls through to the 16-bit case
and comes out as \udXXX, which is where a low surrogate already went.
test_incremental_surrogatepass in UnicodeEscapeTest carried an
expectedFailure for the IndexError. It passes now, so the override goes
with it.
Assisted-by: Claude Code:claude-opus-5
|
||
|
hyoinandout
|
e5c8701d44 |
csv: validate dialect options (#8402)
* csv: validate dialect options Resolve each dialect once and validate the merged options before constructing readers and writers. Handle Unicode character parsing consistently and enable the corresponding CPython CSV tests. Assisted-by: Tau:gpt-5.6-luna * cargo fmt * Validate csv dialect conflicts with line terminators - Reject dialect characters that overlap any lineterminator character - Validate Dialect construction consistently and allow non-ASCII terminators - Unskip csv invalid character coverage now that validation matches behavior * Remove stale CSV rejection snippet Removes a RustPython-only assertion that contradicts the CPython-compatible dialect validation added by this branch. Scope: test-only AI-Assisted-By: Codex * Revert "Remove stale CSV rejection snippet" This reverts commit |
||
|
Jeong, YunWon
|
a7981dccc7 |
Align compiler futures, annotations, and symtable with CPython (#8550)
* Align compiler futures and annotations with CPython Handle Barry parsing consistently across AST, type-comment, REPL, and WASM paths. Preserve deferred annotation source and scope metadata, and make _symtable conversion cached and linear-time. Assisted-by: Codex:gpt-5.4 * Fix f-string debug ranges and update Barry/annotate test expectations Compute the debug f-string source range from the raw leading and trailing text instead of the comment-stripped text, so a comment inside a multi-line replacement field no longer shifts the emitted LOAD_CONST location. Restrict the obsolete-operator location shift to `<>` by also requiring a `>` at the reported offset; without it any `ExpectedExpression` that followed a `<` (`2 <;`) was reported one column to the left. Add `CodeFlags::FUTURE_MASK` and use it in builtins, the shell, and the WASM VM instead of repeating the eight-flag union, and collapse the duplicated Barry diagnostic sequences into `BarrySource::diagnostic`. Update the `__annotate__` disassembly snapshot to LOAD_FAST_BORROW and replace the RustPython-only `barry_as_FLUFL` co_flags assertion in extra_tests with the CPython-matching one. Assisted-by: Claude Code:claude-opus-5 * Restore the annotated function in __annotate__ qualnames Signature annotation scopes are compiled in the scope enclosing the annotated function, so the function name has to be folded back into the qualname: `f.__annotate__`, `C.m.__annotate__`, `outer.<locals>.inner.__annotate__`. test_type_annotations test_annotate_qualname covers this. test_pyrepl test_future_barry_as_flufl now passes, so drop its expectedFailure marker. Assisted-by: Claude Code:claude-opus-5 * Pass safepoints in the attached-thread interpreter tests `main_and_subinterpreter_run_sections_overlap` parked its workers on a condvar while attached, and `busy_main_interpreter_does_not_block_subinterpreter` looped over protocol calls that never reach a safepoint. An attached thread that never reaches a safepoint cannot be suspended, so a concurrent process-wide stop-the-world (from the gc test running in parallel) never completes and the already-stopped sibling interpreter stays stopped; the second worker then waits in `wait_while_suspended` until the 30s deadline. Both loops now call `vm.check_signals()` each iteration, and the condvar wait uses `wait_timeout` so the lock is released between safepoints. Reproduced on Linux (4 CPUs): the two tests with `--test-threads=2` failed 2/40 before and 0/40 after; the full `rustpython-vm` lib binary failed 1/30 before and 0/30 after. Assisted-by: Claude Code:claude-opus-5 * Report the obsolete `<>` operator from one place `<>` was located twice: `BarrySource` recorded the token ranges it rewrote to `!=`, and `cpython_parse_diagnostic_override` separately peeked at the bytes around an `ExpectedExpression` location. Both produced "invalid syntax" spanning the whole operator. `prepare_barry_as_flufl_source` now records the operator ranges in the non-Barry case too -- by plain text search, since nothing is rewritten there -- so `BarrySource::invalid_legacy_operator` covers both, and the byte-peeking helper is gone. It also defers to an unclosed bracket earlier in the source, which the diagnostic chain it used to sit in did for it. A `<>` that starts a statement now spans both characters: the parser stops at the `<` rather than one character in, which the byte peek did not handle. Assisted-by: Claude Code:claude-opus-5 * Drop a py object's payload before its class `PyInner` declares `typ` before `payload`, so `drop_in_place` dropped the class first and `PyAtomicRef::drop` left it null. A `PyWeak` payload is still linked into the weakref list of the object it points at until its own `Drop` unlinks it under the stripe lock, which happens after that. `WeakRefList::add` walks the same list looking for a proxy to reuse and reads `node.class()` off every node it passes, so it could dereference the null class of a node on its way out. `PyInner::dealloc` now runs the two destructors in the other order at all three of its exit paths, with a compile-time check that no other field has one. Reproduced by the new test in about 0.1s, as the panic CI hit in the c-api `weakrefobject::tests::new_proxy` run: panicked at crates/vm/src/object/ext.rs:312: unsafe precondition(s) violated: hint::unreachable_unchecked must never be reached Assisted-by: Claude Code:claude-opus-5 |
||
|
Jeong, YunWon
|
4fab5c966e |
Raise where the fuzzer found aborts and stack overflows, and collect with the count in the object header (#8551)
* Correct what the non-unix frame stack is for
The comment said non-unix threading builds have no stop-the-world.
CollectStopTheWorld is gated on `feature = "threading"` alone, and
sys._current_frames stops the world on both paths; the field is the
fallback a reader uses when there is no `top_iframe` to materialize from.
Assisted-by: Claude
* Seek by the whole file position on Windows
`SetFilePointer` answers with the low half of the new position and signals
failure with INVALID_SET_FILE_POINTER, which is also that half of a
position four gigabytes in; telling them apart takes the error code, which
this did not read, so such a seek was reported as an error. Deciding
seekability from it also called such a file unseekable.
`SetFilePointerEx` returns the whole position and a success flag of its
own, which also removes the transmute of the position into halves.
Assisted-by: Claude
* memoryview: tell a value of the wrong kind from one that does not fit
Assigning an item reported every packing failure as a TypeError, so
m[0] = 300 on a 'B' view said the value was the wrong type rather than out
of range. Packing now says which of the two it was, and whether the value's
own code raised, in which case that error is the answer as it is:
m[0] = 300 ValueError: invalid value for format 'B'
m[0] = "x" TypeError: invalid type for format 'B'
m[0] = <__index__ that raises> the raised error
`struct` reports both as `struct.error` and is unchanged; the kind travels
beside the exception for the caller that tells them apart.
Also: None was read as a deletion, so m[0] = None answered "cannot delete
memory" instead of packing it; and deleting through the mapping protocol
never reached the read-only check, which comes first.
Assisted-by: Claude
* Raise MemoryError for allocations sized by Python input
os.read, _RawIOBase.read, int.to_bytes, struct.pack, ctypes array
creation and the _ssl RAND functions sized a Vec from a Python-supplied
length with vec![], which calls handle_alloc_error and, under
panic = "abort", ends the process. They now allocate through
vm.new_zeroed_bytes and raise MemoryError.
itertools.product built its pools without checking that
len(iterables) * repeat is representable; it now raises OverflowError
"repeat argument too large" and reserves the pool and index vectors
fallibly. repeat is read as isize, so a negative one raises ValueError
"repeat argument cannot be negative" instead of the conversion's message.
_ssl.RAND_bytes and RAND_pseudo_bytes read n as i32, matching the int
they are declared with.
Assisted-by: Claude
* Collect with the count in the object header
A collection kept the count it was working with in a table keyed by the
object's address, and the objects it had proved reachable in a second one.
Between them they were hashed once per candidate and twice per edge in the
heap, which is where most of a collection over a live heap went.
The count now lives in `PyInner::gc_refs`, with `GcBits::COLLECTING` saying
it is meaningful, and reachability is `gc_refs == GC_REACHABLE` rather than
membership in a set. Step 5 splits the candidates and clears the bit in one
pass. gcbench, five interleaved pairs, median: a live heap of 423k objects
goes from 0.101s to 0.055s and a dead one from 0.402s to 0.383s.
The bits, generation, owner and count take eight bytes between them. A
64-bit header had those eight as the padding its alignment forces, so it is
unchanged at 48 bytes; a 32-bit header grows from 24 to 28.
Assisted-by: Claude
* Take an explicit thread stack size as a floor in debug builds
A debug build already started Python threads on 8 MiB rather than Rust's
2 MiB default, but an explicit threading.stack_size(N) went through
verbatim. test_threading asks for 256 KiB, and starting a thread on that
walked off the end of the stack: the guard page fault landed in the
prologue of ExecutingFrame::run.
Unoptimized, that prologue reserves 80,848 bytes where the optimized one
reserves 656 -- execute_instruction is #[inline(always)] and LLVM only
colors stack slots from opt-level 1, so the frame is the sum of all 200
instruction arms' temporaries rather than the largest. A Python call costs
88,672 bytes of native stack there, and threading's bootstrap is six
frames deep, so 256 KiB holds less than half of what starting a thread
takes.
The floor reaches thread::Builder only; threading.stack_size() still
answers with what was asked for, and release builds are unchanged.
Assisted-by: Claude
* Check the native stack on every frame entry
The C-stack guard ran on one frame entry in eight. That asks the margin to
cover eight frames rather than one, and it does not: an unoptimized frame
entered through native code takes 88,672 bytes against a debug margin of
262,144. A recursion whose steps re-enter that way -- `__add__` calling
itself, a sort key that sorts -- ran off the end of the stack instead of
raising RecursionError. On a debug build `class Add: __add__ = lambda s, o:
s + o; Add() + 1` segfaulted on the main thread; it now raises, as it does
under CPython and in release builds.
enter_iframe checked and then called enter_iframe_unchecked, which checked
again; it now leaves the check to the one call.
Measured on a call-dominated benchmark, five interleaved pairs: instructions
retired go up 0.17%, about four per call, which is the stack pointer read
and the compare.
Assisted-by: Claude
* Take an iterable's length hint when a list is filled from it
`map_py_iter` read `__length_hint__` only to pass it to `PyIterIter`, and
returned an empty vector when the hint was `isize::MAX` or more. Collecting
through `PyResult` dropped the iterator's lower bound, so nothing reserved
the room the hint asked for.
`list()`, `list.extend()` and `list.__iadd__()` now reserve it and report a
hint they cannot honour as `MemoryError`; a hint that leaves no room for the
elements the list already holds is passed over, as `list_extend()` does.
`tuple()` and the other callers keep filling up without reserving.
`length_hint_opt` errors other than the `TypeError` it already turns into
`None` now reach the caller instead of being dropped.
`__iadd__` and `inplace_concat` went through `extract_cloned`, which reads
`__len__` and not `__length_hint__`; both call `PyList::extend` now, the way
`list_inplace_concat()` calls `list_extend()`.
The tuple, list and dict fast paths of `extract_elements_inner` reserve their
known length, which the `collect()` they used dropped.
Assisted-by: Claude
* Drop map.__length_hint__
`map_methods` has no `__length_hint__`, so `operator.length_hint()` on a map
answers 0, not the length of what it draws from.
The method walked into the length hint of every iterator it holds, and a
chain of maps 10000 long overflowed the native stack answering for the
outermost one. It also took the longest of its iterators, where a map stops
at the shortest.
Assisted-by: Claude
* Ask for a length hint where each caller asks for it
`map_py_iter` asked the iterable it was handed, for every caller, and reported
what asking raised. Only some callers ask it: `list_extend()` and
`_PyBytes_FromIterator()` ask the iterable, `PySequence_Tuple()` asks the
iterator, and the bytearray constructor asks nothing. `tuple()`, `min()`,
`max()`, `collections.deque()` and `f(*x)` raised for an iterable whose
`__len__` or `__length_hint__` does, where they answer.
Which object is asked is now the caller's to say. `sorted()` asks the iterable,
being `PySequence_List()`.
`bytes_from_object()` stood in for `PyBytes_FromObject()`, for
`bytearray_extend()` and for the bytearray constructor, which do not agree on
this: the first two ask, the last does not. It is split, and assigning to a
bytearray slice takes the constructor's side with `PyByteArray_FromObject()`.
`list.extend()` counted what it held before the iterable had been asked, where
`list_extend()` reads `Py_SIZE(self)` after. A `__length_hint__` that adds to
the list made the overflow guard read a count too small and raise `MemoryError`
where nothing is wrong; one that empties it made the guard skip a reservation
that cannot be served.
Assisted-by: Claude
* Settle product's pool count before it reads its arguments
`product_new()` checks `repeat` and works out `npools` before it calls
`PySequence_Tuple()` on any argument, and fills the pools `npools` times.
The pools were filled by repeating the arguments `repeat` times instead, which
walks that many steps even with no arguments to repeat: `product(repeat=2**62)`
counted up to it rather than answering `[()]`. The count was also worked out
after the arguments had been read, so a repeat too large to serve ran their
code first.
Assisted-by: Claude
* Let the bool format answer with the error its value raised
`pack_single()` leaves `'?'` to `PyObject_IsTrue()` and returns what that
raised. Packing classified the error instead, so a `ValueError` from a
`__bool__` came back as "memoryview: invalid value for format '?'".
Assisted-by: Claude
* Release a cell's old value after the lock
`PyCell::set` dropped what it replaced while still holding the mutex guarding
the cell contents. A `__del__` running from that drop and reading the same cell
waited on a lock its own caller held, so `del it` on a closure variable whose
value has such a `__del__` deadlocked.
The replaced value is now released once the guard is gone, as `Py_XSETREF`
stores before it decrefs.
Assisted-by: Claude
* Have a set iterator hold the set it iterates
The iterator kept only a reference to the inner hash table, so in
`it = iter(A(*args))` the `A()` temporary was the last owner and died as the
call returned, before `it` was bound. A `__del__` reading `it` there saw an
unbound name and its error was printed and ignored.
The iterator now holds the set object, as `si_set` does, and releases it once
exhausted. That release happens after the lock is dropped, where
`setiter_iternext()` puts its `Py_DECREF(so)` past `Py_END_CRITICAL_SECTION()`,
so a `__del__` that iterates again does not wait on a lock the call holds.
Assisted-by: Claude
* Release an exhausted iterator's container after the lock
`PositionIterInternal::_next` overwrote its `IterStatus::Active` while the
caller still held the mutex around it. Dropping the container there ran any
`__del__` under that lock, and a `__del__` that iterated the same object again
blocked on it.
`exhaust()` now hands the container back instead of dropping it, and
`locked_step()` releases it after the guard. list, list_reverseiterator, tuple,
str, dict and its views and reverse views, bytes, bytearray, memoryview,
array, deque, and the enumerate and sequence iterators all go through it.
Assisted-by: Claude
* Unskip test_free_after_iterating
Assisted-by: Claude
* Stop listing test_set as an environment polluter
`check_free_after_iterating` no longer leaves an ignored exception behind, and
the job that reruns the listed tests ten times fails once one of them stops
polluting.
Assisted-by: Claude
* Drop winsound imports left unused
`TryFromBorrowedObject`, `exceptions`, and `ToWideString` have no reference in
the module, which fails the Windows clippy line under `-Dwarnings`.
Assisted-by: Claude
* Ask nothing where the caller takes no room
`map_py_iter` asked the iterator for a length hint on behalf of every caller
that does not reserve, and reported what asking raised. Those callers ask
nothing at all: `tuple()`, `f(*x)`, `bytearray(x)`, `min()`, `max()` and
`collections.deque()` answer for an iterator whose `__length_hint__` raises,
where they had been raising it.
The answer was also never spent. It reached `PyIterIter` for a `size_hint()`
the push loop does not read, so the lookup and any call it made were work
thrown away: `tuple()` over a generator drops 22% of its instructions, and 17%
over an iterator with a `__length_hint__` written in Python.
Assisted-by: Claude
* Stop asking an iterator how long it is to walk it
`PyIter::iter` and `PyIter::into_iter` asked for a length hint and reported
what asking raised. Nothing spent the answer: it reached `PyIterIter` for a
`size_hint()` that every caller either loops past or drops, since collecting
into a `Result` reports no lower bound.
23 operations answered for an iterator whose `__length_hint__` raises, where
they had been raising it: `set`, `frozenset` and the nine `set` methods that
take an iterable, `dict.fromkeys` and the dict view operators, `array` and
`array.extend`, `all`, `any`, `sum`, `io.writelines`, `math.fsum`,
`math.prod`, and `csv.writerow` and `writerows`.
Over a generator, `set()` drops 16% of its instructions and `all()` 23%.
`str.join` and `bytes.join` do ask, reaching their elements through
`PySequence_Fast()`, which fills a list from the iterator. They take
`iter_sized()`, which is now the only way to ask. `iter_without_hint` is gone,
its callers being what `iter` already does.
Assisted-by: Claude
* Name the matrix jobs rather than let the matrix name them
A generated job name lists every value in the matrix entry, so `cargo check`
carried the booleans its dependencies and `skip_ssl` keys expand to, and the
snippets job carried its test arguments and timeout. Adding or removing a key
renames the check, which drops it from the required list until that list is
edited to match. Emptying `env_polluting_tests` renamed three checks this way.
`Run rust tests` and `clippy` keep the names they had. `cargo check` drops the
booleans from six of its nine, and the snippets job drops its arguments and
timeout from all three.
Assisted-by: Claude
* Keep a sequence iterator active when an element raises
`PositionIterInternal::_next` exhausted the iterator for any non-`Return`
result, so an error from `__getitem__` ended the walk. `iter_iternext()`
lets go of its sequence for `IndexError` and `StopIteration` alone, and
`PyIterReturn::from_getitem_result` has already turned the first of those
into the second, so only `StopIteration` exhausts now.
Both deque iterators keep exhausting on their own mutation guard, which
`deque_iternext()` does by zeroing the counter before it raises; they
share one step function for it, and the message it raises is lowercased
to match the three other sites in the module.
Assisted-by: Claude
* Name the caller in the bytes conversion TypeError
An object with no iteration protocol reached `PyObject_GetIter`'s "not
iterable" message, and `bytearray.extend()` reported bytes. Each entry
point now checks for the protocol first and names itself:
bytes(object()) cannot convert 'object' object to bytes
bytearray(object()) cannot convert 'object' object to bytearray
bytearray().extend(object()) can't extend bytearray with object
Assisted-by: Claude
* Take a strong count too large to hold as reachable
`start_gc_refs` clipped a count at `GC_REACHABLE - 1`, a number the
per-reference subtraction could still walk down to zero and collect a
live object. It now stores `GC_REACHABLE`, and `subtract_gc_ref` leaves
that value alone.
Assisted-by: Claude
* Raise AssertionError where a snippet asserts False
`assert False` is removed under `-O`, which the snippet suite may run.
Assisted-by: Claude
* Keep raising for a collection that moved under its iterator
A deque, set or dict iterator raised once and then read as spent. The
guard is sticky: `deque_iternext()` looks at the deque's state before the
count it keeps, and `dictiter_iternextkey()` and `setiter_iternext()`
write a size no collection can have, so every later call finds the same
thing and raises again. `dequereviter_next()` is the one exception,
looking at its count first, so it runs out after the first raise.
Both deque iterators now carry `dequeiterobject.counter` rather than
reading a length back from the deque, and the dict and set iterators
compare the size they captured against the collection's own every time
they are asked how much is left, which is what makes that answer nothing
from the moment the collection changes rather than only once the
iterator has raised.
The set's message is capitalized to match `setiter_iternext()`.
Measured against 3.14.6, for each of deque, reversed deque, set, dict
and a reversed dict view: the hint after the change, the error, the hint
after the error, and what a later call answers.
Assisted-by: Claude
* Give apt-get update a deadline to fail on
The step waits on `apt-get update`, which has no deadline of its own, so
a source that takes the connection and then stops answering holds the
job until the workflow's own timeout. The retry that disables the
Microsoft and azure-cli sources runs only when the update exits
non-zero, which a held connection never does; three jobs on this branch
sat on this step for 20 minutes to five and a half hours.
Each attempt is now bounded, and the transports are given a timeout, so
a source that stops answering reaches the retry.
Assisted-by: Claude
|
||
|
mumallaeng
|
b310b6bf6d |
Collapse bare ExpectedExpression to 'invalid syntax'; fix '<>' diagnostic offset (#8540)
* Collapse bare ExpectedExpression to 'invalid syntax'; fix '<>' diagnostic offset `ParseErrorType::ExpectedExpression` currently surfaces as the raw ruff parser message (e.g. "Expected an expression") to callers that only depend on `rustpython-compiler` (no `rustpython-vm`). `rustpython-vm`'s `vm_new.rs` already collapses this to CPython's generic "invalid syntax" for its own callers; mirror that same collapse inside `cpython_parse_diagnostic_override` so non-vm consumers get the same CPython-compatible message. A bare `<>` outside Barry-as-BDFL mode (`2 <> 3`) lexes as `Less` then an unexpected `Greater`, so the resulting `ExpectedExpression` location points at the `>` -- one character past where CPython's tokenizer (which treats `<>` as a single obsolete token) reports the error. Detect the `<` immediately preceding the location and shift the reported range back over it. Assisted-by: Claude Code:claude-sonnet-5 * Require an adjacent '>' before treating '<' as the obsolete <> operator CodeRabbit review on #8540: the previous check only looked at the byte before the ExpectedExpression location for '<', without confirming an adjacent '>' really follows it. For inputs like a trailing '<' at EOF this could misclassify an unrelated ExpectedExpression as the bare '<>' case and report a bogus range. Require both bytes are present before constructing the diagnostic. Also remove the now-stale `@unittest.expectedFailure # TODO: RUSTPYTHON` markers on test_guido_as_bdfl and test_barry_as_bdfl_relative_import, which pass with this fix (test_barry_as_bdfl and test_barry_as_bdfl_look_ma_with_no_compiler_flags still need real Barry-as-BDFL tokenizer support and stay marked). Assisted-by: Claude Code:claude-sonnet-5 |
||
|
Hanjeong LeeandJeong, YunWon
|
9ace9a8920 |
builtins: generate accurate __text_signature__ (#8512)
* Drop the $module marker from generated __text_signature__
CPython's C functions receive the module as their first argument, so
PyCFunction.__self__ is the module and inspect strips the $module
parameter when building a Signature. A #[pyfunction] takes no such
argument, PyNativeFunction::zelf is None, and inspect has nothing to
strip, so the marker surfaced as a parameter that does not exist:
inspect.signature(len)
(module, /, obj) # was
(obj) # now
All 45 builtins shared with CPython carried it. Methods are unaffected;
their $self marker comes from func_sig and both branches now produce the
same string.
Assisted-by: Claude Code:claude-opus-5
* Mark generated __text_signature__ parameters positional-only
Arguments bind through `FuncArgs::take_positional`, which pops from the
positional list and never consults the keyword map, so a #[pyfunction]
argument cannot be passed by name:
>>> len(obj=[1, 2])
TypeError
The generated signature omitted the `/` marker, so inspect reported those
parameters as POSITIONAL_OR_KEYWORD, contradicting the call above. Emit
the marker, except for `*args`/`**kwargs`, which cannot be followed by
`/`, and for empty parameter lists.
14 of the 45 builtins shared with CPython now report an identical
signature, up from 0.
Assisted-by: Claude Code:claude-opus-5
* Emit no __text_signature__ when an argument has no name
Arguments bound by a destructuring pattern, e.g.
fn round(RoundArgs { number, ndigits }: RoundArgs, ..)
have no name to report, and func_sig stringified the pattern verbatim:
>>> round.__text_signature__
'($module, RoundArgs { number, ndigits })'
That is not valid Python, so inspect.signature() raised "builtin has
invalid signature". Return None instead, which leaves
__text_signature__ unset and makes inspect raise "no signature found",
the same as for a CPython builtin that has no signature.
Affects round, sum, os.pathconf, binascii.b2a_base64 and
binascii.b2a_uu. Their docstrings are unchanged; only the signature
prefix is dropped.
Assisted-by: Claude Code:claude-opus-5
* Name builtin parameters after CPython
These parameters are positional-only, so their names only ever appear in
__text_signature__ and cannot be used at a call site. Naming them after
CPython makes the generated signatures directly comparable:
bin x -> number
ord string -> character
divmod a, b -> x, y
setattr attr -> name
delattr attr -> name
hasattr attr -> name
isinstance typ -> class_or_tuple
issubclass subclass,typ -> cls, class_or_tuple
aiter iter_target -> async_iterable
23 of the 45 builtins shared with CPython now report an identical
signature, up from 0 before this branch. The remainder need FromArgs to
report the parameters of its own structs, which is left for a follow-up.
Add extra_tests/snippets/builtin_signature.py covering the phantom
module parameter, the positional-only marker, the names above, and the
signature-less builtins.
Assisted-by: Claude Code:claude-opus-5
* Drop expectedFailure from test_module_level_callable_noargs
pydoc's summary line for time.time was "time(module)" because the
generated signature carried a $module parameter that inspect could not
strip. It now reads "time()", as the test expects.
Assisted-by: Claude Code:claude-opus-5
* Guard the signature-less assertions to RustPython
test_snippets runs every snippet under CPython as well, and CPython does
have Argument Clinic signatures for round and sum, so that block only
holds for RustPython.
Assisted-by: Claude Code:claude-opus-5
* Update crates/derive-impl/src/util.rs
* Fix ord's parameter reference after the merge
The merge of main took ord's signature from this branch, which renamed
the parameter to character, and its body from main, which rewrote ord to
accept bytes and bytearray through a parameter named c. The body then
referenced a name that no longer existed and the build failed.
Assisted-by: Claude Code:claude-opus-5
---------
Co-authored-by: Jeong, YunWon <69878+youknowone@users.noreply.github.com>
|
||
|
|
1f2f7ff012 |
Fix int unicode decimal digits (#8521)
* Accept Unicode decimal digits in int(), Decimal() and complex()
CPython runs a string argument through
_PyUnicode_TransformDecimalAndSpaceToASCII before parsing it, so decimal
digits from any script are accepted:
int('١٢٣') # 123
int('0x١f', 16) # 31
Decimal('١٢٣') # Decimal('123')
complex('１+２j') # (1+2j)
RustPython only did this for float(), which had the transform inlined.
int() handed the raw UTF-8 bytes to bytes_to_int(), whose digit check is
is_ascii_alphanumeric(), so every non-ASCII digit was rejected — even
though float() accepted the same string.
Lift the inlined transform out of float_from_string() into
common::str::transform_decimal_and_space_to_ascii() and apply it to the
str paths of int() and complex() too. The result is always ASCII: as in
CPython, a character that is neither ASCII, whitespace nor a decimal
digit becomes '?' and truncates the string, which no parser accepts at
any base, leaving the caller to raise the error from the original string.
Bytes-like input keeps going straight to the parser, matching CPython's
split between PyLong_FromUnicodeObject and PyLong_FromString.
This unmarks two expectedFailure tests: test_int.test_unicode and
test_decimal.test_unicode_digits.
* Share one PyStr-to-numeric-literal step across int, float and complex
All three constructors need the same thing from a str argument: trim it,
fold Unicode decimal digits and whitespace to ASCII, and give up on a
string holding surrogates. Each expressed that last part differently —
float matched PyKindStr and returned b"", complex leaned on to_str()
returning None, int returned an empty Cow — so the rule lived in three
places at once.
Move it into protocol::numeric_literal_from_str() and have all three call
it. CPython repeats this per type because its wrapper is three lines over
a single PyUnicode representation; ours has to match over Ascii/Utf8/Wtf8,
which is worth writing once.
Only the shared step moves. int keeps its base handling, int and float
keep accepting bytes-like input, complex keeps rejecting it, and each
keeps raising its own error, because none of that is shared.
No behavior change: the CPython differential suite is byte-identical
before and after.
* Drop the now-empty test_unicode_digits override in test_decimal
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Jeong, YunWon <69878+youknowone@users.noreply.github.com>
|
||
|
Jeong, YunWon
|
adf7da508b |
memoryview: count exports, hash the exporter, and index through __index__ (#8553)
* memoryview: index through `__index__` and name the dimension
A tuple key was taken apart only when every item was already an `int`, so
`m[I(), 2]` with an `__index__` on `I` was reported as an invalid slice key
rather than indexed. What kind of key it is now follows from the types in
it, the way `is_multiindex` decides, and each item is converted where it
sits; an item that answers `__index__` but raises on the way reports that
rather than making the whole key invalid.
That conversion runs Python, which can release the view, so the
multi-dimensional read goes through `unpack_single` and its re-check
rather than reading the bytes itself.
`IndexError` named the index that was out of bounds where
`lookup_dimension` names the dimension it was out of bounds on, counted
from one, and the one-dimensional path had a message of its own.
Assisted-by: Claude
* memoryview: count what a view has exported, and hash the exporter
The view kept no count of the buffers taken from it, so `release()` always
succeeded, even while something was still reading through an export. It now
answers `BufferError` the way `_memory_release` does, and `__exit__` reports
the same refusal.
`memory_hash` asks the exporter for its hash and throws the answer away, so
a view is no more hashable than what it looks at; a read-only view over a
`bytearray` hashed here where CPython refuses it. Asking runs Python, so the
view counts as exported for the duration and a release attempted from inside
that hash is refused rather than obeyed.
Assisted-by: Claude
* bytes, bytearray, memoryview: measure the hex separator as `len()` does
`hex()` took the separator's length from its bytes rather than from the
object, so a `bytes` subclass that answers `__len__` was measured by what it
holds instead of what it says: `b'abcd'.hex(S(b'::'))` with `__len__`
returning 1 was refused where `_Py_strhex_impl` accepts it, and a two-byte
answer was accepted where it refuses. The check also came after the
`bytes_per_sep == 0` and empty-data shortcuts, so `b'abcd'.hex('::', 0)`
never reached it at all.
Measuring runs Python, and the bytes to be written out must not be borrowed
while it does, so the separator is now resolved from the arguments before
the buffer is reached. For a memoryview that also means the view counts as
exported for the duration, so a release attempted from inside `__len__` is
refused.
Assisted-by: Claude
* array: refuse to hash an array
`array.array` is mutable but inherited `object.__hash__`, so an array could
be put in a set or used as a dict key and then changed underneath it. It is
unhashable, as `tp_hash = PyObject_HashNotImplemented` makes it.
A read-only memoryview over an array now reports the exporter as unhashable
too, which is what `memory_hash` asks it for.
Assisted-by: Claude
* Remove expectedFailure from three memoryview tests
test_hash_use_after_free, test_hex_use_after_free and
test_use_released_memory pass.
Assisted-by: Claude
* memoryview: bound the dimensions a cast can name
`cast()` accepted a shape of any length, so `mv.cast('B', (1,)*64 + (8,))`
built a 65-dimensional view where `memoryview_cast_impl` refuses anything
past `PyBUF_MAX_NDIM`. The limit is answered before the shape is looked at
any further, as it is there.
Assisted-by: Claude
|
||
|
Kwak Byoung Min
|
62f4c9895a |
Add StringIO newline tracking (#8539)
* Report observed StringIO newline types Constraint: Match CPython StringIO newline reporting without changing configured newline translation. Rejected: Separate newline-tracking representation | reuse the existing SeenNewline bitflags. Confidence: high Scope-risk: narrow Directive: Keep observed newline state separate from the configured newline mode. Tested: prek run --all-files; test_memoryio; cargo clippy -p rustpython-vm --lib -- -D warnings; workspace tests excluding the macOS C-API baseline SIGSEGV. Not-tested: Full macOS workspace suite is blocked by the existing rustpython-capi SIGSEGV; the Linux suite is running. Assisted-by: Codex:gpt-5.6-sol * Preserve StringIO state boundaries Reset observed newline state when StringIO is reinitialized and keep the property subject to closed-stream checks. Constraint: Match CPython StringIO reinitialization and closed-stream behavior. Rejected: Returning stale or cached newline state | violates StringIO lifecycle semantics. Confidence: high Scope-risk: narrow Directive: Keep observed newline state tied to the current open StringIO contents. Tested: cargo test --workspace --exclude rustpython_wasm --exclude rustpython-venvlauncher; cargo run --release -- extra_tests/snippets/stdlib_io_stringio.py; cargo run --release -- -m test test_memoryio; prek run --all-files; cargo clippy -p rustpython-vm --lib -- -D warnings. Assisted-by: Codex:gpt-5.6-sol |
||
|
sijun-yang
|
cd935b8ef6 |
Expose symbol scopes through _symtable (#8538)
Pack symbol scopes with definition flags in _symtable.symbols for Lib/symtable.py. Assisted-by: Codex:gpt-5 |
||
|
Jeong, YunWon
|
819b4b86fd |
Fix malformed Unicode error constructors (#8497)
* Fix malformed Unicode error constructors Replace message-only Unicode decode and encode errors with fully initialized exceptions, preserve source objects and failure ranges, and remove the obsolete constructors. Assisted-by: Codex:gpt-5 * Fix Windows Unicode CI regressions Assisted-by: Codex:gpt-5 |
||
|
Jeong, YunWon
|
a78d0eb577 |
Fix crashes found hunting the last open fuzzing record (#8524)
* specialize: check the member descriptor's type before caching its slot offset
The LOAD_ATTR/STORE_ATTR specializations cached the slot offset of any member
descriptor found on the owner's type and then guarded the specialized
instruction on the type version alone, while descr_get()/descr_set() check on
every access that the instance belongs to the type the descriptor was defined
for. A descriptor taken from a wider class and bound to a narrower one read
past the instance's slot array once the cache warmed up:
class Big:
__slots__ = ("a0", "a1", "a2", "a3", "a4", "a5", "a6", "a7")
class Narrow:
__slots__ = ("z",)
Narrow.x = Big.__dict__["a7"]
o = Narrow()
for _ in range(1000):
try: o.x
except TypeError: pass
# index out of bounds: the len is 1 but the index is 7 (object/core.rs)
A class with no slots at all reached the ext_ref().unwrap() on the same line.
Assisted-by: Claude
* socket: reserve recv()'s buffer fallibly
recv() and recvfrom() handed the caller's bufsize straight to
Vec::with_capacity, so an unreachable size aborted the process through
handle_alloc_error before any syscall was made:
socket.socket().recv(2**62)
# memory allocation of 4611686018427387904 bytes failed -> SIGABRT
try_reserve_exact reports MemoryError instead, which is what CPython raises.
Assisted-by: Claude
* types: count the __call__ and __get__ slot dispatches as recursion
Both wrappers re-enter Python without pushing a frame, so nothing counted the
nesting when the special method named the object it was looked up on:
class C: pass
c = C(); C.__call__ = c
c() # native stack overflow, SIGSEGV
class D: pass
d = D(); D.__get__ = d; D.x = d
d.x # the same, through descr_get
with_recursion around the two dispatches raises RecursionError instead, the
way Py_EnterRecursiveCall bounds a tp_call dispatch. It costs about 5% on a
__call__ dispatch and 3% on a __get__ dispatch through these wrappers.
Assisted-by: Claude
* typevar: show a ParamSpecArgs origin by its repr
ParamSpecArgs and ParamSpecKwargs fell back to a Rust `{:?}` of __origin__ when
it had no __name__. That walks the object graph natively through Debug for
PyInner, where no recursion guard sits, so a single repr() of a deeply nested
chain overflowed the native stack:
a = object()
for _ in range(30000):
a = typing.ParamSpecArgs(a)
repr(a) # SIGSEGV
The origin is formatted with its repr now, which is guarded, and a ParamSpec
origin is recognized by its type rather than by carrying a __name__.
Assisted-by: Claude
* Do not hold a lock across a call back into Python
Three places kept a lock while running code that can reach the same object, so
a callback that touched it wedged the process:
_asyncio.future_add_to_awaited_by(fut, waiter) # waiter.__hash__ adds again
select.select(elements, [], [], 0) # fileno() clears `elements`
select.poll().poll(1000) # SIGALRM handler registers
The future's awaited-by field is read and written under its lock but the set is
built outside it, the list extraction re-reads the list on each step the way
map_iterable_object() does, and poll() waits on a copy of its descriptors. All
three ran forever before and now finish the way they do on CPython.
Assisted-by: Claude
* Validate memoryview.cast() arguments and export negative strides correctly
cast() accepted any struct format and any shape element. A zero-size
format ('0s') and a 0 in the shape both reached a division by zero;
cast() now takes only a native single character format, optionally
'@'-prefixed, and shape elements that are ints greater than zero.
A view with a negative stride starts at its last item, so the bytes it
exported began there and its own offsets walked off the front of them.
Such a view now exports the whole underlying buffer with `start` folded
into the descriptor's offsets, and zip_eq() hands over a whole run only
when both sides are contiguous in the last dimension.
Assisted-by: Claude
Assisted-by: Codex:GPT-5
* Charge native recursion to the stack, not to the frame limit
with_recursion() checked the limit sys.setrecursionlimit() sets and
incremented the same counter that pushing a frame does, so a guard on a
native dispatch spent what Python code had left to call with, and did so
where sys._getframe() cannot see it: test.support.get_recursion_available()
reported frames that were no longer there. Py_EnterRecursiveCall bounds
the native stack instead, which is a separate budget, and the C stack
check with_recursion already performs is that bound.
The snippets pinning the guarded paths nest deep enough to reach the
stack rather than the frame limit.
Assisted-by: Claude
* Report a size that cannot be allocated instead of aborting on it
A size taken from Python went straight into an infallible allocation in
several places, so the process aborted through handle_alloc_error before
any exception could be raised:
- str/bytes/bytearray center(), ljust(), rjust() and zfill() reserved
the padded result for the caller's width
- expandtabs() built its runs of spaces from a tabsize of any width;
the argument is a C int, and a wider one does not fit
- Buffered{Reader,Writer,Random} allocated buffer_size, and read(),
read1() and FileIO.read() their read size
- bytes(n) and bytearray(n) allocated n
- pbkdf2_hmac() allocated the derived key length, which is a C int
Each of these now reports MemoryError, or OverflowError where the
argument does not fit the type it is declared with.
new_zeroed_bytes() leaves the zeroing to the allocator, so a large
request costs the pages that are written to rather than all of them.
Assisted-by: Claude
* marshal: answer allow_code where a code object is written or read
allow_code was answered by walking the whole result a second time, with
no depth counter and no record of what it had already seen, so a value
that referred back to itself or nested deeply enough ran off the native
stack. w_object() and r_object() answer it where the code object is,
inside the walk that already bounds its depth and resolves references.
A container length is read the way r_long() reads one: it is signed, so
a length with the top bit set is out of range rather than four billion
items to reserve room for.
load() no longer holds a borrow of the buffer read() returned across the
seek() it makes afterwards.
Assisted-by: Claude
* Do not lock an object while running code that can reach it
Several places held a lock or a borrow of an object across a call back
into Python, so a callback that touched the same object waited on a lock
its own caller was holding:
- memoryview slice assignment read a source overlapping the destination,
and __setitem__ converted the value while holding the write borrow
- BytesIO.readinto() read into a buffer viewing the same BytesIO
- array.__setitem__ converted the value under the array's write lock,
and mmap.write() read a source viewing the same map
- bytearray.join() and bytearray.__mod__ drove Python with the
bytearray borrowed
- array and bytearray answered "is this resizable" after taking the
write lock, though an export is exactly a borrow someone else holds
A TextIOWrapper cookie now has to name a position inside what was
decoded in characters as well as in bytes; only the byte offset was
checked, and the character count is what read() and tell() index with.
Assisted-by: Claude
Assisted-by: Codex:GPT-5
* Stop asserting a pbkdf2 message that depends on the width of a C long
The snippet asserted "key length is too great.", which pbkdf2_hmac()
only reaches once the length has been converted; where a C long is
narrower than the length asked for, the conversion fails first and says
so instead. Both are OverflowError, which is what the case is about.
test_support.test_get_recursion_depth passes now that a native recursion
guard no longer spends frames get_recursion_depth() cannot see.
Assisted-by: Claude
* Publish and read the same pointer for a thread's top frame
set_current_frame() casts the `Py<FrameObject>` it publishes straight to
`*mut FrameObject`, so ThreadSlot::top_frame holds the object's base.
sys._current_frames() read it back through Py::from_payload_ptr(), which
subtracts the payload offset from what it is given. The reference it took
therefore incremented, and later decremented, a word 48 bytes ahead of
the frame -- inside the object allocated before it, whose OnceLock state
word sits exactly there for two frames adjacent in the size class. The
neighbour then read an initialized-looking cold pointer that had never
been written and locked whatever the uninitialized word addressed, so
the thread that owned it crashed rather than the one that read.
The slot now holds `*mut Py<FrameObject>`, which is what both sides mean.
A thread parked in a call has no FrameObject for its topmost frame, so
top_frame is null there and the reader takes the materialize path
instead: test_sys.test_current_frames never reaches the branch. The
snippet takes _current_frames() against threads that are running.
Assisted-by: Claude
Assisted-by: Codex:GPT-5
* Decide stop-the-world parking under the thread registry lock
do_suspend() published SUSPENDED first and only then re-read `requested`,
restoring itself to ATTACHED if the stop had ended in the meantime. That
made a thread the second writer able to leave SUSPENDED, so a stop whose
completion check had already observed the thread parked could be undone
behind the requester's back:
worker CAS ATTACHED -> SUSPENDED
requester all_non_requester_suspended() -> true, world_stopped = true
requester start_the_world(): requested = false, then walks the registry
worker reads requested == false, stores ATTACHED
With the store landing inside that walk the debug assertion in
start_the_world fires; with the walk already past the slot, a following
stop force-parks the thread DETACHED -> SUSPENDED, counts it as stopped,
and the store then puts it back to ATTACHED with the world declared
stopped and the thread running bytecode.
`requested` is set in init_thread_countdown() and cleared in
start_the_world() with the registry held, and start_the_world() keeps
holding it while releasing every SUSPENDED thread. Taking the registry
around the check and the transition therefore makes the two orders the
only ones possible: park before that release pass and be woken by it, or
find the request already withdrawn and stay ATTACHED. The requester is
left as the only writer that takes a thread out of SUSPENDED, and the
self-restore is gone.
suspend_if_needed() takes the VirtualMachine to reach the registry.
Assisted-by: Claude
Assisted-by: Codex:GPT-5
* Keep an atexit callback alive while it is being compared
atexit.unregister() releases the callback list around each __eq__ call and
identified the entry it had compared by the address of its Box. __eq__ can
call atexit._clear(), which drops that Box, and atexit.register(), whose
new Box lands on the freed allocation; the identity search then matched
the freshly registered callback and removed it.
atexit.register(a); atexit.register(b); atexit.register(c)
# __eq__ runs _clear() then register(d), returns True
atexit.unregister(probe)
left no callbacks registered where CPython leaves d.
Entries are Arc-shared now, so unregister() holds the one it is comparing
and matches it with Arc::ptr_eq: an address cannot be reused while the
comparison that named it is still running.
Assisted-by: Claude
* Hold atexit entries in PyRc rather than Arc
PyObjectRef is Send and Sync only under the threading feature, so an Arc
over a callback entry trips clippy::arc_with_non_send_sync in builds
without it, such as the wasm package. PyRc is Arc there and Rc otherwise.
Assisted-by: Claude
* Do not hold a buffer's storage while waiting for a peer
FileIO.readinto() and socket.recv_into()/recvfrom_into() took the target
buffer's write borrow and kept it for the whole call, including the wait
for data a pipe, socket or terminal may never deliver. What CPython holds
across that wait is the export, which only forbids resizing; the borrow is
a lock every other thread touching the same object waits on, so
threading.Thread(target=lambda: sock.recv_into(buf)).start()
len(buf)
did not answer until the peer sent. A thread parked on that lock is
ATTACHED and never reaches a safepoint, so gc.collect() in a third thread
waited for the peer as well: one incidental read of the buffer stopped the
world from being stopped at all.
The wait now runs against storage of its own and the bytes are copied over
once they arrive, with the export held throughout so the target still
cannot be resized meanwhile. A seekable file answers from itself rather
than from a peer, so FileIO.readinto() writes into the target directly
there and the buffered read path is unchanged.
Assisted-by: Claude
* Do not hold a buffer's storage while waiting to hand it over
socket.send()/sendall()/sendto()/sendmsg() and FileIO.write() kept the
source buffer's read borrow for the whole call, including the wait for a
peer that may never make room. That borrow is a lock every other thread
writing to the same object waits on, so
threading.Thread(target=lambda: sock.sendall(buf)).start()
buf[0] = 1
did not return until the peer read; and a thread parked there is ATTACHED
and never reaches a safepoint, so gc.collect() in a third thread waited
for the peer too -- the same wedge readinto() had on the receiving side.
ArgBytesLike::borrow_buf_unlocked() answers with bytes that survive the
borrow being dropped. An immutable object hands out a plain reference and
locks nothing, so those are sent where they lie and bytes and memoryviews
over them cost nothing; only bytes reached through a lock are copied out
first. The export is held throughout either way, so the source still
cannot be resized while it is being sent.
The regression snippet covers both directions now and is renamed for it.
Assisted-by: Claude
Assisted-by: Codex:GPT-5
* Check select()'s descriptor limit while the sequence is walked
seq2set() collected the whole sequence and compared the result's length
against FD_SETSIZE afterwards. Selectable::try_from_object() calls
fileno(), which runs Python and can append to the list being walked, and
the walk re-reads the list on every step, so the collection had no end to
reach and the comparison was never made. seq2set in Modules/selectmodule.c
checks the count per element instead.
stdlib_select.py gains a fileno() that appends to its own list, and
releases its sockets with close() rather than by dropping the name.
Assisted-by: Claude
* Bound native recursion where the C stack cannot be measured
check_c_stack_overflow() answers no unconditionally under miri and on
musl, where the stack pointer is not read. Since 9c9905aff that check is
all with_recursion() does, so every guard placed on native recursion --
__call__ and __get__ dispatch among them -- was a no-op on those targets
and the nesting ran until the stack ran out.
with_recursion() now counts its own depth on those targets and refuses
past NATIVE_RECURSION_LIMIT_UNMEASURED. The count is separate from the
frame limit sys.setrecursionlimit() sets, and compiles away where the
stack pointer can be read.
Assisted-by: Claude
* Report the failure to allocate pbkdf2's key and Take.readinto's scratch
Both buffers are sized from an argument -- pbkdf2_hmac's dklen accepts up
to i32::MAX, and readinto's from the length of the destination -- and were
built with vec![0u8; n], which aborts the process on allocation failure.
new_zeroed_bytes() raises MemoryError instead.
Assisted-by: Claude
* Read the frozen-code tuple length as a signed length
The '(' branches in read_marshal_str_vec() and read_marshal_const_tuple()
took the length with read_u32() as usize, so a value with the top bit set
read as four billion items rather than as out of range. read_len() is what
every other length in this file goes through, and it reinterprets as i32.
Both readers serve deserialize_code(), which reads only the frozen modules
baked in at build time, so this changes no reachable behavior;
marshal.loads() already went through read_len().
Assisted-by: Claude
* Make the snippets from the fuzzer sweep assert what they check
builtin_str.py expanded a tab to 2**31-1 columns, allocating 2 GiB to
observe that the width is accepted; a string with no tab observes the same
acceptance without laying anything out.
stdlib_array.py caught the refusal of frombytes() on its own exported
buffer and passed silently when nothing was raised. stdlib_hashlib.py
used a bare `assert False` as its failure branch. Both now say so through
the same shapes the other snippets use.
stdlib_socket.py also accepts OverflowError from recv() with a size that
does not fit the platform's C int.
stdlib_threading_current_frames.py indexed the frame chain for "f123"
without first asserting it is there.
Assisted-by: Claude
* Ask for a marshal container's room instead of assuming it
A flagged tuple or list is published in the reference table before its
children are read, and the placeholder was built with vec![none; len].
The length is the input's to choose and read_len() lets it reach
i32::MAX, so marshal.loads(b"\xa8\xff\xff\xff\x7f") -- five bytes -- asks
for 17 GB of element slots and aborts the process where the allocator
cannot serve it. r_object() allocates the container up front too, but
PyTuple_New() reports what it cannot get.
The elements are now reserved with try_reserve_exact() and a refusal is
raised as MemoryError through the decoder's pending-error channel.
PyTuple::new_marshal_placeholder() held nothing but that allocation and
is gone; the caller builds the elements and uses new_ref().
Assisted-by: Claude
* Compare the blocking-buffer snippet against something
measure() asserted len(buf) == len(buf), which holds whatever the length
is; it now takes the length the caller expects. The pipe case compared
the drained total against the whole source, while an unbuffered write()
reports only what it transferred and a signal can cut that short; it now
compares against what write() returned.
Assisted-by: Claude
* Refuse array.frombytes() a source whose items are not bytes
frombytes() reads its argument as bytes, but accepted any contiguous
buffer, so array("i").frombytes(memoryview(array("d", [1.0]))) appended a
double's bytes read as ints instead of raising. array_array_frombytes_impl
requires an itemsize of 1; ArgBytesLike now reports the itemsize so the
same check can be made here.
The BufferError that a resize meets while the array is exported also names
the array rather than repeating bytearray's wording.
stdlib_array.py's frombytes-of-itself case used typecode "i", where the
new check answers before the resize guard is reached; it uses "b" so the
guard is what refuses, and asserts the wider case separately.
Assisted-by: Claude
* Tell apart the ways marshal data can be bad
A type byte no reader knows, a back reference that names nothing, and
TYPE_NULL all came out as a bare "bad marshal data" ValueError. r_object()
answers the first two with "bad marshal data (unknown type code)" and
"bad marshal data (invalid reference)", and read_object() answers the
third with a TypeError, "NULL object in marshal data for object", since
TYPE_NULL stands for no object rather than for a value.
MarshalError gains the three cases and deserialize_value() maps them.
The container-specific wording r_object() uses for a NULL read inside a
tuple or list is not reproduced; the exception type is.
Assisted-by: Claude
* Hash the collector's tables by address rather than by SipHash
A collection keys three sets and two maps by object address, and it visits
every tracked object and every edge between them, so the hashing is a
per-edge cost. Those tables used the default RandomState, whose SipHash
buys resistance against a caller choosing colliding keys -- and nothing
chooses these keys: they are addresses this process handed out into tables
that live and die inside one collection. A profile of gc.collect() over a
423k-object heap spent 45% of its samples in SipHash.
They now hash with a splitmix64 finalizer. The shifts matter: a table
picks its bucket from the low bits and an address arrives with those bits
zeroed by alignment, so a plain multiply leaves every object in a handful
of buckets and is slower than SipHash was.
The reachability walk also copied each object's referent vector out of the
map it was cached in, a second pass over every edge; it reads them in
place, and reference subtraction hands its vector to the map instead of
cloning it.
Measured over 423k live objects: 0.93s to 0.15s. Over 843k dead ones:
3.00s to 0.79s. extra_tests/snippets/stdlib_threading_gc_import.py, whose
collector thread calls gc.collect() in a loop, ran anywhere from 2.7s to
28s and now runs in 3.1-3.5s: a collection that takes longer leaves more
garbage for the next one to walk, so the cost fed back on itself.
Assisted-by: Claude
Assisted-by: Codex:GPT-5
* Keep the collection's candidates and their counts in one table
A collection built a set of candidates and, beside it, a map from the same
addresses to their reference counts. Both were probed for every edge in
the heap -- membership from the set, the count from the map -- so each
edge paid to hash the same address twice, and each candidate paid to be
inserted twice. The map alone answers both questions.
The candidates also keep a walkable order now, which the reference
subtraction pass needs since it writes the counts while reading the
candidates, and which the unreachable set is built from instead of a
set difference.
Over the 423k-object heap measured in the previous commit: 0.15s to
0.13s live, and 0.79s to 0.49s dead.
Assisted-by: Claude
Assisted-by: Codex:GPT-5
* gc: collect referents into one buffer instead of a vector per object
Step 3 allocated a `Vec` for every tracked object to hold its referents
and kept them all in a map until step 4 read them back. The referents now
go into a single growing buffer, with the map holding each object's range
into it.
Adds `PyObject::gc_extend_referent_ptrs`, which appends to a caller's
buffer; `gc_get_referent_ptrs` calls it with a fresh one.
Assisted-by: Claude
* memoryview and struct: match the checks and errors of the reference
memoryview:
- `cast()` accepted a source and destination that are both item types,
which reinterprets the items rather than re-dividing the bytes; one side
now has to be a byte format.
- A cast to `shape=()` returned without checking that the buffer holds
exactly the one item that shape describes.
- `hash()` hashes the bytes, so it now raises ValueError for a view whose
items are not bytes, rather than returning a hash that disagrees with the
value the view compares equal to.
- `tobytes()` takes the `order` argument, with 'F' walking a
multidimensional view down its columns; `BufferDescriptor` gained
`for_each_segment_fortran` for that walk.
struct:
- A value the format has no room for reported "argument out of range"
instead of naming the format and its range. The format character is now
passed to the packing functions to report it.
- `Struct.__new__` no longer reads the format; `__init__` does, so
`__init__` can be called again and a subclass can pass the format up.
Methods raise RuntimeError until it has run, and `Struct` is a base type.
Removes the expectedFailure from test_Struct_reinitialization and
test_struct_subclass_instantiation.
Assisted-by: Claude
Assisted-by: Codex:GPT-5
* io: decide the readinto path by file type, not by seekability
FileIO.readinto wrote straight into the caller's buffer, holding its write
borrow, when the fd was seekable; otherwise it read aside into scratch and
copied. Seekability stood in for "this read answers without waiting on a
peer", which a pipe on Windows breaks: lseek on one succeeds, so the pipe
took the borrow-holding path and every other thread touching that bytearray
waited for the peer.
host_io::reads_without_waiting answers it directly -- seekability elsewhere,
GetFileType() == FILE_TYPE_DISK on Windows.
The regression snippet times each operation separately, so a failure names
the one that waited; it asserts the transfer is still in flight before
checking the export; and the socket case fills the connection until it
refuses rather than assuming a size that outruns it, which SO_SNDBUF on an
already-connected pair does not settle.
Assisted-by: Claude
|
||
|
Jeong, YunWon
|
c603eb7c9a |
Implement PEP 688 and rework the buffer protocol around managed exports (#8523)
* Implement PEP 688 __buffer__ and __release_buffer__ A Python class could not export a buffer: the slot machinery had no bf_getbuffer or bf_releasebuffer, and every consumer acquired buffers as PyBUF_FULL_RO through a module of PyBUF_* constants. Add both slots. PyBuffer::release now runs a Python __release_buffer__ before the exporter's own release, once per acquisition, which PyBuffer tracks with an `acquired` flag that clones do not inherit. An export made by a Python __buffer__ is held by a _buffer_wrapper payload that counts its exports and drops the returned memoryview with the last one, and the view handed to __release_buffer__ is a _buffer_window that owns no export, so releasing it inside the hook is inert instead of re-entering it. Replace the PyBUF_* constants with a BufferFlags bitflags type whose composite requests are supersets of the simpler ones, so `contains` answers the REQ_* questions, and pass the request to PyBuffer::from_object. Each consumer now asks for what its counterpart asks for: y* arguments for SIMPLE, w* for WRITABLE, BytesIO.write for CONTIG_RO, bytes(), bytearray() and memoryview() for FULL_RO. memoryview checks the request in memory_getbuf, and array.array and mmap.mmap expose __release_buffer__. Test buffer support with PyObject::check_buffer (PyObject_CheckBuffer) instead of attempting an acquisition, so an exception raised by __buffer__ is no longer reported as the object not being bytes-like, and a __buffer__ with side effects runs once. PyBytesInner becomes a y* conversion as a result: bytes and bytearray methods no longer accept iterables of ints, and find, index, count and __contains__ take the arguments parse_args_finds_byte and bytes_contains describe. A view exports its start offset in the descriptor rather than in its window, which fixes a panic when collecting from a negative-stride view. BytesIO.write rechecks closed after acquiring its buffer, which __buffer__ can close in between. Assisted-by: Claude Code:claude-opus-5 * Rework the buffer protocol around managed exports and view offsets Give `PyBuffer` the `_PyManagedBufferObject` shape: one `bf_getbuffer` acquisition is shared by every handle taken from it, cloning takes another share instead of re-acquiring, and the exporter's release runs once when the last share goes away. Remove `retain`, the unsafe `drop_without_release`, the three `impl Drop`s and the `ManuallyDrop` that stood in for this. Add `abort_acquisition` so a failed request does not run `bf_releasebuffer`. Move the view start into `BufferDescriptor::offset`, the `Py_buffer.buf` analogue, and drop the separate `start` fields on `PyMemoryView` and `PyBufferWrapper`. Slicing goes through `SaturatedSlice::adjust_indices_start`, which reproduces `PySlice_AdjustIndices` and keeps the adjusted start. Fix `zip_eq` to take its contiguous fast path only when both last dimensions are contiguous, and make `for_each_segment` and `zip_eq` handle zero-length and zero-dimensional views. Add `BufferDescriptor::projected` so a request without `PyBUF_ND`, `PyBUF_STRIDES` or `PyBUF_FORMAT` receives a correspondingly reduced descriptor, and reject a request without `PyBUF_INDIRECT` against an exporter that has suboffsets. Copy the source first in `memoryview` slice assignment when both sides reach the same root exporter. Hold the export across the resize in `bytearray.extend`, take `y*` in `marshal.loads`, stop probing the buffer protocol in `FsPath`, rewrite `ord` over the concrete string types, fold `array`'s buffer slot into one `slot_as_buffer`, take `w*`/`y*` in `_overlapped`, and thread the new `offset` field through the `_ctypes` descriptors. Assisted-by: Claude * Re-check a memoryview after a conversion and tighten cast and release `pack_single` and `unpack_single` addressed the buffer with a position taken before `__index__` ran, so releasing the view from that conversion read or wrote outside the exporter's storage, panicking when it had also shrunk. Check the released flag again once the conversion is done, as `CHECK_RELEASED_AGAIN` does. Reject a cast to a format that is not a single native format character with an optional `@` in front of it. An empty format reached a division by its item size of zero. get_native_fmtchar Report a second `__release_buffer__` on the same view as a `ValueError` rather than accepting it, and check that the view belongs to the object first; the silent case is a view that exports nothing. wrap_releasebuffer Compare against another memoryview by reading its view where it lies instead of acquiring a buffer from it, so the restricted view handed to `__release_buffer__` compares equal rather than unequal in one direction only. memory_richcompare Name the type in the unraisable an exception from `__release_buffer__` reports, as `releasebuffer_call_python` does, instead of reporting the exporter object. Assisted-by: Claude * Re-check the destination of a memoryview slice assignment Acquiring the source runs `__buffer__`, which can release the destination view, so check the released flag again once the source is in hand and before the structures are compared, as `copy_single` does. Build the sliced destination as a view that counts as no export, the way a `Py_buffer dest = *view` copy does. Holding one kept the exporter unresizable for the length of the assignment, so a source that released the view and then resized the exporter met a `BufferError` instead of the assignment reporting the released view. Assisted-by: Claude * Answer the memoryview contiguity getsets with the layout `f_contiguous` reported the row-major answer for one dimension and False for any more, so a view laid out both ways, such as one of shape (1, 8), reported False while `check_buffer_request` accepted a `F_CONTIGUOUS` request on it. `contiguous` answered row-major order alone. Move the Fortran-order test from `PyMemoryView` to `BufferDescriptor`, next to the row-major one it differs from only in iteration order, and answer all three getsets with it. Assisted-by: Claude |
||
|
sijun-yang
|
c3bfefaeb0 |
Remove private comprehension flags (#8537)
* Remove the private assignment-expression marker Why: extend_namedexpr_scope() already propagates assignment-expression targets and records DEF_GLOBAL or DEF_NONLOCAL in comprehension scopes. ASSIGNED_IN_COMPREHENSION remained only for conflict checks. Its private bit overlaps CPython's packed LOCAL scope bit. Changes: - Register assignment-expression targets as ordinary assignments. - Detect later target conflicts from existing declaration flags. Assisted-by: Codex:gpt-5.6-sol * Remove the private comprehension iterator flag Why: ITER duplicates the DEF_LOCAL and DEF_COMP_ITER facts already recorded for comprehension targets. Its private bit also overlaps CPython's packed scope field. Changes: - Use those flags for rebinding checks and local restoration around inlined comprehensions. Assisted-by: Codex:gpt-5.6-sol * Preserve source names in named-expression diagnostics Why: Assignment-expression diagnostics exposed the mangled keys used for symbol-table lookup. CPython reports identifiers as written in source. Changes: - Use source names in both comprehension conflict diagnostics. - Enable test_named_expression_invalid_mangled_class_variables. Assisted-by: Codex:gpt-5.6-sol |
||
|
Jiseok CHOI
|
dbd3a0a6b3 |
sqlite3: add SQLITE_DBCONFIG constants, setconfig() and getconfig() (#8535)
* sqlite3: add SQLITE_DBCONFIG constants, setconfig() and getconfig() Implement Connection.setconfig() and Connection.getconfig() using sqlite3_db_config(), and export all SQLITE_DBCONFIG_* integer constants to the module. Fixes test_connection_config in test_dbapi.py. Assisted-by: GitHub Copilot:claude-sonnet-4-6 * sqlite3: fix setconfig to propagate actual SQLite error When sqlite3_db_config() returns a non-OK result, propagate the actual SQLite error instead of replacing it with a generic 'Unable to set config' message, to match CPython's set_error_from_db() behavior. Assisted-by: GitHub Copilot:claude-sonnet-4-6 |
||
|
Jiseok CHOI
|
e19b622174 |
sqlite3: pass None for NULL authorizer args instead of crashing (#8534)
* sqlite3: pass None for NULL authorizer args instead of crashing CPython's authorizer callback receives NULL for arg1/arg2/db_name/access when not applicable (e.g. SQLITE_READ on a table gives NULL for the database name in some versions). Previously RustPython passed these pointers to ptr_to_str which would crash or produce an error. Now ptr_to_str_or_none is used: NULL pointers become Python None, which matches CPython behavior and allows test_table_access and test_column_access to pass. Assisted-by: GitHub Copilot:claude-sonnet-4-6 * fix(sqlite3): avoid reentrant deadlock in Statement::new sqlite3_prepare_v2 can synchronously invoke the authorizer callback, which may call back into Connection methods (e.g. set_authorizer) that require the same db lock. Holding db_lock across the prepare() call caused a self-deadlock when a callback re-entered the connection. Release the lock after sql_limit check and copy the raw handle before calling prepare(), so FFI calls that can trigger Python re-entrancy happen outside the lock scope. Fixes hang in test_authorizer_concurrent_mutation_in_call |
||
|
Jiseok CHOI
|
a142de9abc |
sqlite3: add Connection exception attributes and fix autocommit ValueError (#8533)
Add DB-API 2.0 optional extension: expose exception classes as attributes on Connection objects (Warning, Error, InterfaceError, DatabaseError, DataError, OperationalError, IntegrityError, InternalError, ProgrammingError, NotSupportedError). Also fix autocommit validation to raise ValueError (not TypeError) when an unsupported type is passed. Assisted-by: GitHub Copilot:claude-sonnet-4-6 |
||
|
Jiseok CHOI
|
7cda53b946 |
sqlite3: validate narg before creating functions/aggregates (#8532)
CPython 3.14 added check_num_params() which raises ProgrammingError when narg/n_arg/num_params is out of range (-1..=SQLITE_LIMIT_FUNCTION_ARG). RustPython passed invalid values directly to SQLite, resulting in an OperationalError instead. Add check_num_params() helper and call it in create_function(), create_aggregate(), and create_window_function(). Assisted-by: GitHub Copilot:claude-sonnet-4-6 |
||
|
Jeong, YunWon
|
259ce8ce66 |
Fix the itertools.tee leak and race, the contextvars and generator races, _asyncio's InvalidStateError lookup, and _imp's frozen data (#8518)
* _imp: read get_frozen_object's data as a whole marshal value
The data argument went through deserialize_code(), which reads a code body
without the type byte in front of it, so nothing marshal.dumps() produces was
accepted. Read it with marshal.loads() and require a code object back.
Assisted-by: Claude
* _imp: bind find_frozen's arguments with FromArgs
The arity check was done by hand on a FuncArgs. Take the arguments through a
FromArgs struct instead, which makes withdata keyword-only, and fill in the
data it asks for: the frozen encoding is not marshal, so the code is
re-serialized into what get_frozen_object() reads back.
Assisted-by: Claude
* itertools: give tee's shared buffer a Python type
The buffer was a PyRc<PyItertoolsTeeData>, which is not a Python object, so the
collector could not walk into it and any cycle running through a tee was
uncollectable. Make it the _tee_dataobject type with a traverse, held by
PyRef, and split the rest along the same lines: tee() is a function, _tee is
the iterator type it builds, and _tee takes a single iterable rather than
returning a tuple from __new__. _tee is weak-referenceable, tee() rejects a
negative n with ValueError and reserves its tuple fallibly.
test_itertools.test_tee passes now; its expectedFailure marker is removed.
Assisted-by: Claude
* contextvars: hold the shared context state under locks
The variable map was a RefCell, the enter flag, the context index, the token's
used flag and the variable hash were Cells, and each carried an unsafe impl
Sync. A Context or ContextVar shared between threads overlapped their borrows
and panicked. The map and the per-variable cache are now PyMutex, the flags and
the index are atomics, entering a context is a compare_exchange, and the three
unsafe impl Sync are gone.
The cache also stopped being read through AtomicCell::as_ptr, which raced a
concurrent store on a value holding a PyObjectRef.
Values displaced from the map or the cache are dropped after the lock is
released: __del__ can come straight back into the same context, and the locks
are not reentrant.
Assisted-by: Claude
* generator: read the frame state under the running claim
send(), send_none(), throw() and close() read `closed` and `frame.lasti()`
before `running` was compare_exchanged, so the frame they went on to resume
could be one another thread had already advanced. A resume that decided from
`lasti() == 0` that the generator had not started pushes no value onto the
value stack, and the code after the yield pops one, which underflows the stack.
The compare_exchange now hands back a guard, taken before those reads and
released after maybe_close(), so the generator is retired while it is still
claimed.
Assisted-by: Claude
* itertools: claim a tee's position and its buffer as one step
`_tee::next` read `index` and moved it on afterwards, and
`_tee_dataobject::get_item` released `running` before the value it fetched
from the source was cached.
Two callers on one `_tee` then read the same index, hand out the same value
twice and advance past a value that was never cached, which leaves `index`
past `values.len()` and indexes the buffer out of bounds. Two callers at the
same index on separate tees each fetch a value from the source, and one of the
two is dropped without reaching a caller.
Both claims now cover the read and the update.
Assisted-by: Claude
* _asyncio: require InvalidStateError to be a type
new_invalid_state_error() called whatever `asyncio.exceptions.InvalidStateError`
names and unwrapped the downcast of the result, so a future asked for a result
it does not have panicked once that attribute was rebound to something that is
not an exception:
asyncio.InvalidStateError = lambda *args: 42
_asyncio.Future(loop=object()).result()
The type is looked up the way get_cancelled_error_type() looks its own up, and
raised with new_exception_msg; a lookup that does not produce an exception type
falls back to RuntimeError, as the other arms already did.
Assisted-by: Claude
|
||
|
Jeong, YunWon
|
1d6c6aece0 |
marshal: refuse a back-reference to an object still being written (#8519)
`w_ref` marks a code or slice entry incomplete until `w_complete`, because the reader rebuilds both from their fields and a `TYPE_REF` issued while those fields are still on the wire names an object that does not exist yet. `WriterRefTable` carries that marker and `write_object_depth` raises `cannot marshal recursion <type> objects` instead of emitting the reference. `test_reference_loop_code`, `test_unmarshallable` and `test_reference_loop_slice` lose their RustPython markers; `test_marshal` is 75 run, 16 skipped. Assisted-by: Claude |
||
|
Jeong, YunWon
|
7108ff0f69 |
marshal: round-trip code constants through the runtime bag (#8516)
* Keep abnormal marshal loop test aligned with CPython Assisted-by: OpenAI Codex:gpt-5.5 * marshal: retain runtime code constants while loading Decode code fields through the runtime MarshalBag so co_consts values that do not fit the compiler constant enum remain available to VM code wrappers while the compiler table receives shape placeholders. Assisted-by: OpenAI Codex:GPT-5.4 * marshal: write and read co_consts through the runtime bag serialize_code gains a serialize_code_with variant that writes each co_consts entry through a caller-supplied writer; serialize_code keeps the BorrowedConstant writer as its default. The VM writer passes its own write_object_depth, so a code constant that Literal holds but BorrowedConstant cannot describe reaches the stream instead of panicking in borrow_obj_constant, and a constant shared with the enclosing object takes an entry in the writer's reference table. PyMarshalBag implements constant_ref_from_value, bytes_from_value, str_from_value and tuple_elements_from_value, which deserialize_code_value_inner requires to decode code fields through the runtime bag. Assisted-by: Claude |
||
|
Jeong, YunWon
|
d5d19e0611 |
Fix 33 reproduced fuzzing and static-review defects (#8514)
* mmap: guard move() against dest/src past the mapping size
An offset larger than the mapping made `size - dest` underflow, producing
an out-of-range slice index panic. Check the high side first, as write() does.
Assisted-by: Claude
* exceptions: keep ImportError name/path/name_from out of an empty instance dict
ImportError.__reduce__ unwrapped get_arg(0), which is None when the exception
was constructed with no positional argument, aborting on pickle.dumps(ImportError()).
Fall back to the full args tuple there, and expose name/path/name_from as class
attributes defaulting to None so a bare ImportError() reduces to (cls, ()).
Assisted-by: Claude
* asyncio: stop unwrapping the _current_tasks downcast in current_task()
_asyncio._current_tasks is a reassignable module attribute; the three sibling
functions already degrade gracefully when it is not a dict, current_task() did not.
Assisted-by: Claude
* asyncio: raise TypeError when FutureIter.throw()'s exception class returns a non-exception
exc_type.call() goes through a Python-controlled __new__, so its result can be
any object; the downcast was unwrapped. Report it as a TypeError instead.
Assisted-by: Claude
* sequence: use a fallible reservation for sequence repetition
The guard only rejects a repeat whose per-element size crosses MAX_MEMORY_SIZE,
so (1,) * (10**12) still reached Vec::with_capacity and aborted in the allocator.
Reserve fallibly and surface a MemoryError.
Assisted-by: Claude
* collections: reject an oversized deque repetition with MemoryError
deque * sys.maxsize reached the allocator and aborted with a capacity overflow.
Apply the MAX_MEMORY_SIZE guard the other sequences already carry.
Assisted-by: Claude
* itertools: raise OverflowError for an out-of-range r argument
combinations/combinations_with_replacement/permutations narrowed r with
to_usize().unwrap(), so r=2**64 panicked instead of raising.
Assisted-by: Claude
* math: stream the generic sumprod path instead of collecting both iterables
The big-int path collected each argument into a Vec before multiplying, so an
unbounded iterable exhausted memory. Advance both iterators in lockstep and
accumulate, reporting a length mismatch when only one is exhausted.
Assisted-by: Claude
* _imp: raise TypeError for a second positional argument to find_frozen
withdata is keyword-only and unimplemented; passing it positionally hit an
unimplemented!() and aborted.
Assisted-by: Claude
* _typing: check _idfunc arity before indexing args
_typing._idfunc() with no argument indexed args[0] out of bounds.
Assisted-by: Claude
* exceptions: require a sequence for the ExceptionGroup excs argument
The argument was collected before being validated, so an unbounded iterable
such as itertools.count() exhausted memory. Reject a non-sequence up front.
Assisted-by: Claude
* mmap: treat an inverted find/rfind range as empty
find(b"x", 5, 2) built a slice whose start exceeded its end and panicked.
Assisted-by: Claude
* collections: give deque and defaultdict a GC traverse
Neither type opted into traversal, so a reference cycle through a deque or a
defaultdict was never collected.
Assisted-by: Claude
* itertools: opt the iterator types into GC traverse
cycle and its siblings hold Python references but declared no traverse, so a
cycle built through one of them leaked.
Assisted-by: Claude
* _ctypes: mask an out-of-range int instead of panicking
c_char_p(2**64) and pointer item assignment called .expect() on the narrowing
conversion. Wrap the value to the target width, which is what the C
implementation stores.
Assisted-by: Claude
* hashlib: import _hashlib when a hash module is loaded
The hash object type is a static type owned by _hashlib; calling _md5.md5()
without _hashlib imported hit an uninitialized static type and panicked.
Assisted-by: Claude
* _csv: fall back to the built-in dialect defaults when none is registered
The dialect table is empty until csv.py registers 'excel', so _csv.reader([])
unwrapped a missing entry and panicked.
Assisted-by: Claude
* builtins: reject surrogates in compile()/eval() source instead of panicking
expect_str() panics on a str containing surrogates; convert with try_as_utf8
so eval(chr(0xd800)) raises.
Assisted-by: Claude
* _suggestions: require a list for _generate_suggestions candidates
The argument was collected before validation, so an unbounded iterable
exhausted memory.
Assisted-by: Claude
* lzma: size the filter chain before consuming it
filters= was collected into a Vec before the length check, so an unbounded
iterable exhausted memory. Take the length through the sequence protocol first.
Assisted-by: Claude
* classmethod: opt into GC traverse
staticmethod already declares traverse; classmethod did not, so a cycle through
the wrapped callable leaked.
Assisted-by: Claude
* posix: reject unbounded iterables in posix_spawn and setgroups
argv, setsigdef, setsigmask and setgroups bound an ArgIterable and collected it
before validating, so an infinite generator exhausted memory. Require a
list/tuple for argv, validate signals while streaming, and take setgroups
through the sequence protocol.
Assisted-by: Claude
* _ctypes: size Array slice assignment and _argtypes_ before collecting
Both eagerly collected their argument, so an unbounded iterable exhausted
memory before the length check ran.
Assisted-by: Claude
* sys: propagate the breakpointhook warning failure
warn() was unwrapped, so an unimportable $PYTHONBREAKPOINT under
-W error panicked instead of raising.
Assisted-by: Claude
* structseq: require the sequence argument when constructing a struct sequence
A no-argument construction produced an empty backing tuple, and reading any
named field then indexed out of bounds.
Assisted-by: Claude
* Guard the hash slot dispatch against unbounded recursion
`PyObject::hash` invoked the type's hash slot directly, so an element-wise
`__hash__` following a deeply nested object graph recursed one native frame
per level and overflowed the stack. Wrap the dispatch in `with_recursion`,
matching the repr and rich-compare dispatches in the same file, so
`hash(x)` on a nested tuple/GenericAlias/slice raises `RecursionError`.
Assisted-by: Claude
* genericalias: guard the __parameters__ walk against unbounded recursion
`make_parameters_from_slice` recursed into every list/tuple argument with
nothing counting the frames, so `list[L]` for a self-referential or deeply
nested `L` overflowed the native stack at subscript time. Wrap the descent
in `with_recursion`, which makes the walk fallible; `PyGenericAlias::new`,
`from_args` and `make_parameters` now return `PyResult` and every caller
propagates.
Assisted-by: Claude
* Fix the type confusion in PyAtomicRef's Debug impl
`PyAtomicRef<T>` stores a pointer to a `Py<T>`, as `Deref`, `load_raw`,
`swap` and `Drop` all read it, but `Debug` cast it to a bare `T` and
formatted the object header as payload bytes. For `PyFunction`, whose
`code: PyAtomicRef<PyCode>` has a pointer-chasing `Debug`, that
dereferenced header words and segfaulted.
Cast to `PyObject` instead, which is what `Drop` already does and which
also covers the `PyAtomicRef<PyObject>` and `PyAtomicRef<Option<T>>`
instantiations that have no `Py<T>`.
`_asyncio._enter_task` reached this through `{:?}` in its "Cannot enter
into task" message; format the two tasks with their Python repr, which is
what the message is meant to show.
Assisted-by: Claude
* _sre: disallow instantiating Match
`Match` inherited `object.__new__`, so `M.__new__(M)` produced an instance
whose `regs`/`string`/`pattern` were never filled in by a match run; the
mapping subscript path read them and dereferenced garbage. The type has no
public constructor, so mark it `DISALLOW_INSTANTIATION`, which makes
`M.__new__(M)` raise `TypeError: cannot create 're.Match' instances`.
Assisted-by: Claude
* utils: return the empty repr instead of asserting a non-empty collection
`collection_repr` took the first element with an `.expect()` justified by the
caller's preceding non-empty check. Another thread clearing the collection
between that check and the iteration made the iterator yield nothing and
panicked the worker. Fall back to the caller-supplied empty form, which is
the text those callers already produce for an empty collection.
Assisted-by: Claude
* itertools: advance cycle's index atomically
`cycle.__next__` did `fetch_add(1)` and then reset the index to 0 in a
separate store, so two threads replaying the saved items could both read an
index past the end of `saved` and panic on the slice access. Do the advance
and the wrap in one `fetch_update`.
Assisted-by: Claude
* _ctypes: reject a float argument to a foreign call without argtypes
`conv_param`, the conversion used when `argtypes` is not set, converted its
argument with `try_int`, which goes through `__int__` and so accepts a
float. `libc.strlen(1.5)` therefore passed 1 where the callee expects a
`char *` and the callee dereferenced it. Match `ConvParam`, which does a
`PyLong_Check` and converts nothing: take the branch only for an `int` (or
a subclass, so `True` still converts), and let a float fall through to
"Don't know how to convert parameter".
The branch below it converted a float to a C double, but `try_int` claimed
every float before it could run, so it was dead; `CArgValue::Double`
existed only for that branch and both go. Typed doubles are unaffected —
they travel as `CArgValue::Typed` with code 'd'.
Assisted-by: Claude
* Report the iterator itself from PyIter's traverse
`Traverse for PyIter<O>` delegated to the inherent `PyObject::traverse` of
the object it wraps, so it reported that iterator's referents instead of
the iterator. The iterator's own reference to those referents was then
never subtracted during the collector's reference-subtraction pass, the
referents kept a non-zero gc_refs, and every object reachable from them
was classified as a root.
Any cycle running through a type with a `PyIter` field therefore survived
collection: `map`, `filter`, `zip`, `enumerate`, `reversed` and the
`itertools` iterators all leaked, while the same cycle through a `list`,
`tuple` or `list_iterator` collected. Report the wrapped object, as the
`PyObjectRef`, `PyRef<T>` and `PyStackRef` impls do.
`itertools.tee` still leaks: its shared buffer is a `PyRc<PyItertoolsTeeData>`
rather than a Python object, so the collector cannot see through it.
Assisted-by: Claude
* Remove the obsolete expectedFailure on test_code_module.test_unicode_error
Compiling a source string containing a lone surrogate now raises
UnicodeEncodeError, so the test passes.
Assisted-by: Claude
* itertools: reserve the combination indices fallibly
`combinations` and `combinations_with_replacement` built their index vector
with an infallible allocation, so an `r` that passes the ssize_t check but
does not fit in memory aborted the process instead of raising MemoryError.
Assisted-by: Claude
* Apply the struct sequence constructor's dict argument
`structseq(sequence, dict)` discarded its second argument, so the hidden
fields past `n_sequence_fields` — `tm_zone`, `st_atime` and friends — were
always None when constructed directly or restored from a `(sequence, dict)`
pickle, and a non-dict second argument was accepted silently.
Take the dict, require it to be a dict, and fill the hidden slots the
sequence did not cover from it. A key that names a field the sequence
already supplied, or no field at all, is now a
"got duplicate or unexpected field name(s)" TypeError instead of being
dropped. Both arguments are bindable by name, as `sequence` and `dict`.
`os.stat_result` and `os.statvfs_result` did not accept a second argument
at all; they and `time.struct_time` now share the parsing.
Assisted-by: Claude
* _imp: report the argument count in find_frozen's arity error
Assisted-by: Claude
* Add regression tests for the reproduced crashers
One case per catalog entry, each asserting the behavior the fix produces:
recursion guards, the memory-unsafety sites, the overflow and unbounded
allocation guards, the eager-collection rejections, the unwrap sites, and
the cycles the collector now breaks. Every expected value was checked
against CPython 3.14.
Assisted-by: Claude
* Tolerate a changed-size RuntimeError in the set repr stress test
The mutator and reader threads race on purpose; a "changed size during
iteration" RuntimeError is a valid outcome of that race and should not fail
the test. The panic it guards against is not.
Assisted-by: Claude
* Move the crash regression tests into per-module snippets
crash_regressions.py collected every reproduced crasher in one file. Split it
into the snippet for the module each case exercises, and add stdlib_gc.py,
stdlib_asyncio.py, stdlib_lzma.py, stdlib_threading_set_repr.py and
stdlib_threading_itertools_cycle.py for the cases with no existing home.
The suite runs every snippet under the host CPython too, so the checks only
RustPython raises are guarded by sys.implementation.name: the hash and
__parameters__ recursion depth, and the deque repeat overflow. The float
ctypes argument accepts either TypeError or ctypes.ArgumentError.
Assisted-by: Claude
|
||
|
이광효
|
1178841eab |
itertools: defer dropwhile predicate validation (#8504)
Store the `dropwhile` predicate as a Python object and call it while advancing the iterator. This defers callable validation until the predicate is first needed, matching CPython for empty input while preserving exception propagation during iteration. Remove the now-passing `test_dropwhile` expected-failure marker. Assisted-by: Codex:gpt-5.6-sol |
||
|
Sanghun LeeandClaude Opus 5
|
7e6eaa8e3c |
Reuse stored hashes in dict.fromkeys() (#8503)
* Reuse stored hashes in dict.fromkeys() Closes #8490. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Make fromkeys_known_hashes a PyDict associated fn --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Jeong, YunWon
|
84a0ba7864 |
Fix marshal recursive reference loading (#8501)
* Fix marshal recursive reference loading Create reference-tracked containers before reading their children so recursive list, dict, set, and tuple graphs can be unmarshaled. Preserve interned string markers through the runtime bag and add an initialization-only tuple construction path. Assisted-by: Codex:gpt-5 * Preserve marshal container insertion errors Keep Python exceptions raised while constructing unmarshaled sets, frozensets, and dictionaries instead of collapsing them into ValueError. This makes abnormal recursive hash-container streams report TypeError like CPython and removes the remaining test_marshal expected failure. Assisted-by: Codex:gpt-5 * Enable full abnormal marshal reference loop test * Test direct marshal tuple reference loop Assisted-by: Codex:gpt-5 |