mirror of
https://github.com/RustPython/RustPython.git
synced 2026-10-29 11:14:38 +08:00
stack/python-3.15-8950
489
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Jeong, YunWon
|
7f9fa32d6e |
ssl: add key logging with a shared host append sink (#8655)
* ssl: add key logging with a shared host append sink Port the context-owned keylog sink design from Pyre, retaining CPython 3.14 path and context-switch semantics. Keep host file lifecycle in host_env and TLS formatting in stdlib. Assisted-by: Codex:GPT-5 * ssl: use core and alloc imports for key logging Assisted-by: Codex:GPT-5 |
||
|
Jeong, YunWon
|
e3bf84d1e7 |
Fix resource index conversion and REPL warning (#8654)
Accept __index__ values when converting resource limits and only report PyREPL fallback reasons for terminal-backed interactive sessions. Assisted-by: Codex:gpt-5 |
||
|
Jeong, YunWon
|
ad4532c0e0 |
ssl: retain WebPKI-selected certificate chains (#8653)
Port the connection-owned chain builder from Pyre. Preserve verification inputs with TLS configurations and sessions, report the WebPKI-selected path, and publish only its selected capath anchor. Match CPython chain reporting for CERT_NONE and resumed sessions. Assisted-by: Codex:GPT-6 |
||
|
Jeong, YunWon
|
25b5dfe5ca |
Fix /dev/fd, rlimit, spawn scheduler, and context cache (#8651)
* Fix /dev/fd, rlimit, spawn scheduler, and context cache Honor PYTHONEXECUTABLE when computing sys.executable. Rewind the file offset after a pyc magic probe. Stop masking rlimit values with RLIM_INFINITY. Print the pyrepl-unavailable warning before the rustyline shell. Parse posix_spawn scheduler and apply it where the host supports it. Update ContextVar cache fields under one lock. Remove expectedFailure and skip markers for tests that now pass. Assisted-by: Grok:4.6 * address review: check rlim_t overflow on convert Replace the u64-to-rlim_t as-cast in py2rlim_limit with a checked try_from, matching py2rlim. Assisted-by: Grok:4.6 |
||
|
강동안
|
7376e96665 |
Raise MemoryError without allocating via a MemoryError freelist (#8645)
* Add no_memory_error for allocation failures Assisted-by: Claude Code:claude-fable-5-1 * Use no_memory_error at allocation failure sites Assisted-by: Claude Code:claude-fable-5-1 * Raise MemoryError instead of aborting on sequence repeat allocation failure Vec::with_capacity and Vec::reserve in SequenceExt::mul and SequenceMutExt::imul call handle_alloc_error on failure, which aborts the process under panic = "abort". `[1, 2, 3] * 10**17` passes the MAX_MEMORY_SIZE guard (2.4e18 bytes < isize::MAX) and aborted at the allocation instead of raising MemoryError. Use try_reserve_exact and map failures to vm.no_memory_error(). This also removes the n != 2 special case in imul: with the reserve done unconditionally, the copy loop runs zero times for n == 2 and the final append uses the reserved capacity. Assisted-by: Claude Code:claude-fable-5-1 * Recycle MemoryError instances through a dedicated payload freelist no_memory_error allocated the exception object on every raise, so under real memory exhaustion the raise itself could abort. Keep dead MemoryError husks in a process-wide reserve and revive them on the next raise without touching the allocator. The freelist is scoped to MemoryError at the type level. PyMemoryError already exists as a transparent wrapper payload, but instances never used it: the default exception slot_new delegates to the base class, so everything was constructed as PyBaseException. Route no_memory_error through a PyMemoryError-typed construction instead; the freelist hooks live on PyMemoryError's own PyPayload impl, the dealloc glue is monomorphized per payload type, and the existing exact-type push gate admits exactly MemoryError husks. The #[pyexception] macro always generates the PyPayload impl, which left no way to attach freelist configuration; a payload = "manual" attribute now skips that generation so a type can write its own impl, mirroring the generated PAYLOAD_TYPE_ID chain and class-based downcast validation that keep the payload flavors interchangeable. Exceptions are the first HAS_DICT freelist type, so their husks carry an ObjExt prefix that PyInner::dealloc would normally drop; clear its instance dict and member slots before pushing so no stale attributes survive into a revived instance. The reserve is a process-wide Mutex rather than a thread_local so a single prewarm at interpreter init covers every thread; MemoryError is a cold path, so the lock is uncontended. Python-level MemoryError() still constructs a PyBaseException payload through the delegated slot_new and simply does not participate, matching CPython, where only the PyErr_NoMemory path uses the preallocated reserve. Assisted-by: Claude Code:claude-fable-5-1 * Clear MemoryError husks before they enter the freelist Assisted-by: Claude Code:claude-fable-5-1 * Use no_memory_error for the bz2 allocation failure The shared bz2 engine maps BZ_MEM_ERROR to Bz2Error::Mem; report it through no_memory_error like the other allocation-failure sites, matching PyErr_NoMemory in CPython's _bz2module. Assisted-by: Claude Code:claude-fable-5-1 * Change MemoryError husks Vec to a fixed-size array Assisted-by: Claude Code:claude-fable-5-1 * Increase MemoryError freelist size from 4 to 16 * Address review: keep repeat and husk clearing allocation-safe Assisted-by: Claude Code:claude-fable-5-1 |
||
|
Jeong, YunWon
|
699913d8cd |
Align codec, call, exception, and Unix module behavior with CPython (#8647)
* Collect consecutive charmap encoding failures Pass one contiguous run of unmappable characters to the codec error handler, and use the full run when a text replacement is itself unmappable. Assisted-by: Codex:gpt-5 * Leave conversions out of standalone format specs A conversion belongs to a replacement field rather than the format specification consumed by format(), so stop accepting and storing it in FormatSpec. Assisted-by: Codex:gpt-5 * Decode empty byte strings without looking up a codec Match PyUnicode_FromEncodedObject by returning the shared empty string before resolving the requested encoding for empty bytes and bytearray inputs. Assisted-by: Codex:gpt-5 * Validate call shape before converting builtin arguments Reject unknown keywords and excess arguments before conversions can raise a different error. Align Python-function positional-only handling, use declared list arguments in heapq, keep delegated codecs positional-only, and preserve CPython parameter names and validation order. Assisted-by: Codex:gpt-5 * Align attribute and exception initialization errors Give __dict__ and type __qualname__ their CPython diagnostics, preserve OSError filename fields correctly, accept seven-item SyntaxError details, and report Unicode error constructor arity before converting values. Assisted-by: Codex:gpt-5 * Expose missing Unix and Darwin module APIs Add the platform constants exported by fcntl, mmap, select, socket, syslog, termios, stat, errno, time, and _sre. Implement posix.killpg and resource.getpagesize, and expose resource.error as OSError. Assisted-by: Codex:gpt-5 * Honor the configured Python CPU count Parse -X cpu_count and PYTHON_CPU_COUNT, expose the stored value through sys._get_cpu_count_config, and let os.cpu_count prefer it over the host count. Also publish getsizeof with its CPython parameter name. Assisted-by: Codex:gpt-5 * Fix CPU count and WASI CI coverage Remove the obsolete expected-failure marker now that CPU count overrides pass, and avoid exporting FASYNC where WASI libc has no O_ASYNC constant. Assisted-by: Codex:gpt-5 * self review |
||
|
Jeong, YunWonandIvan Mironov
|
4a267b2616 |
ssl: discover default certificate paths in host_env (#8649)
Keep host certificate path probing with the native certificate provider, select the first available OpenSSL-style file and directory, and leave environment overrides to ssl.py. This also gives Android string defaults instead of the previous None values. Co-authored-by: Ivan Mironov <mironov.ivan@gmail.com> Assisted-by: OpenAI Codex: GPT-5 |
||
|
Jeong, YunWonandIvan Mironov
|
d67526cb08 |
ssl: load hashed capath certificates lazily (#8648)
Retain CA directories on SSLContext, materialize their OpenSSL-style hashed entries when building rustls configurations, and cache parsed DER until directory mtimes change. Publish only the trust anchor used by a completed handshake. Co-authored-by: Ivan Mironov <mironov.ivan@gmail.com> Assisted-by: OpenAI Codex: GPT-5 |
||
|
Joshua Megnauth
|
3442228b59 |
unicodedata: Fix name, unmask test_name (#8599)
This commit fixes most of UCD's names. The two main fixes are: * Handling chars missing in 3.2.0 by checking for membership #8548 * Handling names not present in unicodenames AI disclosure: I used AI for the initial research of this patch to determine why the Tangut characters are missing. AI also assisted with a bug fix where I neglected to parse the Tangut u32 as hex. I wrote all of the code. Assisted-by: GPT 5.6 Luna |
||
|
Jeong, YunWonandIvan Mironov
|
de6053099a |
ssl: take the OID table and the OpenSSL cipher-string parser from #8007 (#8646)
* ssl: read the OID table out of OpenSSL's object database
`txt2obj` and `nid2obj` have to answer with OpenSSL's own numbers, and the
table behind them was 39 entries transcribed by hand, so every object
outside that set was reported unknown.
Read OpenSSL's own files instead: `obj_mac.num` carries each object's NID
and `objects.txt` carries its OID, short name and description, and the
identifier `objects.pl` derives per object -- the `!Cname` override, else
the description, else the short name, each qualified by the enclosing
`!module` with `-` spelled `_` -- joins the two. `OidEntry`, `OidTable`
and the `find_by_*` functions keep their shape, so `ssl.rs` only changes
where an object has no OID to report.
Three details the object database settles that a curated table did not
have to:
- an object written with one name carries it as both, so
`nid2obj(48)` is `('emailAddress', 'emailAddress', ...)`;
- an object need not have an OID at all -- `DES-EDE3` and 139 other
algorithm names have a NID and nothing to encode -- so the fourth
field of the tuple is None for those, as it is on OpenSSL;
- one arc does not encode, since DER packs the first two into a byte,
so the bare roots `iso` and `itu-t` are OID-less too, while two arcs
such as X500's `2.5` are a real object.
Checked against `_ssl.nid2obj` on CPython built with OpenSSL, over every
NID it answers: 1387 of its 1390 present, with no disagreement on any of
them, up from 39. The three absent are OpenSSL's unnamed placeholder
slots, which it reports as ('NULL', 'NULL', None).
The data files come from openssl/openssl `crypto/objects/`, via the
rustls integration rewrite in #8007.
Co-authored-by: Ivan Mironov <mironov.ivan@gmail.com>
Assisted-by: Claude
* ssl: parse OpenSSL cipher strings in set_ciphers
Replaces the pattern matcher `set_ciphers` used with the grammar
`man openssl-ciphers` documents, in a new `ssl/cipher.rs` taken from
#8007: `!`, `-`, `+`, `@STRENGTH`, `@SECLEVEL=n`, `+`-joined
conjunctions, the DEFAULT/ALL/COMPLEMENTOF* groups, protocol versions,
SUITEB*, and the aXXX/kXXX/eXXX prefixes.
`set_ciphers` raises "No cipher can be selected." for a string that
names nothing the provider offers, which is the one failure
SSL_CTX_set_cipher_list reports, and keeps the TLS 1.3 suites selected
in provider order because that setter does not reach them. A SUITEB*
string pins its own key exchange groups, which `prepare_kx_groups`
now reads.
Against CPython's OpenSSL over 78 cipher strings, the selections agree
except under `@SECLEVEL=n`, which OpenSSL applies at handshake time and
this applies to the selection.
`CryptoExt` gains `default_cipher_suites` so that `DEFAULT` and `ALL`
are distinguishable.
Co-authored-by: Ivan Mironov <mironov.ivan@gmail.com>
Assisted-by: Claude
* ssl: finish cipher reporting and provider handling
Use the provider-ordered cipher metadata table as the single source for parsing, reporting, OpenSSL names and descriptions. This makes get_ciphers reflect the context selection and validates curves and Suite B against the active provider.
Keep set_ciphers from changing TLS 1.3 suites, preserve the provider's order, fix empty conjunctions, and keep FIPS initialization within the provider's filtered cipher and key-exchange defaults.
Co-authored-by: Ivan Mironov <mironov.ivan@gmail.com>
Assisted-by: Claude
Assisted-by: CodeRabbit
Assisted-by: OpenAI Codex: GPT-5
* ssl: preserve OpenSSL OID lookup rules
Keep object-name lookup case-sensitive and canonicalize numeric OID arcs before consulting the generated OpenSSL object table.
Assisted-by: OpenAI Codex: GPT-5
* ssl: preserve cleartext after MemoryBIO shutdown
Read at most one TLS record while waiting for close_notify in BIO mode, leaving any post-TLS cleartext queued for the BIO owner.
Co-authored-by: Ivan Mironov <mironov.ivan@gmail.com>
Assisted-by: OpenAI Codex: GPT-5
* ssl: reject empty cipher conjunction operands
Keep empty operands visible to the parser so malformed OpenSSL cipher strings such as AES+ and ALL++AES are rejected instead of normalized.
Assisted-by: CodeRabbit
Assisted-by: OpenAI Codex: GPT-5
* ssl: make Suite B cipher ids unambiguous
Spell out the u16 conversion so Windows-only iterator implementations cannot make collection type inference ambiguous.
Assisted-by: OpenAI Codex: GPT-5
* ssl: move native certificate loading behind host_env
Keep platform trust-store loading and filesystem predicates on the host boundary while exposing backend-neutral certificate bytes to the rustls integration.
Assisted-by: OpenAI Codex: GPT-5
* ssl: invalidate server config after cipher changes
Rebuild cached server configuration after set_ciphers updates the selected suites or Suite B key-exchange groups.
Assisted-by: CodeRabbit
Assisted-by: OpenAI Codex: GPT-5
* ssl: load default certificate sources independently
Honor SSL_CERT_FILE and SSL_CERT_DIR independently. Keep directory certificates as lazy capath candidates, and fall back to native roots only when no environment certificate file was loaded.
Co-authored-by: Ivan Mironov <mironov.ivan@gmail.com>
Assisted-by: OpenAI Codex: GPT-5
---------
Co-authored-by: Ivan Mironov <mironov.ivan@gmail.com>
|
||
|
Jeong, YunWon
|
6e6792ee5c |
common: share the hashlib engine, and publish blake2b/blake2s as types (#8643)
The digest, HMAC, PBKDF2 and scrypt state moves out of the `_hashlib` module into a feature-gated `rustpython_common::hashlib` that owns each state behind a `Mutex` and reports plain Rust results; `_hashlib` and `_blake2` keep the object glue. The engine writes its state through a raw pointer into a caller-owned buffer, so the buffer is boxed and only the pointer travels with the Python object, and it carries the alignment the engine asserts. The wrapper is built only after a successful init, so a rejected digest name leaves nothing to drop over storage the engine never wrote. blake2 moves from `blake2` to `blake2b_simd` / `blake2s_simd`, whose `Params` carry the whole parameter block. `_blake2.blake2b` and `blake2s` become immutable types with `SALT_SIZE`, `PERSON_SIZE`, `MAX_KEY_SIZE` and `MAX_DIGEST_SIZE` as class attributes, and their instances are instances of those types. They accept `key`, `salt`, `person`, `fanout`, `depth`, `leaf_size`, `node_offset`, `node_depth`, `inner_size` and `last_node`. `_hashlib.scrypt` is published on the same engine. Six `test_hashlib` expectedFailure markers drop with it. `blake2`, `digest`, `dyn-clone`, `hmac` and `pbkdf2` lose their last consumer. Assisted-by: Grok Assisted-by: Claude |
||
|
Jeong, YunWon
|
aa59f1ee53 |
common: share the bzip2 stream engine (#8638)
Move the VM-independent bzip2 stream owner into a `bz2`-gated `rustpython-common` module beside the zlib and lzma engines, and keep `rustpython-stdlib` as the Python object and exception adapter. The adapter no longer names the `bzip2` crate; the `bz2` feature carries it. `_bz2` was the last user of the generic `DecompressState` / `DecompressStatus` / `Decompressor` machinery in `stdlib/src/compression.rs`, so that goes with it; `DecompressArgs` stays for zlib and lzma. The decompressor feeds the stream through the existing `Chunker`, so buffered and freshly supplied input are no longer joined into a new allocation first. `test_decompress_after_data_error` passes on the ported engine, so its `expectedFailure` marker goes. Assisted-by: Grok |
||
|
Jeong, YunWon
|
39682562b3 |
Raise the shared argument-binding errors from the slots that count for themselves (#8634)
* Raise the shared argument-binding errors from the slots that count for themselves
Sites all over the tree checked their own argument counts and keywords and
wrote the message by hand, so they drifted: `attrgetter expected 1 argument,
got 0.` carried a full stop, `TypeVar() got unexpected keyword argument(s):
bogus` named neither the keyword the way its counterpart does nor the
function the way `typevar()` is named, and `_csv`'s dialect parser handed a
whole formatted sentence to `InvalidKeywordArgument`, which then wrapped it
in another one.
`Callee::arity_error` and `Callee::unexpected_keyword` are now public, and
these sites raise through them, so there is one place the wording lives.
`FrameLocalsProxy` counts its arguments before reading its keywords, and
`min`/`max` do too, which is the order `framelocalsproxy_new` and `min_max`
check in. `weakref.ref`, `GenericAlias`, `frozenset`, `attrgetter`,
`itemgetter`, `islice`, `start_new_thread`, `TextIOWrapper`, `AttributeError`,
`NameError`, `TypeVar`, `ParamSpec`, `TypeVarTuple` and `TypeAliasType` all
follow.
Of 25 calls measured against CPython 3.14.6, 21 now produce the identical
string. The four left over need what `Callee` cannot carry: `weakref.proxy`
is a function of its own there rather than a type, `frozenset` names the
subclass being constructed, and `TextIOWrapper` names its first parameter
and counts before converting.
Assisted-by: Claude
* Give the argument-binding errors the constructors the rest of them have
Raising one of these meant writing `Callee::of::<Self>(vm).arity_error(1..=3,
0, vm)` — the vm twice, and a type to know about before you could say what
you meant. Every other error in the tree is a `vm.new_*_error`.
So the message texts move out of `Callee` into functions the vm can call, and
the vm gets `new_arity_type_error` and `new_unexpected_keyword_type_error`
alongside `new_unsupported_bin_op_error` and the rest. A slot names itself
with the `NAME` its class already carries:
return Err(vm.new_arity_type_error(Self::NAME, 1..=3, 0));
return Err(vm.new_unexpected_keyword_type_error(Some("typevar"), &key));
The keyword one takes the name as an `Option` because the parser sometimes
has none of its own, which is what `_PyArg_Parser.fname` being NULL means and
what `_csv`'s dialect parser wants.
`bind_for` and `check_kwargs_empty_for` take anything a `Callee` converts
from, so the slots pass `Self::NAME` there too. `Callee` itself is left to the
binding machinery, which is the only place that needs to carry a name around
rather than use one.
No message changes: 22 of 26 calls still match CPython 3.14.6 exactly, and the
43-call set still stands at 23.
Assisted-by: Claude
|
||
|
Jeong, YunWon
|
c716a7d070 |
common: share the zlib stream engine (#8631)
* common: share the zlib stream engine Move the VM-independent zlib-rs stream owner and state machine into a feature-gated rustpython-common module. Make rustpython-stdlib adapt that engine instead of carrying its own flate2 implementation, including native stream copy/deepcopy, Z_BLOCK flushes, and wbits=0 decompression. Assisted-by: Codex:GPT-5 * common: share the lzma stream engine Move the VM-independent liblzma stream owner, filter properties, and compression state into a target-gated rustpython-common module. Keep rustpython-stdlib as the Python object and exception adapter, and remove its direct xz dependencies and lzma-only generic compressor machinery. Assisted-by: Codex:GPT-5 * common: address compression engine reviews |
||
|
Jeong, YunWon
|
516a116f9a |
Name the function a failed argument binding was binding for (#8630)
* Name the function a failed argument binding was binding for A built-in call that binds badly reported neither the function nor, for a method, the right numbers: `self` was prepended before binding, so it was counted as both an expected parameter and a given argument. `[].append()` read `expected at least 2 arguments, got 1`. `Callee` carries what a message says about the function: the name, and whether the leading argument fills the instance parameter. `PyNativeFn` takes one, `PyNativeFunction` and `PyMethodDescriptor` supply it, and the four method-descriptor call specializations in `frame.rs` do the same. `bind_for` subtracts the instance from both numbers. The messages now read as their counterparts do: - `_PyArg_CheckPositional`: `find expected at least 1 argument, got 0`, with the singular when the count is one. - `_PyArg_UnpackKeywords`: `split() got an unexpected keyword argument 'bogus'`, and `cast() missing required argument 'format' (pos 1)` for a parameter a call may pass by name. A positional-only parameter is only ever counted, so it keeps the first form. A binding that happens where the name isn't known leaves it off, the way `_PyArg_Parser.fname` is NULL. `test_find_etc_raise_correct_error_messages` now passes in both `test_bytes` and `string_tests`. Assisted-by: Claude * Name the type whose constructor or initializer bound the arguments `Constructor::slot_new` and `Initializer::slot_init` bound their arguments without a name, and so did the slots that override them, so a constructor that was called wrongly said only `expected 1 argument, got 0`. Both defaults and every override now name the type the slot was written for, not the subclass being constructed and not the module `tp_name` carries: `float expected at most 1 argument, got 2`, and `deque expected at most 2 arguments, got 3` for a `deque` subclass. `slice()` and `range()` counted their own arguments and raised messages of their own; they now raise the one `_PyArg_CheckPositional` raises, which makes `test_range_constructor_error_messages` pass. Assisted-by: Claude |
||
|
Jeong, YunWon
|
306f104847 |
codecs: address post-merge review findings (#8623)
Assisted-by: Codex:GPT-5 |
||
|
Jeong, YunWon
|
322f294a21 |
common: add VM-independent codec and text engines (#8622)
* common: add PyPy CJK codec engines
Assisted-by: Codex:GPT-5
* common: mark CJK mapping tables generated
Assisted-by: Codex:GPT-5
* common: add reusable byte and text engines
Assisted-by: Codex:GPT-5
* common: record opt-in engine dependencies
Assisted-by: Codex:GPT-5
* stdlib: add `_multibytecodec` and the six `_codecs_*` modules
Wire the CJK codec engines in `rustpython-common` up to the VM. `_multibytecodec`
holds `MultibyteCodec`, `MultibyteIncrementalEncoder`, `MultibyteIncrementalDecoder`,
`MultibyteStreamReader` and `MultibyteStreamWriter`, the encode/decode drivers and
the error-handler protocol; each `_codecs_*` module exposes `getcodec` for the
encodings of its region.
`rustpython-stdlib` now enables the `cjk-codecs` feature of `rustpython-common`.
Add `cjk::decode_reset`, which the decoder `reset()` methods need. It differs from
building a fresh decoder state: ISO 2022 keeps its G1..G3 designations and the
escape-throughout flag, and resets only G0 and the shifted flag.
Drop the markers from the CJK codec tests and from the `test_codecs`, `test_io`,
`test_email`, `test_zipfile` and `test_traceback` cases that needed a CJK codec.
`test_codecs.BasicUnicodeTest.{test_basics,test_decoder_state}` stay expected
failures for an unrelated reason: `codecs.charmap_encode` rejects a `None` mapping.
Assisted-by: Claude Code:claude-opus-5
* stdlib: use the byte and text engines from `rustpython-common`
`binascii`, `_json` and `_socket`'s address converters now call the engines in
`rustpython-common` instead of carrying their own copies, and `rustpython-stdlib`
enables the `binascii`, `inet` and `json` features for them. That leaves
`base64` and `crc32fast` with no caller in `rustpython-stdlib`, and
`crates/stdlib/src/json/machinery.rs`, the last of the json_in_type-derived code,
goes away with its last caller.
The address converters change behaviour. `inet_aton` reads the lenient forms
(`127.1`, `16909060`, `0x7f.1`, `0177.0.0.1`), `inet_pton` rejects an octet with a
redundant leading zero and a group of five hex digits, `inet_ntop` writes the
IPv4-compatible form as `::c000:201`, and an address family with no converter
raises `EAFNOSUPPORT` instead of a bare `OSError`.
Give the JSON engine the closing-quote scan and the allocation-free `\uXXXX`
escape it needs to carry `_json`'s workload. Against the code it replaces,
`json.dumps` runs 2.3x faster with `ensure_ascii=True` and 1.7x without it, and
`json.loads` 1.2x.
`binascii.a2b_base64` now reports the messages `binascii.c` does: no
`error decoding base64: ` prefix, and the data-character count in parentheses.
Assisted-by: Claude Code:claude-opus-5
* test: enable the Windows cp932 codec case
The shared CJK engine now satisfies the Windows code-page adapter, so keeping this test marked as an expected failure turns its success into a CI failure.
Assisted-by: Codex:GPT-5
|
||
|
Hanjeong Lee
|
d1bbf4bc1a |
Report every generated __text_signature__, and keep it out of __doc__ (#8614)
* Strip the signature prefix from builtin __doc__
A function's signature and its documentation share one string, separated
by `\n--\n\n`, as in CPython. __text_signature__ read the front of it,
but __doc__ returned the whole thing:
>>> len.__doc__
'len(obj, /)\n--\n\nReturn the number of items in a container.' # was
'Return the number of items in a container.' # now
44 of the 46 builtin functions leaked the prefix.
get_doc_from_internal_doc already stripped it but was reachable only from
PyType.__doc__. Split it the way CPython does, into doc_without_signature
(_PyType_DocWithoutSignature) and a wrapper mapping an empty result to
None (_PyType_GetDocFromInternalDoc), and call it from PyNativeFunction
and PyMethodDescriptor as well.
Assisted-by: Claude Code:claude-opus-5
* Terminate a generated __text_signature__ that carries no documentation
A signature and its documentation share one string, separated by
`\n--\n\n`, and readers find the signature by searching for that marker.
A #[pyfunction] with no doc comment stored the bare signature, which no
reader could parse:
>>> time.clock_getres.__doc__
'clock_getres(clk_id, /)'
>>> time.clock_getres.__text_signature__
None
Emit the marker in that case too, as Argument Clinic does for an
undocumented function. 136 of the 915 functions reachable from the
importable modules were affected.
__doc__ now reports None for them, matching CPython, because nothing
follows the marker.
Assisted-by: Claude Code:claude-opus-5
* Report a __text_signature__ for every method
A #[pymethod] built its doc only when the method carried a doc comment,
so one without a comment dropped the generated signature too:
>>> list.append.__text_signature__
None
Build it from either part, as #[pyfunction] does. All 1101 method
descriptors reachable from the builtin types now report a signature, up
from 1, and 713 of the 1030 that CPython also describes match it exactly.
A method that takes its receiver as an ordinary first argument instead of
`&self` reported that argument by name, so binding the method could not
drop it:
>>> inspect.signature([].__dir__)
(obj, /) # was
() # now
Mark it $self, or $type for a classmethod, as CPython does. 310 methods
were affected.
test_unbound_builtin_method_noargs and test_bound_builtin_method_noargs
pass now.
Assisted-by: Claude Code:claude-opus-5
* Drop the expectedFailure on test_simple_completion
rlcompleter closes the parenthesis for a callable whose signature takes
no parameters, and leaves it open when the signature cannot be read:
def _callable_postfix(self, val, word):
if callable(val):
word += "("
try:
if not inspect.signature(val).parameters:
word += ")"
except ValueError:
pass
os.getpid reports `()` since the generated signature carries the `--`
terminator, so completing `os.getpid` yields `os.getpid()` and the test
passes.
Assisted-by: Claude Code:claude-opus-5
* Name every #[pymethod] receiver zelf
A method that needs an owned handle cannot take `&self`, so it takes the
receiver as its first argument and names it `zelf`, `self` being a Rust
keyword. 24 of the 163 such methods named it something else: `instance`
(11), `obj` (4), `exc` (3), `_self` (3), `_instance` (2).
Rename them. The generated signature is unaffected either way, since the
macro marks the receiver by position rather than by name.
Assisted-by: Claude Code:claude-opus-5
* Drop the unreachable zelf name check in func_sig
The receiver is now marked by position, so the name it carries no longer
matters, and no path reaches this check with an argument named zelf:
- a #[pymethod] or #[pyclassmethod] without a `&self` receiver has its
first argument replaced by the marker before the name is read
- a `&self` receiver is handled as syn::FnArg::Receiver
- a #[pymethod(raw)] is passed to static_raw_func, whose PyNativeFn bound
fixes its signature to (&VirtualMachine, FuncArgs)
- no #[pystaticmethod] or #[pyfunction] names an argument zelf
- #[pymember], #[pygetset] and #[pyslot] never reach func_sig
Every generated signature is byte for byte identical without it.
Assisted-by: Claude Code:claude-opus-5
* Keep the argument bundle when marking the receiver
A #[pyclassmethod] that takes FuncArgs receives the class inside the
bundle along with every other argument, but the marker was spent on the
bundle and the arguments went unreported:
>>> object.__subclasshook__.__text_signature__
'($type, /)' # was, claims it takes nothing
'($type, *args, **kwargs)' # now
Report both, the shape CPython uses for __new__. Two methods take their
arguments this way, __subclasshook__ on object and on type.
Assisted-by: Claude Code:claude-opus-5
|
||
|
Lee Dogeon
|
a21377a97b |
Match uuid.getnode() native node detection to CPython (#8612)
* uuid: expose stable hardware node Cache native MAC address detection and advertise an extractable node only when a hardware address is available. This lets uuid.getnode() use the native UUID path while preserving its existing fallbacks. Assisted-by: Codex:gpt-5.6-sol * Prefer universally administered MAC addresses Match uuid.getnode()'s documented interface-selection rule by scanning available addresses before falling back to a locally administered one. Ignore zero addresses and cover the selection behavior with Rust unit tests. Assisted-by: Codex:gpt-5.6-sol |
||
|
Jeong, YunWon
|
7c6c78f955 |
Implement CPython 3.14 subinterpreter APIs (#8605)
* Add _interpreters and _interpchannels modules
Apply the patch baseline for the low-level subinterpreter modules:
- crates/vm/src/stdlib/_interpreters.rs: create/destroy/list_all/get_main/
get_current/is_running/exec/call/run_string/run_func/set___main___attrs/
is_shareable/whence and the Interpreter*Error exceptions
- crates/vm/src/stdlib/_interpchannels.rs: channel create/destroy/send/recv/
list_all/list_interpreters/release/close and the Channel*Error exceptions
- crates/vm/src/vm/crossinterp.rs: cross-interpreter data protocol
- crates/vm/src/vm/{interpreter,mod,runtime,vm_new}.rs: interpreter registry
and lifecycle plumbing the modules need
- Lib/concurrent/interpreters/: high-level PEP 734 package
Assisted-by: Claude:claude-opus-5
* Correct _interpreters and _interpchannels against CPython 3.14
Argument matching:
- Add `function::ArgSpec`, a `PyArg_ParseTupleAndKeywords` equivalent that
applies the `|`, `$` and `:` markers of a format string to a `kwlist`, and
route every module function of both modules through it. This enforces
keyword-only parameters, rejects unexpected keywords, accepts keyword forms
that were previously positional-only (`whence(id=)`, `get_config(id=)`,
`set___main___attrs(id=, updates=)`, `capture_exception(exc=)`,
`_register_end_types(send=, recv=)`), and reproduces the arity messages.
- `O!` slots (`shared`, `updates`, `call`'s `args`/`kwargs`) now reject None,
and `_PyArg_BadArgument` renders None as "None" rather than "NoneType".
- `new_config` takes at most one positional `str`; `list_all` of
`_interpchannels` is argument-less.
- Convert arguments in `kwlist` order, so an argument's own error is raised
ahead of the checks that follow it.
_interpreters:
- Add the `CrossInterpreterBufferView` type and build the received memoryview
on it, keeping the sending interpreter's exporter out of the destination.
- Build `excinfo.errdisplay` from `traceback.TracebackException`.
Feature flags:
- `os.fork` checks finalization before `allow_fork`, matching `os_fork_impl`.
- `_thread.start_new_thread` and `start_joinable_thread` reject an interpreter
without `allow_threads`.
Lib/concurrent/interpreters/_crossinterp.py: restore the stripped docstrings
and comments.
Assisted-by: Claude:claude-opus-5
* Correct the stateless checks and error shapes of _interpreters against CPython 3.14
verify_stateless_function follows _PyFunction_VerifyStateless: it rejects
non-dict builtins, a non-empty __defaults__, __kwdefaults__ or __closure__,
and code whose LOAD_GLOBAL names are held by the function's globals or
absent from its builtins.
code_returns_only_none follows _PyCode_ReturnsOnlyNone: generator,
coroutine and async-generator code is rejected up front, the instruction
walk skips inline caches and maps specialized and instrumented opcodes
back, and the LOAD_CONST preceding each RETURN_VALUE is compared against
the index of None in co_consts.
ExcInfo::capture keeps an empty exception message instead of dropping it,
and builds `formatted` as `module.qualname: msg`, leaving out the builtins
and __main__ modules.
_interpreters.call packs the callable through _PyFunction_GetXIData before
pickle and re-raises the stateless-check failure when both fail. A func,
args or kwargs that cannot be rebuilt in the target, and a result without
cross-interpreter data, now raise NotShareableError with the snapshot as
the cause rather than being returned as excinfo.
_interpreters.capture_exception() without an argument returns None.
pickle_loads and _PyFunction_GetXIData attach the underlying failure as
__cause__.
channel_send converts the object to cross-interpreter data after the
channel's `closing` check.
ChannelID rich comparison returns the result of comparing its id with the
other number instead of coercing that result to bool.
PyMemoryView_FromObjectAndFlags raises "memoryview: a bytes-like object is
required, not 'X'" for an object that is not a buffer.
Assisted-by: Claude:claude-opus-5
* Correct cross-interpreter data conversion and channel argument parsing
Bound shareable ints by isize, raising OverflowError("try sending as
bytes") outside that range and keeping it as the NotShareableError cause.
Report an unshareable object by its repr, and attach a failing conversion
as __cause__ of the NotShareableError raised for the object it was called
for, so each level of a nested tuple appears in the chain.
Guard each tuple item conversion with with_recursion("while sharing a
tuple").
Gate the memoryview getdata function on `_interpreters` having been
imported, and route channel_send_buffer through a memoryview so it uses
that function and the resolved fallback.
Create the cross-interpreter exception types from the module_exec of
either module that raises them, instead of only `_interpreters`.
parse_cid raises OverflowError("int too big to convert"); int_arg
reproduces the `i` converter's three overflow messages and is also used
while parsing channel_send arguments.
Add BASETYPE to _queue.Empty, and gate nt.execv/nt.execve on allow_exec.
Assisted-by: Claude:claude-opus-5
* Correct interpreter ID parsing and the channel_send defaults lookup
_PyInterpreterState_ObjectToID accepts any object with __index__, raises
OverflowError("int too big to convert") outside the int64 range, and
reports a negative ID with the repr of the original object.
channelsmod_send reads the channel's default unboundop and fallback only
when one of the two arguments is negative, so an explicit pair is
validated before the channel is looked up.
Assisted-by: Claude:claude-opus-5
* Add the _interpqueues module and concurrent.interpreters._queues
_interpqueues holds a process-wide queue table and exposes create,
destroy, list_all, put, get, bind, release, get_maxsize,
get_queue_defaults, is_full, get_count and _register_heap_types, raising
QueueError and QueueNotFoundError. QueueEmpty and QueueFull, which
subclass queue.Empty and queue.Full, are registered per interpreter by
concurrent.interpreters._queues.
Cross-interpreter data carries a queue as a refcounted queue ID that
binds when the data is captured and releases when it is dropped, and
is_shareable reports a registered Queue as shareable.
Queue and channel items owned by an interpreter are now cleared from
Interpreter::finalize, after module finalization, rather than from
destroy_owned_interpreter, so values sent by an atexit callback also
become unbound.
is_shareable gates memoryview on the same registration flag as the
getdata lookup.
Assisted-by: Claude:claude-opus-5
* Add the high-level concurrent.interpreters API
Interpreter, ExecutionFailed and the queue aliases, copied verbatim from
CPython 3.14.
Assisted-by: Claude:claude-opus-5
* Destroy leftover subinterpreters when the main interpreter finalizes
Interpreter::finalize now runs finalize_subinterpreters between the atexit
handlers and the finalizing flag: when the main interpreter still owns
subinterpreters it emits the RuntimeWarning "remaining subinterpreters;
close them with Interpreter.close()" and then finalizes each of them.
runtime::owned_interpreter_ids lists the runtime-owned interpreters.
Assisted-by: Claude:claude-opus-5
* Absolutize the script path in run_file
_Py_abspath and _PyPathConfig_ComputeSysPath0 arrive as abs_path and
script_sys_path0. run_file hands the absolutized path to get_importer
and to the runner, so __file__ and co_filename no longer stay relative,
and inserts the symlink-resolved script directory as sys.path[0]
instead of the argument's parent.
test_cmd_line_script.test_script_abspath no longer expects a failure.
Assisted-by: Claude:claude-opus-5
* Name types in error messages the way tp_name does
The object, number, sequence and mapping protocols, the
unsupported-operand helpers, and the type-specific messages fed from
them now read PyType::slot_name() where they read PyType::name(),
so a type declared with a module keeps it.
Message texts corrected along the way:
- PySequence_Size, PyMapping_Size, PySequence_GetItem and
PySequence_SetItem/DelItem split their merged wording in two, and
PyObject_SetItem/DelItem say "object".
- PyObject_GenericSetAttr reports a read-only attribute and the
missing __dict__.
- check_class drops the type name it appended.
- __int__ and __index__ drop the class prefix; __float__ keeps it.
- float(), complex(), round(), next(), str subscripting and list
subscripting take their own wording.
- str.join and bytes.join number the item they turn down and name what
they wanted, and report a non-iterable as "can only join an
iterable".
test_class.testObjectAttributeAccessErrorMessages,
string_tests.test_subscript and the test_descrtut doctest no longer
expect a failure.
Assisted-by: Claude:claude-opus-5
* Gate script_code's source compilation on the compiler feature
crossinterp::script_code called vm.compile unconditionally, which only
exists with the rustpython-compiler feature. Builds without it, such as
example_projects/barebone, now raise a TypeError instead of failing to
compile.
Assisted-by: Claude:claude-opus-5
* Skip test_pyinit_function_raises_exception
The test imports _testsinglephase in its body without a decorator
guarding it, and now that _interpreters exists requires_subinterpreters
no longer skips it.
Assisted-by: Claude:claude-opus-5
|
||
|
Jeong, YunWon
|
0ae835a073 |
Give heap types a dict namespace (#8606)
Replace `PyType::attributes` (`PyRwLock<PyAttributes>`) with a `TypeNamespace` enum. A type created while an interpreter is running holds a `PyDict`; the types `Context::genesis` and `PyType::new_static` build keep the interned-key `IndexMap`, since hashing a string needs a VM. `type.__new__` binds `__classdictcell__` to the type's own namespace rather than the namespace dict passed to it, so an annotation scope reads attribute changes made after the class body ran. Removes the `expectedFailure` on `test_type_params.TypeParamsClassScopeTest.test_modified_later`. `PyGetSet` holds its class as a `PyRef<PyType>` instead of a non-owning `PointerSlot`, and traverses it, because a namespace can outlive the type it belongs to. `PointerSlot` and the `unsafe` on `Context::new_getset` are gone. `subs_parameters` raises `TypeError` when `__typing_subst__` returns a non-tuple in an unpack position, using the new `PyType::fully_qualified_name` for the message. Removes the `expectedFailure` on `test_typing.GenericTests.test_return_non_tuple_while_unpacking`. Assisted-by: Claude |
||
|
Jeong, YunWon
|
17deb6af5b |
lock: add an opt-in rwlock that detaches before it blocks (#8556)
* lock: add an opt-in rwlock that detaches before it blocks A thread blocked acquiring a lock reaches no safepoint, so stop-the-world cannot stop it, and the lock it waits for is routinely one a thread the requester already suspended is holding. `RawDetachingRwLock` wraps the raw rwlock and hands the wait for a contended acquire to a hook that leaves the interpreter first; an acquire that takes the lock on its first try does not reach the hook. The vm installs the hook during interpreter init and implements it with `allow_threads`. The wait ends with the lock acquired while detached, so re-attaching can park the thread holding it. That is only safe where nothing reachable from a stop-the-world section takes the same lock, so it is opt-in per lock: `PyDetachingRwLock` is a separate type from `PyRwLock`, and `Traverse` is not implemented for it, so a payload holding one cannot derive `Traverse`. `PyByteArray::inner` takes it. `BorrowedValue`/`BorrowedValueMut` gain the matching mapped-guard variants. `a_thread_blocked_on_a_lock_does_not_stall_stop_the_world` blocks an interpreter thread on a `PyDetachingRwLock` and asserts stop-the-world still completes, running the stop on its own thread with a timeout so a stop that never completes fails rather than hangs. Without the hook installed it fails on the 10 s timeout; with it, it passes in 0.07 s. Assisted-by: Claude * lock: stop detaching where the lock may already be held `upgrade` runs with the upgradable lock held, and `lock_shared_recursive` may be the re-entrant take of a lock the calling thread holds; detaching there parks a thread holding the lock, which is what this type documents it must not do. They forward to the wrapped lock instead. `lock_upgradable` starts from holding nothing, but nothing takes an upgradable read of one of these, so it forwards too. `lock_shared` and `lock_exclusive` still detach. Also narrow two claims the comments overstated. Not implementing `Traverse` enforces the opt-in rule only against collections, not against the other things that stop the world. And the requester exemption the hook relies on is wider than `_PyEval_StopTheWorld` gives, so it is a local invariant. Assisted-by: Claude * os.readinto: read aside when the fd can wait `readinto` held the destination's write lock for the whole call, including the `read(2)` inside `allow_threads`. On a pipe, socket or terminal that read returns only when the other end writes, so a thread reaching the same object waited on a lock for an unbounded time, reaching no safepoint while it did. Take the fd that answers without waiting directly, as before, and otherwise read into scratch and take the lock only for the copy. This is what `FileIO.readinto`, `socket.recv_into` and `socket.recvfrom_into` already do; `os.readinto` was the site left over. The EINTR retry moves to `read_into_slice`, unchanged. Assisted-by: Claude * fcntl: detach around the calls that wait Every call in this module ran with the thread attached, so a thread inside one reached no safepoint until it returned. `flock(LOCK_EX)` and `lockf(F_LOCK)` return when whoever holds the lock gives it up, which may be never, and an ioctl on a terminal or socket answers when the device is ready to; the world could not be stopped for that long. `fcntl_fcntl_impl`, `fcntl_ioctl_impl`, `fcntl_flock_impl` and `fcntl_lockf_impl` all release around the call. `ioctl` with `mutate_flag` additionally held the target's write lock for the whole call, so a thread reaching the same object waited on a lock for as long as the device took. Its bytes now go in and come back through a buffer of our own, as `fcntl_ioctl_impl` copies through one of its own for anything up to IOCTL_BUFSZ. The export the argument holds is what keeps the length from changing in between. test_fcntl and test_ioctl pass. Assisted-by: Claude * openssl: read aside instead of holding the caller's buffer locked `SSLSocket.read` wrote straight into the destination buffer, holding the lock that reaching its bytes takes for the whole call. That read returns when the peer writes, which may be never, so a thread touching the same object waited on that lock for as long as the peer took, reaching no safepoint while it did. Read into a buffer of our own and take the destination's lock only for the copy. The rustls backend already reads this way, and `_ssl__SSLSocket_read_impl` works from a `Py_buffer` whose critical section ended before the read. `test_ssl` on this backend fails the same 16 tests before and after. Assisted-by: Claude * vm: add the inverse of allow_threads for callbacks A call that detaches hands the thread to stop-the-world, which counts it as parked. A callback that reaches Python from inside such a call would then run on a thread the requester believes is stopped, and nothing in the vm could stop it: `attach_thread` and `detach_thread` are private to `thread.rs`, and `allow_threads` only goes the one way. `attach_for_callback` attaches for the duration of the closure and returns the thread to where it was, the way `PyGILState_Ensure` and `PyGILState_Release` bracket `_servername_callback`. It tests for "not ATTACHED" rather than for DETACHED, so a thread a stop-the-world has already moved to SUSPENDED routes through `attach_thread` and parks there until the world starts again. `a_callback_inside_a_detached_call_waits_for_the_world` stops the world with a thread detached, then turns that thread loose at a callback and asserts it does not run until the world starts. With the transition disabled it fails. Assisted-by: Claude * openssl: attach around the callbacks that run Python `_servername_callback` and `_msg_callback` reach the interpreter from inside an SSL call -- they take a reference to the Python callback, build arguments and call it. That call is about to detach, and running Python from a detached thread runs it on a thread a stop-the-world requester counts as parked. `_servername_callback` opens with `PyGILState_Ensure()` for the same reason. Both now rejoin the interpreter for the duration of the callback and give the thread back afterwards. The reference to the callback moves inside that section, since taking it is itself an interpreter operation; the check for whether a callback is set at all stays outside, so a socket with none set never reaches the interpreter. No behavior change yet: nothing detaches around these calls, so `attach_for_callback` finds the thread already attached and just runs. Assisted-by: Claude * openssl: detach around the SSL calls that wait On a socket with a timeout the wait lands in `select` -> `sock_wait`, which already detaches. On a blocking socket there is no such return: `SSL_read` blocks in `recv(2)` through `impl Read for &PySocket`, with the thread attached and the connection's write lock held, so the world could not be stopped for as long as the peer stayed silent. `SSL_do_handshake`, `SSL_read_ex`, `SSL_write_ex` and `SSL_shutdown` all run between `Py_BEGIN_ALLOW_THREADS` and `Py_END_ALLOW_THREADS`; these now do too, in both socket and BIO mode as there. The connection lock stays held across the call and so becomes a detaching lock: a thread reaching the same socket gives up its interpreter rather than wait for it attached. `connection` is `#[pytraverse(skip)]`, so a collection does not walk into it and never takes that lock -- which is the rule for opting in, though the skip is what supplies it here rather than the missing `Traverse`. A server that completes a handshake and then says nothing used to deadlock the whole process: the collector suspended the main thread at a safepoint and then waited forever for the reader, so even the test's own timeout could not fire. It now collects in 3 ms. Verified separately that a Python `_msg_callback` still runs from inside the handshake -- 920 invocations across 40 handshakes with a collector looping, five runs clean. test_ssl on this backend fails the same 8 tests before and after, by name, and openssl.rs draws no clippy warning it did not draw before. Assisted-by: Claude * lock: assert the opt-in rule against every stop-the-world section Not implementing `Traverse` for `PyDetachingRwLock` states the rule to a collection: a payload holding one cannot derive `Traverse`, so a collection cannot walk into it. It says nothing to the other sections that stop the world -- fork, traceback dumps, frame enumeration -- and `#[pytraverse(skip)]` steps around it besides, which is how `_SSLSocket.connection` holds one. For those the rule was a comment. `set_world_stopped` records, on the one thread still running inside a stopped world, that it is that thread; `lock_shared` and `lock_exclusive` assert it is not set. A section that took one of these could block on a lock a parked thread holds and only that section can release, which is the deadlock the rule exists to prevent. Debug builds only; release builds track nothing. Nothing in the tree trips it: 180 rounds of collect, `sys._current_frames`, `faulthandler.dump_traceback` and 18 forks with four threads churning bytearrays, plus test_gc/test_bytes/test_threading/test_memoryview/test_buffer on a debug build, all clean. `taking_one_while_stopping_the_world_is_caught` takes one with the flag set and asserts the panic, so the guard is not dead code. Assisted-by: Claude * clippy: drop std_instead_of_core suppressions 1.98 no longer reports `std::io` items for the lint, so the six `expect` attributes for it are unfulfilled. Also drops `from_iter_instead_of_collect` from the workspace lint table, which 1.98 removed. Assisted-by: Claude * lock: spell out why the detaching rwlock exists States the motivation as the three-thread cycle it avoids: how a stop reaches a DETACHED thread but not an ATTACHED one, why a thread blocked on a lock reaches no safepoint, and why the waiter rather than the holder gives way. Puts it on `RawDetachingRwLock`, which is public and so rendered; the module doc is private and keeps only the hook description. Assisted-by: Claude * fcntl: raise the error the call returned The six detached calls discarded the `io::Error` and read `errno` again after `allow_threads`, which re-attaches in between and can park on the way. `lockf` already converts the returned error; these now do too. Assisted-by: Claude * openssl: allocate the read buffer through the vm Without a `buffer` argument `read_len` is whatever non-negative size the caller passed, so `vec![0u8; read_len]` aborts on allocation failure. `new_zeroed_bytes` raises `MemoryError` instead, as the two other reads in this file already do. Assisted-by: Claude * lock: leave no attached blocking acquire on a detaching lock `lock_upgradable` starts from holding nothing, so it detaches like `lock_shared`. A recursive read cannot: it may be the re-entrant take of a lock this thread holds, so detaching there parks a thread holding it, and staying attached stalls stop-the-world. `RawRwLockRecursive` is no longer implemented, which removes `read_recursive` from these locks; nothing took one. The assertion test no longer replaces the panic hook, which is process-wide and was suppressing panic output from whatever else ran beside it. Assisted-by: Claude * vm: wait for the actual block in the stop-the-world test The worker signalled before `read()` and the test slept 50ms, so a stop could complete with nothing blocked on the lock and the test would pass having checked nothing. It now publishes its thread id from inside the interpreter and the test waits for that slot to reach DETACHED, bounded so an acquire that never detaches fails rather than hangs. Assisted-by: Claude |
||
|
Jeong, YunWon
|
f87cfe7e7a |
Match CPython on code objects, ast, symtable and syntax error positions (#8580)
* Guard all/any/tuple(genexpr) calls when the generator is a coroutine
`maybe_optimize_function_call()` reserves a `skip_optimization` label for
every `name(genexpr)` call and, for `all`, `any` and `tuple`, emits an
identity guard against the builtin plus an inlined loop. The port skipped
both whenever the generator expression's symbol table was a coroutine, so
`all(await x for x in xs)` and `all(x async for x in xs)` compiled to a
plain call.
An `await` or an `async for` in the generator does not disqualify the
shape: the inlined `FOR_ITER` raises the same `TypeError` that calling the
builtin on an async generator raises, which is what
test_builtin test_builtin_call_async_genexpr_no_crash asserts.
This was the last code-level difference against CPython across `Lib/`:
comparing every file's code tree by structure, opcode and constant value
(qualnames aside, which moved in 3.14.3+, and set constants compared by
value rather than by hash order) now matches on 1720 of 1720 comparable
files, up from 1719.
Assisted-by: Claude Code:claude-opus-5
* Name the ellipsis type `ellipsis`
`type(...).__name__`, its `__qualname__`, and `types.EllipsisType.__name__`
read `EllipsisType`.
Assisted-by: Claude Code:claude-opus-5
* Align compiler front end and instruction positions with CPython
Reject a bare generator expression in a class header and a format spec
nested more than two deep. Both parse here; the reference grammar has no
rule for either, since a class header only takes `arguments` and the
tokenizer runs out of nesting levels at the third format spec.
Scan the interpolations a format spec carries inside another format spec,
so `f"{x:{y:{z}}}"` reaches `z` when building the symbol table.
Give every generic function's type params scope a `.defaults` slot,
whether or not the signature has any default, so `co_varnames` and
`co_nlocals` match. Classes and type aliases keep theirs empty.
Decide an unparenthesized sole generator argument by the parser's
`parenthesized` flag instead of scanning the source for the shape of its
element. `_sum((d := x - c) * d for x in data)` opens with a parenthesized
group the rest of the expression continues, which the scan read as the
generator's own parentheses and left the call's out of its range.
Drop empty f-string fragments unconditionally. A concatenation whose
fragments are all empty now loads its empty string at the whole
concatenation instead of at whichever fragments happened to be plain
string literals.
Keep a byte order mark in source handed over as text, and report it as an
invalid non-printable character. Reading a file to freeze strips it, as
the encoded-bytes path already did.
Carry the filename into the errors `compile(..., PyCF_ONLY_AST)` raises;
they reported `<unknown>` or an empty name.
Across `Lib/`, the 1729 files that compile on both sides now agree on
opcodes, constants, flags, variable names, names, and exception tables,
and on instruction positions in all but 7 files, where a compound
statement whose body ends in a trailing `;` ends one column short.
Assisted-by: Claude Code:claude-opus-5
* Align ast module output with CPython
- Keep lone surrogates in string, f-string and t-string literal values by
reading such a literal from its source; the literal-value helpers move
out of `Compiler` into `rustpython_codegen` and `ConstantLiteral::Str`
holds `Wtf8Buf`.
- Give `{expr=}` interpolations their leading text as a Constant and
default their conversion to `repr`, in both f-strings and t-strings.
- Span folded consecutive literals from the first of them to the last.
- Take the `:` into a format spec's range only when the parser left it
out, instead of always widening by one.
- Report a node end that lands on a line start as that line, column 0.
- Give a generator expression written straight into a call's parentheses
the range of those parentheses, in the preprocess pass, replacing
codegen's source scan and `_ast`'s one-column adjustment.
- Drop the expectedFailure on test_ast_line_numbers_with_parentheses.
Assisted-by: Claude Code:claude-opus-5
* Align symbol table output with CPython
- Register an import as `DEF_IMPORT` alone, and an annotated parameter as
a plain parameter, so `import x; global x` is accepted and annotated
parameters are not reported as annotated names.
- Record a `global` declaration in the module block as well, and resolve a
name a `global` in an enclosing scope covers as an implicit global, so a
class body reading such a name emits `LOAD_NAME`.
- Resolve a name bound nowhere as an implicit global instead of leaving
its scope unknown.
- Require a binding only for a name an explicit `nonlocal` declared;
a free variable that reached the scope any other way already resolved.
- Name a generator expression's block `genexpr`; the code object keeps
`<genexpr>`.
- Locate a `def`/`class` block at its keyword rather than at the first
decorator. `decorated_definition_range()` moves from `Compiler` to
`rustpython_codegen` for that.
- Carry a comprehension symbol's flags over when inlining it into a parent
entry that only holds a free variable propagated from a child.
- Leave `__conditional_annotations__` out of an annotation block's
symbols and have the compiler cook up the free variable, which places
it after the ones the symbol table supplies.
- Check `from __future__` placement while building the symbol table, and
give `symtable.symtable()` the same docstring stripping and future
feature validation the compiler does.
- Floor `co_stacksize` at 1.
Assisted-by: Claude Code:claude-opus-5
* Locate the eval return and the interactive print like CPython
Leave `eval` mode's `RETURN_VALUE` without a location, so the exit block
is duplicated per reaching path and each copy carries the location of the
branch that jumps to it. `a and b` now ends with two returns as it does
in CPython, and `a if b else c` gives the first one the `a` it returns.
Emit an interactive `CALL_INTRINSIC_1(Print)` at the expression statement
rather than wherever the expression left the location, which differ for
`(a := 10)` and `f'{0:fz}'`.
Assisted-by: Claude Code:claude-opus-5
* Give interactive mode the conditional annotations cell
`compile_program_single()` emitted the `__conditional_annotations__` set
and its name accesses but never added the cell that holds it, so a module
compiled in `single` mode was missing the `MAKE_CELL` and the cellvar that
the same source gets in `exec` mode.
Assisted-by: Claude Code:claude-opus-5
* Give compiler-raised SyntaxErrors an end position
`CodegenError` and `SymbolTableError` now carry an `end_location`
alongside `location`, and `CompileError::python_end_location()` returns
it, so errors such as "'await' outside function" set `end_lineno` and
`end_offset` instead of leaving them None.
`SymbolTableBuilder::error_ranged()` builds both positions from a range,
replacing the inline `source_location()` calls at the error sites.
Drop the `expectedFailure` on
`test_exceptions.SyntaxErrorTests.test_file_source`, which now passes.
Assisted-by: Claude Code:claude-opus-5
* Report parser diagnostics with CPython's wording
`InvalidAnnotatedAssignmentTarget` becomes "illegal target for
annotation", `UnexpectedExpressionToken` becomes plain "invalid syntax"
rather than prefixing the parser's text, and the list-recovery
diagnostics ("Expected an expression or a '}'" and its siblings) join the
two that already mapped to "invalid syntax".
Assisted-by: Claude Code:claude-opus-5
* Punctuate the code object repr like CPython
`repr(code)` was missing the comma after the address and escaped the
filename through Rust's Debug formatting.
Drop the `expectedFailure` on the six `test_dis` tests that compare
disassembly text, which now pass.
Assisted-by: Claude Code:claude-opus-5
* Match CPython's parser error positions
A syntax error the parser reports counts its columns in characters, so
`source_location()` converts with UTF-32 rather than UTF-8; an offset
inside a character walks back to where that character starts.
Spans the parser reported differently:
- an error between two tokens covered nothing, and now covers one
character, since the narrowest token still covers one
- a "was never closed" bracket ends at column 0
- an unterminated string literal ends where it starts
- "leading zeros in decimal integer literals" spans the run of zeros
- an exponent with no digits and no sign is blamed on the digit before
the exponent letter, which the tokenizer puts back
Assisted-by: Claude Code:claude-opus-5
* codegen: seek to the finally body's own scopes before copying it
An early exit from a try block emits an extra copy of the finally body.
Nested scopes are handed out by position, and the cursors still sit inside
the try block's own run of scopes there, so the copy took those instead of
the ones the finally body opens: a generator expression got a sibling's
symbol table and raised "the symbol 'k' must be present in the symbol
table", or, where the sibling was a named scope, built a code object with
no `.0` argument that raised UnboundLocalError when it ran.
Seek the cursors to the first scope beginning on or after the finally
body's first line before compiling the copy. The existing restore then
leaves the try statement's own copies starting from the same place.
Assisted-by: Claude
|
||
|
Luan Taraschi
|
b9f591b665 |
Follow CPython's padding rules in a2b_base64 (#8570)
With strict_mode=True the decoder accepted padding CPython rejects:
>>> binascii.a2b_base64(b'YWJj=', strict_mode=True)
b'abc'
It also dropped data in the default mode, because it returned at the
first pad that completed a quad:
>>> binascii.a2b_base64(b'abc=a')
b'i\xb7' # CPython: b'i\xb7\x1a'
The loop now counts pads and decides at the end, the way CPython does,
so a stray pad is ignored outside strict mode and named as leading,
excess or discontinuous inside it. DecodeError::InvalidPadding was dead
in this file and now carries the excess padding message.
test_base64_strict_mode and test_base64_excess_data were marked as
expected failures and pass now.
Assisted-by: Claude Code:claude-opus-5
|
||
|
Luan Taraschi
|
c3d22bb225 |
Insert the item rather than its key in bisect.insort (#8565)
insort_left and insort_right rebound `x` to `key(x)` and then handed that
same value to both the search and `a.insert`, so the object the caller
passed in never reached the list:
>>> words = ["a", "ccc"]
>>> bisect.insort(words, "bb", key=len)
>>> words
['a', 2, 'ccc']
The key now feeds the search only, and the insert keeps the original.
Assisted-by: Claude Code:claude-opus-5
|
||
|
hyoinandout
|
e5c8701d44 |
csv: validate dialect options (#8402)
* csv: validate dialect options Resolve each dialect once and validate the merged options before constructing readers and writers. Handle Unicode character parsing consistently and enable the corresponding CPython CSV tests. Assisted-by: Tau:gpt-5.6-luna * cargo fmt * Validate csv dialect conflicts with line terminators - Reject dialect characters that overlap any lineterminator character - Validate Dialect construction consistently and allow non-ASCII terminators - Unskip csv invalid character coverage now that validation matches behavior * Remove stale CSV rejection snippet Removes a RustPython-only assertion that contradicts the CPython-compatible dialect validation added by this branch. Scope: test-only AI-Assisted-By: Codex * Revert "Remove stale CSV rejection snippet" This reverts commit |
||
|
Luan Taraschi
|
3144a00bda |
Drop the lint expectations clippy 1.98 no longer fulfils (#8564)
`clippy::std_instead_of_core` stopped firing at the four places that carry an `#[expect]` for it. All four were added for the same false positive, on items whose `core` counterpart is still unstable. `unfulfilled_lint_expectations` is denied through `-D warnings`, so the three clippy jobs and the WASM check fail on current stable for every pull request, whatever it touches. Assisted-by: Claude Code:claude-opus-5 |
||
|
Jeong, YunWon
|
a7981dccc7 |
Align compiler futures, annotations, and symtable with CPython (#8550)
* Align compiler futures and annotations with CPython Handle Barry parsing consistently across AST, type-comment, REPL, and WASM paths. Preserve deferred annotation source and scope metadata, and make _symtable conversion cached and linear-time. Assisted-by: Codex:gpt-5.4 * Fix f-string debug ranges and update Barry/annotate test expectations Compute the debug f-string source range from the raw leading and trailing text instead of the comment-stripped text, so a comment inside a multi-line replacement field no longer shifts the emitted LOAD_CONST location. Restrict the obsolete-operator location shift to `<>` by also requiring a `>` at the reported offset; without it any `ExpectedExpression` that followed a `<` (`2 <;`) was reported one column to the left. Add `CodeFlags::FUTURE_MASK` and use it in builtins, the shell, and the WASM VM instead of repeating the eight-flag union, and collapse the duplicated Barry diagnostic sequences into `BarrySource::diagnostic`. Update the `__annotate__` disassembly snapshot to LOAD_FAST_BORROW and replace the RustPython-only `barry_as_FLUFL` co_flags assertion in extra_tests with the CPython-matching one. Assisted-by: Claude Code:claude-opus-5 * Restore the annotated function in __annotate__ qualnames Signature annotation scopes are compiled in the scope enclosing the annotated function, so the function name has to be folded back into the qualname: `f.__annotate__`, `C.m.__annotate__`, `outer.<locals>.inner.__annotate__`. test_type_annotations test_annotate_qualname covers this. test_pyrepl test_future_barry_as_flufl now passes, so drop its expectedFailure marker. Assisted-by: Claude Code:claude-opus-5 * Pass safepoints in the attached-thread interpreter tests `main_and_subinterpreter_run_sections_overlap` parked its workers on a condvar while attached, and `busy_main_interpreter_does_not_block_subinterpreter` looped over protocol calls that never reach a safepoint. An attached thread that never reaches a safepoint cannot be suspended, so a concurrent process-wide stop-the-world (from the gc test running in parallel) never completes and the already-stopped sibling interpreter stays stopped; the second worker then waits in `wait_while_suspended` until the 30s deadline. Both loops now call `vm.check_signals()` each iteration, and the condvar wait uses `wait_timeout` so the lock is released between safepoints. Reproduced on Linux (4 CPUs): the two tests with `--test-threads=2` failed 2/40 before and 0/40 after; the full `rustpython-vm` lib binary failed 1/30 before and 0/30 after. Assisted-by: Claude Code:claude-opus-5 * Report the obsolete `<>` operator from one place `<>` was located twice: `BarrySource` recorded the token ranges it rewrote to `!=`, and `cpython_parse_diagnostic_override` separately peeked at the bytes around an `ExpectedExpression` location. Both produced "invalid syntax" spanning the whole operator. `prepare_barry_as_flufl_source` now records the operator ranges in the non-Barry case too -- by plain text search, since nothing is rewritten there -- so `BarrySource::invalid_legacy_operator` covers both, and the byte-peeking helper is gone. It also defers to an unclosed bracket earlier in the source, which the diagnostic chain it used to sit in did for it. A `<>` that starts a statement now spans both characters: the parser stops at the `<` rather than one character in, which the byte peek did not handle. Assisted-by: Claude Code:claude-opus-5 * Drop a py object's payload before its class `PyInner` declares `typ` before `payload`, so `drop_in_place` dropped the class first and `PyAtomicRef::drop` left it null. A `PyWeak` payload is still linked into the weakref list of the object it points at until its own `Drop` unlinks it under the stripe lock, which happens after that. `WeakRefList::add` walks the same list looking for a proxy to reuse and reads `node.class()` off every node it passes, so it could dereference the null class of a node on its way out. `PyInner::dealloc` now runs the two destructors in the other order at all three of its exit paths, with a compile-time check that no other field has one. Reproduced by the new test in about 0.1s, as the panic CI hit in the c-api `weakrefobject::tests::new_proxy` run: panicked at crates/vm/src/object/ext.rs:312: unsafe precondition(s) violated: hint::unreachable_unchecked must never be reached Assisted-by: Claude Code:claude-opus-5 |
||
|
Jeong, YunWon
|
4fab5c966e |
Raise where the fuzzer found aborts and stack overflows, and collect with the count in the object header (#8551)
* Correct what the non-unix frame stack is for
The comment said non-unix threading builds have no stop-the-world.
CollectStopTheWorld is gated on `feature = "threading"` alone, and
sys._current_frames stops the world on both paths; the field is the
fallback a reader uses when there is no `top_iframe` to materialize from.
Assisted-by: Claude
* Seek by the whole file position on Windows
`SetFilePointer` answers with the low half of the new position and signals
failure with INVALID_SET_FILE_POINTER, which is also that half of a
position four gigabytes in; telling them apart takes the error code, which
this did not read, so such a seek was reported as an error. Deciding
seekability from it also called such a file unseekable.
`SetFilePointerEx` returns the whole position and a success flag of its
own, which also removes the transmute of the position into halves.
Assisted-by: Claude
* memoryview: tell a value of the wrong kind from one that does not fit
Assigning an item reported every packing failure as a TypeError, so
m[0] = 300 on a 'B' view said the value was the wrong type rather than out
of range. Packing now says which of the two it was, and whether the value's
own code raised, in which case that error is the answer as it is:
m[0] = 300 ValueError: invalid value for format 'B'
m[0] = "x" TypeError: invalid type for format 'B'
m[0] = <__index__ that raises> the raised error
`struct` reports both as `struct.error` and is unchanged; the kind travels
beside the exception for the caller that tells them apart.
Also: None was read as a deletion, so m[0] = None answered "cannot delete
memory" instead of packing it; and deleting through the mapping protocol
never reached the read-only check, which comes first.
Assisted-by: Claude
* Raise MemoryError for allocations sized by Python input
os.read, _RawIOBase.read, int.to_bytes, struct.pack, ctypes array
creation and the _ssl RAND functions sized a Vec from a Python-supplied
length with vec![], which calls handle_alloc_error and, under
panic = "abort", ends the process. They now allocate through
vm.new_zeroed_bytes and raise MemoryError.
itertools.product built its pools without checking that
len(iterables) * repeat is representable; it now raises OverflowError
"repeat argument too large" and reserves the pool and index vectors
fallibly. repeat is read as isize, so a negative one raises ValueError
"repeat argument cannot be negative" instead of the conversion's message.
_ssl.RAND_bytes and RAND_pseudo_bytes read n as i32, matching the int
they are declared with.
Assisted-by: Claude
* Collect with the count in the object header
A collection kept the count it was working with in a table keyed by the
object's address, and the objects it had proved reachable in a second one.
Between them they were hashed once per candidate and twice per edge in the
heap, which is where most of a collection over a live heap went.
The count now lives in `PyInner::gc_refs`, with `GcBits::COLLECTING` saying
it is meaningful, and reachability is `gc_refs == GC_REACHABLE` rather than
membership in a set. Step 5 splits the candidates and clears the bit in one
pass. gcbench, five interleaved pairs, median: a live heap of 423k objects
goes from 0.101s to 0.055s and a dead one from 0.402s to 0.383s.
The bits, generation, owner and count take eight bytes between them. A
64-bit header had those eight as the padding its alignment forces, so it is
unchanged at 48 bytes; a 32-bit header grows from 24 to 28.
Assisted-by: Claude
* Take an explicit thread stack size as a floor in debug builds
A debug build already started Python threads on 8 MiB rather than Rust's
2 MiB default, but an explicit threading.stack_size(N) went through
verbatim. test_threading asks for 256 KiB, and starting a thread on that
walked off the end of the stack: the guard page fault landed in the
prologue of ExecutingFrame::run.
Unoptimized, that prologue reserves 80,848 bytes where the optimized one
reserves 656 -- execute_instruction is #[inline(always)] and LLVM only
colors stack slots from opt-level 1, so the frame is the sum of all 200
instruction arms' temporaries rather than the largest. A Python call costs
88,672 bytes of native stack there, and threading's bootstrap is six
frames deep, so 256 KiB holds less than half of what starting a thread
takes.
The floor reaches thread::Builder only; threading.stack_size() still
answers with what was asked for, and release builds are unchanged.
Assisted-by: Claude
* Check the native stack on every frame entry
The C-stack guard ran on one frame entry in eight. That asks the margin to
cover eight frames rather than one, and it does not: an unoptimized frame
entered through native code takes 88,672 bytes against a debug margin of
262,144. A recursion whose steps re-enter that way -- `__add__` calling
itself, a sort key that sorts -- ran off the end of the stack instead of
raising RecursionError. On a debug build `class Add: __add__ = lambda s, o:
s + o; Add() + 1` segfaulted on the main thread; it now raises, as it does
under CPython and in release builds.
enter_iframe checked and then called enter_iframe_unchecked, which checked
again; it now leaves the check to the one call.
Measured on a call-dominated benchmark, five interleaved pairs: instructions
retired go up 0.17%, about four per call, which is the stack pointer read
and the compare.
Assisted-by: Claude
* Take an iterable's length hint when a list is filled from it
`map_py_iter` read `__length_hint__` only to pass it to `PyIterIter`, and
returned an empty vector when the hint was `isize::MAX` or more. Collecting
through `PyResult` dropped the iterator's lower bound, so nothing reserved
the room the hint asked for.
`list()`, `list.extend()` and `list.__iadd__()` now reserve it and report a
hint they cannot honour as `MemoryError`; a hint that leaves no room for the
elements the list already holds is passed over, as `list_extend()` does.
`tuple()` and the other callers keep filling up without reserving.
`length_hint_opt` errors other than the `TypeError` it already turns into
`None` now reach the caller instead of being dropped.
`__iadd__` and `inplace_concat` went through `extract_cloned`, which reads
`__len__` and not `__length_hint__`; both call `PyList::extend` now, the way
`list_inplace_concat()` calls `list_extend()`.
The tuple, list and dict fast paths of `extract_elements_inner` reserve their
known length, which the `collect()` they used dropped.
Assisted-by: Claude
* Drop map.__length_hint__
`map_methods` has no `__length_hint__`, so `operator.length_hint()` on a map
answers 0, not the length of what it draws from.
The method walked into the length hint of every iterator it holds, and a
chain of maps 10000 long overflowed the native stack answering for the
outermost one. It also took the longest of its iterators, where a map stops
at the shortest.
Assisted-by: Claude
* Ask for a length hint where each caller asks for it
`map_py_iter` asked the iterable it was handed, for every caller, and reported
what asking raised. Only some callers ask it: `list_extend()` and
`_PyBytes_FromIterator()` ask the iterable, `PySequence_Tuple()` asks the
iterator, and the bytearray constructor asks nothing. `tuple()`, `min()`,
`max()`, `collections.deque()` and `f(*x)` raised for an iterable whose
`__len__` or `__length_hint__` does, where they answer.
Which object is asked is now the caller's to say. `sorted()` asks the iterable,
being `PySequence_List()`.
`bytes_from_object()` stood in for `PyBytes_FromObject()`, for
`bytearray_extend()` and for the bytearray constructor, which do not agree on
this: the first two ask, the last does not. It is split, and assigning to a
bytearray slice takes the constructor's side with `PyByteArray_FromObject()`.
`list.extend()` counted what it held before the iterable had been asked, where
`list_extend()` reads `Py_SIZE(self)` after. A `__length_hint__` that adds to
the list made the overflow guard read a count too small and raise `MemoryError`
where nothing is wrong; one that empties it made the guard skip a reservation
that cannot be served.
Assisted-by: Claude
* Settle product's pool count before it reads its arguments
`product_new()` checks `repeat` and works out `npools` before it calls
`PySequence_Tuple()` on any argument, and fills the pools `npools` times.
The pools were filled by repeating the arguments `repeat` times instead, which
walks that many steps even with no arguments to repeat: `product(repeat=2**62)`
counted up to it rather than answering `[()]`. The count was also worked out
after the arguments had been read, so a repeat too large to serve ran their
code first.
Assisted-by: Claude
* Let the bool format answer with the error its value raised
`pack_single()` leaves `'?'` to `PyObject_IsTrue()` and returns what that
raised. Packing classified the error instead, so a `ValueError` from a
`__bool__` came back as "memoryview: invalid value for format '?'".
Assisted-by: Claude
* Release a cell's old value after the lock
`PyCell::set` dropped what it replaced while still holding the mutex guarding
the cell contents. A `__del__` running from that drop and reading the same cell
waited on a lock its own caller held, so `del it` on a closure variable whose
value has such a `__del__` deadlocked.
The replaced value is now released once the guard is gone, as `Py_XSETREF`
stores before it decrefs.
Assisted-by: Claude
* Have a set iterator hold the set it iterates
The iterator kept only a reference to the inner hash table, so in
`it = iter(A(*args))` the `A()` temporary was the last owner and died as the
call returned, before `it` was bound. A `__del__` reading `it` there saw an
unbound name and its error was printed and ignored.
The iterator now holds the set object, as `si_set` does, and releases it once
exhausted. That release happens after the lock is dropped, where
`setiter_iternext()` puts its `Py_DECREF(so)` past `Py_END_CRITICAL_SECTION()`,
so a `__del__` that iterates again does not wait on a lock the call holds.
Assisted-by: Claude
* Release an exhausted iterator's container after the lock
`PositionIterInternal::_next` overwrote its `IterStatus::Active` while the
caller still held the mutex around it. Dropping the container there ran any
`__del__` under that lock, and a `__del__` that iterated the same object again
blocked on it.
`exhaust()` now hands the container back instead of dropping it, and
`locked_step()` releases it after the guard. list, list_reverseiterator, tuple,
str, dict and its views and reverse views, bytes, bytearray, memoryview,
array, deque, and the enumerate and sequence iterators all go through it.
Assisted-by: Claude
* Unskip test_free_after_iterating
Assisted-by: Claude
* Stop listing test_set as an environment polluter
`check_free_after_iterating` no longer leaves an ignored exception behind, and
the job that reruns the listed tests ten times fails once one of them stops
polluting.
Assisted-by: Claude
* Drop winsound imports left unused
`TryFromBorrowedObject`, `exceptions`, and `ToWideString` have no reference in
the module, which fails the Windows clippy line under `-Dwarnings`.
Assisted-by: Claude
* Ask nothing where the caller takes no room
`map_py_iter` asked the iterator for a length hint on behalf of every caller
that does not reserve, and reported what asking raised. Those callers ask
nothing at all: `tuple()`, `f(*x)`, `bytearray(x)`, `min()`, `max()` and
`collections.deque()` answer for an iterator whose `__length_hint__` raises,
where they had been raising it.
The answer was also never spent. It reached `PyIterIter` for a `size_hint()`
the push loop does not read, so the lookup and any call it made were work
thrown away: `tuple()` over a generator drops 22% of its instructions, and 17%
over an iterator with a `__length_hint__` written in Python.
Assisted-by: Claude
* Stop asking an iterator how long it is to walk it
`PyIter::iter` and `PyIter::into_iter` asked for a length hint and reported
what asking raised. Nothing spent the answer: it reached `PyIterIter` for a
`size_hint()` that every caller either loops past or drops, since collecting
into a `Result` reports no lower bound.
23 operations answered for an iterator whose `__length_hint__` raises, where
they had been raising it: `set`, `frozenset` and the nine `set` methods that
take an iterable, `dict.fromkeys` and the dict view operators, `array` and
`array.extend`, `all`, `any`, `sum`, `io.writelines`, `math.fsum`,
`math.prod`, and `csv.writerow` and `writerows`.
Over a generator, `set()` drops 16% of its instructions and `all()` 23%.
`str.join` and `bytes.join` do ask, reaching their elements through
`PySequence_Fast()`, which fills a list from the iterator. They take
`iter_sized()`, which is now the only way to ask. `iter_without_hint` is gone,
its callers being what `iter` already does.
Assisted-by: Claude
* Name the matrix jobs rather than let the matrix name them
A generated job name lists every value in the matrix entry, so `cargo check`
carried the booleans its dependencies and `skip_ssl` keys expand to, and the
snippets job carried its test arguments and timeout. Adding or removing a key
renames the check, which drops it from the required list until that list is
edited to match. Emptying `env_polluting_tests` renamed three checks this way.
`Run rust tests` and `clippy` keep the names they had. `cargo check` drops the
booleans from six of its nine, and the snippets job drops its arguments and
timeout from all three.
Assisted-by: Claude
* Keep a sequence iterator active when an element raises
`PositionIterInternal::_next` exhausted the iterator for any non-`Return`
result, so an error from `__getitem__` ended the walk. `iter_iternext()`
lets go of its sequence for `IndexError` and `StopIteration` alone, and
`PyIterReturn::from_getitem_result` has already turned the first of those
into the second, so only `StopIteration` exhausts now.
Both deque iterators keep exhausting on their own mutation guard, which
`deque_iternext()` does by zeroing the counter before it raises; they
share one step function for it, and the message it raises is lowercased
to match the three other sites in the module.
Assisted-by: Claude
* Name the caller in the bytes conversion TypeError
An object with no iteration protocol reached `PyObject_GetIter`'s "not
iterable" message, and `bytearray.extend()` reported bytes. Each entry
point now checks for the protocol first and names itself:
bytes(object()) cannot convert 'object' object to bytes
bytearray(object()) cannot convert 'object' object to bytearray
bytearray().extend(object()) can't extend bytearray with object
Assisted-by: Claude
* Take a strong count too large to hold as reachable
`start_gc_refs` clipped a count at `GC_REACHABLE - 1`, a number the
per-reference subtraction could still walk down to zero and collect a
live object. It now stores `GC_REACHABLE`, and `subtract_gc_ref` leaves
that value alone.
Assisted-by: Claude
* Raise AssertionError where a snippet asserts False
`assert False` is removed under `-O`, which the snippet suite may run.
Assisted-by: Claude
* Keep raising for a collection that moved under its iterator
A deque, set or dict iterator raised once and then read as spent. The
guard is sticky: `deque_iternext()` looks at the deque's state before the
count it keeps, and `dictiter_iternextkey()` and `setiter_iternext()`
write a size no collection can have, so every later call finds the same
thing and raises again. `dequereviter_next()` is the one exception,
looking at its count first, so it runs out after the first raise.
Both deque iterators now carry `dequeiterobject.counter` rather than
reading a length back from the deque, and the dict and set iterators
compare the size they captured against the collection's own every time
they are asked how much is left, which is what makes that answer nothing
from the moment the collection changes rather than only once the
iterator has raised.
The set's message is capitalized to match `setiter_iternext()`.
Measured against 3.14.6, for each of deque, reversed deque, set, dict
and a reversed dict view: the hint after the change, the error, the hint
after the error, and what a later call answers.
Assisted-by: Claude
* Give apt-get update a deadline to fail on
The step waits on `apt-get update`, which has no deadline of its own, so
a source that takes the connection and then stops answering holds the
job until the workflow's own timeout. The retry that disables the
Microsoft and azure-cli sources runs only when the update exits
non-zero, which a held connection never does; three jobs on this branch
sat on this step for 20 minutes to five and a half hours.
Each attempt is now bounded, and the transports are given a timeout, so
a source that stops answering reaches the retry.
Assisted-by: Claude
|
||
|
|
6ea492207d |
ffi: No interior NULs (part 1) (#8245)
* ffi: No interior NULs (part 1) Interior NULs is a security hazard for C-style strings. A NUL byte truncates a string which can lead the caller and callee to see two different strings. It can cause path traversal attacks where a path in Python looks complete but it is interpreted differently through FFI. RustPython needs to handle this for some of its C-API as well as raw libc or Windows calls. Both Rust's standard library as well as Rustix handle interior NULs for us with CStrings, so this mostly affects a handful of Windows functions or areas where we have raw bytes that weren't checked by CString. Finally, this PR is non-exhaustive. I will have to rely heavily on CodeRabbit to help lint it to ensure that interior NUL checks are only introduced for FFI and not outside of it. Most of RustPython seems to handle interior NULs already due to CString as well as WideCString. **Sources:** * https://owasp.org/www-community/attacks/Embedding_Null_Code * python/cpython#11656 * rename null_terminated_bytes --------- Co-authored-by: Jeong, YunWon <69878+youknowone@users.noreply.github.com> Co-authored-by: Jeong, YunWon <jeong@youknowone.org> |
||
|
Jeong, YunWon
|
adf7da508b |
memoryview: count exports, hash the exporter, and index through __index__ (#8553)
* memoryview: index through `__index__` and name the dimension
A tuple key was taken apart only when every item was already an `int`, so
`m[I(), 2]` with an `__index__` on `I` was reported as an invalid slice key
rather than indexed. What kind of key it is now follows from the types in
it, the way `is_multiindex` decides, and each item is converted where it
sits; an item that answers `__index__` but raises on the way reports that
rather than making the whole key invalid.
That conversion runs Python, which can release the view, so the
multi-dimensional read goes through `unpack_single` and its re-check
rather than reading the bytes itself.
`IndexError` named the index that was out of bounds where
`lookup_dimension` names the dimension it was out of bounds on, counted
from one, and the one-dimensional path had a message of its own.
Assisted-by: Claude
* memoryview: count what a view has exported, and hash the exporter
The view kept no count of the buffers taken from it, so `release()` always
succeeded, even while something was still reading through an export. It now
answers `BufferError` the way `_memory_release` does, and `__exit__` reports
the same refusal.
`memory_hash` asks the exporter for its hash and throws the answer away, so
a view is no more hashable than what it looks at; a read-only view over a
`bytearray` hashed here where CPython refuses it. Asking runs Python, so the
view counts as exported for the duration and a release attempted from inside
that hash is refused rather than obeyed.
Assisted-by: Claude
* bytes, bytearray, memoryview: measure the hex separator as `len()` does
`hex()` took the separator's length from its bytes rather than from the
object, so a `bytes` subclass that answers `__len__` was measured by what it
holds instead of what it says: `b'abcd'.hex(S(b'::'))` with `__len__`
returning 1 was refused where `_Py_strhex_impl` accepts it, and a two-byte
answer was accepted where it refuses. The check also came after the
`bytes_per_sep == 0` and empty-data shortcuts, so `b'abcd'.hex('::', 0)`
never reached it at all.
Measuring runs Python, and the bytes to be written out must not be borrowed
while it does, so the separator is now resolved from the arguments before
the buffer is reached. For a memoryview that also means the view counts as
exported for the duration, so a release attempted from inside `__len__` is
refused.
Assisted-by: Claude
* array: refuse to hash an array
`array.array` is mutable but inherited `object.__hash__`, so an array could
be put in a set or used as a dict key and then changed underneath it. It is
unhashable, as `tp_hash = PyObject_HashNotImplemented` makes it.
A read-only memoryview over an array now reports the exporter as unhashable
too, which is what `memory_hash` asks it for.
Assisted-by: Claude
* Remove expectedFailure from three memoryview tests
test_hash_use_after_free, test_hex_use_after_free and
test_use_released_memory pass.
Assisted-by: Claude
* memoryview: bound the dimensions a cast can name
`cast()` accepted a shape of any length, so `mv.cast('B', (1,)*64 + (8,))`
built a 65-dimensional view where `memoryview_cast_impl` refuses anything
past `PyBUF_MAX_NDIM`. The limit is answered before the shape is looked at
any further, as it is there.
Assisted-by: Claude
|
||
|
Jeong, YunWon
|
819b4b86fd |
Fix malformed Unicode error constructors (#8497)
* Fix malformed Unicode error constructors Replace message-only Unicode decode and encode errors with fully initialized exceptions, preserve source objects and failure ranges, and remove the obsolete constructors. Assisted-by: Codex:gpt-5 * Fix Windows Unicode CI regressions Assisted-by: Codex:gpt-5 |
||
|
Jeong, YunWon
|
a78d0eb577 |
Fix crashes found hunting the last open fuzzing record (#8524)
* specialize: check the member descriptor's type before caching its slot offset
The LOAD_ATTR/STORE_ATTR specializations cached the slot offset of any member
descriptor found on the owner's type and then guarded the specialized
instruction on the type version alone, while descr_get()/descr_set() check on
every access that the instance belongs to the type the descriptor was defined
for. A descriptor taken from a wider class and bound to a narrower one read
past the instance's slot array once the cache warmed up:
class Big:
__slots__ = ("a0", "a1", "a2", "a3", "a4", "a5", "a6", "a7")
class Narrow:
__slots__ = ("z",)
Narrow.x = Big.__dict__["a7"]
o = Narrow()
for _ in range(1000):
try: o.x
except TypeError: pass
# index out of bounds: the len is 1 but the index is 7 (object/core.rs)
A class with no slots at all reached the ext_ref().unwrap() on the same line.
Assisted-by: Claude
* socket: reserve recv()'s buffer fallibly
recv() and recvfrom() handed the caller's bufsize straight to
Vec::with_capacity, so an unreachable size aborted the process through
handle_alloc_error before any syscall was made:
socket.socket().recv(2**62)
# memory allocation of 4611686018427387904 bytes failed -> SIGABRT
try_reserve_exact reports MemoryError instead, which is what CPython raises.
Assisted-by: Claude
* types: count the __call__ and __get__ slot dispatches as recursion
Both wrappers re-enter Python without pushing a frame, so nothing counted the
nesting when the special method named the object it was looked up on:
class C: pass
c = C(); C.__call__ = c
c() # native stack overflow, SIGSEGV
class D: pass
d = D(); D.__get__ = d; D.x = d
d.x # the same, through descr_get
with_recursion around the two dispatches raises RecursionError instead, the
way Py_EnterRecursiveCall bounds a tp_call dispatch. It costs about 5% on a
__call__ dispatch and 3% on a __get__ dispatch through these wrappers.
Assisted-by: Claude
* typevar: show a ParamSpecArgs origin by its repr
ParamSpecArgs and ParamSpecKwargs fell back to a Rust `{:?}` of __origin__ when
it had no __name__. That walks the object graph natively through Debug for
PyInner, where no recursion guard sits, so a single repr() of a deeply nested
chain overflowed the native stack:
a = object()
for _ in range(30000):
a = typing.ParamSpecArgs(a)
repr(a) # SIGSEGV
The origin is formatted with its repr now, which is guarded, and a ParamSpec
origin is recognized by its type rather than by carrying a __name__.
Assisted-by: Claude
* Do not hold a lock across a call back into Python
Three places kept a lock while running code that can reach the same object, so
a callback that touched it wedged the process:
_asyncio.future_add_to_awaited_by(fut, waiter) # waiter.__hash__ adds again
select.select(elements, [], [], 0) # fileno() clears `elements`
select.poll().poll(1000) # SIGALRM handler registers
The future's awaited-by field is read and written under its lock but the set is
built outside it, the list extraction re-reads the list on each step the way
map_iterable_object() does, and poll() waits on a copy of its descriptors. All
three ran forever before and now finish the way they do on CPython.
Assisted-by: Claude
* Validate memoryview.cast() arguments and export negative strides correctly
cast() accepted any struct format and any shape element. A zero-size
format ('0s') and a 0 in the shape both reached a division by zero;
cast() now takes only a native single character format, optionally
'@'-prefixed, and shape elements that are ints greater than zero.
A view with a negative stride starts at its last item, so the bytes it
exported began there and its own offsets walked off the front of them.
Such a view now exports the whole underlying buffer with `start` folded
into the descriptor's offsets, and zip_eq() hands over a whole run only
when both sides are contiguous in the last dimension.
Assisted-by: Claude
Assisted-by: Codex:GPT-5
* Charge native recursion to the stack, not to the frame limit
with_recursion() checked the limit sys.setrecursionlimit() sets and
incremented the same counter that pushing a frame does, so a guard on a
native dispatch spent what Python code had left to call with, and did so
where sys._getframe() cannot see it: test.support.get_recursion_available()
reported frames that were no longer there. Py_EnterRecursiveCall bounds
the native stack instead, which is a separate budget, and the C stack
check with_recursion already performs is that bound.
The snippets pinning the guarded paths nest deep enough to reach the
stack rather than the frame limit.
Assisted-by: Claude
* Report a size that cannot be allocated instead of aborting on it
A size taken from Python went straight into an infallible allocation in
several places, so the process aborted through handle_alloc_error before
any exception could be raised:
- str/bytes/bytearray center(), ljust(), rjust() and zfill() reserved
the padded result for the caller's width
- expandtabs() built its runs of spaces from a tabsize of any width;
the argument is a C int, and a wider one does not fit
- Buffered{Reader,Writer,Random} allocated buffer_size, and read(),
read1() and FileIO.read() their read size
- bytes(n) and bytearray(n) allocated n
- pbkdf2_hmac() allocated the derived key length, which is a C int
Each of these now reports MemoryError, or OverflowError where the
argument does not fit the type it is declared with.
new_zeroed_bytes() leaves the zeroing to the allocator, so a large
request costs the pages that are written to rather than all of them.
Assisted-by: Claude
* marshal: answer allow_code where a code object is written or read
allow_code was answered by walking the whole result a second time, with
no depth counter and no record of what it had already seen, so a value
that referred back to itself or nested deeply enough ran off the native
stack. w_object() and r_object() answer it where the code object is,
inside the walk that already bounds its depth and resolves references.
A container length is read the way r_long() reads one: it is signed, so
a length with the top bit set is out of range rather than four billion
items to reserve room for.
load() no longer holds a borrow of the buffer read() returned across the
seek() it makes afterwards.
Assisted-by: Claude
* Do not lock an object while running code that can reach it
Several places held a lock or a borrow of an object across a call back
into Python, so a callback that touched the same object waited on a lock
its own caller was holding:
- memoryview slice assignment read a source overlapping the destination,
and __setitem__ converted the value while holding the write borrow
- BytesIO.readinto() read into a buffer viewing the same BytesIO
- array.__setitem__ converted the value under the array's write lock,
and mmap.write() read a source viewing the same map
- bytearray.join() and bytearray.__mod__ drove Python with the
bytearray borrowed
- array and bytearray answered "is this resizable" after taking the
write lock, though an export is exactly a borrow someone else holds
A TextIOWrapper cookie now has to name a position inside what was
decoded in characters as well as in bytes; only the byte offset was
checked, and the character count is what read() and tell() index with.
Assisted-by: Claude
Assisted-by: Codex:GPT-5
* Stop asserting a pbkdf2 message that depends on the width of a C long
The snippet asserted "key length is too great.", which pbkdf2_hmac()
only reaches once the length has been converted; where a C long is
narrower than the length asked for, the conversion fails first and says
so instead. Both are OverflowError, which is what the case is about.
test_support.test_get_recursion_depth passes now that a native recursion
guard no longer spends frames get_recursion_depth() cannot see.
Assisted-by: Claude
* Publish and read the same pointer for a thread's top frame
set_current_frame() casts the `Py<FrameObject>` it publishes straight to
`*mut FrameObject`, so ThreadSlot::top_frame holds the object's base.
sys._current_frames() read it back through Py::from_payload_ptr(), which
subtracts the payload offset from what it is given. The reference it took
therefore incremented, and later decremented, a word 48 bytes ahead of
the frame -- inside the object allocated before it, whose OnceLock state
word sits exactly there for two frames adjacent in the size class. The
neighbour then read an initialized-looking cold pointer that had never
been written and locked whatever the uninitialized word addressed, so
the thread that owned it crashed rather than the one that read.
The slot now holds `*mut Py<FrameObject>`, which is what both sides mean.
A thread parked in a call has no FrameObject for its topmost frame, so
top_frame is null there and the reader takes the materialize path
instead: test_sys.test_current_frames never reaches the branch. The
snippet takes _current_frames() against threads that are running.
Assisted-by: Claude
Assisted-by: Codex:GPT-5
* Decide stop-the-world parking under the thread registry lock
do_suspend() published SUSPENDED first and only then re-read `requested`,
restoring itself to ATTACHED if the stop had ended in the meantime. That
made a thread the second writer able to leave SUSPENDED, so a stop whose
completion check had already observed the thread parked could be undone
behind the requester's back:
worker CAS ATTACHED -> SUSPENDED
requester all_non_requester_suspended() -> true, world_stopped = true
requester start_the_world(): requested = false, then walks the registry
worker reads requested == false, stores ATTACHED
With the store landing inside that walk the debug assertion in
start_the_world fires; with the walk already past the slot, a following
stop force-parks the thread DETACHED -> SUSPENDED, counts it as stopped,
and the store then puts it back to ATTACHED with the world declared
stopped and the thread running bytecode.
`requested` is set in init_thread_countdown() and cleared in
start_the_world() with the registry held, and start_the_world() keeps
holding it while releasing every SUSPENDED thread. Taking the registry
around the check and the transition therefore makes the two orders the
only ones possible: park before that release pass and be woken by it, or
find the request already withdrawn and stay ATTACHED. The requester is
left as the only writer that takes a thread out of SUSPENDED, and the
self-restore is gone.
suspend_if_needed() takes the VirtualMachine to reach the registry.
Assisted-by: Claude
Assisted-by: Codex:GPT-5
* Keep an atexit callback alive while it is being compared
atexit.unregister() releases the callback list around each __eq__ call and
identified the entry it had compared by the address of its Box. __eq__ can
call atexit._clear(), which drops that Box, and atexit.register(), whose
new Box lands on the freed allocation; the identity search then matched
the freshly registered callback and removed it.
atexit.register(a); atexit.register(b); atexit.register(c)
# __eq__ runs _clear() then register(d), returns True
atexit.unregister(probe)
left no callbacks registered where CPython leaves d.
Entries are Arc-shared now, so unregister() holds the one it is comparing
and matches it with Arc::ptr_eq: an address cannot be reused while the
comparison that named it is still running.
Assisted-by: Claude
* Hold atexit entries in PyRc rather than Arc
PyObjectRef is Send and Sync only under the threading feature, so an Arc
over a callback entry trips clippy::arc_with_non_send_sync in builds
without it, such as the wasm package. PyRc is Arc there and Rc otherwise.
Assisted-by: Claude
* Do not hold a buffer's storage while waiting for a peer
FileIO.readinto() and socket.recv_into()/recvfrom_into() took the target
buffer's write borrow and kept it for the whole call, including the wait
for data a pipe, socket or terminal may never deliver. What CPython holds
across that wait is the export, which only forbids resizing; the borrow is
a lock every other thread touching the same object waits on, so
threading.Thread(target=lambda: sock.recv_into(buf)).start()
len(buf)
did not answer until the peer sent. A thread parked on that lock is
ATTACHED and never reaches a safepoint, so gc.collect() in a third thread
waited for the peer as well: one incidental read of the buffer stopped the
world from being stopped at all.
The wait now runs against storage of its own and the bytes are copied over
once they arrive, with the export held throughout so the target still
cannot be resized meanwhile. A seekable file answers from itself rather
than from a peer, so FileIO.readinto() writes into the target directly
there and the buffered read path is unchanged.
Assisted-by: Claude
* Do not hold a buffer's storage while waiting to hand it over
socket.send()/sendall()/sendto()/sendmsg() and FileIO.write() kept the
source buffer's read borrow for the whole call, including the wait for a
peer that may never make room. That borrow is a lock every other thread
writing to the same object waits on, so
threading.Thread(target=lambda: sock.sendall(buf)).start()
buf[0] = 1
did not return until the peer read; and a thread parked there is ATTACHED
and never reaches a safepoint, so gc.collect() in a third thread waited
for the peer too -- the same wedge readinto() had on the receiving side.
ArgBytesLike::borrow_buf_unlocked() answers with bytes that survive the
borrow being dropped. An immutable object hands out a plain reference and
locks nothing, so those are sent where they lie and bytes and memoryviews
over them cost nothing; only bytes reached through a lock are copied out
first. The export is held throughout either way, so the source still
cannot be resized while it is being sent.
The regression snippet covers both directions now and is renamed for it.
Assisted-by: Claude
Assisted-by: Codex:GPT-5
* Check select()'s descriptor limit while the sequence is walked
seq2set() collected the whole sequence and compared the result's length
against FD_SETSIZE afterwards. Selectable::try_from_object() calls
fileno(), which runs Python and can append to the list being walked, and
the walk re-reads the list on every step, so the collection had no end to
reach and the comparison was never made. seq2set in Modules/selectmodule.c
checks the count per element instead.
stdlib_select.py gains a fileno() that appends to its own list, and
releases its sockets with close() rather than by dropping the name.
Assisted-by: Claude
* Bound native recursion where the C stack cannot be measured
check_c_stack_overflow() answers no unconditionally under miri and on
musl, where the stack pointer is not read. Since 9c9905aff that check is
all with_recursion() does, so every guard placed on native recursion --
__call__ and __get__ dispatch among them -- was a no-op on those targets
and the nesting ran until the stack ran out.
with_recursion() now counts its own depth on those targets and refuses
past NATIVE_RECURSION_LIMIT_UNMEASURED. The count is separate from the
frame limit sys.setrecursionlimit() sets, and compiles away where the
stack pointer can be read.
Assisted-by: Claude
* Report the failure to allocate pbkdf2's key and Take.readinto's scratch
Both buffers are sized from an argument -- pbkdf2_hmac's dklen accepts up
to i32::MAX, and readinto's from the length of the destination -- and were
built with vec![0u8; n], which aborts the process on allocation failure.
new_zeroed_bytes() raises MemoryError instead.
Assisted-by: Claude
* Read the frozen-code tuple length as a signed length
The '(' branches in read_marshal_str_vec() and read_marshal_const_tuple()
took the length with read_u32() as usize, so a value with the top bit set
read as four billion items rather than as out of range. read_len() is what
every other length in this file goes through, and it reinterprets as i32.
Both readers serve deserialize_code(), which reads only the frozen modules
baked in at build time, so this changes no reachable behavior;
marshal.loads() already went through read_len().
Assisted-by: Claude
* Make the snippets from the fuzzer sweep assert what they check
builtin_str.py expanded a tab to 2**31-1 columns, allocating 2 GiB to
observe that the width is accepted; a string with no tab observes the same
acceptance without laying anything out.
stdlib_array.py caught the refusal of frombytes() on its own exported
buffer and passed silently when nothing was raised. stdlib_hashlib.py
used a bare `assert False` as its failure branch. Both now say so through
the same shapes the other snippets use.
stdlib_socket.py also accepts OverflowError from recv() with a size that
does not fit the platform's C int.
stdlib_threading_current_frames.py indexed the frame chain for "f123"
without first asserting it is there.
Assisted-by: Claude
* Ask for a marshal container's room instead of assuming it
A flagged tuple or list is published in the reference table before its
children are read, and the placeholder was built with vec![none; len].
The length is the input's to choose and read_len() lets it reach
i32::MAX, so marshal.loads(b"\xa8\xff\xff\xff\x7f") -- five bytes -- asks
for 17 GB of element slots and aborts the process where the allocator
cannot serve it. r_object() allocates the container up front too, but
PyTuple_New() reports what it cannot get.
The elements are now reserved with try_reserve_exact() and a refusal is
raised as MemoryError through the decoder's pending-error channel.
PyTuple::new_marshal_placeholder() held nothing but that allocation and
is gone; the caller builds the elements and uses new_ref().
Assisted-by: Claude
* Compare the blocking-buffer snippet against something
measure() asserted len(buf) == len(buf), which holds whatever the length
is; it now takes the length the caller expects. The pipe case compared
the drained total against the whole source, while an unbuffered write()
reports only what it transferred and a signal can cut that short; it now
compares against what write() returned.
Assisted-by: Claude
* Refuse array.frombytes() a source whose items are not bytes
frombytes() reads its argument as bytes, but accepted any contiguous
buffer, so array("i").frombytes(memoryview(array("d", [1.0]))) appended a
double's bytes read as ints instead of raising. array_array_frombytes_impl
requires an itemsize of 1; ArgBytesLike now reports the itemsize so the
same check can be made here.
The BufferError that a resize meets while the array is exported also names
the array rather than repeating bytearray's wording.
stdlib_array.py's frombytes-of-itself case used typecode "i", where the
new check answers before the resize guard is reached; it uses "b" so the
guard is what refuses, and asserts the wider case separately.
Assisted-by: Claude
* Tell apart the ways marshal data can be bad
A type byte no reader knows, a back reference that names nothing, and
TYPE_NULL all came out as a bare "bad marshal data" ValueError. r_object()
answers the first two with "bad marshal data (unknown type code)" and
"bad marshal data (invalid reference)", and read_object() answers the
third with a TypeError, "NULL object in marshal data for object", since
TYPE_NULL stands for no object rather than for a value.
MarshalError gains the three cases and deserialize_value() maps them.
The container-specific wording r_object() uses for a NULL read inside a
tuple or list is not reproduced; the exception type is.
Assisted-by: Claude
* Hash the collector's tables by address rather than by SipHash
A collection keys three sets and two maps by object address, and it visits
every tracked object and every edge between them, so the hashing is a
per-edge cost. Those tables used the default RandomState, whose SipHash
buys resistance against a caller choosing colliding keys -- and nothing
chooses these keys: they are addresses this process handed out into tables
that live and die inside one collection. A profile of gc.collect() over a
423k-object heap spent 45% of its samples in SipHash.
They now hash with a splitmix64 finalizer. The shifts matter: a table
picks its bucket from the low bits and an address arrives with those bits
zeroed by alignment, so a plain multiply leaves every object in a handful
of buckets and is slower than SipHash was.
The reachability walk also copied each object's referent vector out of the
map it was cached in, a second pass over every edge; it reads them in
place, and reference subtraction hands its vector to the map instead of
cloning it.
Measured over 423k live objects: 0.93s to 0.15s. Over 843k dead ones:
3.00s to 0.79s. extra_tests/snippets/stdlib_threading_gc_import.py, whose
collector thread calls gc.collect() in a loop, ran anywhere from 2.7s to
28s and now runs in 3.1-3.5s: a collection that takes longer leaves more
garbage for the next one to walk, so the cost fed back on itself.
Assisted-by: Claude
Assisted-by: Codex:GPT-5
* Keep the collection's candidates and their counts in one table
A collection built a set of candidates and, beside it, a map from the same
addresses to their reference counts. Both were probed for every edge in
the heap -- membership from the set, the count from the map -- so each
edge paid to hash the same address twice, and each candidate paid to be
inserted twice. The map alone answers both questions.
The candidates also keep a walkable order now, which the reference
subtraction pass needs since it writes the counts while reading the
candidates, and which the unreachable set is built from instead of a
set difference.
Over the 423k-object heap measured in the previous commit: 0.15s to
0.13s live, and 0.79s to 0.49s dead.
Assisted-by: Claude
Assisted-by: Codex:GPT-5
* gc: collect referents into one buffer instead of a vector per object
Step 3 allocated a `Vec` for every tracked object to hold its referents
and kept them all in a map until step 4 read them back. The referents now
go into a single growing buffer, with the map holding each object's range
into it.
Adds `PyObject::gc_extend_referent_ptrs`, which appends to a caller's
buffer; `gc_get_referent_ptrs` calls it with a fresh one.
Assisted-by: Claude
* memoryview and struct: match the checks and errors of the reference
memoryview:
- `cast()` accepted a source and destination that are both item types,
which reinterprets the items rather than re-dividing the bytes; one side
now has to be a byte format.
- A cast to `shape=()` returned without checking that the buffer holds
exactly the one item that shape describes.
- `hash()` hashes the bytes, so it now raises ValueError for a view whose
items are not bytes, rather than returning a hash that disagrees with the
value the view compares equal to.
- `tobytes()` takes the `order` argument, with 'F' walking a
multidimensional view down its columns; `BufferDescriptor` gained
`for_each_segment_fortran` for that walk.
struct:
- A value the format has no room for reported "argument out of range"
instead of naming the format and its range. The format character is now
passed to the packing functions to report it.
- `Struct.__new__` no longer reads the format; `__init__` does, so
`__init__` can be called again and a subclass can pass the format up.
Methods raise RuntimeError until it has run, and `Struct` is a base type.
Removes the expectedFailure from test_Struct_reinitialization and
test_struct_subclass_instantiation.
Assisted-by: Claude
Assisted-by: Codex:GPT-5
* io: decide the readinto path by file type, not by seekability
FileIO.readinto wrote straight into the caller's buffer, holding its write
borrow, when the fd was seekable; otherwise it read aside into scratch and
copied. Seekability stood in for "this read answers without waiting on a
peer", which a pipe on Windows breaks: lseek on one succeeds, so the pipe
took the borrow-holding path and every other thread touching that bytearray
waited for the peer.
host_io::reads_without_waiting answers it directly -- seekability elsewhere,
GetFileType() == FILE_TYPE_DISK on Windows.
The regression snippet times each operation separately, so a failure names
the one that waited; it asserts the transfer is still in flight before
checking the export; and the socket case fills the connection until it
refuses rather than assuming a size that outruns it, which SO_SNDBUF on an
already-connected pair does not settle.
Assisted-by: Claude
|
||
|
Jeong, YunWon
|
c603eb7c9a |
Implement PEP 688 and rework the buffer protocol around managed exports (#8523)
* Implement PEP 688 __buffer__ and __release_buffer__ A Python class could not export a buffer: the slot machinery had no bf_getbuffer or bf_releasebuffer, and every consumer acquired buffers as PyBUF_FULL_RO through a module of PyBUF_* constants. Add both slots. PyBuffer::release now runs a Python __release_buffer__ before the exporter's own release, once per acquisition, which PyBuffer tracks with an `acquired` flag that clones do not inherit. An export made by a Python __buffer__ is held by a _buffer_wrapper payload that counts its exports and drops the returned memoryview with the last one, and the view handed to __release_buffer__ is a _buffer_window that owns no export, so releasing it inside the hook is inert instead of re-entering it. Replace the PyBUF_* constants with a BufferFlags bitflags type whose composite requests are supersets of the simpler ones, so `contains` answers the REQ_* questions, and pass the request to PyBuffer::from_object. Each consumer now asks for what its counterpart asks for: y* arguments for SIMPLE, w* for WRITABLE, BytesIO.write for CONTIG_RO, bytes(), bytearray() and memoryview() for FULL_RO. memoryview checks the request in memory_getbuf, and array.array and mmap.mmap expose __release_buffer__. Test buffer support with PyObject::check_buffer (PyObject_CheckBuffer) instead of attempting an acquisition, so an exception raised by __buffer__ is no longer reported as the object not being bytes-like, and a __buffer__ with side effects runs once. PyBytesInner becomes a y* conversion as a result: bytes and bytearray methods no longer accept iterables of ints, and find, index, count and __contains__ take the arguments parse_args_finds_byte and bytes_contains describe. A view exports its start offset in the descriptor rather than in its window, which fixes a panic when collecting from a negative-stride view. BytesIO.write rechecks closed after acquiring its buffer, which __buffer__ can close in between. Assisted-by: Claude Code:claude-opus-5 * Rework the buffer protocol around managed exports and view offsets Give `PyBuffer` the `_PyManagedBufferObject` shape: one `bf_getbuffer` acquisition is shared by every handle taken from it, cloning takes another share instead of re-acquiring, and the exporter's release runs once when the last share goes away. Remove `retain`, the unsafe `drop_without_release`, the three `impl Drop`s and the `ManuallyDrop` that stood in for this. Add `abort_acquisition` so a failed request does not run `bf_releasebuffer`. Move the view start into `BufferDescriptor::offset`, the `Py_buffer.buf` analogue, and drop the separate `start` fields on `PyMemoryView` and `PyBufferWrapper`. Slicing goes through `SaturatedSlice::adjust_indices_start`, which reproduces `PySlice_AdjustIndices` and keeps the adjusted start. Fix `zip_eq` to take its contiguous fast path only when both last dimensions are contiguous, and make `for_each_segment` and `zip_eq` handle zero-length and zero-dimensional views. Add `BufferDescriptor::projected` so a request without `PyBUF_ND`, `PyBUF_STRIDES` or `PyBUF_FORMAT` receives a correspondingly reduced descriptor, and reject a request without `PyBUF_INDIRECT` against an exporter that has suboffsets. Copy the source first in `memoryview` slice assignment when both sides reach the same root exporter. Hold the export across the resize in `bytearray.extend`, take `y*` in `marshal.loads`, stop probing the buffer protocol in `FsPath`, rewrite `ord` over the concrete string types, fold `array`'s buffer slot into one `slot_as_buffer`, take `w*`/`y*` in `_overlapped`, and thread the new `offset` field through the `_ctypes` descriptors. Assisted-by: Claude * Re-check a memoryview after a conversion and tighten cast and release `pack_single` and `unpack_single` addressed the buffer with a position taken before `__index__` ran, so releasing the view from that conversion read or wrote outside the exporter's storage, panicking when it had also shrunk. Check the released flag again once the conversion is done, as `CHECK_RELEASED_AGAIN` does. Reject a cast to a format that is not a single native format character with an optional `@` in front of it. An empty format reached a division by its item size of zero. get_native_fmtchar Report a second `__release_buffer__` on the same view as a `ValueError` rather than accepting it, and check that the view belongs to the object first; the silent case is a view that exports nothing. wrap_releasebuffer Compare against another memoryview by reading its view where it lies instead of acquiring a buffer from it, so the restricted view handed to `__release_buffer__` compares equal rather than unequal in one direction only. memory_richcompare Name the type in the unraisable an exception from `__release_buffer__` reports, as `releasebuffer_call_python` does, instead of reporting the exporter object. Assisted-by: Claude * Re-check the destination of a memoryview slice assignment Acquiring the source runs `__buffer__`, which can release the destination view, so check the released flag again once the source is in hand and before the structures are compared, as `copy_single` does. Build the sliced destination as a view that counts as no export, the way a `Py_buffer dest = *view` copy does. Holding one kept the exporter unresizable for the length of the assignment, so a source that released the view and then resized the exporter met a `BufferError` instead of the assignment reporting the released view. Assisted-by: Claude * Answer the memoryview contiguity getsets with the layout `f_contiguous` reported the row-major answer for one dimension and False for any more, so a view laid out both ways, such as one of shape (1, 8), reported False while `check_buffer_request` accepted a `F_CONTIGUOUS` request on it. `contiguous` answered row-major order alone. Move the Fortran-order test from `PyMemoryView` to `BufferDescriptor`, next to the row-major one it differs from only in iteration order, and answer all three getsets with it. Assisted-by: Claude |
||
|
Jeong, YunWon
|
11de045ef8 |
Per-interpreter runtime state and a process-wide GC stop-the-world (#8517)
* vm: add per-interpreter runtime state and interpreter registry - vm/runtime.rs: process-global interpreter registry (monotonic ids, weak entries), InterpreterWhence/InterpreterInfo, process main id recording via main_interpreter_id(), a threading-gated owner map (store_owned_interpreter/ take_owned_interpreter/is_owned_interpreter/owned_interpreter_count), and the SUPPORTS_ISOLATED_INTERPRETERS constant. - PyGlobalState gains interpreter_id/whence/is_main and is_main_interpreter(); PyConfig/Settings derive Clone so a subinterpreter can clone parent config. - Interpreter: id()/whence()/is_main()/is_process_main(), create_subinterpreter() and create_owned_subinterpreter(); unregister on Drop. - thread.rs: per-interpreter thread slots (INTERP_THREAD_SLOTS), slot swap when switching interpreters on one OS thread, cleanup keyed by interpreter id. - Install signal handlers and init the main-thread ident only on the main interpreter; _thread._is_main_interpreter reflects the current interpreter. - sys.implementation.supports_isolated_interpreters reads the constant. - Guard the registry static for non-threading builds where rc::Weak is !Send. Assisted-by: Claude Code:claude-opus-4-8 * gc: stop every interpreter while collecting The generation lists are process-global, so a collection reads and frees objects owned by every interpreter. CollectStopTheWorld stopped only the collecting interpreter, leaving other interpreters' threads free to mutate the same object graph during the reference-subtraction, reachability and snapshot phases. Stop all live interpreters instead, in runtime id order, and restart them in reverse. The global `collecting` mutex serializes collectors process-wide, so no second collector takes these exclusions in another order; fork acquires a single interpreter's exclusion, so the orders cannot cycle. - runtime: add live_interpreter_states(), ordered by interpreter id. - StopTheWorldState methods take &PyGlobalState instead of &VirtualMachine, so an interpreter's world can be stopped without a VM for it; update the call sites in frame, _thread, posix, faulthandler and capi. - Document in gc_state() that the collector is process-wide: gc.disable(), thresholds, gc.garbage and gc.get_objects() observe process-wide state, and a per-interpreter collector additionally needs untrack_object (called from default_dealloc with no VM in scope) routed to the owning interpreter. Tests: stop_the_world_parks_threads_of_another_interpreter asserts a thread entered in one interpreter parks another interpreter's threads (it fails without this change), plus a collect-while-another-interpreter-churns test. Assisted-by: Claude Code:claude-opus-4-8 * vm: scope the interpreter registry per thread without threading The registry held `rc::Weak` in a process-global `OnceLock` and covered the resulting `!Send`/`!Sync` with an `unsafe impl` justifying it as "non-threading builds are single-threaded". That is not this codebase's model: `static_cell!` is thread-local without the `threading` feature precisely so each OS thread can own its own `Context::genesis()` and `GcState`, so two threads could reach the same `Rc` counts through the registry. Use `static_cell!` for the registry as well, matching `gc_state()`, and drop the `unsafe impl`. Ids are consequently unique per registry rather than per process in non-threading builds, which is documented on `alloc_interpreter_id`. Move the recorded main interpreter id into the registry so it follows the same scoping instead of living in a separate global `OnceLock`. Also make `CollectStopTheWorld::new` accumulate into a live guard: it built a bare `Vec` and only moved it into the restarting `Drop` type after stopping every interpreter, so an unwind partway through the loop left the already stopped interpreters parked forever with their exclusion held. Assisted-by: Claude Code:claude-opus-4-8 * vm: keep the interpreter registry usable across bootstrap, drop and fork Three gaps where the registry did not describe the interpreters that actually exist, each of which hides an interpreter from the collector's stop-the-world. Register before `initialize()`. Registration ran as the last step of `initialize_vm`, so the whole bootstrap — which executes Python bytecode and allocates GC-tracked objects — was invisible to `live_interpreter_states()`. It still cannot run any earlier than this: the init hooks take `PyRc::get_mut` on the state, which fails as soon as the registry holds a weak reference to it. Stop unregistering in `Interpreter::drop`. The handle does not decide the interpreter's lifetime — every `ThreadedVirtualMachine` from `new_thread()` holds its own `PyRc<PyGlobalState>` — so an interpreter with running workers disappeared from the registry while its threads kept mutating the object graph. The entries are weak, so lifetime is already correct without the removal; dead entries are now reaped when registering instead. Interpreters are consequently released rather than unregistered at a fixed point, so the two tests asserting disappearance now wait for it: a collection in progress legitimately holds a reference to every live interpreter. Repair other interpreters after fork. `py_os_after_fork_child` only fixed the forking interpreter, leaving every other one with slots for threads that did not survive (still ATTACHED if they were running bytecode) plus locks and stop-the-world flags held by them. Since a collection stops all interpreters, the child's first collection would wait for threads that no longer exist. Reset their locks, stop-the-world state and thread tables, drop this thread's cached slots for them, and reinit the registry's own locks first, since enumerating interpreters now takes them. Tests: test_gc, test_threading and test_fork1 pass, as do the vm tests in both the threading and default configurations. Assisted-by: Claude Code:claude-opus-4-8 * vm: attach and detach thread slots when switching interpreters `enter_vm` decided whether to attach from `was_outermost` (an empty VM_STACK), which held while a thread could only ever be in one interpreter. With a slot per (thread, interpreter) pair, entering interpreter B from a thread already inside interpreter A's section switched CURRENT_THREAD_SLOT to B's slot but attached nothing: the thread then ran B's bytecode with B's slot DETACHED while A's slot stayed ATTACHED. A collector stopping B force-parks the DETACHED slot and concludes B is stopped, and then walks the object graph this thread is still mutating. Pair the attach/detach with the slot switch instead (≈ `_PyThreadState_Swap`): `begin_interpreter_section` detaches the enclosing interpreter's slot, makes the target slot current and attaches it, and `end_interpreter_section` undoes that and re-attaches the enclosing interpreter. Both live in `set_current_vm`, which every path making a VM current already goes through, so `enter_vm` and `VmBootstrapGuard` no longer track outermost-ness themselves. `nested_enter_of_subinterpreter_is_stoppable` covers this: it runs a subinterpreter nested inside the parent's section and asserts the sub's threads park when the sub's world is stopped. It fails with the previous attach-at-outermost-only behavior. Assisted-by: Claude Code:claude-opus-4-8 * vm: list only the current interpreter's subclasses Interpreters share the context, so `class Foo(int)` in one of them pushes onto the same `int.subclasses` every other one reads, and `int.__subclasses__()` returned types no other interpreter can reach. Record the creating interpreter on `HeapTypeExt` and filter `__subclasses__` by it, the way `lookup_tp_subclasses` reads `tp_subclasses` out of per-interpreter state for static builtin types. Types built before any interpreter exists — the ones the shared context creates, including the exception hierarchy — carry no id and stay visible to every interpreter, which is what `_PyStaticType_InitBuiltin` produces by registering the builtin subclass links once per interpreter. The other walks over `subclasses` (version-tag invalidation, abc flag propagation, mro updates, slot propagation) are left as they are: each starts from a type being mutated, so from a heap type, whose subclasses all live in the interpreter that created it. `subinterpreter_subclasses_are_scoped_to_their_interpreter` covers this and fails without the filter. Assisted-by: Claude Code:claude-opus-5 * gc: give each interpreter its own collector state The generation lists stay process-wide, because an object is untracked from `default_dealloc`, where no interpreter is in scope to route to. What changes is that a collection no longer acts on every interpreter's objects, and the gc module no longer reports one interpreter's state to another. `track_object` stamps the running interpreter into a new `gc_owner` word on the object header, and a collection takes as candidates only the objects carrying its own tag plus the ones carrying none. `gc_owner` fits in the padding the header's alignment already forces, so objects do not grow; an assertion on the header size keeps it that way. Objects allocated with no interpreter running — everything the shared context builds — carry no owner and stay candidates for every interpreter, which is where they were before. Objects that outlive the interpreter that tracked them are adopted the same way by the next full collection, rather than being left to a collector that will never come. `enabled`, the thresholds, the debug flags, the statistics, `gc.garbage` and `gc.callbacks` move onto `PyGlobalState` — the last two off the shared `Context` — so `gc.disable()`, `gc.set_threshold()`, `gc.get_stats()`, `gc.get_objects()` and `gc.garbage` describe the interpreter that asks. The occupancy counts behind `gc.get_count()` and `gc.get_freeze_count()` stay process-wide: they measure how full the shared lists are. Their decrements are saturating now, since a collection zeroes the generations it emptied while another interpreter's objects are still sitting in them. Stop-the-world still stops every interpreter. Unowned objects are candidates and any interpreter can incref one, so the refcounts a collection reads are only stable while all of them are parked. This also fixes a deadlock it exposed: `CollectStopTheWorld` dropped its references to the stopped interpreters while the collection still held the generation read locks, so releasing the last reference to one — which frees its objects, and so untracks them — waited for a write lock behind that read lock. The references are now held until the guard itself drops. `collections_only_reach_the_collecting_interpreter` and `get_objects_only_reports_the_calling_interpreter` cover this and both fail without the owner check. Assisted-by: Claude Code:claude-opus-5 * _queue, _thread: detach before taking locks held across waits `_queue.Semaphore` holds its mutex across the `allow_threads` condvar wait, and `join_internal` holds a thread handle's completion mutex the same way. Stop-the-world can stop a thread while it holds either one. The remaining acquisitions ran attached, so a thread blocking on such a mutex had no safepoint left to reach: the stop never completed, and the holder was never resumed to release it. Route those acquisitions through helpers that detach first. The fork-child reinit paths keep their direct locks. Assisted-by: Claude * _io, _winapi: detach on the remaining stopped-holdable lock takes `TextIOWrapper.__repr__` took `data` directly while every other method takes it through `lock_opt`, which detaches. `Overlapped` holds `inner` across the `allow_threads` in `GetOverlappedResult`, and all four of its takes were direct. A thread stopped by stop-the-world can be holding either mutex, so taking one while attached left the blocked thread with no safepoint to reach. Assisted-by: Claude * gc: size the interpreter owner tag to the header padding `gc_owner` was a u32. With 4-byte pointers its alignment pushed it out of the padding that follows the gc bits and generation, growing every object by a word and tripping the `SIZEOF_PYOBJECT_HEAD` assertion on 32-bit targets. Introduce `GcOwner = u16`, which the `repr(C)` layout places at offset 10 on 32-bit and 18 on 64-bit, leaving the header at 6 words on both. Tags now run out after 65535 interpreters; `alloc_owner` already falls back to `GC_NO_OWNER`, so an interpreter past that collects as it did before tagging. Also widen three test deadlines that measure liveness, not speed. Assisted-by: Claude * vm: reuse cleared frame blocks and shorten interpreter hot paths - `datastack` remembers the most recently popped frame block. `push_frame` reports an exact LIFO reuse, and `setup_datastack_frame` then skips zero-filling localsplus. - Small-int loads push the context's cached int as a borrowed stack reference instead of taking a new one. - Calls to jitted functions go straight to `execute_call_vectorcall`. - Binary-op specialization reads ints through the new `PyInt::try_to_i64_fast` instead of the generic primitive conversion, and `try_to_bool` moves its non-bool path into a `#[cold]` helper. - Dict caches read an entry through a keys-version stamp (`get_index_if_keys_version`) rather than an entry-index hint. - Frame publishing caches a pointer to `ThreadSlot::top_iframe` in a thread-local `Cell` instead of borrowing `CURRENT_THREAD_SLOT`. * gc, vm: address review notes on owner tags and interpreter docs Look up retired owner tags with a sorted binary search instead of a linear scan, which every scanned object paid for once per dropped interpreter. Drop the claim that clearing a tag frees it for reuse; `alloc_owner` only ever hands out new tags. Also correct the tag space it mentions, which is 16-bit since the tag was sized to the header padding. Document `is_main` as "top-level interpreter" rather than "the process main": every top-level interpreter sets it, and only the first registered one becomes the main `main_interpreter_id` reports. Assert membership rather than an exact owned-interpreter count delta; the owned table is process-global and other tests store into it in parallel. Assisted-by: Claude * vm: gate interpreter registration on stop-the-world admission A collection snapshots the registry, stops the interpreters it found, and then reads tracked objects with their threads parked. An interpreter that registered after the snapshot was taken was absent from it, so nothing stopped it and its bootstrap ran Python — allocating and mutating the shared generation lists — underneath that scan. Registration and the stop now share a process-global gate: the collection holds it from before the snapshot until the restart, and registration takes it around the registry insert. The insert runs detached, since a thread that waited for the gate, or re-attached while holding it, would leave the stop it waits for no safepoint to complete at. Assisted-by: Claude * gc: keep the tracking counters off the barrier path Every tracked allocation and every free went through sequentially consistent counter updates, and each free through a `fetch_update` CAS loop. The counters drive only the gen0 threshold and `gc.get_count()`, and the generation locks — not the counters — order the list changes they describe, so they are relaxed now and the decrement is a load plus a conditional `fetch_sub`. `is_enabled` and `threshold`, both read once per allocation, are relaxed for the same reason: an allocation racing `gc.disable()` may use either value. Drop `alloc_count`, which nothing has ever read. Assisted-by: Claude * dict: settle lookups and iteration steps under one read guard A lookup probed under a read guard, dropped it with the matched entry in hand, and then took the lock again to re-find the entry. `lookup_extract` reads the entry while the probe still holds the guard when key identity settles the match, which is the case that cannot run Python. Dict and set iterators took one lock to compare the size and another to read the entry, then cloned both the key and the value even though a keys or values view keeps only one of them. `next_entry_checked` does the size check and the read under one guard, and clones through a per-view projection. A store that missed its inline-cache hint re-probed the dict afterwards only to recover the entry index the store had just computed; `unchecked_push` reports that index instead. Assisted-by: Claude * vm: build a call's argument vector once The specialized CALL handlers collected the positional arguments into one vector and then copied them into a second one to put `self` in front, so every specialized builtin, method-descriptor, class and non-Python call paid two allocations, a copy and two frees. The arguments are already laid out on the value stack in vectorcall order, so `take_call_args` fills one vector by index — the shape `execute_call_vectorcall` already used. `vectorcall_native_function` and the keyword path of `vectorcall_function` then cloned that vector again to build `FuncArgs`; both now move it in through `from_vectorcall_owned`, as the other vectorcall slots do. Assisted-by: Claude * vm: reach an instance dict without cloning it The specialized attribute instructions cloned the instance dict — a rwlock round-trip plus a refcount round-trip — for two things that only look at it: `LoadAttrMethodLazyDict` asking whether the dict exists, and the keys-version stamp check that is the whole of `shadowing_instance_attr`'s fast path. Both now read it borrowed, through `has_instance_dict` / `with_instance_dict`. `generic_getattr_opt` probed the dict with the name's `&Wtf8`, which hashes the name on every lookup and can never match a key by pointer. Passing the `Py<PyStr>` uses the string's cached hash and the interned-key identity check, and drops an allocation when a stored key is not an exact `str`. Assisted-by: Claude * vm: shorten the per-instruction safepoint and the call preamble The dispatch loop asks once per instruction whether stop-the-world wants this thread, and that read went through `CURRENT_THREAD_SLOT` — a `RefCell` borrow, so two stores to thread-local memory around an `Option` test. Cache the `stop_requested` pointer in a plain `Cell` at the same three places the frame pointers are cached, and the safepoint becomes one relaxed load. `lasti` is advanced from the index the loop just read, rather than reloaded to increment it, and `Resume` reads `quickened` before swapping it, so a call to an already-quickened code object costs a load instead of an atomic read-modify-write. An exact-args vectorcall to a Python function built a heap `FrameObject` where the equivalent `invoke` path uses a data stack frame; it now does the same when tracing is off. Assisted-by: Claude * vm: give KwArgs a zero-sized hasher `KwArgs::default()` is built for every call, keyword-less ones included, and it seeded a `RandomState` each time — a thread-local read and 16 bytes in every `FuncArgs`. Keyword names come from the program text, so the map now uses `BuildHasherDefault`, whose `Default` is a zero-init. Assisted-by: Claude * vm: promote borrowed stack refs before a yield `yield <small int>` tripped the "borrowed refs on stack at yield point" assertion: `LoadSmallInt` pushes a borrowed ref, and a yield saves the stack with the frame, which is exactly what that assertion forbids. Promote the stack first, so a suspended frame owns everything it holds. Assisted-by: Claude * vm: run the exact-args vectorcall on a heap frame again `vectorcall_function` took the data stack path for an exact-args call whenever tracing was off. A data stack frame that materializes — one traceback entry is enough — copies its localsplus into the `FrameObject`, so every value it holds gains a reference for as long as the frame runs, and a tail call holds the callee's function object until the caller's next call or return. Both are reachable without this hunk, through the `CallPyExactArgs` specialization, but routing the vectorcall entry point through the data stack made them apply from a call site's first execution: test_enumerate, test_memoryio, test_sys, test_tempfile and test_traceback all read a refcount or a finalization that moved. Assisted-by: Claude * vm: release a tail call's function at the callee's return The trampoline moved the callee's function object into the caller's `SuspendedFrame` and dropped it only once the caller had run again, so a function entered through a tail call kept one reference past its return: `sys.getrefcount` on it read one high for the caller's next stretch of bytecode, and anything the function reached stayed alive that much longer. The callee's frame is released before the `ReturnValue` or `Unwind` action is formed, and a materialized frame object holds its own references, so the owner is dropped where the suspended caller is popped instead. Assisted-by: Claude * vm: stop copying a running frame's locals when it materializes `materialize_slow` cloned the source frame's fast locals into the new `FrameObject`, so every value in a data stack frame gained a reference lasting as long as that frame object: one traceback entry through a frame kept all of its locals alive, and `sys.getrefcount` read one high for each of them. Nothing read that copy while the frame ran. `find_live_source_iframe` already routes reads to the live frame, and `exit_iframe` fills the slots from it at return, so the copy now starts empty as `materialize_slow_chain`'s does, and `snapshot_to_heap` goes with its last caller. Two readers had to be brought to the live frame the same way. `has_active_hidden_locals` read the frame object's own slots, which let `locals()` in a class body write an inlined comprehension's hidden variable into the class namespace; it now shares `live_fastlocals` with the other two. Another thread's frame has no live frame to reach from here, so the six cross-thread sites fill the copy under stop-the-world through `materialize_with_locals`. Assisted-by: Claude * vm: key data stack frame reuse on the exact frame size push_frame and pop_frame compared 16-byte-aligned sizes, so two frames whose sizes differ by less than the alignment matched as a reuse. The caller reads that as "every slot was cleared by the previous frame", but the larger frame's tail slots were never touched by it. Assisted-by: Claude * vm: store the frame object payload address in the cross-thread slot set_current_frame and reinit_frame_slot_after_fork wrote the address of the Py<FrameObject>, while the cross-thread readers hand that pointer to Py::from_payload_ptr, which subtracts the object header from it. Both writers now store the payload address. Assisted-by: Claude * vm: copy a foreign thread's frames instead of linking them sys._current_frames(), _thread._current_exceptions() and the cross-thread f_back path handed back frame objects still linked to the running frames. The owning thread wrote its locals into those same buffers at exit_iframe while the reader was cloning out of them. materialize_detached and materialize_detached_chain copy the values under stop-the-world and leave nothing for the owner to write into; materialize_with_locals is gone. FrameColdData gains attached_tid, the thread still running the frame a frame object was materialized from, which exit_iframe clears once it has written the values in. check_locals_access now rejects a read from any other thread while it is set; the FrameOwner test could not, because materialize_slow forces FrameObject ownership. Also removes with_iframe, which had no callers. Assisted-by: Claude * vm: resolve the two rustdoc links this branch added MAIN_INTERPRETER_ID is not in scope in interpreter.rs, and new_thread only exists in a threading build, so neither link resolved. Assisted-by: Claude |
||
|
Jiseok CHOI
|
dbd3a0a6b3 |
sqlite3: add SQLITE_DBCONFIG constants, setconfig() and getconfig() (#8535)
* sqlite3: add SQLITE_DBCONFIG constants, setconfig() and getconfig() Implement Connection.setconfig() and Connection.getconfig() using sqlite3_db_config(), and export all SQLITE_DBCONFIG_* integer constants to the module. Fixes test_connection_config in test_dbapi.py. Assisted-by: GitHub Copilot:claude-sonnet-4-6 * sqlite3: fix setconfig to propagate actual SQLite error When sqlite3_db_config() returns a non-OK result, propagate the actual SQLite error instead of replacing it with a generic 'Unable to set config' message, to match CPython's set_error_from_db() behavior. Assisted-by: GitHub Copilot:claude-sonnet-4-6 |
||
|
Jiseok CHOI
|
e19b622174 |
sqlite3: pass None for NULL authorizer args instead of crashing (#8534)
* sqlite3: pass None for NULL authorizer args instead of crashing CPython's authorizer callback receives NULL for arg1/arg2/db_name/access when not applicable (e.g. SQLITE_READ on a table gives NULL for the database name in some versions). Previously RustPython passed these pointers to ptr_to_str which would crash or produce an error. Now ptr_to_str_or_none is used: NULL pointers become Python None, which matches CPython behavior and allows test_table_access and test_column_access to pass. Assisted-by: GitHub Copilot:claude-sonnet-4-6 * fix(sqlite3): avoid reentrant deadlock in Statement::new sqlite3_prepare_v2 can synchronously invoke the authorizer callback, which may call back into Connection methods (e.g. set_authorizer) that require the same db lock. Holding db_lock across the prepare() call caused a self-deadlock when a callback re-entered the connection. Release the lock after sql_limit check and copy the raw handle before calling prepare(), so FFI calls that can trigger Python re-entrancy happen outside the lock scope. Fixes hang in test_authorizer_concurrent_mutation_in_call |
||
|
Jiseok CHOI
|
a142de9abc |
sqlite3: add Connection exception attributes and fix autocommit ValueError (#8533)
Add DB-API 2.0 optional extension: expose exception classes as attributes on Connection objects (Warning, Error, InterfaceError, DatabaseError, DataError, OperationalError, IntegrityError, InternalError, ProgrammingError, NotSupportedError). Also fix autocommit validation to raise ValueError (not TypeError) when an unsupported type is passed. Assisted-by: GitHub Copilot:claude-sonnet-4-6 |
||
|
Jiseok CHOI
|
7cda53b946 |
sqlite3: validate narg before creating functions/aggregates (#8532)
CPython 3.14 added check_num_params() which raises ProgrammingError when narg/n_arg/num_params is out of range (-1..=SQLITE_LIMIT_FUNCTION_ARG). RustPython passed invalid values directly to SQLite, resulting in an OperationalError instead. Add check_num_params() helper and call it in create_function(), create_aggregate(), and create_window_function(). Assisted-by: GitHub Copilot:claude-sonnet-4-6 |
||
|
Jeong, YunWon
|
259ce8ce66 |
Fix the itertools.tee leak and race, the contextvars and generator races, _asyncio's InvalidStateError lookup, and _imp's frozen data (#8518)
* _imp: read get_frozen_object's data as a whole marshal value
The data argument went through deserialize_code(), which reads a code body
without the type byte in front of it, so nothing marshal.dumps() produces was
accepted. Read it with marshal.loads() and require a code object back.
Assisted-by: Claude
* _imp: bind find_frozen's arguments with FromArgs
The arity check was done by hand on a FuncArgs. Take the arguments through a
FromArgs struct instead, which makes withdata keyword-only, and fill in the
data it asks for: the frozen encoding is not marshal, so the code is
re-serialized into what get_frozen_object() reads back.
Assisted-by: Claude
* itertools: give tee's shared buffer a Python type
The buffer was a PyRc<PyItertoolsTeeData>, which is not a Python object, so the
collector could not walk into it and any cycle running through a tee was
uncollectable. Make it the _tee_dataobject type with a traverse, held by
PyRef, and split the rest along the same lines: tee() is a function, _tee is
the iterator type it builds, and _tee takes a single iterable rather than
returning a tuple from __new__. _tee is weak-referenceable, tee() rejects a
negative n with ValueError and reserves its tuple fallibly.
test_itertools.test_tee passes now; its expectedFailure marker is removed.
Assisted-by: Claude
* contextvars: hold the shared context state under locks
The variable map was a RefCell, the enter flag, the context index, the token's
used flag and the variable hash were Cells, and each carried an unsafe impl
Sync. A Context or ContextVar shared between threads overlapped their borrows
and panicked. The map and the per-variable cache are now PyMutex, the flags and
the index are atomics, entering a context is a compare_exchange, and the three
unsafe impl Sync are gone.
The cache also stopped being read through AtomicCell::as_ptr, which raced a
concurrent store on a value holding a PyObjectRef.
Values displaced from the map or the cache are dropped after the lock is
released: __del__ can come straight back into the same context, and the locks
are not reentrant.
Assisted-by: Claude
* generator: read the frame state under the running claim
send(), send_none(), throw() and close() read `closed` and `frame.lasti()`
before `running` was compare_exchanged, so the frame they went on to resume
could be one another thread had already advanced. A resume that decided from
`lasti() == 0` that the generator had not started pushes no value onto the
value stack, and the code after the yield pops one, which underflows the stack.
The compare_exchange now hands back a guard, taken before those reads and
released after maybe_close(), so the generator is retired while it is still
claimed.
Assisted-by: Claude
* itertools: claim a tee's position and its buffer as one step
`_tee::next` read `index` and moved it on afterwards, and
`_tee_dataobject::get_item` released `running` before the value it fetched
from the source was cached.
Two callers on one `_tee` then read the same index, hand out the same value
twice and advance past a value that was never cached, which leaves `index`
past `values.len()` and indexes the buffer out of bounds. Two callers at the
same index on separate tees each fetch a value from the source, and one of the
two is dropped without reaching a caller.
Both claims now cover the read and the update.
Assisted-by: Claude
* _asyncio: require InvalidStateError to be a type
new_invalid_state_error() called whatever `asyncio.exceptions.InvalidStateError`
names and unwrapped the downcast of the result, so a future asked for a result
it does not have panicked once that attribute was rebound to something that is
not an exception:
asyncio.InvalidStateError = lambda *args: 42
_asyncio.Future(loop=object()).result()
The type is looked up the way get_cancelled_error_type() looks its own up, and
raised with new_exception_msg; a lookup that does not produce an exception type
falls back to RuntimeError, as the other arms already did.
Assisted-by: Claude
|
||
|
Jeong, YunWon
|
d5d19e0611 |
Fix 33 reproduced fuzzing and static-review defects (#8514)
* mmap: guard move() against dest/src past the mapping size
An offset larger than the mapping made `size - dest` underflow, producing
an out-of-range slice index panic. Check the high side first, as write() does.
Assisted-by: Claude
* exceptions: keep ImportError name/path/name_from out of an empty instance dict
ImportError.__reduce__ unwrapped get_arg(0), which is None when the exception
was constructed with no positional argument, aborting on pickle.dumps(ImportError()).
Fall back to the full args tuple there, and expose name/path/name_from as class
attributes defaulting to None so a bare ImportError() reduces to (cls, ()).
Assisted-by: Claude
* asyncio: stop unwrapping the _current_tasks downcast in current_task()
_asyncio._current_tasks is a reassignable module attribute; the three sibling
functions already degrade gracefully when it is not a dict, current_task() did not.
Assisted-by: Claude
* asyncio: raise TypeError when FutureIter.throw()'s exception class returns a non-exception
exc_type.call() goes through a Python-controlled __new__, so its result can be
any object; the downcast was unwrapped. Report it as a TypeError instead.
Assisted-by: Claude
* sequence: use a fallible reservation for sequence repetition
The guard only rejects a repeat whose per-element size crosses MAX_MEMORY_SIZE,
so (1,) * (10**12) still reached Vec::with_capacity and aborted in the allocator.
Reserve fallibly and surface a MemoryError.
Assisted-by: Claude
* collections: reject an oversized deque repetition with MemoryError
deque * sys.maxsize reached the allocator and aborted with a capacity overflow.
Apply the MAX_MEMORY_SIZE guard the other sequences already carry.
Assisted-by: Claude
* itertools: raise OverflowError for an out-of-range r argument
combinations/combinations_with_replacement/permutations narrowed r with
to_usize().unwrap(), so r=2**64 panicked instead of raising.
Assisted-by: Claude
* math: stream the generic sumprod path instead of collecting both iterables
The big-int path collected each argument into a Vec before multiplying, so an
unbounded iterable exhausted memory. Advance both iterators in lockstep and
accumulate, reporting a length mismatch when only one is exhausted.
Assisted-by: Claude
* _imp: raise TypeError for a second positional argument to find_frozen
withdata is keyword-only and unimplemented; passing it positionally hit an
unimplemented!() and aborted.
Assisted-by: Claude
* _typing: check _idfunc arity before indexing args
_typing._idfunc() with no argument indexed args[0] out of bounds.
Assisted-by: Claude
* exceptions: require a sequence for the ExceptionGroup excs argument
The argument was collected before being validated, so an unbounded iterable
such as itertools.count() exhausted memory. Reject a non-sequence up front.
Assisted-by: Claude
* mmap: treat an inverted find/rfind range as empty
find(b"x", 5, 2) built a slice whose start exceeded its end and panicked.
Assisted-by: Claude
* collections: give deque and defaultdict a GC traverse
Neither type opted into traversal, so a reference cycle through a deque or a
defaultdict was never collected.
Assisted-by: Claude
* itertools: opt the iterator types into GC traverse
cycle and its siblings hold Python references but declared no traverse, so a
cycle built through one of them leaked.
Assisted-by: Claude
* _ctypes: mask an out-of-range int instead of panicking
c_char_p(2**64) and pointer item assignment called .expect() on the narrowing
conversion. Wrap the value to the target width, which is what the C
implementation stores.
Assisted-by: Claude
* hashlib: import _hashlib when a hash module is loaded
The hash object type is a static type owned by _hashlib; calling _md5.md5()
without _hashlib imported hit an uninitialized static type and panicked.
Assisted-by: Claude
* _csv: fall back to the built-in dialect defaults when none is registered
The dialect table is empty until csv.py registers 'excel', so _csv.reader([])
unwrapped a missing entry and panicked.
Assisted-by: Claude
* builtins: reject surrogates in compile()/eval() source instead of panicking
expect_str() panics on a str containing surrogates; convert with try_as_utf8
so eval(chr(0xd800)) raises.
Assisted-by: Claude
* _suggestions: require a list for _generate_suggestions candidates
The argument was collected before validation, so an unbounded iterable
exhausted memory.
Assisted-by: Claude
* lzma: size the filter chain before consuming it
filters= was collected into a Vec before the length check, so an unbounded
iterable exhausted memory. Take the length through the sequence protocol first.
Assisted-by: Claude
* classmethod: opt into GC traverse
staticmethod already declares traverse; classmethod did not, so a cycle through
the wrapped callable leaked.
Assisted-by: Claude
* posix: reject unbounded iterables in posix_spawn and setgroups
argv, setsigdef, setsigmask and setgroups bound an ArgIterable and collected it
before validating, so an infinite generator exhausted memory. Require a
list/tuple for argv, validate signals while streaming, and take setgroups
through the sequence protocol.
Assisted-by: Claude
* _ctypes: size Array slice assignment and _argtypes_ before collecting
Both eagerly collected their argument, so an unbounded iterable exhausted
memory before the length check ran.
Assisted-by: Claude
* sys: propagate the breakpointhook warning failure
warn() was unwrapped, so an unimportable $PYTHONBREAKPOINT under
-W error panicked instead of raising.
Assisted-by: Claude
* structseq: require the sequence argument when constructing a struct sequence
A no-argument construction produced an empty backing tuple, and reading any
named field then indexed out of bounds.
Assisted-by: Claude
* Guard the hash slot dispatch against unbounded recursion
`PyObject::hash` invoked the type's hash slot directly, so an element-wise
`__hash__` following a deeply nested object graph recursed one native frame
per level and overflowed the stack. Wrap the dispatch in `with_recursion`,
matching the repr and rich-compare dispatches in the same file, so
`hash(x)` on a nested tuple/GenericAlias/slice raises `RecursionError`.
Assisted-by: Claude
* genericalias: guard the __parameters__ walk against unbounded recursion
`make_parameters_from_slice` recursed into every list/tuple argument with
nothing counting the frames, so `list[L]` for a self-referential or deeply
nested `L` overflowed the native stack at subscript time. Wrap the descent
in `with_recursion`, which makes the walk fallible; `PyGenericAlias::new`,
`from_args` and `make_parameters` now return `PyResult` and every caller
propagates.
Assisted-by: Claude
* Fix the type confusion in PyAtomicRef's Debug impl
`PyAtomicRef<T>` stores a pointer to a `Py<T>`, as `Deref`, `load_raw`,
`swap` and `Drop` all read it, but `Debug` cast it to a bare `T` and
formatted the object header as payload bytes. For `PyFunction`, whose
`code: PyAtomicRef<PyCode>` has a pointer-chasing `Debug`, that
dereferenced header words and segfaulted.
Cast to `PyObject` instead, which is what `Drop` already does and which
also covers the `PyAtomicRef<PyObject>` and `PyAtomicRef<Option<T>>`
instantiations that have no `Py<T>`.
`_asyncio._enter_task` reached this through `{:?}` in its "Cannot enter
into task" message; format the two tasks with their Python repr, which is
what the message is meant to show.
Assisted-by: Claude
* _sre: disallow instantiating Match
`Match` inherited `object.__new__`, so `M.__new__(M)` produced an instance
whose `regs`/`string`/`pattern` were never filled in by a match run; the
mapping subscript path read them and dereferenced garbage. The type has no
public constructor, so mark it `DISALLOW_INSTANTIATION`, which makes
`M.__new__(M)` raise `TypeError: cannot create 're.Match' instances`.
Assisted-by: Claude
* utils: return the empty repr instead of asserting a non-empty collection
`collection_repr` took the first element with an `.expect()` justified by the
caller's preceding non-empty check. Another thread clearing the collection
between that check and the iteration made the iterator yield nothing and
panicked the worker. Fall back to the caller-supplied empty form, which is
the text those callers already produce for an empty collection.
Assisted-by: Claude
* itertools: advance cycle's index atomically
`cycle.__next__` did `fetch_add(1)` and then reset the index to 0 in a
separate store, so two threads replaying the saved items could both read an
index past the end of `saved` and panic on the slice access. Do the advance
and the wrap in one `fetch_update`.
Assisted-by: Claude
* _ctypes: reject a float argument to a foreign call without argtypes
`conv_param`, the conversion used when `argtypes` is not set, converted its
argument with `try_int`, which goes through `__int__` and so accepts a
float. `libc.strlen(1.5)` therefore passed 1 where the callee expects a
`char *` and the callee dereferenced it. Match `ConvParam`, which does a
`PyLong_Check` and converts nothing: take the branch only for an `int` (or
a subclass, so `True` still converts), and let a float fall through to
"Don't know how to convert parameter".
The branch below it converted a float to a C double, but `try_int` claimed
every float before it could run, so it was dead; `CArgValue::Double`
existed only for that branch and both go. Typed doubles are unaffected —
they travel as `CArgValue::Typed` with code 'd'.
Assisted-by: Claude
* Report the iterator itself from PyIter's traverse
`Traverse for PyIter<O>` delegated to the inherent `PyObject::traverse` of
the object it wraps, so it reported that iterator's referents instead of
the iterator. The iterator's own reference to those referents was then
never subtracted during the collector's reference-subtraction pass, the
referents kept a non-zero gc_refs, and every object reachable from them
was classified as a root.
Any cycle running through a type with a `PyIter` field therefore survived
collection: `map`, `filter`, `zip`, `enumerate`, `reversed` and the
`itertools` iterators all leaked, while the same cycle through a `list`,
`tuple` or `list_iterator` collected. Report the wrapped object, as the
`PyObjectRef`, `PyRef<T>` and `PyStackRef` impls do.
`itertools.tee` still leaks: its shared buffer is a `PyRc<PyItertoolsTeeData>`
rather than a Python object, so the collector cannot see through it.
Assisted-by: Claude
* Remove the obsolete expectedFailure on test_code_module.test_unicode_error
Compiling a source string containing a lone surrogate now raises
UnicodeEncodeError, so the test passes.
Assisted-by: Claude
* itertools: reserve the combination indices fallibly
`combinations` and `combinations_with_replacement` built their index vector
with an infallible allocation, so an `r` that passes the ssize_t check but
does not fit in memory aborted the process instead of raising MemoryError.
Assisted-by: Claude
* Apply the struct sequence constructor's dict argument
`structseq(sequence, dict)` discarded its second argument, so the hidden
fields past `n_sequence_fields` — `tm_zone`, `st_atime` and friends — were
always None when constructed directly or restored from a `(sequence, dict)`
pickle, and a non-dict second argument was accepted silently.
Take the dict, require it to be a dict, and fill the hidden slots the
sequence did not cover from it. A key that names a field the sequence
already supplied, or no field at all, is now a
"got duplicate or unexpected field name(s)" TypeError instead of being
dropped. Both arguments are bindable by name, as `sequence` and `dict`.
`os.stat_result` and `os.statvfs_result` did not accept a second argument
at all; they and `time.struct_time` now share the parsing.
Assisted-by: Claude
* _imp: report the argument count in find_frozen's arity error
Assisted-by: Claude
* Add regression tests for the reproduced crashers
One case per catalog entry, each asserting the behavior the fix produces:
recursion guards, the memory-unsafety sites, the overflow and unbounded
allocation guards, the eager-collection rejections, the unwrap sites, and
the cycles the collector now breaks. Every expected value was checked
against CPython 3.14.
Assisted-by: Claude
* Tolerate a changed-size RuntimeError in the set repr stress test
The mutator and reader threads race on purpose; a "changed size during
iteration" RuntimeError is a valid outcome of that race and should not fail
the test. The panic it guards against is not.
Assisted-by: Claude
* Move the crash regression tests into per-module snippets
crash_regressions.py collected every reproduced crasher in one file. Split it
into the snippet for the module each case exercises, and add stdlib_gc.py,
stdlib_asyncio.py, stdlib_lzma.py, stdlib_threading_set_repr.py and
stdlib_threading_itertools_cycle.py for the cases with no existing home.
The suite runs every snippet under the host CPython too, so the checks only
RustPython raises are guarded by sys.implementation.name: the hash and
__parameters__ recursion depth, and the deque repeat overflow. The float
ctypes argument accepts either TypeError or ctypes.ArgumentError.
Assisted-by: Claude
|
||
|
Jeong, YunWon
|
161ba746fc |
ssl: pass test_ssl with the rustls backend (#8502)
* Fix rustls test_ssl compatibility Assisted-by: OpenAI Codex:GPT-5 * Keep urllib3 compatible SSL version prefix Assisted-by: OpenAI Codex:GPT-5 |
||
|
Jeaeun Kim
|
6b747b5a76 |
ssl: support explicit session reuse (#8466)
* ssl: support explicit session reuse Assisted-by: Codex-5.6-sol * ssl: support explicit TLS 1.3 session reuse Assisted-by: Codex-5.6-sol |
||
|
lms0806
|
25e2df1731 | Implement TclObject support in tkinter varname_converter (#8465) | ||
|
Joshua Megnauth
|
3c70a1ba58 |
Replace deprecated chrono with jiff (#8436)
See: chronotope/chrono#1768 |
||
|
Jeong, YunWon
|
cbd77af58c |
socket: accept a filesystem-encoded hostname in sethostname (#8437)
`socket.sethostname` took `PyUtf8StrRef`, so it rejected `bytes` outright and refused a `str` carrying a surrogate escape. `socketmodule.c socket_sethostname` accepts a bytes object directly, falls back to `PyUnicode_FSConverter` for anything else, and hands the syscall the resulting buffer and its length — the name is never required to be UTF-8. Take `FsPath`, the converter `if_nametoindex` in this same module already uses, and pass its bytes down. `host_env::socket::sethostname` correspondingly takes `&[u8]` and builds the `OsStr` from them; `nix::unistd::sethostname` accepts `AsRef<OsStr>` and passes pointer and length to the syscall, so nothing on the path needs a NUL terminator or valid UTF-8. `Lib/test/test_socket.py test_sethostname` covers this: it calls `socket.sethostname(b'bar')` and asserts the hostname changed. The test is skipped unless run as root, which is why the gap went unnoticed. Assisted-by: Claude |
||
|
Jeaeun Kim
|
4ad64a99a2 |
ssl: store SSL socket owners as weak references (#8423)
Assisted-by: Codex:gpt-5.6-sol |
||
|
Jeaeun Kim
|
8c241d86bc |
fix(socket): use object repr in deallocation warnings (#8422)
Assisted-by: Codex:gpt-5.6-sol |
||
|
이광효
|
beff2b02ac |
csv: port CPython reader state machine (#8381)
* csv: port CPython reader state machine Replace the csv-core reader and per-item quote scanner with one Rust implementation of CPython's nine-state reader parser. Keep parser state across iterator items and distinguish virtual item boundaries from true iterator exhaustion. Centralize field completion so quote provenance, empty-field None conversion, float conversion, strict parsing, field limits, blank rows, and escaped or quoted newlines share one path. Apply the existing reentrant-iterator generation check to every iterator item consumed by a record. Also accept empty ASCII line terminators like CPython, report a missing csv-core writer sentinel as _csv.Error instead of panicking, and name the FSM entry point process_parser_input. Remove the expected-failure markers from the eleven reader tests that now pass. Assisted-by: Codex:gpt-5.6-sol * csv: test CRLF boundaries with custom lineterminator |
||
|
Chanho Lee
|
4d890b33db |
Fix sqlite autocommit cursor transactions (#8419)
Assisted-by: Codex:gpt-5.6-sol |