* unicodedata: resolve name aliases and named sequences
Vendor Unicode 17 NameAliases.txt and NamedSequences.txt into
rustpython-unicode so unicodedata.lookup matches CPython: formal
aliases and named sequences on the modern view, none on ucd_3_2_0.
Tangut derived-name lookup is case-insensitive. \N{} still rejects
named sequences because they are not a single character.
Assisted-by: Claude
* test_ucn: drop expectedFailure on test_aliases
The name-alias tables make the test pass; leaving the marker
fails CI as an unexpected success.
Assisted-by: Claude
* unicode: reject whitespace in Tangut derived names
parse_hex_code does not skip spaces around the hex digits.
Assisted-by: Claude
TlsConnection, RecordCursor, rustls error mapping, hostname/ALPN/version
helpers, and the capturing client session store now live in host_env.
stdlib _ssl keeps socket/MemoryBIO I/O and Python objects.
Remove unused stdlib sha2 and leftover workspace TLS crates.
Assisted-by: Grok:grok-4.6
* fix(vm): report which format string mistake was made
`ToPyException for FormatParseError` mapped two variants and sent the rest to
"Unexpected error parsing format string", so most format-string mistakes gave
no hint about the mistake. Against CPython 3.14.0:
| expression | before | CPython |
| --- | --- | --- |
| `"a}b".format()` | Unexpected error… | Single `}` encountered in format string |
| `"{0.}".format(1)` | Unexpected error… | Empty attribute in format string |
| `"{0[]}".format([1])` | Unexpected error… | Empty attribute in format string |
| `"{0[1}".format([1,2])` | Unexpected error… | expected `}` before end of string |
| `"{0[0]x}".format([1])` | Unexpected error… | Only `.` or `[` may follow `]` in format field specifier |
| `"{:{{}}".format(1)` | Unexpected error… | unmatched `{` in format spec |
The match is now exhaustive, so a new variant cannot silently inherit the
generic message.
Two cases are deliberately left generic, with the reason in a comment beside
each. `UnknownConversion` is raised both for a multi-character conversion
(`{0!xy}`, where CPython says "expected ':' after conversion specifier") and
for an empty one (`{0!}`, "unmatched '{' in format spec"), so one message
cannot serve both without the parser saying which it hit. And a trailing lone
`{` reaches `UnmatchedBracket` rather than
`UnescapedStartBracketInLiteral`, so `"{".format()` still differs from CPython;
that needs a parser change, not a message.
Expectations in the new snippet are CPython 3.14.0's output and pass under both
interpreters. Reverting only format.rs fails the snippet's first assertion.
Assisted-by: Claude Code:claude-opus-5
* fix(common): separate a stray `{` from an unclosed field
`parse_literal` already reports `UnescapedStartBracketInLiteral` for a brace
that was not doubled, but `from_str` discards it with `or_else` and retries as a
field, so `"{".format()` surfaced `UnmatchedBracket` and read as
"expected '}' before end of string". CPython separates the two by whether
anything follows the brace:
"{" "a{" Single '{' encountered in format string
"{s" "a{b" expected '}' before end of string
"{0" expected '}' before end of string
`parse_spec` now returns the literal error when the text is the brace alone,
which is also what makes `UnescapedStartBracketInLiteral` reachable — the
previous commit could map it but not trigger it.
`format_parse_lone_start_bracket` pins both directions at the parser level, and
the snippet covers the four messages under CPython 3.14.0 as well. The existing
`format_parse_fail` assertion for `"{s"` is unchanged.
Assisted-by: Claude Code:claude-opus-5
* ssl: move remaining rustls engine into host_env
Cert loading, rustls verifiers, ClientConfig/ServerConfig construction,
key logging, acceptor helpers, and X509 error mapping now live next to
the trust store. stdlib _ssl keeps the Python object and socket adapters.
Assisted-by: Claude
* Remove unused workspace SSL dependencies
Drop oid-registry, rustls-platform-verifier, and x509-cert from
workspace.dependencies.
Assisted-by: Grok 4.6
* ssl: fire msg callbacks and compute tls-unique
Observe TLS records to invoke _msg_callback, and derive TLS 1.2
tls-unique from the KeyLog master secret plus rustls PRF.
Restore CHANNEL_BINDING_TYPES and drop the remaining test_ssl
RUSTPYTHON skips.
Assisted-by: Claude
* lzma: build the xz-core engine on wasm32
Expose `_lzma` on wasm32 and Android by compiling the xz-core
engine on those targets instead of gating it out.
Assisted-by: Claude
* ssl: move the rustls engine into rustpython-common
Extract the VM-independent rustls helpers (MemoryBio, certificate
decode, cipher strings, OID table, chain builder, providers, and
TLS record observation) so stdlib _ssl wraps a shared engine.
Assisted-by: Claude
* ssl: keep the rustls engine in host_env
TLS helpers share the host trust store and default verify paths, so
they belong next to native_certs rather than in rustpython-common.
Assisted-by: Claude
* stdlib: drop unused jiff after moving cert decode
Assisted-by: Claude
Hold the rustls connection mutex only around rustls operations
in handshake, read, write, and shutdown. Socket and MemoryBIO
waits run without that lock so a blocking recv cannot deadlock
a concurrent send.
Remove the hang skip on test_thread_recv_while_main_thread_sends.
Assisted-by: Claude
* Align contextvars, CLI startup, and stat validation with CPython
Validate Context and ContextVar arguments and mapping keys, hash names through the Python protocol, and preserve string contents without invoking __str__. Set the main module loader for command and stdin execution, report missing scripts with exit status 2, and reject file descriptors combined with follow_symlinks=False.
Enable the nine corresponding stdlib regression tests.
Assisted-by: Codex:gpt-6
* Move set_main_builtin_importer off the compiler feature
Define the method in vm_new.rs so builds without rustpython-compiler,
such as example_projects/barebone, can compile. Also set BuiltinImporter
on an existing __main__ when its loader is missing or None.
Assisted-by: Grok:4.6
Keep BIO writes from filling asyncio's socket transport past its
high-water mark when SO_SNDBUF is small. Drop the Unix skip on
test_sendfile_ssl_pre_and_post_data.
Assisted-by: Claude
Pin the RustPython Ruff fork to the patch commit based on Ruff 0.16.5 and adapt compiler and AST integration to the updated APIs.
Assisted-by: Codex:GPT-5
* ssl: accept ClientHello before selecting the SNI context
Replace the provisional server connection and replay flags with explicit handshake states. Retain fatal alerts across nonblocking retries and preserve fragmented input on timeout. Enable the three SNI alert regressions.
Assisted-by: Codex:GPT-5
* ssl: share socket and MemoryBIO transport progress
Queue drained TLS records before fallible output and use one handshake record loop for both transports.
Assisted-by: Codex:GPT-5
* ssl: unify shutdown and fatal alert lifecycle
Track close-notify progress in TlsState, preserve pending alerts across retries, and share record-boundary shutdown for sockets and MemoryBIO. Retain rustls errors until Python exception conversion.
Assisted-by: Codex:GPT-5
* ssl: emit the full peer certificate dict
Use the same certificate dict helper for getpeercert() and
Certificate.get_info() so OCSP, caIssuers, and CRL URLs are present.
Clear CHANNEL_BINDING_TYPES on the rustls backend, and add a MemoryBIO
leftover-plaintext snippet.
Assisted-by: Claude
* ssl: add key logging with a shared host append sink
Port the context-owned keylog sink design from Pyre, retaining CPython 3.14 path and context-switch semantics. Keep host file lifecycle in host_env and TLS formatting in stdlib.
Assisted-by: Codex:GPT-5
* ssl: use core and alloc imports for key logging
Assisted-by: Codex:GPT-5
Accept __index__ values when converting resource limits and only report PyREPL fallback reasons for terminal-backed interactive sessions.
Assisted-by: Codex:gpt-5
Port the connection-owned chain builder from Pyre. Preserve verification inputs with TLS configurations and sessions, report the WebPKI-selected path, and publish only its selected capath anchor. Match CPython chain reporting for CERT_NONE and resumed sessions.
Assisted-by: Codex:GPT-6
`PyStr::__add__` delegated to the right operand's `__radd__` whenever it had
one, and returned whatever came back. For every numeric type that is a
`NotImplemented`, which then reached the generic binary-op handler:
>>> "a" + 1
TypeError: unsupported operand type(s) for +: 'str' and 'int'
CPython falls back to `str`'s `sq_concat` once the reflected call declines, so
it names the concatenation instead:
TypeError: can only concatenate str (not "int") to str
The specific message was already here, but only reachable for operands with no
`__radd__` at all, so `list`, `object`, `bytes` and `None` were correct while
`int`, `float`, `bool` and any class with a declining `__radd__` were not.
A `__radd__` that returns a value still wins, unchanged.
Verified against CPython 3.14.0: the new snippet asserts the message for
`int`, `float`, `bool`, `list`, `None`, `bytes` and a declining `__radd__`,
and passes under both interpreters. Reverting only the `str.rs` change fails
its first assertion.
Assisted-by: Claude Code:claude-opus-5
* Fix /dev/fd, rlimit, spawn scheduler, and context cache
Honor PYTHONEXECUTABLE when computing sys.executable.
Rewind the file offset after a pyc magic probe.
Stop masking rlimit values with RLIM_INFINITY.
Print the pyrepl-unavailable warning before the rustyline shell.
Parse posix_spawn scheduler and apply it where the host supports it.
Update ContextVar cache fields under one lock.
Remove expectedFailure and skip markers for tests that now pass.
Assisted-by: Grok:4.6
* address review: check rlim_t overflow on convert
Replace the u64-to-rlim_t as-cast in py2rlim_limit with
a checked try_from, matching py2rlim.
Assisted-by: Grok:4.6
* Add no_memory_error for allocation failures
Assisted-by: Claude Code:claude-fable-5-1
* Use no_memory_error at allocation failure sites
Assisted-by: Claude Code:claude-fable-5-1
* Raise MemoryError instead of aborting on sequence repeat allocation failure
Vec::with_capacity and Vec::reserve in SequenceExt::mul and
SequenceMutExt::imul call handle_alloc_error on failure, which aborts
the process under panic = "abort". `[1, 2, 3] * 10**17` passes the
MAX_MEMORY_SIZE guard (2.4e18 bytes < isize::MAX) and aborted at the
allocation instead of raising MemoryError.
Use try_reserve_exact and map failures to vm.no_memory_error(). This
also removes the n != 2 special case in imul: with the reserve done
unconditionally, the copy loop runs zero times for n == 2 and the
final append uses the reserved capacity.
Assisted-by: Claude Code:claude-fable-5-1
* Recycle MemoryError instances through a dedicated payload freelist
no_memory_error allocated the exception object on every raise, so
under real memory exhaustion the raise itself could abort. Keep dead
MemoryError husks in a process-wide reserve and revive them on the
next raise without touching the allocator.
The freelist is scoped to MemoryError at the type level. PyMemoryError
already exists as a transparent wrapper payload, but instances never
used it: the default exception slot_new delegates to the base class,
so everything was constructed as PyBaseException. Route
no_memory_error through a PyMemoryError-typed construction instead;
the freelist hooks live on PyMemoryError's own PyPayload impl, the
dealloc glue is monomorphized per payload type, and the existing
exact-type push gate admits exactly MemoryError husks.
The #[pyexception] macro always generates the PyPayload impl, which
left no way to attach freelist configuration; a payload = "manual"
attribute now skips that generation so a type can write its own impl,
mirroring the generated PAYLOAD_TYPE_ID chain and class-based
downcast validation that keep the payload flavors interchangeable.
Exceptions are the first HAS_DICT freelist type, so their husks carry
an ObjExt prefix that PyInner::dealloc would normally drop; clear its
instance dict and member slots before pushing so no stale attributes
survive into a revived instance.
The reserve is a process-wide Mutex rather than a thread_local so a
single prewarm at interpreter init covers every thread; MemoryError
is a cold path, so the lock is uncontended. Python-level MemoryError()
still constructs a PyBaseException payload through the delegated
slot_new and simply does not participate, matching CPython, where
only the PyErr_NoMemory path uses the preallocated reserve.
Assisted-by: Claude Code:claude-fable-5-1
* Clear MemoryError husks before they enter the freelist
Assisted-by: Claude Code:claude-fable-5-1
* Use no_memory_error for the bz2 allocation failure
The shared bz2 engine maps BZ_MEM_ERROR to Bz2Error::Mem; report it
through no_memory_error like the other allocation-failure sites, matching
PyErr_NoMemory in CPython's _bz2module.
Assisted-by: Claude Code:claude-fable-5-1
* Change MemoryError husks Vec to a fixed-size array
Assisted-by: Claude Code:claude-fable-5-1
* Increase MemoryError freelist size from 4 to 16
* Address review: keep repeat and husk clearing allocation-safe
Assisted-by: Claude Code:claude-fable-5-1
* Collect consecutive charmap encoding failures
Pass one contiguous run of unmappable characters to the codec error handler, and use the full run when a text replacement is itself unmappable.
Assisted-by: Codex:gpt-5
* Leave conversions out of standalone format specs
A conversion belongs to a replacement field rather than the format specification consumed by format(), so stop accepting and storing it in FormatSpec.
Assisted-by: Codex:gpt-5
* Decode empty byte strings without looking up a codec
Match PyUnicode_FromEncodedObject by returning the shared empty string before resolving the requested encoding for empty bytes and bytearray inputs.
Assisted-by: Codex:gpt-5
* Validate call shape before converting builtin arguments
Reject unknown keywords and excess arguments before conversions can raise a different error. Align Python-function positional-only handling, use declared list arguments in heapq, keep delegated codecs positional-only, and preserve CPython parameter names and validation order.
Assisted-by: Codex:gpt-5
* Align attribute and exception initialization errors
Give __dict__ and type __qualname__ their CPython diagnostics, preserve OSError filename fields correctly, accept seven-item SyntaxError details, and report Unicode error constructor arity before converting values.
Assisted-by: Codex:gpt-5
* Expose missing Unix and Darwin module APIs
Add the platform constants exported by fcntl, mmap, select, socket, syslog, termios, stat, errno, time, and _sre. Implement posix.killpg and resource.getpagesize, and expose resource.error as OSError.
Assisted-by: Codex:gpt-5
* Honor the configured Python CPU count
Parse -X cpu_count and PYTHON_CPU_COUNT, expose the stored value through sys._get_cpu_count_config, and let os.cpu_count prefer it over the host count. Also publish getsizeof with its CPython parameter name.
Assisted-by: Codex:gpt-5
* Fix CPU count and WASI CI coverage
Remove the obsolete expected-failure marker now that CPU count overrides pass, and avoid exporting FASYNC where WASI libc has no O_ASYNC constant.
Assisted-by: Codex:gpt-5
* self review
Keep host certificate path probing with the native certificate provider, select the first available OpenSSL-style file and directory, and leave environment overrides to ssl.py. This also gives Android string defaults instead of the previous None values.
Co-authored-by: Ivan Mironov <mironov.ivan@gmail.com>
Assisted-by: OpenAI Codex: GPT-5
Retain CA directories on SSLContext, materialize their OpenSSL-style hashed entries when building rustls configurations, and cache parsed DER until directory mtimes change. Publish only the trust anchor used by a completed handshake.
Co-authored-by: Ivan Mironov <mironov.ivan@gmail.com>
Assisted-by: OpenAI Codex: GPT-5
This commit fixes most of UCD's names. The two main fixes are:
* Handling chars missing in 3.2.0 by checking for membership #8548
* Handling names not present in unicodenames
AI disclosure: I used AI for the initial research of this patch to
determine why the Tangut characters are missing. AI also assisted with a
bug fix where I neglected to parse the Tangut u32 as hex. I wrote all of
the code.
Assisted-by: GPT 5.6 Luna
* ssl: read the OID table out of OpenSSL's object database
`txt2obj` and `nid2obj` have to answer with OpenSSL's own numbers, and the
table behind them was 39 entries transcribed by hand, so every object
outside that set was reported unknown.
Read OpenSSL's own files instead: `obj_mac.num` carries each object's NID
and `objects.txt` carries its OID, short name and description, and the
identifier `objects.pl` derives per object -- the `!Cname` override, else
the description, else the short name, each qualified by the enclosing
`!module` with `-` spelled `_` -- joins the two. `OidEntry`, `OidTable`
and the `find_by_*` functions keep their shape, so `ssl.rs` only changes
where an object has no OID to report.
Three details the object database settles that a curated table did not
have to:
- an object written with one name carries it as both, so
`nid2obj(48)` is `('emailAddress', 'emailAddress', ...)`;
- an object need not have an OID at all -- `DES-EDE3` and 139 other
algorithm names have a NID and nothing to encode -- so the fourth
field of the tuple is None for those, as it is on OpenSSL;
- one arc does not encode, since DER packs the first two into a byte,
so the bare roots `iso` and `itu-t` are OID-less too, while two arcs
such as X500's `2.5` are a real object.
Checked against `_ssl.nid2obj` on CPython built with OpenSSL, over every
NID it answers: 1387 of its 1390 present, with no disagreement on any of
them, up from 39. The three absent are OpenSSL's unnamed placeholder
slots, which it reports as ('NULL', 'NULL', None).
The data files come from openssl/openssl `crypto/objects/`, via the
rustls integration rewrite in #8007.
Co-authored-by: Ivan Mironov <mironov.ivan@gmail.com>
Assisted-by: Claude
* ssl: parse OpenSSL cipher strings in set_ciphers
Replaces the pattern matcher `set_ciphers` used with the grammar
`man openssl-ciphers` documents, in a new `ssl/cipher.rs` taken from
#8007: `!`, `-`, `+`, `@STRENGTH`, `@SECLEVEL=n`, `+`-joined
conjunctions, the DEFAULT/ALL/COMPLEMENTOF* groups, protocol versions,
SUITEB*, and the aXXX/kXXX/eXXX prefixes.
`set_ciphers` raises "No cipher can be selected." for a string that
names nothing the provider offers, which is the one failure
SSL_CTX_set_cipher_list reports, and keeps the TLS 1.3 suites selected
in provider order because that setter does not reach them. A SUITEB*
string pins its own key exchange groups, which `prepare_kx_groups`
now reads.
Against CPython's OpenSSL over 78 cipher strings, the selections agree
except under `@SECLEVEL=n`, which OpenSSL applies at handshake time and
this applies to the selection.
`CryptoExt` gains `default_cipher_suites` so that `DEFAULT` and `ALL`
are distinguishable.
Co-authored-by: Ivan Mironov <mironov.ivan@gmail.com>
Assisted-by: Claude
* ssl: finish cipher reporting and provider handling
Use the provider-ordered cipher metadata table as the single source for parsing, reporting, OpenSSL names and descriptions. This makes get_ciphers reflect the context selection and validates curves and Suite B against the active provider.
Keep set_ciphers from changing TLS 1.3 suites, preserve the provider's order, fix empty conjunctions, and keep FIPS initialization within the provider's filtered cipher and key-exchange defaults.
Co-authored-by: Ivan Mironov <mironov.ivan@gmail.com>
Assisted-by: Claude
Assisted-by: CodeRabbit
Assisted-by: OpenAI Codex: GPT-5
* ssl: preserve OpenSSL OID lookup rules
Keep object-name lookup case-sensitive and canonicalize numeric OID arcs before consulting the generated OpenSSL object table.
Assisted-by: OpenAI Codex: GPT-5
* ssl: preserve cleartext after MemoryBIO shutdown
Read at most one TLS record while waiting for close_notify in BIO mode, leaving any post-TLS cleartext queued for the BIO owner.
Co-authored-by: Ivan Mironov <mironov.ivan@gmail.com>
Assisted-by: OpenAI Codex: GPT-5
* ssl: reject empty cipher conjunction operands
Keep empty operands visible to the parser so malformed OpenSSL cipher strings such as AES+ and ALL++AES are rejected instead of normalized.
Assisted-by: CodeRabbit
Assisted-by: OpenAI Codex: GPT-5
* ssl: make Suite B cipher ids unambiguous
Spell out the u16 conversion so Windows-only iterator implementations cannot make collection type inference ambiguous.
Assisted-by: OpenAI Codex: GPT-5
* ssl: move native certificate loading behind host_env
Keep platform trust-store loading and filesystem predicates on the host boundary while exposing backend-neutral certificate bytes to the rustls integration.
Assisted-by: OpenAI Codex: GPT-5
* ssl: invalidate server config after cipher changes
Rebuild cached server configuration after set_ciphers updates the selected suites or Suite B key-exchange groups.
Assisted-by: CodeRabbit
Assisted-by: OpenAI Codex: GPT-5
* ssl: load default certificate sources independently
Honor SSL_CERT_FILE and SSL_CERT_DIR independently. Keep directory certificates as lazy capath candidates, and fall back to native roots only when no environment certificate file was loaded.
Co-authored-by: Ivan Mironov <mironov.ivan@gmail.com>
Assisted-by: OpenAI Codex: GPT-5
---------
Co-authored-by: Ivan Mironov <mironov.ivan@gmail.com>
* ci: hold a CodSpeed run to the machine main's baseline was measured on
CodSpeed compares a run against a base run and refuses to call the
comparison meaningful when the two ran on different machines: the runner
uploads a `SystemInfo` record beside the profile carrying the OS, the
architecture, the CPU's vendor id, brand and flags, the physical core
count and the installed memory, and a hosted pool hands out whatever
machine is free. A simulated measurement is fixed against how fast the
machine is, but not against what it makes the process execute -- glibc
selects `memcpy` and the string routines through IFUNC on CPU flags, a
new stable rustc emits different code, and these benchmarks run each
program under CPython through pyo3 as well as under RustPython, so the
runner's default interpreter is an input too.
`scripts/codspeed-environment.py` records those fields plus the rustc,
libc and CPython versions, and compares a run against a recording. The
benchmark job now restores the recording main last wrote, records over it
when the ref is main, and compares against it otherwise; the install,
build and measure steps carry `if: steps.environment.outputs.match !=
'false'`, so a run that drew a different machine finishes green having
uploaded nothing. A missing or unfetchable recording is read as "nothing
to disagree with" rather than as a mismatch.
Assisted-by: Claude
* ci: read an unreadable environment recording as not a match
A cache miss, an eviction or a cache-service outage leaves the reference file
absent, and the comparison then reported `match=true`: every guarded step ran
and the run uploaded a measurement without having identified the machine it was
taken on.
An absent recording now reports `match=false`, and says so as a `::warning`
rather than the `::notice` a mismatch emits, so a job that measures nothing
because it has nothing to compare against is distinguishable from one that
measures nothing because the machine differs.
Assisted-by: Claude
The digest, HMAC, PBKDF2 and scrypt state moves out of the `_hashlib`
module into a feature-gated `rustpython_common::hashlib` that owns each
state behind a `Mutex` and reports plain Rust results; `_hashlib` and
`_blake2` keep the object glue.
The engine writes its state through a raw pointer into a caller-owned
buffer, so the buffer is boxed and only the pointer travels with the
Python object, and it carries the alignment the engine asserts. The
wrapper is built only after a successful init, so a rejected digest name
leaves nothing to drop over storage the engine never wrote.
blake2 moves from `blake2` to `blake2b_simd` / `blake2s_simd`, whose
`Params` carry the whole parameter block. `_blake2.blake2b` and
`blake2s` become immutable types with `SALT_SIZE`, `PERSON_SIZE`,
`MAX_KEY_SIZE` and `MAX_DIGEST_SIZE` as class attributes, and their
instances are instances of those types. They accept `key`, `salt`,
`person`, `fanout`, `depth`, `leaf_size`, `node_offset`, `node_depth`,
`inner_size` and `last_node`. `_hashlib.scrypt` is published on the
same engine.
Six `test_hashlib` expectedFailure markers drop with it. `blake2`,
`digest`, `dyn-clone`, `hmac` and `pbkdf2` lose their last consumer.
Assisted-by: Grok
Assisted-by: Claude
Move the VM-independent bzip2 stream owner into a `bz2`-gated
`rustpython-common` module beside the zlib and lzma engines, and keep
`rustpython-stdlib` as the Python object and exception adapter. The adapter no
longer names the `bzip2` crate; the `bz2` feature carries it.
`_bz2` was the last user of the generic `DecompressState` / `DecompressStatus` /
`Decompressor` machinery in `stdlib/src/compression.rs`, so that goes with it;
`DecompressArgs` stays for zlib and lzma. The decompressor feeds the stream
through the existing `Chunker`, so buffered and freshly supplied input are no
longer joined into a new allocation first.
`test_decompress_after_data_error` passes on the ported engine, so its
`expectedFailure` marker goes.
Assisted-by: Grok
* builtins: name the other operand with tp_name in str and Template __add__
`str.__add__` and `Template.__add__` built their TypeError with
`PyType::name()`, which drops the module, where CPython formats `tp_name`:
>>> t"a" + "b"
TypeError: can only concatenate Template (not 'str') to Template
CPython 3.14 reports the qualified name and quotes the operand with double
quotes, as `Objects/unicodeobject.c` and `Objects/templateobject.c` do:
TypeError: can only concatenate string.templatelib.Template
(not "str") to string.templatelib.Template
`Template.__add__` also spelled its own name literally rather than taking it
from `PyClassDef::TP_NAME`, so the two halves of the message could drift apart.
Known gap: `slot_name()` still does not qualify a class defined in a module, so
`"a" + collections.OrderedDict()` names `OrderedDict` where CPython names
`collections.OrderedDict`. This affects both methods.
Unblocks test_template_concatenation in test_tstring.
Assisted-by: Claude Code:claude-opus-5
* interpolated strings: align f-string and t-string diagnostics with CPython
RustPython re-derives CPython's syntax diagnostics by scanning the source rather
than by translating ruff's parse errors, and the scanner shared by f-strings and
t-strings misread several replacement field states:
- A field that ran off the end of the literal (`f'{'`) was reported as an
expression that failed to start, where CPython asks for the closing brace.
- A field whose expression cannot start (`f'{;'`) and one whose expression runs
into a stray character (`f'{a;'`) both fell through to the missing brace
message instead of CPython's two distinct ones.
- A conversion or format spec following `=` was never validated, and a format
spec following a valid conversion was skipped entirely.
- Format specs were scanned with the `{{` escape rule that only applies to
literal text, so nested replacement fields inside them went unchecked.
- Brackets opened inside a field were never matched, so `f'{a[4)}'` and
`f'{3)+(4}'` fell back to a bare "invalid syntax".
- Comments inside a field were not recognised, so a `#` that swallows the
closing brace was reported as an unterminated literal, and a comment in a
multi-line field hid the expression behind it.
- Unterminated literals were named "string" whatever their prefix, and an
interpolated literal left with a replacement field open reported the missing
quote rather than the brace.
A field is only "never closed" while the tokenizer is still reading its
expression. Past the field's own `:` it is emitting literal text again, so
`f'{a:>5` runs out of input as an ordinary unterminated literal while `f'{a`
reports the brace. The field's own `:` is also not the one in a slice, a display
or a lambda, and an unclosed bracket inside the expression is what CPython names
rather than the field around it, so this walk tracks the whole delimiter stack.
Two messages were worded from the wrong branch of CPython's tokenizer. The hint
"perhaps you escaped the end quote?" is raised only where lexer.c handles a
literal without an interpolation prefix; its `%c-string` branch has just the
triple-quoted and plain forms, so pairing the hint with a prefix produced
`unterminated f-string literal (...); perhaps you escaped the end quote?`, which
CPython never emits. And a bracket mismatch inside a field dropped the
` on line %d` clause that lexer.c adds whenever `parenlinenostack[level]` differs
from the current `lineno`; the general bracket scanner in this file already
computed that suffix, so `f"""{a[\n4)}"""` now names line 1 as CPython does.
The rules these now follow are the ones CPython spells out in
Grammar/python.gram (`invalid_fstring_replacement_field` and its t-string twin)
and in Parser/lexer/lexer.c, which likewise parameterises the literal's prefix
rather than duplicating the messages.
Only a field's expression is code. A format spec is text, so a `(` there opens
nothing and a `#` selects the alternate form instead of starting a comment;
inside the expression a `#` does start one. Scanning the whole field for either
therefore reported diagnostics against valid literals, and because these
scanners only run once the source has already failed to parse, that let a good
literal take the blame for an error further down the file. Expression-level
checks now stop at the field's top-level separator and skip comments, and the
literal is walked field by field rather than byte by byte so that a `{` inside a
comment no longer opens one. A pre-existing instance of the same bug in the
line-continuation check is fixed along with them.
Ruff reports a mixed literal concatenation only as a bytes/non-bytes mix, so
`t"x" b"y"` arrived as a bytes error where CPython names the t-string. CPython's
`invalid_string_tstring_concat` is an `invalid_` rule, reached only on the error
pass, and `strings` tries `(fstring|string)+` ahead of it: that alternative
consumes the concatenation's leading run of non-t-string literals, so a mix among
those raises from `_PyPegen_concatenate_strings` on the first pass and sets
`error_indicator`, which suppresses the t-string rule. The t-string message
therefore wins for `t"x" b"y"` and the bytes message keeps precedence for
`"a" b"b" t"c"`. Both are now settled here rather than left to whichever scanner
runs next, which had been answering the second case with an unrelated
"Is this intended to be part of the string?".
A leading doubled `=` or `!` was read as a marker with an empty expression before it,
so `f"{==a}"` reported `valid expression required before '='` where CPython has no
expression to report at all. The same lookahead now guards that branch, and a
doubled `=` or `!` counts as an expression that cannot start.
A top-level `=` or `!` was read as a debug or conversion marker without checking
whether it belonged to a longer operator, so `f"{a==b}"` and `f"{a!=b}"` were also
read as fields ending early. CPython tokenises `==`, `!=`, `<=` and the rest as
single tokens before it ever considers the debug marker, so the separator scan now
skips a `=` that follows one of `= ! < > + - * / % & | ^ @ :` or precedes another
`=`, and a `!` that precedes one.
An expression cannot end on an operator that still wants an operand, and CPython
points at that operator rather than at the brace. `f"{a==}"`, `f"{a and}"` and
`f"{a.b.}"` fell through to a plain `invalid syntax`; they now carry the same
message and column CPython gives, reusing the scan that already handled `;` and
`$` for exactly this shape. `is not` and `not in` are single operators, so the
first word is what gets pointed at, and `...` is consumed in whole triples so
that `f"{....}"` blames the fourth dot rather than the first. The region has to
be walked forwards: a comment's terminating newline is whitespace, so trimming
backwards from the end would step into the comment body and read a triple-quoted
field whose comment ends in `+` as an expression ending in `+`.
That check runs only once the field is known to close, which is where a stray
character and a dangling operator part ways. A stray character is a finished
token, so the parser rejects it on the lookahead in
`annotated_rhs !('='|'!'|':'|'}')` and `f"{a;"` gets the separator message even
with no closing brace. A dangling operator instead makes the parser ask for one
more token, and producing it runs into the literal's closing quote, where
lexer.c answers from its `INSIDE_FSTRING(tok)` branch with
`%c-string: expecting '}'` before any `invalid_` rule is reached. So `f"{a and"`
is a missing brace while `f"{a and}"` names the operator.
A leading `.` was read as a character that cannot start an expression, so the
Ellipsis literal `f"{...}"`, the float `f"{.5}"` and its signed form `f"{-.5}"`
were reported as broken whenever the file failed to parse somewhere else.
`pegen` joins the cpython spelling dictionary, for the `_PyPegen_*` names these
comments cite.
Known gaps, none covered by either test file. Needing the longest valid expression
prefix, which a character scanner cannot compute: a starred tuple (`f"{*a,}"`) and
a dict-unpacking display are read as unable to start an expression, and
`f"{a===b}"`, `f"{a b}"` and `f"{a,,}"` fall through to a bare `invalid syntax`.
The caret still differs from CPython's on `expecting a valid expression after '{'`,
`expecting '}'`, `expecting '}', or format specs` and `unterminated ... literal`,
and on the messages CPython points at a whole token for, such as
`f"{lambda x: x}"` and `f"{x! r}"`. A quote inside a format spec is still treated
as a string delimiter.
Two more are reachable only through a t-string concatenation that ruff does not
report as a bytes mix, so `mixed_tstring_literal_error` never runs: three or more
literals with no bytes literal among them (`t"a" t"b" "c"`, `"a" "b" t"c"`) answer
with `invalid syntax. Is this intended to be part of the string?`, and so does a
tokenizer-level nesting overflow where CPython has
`too many nested f-strings or t-strings`. Two-literal mixes are correct because
ruff's own error carries the wording.
Two pre-existing bugs outside this change are worth naming, since it touches
their neighbourhood. `unterminated triple-quoted ... literal` reports
`detected at line 2` for a one-line source where CPython reports line 1, for every
prefix including none, so it is in the shared line arithmetic rather than the
interpolated path. And `str.__add__`'s message is unreachable for an operand that
defines `__radd__`: `"a" + 1` falls through to the generic
`unsupported operand type(s)` from the binop dispatch, where CPython has
`can only concatenate str (not "int") to str`.
Unblocks test_syntax_errors and test_literal_concatenation in test_tstring, and
test_comments,
test_conversions, test_invalid_syntax_error_message, test_mismatched_braces,
test_mismatched_parens, test_parens_in_expressions and
test_syntax_error_after_debug in test_fstring.
Assisted-by: Claude Code:claude-opus-5
* diagnostics: give the source scanners a diagnostic type
Every scanner in `cpython_parse_diagnostic_override` returned a bare
`Option<(String, usize, usize)>` — 69 signatures and 114 construction sites of
message, start offset, end offset — which the consumer then reassembled into a
message and a range one call later. They return a `CpythonDiagnostic` now, and
the two things that consume one, `NormalizedParseDiagnostic::other` and
`CompileError::from_source_error`, take it whole.
The type earns its name. These scanners run *after* ruff's parse has failed, so
what they produce is not a parse error but a reconstruction of what CPython
would have said about the same source; nothing here ever hands one back to ruff.
Reusing `ruff_python_parser::ParseError` would have fit the shape — it is
`{ ParseErrorType, TextRange }` — but every one of the 114 sites would have
wrapped its message in the single `OtherError` variant of a ninety-variant enum,
widened `other` and `from_source_error` to accept variants no caller
constructs, and, being a foreign type, ruled out the constructor that now
carries the `u32` cast and its justification. The `OtherError` wrapping happens
once per consumer instead, at the boundary where this does cross into ruff's
vocabulary.
Two nameless tuples that carried more than a diagnostic get names, following
`CallArgFrame` and `AssignmentContext` in this file:
- `bracket_syntax_error` returned the message alongside a bare `bool` for
whether the bracket was left open, which the caller needs apart from the
message because ruff reports that case as an EOF error. It returns a
`BracketError` now.
- `unclosed_replacement_field_error` walked its delimiter stack as
`Vec<(usize, u8, bool)>`, where the third element decided whether the field
had reached its format spec — `.2 = true` and `Some((_, b'{', false))` at the
use sites. `OpenDelimiter` names all three.
The `source_error!` macro is untouched at the 34 call sites it already had and
drops to five lines. Two copies of its body had been written out by hand
(`invalid_number_literal_error`, `unterminated_string_error`) and fold into it,
bringing it to 36.
One thing does change beyond the packaging. `TextRange::new` asserts that its
start does not exceed its end, and it asserts unconditionally. Building the
range at the scanner rather than at the consumer therefore puts that assert in
front of the hundred-odd scanners that reach `NormalizedParseDiagnostic::other`,
which never built a range before — they handed their two offsets straight to
`source_locations`. A scanner that emitted a reversed span used to produce a
garbled location; now it aborts. Every site was read for this and each clamp is
anchored to a bound at or after its start, and neither two hundred thousand
generated sources nor eleven thousand mutations of stdlib files reached it, so
this is a latent invariant made loud rather than a new failure — but it is not
packaging, so it is written down here.
Otherwise a pure refactor: no diagnostic changes message, column or line.
Verified by building the parent commit and diffing its output against this one
over a malformed-source corpus and over every `Lib/**/*.py` that parses clean
(1728 files, each recompiled with a trailing `$` so a scanner that misreads
valid code shows up as a moved line) — byte-identical, and the count of files
where the reported line moves is 1149 on both. test_fstring, test_tstring,
test_string_literals, test_syntax, test_exceptions, test_grammar and
test_compile all pass.
Addresses the review on #8601.
Assisted-by: Claude Code:claude-opus-5
* diagnostics: name the operator-character set behind the debug marker
`is_replacement_field_marker` carried its list of operator characters inline, a
sixteen-line `matches!` wrapped around three lines of logic, and it looked very
much like `is_dangling_operator_byte` a few functions down — enough that a
reviewer asked whether they were the same set.
They are not, and neither contains the other. Written in the same order the
difference is three characters:
precedes_equals_in_one_operator ! % & * + - / : < = > @ ^ |
is_dangling_operator_byte ! % & * + - . / < = > @ ^ | ~
The two are derived from different things. One asks which bytes pair with a
following `=` to make a single token, so that the `=` is part of that operator
rather than the start of a debug specifier — read off Python's token table, and
`.=` and `~=` are not on it, while `:=` is. The other asks which bytes make up
an operator that still wants an operand, which is why `~a` and `a.b.` put `.`
and `~` there and why the separator `:` stays out.
So the list moves next to the one it resembles, under a name, both sorted the
same way, and each says what the other has that it does not. Neither is defined
in terms of the other: the thirteen characters they share are two answers that
happen to coincide, not a set with a meaning of its own, and a common list would
tie each to the other's reasons to change.
No behavior change: the extracted list is character for character what was
inline. Verified against a build of the parent over a malformed-source corpus
and every `Lib/**/*.py` that parses clean — byte-identical — and with
test_fstring, test_tstring, test_string_literals, test_syntax, test_exceptions,
test_grammar and test_compile.
Addresses the review on #8601.
Assisted-by: Claude Code:claude-opus-5
* diagnostics: give the unterminated-literal scan one exit
A single-quoted literal that meets a newline and one that runs off the end of
the source are the same failure, and CPython's lexer says so in one condition:
if (c == EOF || (quote_size == 1 && c == '\n')) {
Everything after that — the check for a replacement field left open, and the
choice of message — is written once there. This scan had grown a second exit
for the newline case with its own copy of both, so the guard added for
interpolated literals had to be added twice and the message assembled twice.
The newline case now breaks out of the scan and falls into the exit that was
already handling end of source. `line` cannot have moved from `start_line` on
that path, and `quote_size` is 1, so the surviving exit computes the same
`detected_line` and the same `triple` the deleted one did.
No behavior change: verified against a build of the parent over a
malformed-source corpus and every `Lib/**/*.py` that parses clean —
byte-identical — and with test_fstring, test_tstring, test_string_literals,
test_syntax, test_exceptions, test_grammar and test_compile.
Assisted-by: Claude Code:claude-opus-5
* diagnostics: classify an unterminated triple-quoted t-string as incomplete input
`analyze_compile_error` pairs the f-string and t-string error variants
everywhere it names a message, but the match that decides whether a failure is
a `SyntaxError` or an `IncompleteInputError` listed only the f-string one. So a
`t'''` typed at the prompt would not have asked for another line where an
`f'''` would.
Would not have, rather than does not: the arm is unreachable today. The
compiler's `unterminated_string_error` scanner claims these first and hands
back an `OtherError`, so `f'''` and a bare `'''` get a `SyntaxError` at the
prompt too, on `main` as much as here — which is why
`test_codeop.test_incomplete` carries an `expectedFailure`. Whoever untangles
that precedence should not then find that f-strings work and t-strings
silently do not.
CPython does not separate the two either: `lexer.c` emits one message
parameterised with `%c` for the prefix and sets `E_EOFS` the same way for both,
and `_is_end_of_source` (`Parser/pegen.c`) looks only at that code, never at
which kind of literal produced it.
No behavior change, and none is testable while the arm cannot be reached.
Assisted-by: Claude Code:claude-opus-5
Follow up to #8591 because I figured out how to deduplicate the code.
This is nicer for future patches that improve WASI support because
Unix-likes and WASI can often share the same dispatch functions.
* Raise the shared argument-binding errors from the slots that count for themselves
Sites all over the tree checked their own argument counts and keywords and
wrote the message by hand, so they drifted: `attrgetter expected 1 argument,
got 0.` carried a full stop, `TypeVar() got unexpected keyword argument(s):
bogus` named neither the keyword the way its counterpart does nor the
function the way `typevar()` is named, and `_csv`'s dialect parser handed a
whole formatted sentence to `InvalidKeywordArgument`, which then wrapped it
in another one.
`Callee::arity_error` and `Callee::unexpected_keyword` are now public, and
these sites raise through them, so there is one place the wording lives.
`FrameLocalsProxy` counts its arguments before reading its keywords, and
`min`/`max` do too, which is the order `framelocalsproxy_new` and `min_max`
check in. `weakref.ref`, `GenericAlias`, `frozenset`, `attrgetter`,
`itemgetter`, `islice`, `start_new_thread`, `TextIOWrapper`, `AttributeError`,
`NameError`, `TypeVar`, `ParamSpec`, `TypeVarTuple` and `TypeAliasType` all
follow.
Of 25 calls measured against CPython 3.14.6, 21 now produce the identical
string. The four left over need what `Callee` cannot carry: `weakref.proxy`
is a function of its own there rather than a type, `frozenset` names the
subclass being constructed, and `TextIOWrapper` names its first parameter
and counts before converting.
Assisted-by: Claude
* Give the argument-binding errors the constructors the rest of them have
Raising one of these meant writing `Callee::of::<Self>(vm).arity_error(1..=3,
0, vm)` — the vm twice, and a type to know about before you could say what
you meant. Every other error in the tree is a `vm.new_*_error`.
So the message texts move out of `Callee` into functions the vm can call, and
the vm gets `new_arity_type_error` and `new_unexpected_keyword_type_error`
alongside `new_unsupported_bin_op_error` and the rest. A slot names itself
with the `NAME` its class already carries:
return Err(vm.new_arity_type_error(Self::NAME, 1..=3, 0));
return Err(vm.new_unexpected_keyword_type_error(Some("typevar"), &key));
The keyword one takes the name as an `Option` because the parser sometimes
has none of its own, which is what `_PyArg_Parser.fname` being NULL means and
what `_csv`'s dialect parser wants.
`bind_for` and `check_kwargs_empty_for` take anything a `Callee` converts
from, so the slots pass `Self::NAME` there too. `Callee` itself is left to the
binding machinery, which is the only place that needs to carry a name around
rather than use one.
No message changes: 22 of 26 calls still match CPython 3.14.6 exactly, and the
43-call set still stands at 23.
Assisted-by: Claude
* common: share the zlib stream engine
Move the VM-independent zlib-rs stream owner and state machine into a feature-gated rustpython-common module. Make rustpython-stdlib adapt that engine instead of carrying its own flate2 implementation, including native stream copy/deepcopy, Z_BLOCK flushes, and wbits=0 decompression.
Assisted-by: Codex:GPT-5
* common: share the lzma stream engine
Move the VM-independent liblzma stream owner, filter properties, and compression state into a target-gated rustpython-common module. Keep rustpython-stdlib as the Python object and exception adapter, and remove its direct xz dependencies and lzma-only generic compressor machinery.
Assisted-by: Codex:GPT-5
* common: address compression engine reviews
Add dependency-free CPU-bound workloads adapted from pyperformance 1.14.0 to cover interpreter, standard-library serialization, regex, numeric, and garbage-collection paths under the existing Criterion and CodSpeed harnesses. Keep inputs deterministic and scale expensive cases for simulation while checking that benchmark execution does not leak interpreter state.
Assisted-by: Codex:gpt-5.6-sol
Related: #8599
My original parser was monolithic and inflexible. It worked as intended
for the derived data files, but anything more complicated required hacky
code. For example, the original parser always expected to build a vector
of values, yet sometimes we required other data structures such as
BTreeMaps.
I split up the parser into helper functions that are both cleaner and
more flexible. The actual parser should still yield the same results -
this doesn't "fix" anything yet. However, this is preliminary work for
fixing more of UCD since we will need a more flexible parser for fixes
like the linked PR.
Preserve logical memoryview descriptors across interpreter transfers, make channel-end release idempotent, and scope runtime-owned interpreter cleanup to the creating top-level runtime. Re-read owned interpreters during shutdown so finalizers cannot leave newly-created children behind.
Add regression coverage for sliced and cast memoryviews, repeated channel release, and multiple embedded runtimes.
Assisted-by: Codex:GPT-5
* Name the function a failed argument binding was binding for
A built-in call that binds badly reported neither the function nor, for a
method, the right numbers: `self` was prepended before binding, so it was
counted as both an expected parameter and a given argument. `[].append()`
read `expected at least 2 arguments, got 1`.
`Callee` carries what a message says about the function: the name, and
whether the leading argument fills the instance parameter. `PyNativeFn`
takes one, `PyNativeFunction` and `PyMethodDescriptor` supply it, and the
four method-descriptor call specializations in `frame.rs` do the same.
`bind_for` subtracts the instance from both numbers.
The messages now read as their counterparts do:
- `_PyArg_CheckPositional`: `find expected at least 1 argument, got 0`,
with the singular when the count is one.
- `_PyArg_UnpackKeywords`: `split() got an unexpected keyword argument
'bogus'`, and `cast() missing required argument 'format' (pos 1)` for a
parameter a call may pass by name. A positional-only parameter is only
ever counted, so it keeps the first form.
A binding that happens where the name isn't known leaves it off, the way
`_PyArg_Parser.fname` is NULL.
`test_find_etc_raise_correct_error_messages` now passes in both
`test_bytes` and `string_tests`.
Assisted-by: Claude
* Name the type whose constructor or initializer bound the arguments
`Constructor::slot_new` and `Initializer::slot_init` bound their arguments
without a name, and so did the slots that override them, so a constructor
that was called wrongly said only `expected 1 argument, got 0`.
Both defaults and every override now name the type the slot was written
for, not the subclass being constructed and not the module `tp_name`
carries: `float expected at most 1 argument, got 2`, and `deque expected
at most 2 arguments, got 3` for a `deque` subclass.
`slice()` and `range()` counted their own arguments and raised messages of
their own; they now raise the one `_PyArg_CheckPositional` raises, which
makes `test_range_constructor_error_messages` pass.
Assisted-by: Claude
Jeong, YunWon