mirror of
https://github.com/RustPython/RustPython.git
synced 2026-10-28 18:59:04 +08:00
dependabot/pre_commit/https-/github.com/astral-sh/ruff-pre-commit-0.17.1
31
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 youknowdot
|
73b6806608 |
Limit single-input parser warning emission to the first logical input (#9304)
Keep later numeric warnings available after parser warning escalation. Preserve compound tokenizer warning emission and legacy malformed-source fallback. Assisted-by: OpenAI Codex:gpt-6-astra |
||
|
youknowdot
|
e50103281d |
Fix single-input multiple-statement diagnostic ranges (#9289)
Preserve comment spans and count a logical newline once when converting single-mode multiple-statement diagnostics to character locations. Assisted-by: OpenAI Codex:model-version-unavailable |
||
|
Jeong, YunWon
|
cdb5cca177 |
Use the Ruff fork 0.17.1 for CPython syntax error messages and the obsolete '<>' hint (#9212)
* Use the RustPython Ruff fork 0.17.1 from git and place EOF errors by kind - The ruff crates come from the RustPython/ruff `rustpython-0.17.1` branch at 93967836c8 instead of the 0.16.10 crates.io release. - `EofToken.known_location` is replaced by `EofErrorLocation` (Tokenizer, Token, StartingFromToken), chosen from the parse error kind. The keyword scan in `clause_keyword` is removed, since the parser now reports a forced `:` as `ExpectedForcedToken`. - An empty import at EOF ends at the end of the line read, and an empty import before a newline is an empty range at the newline. Assisted-by: Claude Code:claude-opus-5-5 * Report the obsolete `<>` from the parser outside Barry mode - Bump the RustPython/ruff fork to 9290a88495, where the parser reports an adjacent `<` and `>` in a comparison as "invalid syntax. Maybe you meant '!=' instead of '<>'?" over both tokens. - Outside Barry mode, `BarrySource` no longer searches the source text for `<>`, so the parser's error is reported unchanged. A `<>` that starts a statement is reported at the `<` only, and `<<>` at the `>`. Assisted-by: Claude Code:claude-opus-5-5 |
||
|
youknowdotandJeong YunWon
|
3f55cf4af0 |
Implement basic REPL execution and fix piped stdin handling (#9163)
* Preserve original main VM through C API execution
Run and finalize the original main VM while keeping the registered attachment template available without holding its mutex across Python callbacks.
Assisted-by: OpenAI Codex:model version unavailable
* Make unraisable reporting robust at hook boundaries
Keep the existing message API and traceback formatting while handling missing, disabled, audited and failing hooks without panicking.
Assisted-by: OpenAI Codex:model version unavailable
* Separate interactive warning collection from compilation
Retain ordinary warning behavior and typed module APIs while exposing record-local parser warnings and post-warning code generation. Add a source-only lazy future-import guard.
Assisted-by: OpenAI Codex:model version unavailable
* Evaluate native stdin with shared VM record semantics
Use one evaluator for basic REPL and root transports, preserve effective inspect snapshots and future flags, and carry owned host exit status through original-main finalization.
Assisted-by: OpenAI Codex:model version unavailable
* Refresh canonical REPL tests and retire fixed failures
Run update_lib migrate from pinned CPython 3.15.0, preserving three remaining platform/runtime markers and retiring the two traceback/startup failures verified on the native evaluator.
Assisted-by: OpenAI Codex:model version unavailable
* Keep the root compiler dependency for development targets
The native evaluator now compiles through the VM. Only root benchmarks and the parse_folder example use rustpython-compiler directly, so declare it as a dev dependency while preserving VM compiler features.
Assisted-by: OpenAI Codex:model version unavailable
* Validate explicit conditional failure marker retirement
Require exact selector and original decorator evidence for each conditional expected-failure marker retired by migrate. Preserve undeclared guards and reject missing, duplicate, modified or unconsumed declarations.
Assisted-by: OpenAI Codex:model version unavailable
* Refresh canonical stdin and exception audit tests
Migrate the pinned CPython 3.15.0 modules and their canonical helper fixtures through update_lib. Retire exactly four failure markers proven obsolete by hosted CI, preserving all other existing guards.
Assisted-by: OpenAI Codex:model version unavailable
* Synchronize runpy warning attribution and document remaining capabilities
Migrate whole canonical runpy.py to pass module names to compile. Use update_lib to annotate only reproduced missing-capability cases: absent _testcapi import-event tests, unsupported startup/import-time features, and the deferred frozen-loader module-attribution caller.
Assisted-by: OpenAI Codex:model version unavailable
* Propagate interrupted native stdin reads to signal dispatch
Preserve the shared input buffer and partial bytes while returning EINTR to the VM before retrying. Exercise the transport deterministically and verify a returning SIGINT handler while pipe stdin stays open.
Assisted-by: OpenAI Codex:model version unavailable
* Reject statements after a compound statement in single mode
single_mode_multiple_statements_error now reports "invalid syntax" at the
statement that follows a compound first statement, and interactive_input
runs it right after parsing as compile() does.
The lazy `from __future__` import check moved from lazy_future_import_error
into unsupported_grammar_error, so it applies to every compile mode.
Assisted-by: Claude Code:claude-opus-5-5
* Remove passing expected failure from test_runpy
test_run_module_filter_syntax_warnings_by_module passes.
Assisted-by: Claude Code:claude-opus-5-5
* Restructure stdin execution after Modules/main.c and pythonrun.c
- src/lib.rs follows pymain_run_python: pymain_run_stdin, pymain_repl,
pymain_run_startup, pymain_run_interactive_hook and pymain_run_pyrepl
return exit codes instead of raising SystemExit.
- VirtualMachine gains run_any_stdin (_PyRun_AnyFile),
run_interactive_loop (_PyRun_InteractiveLoop) and run_simple_stdin;
run_shell feeds rustyline lines into run_interactive_loop, and
sys._baserepl calls run_any_stdin.
- print_uncaught_exception (PyErr_Print) replaces
report_uncaught_exception, and handle_system_exit_and_keyboard_interrupt
replaces handle_reported_exception.
- An unhandled KeyboardInterrupt is recorded and the process exits
through SIGINT after finalization.
- Interpreter::run_main replaces run_with_exit_status and InterpreterExit;
rustpython_capi::run_main_interpreter takes an exit-code callback.
- config_set_inspect audits cpython.PyConfig_Set and rebuilds sys.flags
through set_flag_obj (_PySys_SetFlagObj); the inspect state is an
AtomicBool and dict replace_by_identity is removed.
- add_main_module (PyImport_AddModuleRef("__main__")) is used by the
interactive and simple-file paths.
- The interactive parser warning helpers take their state directly.
Assisted-by: Claude Code:claude-opus-5-5
* Exit through Py_Exit from PyErr_Print and run the basic REPL via stdin
- print_uncaught_exception (PyErr_Print) calls exit_process (Py_Exit) for a
SystemExit when inspect mode is off, instead of returning the exception.
run_interactive_loop no longer checks SystemExit itself and flushes after
printing an error; sys._baserepl returns None.
- exit_process finalizes through VirtualMachine::finalize_ex, which holds
the Py_FinalizeEx steps split out of Interpreter::finalize, and exits
through host_env::os::exit_with_cleanup (C exit()).
- The basic REPL on a terminal (PYTHON_BASIC_REPL, or _pyrepl missing)
runs run_any_stdin instead of the rustyline shell.
- run_shell saves the history after every line.
Assisted-by: Claude Code:claude-opus-5-5
* Display exceptions through PyErr_Display and track the main thread running __main__
- display_exception (PyErr_Display) and display_exception_to (_PyErr_Display)
try traceback._print_exception_bltin before the native printer and flush
the file. PyErr_Print's excepthook fallback, print_exception and
sys.excepthook use them. A missing sys.stderr is reported with
dump_object (PyObject_Dump); a None sys.stderr prints nothing.
- Paths gains program_name (argv[0]) from getpath. The "can't open file"
message uses it with strerror(errno) and goes to sys.stderr.
- running_main holds the ident of the thread running __main__.
pymain_run_python sets it around running the main code, Py_Exit clears it,
the fork child reinitializes it, and _interpreters.is_running reads it
instead of treating the main interpreter as always running.
- Interpreter::run_main clears the unhandled KeyboardInterrupt flag before
running and applies exit_sigint after finalizing; finalize no longer
does.
- run_interactive_loop returns a SystemExit raised outside inspect mode
again and no longer flushes after printing an error; run_shell saves the
history once after the loop.
- pymain_set_inspect reports its error through print_uncaught_exception.
Assisted-by: Claude Code:claude-opus-5-5
* Treat a parse error at the end of the source as incomplete input
A parser error that starts at the end of the source is reported as
_IncompleteInputError when incomplete input is allowed, so the
interactive loop and codeop keep reading after `try:` and its block or
after a decorator line.
Assisted-by: Claude Code:claude-opus-5-5
* Place syntax errors on tokens made up at the end of the input
A parse error on the NEWLINE, DEDENT or ENDMARKER the tokenizer makes up
at the end of the input now carries an EofToken: the number of lines
read and the last line. SyntaxError takes its lineno and text from it,
and its offset is that line's length with end_offset -1, or 0 for both
when the error is raised at the token itself (the generic syntax error
and a missing forced `:` or `(`). Exec input that does not end with a
newline is checked with one appended.
Assisted-by: Claude Code:claude-opus-5-5
* Slice the source with get when checking an indented block error at EOF
Assisted-by: Claude Code:claude-opus-5-5
* Rename the host exits after C exit() and fix wasi and Windows builds
- host_env::os::exit now calls the C exit() on every platform; on Windows
std::process::exit ends in ExitProcess, so it flushes the Rust stdout
and calls libc::exit there. The former exit, which calls _exit(), is
now exit_immediately, used by os._exit, faulthandler and
posixsubprocess.
- errno::strerror_string is available on wasi.
- PyGlobalState::reinit_running_main is built only where its fork caller
is.
Assisted-by: Claude Code:claude-opus-5-5
---------
Co-authored-by: Jeong YunWon <jeong@youknowone.org>
|
||
|
youknowdot
|
9f6f771965 |
Align lazy imports and f/t-string interpolation with Python 3.15 (#9179)
* Preserve interpolation source text and canonical metadata Source-only checkpoint; candidate validation and the existing annotation expectation reconciliation remain pending. Assisted-by: OpenAI Codex:model version unavailable * Align existing annotation expectations with CPython metadata Update only the nine existing expected strings confirmed against pinned CPython 3.15, with explicit user approval. Assisted-by: OpenAI Codex:model version unavailable * Preserve template metadata through compilation and interactive input Escape literal debug prefixes, retain None AST expression metadata internally, and continue incomplete triple-quoted t-strings in the existing shell. Assisted-by: OpenAI Codex:model version unavailable * Classify normalized unclosed brackets as REPL line continuations Assisted-by: OpenAI Codex:model version unavailable * Nest equivalent REPL parse-error alternatives for Clippy Assisted-by: OpenAI Codex:model version unavailable * Remove fulfilled annotation migration lint workaround The canonical expected strings no longer trigger literal_string_with_formatting_args, so the previous lint expectation is obsolete. Assisted-by: OpenAI Codex:model version unavailable * Complete lazy-import exception and traceback library closure Copy canonical traceback and legacy exception mappings with preserved RustPython guards. Copy explicit non-Python library assets by exact path so exception hierarchy data does not expand into the entire test package. Assisted-by: OpenAI Codex:model version unavailable * Provide Python 3.15 traceback metadata for missing modules and attributes Preserve AttributeError context for missing class attribute deletion, matching type_update_dict. Expose canonical generated missing-stdlib messages through the native sysconfig modules without importing CPython build-specific disabled-module assumptions. Assisted-by: OpenAI Codex:model version unavailable |
||
|
|
a403341791 |
Take SyntaxError messages and ranges from the parser (#8983)
* Match CPython's SyntaxError range for unparenthesized `except` types
CPython's `invalid_except_stmt` rule raises the error only once the whole
clause has matched, and reports a range that starts at the first exception
type and ends at the `:` closing the clause, so it covers the `as NAME`
part as well:
invalid_except_stmt:
| 'except' a=expression ',' expressions 'as' NAME ':' {
RAISE_SYNTAX_ERROR_STARTING_FROM(a, "multiple exception types
must be parenthesized when using 'as'") }
The parser reports the exception types alone, so look up that `:` in the
source and widen the range to match. Only `as NAME` may follow the
exception types, which is why the first `:` after them is the one closing
the clause.
try:
pass
except A, B, C as e:
pass
CPython 3.14: ('x.py', 3, 8, 'except A, B, C as e:\n', 3, 20)
before: ('x.py', 3, 8, 'except A, B, C as e:\n', 3, 15)
Fixes #8496
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Clarify that the `except` error range stops before the closing colon
`invalid_except_stmt_end` returns the exclusive end of the range, so the
column it yields is the one the `:` sits on even though the `:` is not part
of the range. The previous "ending at the `:`" wording read as if the colon
were included, which invites an off-by-one "fix".
Comments only; no behavior change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Fix the `except ... as` error column for non-ASCII names and line joins
`invalid_except_stmt_end` recovers the end of CPython's range by scanning the
source, which has to redo by hand what CPython gets from its tokenizer:
- The column was the byte distance from the line start, but `end_offset` is a
character column. `except Ä, B as e:` reported 18 instead of 17. CPython
converts explicitly, in `_PyPegen_byte_offset_to_character_offset`.
- The colon scan already spans explicit line joins, but the `as` check ran on
the raw slice, so a `\` before `as` left the range unextended.
`except A, B \` + `as exc:` reported (3, 12) instead of (4, 7).
Add regression cases to `syntax_invalid.py` covering both, plus the whitespace,
`except*` and line-join placements that already worked. All values are verified
against CPython 3.14.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Update the Ruff fork to 0.16.10
Point the rustpython-ruff_* dependencies at the 0.16.10-rustpython tag
of the RustPython Ruff fork and port to its AST changes:
`ExprCompare` stores all operands in `operands`, call arguments use
`ThinVec`, dict comprehension generators and f-string elements are boxed
slices, `FStringValue` iterates `FStringPartRef`, and `TokenKind::Name`
is now `TokenKind::Identifier`.
Assisted-by: Grok:grok-4.6
Assisted-by: Claude Code:claude-opus-5-5
* Drop the source scan for the `except ... as` error range
The parser now reports the unparenthesized exception types error with a
range ending at the `:` that closes the clause, so
`invalid_except_stmt_end` and the message match that widened the range
are removed.
Assisted-by: Claude Code:claude-opus-5-5
* Take parse error messages from the parser as they are
The Ruff fork now emits the final message texts, so `vm_new.rs` no
longer rewrites parse error messages by matching their text and no
longer lowercases the first character of every message.
`analyze_compile_error` keeps only the caret narrowing for starred
expressions and the line number in the unterminated string message.
The compiler matches the new lowercase texts for indented block and
mixed bytes literal errors.
The fork is referenced by revision until it is published.
Assisted-by: Claude Code:claude-opus-5-5
* Take starred and bytes literal error ranges from the parser
The Ruff fork now reports misplaced starred expressions at the `*` and
non-ASCII bytes literal errors over the whole literal. `vm_new.rs` drops
`narrow_caret` together with `SyntaxErrorInfo`, which only carried the
message after that, and the compiler drops `bytes_literal_span`.
Assisted-by: Claude Code:claude-opus-5-5
* Take indentation error kinds and positions from the parser
The Ruff fork now reports `TabError` and `TooDeepIndentation` as their
own lexer errors and places indentation and line continuation errors
itself. `vm_new.rs` picks TabError or IndentationError from the error
kind instead of scanning the source for mixed indentation, drops the
TabError message override, and sets the end offset of tokenizer errors
to 0 or -1 by kind. The compiler drops the line-end relocation of
unindent errors and the line continuation rewrite. `_tokenize` raises
TabError and depth errors for the token they replace, also for sources
containing tabs.
Remove expectedFailure from test_tokenize.test_max_indent and four
test_tabnanny tests.
Assisted-by: Claude Code:claude-opus-5-5
* Take unterminated string messages from the parser
The Ruff fork's unterminated string errors now include the detected
line and are reported at the start of the string. `vm_new.rs` drops its
message override for `UnclosedStringError`, and the shell continues a
line on an unterminated triple-quoted string by the error's
`triple_quoted` flag instead of reading the quotes from the source.
Assisted-by: Claude Code:claude-opus-5-5
* Take bracket errors and tokenizer error priority from the parser
Bump the ruff fork, which now reports unmatched, mismatched, unclosed
and too deeply nested brackets and decides whether a tokenizer error
replaces the parser's first error.
Remove the bracket, unterminated string and nesting depth source
scanners and `pre_parse_source_error`. A tokenizer error from the
parser now wins over source scanner diagnostics found after it. An
unclosed bracket is incomplete input only when the parser reports it
as such, and other tokenizer errors end where they start.
Remove the expected failure markers from test_unicode_identifiers
test_invalid and the `import ä £` doctest in test_syntax.
Assisted-by: Claude Code:claude-opus-5-5
* Take number literal, string prefix and non-printable errors from the parser
Bump the ruff fork, which now reports malformed number literals,
incompatible string prefixes and non-printable characters itself.
Remove the number literal, string prefix and non-printable character
source scanners and the override classes that ranked them. Leading zero
and string prefix errors keep their range instead of ending where they
start.
Assisted-by: Claude Code:claude-opus-5-5
* Take the missing indented block error from the parser
Bump the ruff fork, which now reports `ExpectedIndentedBlock` with the
clause and the line of its header keyword.
Remove the message rewrite that derived the clause and line from the
source. The VM chooses IndentationError or incomplete input by the
error variant instead of its message.
Assisted-by: Claude Code:claude-opus-5-5
* Take binding target errors from the parser
Bump the ruff fork, which now names the invalid expression in
assignment, augmented assignment, delete, `for`, `with`, import,
pattern and `except` target errors.
Remove the source scanners for those targets and the message rewrites
of `InvalidAssignmentTarget` and `InvalidNamedAssignmentTarget`.
Remove the expected failure markers from test_syntax doctests that now
pass.
Assisted-by: Claude Code:claude-opus-5-5
* Take parameter, argument, star, comma and dictionary errors from the parser
Bump the ruff fork to 36919b62fa. Remove the source scanners for type
parameters, parameter lists, star annotations, call arguments, def type
parameters, parenthesized groups, missing commas and dictionary items.
Set end_offset 0 for ExpectedColonAfterDictionaryKey. Remove 14
EXPECTED_FAILURE markers in test_syntax and test_unpack_ex.
Assisted-by: Claude Code:claude-opus-5-5
* Take expression errors from the parser
Bump the ruff fork to 4f922d2124. Remove the source scanners for
comprehensions, collection and condition assignments, expressions
between strings, named expression targets, missing `in` after for-loop
variables and statements in `if` expressions. Remove the now passing
expected failure markers in test_syntax and test_named_expressions.
Assisted-by: Claude Code:claude-opus-5-5
* Take statement errors from the parser
Bump the ruff fork to c7f6594ed6. Remove the source scanners for
standalone except, `import ... from`, `**_` mapping rests, `elif` after
`else` and mixed except handlers, and the post-parse scanners for call
arguments, match targets, yield after a comma and parenthesized star
imports.
Assisted-by: Claude Code:claude-opus-5-5
* Take string and f-string errors from the parser
Bump the ruff fork to 6b46d0dfc8. Remove the source scanners for
malformed `\N` escapes, f-string and t-string replacement fields, mixed
t-string literals and `print`/`exec` statements, and the override
plumbing they used. Drop the `f'{x'; '` case from the compiler test,
whose expected message differs from CPython 3.14.
Assisted-by: Claude Code:claude-opus-5-5
* Restore the bracket depth check and fix incomplete unclosed strings
Restore `pre_parse_source_error`, which rejects bracket nesting past 200
levels before parsing. The parser reports the same error but still
builds the full tree, which overflows the native stack when dropped.
Remove the VM's unclosed string scanner, which ignored the mode and
treated `'abc` in exec mode as incomplete input. The compiler's check
now skips escaped newlines.
Assisted-by: Claude Code:claude-opus-5-5
* Report multiple statements before later errors in single mode
In single mode, when the first statement is a simple statement, report
code after its line as multiple statements before any error found
later, at the newline token or the comment before it. This replaces the
multiple statements check on the parsed body and also applies to
`ast.parse(mode='single')`.
An unclosed string that follows an earlier syntax error is no longer
incomplete input. The bracket depth error range is now empty.
Assisted-by: Claude Code:claude-opus-5-5
* Update the Ruff fork for parser speed and starred annotations
Bump the ruff fork to 8b59b85344, which skips the tokenizer error pass
when parsing finds no errors, stops allocating for every identifier's
string prefix check, and reports `def f(*args: *b = ...)` as invalid
syntax.
Remove expected failure markers from test_code_module
test_indentation_error and test_sysexcepthook_indentation_error, and
from test_dictcomps test_illegal_assignment.
Assisted-by: Claude Code:claude-opus-5-5
* Use the published Ruff fork 0.16.10 crates
Switch the rustpython-ruff_* dependencies from the git revision to the
0.16.10 crates.io release, published from the same commit (8b59b85344).
Assisted-by: Claude Code:claude-opus-5-5
---------
Co-authored-by: zzarbttoo <qodbwls70@naver.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Sumi Jeong
|
c9c78db91c |
Bound AST nesting so deep source raises instead of crashing (#8827)
* Bound AST nesting so deep source raises instead of crashing Deeply nested source exhausted the native stack and killed the process with SIGSEGV, which no Python-level `except` can catch. The parser grows its own stack when it runs low on one, so it returns trees deeper than the passes after it can walk. Those passes recurse without a limit of their own, and the symbol table, which does apply one, runs last. `compile(..., PyCF_ONLY_AST)` returns before reaching it at all. Apply that same limit to the tree right after it is parsed, and count bracket nesting before parsing rather than after, as CPython's tokenizer does. Checking afterwards built the tree first, and one nested that deep exhausts the stack when it is dropped. Fixes #7655 Assisted-by: Claude Code:Opus 5 * Return Result from too_deeply_nested_error Lets the four call sites use `?` instead of matching on `Some` and returning the error by hand, which drops 11 lines. `Result` is already `#[must_use]`, so the attribute on the function goes away with it. Assisted-by: Claude Code:Opus 5 * Return Result from pre_parse_source_error too Same shape as the previous commit, for the other check this PR adds, so the two are consistent with each other. Assisted-by: Claude Code:Opus 5 * Drop exceeds_max_nesting for the check beside it The escape-warning pass needs the bracket check but had no `SourceFile` to build a `CompileError` from, so it got a `bool` wrapper of its own. Building the `SourceFile` there lets it call `pre_parse_source_error` instead, which is shaped like the other checks in that file, and costs nothing measurable. Assisted-by: Claude Code:Opus 5 |
||
|
Jeong, YunWon
|
7bc3ef6c0a |
ExceptionGroup fields, OSError use-init, and live f_lineno (#8738)
* Add ExceptionGroup fields and OSError use-init BaseExceptionGroup stores msg, excs, and excs_str; args keep the original sequence. OSError defers parse when a subclass overrides only __init__. characters_written can be deleted. Exception args/traceback/cause/context reject invalid setattr/delattr. UnicodeError.object is a deletable field. except* remainder projection copies traceback/context/cause. SyntaxError.__init__ clears omitted location attributes. Assisted-by: Grok:grok-4.6 * Fix OSError payload init and UnicodeError construction new_payload_exception now runs slot_new so BlockingIOError parses characters_written. new_exception falls back to invoke_exception for extra-payload types. Mixed tab/space UnexpectedIndentation is TabError. AttributeError pickle omits obj. Assisted-by: Grok:grok-4.6 * address review: classify TabError from error line Assisted-by: Grok:grok-4.6 * Add new_simple_exception; fill SyntaxError and f_lineno new_exception constructs extra-payload types; BaseException-sized types use new_simple_exception. Non-UTF-8 SyntaxError sets location and text. Codegen errors load text via ProgramText. Keep Missing parentheses capitalization. f-string adjacent atoms report a missing comma. f_lineno may be None; lasti is not restored on unwind. Assisted-by: Grok:grok-4.6 * address review: skip f-string continuation keywords Assisted-by: Grok:grok-4.6 * Fix clippy map_unwrap_or in Non-UTF-8 offset Assisted-by: Grok:grok-4.6 * Rename encode error ctor and exception setters new_unicode_encode_error_real is new_unicode_encode_error. set___traceback__/__cause__/__context__ are pygetset setters; Rust uses set_traceback/set_cause/set_context. Assisted-by: Grok:grok-4.6 * Use set_context in native pickle Assisted-by: Grok:grok-4.6 * Collapse nested if in asyncio traceback restore Assisted-by: Grok:grok-4.6 * Use current instruction for f_lineno addr2line lasti is stored as the next instruction index. addr2line of that byte offset is past RETURN_VALUE, so f_lineno was None and stdlib_threading_current_frames hung on leftover workers. Assisted-by: Grok:grok-4.6 * Look up format args, dir class, and frame builtins Consume printf args before rejecting unknown conversions. object.__dir__ uses getattr of __class__/__dict__. _PyEval_GetBuiltin looks up iter/__import__ on the frame. Assisted-by: Grok:grok-4.6 * Fix builtin lookups after rebase and unxfail passing tests Assisted-by: Grok:grok-4.6 * Restore RERAISE lasti and SyntaxError locations RERAISE with oparg restores lasti from the exception table before unwind, so leftover f_lineno matches the original raise. Handler lookup still uses the raising instruction. Indentation errors outrank print/exec missing-parentheses. BOM-with-non-utf8, bytes literals, and global+nonlocal now report the same offsets as the parser/symtable. object.__dir__ keeps string keys; type.__type_params__ skips the getset wrapper stored on builtin types. Assisted-by: Grok:grok-4.6 * Unxfail syntax/import tests and bind jump locals IMPORT_NAME passes fromlist None instead of an empty tuple. Jumping f_lineno binds None into unbound localsplus slots. str lstrip/rstrip copy subclasses to exact str when unchanged. Assisted-by: Grok:grok-4.6 * Fix import_from callers and rename frame lineno sqlite3.iterdump now passes fromlist as a PyObject. FrameObject::lineno is the i32 helper; f_lineno is the getset. Assisted-by: Grok:grok-4.6 * Pass sqlite fromlist without Into conversion from_list.into() could not infer Into<PyObjectRef> at the import_from call. The typed tuple already converts. Assisted-by: Grok:grok-4.6 |
||
|
Jeong, YunWon
|
3191ecaf38 |
host_env: share thread, locale, loader and BSTR (#8702)
* host_env: share thread, locale, loader and BSTR
Add Windows thread name, locale-info/wcscoll, CompareStringOrdinal,
LoadLibraryEx/GetProcAddress/FreeLibrary, BSTR and COM helpers, and
AF_HYPERV UUID conversion. Wire _thread, locale, ctypes LoadLibrary
and WMI BSTRs through those wrappers. Ensure =X: before spawnve/execve.
Assisted-by: Claude
* host_env: expose rustls-free ssl core on wasm
Compile MemoryBIO, protocol constants, OID tables, hostname
checks, and ALPN parsing without the rustls feature so wasm
and other hosts can implement _ssl on the shared types.
Assisted-by: Claude
* stdlib: add wasm _ssl on rustls-free host_env
Install _ssl on wasm32 against rustpython_host_env::ssl.
MemoryBIO, constants, OID, hostname, and ALPN work. Context
settings are stored; wrap/handshake and rustls-only loaders
raise SSLError until a wasm TLS engine is wired.
Assisted-by: Claude
* host_env: address #8702 review comments
Cache-miss Windows ctypes lookup uses GetProcAddress. FreeLibrary
propagates OSError. start_new_thread returns GetThreadId. Export
_thread._get_name and drop pyfunction doc comments. Reject
embedded NULs in Windows locale.strcoll/strxfrm. Terminate
wcscoll/wcsxfrm and SetThreadDescription inputs.
wasm _ssl accepts wrap_bio owner and get_ca_certs(binary_form),
and MemoryBIO.read(-1) returns all pending bytes.
Assisted-by: Claude
* host_env: gate the whole ssl module on feature ssl
Compile rustpython_host_env::ssl only with the ssl feature.
Rustls crates stay native-only, so enabling ssl on wasm does
not pull rustls. Inside the module, rustls files follow
not(wasm32) rather than a second feature flag.
Assisted-by: Claude
* host_env: fix Windows types and thread names
Cast SysAllocStringLen to *mut u16 and RPC status to u32 so
windows-gnu and MSVC agree. Export _NAME_MAXLEN, truncate names
at NUL and the OS byte cap, and decode get_name with fsdecode.
Assisted-by: Claude
* host_env: compile rustls engine on WASI
Keep rustls-free MemoryBIO/constants/OID/ALPN on all wasm
targets. Compile the rustls engine on WASI, where UnixTime::now
exists. Allow the rustpython ssl feature on wasm without
ssl-rustls or ssl-openssl.
Assisted-by: Claude
* compiler: pick the earliest CPython parse override
A later incompatible-prefix scan must not hide an earlier
number or bracket diagnostic. Parser-driven tokenization
stops at the first failure.
Assisted-by: Claude
* host_env: address remaining #8702 review comments
Map a missing Windows export to a symbol-load error, keep
LoadLibrary's Win32 code on the OSError, drop incomplete
UTF-16 pairs when truncating thread names, hide set_name
and _get_name off linux/macOS/Windows, and let ssl-rustls
enable host_env.
Assisted-by: Claude
* stdlib: add wasm _socket and read-all MemoryBIO
Register a rustls-free-style _socket shim on wasm32 so
Lib/ssl.py can import. Treat MemoryBIO.read(-1) as read-all,
matching _ssl_MemoryBIO_read_impl.
Assisted-by: Claude
* host_env: fix clippy, WASI ssl cfg, and wasm rustls
Defer unclosed-bracket overrides until no token diagnostic
exists, keep WASI rustls and wasm ssl modules exclusive,
preserve lone Windows surrogates, compile the rustls engine
on browser wasm, and clear clippy denials.
Assisted-by: Claude
* host_env: fix Windows clippy and wasm UnixTime
Gate thread-name byte truncation to linux/macOS, allow
GetProcAddress/FreeLibrary pointer deref, and enable
rustls-pki-types web time on browser wasm.
Assisted-by: Claude
* host_env: lock rustls-pki-types web for wasm
Assisted-by: Claude
* host_env: silence Windows unused thread id and shear
Assisted-by: Claude
* stdlib: drop Windows locale clippy needless wraps
Windows clippy failed on locale.strcoll/strxfrm needless return
and _getdefaultlocale always wrapping Ok.
Assisted-by: Claude
* compiler: keep lexer errors over print hints
print x; 0x must report an invalid hex literal. Ranking by source
offset let the earlier print hint win. Treat a finished radix
literal as valid so print x; 0x1 still gets the print message.
Assisted-by: Claude
* compiler: keep decode errors over unclosed openers
Ranking by offset made ( '\\N' report an unclosed paren and
wiped the unclosed-bracket flag on f'{, so compile snippets
raised SyntaxError instead of incomplete input. Export wasm
socket._accept for Lib/socket.py.
Assisted-by: Claude
* wasm: drop duplicate _socket; split locale cfgs
_socket lives in rustpython-stdlib's socket_wasm. The wasm crate
only re-exported that file when freeze-stdlib was off. Split
strcoll, strxfrm, and getencoding into cfg'd function defs.
Assisted-by: Claude
* compiler: keep decode blockers over unclosed openers
(print x; '\\N' must keep the print hint. A later malformed
escape still suppresses the EOF opener even when print wins.
Assisted-by: Claude
* compiler: keep earlier f-string over later lexer errors
f'{x'; ' and f'{x'; 0x must not report the trailing quote or
hex literal. Decode and f-string diagnostics now compete with
lexer failures by offset.
Assisted-by: Claude
* compiler: rank decode vs lexer vs print by class
Unterminated quotes no longer hide format-spec newlines or
empty f-string fields. Print is considered only after the
final winner is not a lexer failure, so print x; f'{x'; 0x
keeps the print hint. Line-continuation uses CPython wording.
Assisted-by: Claude
* compiler: keep unclosed paren over line-cont at EOF
test_exceptions wants '(' was never closed at the opener when
a final backslash follows. Skip escaped quotes in the bracket
scan so {\'a\'} still reports the continuation error. Un-xfail
test_join_daemon_thread_in_finalization.
Assisted-by: Claude
* test_threading: xfail daemon join finalization off Linux
The test passes on Linux. macOS and Windows leave an ignored
deallocator exception on stderr.
Assisted-by: Claude
* vm: hang daemon threads at interpreter finalize
Forced-off daemons must stay not-done so join() during
finalization GC raises PythonFinalizationError and
Thread.is_alive() stays true. SystemExit made the handle
look finished on macOS/Windows. Drop the Lib/test xfail.
Assisted-by: Claude
(cherry picked from commit
|
||
|
Jeong, YunWon
|
98b245f987 |
address review: assignment scan and wasm TLS (#8716)
* address review: assignment scan and wasm TLS Skip comments and lambda defaults in the condition assignment scanner. Restrict the suite-header colon fallback to yield targets. Capture __del__ before calling it. Store browser socket timeouts as f64 and accept integers. Fail closed when wasm verify_mode is not CERT_NONE. Export getpeercert and feed MemoryBIO EOF. Assisted-by: Grok:grok-4.6 * address review: parenthesized yield suite header Walk through grouping parentheses before requiring yield after a suite-header colon. Assisted-by: Grok:grok-4.6 |
||
|
Jeong, YunWon
|
73e547a9c3 |
Honor generator close, throw context, and docs (#8701)
* Honor close unraisable and deallocator messages A failed close lookup on a yield-from target is unraisable. Ignored GeneratorExit from finalize gets a traceback and the closing-generator err_msg. __del__ uses the deallocator message. Assisted-by: Grok:grok-4.6 * address review: claim, ag_running, frame steal Keep the running claim through close() cleanup. ag_running reads running_async. aclose ignore-yield does not mark the generator closed. Steal the frame slot before uniqueness and expose it with try_to_owned. Assisted-by: Grok:grok-4.6 * Honor wrapper docs, yield assign, throw context method-wrapper exposes the slot __doc__. Parenthesized yield assignment uses the invalid-target messages. throw() restores without overwriting __context__, then chains only from the generator's own exc_info slot. Assisted-by: Grok:grok-4.6 * Remove passing expectedFailure markers method-wrapper docs, deallocator messages, and help pager output now match, so drop the leftover markers. Assisted-by: Grok:grok-4.6 * Honor wrapper signatures and assignment diagnostics Slot docs include text signatures. method-wrapper compares and hashes by identity. Keyword genexp and if/while assignment use the invalid-target messages. Assisted-by: Grok:grok-4.6 * address review: aclose, unraisable, frame publish aclose ignore-yield does not close the generator. Format unraisable object is None. close() leaves traceback to the caller; finalize attaches the generator frame. Publish the iframe before the lock-free slot. Assisted-by: Grok:grok-4.6 * Honor member descriptors and wrapper annotations Assisted-by: Grok:grok-4.6 * address review: getitem, bool doc, if assign Assisted-by: Grok:grok-4.6 * Honor eval-mode assignment invalid syntax Assisted-by: Grok:grok-4.6 * Supply getset and member docs from rustpython-doc Assisted-by: Grok:grok-4.6 * Enable rustls SSL on WASI targets Compile the rustls _ssl module for WASI and stub socket waits plus the native certificate store. Assisted-by: Grok:grok-4.6 * Add browser TLS _ssl in rustpython-wasm Register rustls _ssl and a MemoryBIO-oriented _socket from rustpython-wasm. Use rustls-rustcrypto because ring cannot compile for wasm32-unknown-unknown. Assisted-by: Grok:grok-4.6 * Define SSL protocol and cert constants as enums Keep OpenSSL bitmask defines as integer constants. Assisted-by: Grok:grok-4.6 * Ignore rustls deps cargo-shear cannot see The rustpython-wasm _ssl sources are compiled through a symlink, so shear does not count their crate uses. Assisted-by: Grok:grok-4.6 * Replace wasm ssl symlink with a BIO module Drop the stdlib ssl.rs link and keep a rustpython-wasm _ssl that only covers MemoryBIO wrap_bio TLS. Assisted-by: Grok:grok-4.6 |
||
|
Jeong, YunWon
|
a7217a087b |
Classify incomplete input from tokenizer EOF flags (#8703)
* Classify incomplete input from tokenizer EOF flags Treat exec's implicit newline, single-quoted f/t-strings, blank single input, and unclosed field braces the way E_EOF, E_EOFS, and E_EOLS do. Assisted-by: Grok:4.6 * address review: lambda defaults, unicode prefixes, comments Keep lambda parameter defaults and Unicode identifiers continuable, and report comment-only single input as SyntaxError. Assisted-by: Grok:4.6 |
||
|
Jeong, YunWon
|
850fadd388 |
Classify continuable unclosed strings as incomplete (#8694)
* Classify continuable unclosed strings as incomplete Mark EOF-open strings for PyCF_ALLOW_INCOMPLETE_INPUT, treat blank eval input as incomplete, detect unparenthesized lambda after a comma in f-strings, and advance parameter scans past default values. Assisted-by: Grok:4.6 * Diagnose incompatible prefixes and nested f-strings Report incompatible string prefixes, malformed \\N escapes, and f/t-string nesting beyond 150. Omit the extra newline on unterminated triple-quoted f-string SyntaxError.text. Assisted-by: Grok:4.6 * Fix empty defaults and longer incompatible prefixes Report an empty parameter default before later parameters, collect the full prefix before a quote, and prefer the u/b pair. Assisted-by: Grok:4.6 |
||
|
Jeong, YunWon
|
96925f49f8 |
Update Ruff to 0.16.5 (#8617)
Pin the RustPython Ruff fork to the patch commit based on Ruff 0.16.5 and adapt compiler and AST integration to the updated APIs. Assisted-by: Codex:GPT-5 |
||
|
OkJa
|
93f151691c |
interpolated strings: align f-string and t-string diagnostics with CPython (#8601)
* builtins: name the other operand with tp_name in str and Template __add__
`str.__add__` and `Template.__add__` built their TypeError with
`PyType::name()`, which drops the module, where CPython formats `tp_name`:
>>> t"a" + "b"
TypeError: can only concatenate Template (not 'str') to Template
CPython 3.14 reports the qualified name and quotes the operand with double
quotes, as `Objects/unicodeobject.c` and `Objects/templateobject.c` do:
TypeError: can only concatenate string.templatelib.Template
(not "str") to string.templatelib.Template
`Template.__add__` also spelled its own name literally rather than taking it
from `PyClassDef::TP_NAME`, so the two halves of the message could drift apart.
Known gap: `slot_name()` still does not qualify a class defined in a module, so
`"a" + collections.OrderedDict()` names `OrderedDict` where CPython names
`collections.OrderedDict`. This affects both methods.
Unblocks test_template_concatenation in test_tstring.
Assisted-by: Claude Code:claude-opus-5
* interpolated strings: align f-string and t-string diagnostics with CPython
RustPython re-derives CPython's syntax diagnostics by scanning the source rather
than by translating ruff's parse errors, and the scanner shared by f-strings and
t-strings misread several replacement field states:
- A field that ran off the end of the literal (`f'{'`) was reported as an
expression that failed to start, where CPython asks for the closing brace.
- A field whose expression cannot start (`f'{;'`) and one whose expression runs
into a stray character (`f'{a;'`) both fell through to the missing brace
message instead of CPython's two distinct ones.
- A conversion or format spec following `=` was never validated, and a format
spec following a valid conversion was skipped entirely.
- Format specs were scanned with the `{{` escape rule that only applies to
literal text, so nested replacement fields inside them went unchecked.
- Brackets opened inside a field were never matched, so `f'{a[4)}'` and
`f'{3)+(4}'` fell back to a bare "invalid syntax".
- Comments inside a field were not recognised, so a `#` that swallows the
closing brace was reported as an unterminated literal, and a comment in a
multi-line field hid the expression behind it.
- Unterminated literals were named "string" whatever their prefix, and an
interpolated literal left with a replacement field open reported the missing
quote rather than the brace.
A field is only "never closed" while the tokenizer is still reading its
expression. Past the field's own `:` it is emitting literal text again, so
`f'{a:>5` runs out of input as an ordinary unterminated literal while `f'{a`
reports the brace. The field's own `:` is also not the one in a slice, a display
or a lambda, and an unclosed bracket inside the expression is what CPython names
rather than the field around it, so this walk tracks the whole delimiter stack.
Two messages were worded from the wrong branch of CPython's tokenizer. The hint
"perhaps you escaped the end quote?" is raised only where lexer.c handles a
literal without an interpolation prefix; its `%c-string` branch has just the
triple-quoted and plain forms, so pairing the hint with a prefix produced
`unterminated f-string literal (...); perhaps you escaped the end quote?`, which
CPython never emits. And a bracket mismatch inside a field dropped the
` on line %d` clause that lexer.c adds whenever `parenlinenostack[level]` differs
from the current `lineno`; the general bracket scanner in this file already
computed that suffix, so `f"""{a[\n4)}"""` now names line 1 as CPython does.
The rules these now follow are the ones CPython spells out in
Grammar/python.gram (`invalid_fstring_replacement_field` and its t-string twin)
and in Parser/lexer/lexer.c, which likewise parameterises the literal's prefix
rather than duplicating the messages.
Only a field's expression is code. A format spec is text, so a `(` there opens
nothing and a `#` selects the alternate form instead of starting a comment;
inside the expression a `#` does start one. Scanning the whole field for either
therefore reported diagnostics against valid literals, and because these
scanners only run once the source has already failed to parse, that let a good
literal take the blame for an error further down the file. Expression-level
checks now stop at the field's top-level separator and skip comments, and the
literal is walked field by field rather than byte by byte so that a `{` inside a
comment no longer opens one. A pre-existing instance of the same bug in the
line-continuation check is fixed along with them.
Ruff reports a mixed literal concatenation only as a bytes/non-bytes mix, so
`t"x" b"y"` arrived as a bytes error where CPython names the t-string. CPython's
`invalid_string_tstring_concat` is an `invalid_` rule, reached only on the error
pass, and `strings` tries `(fstring|string)+` ahead of it: that alternative
consumes the concatenation's leading run of non-t-string literals, so a mix among
those raises from `_PyPegen_concatenate_strings` on the first pass and sets
`error_indicator`, which suppresses the t-string rule. The t-string message
therefore wins for `t"x" b"y"` and the bytes message keeps precedence for
`"a" b"b" t"c"`. Both are now settled here rather than left to whichever scanner
runs next, which had been answering the second case with an unrelated
"Is this intended to be part of the string?".
A leading doubled `=` or `!` was read as a marker with an empty expression before it,
so `f"{==a}"` reported `valid expression required before '='` where CPython has no
expression to report at all. The same lookahead now guards that branch, and a
doubled `=` or `!` counts as an expression that cannot start.
A top-level `=` or `!` was read as a debug or conversion marker without checking
whether it belonged to a longer operator, so `f"{a==b}"` and `f"{a!=b}"` were also
read as fields ending early. CPython tokenises `==`, `!=`, `<=` and the rest as
single tokens before it ever considers the debug marker, so the separator scan now
skips a `=` that follows one of `= ! < > + - * / % & | ^ @ :` or precedes another
`=`, and a `!` that precedes one.
An expression cannot end on an operator that still wants an operand, and CPython
points at that operator rather than at the brace. `f"{a==}"`, `f"{a and}"` and
`f"{a.b.}"` fell through to a plain `invalid syntax`; they now carry the same
message and column CPython gives, reusing the scan that already handled `;` and
`$` for exactly this shape. `is not` and `not in` are single operators, so the
first word is what gets pointed at, and `...` is consumed in whole triples so
that `f"{....}"` blames the fourth dot rather than the first. The region has to
be walked forwards: a comment's terminating newline is whitespace, so trimming
backwards from the end would step into the comment body and read a triple-quoted
field whose comment ends in `+` as an expression ending in `+`.
That check runs only once the field is known to close, which is where a stray
character and a dangling operator part ways. A stray character is a finished
token, so the parser rejects it on the lookahead in
`annotated_rhs !('='|'!'|':'|'}')` and `f"{a;"` gets the separator message even
with no closing brace. A dangling operator instead makes the parser ask for one
more token, and producing it runs into the literal's closing quote, where
lexer.c answers from its `INSIDE_FSTRING(tok)` branch with
`%c-string: expecting '}'` before any `invalid_` rule is reached. So `f"{a and"`
is a missing brace while `f"{a and}"` names the operator.
A leading `.` was read as a character that cannot start an expression, so the
Ellipsis literal `f"{...}"`, the float `f"{.5}"` and its signed form `f"{-.5}"`
were reported as broken whenever the file failed to parse somewhere else.
`pegen` joins the cpython spelling dictionary, for the `_PyPegen_*` names these
comments cite.
Known gaps, none covered by either test file. Needing the longest valid expression
prefix, which a character scanner cannot compute: a starred tuple (`f"{*a,}"`) and
a dict-unpacking display are read as unable to start an expression, and
`f"{a===b}"`, `f"{a b}"` and `f"{a,,}"` fall through to a bare `invalid syntax`.
The caret still differs from CPython's on `expecting a valid expression after '{'`,
`expecting '}'`, `expecting '}', or format specs` and `unterminated ... literal`,
and on the messages CPython points at a whole token for, such as
`f"{lambda x: x}"` and `f"{x! r}"`. A quote inside a format spec is still treated
as a string delimiter.
Two more are reachable only through a t-string concatenation that ruff does not
report as a bytes mix, so `mixed_tstring_literal_error` never runs: three or more
literals with no bytes literal among them (`t"a" t"b" "c"`, `"a" "b" t"c"`) answer
with `invalid syntax. Is this intended to be part of the string?`, and so does a
tokenizer-level nesting overflow where CPython has
`too many nested f-strings or t-strings`. Two-literal mixes are correct because
ruff's own error carries the wording.
Two pre-existing bugs outside this change are worth naming, since it touches
their neighbourhood. `unterminated triple-quoted ... literal` reports
`detected at line 2` for a one-line source where CPython reports line 1, for every
prefix including none, so it is in the shared line arithmetic rather than the
interpolated path. And `str.__add__`'s message is unreachable for an operand that
defines `__radd__`: `"a" + 1` falls through to the generic
`unsupported operand type(s)` from the binop dispatch, where CPython has
`can only concatenate str (not "int") to str`.
Unblocks test_syntax_errors and test_literal_concatenation in test_tstring, and
test_comments,
test_conversions, test_invalid_syntax_error_message, test_mismatched_braces,
test_mismatched_parens, test_parens_in_expressions and
test_syntax_error_after_debug in test_fstring.
Assisted-by: Claude Code:claude-opus-5
* diagnostics: give the source scanners a diagnostic type
Every scanner in `cpython_parse_diagnostic_override` returned a bare
`Option<(String, usize, usize)>` — 69 signatures and 114 construction sites of
message, start offset, end offset — which the consumer then reassembled into a
message and a range one call later. They return a `CpythonDiagnostic` now, and
the two things that consume one, `NormalizedParseDiagnostic::other` and
`CompileError::from_source_error`, take it whole.
The type earns its name. These scanners run *after* ruff's parse has failed, so
what they produce is not a parse error but a reconstruction of what CPython
would have said about the same source; nothing here ever hands one back to ruff.
Reusing `ruff_python_parser::ParseError` would have fit the shape — it is
`{ ParseErrorType, TextRange }` — but every one of the 114 sites would have
wrapped its message in the single `OtherError` variant of a ninety-variant enum,
widened `other` and `from_source_error` to accept variants no caller
constructs, and, being a foreign type, ruled out the constructor that now
carries the `u32` cast and its justification. The `OtherError` wrapping happens
once per consumer instead, at the boundary where this does cross into ruff's
vocabulary.
Two nameless tuples that carried more than a diagnostic get names, following
`CallArgFrame` and `AssignmentContext` in this file:
- `bracket_syntax_error` returned the message alongside a bare `bool` for
whether the bracket was left open, which the caller needs apart from the
message because ruff reports that case as an EOF error. It returns a
`BracketError` now.
- `unclosed_replacement_field_error` walked its delimiter stack as
`Vec<(usize, u8, bool)>`, where the third element decided whether the field
had reached its format spec — `.2 = true` and `Some((_, b'{', false))` at the
use sites. `OpenDelimiter` names all three.
The `source_error!` macro is untouched at the 34 call sites it already had and
drops to five lines. Two copies of its body had been written out by hand
(`invalid_number_literal_error`, `unterminated_string_error`) and fold into it,
bringing it to 36.
One thing does change beyond the packaging. `TextRange::new` asserts that its
start does not exceed its end, and it asserts unconditionally. Building the
range at the scanner rather than at the consumer therefore puts that assert in
front of the hundred-odd scanners that reach `NormalizedParseDiagnostic::other`,
which never built a range before — they handed their two offsets straight to
`source_locations`. A scanner that emitted a reversed span used to produce a
garbled location; now it aborts. Every site was read for this and each clamp is
anchored to a bound at or after its start, and neither two hundred thousand
generated sources nor eleven thousand mutations of stdlib files reached it, so
this is a latent invariant made loud rather than a new failure — but it is not
packaging, so it is written down here.
Otherwise a pure refactor: no diagnostic changes message, column or line.
Verified by building the parent commit and diffing its output against this one
over a malformed-source corpus and over every `Lib/**/*.py` that parses clean
(1728 files, each recompiled with a trailing `$` so a scanner that misreads
valid code shows up as a moved line) — byte-identical, and the count of files
where the reported line moves is 1149 on both. test_fstring, test_tstring,
test_string_literals, test_syntax, test_exceptions, test_grammar and
test_compile all pass.
Addresses the review on #8601.
Assisted-by: Claude Code:claude-opus-5
* diagnostics: name the operator-character set behind the debug marker
`is_replacement_field_marker` carried its list of operator characters inline, a
sixteen-line `matches!` wrapped around three lines of logic, and it looked very
much like `is_dangling_operator_byte` a few functions down — enough that a
reviewer asked whether they were the same set.
They are not, and neither contains the other. Written in the same order the
difference is three characters:
precedes_equals_in_one_operator ! % & * + - / : < = > @ ^ |
is_dangling_operator_byte ! % & * + - . / < = > @ ^ | ~
The two are derived from different things. One asks which bytes pair with a
following `=` to make a single token, so that the `=` is part of that operator
rather than the start of a debug specifier — read off Python's token table, and
`.=` and `~=` are not on it, while `:=` is. The other asks which bytes make up
an operator that still wants an operand, which is why `~a` and `a.b.` put `.`
and `~` there and why the separator `:` stays out.
So the list moves next to the one it resembles, under a name, both sorted the
same way, and each says what the other has that it does not. Neither is defined
in terms of the other: the thirteen characters they share are two answers that
happen to coincide, not a set with a meaning of its own, and a common list would
tie each to the other's reasons to change.
No behavior change: the extracted list is character for character what was
inline. Verified against a build of the parent over a malformed-source corpus
and every `Lib/**/*.py` that parses clean — byte-identical — and with
test_fstring, test_tstring, test_string_literals, test_syntax, test_exceptions,
test_grammar and test_compile.
Addresses the review on #8601.
Assisted-by: Claude Code:claude-opus-5
* diagnostics: give the unterminated-literal scan one exit
A single-quoted literal that meets a newline and one that runs off the end of
the source are the same failure, and CPython's lexer says so in one condition:
if (c == EOF || (quote_size == 1 && c == '\n')) {
Everything after that — the check for a replacement field left open, and the
choice of message — is written once there. This scan had grown a second exit
for the newline case with its own copy of both, so the guard added for
interpolated literals had to be added twice and the message assembled twice.
The newline case now breaks out of the scan and falls into the exit that was
already handling end of source. `line` cannot have moved from `start_line` on
that path, and `quote_size` is 1, so the surviving exit computes the same
`detected_line` and the same `triple` the deleted one did.
No behavior change: verified against a build of the parent over a
malformed-source corpus and every `Lib/**/*.py` that parses clean —
byte-identical — and with test_fstring, test_tstring, test_string_literals,
test_syntax, test_exceptions, test_grammar and test_compile.
Assisted-by: Claude Code:claude-opus-5
* diagnostics: classify an unterminated triple-quoted t-string as incomplete input
`analyze_compile_error` pairs the f-string and t-string error variants
everywhere it names a message, but the match that decides whether a failure is
a `SyntaxError` or an `IncompleteInputError` listed only the f-string one. So a
`t'''` typed at the prompt would not have asked for another line where an
`f'''` would.
Would not have, rather than does not: the arm is unreachable today. The
compiler's `unterminated_string_error` scanner claims these first and hands
back an `OtherError`, so `f'''` and a bare `'''` get a `SyntaxError` at the
prompt too, on `main` as much as here — which is why
`test_codeop.test_incomplete` carries an `expectedFailure`. Whoever untangles
that precedence should not then find that f-strings work and t-strings
silently do not.
CPython does not separate the two either: `lexer.c` emits one message
parameterised with `%c` for the prefix and sets `E_EOFS` the same way for both,
and `_is_end_of_source` (`Parser/pegen.c`) looks only at that code, never at
which kind of literal produced it.
No behavior change, and none is testable while the arm cannot be reached.
Assisted-by: Claude Code:claude-opus-5
|
||
|
Jeong, YunWon
|
b4e282a6b9 |
compiler: end the missing-comma diagnostic past the whole second atom (#8594)
`missing_comma_expression_error` returned its span as `(index, next + 1)`,
where `next` is the byte index at which the second atom starts. The `+ 1`
is a fixed one-byte width, so the reported `end_offset` matched CPython
only when that atom was a single ASCII character:
(a b) end 5, CPython 5
(a bb) end 5, CPython 6
(a bbb) end 5, CPython 7
(a β) end 4, CPython 5
`adjacent_atom_end` already measures an atom -- identifiers including
non-ASCII ones, string literals and numbers -- and is what the first atom
is measured with three lines above. Use it for the second one too.
Assisted-by: Claude
|
||
|
Jeong, YunWon
|
f87cfe7e7a |
Match CPython on code objects, ast, symtable and syntax error positions (#8580)
* Guard all/any/tuple(genexpr) calls when the generator is a coroutine
`maybe_optimize_function_call()` reserves a `skip_optimization` label for
every `name(genexpr)` call and, for `all`, `any` and `tuple`, emits an
identity guard against the builtin plus an inlined loop. The port skipped
both whenever the generator expression's symbol table was a coroutine, so
`all(await x for x in xs)` and `all(x async for x in xs)` compiled to a
plain call.
An `await` or an `async for` in the generator does not disqualify the
shape: the inlined `FOR_ITER` raises the same `TypeError` that calling the
builtin on an async generator raises, which is what
test_builtin test_builtin_call_async_genexpr_no_crash asserts.
This was the last code-level difference against CPython across `Lib/`:
comparing every file's code tree by structure, opcode and constant value
(qualnames aside, which moved in 3.14.3+, and set constants compared by
value rather than by hash order) now matches on 1720 of 1720 comparable
files, up from 1719.
Assisted-by: Claude Code:claude-opus-5
* Name the ellipsis type `ellipsis`
`type(...).__name__`, its `__qualname__`, and `types.EllipsisType.__name__`
read `EllipsisType`.
Assisted-by: Claude Code:claude-opus-5
* Align compiler front end and instruction positions with CPython
Reject a bare generator expression in a class header and a format spec
nested more than two deep. Both parse here; the reference grammar has no
rule for either, since a class header only takes `arguments` and the
tokenizer runs out of nesting levels at the third format spec.
Scan the interpolations a format spec carries inside another format spec,
so `f"{x:{y:{z}}}"` reaches `z` when building the symbol table.
Give every generic function's type params scope a `.defaults` slot,
whether or not the signature has any default, so `co_varnames` and
`co_nlocals` match. Classes and type aliases keep theirs empty.
Decide an unparenthesized sole generator argument by the parser's
`parenthesized` flag instead of scanning the source for the shape of its
element. `_sum((d := x - c) * d for x in data)` opens with a parenthesized
group the rest of the expression continues, which the scan read as the
generator's own parentheses and left the call's out of its range.
Drop empty f-string fragments unconditionally. A concatenation whose
fragments are all empty now loads its empty string at the whole
concatenation instead of at whichever fragments happened to be plain
string literals.
Keep a byte order mark in source handed over as text, and report it as an
invalid non-printable character. Reading a file to freeze strips it, as
the encoded-bytes path already did.
Carry the filename into the errors `compile(..., PyCF_ONLY_AST)` raises;
they reported `<unknown>` or an empty name.
Across `Lib/`, the 1729 files that compile on both sides now agree on
opcodes, constants, flags, variable names, names, and exception tables,
and on instruction positions in all but 7 files, where a compound
statement whose body ends in a trailing `;` ends one column short.
Assisted-by: Claude Code:claude-opus-5
* Align ast module output with CPython
- Keep lone surrogates in string, f-string and t-string literal values by
reading such a literal from its source; the literal-value helpers move
out of `Compiler` into `rustpython_codegen` and `ConstantLiteral::Str`
holds `Wtf8Buf`.
- Give `{expr=}` interpolations their leading text as a Constant and
default their conversion to `repr`, in both f-strings and t-strings.
- Span folded consecutive literals from the first of them to the last.
- Take the `:` into a format spec's range only when the parser left it
out, instead of always widening by one.
- Report a node end that lands on a line start as that line, column 0.
- Give a generator expression written straight into a call's parentheses
the range of those parentheses, in the preprocess pass, replacing
codegen's source scan and `_ast`'s one-column adjustment.
- Drop the expectedFailure on test_ast_line_numbers_with_parentheses.
Assisted-by: Claude Code:claude-opus-5
* Align symbol table output with CPython
- Register an import as `DEF_IMPORT` alone, and an annotated parameter as
a plain parameter, so `import x; global x` is accepted and annotated
parameters are not reported as annotated names.
- Record a `global` declaration in the module block as well, and resolve a
name a `global` in an enclosing scope covers as an implicit global, so a
class body reading such a name emits `LOAD_NAME`.
- Resolve a name bound nowhere as an implicit global instead of leaving
its scope unknown.
- Require a binding only for a name an explicit `nonlocal` declared;
a free variable that reached the scope any other way already resolved.
- Name a generator expression's block `genexpr`; the code object keeps
`<genexpr>`.
- Locate a `def`/`class` block at its keyword rather than at the first
decorator. `decorated_definition_range()` moves from `Compiler` to
`rustpython_codegen` for that.
- Carry a comprehension symbol's flags over when inlining it into a parent
entry that only holds a free variable propagated from a child.
- Leave `__conditional_annotations__` out of an annotation block's
symbols and have the compiler cook up the free variable, which places
it after the ones the symbol table supplies.
- Check `from __future__` placement while building the symbol table, and
give `symtable.symtable()` the same docstring stripping and future
feature validation the compiler does.
- Floor `co_stacksize` at 1.
Assisted-by: Claude Code:claude-opus-5
* Locate the eval return and the interactive print like CPython
Leave `eval` mode's `RETURN_VALUE` without a location, so the exit block
is duplicated per reaching path and each copy carries the location of the
branch that jumps to it. `a and b` now ends with two returns as it does
in CPython, and `a if b else c` gives the first one the `a` it returns.
Emit an interactive `CALL_INTRINSIC_1(Print)` at the expression statement
rather than wherever the expression left the location, which differ for
`(a := 10)` and `f'{0:fz}'`.
Assisted-by: Claude Code:claude-opus-5
* Give interactive mode the conditional annotations cell
`compile_program_single()` emitted the `__conditional_annotations__` set
and its name accesses but never added the cell that holds it, so a module
compiled in `single` mode was missing the `MAKE_CELL` and the cellvar that
the same source gets in `exec` mode.
Assisted-by: Claude Code:claude-opus-5
* Give compiler-raised SyntaxErrors an end position
`CodegenError` and `SymbolTableError` now carry an `end_location`
alongside `location`, and `CompileError::python_end_location()` returns
it, so errors such as "'await' outside function" set `end_lineno` and
`end_offset` instead of leaving them None.
`SymbolTableBuilder::error_ranged()` builds both positions from a range,
replacing the inline `source_location()` calls at the error sites.
Drop the `expectedFailure` on
`test_exceptions.SyntaxErrorTests.test_file_source`, which now passes.
Assisted-by: Claude Code:claude-opus-5
* Report parser diagnostics with CPython's wording
`InvalidAnnotatedAssignmentTarget` becomes "illegal target for
annotation", `UnexpectedExpressionToken` becomes plain "invalid syntax"
rather than prefixing the parser's text, and the list-recovery
diagnostics ("Expected an expression or a '}'" and its siblings) join the
two that already mapped to "invalid syntax".
Assisted-by: Claude Code:claude-opus-5
* Punctuate the code object repr like CPython
`repr(code)` was missing the comma after the address and escaped the
filename through Rust's Debug formatting.
Drop the `expectedFailure` on the six `test_dis` tests that compare
disassembly text, which now pass.
Assisted-by: Claude Code:claude-opus-5
* Match CPython's parser error positions
A syntax error the parser reports counts its columns in characters, so
`source_location()` converts with UTF-32 rather than UTF-8; an offset
inside a character walks back to where that character starts.
Spans the parser reported differently:
- an error between two tokens covered nothing, and now covers one
character, since the narrowest token still covers one
- a "was never closed" bracket ends at column 0
- an unterminated string literal ends where it starts
- "leading zeros in decimal integer literals" spans the run of zeros
- an exponent with no digits and no sign is blamed on the digit before
the exponent letter, which the tokenizer puts back
Assisted-by: Claude Code:claude-opus-5
* codegen: seek to the finally body's own scopes before copying it
An early exit from a try block emits an extra copy of the finally body.
Nested scopes are handed out by position, and the cursors still sit inside
the try block's own run of scopes there, so the copy took those instead of
the ones the finally body opens: a generator expression got a sibling's
symbol table and raised "the symbol 'k' must be present in the symbol
table", or, where the sibling was a named scope, built a code object with
no `.0` argument that raised UnboundLocalError when it ran.
Seek the cursors to the first scope beginning on or after the finally
body's first line before compiling the copy. The existing restore then
leaves the try statement's own copies starting from the same place.
Assisted-by: Claude
|
||
|
Jeong, YunWon
|
a7981dccc7 |
Align compiler futures, annotations, and symtable with CPython (#8550)
* Align compiler futures and annotations with CPython Handle Barry parsing consistently across AST, type-comment, REPL, and WASM paths. Preserve deferred annotation source and scope metadata, and make _symtable conversion cached and linear-time. Assisted-by: Codex:gpt-5.4 * Fix f-string debug ranges and update Barry/annotate test expectations Compute the debug f-string source range from the raw leading and trailing text instead of the comment-stripped text, so a comment inside a multi-line replacement field no longer shifts the emitted LOAD_CONST location. Restrict the obsolete-operator location shift to `<>` by also requiring a `>` at the reported offset; without it any `ExpectedExpression` that followed a `<` (`2 <;`) was reported one column to the left. Add `CodeFlags::FUTURE_MASK` and use it in builtins, the shell, and the WASM VM instead of repeating the eight-flag union, and collapse the duplicated Barry diagnostic sequences into `BarrySource::diagnostic`. Update the `__annotate__` disassembly snapshot to LOAD_FAST_BORROW and replace the RustPython-only `barry_as_FLUFL` co_flags assertion in extra_tests with the CPython-matching one. Assisted-by: Claude Code:claude-opus-5 * Restore the annotated function in __annotate__ qualnames Signature annotation scopes are compiled in the scope enclosing the annotated function, so the function name has to be folded back into the qualname: `f.__annotate__`, `C.m.__annotate__`, `outer.<locals>.inner.__annotate__`. test_type_annotations test_annotate_qualname covers this. test_pyrepl test_future_barry_as_flufl now passes, so drop its expectedFailure marker. Assisted-by: Claude Code:claude-opus-5 * Pass safepoints in the attached-thread interpreter tests `main_and_subinterpreter_run_sections_overlap` parked its workers on a condvar while attached, and `busy_main_interpreter_does_not_block_subinterpreter` looped over protocol calls that never reach a safepoint. An attached thread that never reaches a safepoint cannot be suspended, so a concurrent process-wide stop-the-world (from the gc test running in parallel) never completes and the already-stopped sibling interpreter stays stopped; the second worker then waits in `wait_while_suspended` until the 30s deadline. Both loops now call `vm.check_signals()` each iteration, and the condvar wait uses `wait_timeout` so the lock is released between safepoints. Reproduced on Linux (4 CPUs): the two tests with `--test-threads=2` failed 2/40 before and 0/40 after; the full `rustpython-vm` lib binary failed 1/30 before and 0/30 after. Assisted-by: Claude Code:claude-opus-5 * Report the obsolete `<>` operator from one place `<>` was located twice: `BarrySource` recorded the token ranges it rewrote to `!=`, and `cpython_parse_diagnostic_override` separately peeked at the bytes around an `ExpectedExpression` location. Both produced "invalid syntax" spanning the whole operator. `prepare_barry_as_flufl_source` now records the operator ranges in the non-Barry case too -- by plain text search, since nothing is rewritten there -- so `BarrySource::invalid_legacy_operator` covers both, and the byte-peeking helper is gone. It also defers to an unclosed bracket earlier in the source, which the diagnostic chain it used to sit in did for it. A `<>` that starts a statement now spans both characters: the parser stops at the `<` rather than one character in, which the byte peek did not handle. Assisted-by: Claude Code:claude-opus-5 * Drop a py object's payload before its class `PyInner` declares `typ` before `payload`, so `drop_in_place` dropped the class first and `PyAtomicRef::drop` left it null. A `PyWeak` payload is still linked into the weakref list of the object it points at until its own `Drop` unlinks it under the stripe lock, which happens after that. `WeakRefList::add` walks the same list looking for a proxy to reuse and reads `node.class()` off every node it passes, so it could dereference the null class of a node on its way out. `PyInner::dealloc` now runs the two destructors in the other order at all three of its exit paths, with a compile-time check that no other field has one. Reproduced by the new test in about 0.1s, as the panic CI hit in the c-api `weakrefobject::tests::new_proxy` run: panicked at crates/vm/src/object/ext.rs:312: unsafe precondition(s) violated: hint::unreachable_unchecked must never be reached Assisted-by: Claude Code:claude-opus-5 |
||
|
mumallaeng
|
b310b6bf6d |
Collapse bare ExpectedExpression to 'invalid syntax'; fix '<>' diagnostic offset (#8540)
* Collapse bare ExpectedExpression to 'invalid syntax'; fix '<>' diagnostic offset `ParseErrorType::ExpectedExpression` currently surfaces as the raw ruff parser message (e.g. "Expected an expression") to callers that only depend on `rustpython-compiler` (no `rustpython-vm`). `rustpython-vm`'s `vm_new.rs` already collapses this to CPython's generic "invalid syntax" for its own callers; mirror that same collapse inside `cpython_parse_diagnostic_override` so non-vm consumers get the same CPython-compatible message. A bare `<>` outside Barry-as-BDFL mode (`2 <> 3`) lexes as `Less` then an unexpected `Greater`, so the resulting `ExpectedExpression` location points at the `>` -- one character past where CPython's tokenizer (which treats `<>` as a single obsolete token) reports the error. Detect the `<` immediately preceding the location and shift the reported range back over it. Assisted-by: Claude Code:claude-sonnet-5 * Require an adjacent '>' before treating '<' as the obsolete <> operator CodeRabbit review on #8540: the previous check only looked at the byte before the ExpectedExpression location for '<', without confirming an adjacent '>' really follows it. For inputs like a trailing '<' at EOF this could misclassify an unrelated ExpectedExpression as the bare '<>' case and report a bogus range. Require both bytes are present before constructing the diagnostic. Also remove the now-stale `@unittest.expectedFailure # TODO: RUSTPYTHON` markers on test_guido_as_bdfl and test_barry_as_bdfl_relative_import, which pass with this fix (test_barry_as_bdfl and test_barry_as_bdfl_look_ma_with_no_compiler_flags still need real Barry-as-BDFL tokenizer support and stay marked). Assisted-by: Claude Code:claude-sonnet-5 |
||
|
chestnut1717
|
d6161ac321 |
Fix EOF SyntaxError diagnostics (#8429)
* Fix EOF SyntaxError diagnostics * Fix Scope EOF SyntaxError location conversion * Fix Additional Unexpected Test * Docs Add Annotation source_location_in_code_points() |
||
|
이광효
|
3b9c64b151 |
ast: enforce integer string limits in AST-only parsing (#8454)
Validate decimal integer literals from the parser token stream before lowering the parsed AST. Reuse the validator for normal compilation and the `_ast` parsing path so `PyCF_ONLY_AST`, `ast.parse()`, and `ast.literal_eval()` honor `int_max_str_digits` consistently. Remove the now-passing `test_literal_eval_str_int_limit` expected-failure marker. The token-based validator excludes non-decimal integers, floats, complex literals, strings, and comments without duplicating lexer logic. Assisted-by: Codex:gpt-5.6-sol |
||
|
Jeong, YunWon
|
a1ba6a662c |
Align compiler and AST behavior with CPython (#8138)
* Align compiler and AST behavior with CPython
Update the compiler pipeline, AST module support, and related VM compile/eval paths toward CPython 3.14 behavior.
Use upstream Ruff parser crates directly and keep RustPython-specific syntax preflight handling in the compile path.
Refresh opcode metadata, snapshots, and targeted tests for the aligned bytecode and AST behavior.
* fix
* Fix PR 8138 CI failures
* Fix match mapping wildcard scan complexity
* fixes
* Use Ruff runtime AST fields
* Minimize residual patch diff
* refactor error handling
* Remove _ast conversion context wrappers
* Clean up _ast conversion glue and compiler warning escalation
Rename the Node conversion parameter `ctx` to `vm` throughout the _ast
module.
Remove the unused compile_program, compile_program_single,
compile_block_expression, and compile_expression forwarders; compile_top
stays as the entry point.
Stash an escalated compiler SyntaxWarning so a non-SyntaxWarning category
propagates unchanged instead of being rewritten to SyntaxError, matching
PyErr_ExceptionMatches(SyntaxWarning) in compiler_warn. Drop the now-unused
CompileWarningError::into_codegen_error.
Replace the PositionalArguments two-variant enum with a struct holding a
shared range and a PositionalArgumentsKind enum.
Assisted-by: Claude
* Represent compiler flags as a CompilerFlags bitflags
Replace the loose PY_CF_*/CO_* i32 constants in compile_mode with a
CompilerFlags bitflags covering the PyCF_* flags and __future__
CO_FUTURE_* bits. The Python-visible ast.PyCF_* attribute values are
derived from it via bits(); parser input modes stay plain consts.
builtins::compile and the compile_string flag handling test flags
through CompilerFlags::contains instead of raw bit math.
Assisted-by: Claude
* Fix PR 8138 test regressions
* Fix PR 8138 CI failures
* Avoid shell interpolation in Linux deps action
* Fix review findings in compiler/AST changes
- frame.rs: compute the MATCH_CLASS class name lazily so a successful
match no longer takes the type name lock or allocates; the
__match_args__-not-a-tuple branch reuses it instead of recomputing.
- _ast/constant.rs: add the missing leading space to the with_recursion
label ("maximum recursion depth exceeded during compilation").
- sys.set_int_max_str_digits: restore the "maxdigits must be 0 or larger
than {n}" error text.
- codegen: set CO_MAXBLOCKS to 20 in both ir.rs and compile.rs.
- _ast/exception.rs: extract the duplicated excepthandler node guard
into ensure_excepthandler_node.
- codegen: correct the validate_keywords doc comment.
- Drop redundant "CPython" qualifiers from comments added in this branch.
Assisted-by: Claude
|
||
|
Shahar Naveh
|
fbdf7c1052 |
Align more error messages with CPython 3.14.5 (#7988)
* Cannot assign to True/False/None * cannot use assignment expressions with ... * cannot assign to function call/expression |
||
|
Shahar Naveh
|
9a0291188c | Clippy rules (test related) (#7968) | ||
|
Shahar Naveh
|
acf12468c4 | Clippy nursery lints (#7875) | ||
|
Shahar Naveh
|
54be99bbc9 |
Clippy warn uninlined_format_args & redundant_else (#7873)
|
||
|
Shahar Naveh
|
323a45ef07 | Enable some pedantic clippy lints (#7764) | ||
|
Jeong, YunWonandCPython Developers
|
3df178298a |
Fix traceback, syntax errors, and exception handling (#7015)
* Update codeop from v3.14.3
* Fix traceback, syntax errors, and exception handling
- Improve unclosed bracket detection with "'(' was never closed" message
- Fix IndentationError location to point to end of line
- Implement frame.clear() with proper checks for executing/suspended frames
- Fix exception context chaining for propagated exceptions
- Add traceback.__dir__() and prevent tb_next deletion
- Fix subscript operation source range restoration in compiler
- Change "duplicate parameter" to "duplicate argument" error message
- Refactor duplicate code in asyncgenerator.rs and frame.rs
---------
Co-authored-by: CPython Developers <>
Assisted-by: Codex:gpt-6
|
||
|
Jeong, YunWon
|
bdf3b3654e | fix syntaxerror ast end_offset (#6646) | ||
|
Terry Tianlin Luan
|
1464d5ca43 |
Adding + Fixing Clippy rules to better align with #[no_std] (#6570)
* * Added alloc_instead_of_core, std_instead_of_alloc, and std_instead_of_core clippy rules * Manually changed part of the code to use core/alloc * use clippy --fix to fix issues in stdlib * * Used clippy --fix to fix issues in vm * Imported Range in vm/src/anystr.rs * * Used clippy --fix to fix issues in common |
||
|
Shahar Naveh
|
041dd30602 |
Move compiler/src -> crates/compiler (#6270)
|