Files
RustPython/extra_tests/snippets/syntax_invalid.py
a403341791 Take SyntaxError messages and ranges from the parser (#8983)
* Match CPython's SyntaxError range for unparenthesized `except` types

CPython's `invalid_except_stmt` rule raises the error only once the whole
clause has matched, and reports a range that starts at the first exception
type and ends at the `:` closing the clause, so it covers the `as NAME`
part as well:

    invalid_except_stmt:
        | 'except' a=expression ',' expressions 'as' NAME  ':' {
            RAISE_SYNTAX_ERROR_STARTING_FROM(a, "multiple exception types
            must be parenthesized when using 'as'") }

The parser reports the exception types alone, so look up that `:` in the
source and widen the range to match. Only `as NAME` may follow the
exception types, which is why the first `:` after them is the one closing
the clause.

    try:
        pass
    except A, B, C as e:
        pass

    CPython 3.14:  ('x.py', 3, 8, 'except A, B, C as e:\n', 3, 20)
    before:        ('x.py', 3, 8, 'except A, B, C as e:\n', 3, 15)

Fixes #8496

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* Clarify that the `except` error range stops before the closing colon

`invalid_except_stmt_end` returns the exclusive end of the range, so the
column it yields is the one the `:` sits on even though the `:` is not part
of the range. The previous "ending at the `:`" wording read as if the colon
were included, which invites an off-by-one "fix".

Comments only; no behavior change.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Fix the `except ... as` error column for non-ASCII names and line joins

`invalid_except_stmt_end` recovers the end of CPython's range by scanning the
source, which has to redo by hand what CPython gets from its tokenizer:

- The column was the byte distance from the line start, but `end_offset` is a
  character column. `except Ä, B as e:` reported 18 instead of 17. CPython
  converts explicitly, in `_PyPegen_byte_offset_to_character_offset`.
- The colon scan already spans explicit line joins, but the `as` check ran on
  the raw slice, so a `\` before `as` left the range unextended.
  `except A, B \` + `as exc:` reported (3, 12) instead of (4, 7).

Add regression cases to `syntax_invalid.py` covering both, plus the whitespace,
`except*` and line-join placements that already worked. All values are verified
against CPython 3.14.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Update the Ruff fork to 0.16.10

Point the rustpython-ruff_* dependencies at the 0.16.10-rustpython tag
of the RustPython Ruff fork and port to its AST changes:
`ExprCompare` stores all operands in `operands`, call arguments use
`ThinVec`, dict comprehension generators and f-string elements are boxed
slices, `FStringValue` iterates `FStringPartRef`, and `TokenKind::Name`
is now `TokenKind::Identifier`.

Assisted-by: Grok:grok-4.6
Assisted-by: Claude Code:claude-opus-5-5

* Drop the source scan for the `except ... as` error range

The parser now reports the unparenthesized exception types error with a
range ending at the `:` that closes the clause, so
`invalid_except_stmt_end` and the message match that widened the range
are removed.

Assisted-by: Claude Code:claude-opus-5-5

* Take parse error messages from the parser as they are

The Ruff fork now emits the final message texts, so `vm_new.rs` no
longer rewrites parse error messages by matching their text and no
longer lowercases the first character of every message.
`analyze_compile_error` keeps only the caret narrowing for starred
expressions and the line number in the unterminated string message.
The compiler matches the new lowercase texts for indented block and
mixed bytes literal errors.

The fork is referenced by revision until it is published.

Assisted-by: Claude Code:claude-opus-5-5

* Take starred and bytes literal error ranges from the parser

The Ruff fork now reports misplaced starred expressions at the `*` and
non-ASCII bytes literal errors over the whole literal. `vm_new.rs` drops
`narrow_caret` together with `SyntaxErrorInfo`, which only carried the
message after that, and the compiler drops `bytes_literal_span`.

Assisted-by: Claude Code:claude-opus-5-5

* Take indentation error kinds and positions from the parser

The Ruff fork now reports `TabError` and `TooDeepIndentation` as their
own lexer errors and places indentation and line continuation errors
itself. `vm_new.rs` picks TabError or IndentationError from the error
kind instead of scanning the source for mixed indentation, drops the
TabError message override, and sets the end offset of tokenizer errors
to 0 or -1 by kind. The compiler drops the line-end relocation of
unindent errors and the line continuation rewrite. `_tokenize` raises
TabError and depth errors for the token they replace, also for sources
containing tabs.

Remove expectedFailure from test_tokenize.test_max_indent and four
test_tabnanny tests.

Assisted-by: Claude Code:claude-opus-5-5

* Take unterminated string messages from the parser

The Ruff fork's unterminated string errors now include the detected
line and are reported at the start of the string. `vm_new.rs` drops its
message override for `UnclosedStringError`, and the shell continues a
line on an unterminated triple-quoted string by the error's
`triple_quoted` flag instead of reading the quotes from the source.

Assisted-by: Claude Code:claude-opus-5-5

* Take bracket errors and tokenizer error priority from the parser

Bump the ruff fork, which now reports unmatched, mismatched, unclosed
and too deeply nested brackets and decides whether a tokenizer error
replaces the parser's first error.

Remove the bracket, unterminated string and nesting depth source
scanners and `pre_parse_source_error`. A tokenizer error from the
parser now wins over source scanner diagnostics found after it. An
unclosed bracket is incomplete input only when the parser reports it
as such, and other tokenizer errors end where they start.

Remove the expected failure markers from test_unicode_identifiers
test_invalid and the `import ä £` doctest in test_syntax.

Assisted-by: Claude Code:claude-opus-5-5

* Take number literal, string prefix and non-printable errors from the parser

Bump the ruff fork, which now reports malformed number literals,
incompatible string prefixes and non-printable characters itself.

Remove the number literal, string prefix and non-printable character
source scanners and the override classes that ranked them. Leading zero
and string prefix errors keep their range instead of ending where they
start.

Assisted-by: Claude Code:claude-opus-5-5

* Take the missing indented block error from the parser

Bump the ruff fork, which now reports `ExpectedIndentedBlock` with the
clause and the line of its header keyword.

Remove the message rewrite that derived the clause and line from the
source. The VM chooses IndentationError or incomplete input by the
error variant instead of its message.

Assisted-by: Claude Code:claude-opus-5-5

* Take binding target errors from the parser

Bump the ruff fork, which now names the invalid expression in
assignment, augmented assignment, delete, `for`, `with`, import,
pattern and `except` target errors.

Remove the source scanners for those targets and the message rewrites
of `InvalidAssignmentTarget` and `InvalidNamedAssignmentTarget`.
Remove the expected failure markers from test_syntax doctests that now
pass.

Assisted-by: Claude Code:claude-opus-5-5

* Take parameter, argument, star, comma and dictionary errors from the parser

Bump the ruff fork to 36919b62fa. Remove the source scanners for type
parameters, parameter lists, star annotations, call arguments, def type
parameters, parenthesized groups, missing commas and dictionary items.
Set end_offset 0 for ExpectedColonAfterDictionaryKey. Remove 14
EXPECTED_FAILURE markers in test_syntax and test_unpack_ex.

Assisted-by: Claude Code:claude-opus-5-5

* Take expression errors from the parser

Bump the ruff fork to 4f922d2124. Remove the source scanners for
comprehensions, collection and condition assignments, expressions
between strings, named expression targets, missing `in` after for-loop
variables and statements in `if` expressions. Remove the now passing
expected failure markers in test_syntax and test_named_expressions.

Assisted-by: Claude Code:claude-opus-5-5

* Take statement errors from the parser

Bump the ruff fork to c7f6594ed6. Remove the source scanners for
standalone except, `import ... from`, `**_` mapping rests, `elif` after
`else` and mixed except handlers, and the post-parse scanners for call
arguments, match targets, yield after a comma and parenthesized star
imports.

Assisted-by: Claude Code:claude-opus-5-5

* Take string and f-string errors from the parser

Bump the ruff fork to 6b46d0dfc8. Remove the source scanners for
malformed `\N` escapes, f-string and t-string replacement fields, mixed
t-string literals and `print`/`exec` statements, and the override
plumbing they used. Drop the `f'{x'; '` case from the compiler test,
whose expected message differs from CPython 3.14.

Assisted-by: Claude Code:claude-opus-5-5

* Restore the bracket depth check and fix incomplete unclosed strings

Restore `pre_parse_source_error`, which rejects bracket nesting past 200
levels before parsing. The parser reports the same error but still
builds the full tree, which overflows the native stack when dropped.

Remove the VM's unclosed string scanner, which ignored the mode and
treated `'abc` in exec mode as incomplete input. The compiler's check
now skips escaped newlines.

Assisted-by: Claude Code:claude-opus-5-5

* Report multiple statements before later errors in single mode

In single mode, when the first statement is a simple statement, report
code after its line as multiple statements before any error found
later, at the newline token or the comment before it. This replaces the
multiple statements check on the parsed body and also applies to
`ast.parse(mode='single')`.

An unclosed string that follows an earlier syntax error is no longer
incomplete input. The bracket depth error range is now empty.

Assisted-by: Claude Code:claude-opus-5-5

* Update the Ruff fork for parser speed and starred annotations

Bump the ruff fork to 8b59b85344, which skips the tokenizer error pass
when parsing finds no errors, stops allocating for every identifier's
string prefix check, and reports `def f(*args: *b = ...)` as invalid
syntax.

Remove expected failure markers from test_code_module
test_indentation_error and test_sysexcepthook_indentation_error, and
from test_dictcomps test_illegal_assignment.

Assisted-by: Claude Code:claude-opus-5-5

* Use the published Ruff fork 0.16.10 crates

Switch the rustpython-ruff_* dependencies from the git revision to the
0.16.10 crates.io release, published from the same commit (8b59b85344).

Assisted-by: Claude Code:claude-opus-5-5

---------

Co-authored-by: zzarbttoo <qodbwls70@naver.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-10-07 12:54:27 +09:00

123 lines
3.0 KiB
Python

from testutils import assert_raises
src = """
def valid_func():
pass
yield 2
"""
with assert_raises(SyntaxError) as ae:
compile(src, 'test.py', 'exec')
assert ae.exception.lineno == 5
src = """
if True:
pass
"""
with assert_raises(IndentationError):
compile(src, '', 'exec')
src = """
if True:
pass
pass
"""
with assert_raises(IndentationError):
compile(src, '', 'exec')
src = """
if True:
pass
pass
"""
with assert_raises(IndentationError):
compile(src, '', 'exec')
src = """
if True:
pass
\tpass
"""
with assert_raises(TabError):
compile(src, '', 'exec')
with assert_raises(SyntaxError):
compile('0xX', 'test.py', 'exec')
src = """
"aaaa" \a
"bbbb"
"""
with assert_raises(SyntaxError):
compile(src, 'test.py', 'exec')
src = """
from __future__ import not_a_real_future_feature
"""
with assert_raises(SyntaxError):
compile(src, 'test.py', 'exec')
src = """
a = 1
from __future__ import print_function
"""
with assert_raises(SyntaxError):
compile(src, 'test.py', 'exec')
src = """
from __future__ import print_function
"""
compile(src, 'test.py', 'exec')
# CPython reports the unparenthesized `except ... as` error with a range that starts at the
# first exception type and stops just before the `:` closing the clause. `end_offset` is an
# exclusive 1-based *character* column, so that end lands on the column the `:` sits on.
#
# The cases below pin the parts that are easy to get wrong: the column counts characters
# rather than UTF-8 bytes, and an explicit line join may sit anywhere between the exception
# types and the `:`. Values verified against CPython 3.14.
except_as_ranges = [
# (clause, (lineno, offset), (end_lineno, end_offset))
("except A, B as e:", (3, 8), (3, 17)),
("except A, B as e :", (3, 8), (3, 18)),
("except A, B as e :", (3, 8), (3, 20)),
("except A, B as e\t:", (3, 8), (3, 18)),
("except A, B, C as blech:", (3, 8), (3, 24)),
("except* A, B, C as e:", (3, 9), (3, 21)),
# A non-ASCII name must not push the column right by its extra UTF-8 bytes.
("except Ä, B as e:", (3, 8), (3, 17)),
("except 사과, B as e:", (3, 8), (3, 18)),
("except A, B as 오:", (3, 8), (3, 17)),
# An explicit line join may precede or follow `as`, or close the clause on its own line.
("except A, B as exc\\\n :", (3, 8), (4, 5)),
("except A, B \\\nas exc:", (3, 8), (4, 7)),
("except A, \\\nB as exc:", (3, 8), (4, 9)),
("except A, B as \\\nexc:", (3, 8), (4, 4)),
]
for clause, start, end in except_as_ranges:
src = "try:\n pass\n%s\n pass\n" % clause
with assert_raises(SyntaxError) as ae:
compile(src, "test.py", "exec")
exc = ae.exception
assert exc.msg == "multiple exception types must be parenthesized when using 'as'", (
clause,
exc.msg,
)
assert (exc.lineno, exc.offset) == start, (clause, (exc.lineno, exc.offset), start)
assert (exc.end_lineno, exc.end_offset) == end, (
clause,
(exc.end_lineno, exc.end_offset),
end,
)