mirrors/RustPython
0
0 Fork 0
mirror of https://github.com/RustPython/RustPython.git synced 2026-10-28 10:44:36 +08:00
Code Activity

Take SyntaxError messages and ranges from the parser (#8983)

Browse Source 
* Match CPython's SyntaxError range for unparenthesized `except` types

CPython's `invalid_except_stmt` rule raises the error only once the whole
clause has matched, and reports a range that starts at the first exception
type and ends at the `:` closing the clause, so it covers the `as NAME`
part as well:

    invalid_except_stmt:
        | 'except' a=expression ',' expressions 'as' NAME  ':' {
            RAISE_SYNTAX_ERROR_STARTING_FROM(a, "multiple exception types
            must be parenthesized when using 'as'") }

The parser reports the exception types alone, so look up that `:` in the
source and widen the range to match. Only `as NAME` may follow the
exception types, which is why the first `:` after them is the one closing
the clause.

    try:
        pass
    except A, B, C as e:
        pass

    CPython 3.14:  ('x.py', 3, 8, 'except A, B, C as e:\n', 3, 20)
    before:        ('x.py', 3, 8, 'except A, B, C as e:\n', 3, 15)

Fixes #8496

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* Clarify that the `except` error range stops before the closing colon

`invalid_except_stmt_end` returns the exclusive end of the range, so the
column it yields is the one the `:` sits on even though the `:` is not part
of the range. The previous "ending at the `:`" wording read as if the colon
were included, which invites an off-by-one "fix".

Comments only; no behavior change.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Fix the `except ... as` error column for non-ASCII names and line joins

`invalid_except_stmt_end` recovers the end of CPython's range by scanning the
source, which has to redo by hand what CPython gets from its tokenizer:

- The column was the byte distance from the line start, but `end_offset` is a
  character column. `except Ä, B as e:` reported 18 instead of 17. CPython
  converts explicitly, in `_PyPegen_byte_offset_to_character_offset`.
- The colon scan already spans explicit line joins, but the `as` check ran on
  the raw slice, so a `\` before `as` left the range unextended.
  `except A, B \` + `as exc:` reported (3, 12) instead of (4, 7).

Add regression cases to `syntax_invalid.py` covering both, plus the whitespace,
`except*` and line-join placements that already worked. All values are verified
against CPython 3.14.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Update the Ruff fork to 0.16.10

Point the rustpython-ruff_* dependencies at the 0.16.10-rustpython tag
of the RustPython Ruff fork and port to its AST changes:
`ExprCompare` stores all operands in `operands`, call arguments use
`ThinVec`, dict comprehension generators and f-string elements are boxed
slices, `FStringValue` iterates `FStringPartRef`, and `TokenKind::Name`
is now `TokenKind::Identifier`.

Assisted-by: Grok:grok-4.6
Assisted-by: Claude Code:claude-opus-5-5

* Drop the source scan for the `except ... as` error range

The parser now reports the unparenthesized exception types error with a
range ending at the `:` that closes the clause, so
`invalid_except_stmt_end` and the message match that widened the range
are removed.

Assisted-by: Claude Code:claude-opus-5-5

* Take parse error messages from the parser as they are

The Ruff fork now emits the final message texts, so `vm_new.rs` no
longer rewrites parse error messages by matching their text and no
longer lowercases the first character of every message.
`analyze_compile_error` keeps only the caret narrowing for starred
expressions and the line number in the unterminated string message.
The compiler matches the new lowercase texts for indented block and
mixed bytes literal errors.

The fork is referenced by revision until it is published.

Assisted-by: Claude Code:claude-opus-5-5

* Take starred and bytes literal error ranges from the parser

The Ruff fork now reports misplaced starred expressions at the `*` and
non-ASCII bytes literal errors over the whole literal. `vm_new.rs` drops
`narrow_caret` together with `SyntaxErrorInfo`, which only carried the
message after that, and the compiler drops `bytes_literal_span`.

Assisted-by: Claude Code:claude-opus-5-5

* Take indentation error kinds and positions from the parser

The Ruff fork now reports `TabError` and `TooDeepIndentation` as their
own lexer errors and places indentation and line continuation errors
itself. `vm_new.rs` picks TabError or IndentationError from the error
kind instead of scanning the source for mixed indentation, drops the
TabError message override, and sets the end offset of tokenizer errors
to 0 or -1 by kind. The compiler drops the line-end relocation of
unindent errors and the line continuation rewrite. `_tokenize` raises
TabError and depth errors for the token they replace, also for sources
containing tabs.

Remove expectedFailure from test_tokenize.test_max_indent and four
test_tabnanny tests.

Assisted-by: Claude Code:claude-opus-5-5

* Take unterminated string messages from the parser

The Ruff fork's unterminated string errors now include the detected
line and are reported at the start of the string. `vm_new.rs` drops its
message override for `UnclosedStringError`, and the shell continues a
line on an unterminated triple-quoted string by the error's
`triple_quoted` flag instead of reading the quotes from the source.

Assisted-by: Claude Code:claude-opus-5-5

* Take bracket errors and tokenizer error priority from the parser

Bump the ruff fork, which now reports unmatched, mismatched, unclosed
and too deeply nested brackets and decides whether a tokenizer error
replaces the parser's first error.

Remove the bracket, unterminated string and nesting depth source
scanners and `pre_parse_source_error`. A tokenizer error from the
parser now wins over source scanner diagnostics found after it. An
unclosed bracket is incomplete input only when the parser reports it
as such, and other tokenizer errors end where they start.

Remove the expected failure markers from test_unicode_identifiers
test_invalid and the `import ä £` doctest in test_syntax.

Assisted-by: Claude Code:claude-opus-5-5

* Take number literal, string prefix and non-printable errors from the parser

Bump the ruff fork, which now reports malformed number literals,
incompatible string prefixes and non-printable characters itself.

Remove the number literal, string prefix and non-printable character
source scanners and the override classes that ranked them. Leading zero
and string prefix errors keep their range instead of ending where they
start.

Assisted-by: Claude Code:claude-opus-5-5

* Take the missing indented block error from the parser

Bump the ruff fork, which now reports `ExpectedIndentedBlock` with the
clause and the line of its header keyword.

Remove the message rewrite that derived the clause and line from the
source. The VM chooses IndentationError or incomplete input by the
error variant instead of its message.

Assisted-by: Claude Code:claude-opus-5-5

* Take binding target errors from the parser

Bump the ruff fork, which now names the invalid expression in
assignment, augmented assignment, delete, `for`, `with`, import,
pattern and `except` target errors.

Remove the source scanners for those targets and the message rewrites
of `InvalidAssignmentTarget` and `InvalidNamedAssignmentTarget`.
Remove the expected failure markers from test_syntax doctests that now
pass.

Assisted-by: Claude Code:claude-opus-5-5

* Take parameter, argument, star, comma and dictionary errors from the parser

Bump the ruff fork to 36919b62fa. Remove the source scanners for type
parameters, parameter lists, star annotations, call arguments, def type
parameters, parenthesized groups, missing commas and dictionary items.
Set end_offset 0 for ExpectedColonAfterDictionaryKey. Remove 14
EXPECTED_FAILURE markers in test_syntax and test_unpack_ex.

Assisted-by: Claude Code:claude-opus-5-5

* Take expression errors from the parser

Bump the ruff fork to 4f922d2124. Remove the source scanners for
comprehensions, collection and condition assignments, expressions
between strings, named expression targets, missing `in` after for-loop
variables and statements in `if` expressions. Remove the now passing
expected failure markers in test_syntax and test_named_expressions.

Assisted-by: Claude Code:claude-opus-5-5

* Take statement errors from the parser

Bump the ruff fork to c7f6594ed6. Remove the source scanners for
standalone except, `import ... from`, `**_` mapping rests, `elif` after
`else` and mixed except handlers, and the post-parse scanners for call
arguments, match targets, yield after a comma and parenthesized star
imports.

Assisted-by: Claude Code:claude-opus-5-5

* Take string and f-string errors from the parser

Bump the ruff fork to 6b46d0dfc8. Remove the source scanners for
malformed `\N` escapes, f-string and t-string replacement fields, mixed
t-string literals and `print`/`exec` statements, and the override
plumbing they used. Drop the `f'{x'; '` case from the compiler test,
whose expected message differs from CPython 3.14.

Assisted-by: Claude Code:claude-opus-5-5

* Restore the bracket depth check and fix incomplete unclosed strings

Restore `pre_parse_source_error`, which rejects bracket nesting past 200
levels before parsing. The parser reports the same error but still
builds the full tree, which overflows the native stack when dropped.

Remove the VM's unclosed string scanner, which ignored the mode and
treated `'abc` in exec mode as incomplete input. The compiler's check
now skips escaped newlines.

Assisted-by: Claude Code:claude-opus-5-5

* Report multiple statements before later errors in single mode

In single mode, when the first statement is a simple statement, report
code after its line as multiple statements before any error found
later, at the newline token or the comment before it. This replaces the
multiple statements check on the parsed body and also applies to
`ast.parse(mode='single')`.

An unclosed string that follows an earlier syntax error is no longer
incomplete input. The bracket depth error range is now empty.

Assisted-by: Claude Code:claude-opus-5-5

* Update the Ruff fork for parser speed and starred annotations

Bump the ruff fork to 8b59b85344, which skips the tokenizer error pass
when parsing finds no errors, stops allocating for every identifier's
string prefix check, and reports `def f(*args: *b = ...)` as invalid
syntax.

Remove expected failure markers from test_code_module
test_indentation_error and test_sysexcepthook_indentation_error, and
from test_dictcomps test_illegal_assignment.

Assisted-by: Claude Code:claude-opus-5-5

* Use the published Ruff fork 0.16.10 crates

Switch the rustpython-ruff_* dependencies from the git revision to the
0.16.10 crates.io release, published from the same commit (8b59b85344).

Assisted-by: Claude Code:claude-opus-5-5

---------

Co-authored-by: zzarbttoo <qodbwls70@naver.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Jeong, YunWonzzarbttooClaude Opus 5
 authored and GitHub committed 2026-10-07 12:54:27 +09:00
1 parent 4b6ad58469
commit a403341791
25 files changed
+562 -6848
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

Cargo.lock
Generated
+20 -16
View File
@@ -572,9 +572,9 @@ dependencies = [
[[package]]
name = "char_str"
version = "0.0.2"
version = "0.0.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "576ba56f6ca18ebb069d0d07260407171712aff4dfe15ee3f8982dc16a455bcf"
checksum = "c9cba78a5abb515610e10af297615434b019755b5d7be14e559ff5340dff50c4"
dependencies = [
"castaway",
"get-size2",
@@ -1612,9 +1612,9 @@ dependencies = [
[[package]]
name = "get-size-derive2"
version = "0.10.3"
version = "0.11.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c736d226c32e496b8377813b52269e11ad3a48d8373b68862d0364f04fd1229d"
checksum = "7efe46d2adb529015748742f1a38594039ce357dbb21e6d178de342cfce468d4"
dependencies = [
"attribute-derive",
"quote",
@@ -1623,9 +1623,9 @@ dependencies = [
[[package]]
name = "get-size2"
version = "0.10.3"
version = "0.11.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b411f34418305908ab15a82ff78958c2a9aee9a272b2a2e663836b20a4e4b9d3"
checksum = "d4d338de392104417fe6fdbe588c52383b9236956d8afe5654f0077176edc632"
dependencies = [
"compact_str",
"get-size-derive2",
@@ -3935,9 +3935,9 @@ dependencies = [
[[package]]
name = "rustpython-ruff_python_ast"
version = "0.16.5"
version = "0.16.10"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bc19eeb4cfa2c35428143693eadf85ee78816f22a829298593581a56700c0a21"
checksum = "5b7ffeb85f9a68c461d538839c79548584768df08f2d68cb1ee693be697e0355"
dependencies = [
"aho-corasick",
"arrayvec",
@@ -3957,14 +3957,16 @@ dependencies = [
[[package]]
name = "rustpython-ruff_python_parser"
version = "0.16.5"
version = "0.16.10"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "480eec9ac4dda77b6724f0c9a54a81d3c57868d6a707c3d7c5133fd83c933f66"
checksum = "7c9729e35cac02839c7a67aaccfea3eba32032f7aeacca2e82ad79e8d7b7fd95"
dependencies = [
"bitflags 2.13.2",
"bstr",
"drop_bomb",
"get-size2",
"hashbrown 0.17.1",
"icu_properties",
"memchr",
"rustc-hash",
"rustpython-ruff_python_ast",
@@ -3980,22 +3982,24 @@ dependencies = [
[[package]]
name = "rustpython-ruff_python_trivia"
version = "0.16.5"
version = "0.16.10"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6d69a5d9bb610a76bee05d23adce0de55f0fa06f251c0e1c7f7a0c625a366a58"
checksum = "764f43ea060f888164fbd360b0e1a36f2a35f5c03def85407bab7039fc62a645"
dependencies = [
"itertools 0.15.0",
"memchr",
"rustc-hash",
"rustpython-ruff_source_file",
"rustpython-ruff_text_size",
"unicode-ident",
"unicode-normalization",
]
[[package]]
name = "rustpython-ruff_source_file"
version = "0.16.5"
version = "0.16.10"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d3aa9c91c8a4e86bf0fa8a67aec284fa6c0c533cdefb900b08584be15748c606"
checksum = "cfe26430c3f2eadb40524b6df7ac9a3970dd2b58f921b01ed75df0121e332fad"
dependencies = [
"memchr",
"rustpython-ruff_text_size",
@@ -4003,9 +4007,9 @@ dependencies = [
[[package]]
name = "rustpython-ruff_text_size"
version = "0.16.5"
version = "0.16.10"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8fedbd1af4f91cd828185123d95119662db9c4a0b21c02b695783d74360d4f46"
checksum = "5e5377a04f2855030600b93a05754242950ea9b070507ef9aef67965a52eeb09"
dependencies = [
"get-size2",
]
Cargo.toml
+4 -4
View File
@@ -188,10 +188,10 @@ rustpython-wtf8 = { path = "crates/wtf8", version = "0.6.1" }
rustpython-doc = { path = "crates/doc", version = "0.6.1" }
# Use the RustPython Ruff fork for RustPython public `_ast` metadata.
ruff_python_parser = { package = "rustpython-ruff_python_parser", version = "0.16.5" }
ruff_python_ast = { package = "rustpython-ruff_python_ast", version = "0.16.5" }
ruff_text_size = { package = "rustpython-ruff_text_size", version = "0.16.5" }
ruff_source_file = { package = "rustpython-ruff_source_file", version = "0.16.5" }
ruff_python_parser = { package = "rustpython-ruff_python_parser", version = "0.16.10" }
ruff_python_ast = { package = "rustpython-ruff_python_ast", version = "0.16.10" }
ruff_text_size = { package = "rustpython-ruff_text_size", version = "0.16.10" }
ruff_source_file = { package = "rustpython-ruff_source_file", version = "0.16.10" }
der = { version = "0.8", features = ["alloc", "oid", "pem", "zeroize"] }
phf = { version = "0.14.0", default-features = false, features = ["macros"]}
Lib/test/test_code_module.py
Vendored
-2
View File
@@ -111,7 +111,6 @@ class TestInteractiveConsole(unittest.TestCase, MockSys):
self.assertIsNone(self.sysmod.last_value.__traceback__)
self.assertIs(self.sysmod.last_exc, self.sysmod.last_value)
@unittest.expectedFailure # TODO: RUSTPYTHON; - 'IndentationError: unexpected indentation']
def test_indentation_error(self):
self.infunc.side_effect = [" 1", EOFError('Finished')]
self.console.interact()
@@ -187,7 +186,6 @@ class TestInteractiveConsole(unittest.TestCase, MockSys):
' ^\n',
'SyntaxError: invalid syntax\n'])
@unittest.expectedFailure # TODO: RUSTPYTHON; + 'IndentationError: unexpected indent\n']
def test_sysexcepthook_indentation_error(self):
self.infunc.side_effect = [" 1", EOFError('Finished')]
hook = mock.Mock()
Lib/test/test_dictcomps.py
Vendored
-1
View File
@@ -75,7 +75,6 @@ class DictComprehensionTest(unittest.TestCase):
self.assertEqual(actual, expected)
self.assertEqual(v, "Local variable")
@unittest.expectedFailure # TODO: RUSTPYTHON
def test_illegal_assignment(self):
with self.assertRaisesRegex(SyntaxError, "cannot assign"):
compile("{x: y for y, x in ((1, 2), (3, 4))} = 5", "<test>",
Lib/test/test_fstring.py
Vendored
-1
View File
@@ -865,7 +865,6 @@ x = (
x = X()
self.assertEqual(f'{x} {x}', '1 2')
@unittest.expectedFailure # TODO: RUSTPYTHON
def test_missing_expression(self):
self.assertAllRaise(SyntaxError,
"f-string: valid expression required before '}'",
Lib/test/test_named_expressions.py
Vendored
-1
View File
@@ -98,7 +98,6 @@ class NamedExpressionInvalidTest(unittest.TestCase):
with self.assertRaisesRegex(SyntaxError, "invalid syntax"):
exec(code, {}, {})
@unittest.expectedFailure # TODO: RUSTPYTHON
def test_named_expression_invalid_17(self):
code = "[i := 0, j := 1 for i, j in [(1, 2), (3, 4)]]"
Lib/test/test_syntax.py
Vendored
+21 -21
View File
@@ -326,7 +326,7 @@ SyntaxError: did you forget parentheses around the comprehension target?
# Incorrectly closed strings
>>> "The interesting object "The important object" is very important" # TODO: RUSTPYTHON; Wrong error message # doctest: +EXPECTED_FAILURE
>>> "The interesting object "The important object" is very important"
Traceback (most recent call last):
SyntaxError: invalid syntax. Is this intended to be part of the string?
@@ -423,7 +423,7 @@ SyntaxError: invalid syntax
Traceback (most recent call last):
SyntaxError: invalid syntax
>>> def foo(/,a,b=,c): # TODO: RUSTPYTHON; Wrong error message # doctest: +EXPECTED_FAILURE
>>> def foo(/,a,b=,c):
... pass
Traceback (most recent call last):
SyntaxError: at least one argument must precede /
@@ -1045,7 +1045,7 @@ This tests assignment-context; there was a bug in Python 2.5 where compiling
a complex 'if' (one with 'elif') would fail to notice an invalid suite,
leading to spurious errors.
>>> if 1: # TODO: RUSTPYTHON; Wrong error message # doctest: +EXPECTED_FAILURE
>>> if 1:
... x() = 1
... elif 1:
... pass
@@ -1053,7 +1053,7 @@ leading to spurious errors.
...
SyntaxError: cannot assign to function call here. Maybe you meant '==' instead of '='?
>>> if 1: # TODO: RUSTPYTHON; Wrong error message # doctest: +EXPECTED_FAILURE
>>> if 1:
... pass
... elif 1:
... x() = 1
@@ -1061,7 +1061,7 @@ leading to spurious errors.
...
SyntaxError: cannot assign to function call here. Maybe you meant '==' instead of '='?
>>> if 1: # TODO: RUSTPYTHON; Wrong error message # doctest: +EXPECTED_FAILURE
>>> if 1:
... x() = 1
... elif 1:
... pass
@@ -1071,7 +1071,7 @@ leading to spurious errors.
...
SyntaxError: cannot assign to function call here. Maybe you meant '==' instead of '='?
>>> if 1: # TODO: RUSTPYTHON; Wrong error message # doctest: +EXPECTED_FAILURE
>>> if 1:
... pass
... elif 1:
... x() = 1
@@ -1081,7 +1081,7 @@ leading to spurious errors.
...
SyntaxError: cannot assign to function call here. Maybe you meant '==' instead of '='?
>>> if 1: # TODO: RUSTPYTHON; Wrong error message # doctest: +EXPECTED_FAILURE
>>> if 1:
... pass
... elif 1:
... pass
@@ -1185,7 +1185,7 @@ Missing ':' before suites:
Traceback (most recent call last):
SyntaxError: expected ':'
>>> with (blech as something) # TODO: RUSTPYTHON; Wrong error message # doctest: +EXPECTED_FAILURE
>>> with (blech as something)
... pass
Traceback (most recent call last):
SyntaxError: expected ':'
@@ -1195,12 +1195,12 @@ Missing ':' before suites:
Traceback (most recent call last):
SyntaxError: expected ':'
>>> with (blech, block as something) # TODO: RUSTPYTHON; Wrong error message # doctest: +EXPECTED_FAILURE
>>> with (blech, block as something)
... pass
Traceback (most recent call last):
SyntaxError: expected ':'
>>> with (blech, block as something, bluch) # TODO: RUSTPYTHON; Wrong error message # doctest: +EXPECTED_FAILURE
>>> with (blech, block as something, bluch)
... pass
Traceback (most recent call last):
SyntaxError: expected ':'
@@ -1464,7 +1464,7 @@ Ensure that early = are not matched by the parser as invalid comparisons
Traceback (most recent call last):
SyntaxError: invalid syntax
>>> dict(x=34, (x for x in range 10), 1); x $ y # TODO: RUSTPYTHON; Wrong error message # doctest: +EXPECTED_FAILURE
>>> dict(x=34, (x for x in range 10), 1); x $ y
Traceback (most recent call last):
SyntaxError: invalid syntax
@@ -2077,7 +2077,7 @@ SyntaxError: cannot use literal as import target
Traceback (most recent call last):
SyntaxError: cannot use literal as import target
>>> from a import (b as c.d) # TODO: RUSTPYTHON; Wrong error message # doctest: +EXPECTED_FAILURE
>>> from a import (b as c.d)
Traceback (most recent call last):
SyntaxError: cannot use attribute as import target
@@ -2085,18 +2085,18 @@ SyntaxError: cannot use attribute as import target
Traceback (most recent call last):
SyntaxError: cannot use literal as import target
>>> from a import ( # TODO: RUSTPYTHON; Wrong error message # doctest: +EXPECTED_FAILURE
>>> from a import (
... b as f())
Traceback (most recent call last):
SyntaxError: cannot use function call as import target
>>> from a import ( # TODO: RUSTPYTHON; Wrong error message # doctest: +EXPECTED_FAILURE
>>> from a import (
... b as [],
... )
Traceback (most recent call last):
SyntaxError: cannot use list as import target
>>> from a import ( # TODO: RUSTPYTHON; Wrong error message # doctest: +EXPECTED_FAILURE
>>> from a import (
... b,
... c as ()
... )
@@ -2233,7 +2233,7 @@ Corner-cases that used to crash:
Traceback (most recent call last):
SyntaxError: cannot assign to __debug__
>>> import ä £ # TODO: RUSTPYTHON; Wrong error message # doctest: +EXPECTED_FAILURE
>>> import ä £
Traceback (most recent call last):
SyntaxError: invalid character '£' (U+00A3)
@@ -2245,7 +2245,7 @@ Corner-cases that used to crash:
Traceback (most recent call last):
SyntaxError: cannot use '_' as a target
>>> match ...: # TODO: RUSTPYTHON; Wrong error message # doctest: +EXPECTED_FAILURE
>>> match ...:
... case 42 as 1+2+4:
... ...
Traceback (most recent call last):
@@ -2263,13 +2263,13 @@ Corner-cases that used to crash:
Traceback (most recent call last):
SyntaxError: cannot use tuple as pattern target
>>> match ...: # TODO: RUSTPYTHON; Wrong error message # doctest: +EXPECTED_FAILURE
>>> match ...:
... case 42 as (a + 1):
... ...
Traceback (most recent call last):
SyntaxError: cannot use expression as pattern target
>>> match ...: # TODO: RUSTPYTHON; Wrong error message # doctest: +EXPECTED_FAILURE
>>> match ...:
... case (32 as x) | (42 as a()):
... ...
Traceback (most recent call last):
@@ -2375,7 +2375,7 @@ A[*(1:2)]
A[*:] and A[:*]
>>> A[*:] # TODO: RUSTPYTHON; Wrong error message # doctest: +EXPECTED_FAILURE
>>> A[*:]
Traceback (most recent call last):
...
SyntaxError: Invalid star expression
@@ -2386,7 +2386,7 @@ A[*:] and A[:*]
A[*]
>>> A[*] # TODO: RUSTPYTHON; Wrong error message # doctest: +EXPECTED_FAILURE
>>> A[*]
Traceback (most recent call last):
...
SyntaxError: Invalid star expression
Lib/test/test_tabnanny.py
Vendored
-4
View File
@@ -217,7 +217,6 @@ class TestCheck(TestCase):
with self.assertRaises(SystemExit):
self.verify_tabnanny_check(file_path, err=err)
@unittest.expectedFailure # TODO: RUSTPYTHON; A python source code file eligible for raising `tabnanny.NannyNag`.
def test_when_nannynag_error_verbose(self):
"""A python source code file eligible for raising `tabnanny.NannyNag`.
@@ -231,7 +230,6 @@ class TestCheck(TestCase):
tabnanny.verbose = 1
self.verify_tabnanny_check(file_path, out=out)
@unittest.expectedFailure # TODO: RUSTPYTHON; A python source code file eligible for raising `tabnanny.NannyNag`.
def test_when_nannynag_error(self):
"""A python source code file eligible for raising `tabnanny.NannyNag`."""
with TemporaryPyFile(SOURCE_CODES["nannynag_errored"]) as file_path:
@@ -341,7 +339,6 @@ class TestCommandLine(TestCase):
stdout = f"{file_path}\n"
self.validate_cmd("-q", file_path, stdout=stdout)
@unittest.expectedFailure # TODO: RUSTPYTHON
def test_verbose_mode(self):
"""Should display more error information if verbose mode is on."""
with TemporaryPyFile(SOURCE_CODES["nannynag_errored"]) as path:
@@ -350,7 +347,6 @@ class TestCommandLine(TestCase):
).strip()
self.validate_cmd("-v", path, stdout=stdout, partial=True)
@unittest.expectedFailure # TODO: RUSTPYTHON
def test_double_verbose_mode(self):
"""Should display detailed error information if double verbose is on."""
with TemporaryPyFile(SOURCE_CODES["nannynag_errored"]) as path:
Lib/test/test_tokenize.py
Vendored
-1
View File
@@ -3193,7 +3193,6 @@ async def f():
with self.subTest(case=case):
self.assertRaises(tokenize.TokenError, get_tokens, case)
@unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: IndentationError not raised by <lambda>
@support.skip_wasi_stack_overflow()
def test_max_indent(self):
MAXINDENT = 100
Lib/test/test_unicode_identifiers.py
Vendored
-1
View File
@@ -17,7 +17,6 @@ class PEP3131Test(unittest.TestCase):
𝔘𝔫𝔦𝔠𝔬𝔡𝔢 = 1
self.assertIn("Unicode", dir())
@unittest.expectedFailure # TODO: RUSTPYTHON
def test_invalid(self):
try:
from test.tokenizedata import badsyntax_3131 # noqa: F401
Lib/test/test_unpack_ex.py
Vendored
+2 -2
View File
@@ -361,7 +361,7 @@ Now some general starred expressions (all fail).
...
SyntaxError: cannot use starred expression here
>>> (((*x))),y = 1, 2 # TODO: RUSTPYTHON # doctest:+ELLIPSIS +EXPECTED_FAILURE
>>> (((*x))),y = 1, 2 # doctest:+ELLIPSIS
Traceback (most recent call last):
...
SyntaxError: cannot use starred expression here
@@ -376,7 +376,7 @@ Now some general starred expressions (all fail).
...
SyntaxError: cannot use starred expression here
>>> ((*x),y) = 1, 2 # TODO: RUSTPYTHON # doctest:+ELLIPSIS +EXPECTED_FAILURE
>>> ((*x),y) = 1, 2 # doctest:+ELLIPSIS
Traceback (most recent call last):
...
SyntaxError: cannot use starred expression here
crates/codegen/src/compile.rs
+36 -34
View File
@@ -8604,14 +8604,15 @@ impl<'warnings> Compiler<'warnings> {
self.use_cpython_label_block(end);
Ok(())
}
ast::Expr::Compare(ast::ExprCompare {
left,
ops,
comparators,
..
}) if ops.len() > 1 => {
ast::Expr::Compare(compare) if compare.ops.len() > 1 => {
self.set_source_range(expression.range());
self.compile_jump_if_compare(left, ops, comparators, condition, target_block)
self.compile_jump_if_compare(
compare.first_operand(),
&compare.ops,
compare.comparators(),
condition,
target_block,
)
}
_ => {
// Fall back case which always will work!
@@ -8940,13 +8941,8 @@ impl<'warnings> Compiler<'warnings> {
self.emit_load_attr(idx);
}
}
ast::Expr::Compare(ast::ExprCompare {
left,
ops,
comparators,
..
}) => {
self.compile_compare(left, ops, comparators)?;
ast::Expr::Compare(compare) => {
self.compile_compare(compare.first_operand(), &compare.ops, compare.comparators())?;
}
ast::Expr::Constant(ast::ExprConstant { value, .. }) => {
self.emit_load_const(ast_constant_value_to_constant_data(value.clone()));
@@ -12521,7 +12517,11 @@ impl<'warnings> Compiler<'warnings> {
ast::Expr::Await(_) => self.found = true,
ast::Expr::ListComp(ast::ExprListComp { generators, .. })
| ast::Expr::SetComp(ast::ExprSetComp { generators, .. })
| ast::Expr::DictComp(ast::ExprDictComp { generators, .. })
if generators.iter().any(|generator| generator.is_async) =>
{
self.found = true
}
ast::Expr::DictComp(ast::ExprDictComp { generators, .. })
if generators.iter().any(|generator| generator.is_async) =>
{
self.found = true
@@ -12556,16 +12556,16 @@ impl<'warnings> Compiler<'warnings> {
fn compile_expr_fstring(&mut self, fstring: &ast::ExprFString) -> CompileResult<()> {
let fstring_range = fstring.range;
let fstring = fstring.value.as_slice();
if self.count_fstring_parts(fstring) > STACK_USE_GUIDELINE {
return self.compile_fstring_parts_joined(fstring, fstring_range);
let value = &fstring.value;
if self.count_fstring_parts(value) > STACK_USE_GUIDELINE {
return self.compile_fstring_parts_joined(value, fstring_range);
}
let mut element_count = 0;
let mut pending_literal = None;
let mut pending_literal_range = None;
let mut pending_literal_no_location = false;
for part in fstring {
for part in value {
self.compile_fstring_part_into(
part,
&mut pending_literal,
@@ -12629,7 +12629,7 @@ impl<'warnings> Compiler<'warnings> {
fn compile_fstring_parts_joined(
&mut self,
fstring: &[ast::FStringPart],
fstring: &ast::FStringValue,
fstring_range: TextRange,
) -> CompileResult<()> {
self.set_source_range(fstring_range);
@@ -12666,7 +12666,7 @@ impl<'warnings> Compiler<'warnings> {
fn compile_fstring_part_into(
&mut self,
part: &ast::FStringPart,
part: ast::FStringPartRef<'_>,
pending_literal: &mut Option<Wtf8Buf>,
pending_literal_range: &mut Option<TextRange>,
pending_literal_no_location: &mut bool,
@@ -12674,7 +12674,7 @@ impl<'warnings> Compiler<'warnings> {
join_append_range: Option<TextRange>,
) -> CompileResult<()> {
match part {
ast::FStringPart::Literal(string) => {
ast::FStringPartRef::Literal(string) => {
let value = string_literal_part_value(&self.source_file, string);
if pending_literal.is_none() {
*pending_literal_range = Some(string.range);
@@ -12687,7 +12687,7 @@ impl<'warnings> Compiler<'warnings> {
}
Ok(())
}
ast::FStringPart::FString(fstring) => self.compile_fstring_elements_into(
ast::FStringPartRef::FString(fstring) => self.compile_fstring_elements_into(
fstring.flags,
&fstring.elements,
pending_literal,
@@ -12801,7 +12801,7 @@ impl<'warnings> Compiler<'warnings> {
}
}
fn count_fstring_parts(&self, fstring: &[ast::FStringPart]) -> u32 {
fn count_fstring_parts(&self, fstring: &ast::FStringValue) -> u32 {
let mut element_count = 0;
let mut pending_literal = None;
for part in fstring {
@@ -12813,12 +12813,12 @@ impl<'warnings> Compiler<'warnings> {
fn count_fstring_part_into(
&self,
part: &ast::FStringPart,
part: ast::FStringPartRef<'_>,
pending_literal: &mut Option<Wtf8Buf>,
element_count: &mut u32,
) {
match part {
ast::FStringPart::Literal(string) => {
ast::FStringPartRef::Literal(string) => {
let value = string_literal_part_value(&self.source_file, string);
if let Some(pending) = pending_literal.as_mut() {
pending.push_wtf8(value.as_ref());
@@ -12826,7 +12826,7 @@ impl<'warnings> Compiler<'warnings> {
*pending_literal = Some(value);
}
}
ast::FStringPart::FString(fstring) => self.count_fstring_elements_into(
ast::FStringPartRef::FString(fstring) => self.count_fstring_elements_into(
fstring.flags,
&fstring.elements,
pending_literal,
@@ -13972,7 +13972,7 @@ mod tests {
arguments: ast::Arguments {
node_index: ast::AtomicNodeIndex::NONE,
range: TextRange::default(),
args: Box::default(),
args: Default::default(),
keywords: Default::default(),
runtime_args: None,
runtime_bases: None,
@@ -14120,7 +14120,7 @@ mod tests {
arguments: ast::Arguments {
node_index: ast::AtomicNodeIndex::NONE,
range: TextRange::default(),
args: Box::default(),
args: Default::default(),
keywords: Default::default(),
runtime_args: None,
runtime_bases: None,
@@ -14196,12 +14196,14 @@ mod tests {
let message = first_ast_constant_warning(ast::Expr::Compare(ast::ExprCompare {
node_index: ast::AtomicNodeIndex::NONE,
range: TextRange::default(),
left: Box::new(left),
ops: Box::new([ast::CmpOp::Is]),
comparators: Box::new([ast::Expr::NoneLiteral(ast::ExprNoneLiteral {
node_index: ast::AtomicNodeIndex::NONE,
range: TextRange::default(),
})]),
operands: Box::new([
left,
ast::Expr::NoneLiteral(ast::ExprNoneLiteral {
node_index: ast::AtomicNodeIndex::NONE,
range: TextRange::default(),
}),
]),
runtime_comparators: None,
}));
assert!(
crates/codegen/src/symboltable.rs
+3 -5
View File
@@ -2207,11 +2207,9 @@ impl SymbolTableBuilder {
Expr::BoolOp(ExprBoolOp { values, .. }) => {
self.scan_expressions(values, context)?;
}
Expr::Compare(ExprCompare {
left, comparators, ..
}) => {
self.scan_expression(left, context)?;
self.scan_expressions(comparators, context)?;
Expr::Compare(compare) => {
self.scan_expression(compare.first_operand(), context)?;
self.scan_expressions(compare.comparators(), context)?;
}
Expr::Subscript(ExprSubscript { value, slice, .. }) => {
self.scan_expression(value, ExpressionContext::Load)?;
crates/codegen/src/unparse.rs
+5 -12
View File
@@ -380,18 +380,11 @@ impl<'a, 'b, 'c> Unparser<'a, 'b, 'c> {
UnparseExpr::new(value, self.source)
)?;
}
ast::Expr::Compare(ast::ExprCompare {
left,
ops,
comparators,
node_index: _,
range: _range,
..
}) => {
ast::Expr::Compare(compare) => {
group_if!(precedence::CMP, {
let new_lvl = precedence::CMP + 1;
self.unparse_expr(left, new_lvl)?;
for (op, cmp) in ops.iter().zip(comparators) {
self.unparse_expr(compare.first_operand(), new_lvl)?;
for (_, op, cmp) in compare.iter() {
self.p(" ")?;
self.p(op.as_str())?;
self.p(" ")?;
@@ -700,8 +693,8 @@ impl<'a, 'b, 'c> Unparser<'a, 'b, 'c> {
self.p("f")?;
let body = fmt::from_fn(|f| {
value.iter().try_for_each(|part| match part {
ast::FStringPart::Literal(lit) => f.write_str(lit),
ast::FStringPart::FString(ast::FString { elements, .. }) => {
ast::FStringPartRef::Literal(lit) => f.write_str(lit),
ast::FStringPartRef::FString(ast::FString { elements, .. }) => {
Unparser::new(f, self.source).unparse_fstring_body(elements)
}
})
crates/compiler/src/lib.rs
+232 -6139
View File
File diff suppressed because it is too large. Load diff
crates/stdlib/src/_tokenize.rs
+27 -22
View File
@@ -210,22 +210,22 @@ mod _tokenize {
let kind = token.kind();
let range = token.range();
// Check for lexical indentation errors.
// Skip when source has tabs — ruff and CPython handle tab
// indentation differently (CPython uses tabsize=8), so ruff may
// report false IndentationErrors for valid mixed-tab code.
if !source.contains('\t') {
for err in errors.iter() {
if !matches!(
err.error,
ParseErrorType::Lexical(LexicalErrorType::IndentationError)
) {
continue;
}
if err.location.start() <= range.start() && range.start() < err.location.end() {
return Err(raise_indentation_error(vm, err, source, line_index));
}
}
// An indentation error replaces the token the lexer raised it for.
if kind == TokenKind::Unknown
&& let Some(err) = errors.iter().find(|err| {
line_index.line_index(err.location.start())
== line_index.line_index(range.start())
&& matches!(
err.error,
ParseErrorType::Lexical(
LexicalErrorType::IndentationError
| LexicalErrorType::TabError
| LexicalErrorType::TooDeepIndentation
)
)
})
{
return Err(raise_indentation_error(vm, err, source, line_index));
}
if kind == TokenKind::EndOfFile {
@@ -480,7 +480,7 @@ mod _tokenize {
exc
}
/// Raise an IndentationError from a parse error.
/// Raise an IndentationError or TabError from a parse error.
fn raise_indentation_error(
vm: &VirtualMachine,
err: &ParseError,
@@ -491,10 +491,15 @@ mod _tokenize {
let err_line_text = source.full_line_str(err.location.start());
let err_text = err_line_text.trim_end_matches('\n').trim_end_matches('\r');
let msg = format!("{}", err.error);
let exc = vm.new_exception_msg(
vm.ctx.exceptions.indentation_error.to_owned(),
msg.clone().into(),
);
let exc_type = if matches!(
err.error,
ParseErrorType::Lexical(LexicalErrorType::TabError)
) {
vm.ctx.exceptions.tab_error
} else {
vm.ctx.exceptions.indentation_error
};
let exc = vm.new_exception_msg(exc_type.to_owned(), msg.clone().into());
let obj = exc.as_object();
let _ = obj.set_attr("lineno", vm.ctx.new_int(err_lc.line.get()), vm);
let _ = obj.set_attr("offset", vm.ctx.new_int(err_text.len() as i64 + 1), vm);
@@ -658,7 +663,7 @@ mod _tokenize {
const fn token_kind_value(kind: TokenKind) -> u8 {
match kind {
TokenKind::EndOfFile => 0,
TokenKind::Name
TokenKind::Identifier
| TokenKind::For
| TokenKind::In
| TokenKind::Pass
crates/vm/src/stdlib/_ast.rs
+6
View File
@@ -1815,6 +1815,12 @@ pub(crate) fn parse(
);
rustpython_compiler::pre_parse_source_error(&source_file)?;
let parsed = parser::parse_unchecked(barry_source.source(), options);
if interactive
&& let Some(error) =
rustpython_compiler::single_mode_multiple_statements_error(&source_file, &parsed)
{
return Err(error);
}
let type_comment_source =
type_comments.then(|| TypeCommentSource::new(source, parsed.tokens()));
if let Some(lines) = &type_comment_source
crates/vm/src/stdlib/_ast/argument.rs
+17 -23
View File
@@ -7,7 +7,7 @@ pub(super) struct PositionalArguments {
}
enum PositionalArgumentsKind {
Args(Box<[ast::Expr]>),
Args(thin_vec::ThinVec<ast::Expr>),
RuntimeValues(Vec<Option<ast::Expr>>),
}
@@ -24,7 +24,7 @@ impl PositionalArguments {
Ok(Self::from_values(TextRange::default(), values))
}
fn from_args(range: TextRange, args: Box<[ast::Expr]>) -> Self {
fn from_args(range: TextRange, args: thin_vec::ThinVec<ast::Expr>) -> Self {
Self {
range,
kind: PositionalArgumentsKind::Args(args),
@@ -42,14 +42,7 @@ impl PositionalArguments {
if values.iter().any(Option::is_none) {
Self::from_runtime_values(range, values)
} else {
Self::from_args(
range,
values
.into_iter()
.flatten()
.collect::<Vec<_>>()
.into_boxed_slice(),
)
Self::from_args(range, values.into_iter().flatten().collect())
}
}
@@ -57,13 +50,14 @@ impl PositionalArguments {
self.range
}
fn into_args_and_runtime_values(self) -> (Box<[ast::Expr]>, Option<Vec<Option<ast::Expr>>>) {
fn into_args_and_runtime_values(
self,
) -> (thin_vec::ThinVec<ast::Expr>, Option<Vec<Option<ast::Expr>>>) {
match self.kind {
PositionalArgumentsKind::Args(args) => (args, None),
PositionalArgumentsKind::RuntimeValues(values) => (
lower_runtime_expr_list(values.clone()).into_boxed_slice(),
Some(values),
),
PositionalArgumentsKind::RuntimeValues(values) => {
(lower_runtime_expr_list(values.clone()).into(), Some(values))
}
}
}
}
@@ -71,7 +65,7 @@ impl PositionalArguments {
impl Node for PositionalArguments {
fn ast_to_object(self, vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef {
match self.kind {
PositionalArgumentsKind::Args(args) => BoxedSlice(args).ast_to_object(vm, source_file),
PositionalArgumentsKind::Args(args) => args.ast_to_object(vm, source_file),
PositionalArgumentsKind::RuntimeValues(values) => values.ast_to_object(vm, source_file),
}
}
@@ -81,8 +75,8 @@ impl Node for PositionalArguments {
source_file: &SourceFile,
object: PyObjectRef,
) -> PyResult<Self> {
let args: BoxedSlice<_> = Node::ast_from_object(vm, source_file, object)?;
Ok(Self::from_args(TextRange::default(), args.0))
let args: thin_vec::ThinVec<_> = Node::ast_from_object(vm, source_file, object)?;
Ok(Self::from_args(TextRange::default(), args))
}
}
@@ -137,7 +131,7 @@ pub(super) fn merge_function_call_arguments(
node_index: Default::default(),
range,
args,
keywords: key_args.keywords.into(),
keywords: key_args.keywords.into_vec().into(),
runtime_args,
runtime_bases: None,
}
@@ -173,7 +167,7 @@ pub(super) fn split_function_call_arguments(
// debug_assert!(range.contains_range(keyword_arguments_range));
let keyword_arguments = KeywordArguments {
range: keyword_arguments_range,
keywords: keywords.into(),
keywords: keywords.into_iter().collect::<Vec<_>>().into_boxed_slice(),
};
(positional_arguments, keyword_arguments)
@@ -214,7 +208,7 @@ pub(super) fn split_class_def_args(
// debug_assert!(range.contains_range(keyword_arguments_range));
let keyword_arguments = KeywordArguments {
range: keyword_arguments_range,
keywords: keywords.into(),
keywords: keywords.into_iter().collect::<Vec<_>>().into_boxed_slice(),
};
(Some(positional_arguments), Some(keyword_arguments))
@@ -231,7 +225,7 @@ pub(super) fn merge_class_def_args(
let (args, runtime_bases) = if let Some(positional_arguments) = positional_arguments {
positional_arguments.into_args_and_runtime_values()
} else {
(vec![].into_boxed_slice(), None)
(thin_vec::ThinVec::new(), None)
};
let keywords = if let Some(keyword_arguments) = keyword_arguments {
keyword_arguments.keywords
@@ -243,7 +237,7 @@ pub(super) fn merge_class_def_args(
node_index: Default::default(),
range: Default::default(), // TODO
args,
keywords: keywords.into(),
keywords: keywords.into_vec().into(),
runtime_args: None,
runtime_bases,
}))
crates/vm/src/stdlib/_ast/expression.rs
+17 -7
View File
@@ -866,7 +866,8 @@ fn expr_dict_comp_from_object_with_range(
"DictComp",
)?),
value: get_required_node_field(vm, source_file, object, "value", "DictComp")?,
generators: get_node_list_field(vm, source_file, object, "generators", "DictComp")?,
generators: get_node_list_field(vm, source_file, object, "generators", "DictComp")?
.into_boxed_slice(),
range,
})
}
@@ -888,8 +889,12 @@ impl Node for ast::ExprDictComp {
.unwrap();
dict.set_item("value", value.ast_to_object(vm, source_file), vm)
.unwrap();
dict.set_item("generators", generators.ast_to_object(vm, source_file), vm)
.unwrap();
dict.set_item(
"generators",
BoxedSlice(generators).ast_to_object(vm, source_file),
vm,
)
.unwrap();
node_add_location(&dict, range, vm, source_file);
node.into()
}
@@ -1082,14 +1087,17 @@ fn expr_compare_from_object_with_range(
object: &PyObject,
range: TextRange,
) -> PyResult<ast::ExprCompare> {
let left: ast::Expr = get_required_node_field(vm, source_file, object, "left", "Compare")?;
let comparators: Vec<Option<ast::Expr>> =
get_node_list_field(vm, source_file, object, "comparators", "Compare")?;
let (runtime_comparators, comparators) = runtime_expr_boxed_slice_from_values(comparators);
let mut operands = Vec::with_capacity(comparators.len() + 1);
operands.push(left);
operands.extend(comparators);
Ok(ast::ExprCompare {
node_index: Default::default(),
left: get_required_node_field(vm, source_file, object, "left", "Compare")?,
ops: get_node_boxed_slice_field(vm, source_file, object, "ops", "Compare")?,
comparators,
operands: operands.into_boxed_slice(),
range,
runtime_comparators,
})
@@ -1099,9 +1107,8 @@ impl Node for ast::ExprCompare {
fn ast_to_object(self, vm: &VirtualMachine, source_file: &SourceFile) -> PyObjectRef {
let Self {
node_index: _,
left,
ops,
comparators,
operands,
range,
runtime_comparators,
} = self;
@@ -1109,10 +1116,13 @@ impl Node for ast::ExprCompare {
.into_ref_with_type(vm, pyast::NodeExprCompare::static_type().to_owned())
.unwrap();
let dict = node.as_object().dict().unwrap();
let mut operands = operands.into_vec().into_iter();
let left = operands.next().expect("A comparison has a left operand");
dict.set_item("left", left.ast_to_object(vm, source_file), vm)
.unwrap();
dict.set_item("ops", BoxedSlice(ops).ast_to_object(vm, source_file), vm)
.unwrap();
let comparators: Box<[ast::Expr]> = operands.collect();
let comparators = runtime_comparators.map_or_else(
|| BoxedSlice(comparators).ast_to_object(vm, source_file),
|values| values.ast_to_object(vm, source_file),
crates/vm/src/stdlib/_ast/string.rs
+33 -18
View File
@@ -320,6 +320,35 @@ fn ruff_format_spec_to_joined_str(
}
}
fn dummy_string_literal() -> ast::StringLiteral {
ast::StringLiteral {
node_index: Default::default(),
range: Default::default(),
value: "".into(),
flags: ast::StringLiteralFlags::empty(),
}
}
fn dummy_fstring() -> ast::FString {
ast::FString {
node_index: Default::default(),
range: Default::default(),
elements: Default::default(),
flags: ast::FStringFlags::empty(),
}
}
fn take_fstring_part(part: ast::FStringPartMut<'_>) -> ast::FStringPart {
match part {
ast::FStringPartMut::Literal(literal) => {
ast::FStringPart::Literal(core::mem::replace(literal, dummy_string_literal()))
}
ast::FStringPartMut::FString(fstring) => {
ast::FStringPart::FString(core::mem::replace(fstring, dummy_fstring()))
}
}
}
fn ruff_fstring_element_to_ruff_fstring_part(
element: ast::InterpolatedStringElement,
) -> ast::FStringPart {
@@ -362,16 +391,9 @@ fn format_spec_expr_to_ruff_format_spec(
runtime_joined_str: _,
runtime_values: _,
} = fstring;
let default_part = ast::FStringPart::FString(ast::FString {
node_index: Default::default(),
range: Default::default(),
elements: Default::default(),
flags: ast::FStringFlags::empty(),
});
let mut elements = Vec::new();
for i in 0..value.as_slice().len() {
let part = core::mem::replace(value.iter_mut().nth(i).unwrap(), default_part.clone());
match part {
for part in value {
match take_fstring_part(part) {
ast::FStringPart::Literal(ast::StringLiteral {
range,
value,
@@ -718,16 +740,9 @@ pub(super) fn fstring_to_object(
.ast_to_object(vm, source_file);
}
let default_part = ast::FStringPart::FString(ast::FString {
node_index: Default::default(),
range: Default::default(),
elements: Default::default(),
flags: ast::FStringFlags::empty(),
});
let mut values = Vec::new();
for i in 0..value.as_slice().len() {
let part = core::mem::replace(value.iter_mut().nth(i).unwrap(), default_part.clone());
match part {
for part in &mut value {
match take_fstring_part(part) {
ast::FStringPart::Literal(literal) => {
values.push(JoinedStrPart::Constant(Constant::new_str(
rustpython_codegen::string_literal_part_value(source_file, &literal),
crates/vm/src/stdlib/_ast/validate.rs
+5 -4
View File
@@ -616,10 +616,11 @@ fn validate_expr(vm: &VirtualMachine, expr: &ast::Expr, ctx: ast::ExprContext) -
validate_expr(vm, &await_expr.value, ast::ExprContext::Load)
}
ast::Expr::Compare(compare) => {
if compare.comparators.is_empty() {
let comparators = compare.operands.get(1..).unwrap_or(&[]);
if comparators.is_empty() {
return Err(vm.new_value_error("Compare with no comparators"));
}
if compare.comparators.len() != compare.ops.len() {
if comparators.len() != compare.ops.len() {
return Err(vm.new_value_error(
"Compare has a different number of comparators and operands",
));
@@ -629,8 +630,8 @@ fn validate_expr(vm: &VirtualMachine, expr: &ast::Expr, ctx: ast::ExprContext) -
compare.runtime_comparators.as_ref(),
ast::ExprContext::Load,
)?;
validate_exprs(vm, &compare.comparators, ast::ExprContext::Load, false)?;
validate_expr(vm, &compare.left, ast::ExprContext::Load)
validate_exprs(vm, comparators, ast::ExprContext::Load, false)?;
validate_expr(vm, compare.first_operand(), ast::ExprContext::Load)
}
ast::Expr::Call(call) => {
validate_expr(vm, &call.func, ast::ExprContext::Load)?;
crates/vm/src/vm/compile.rs
+3 -3
View File
@@ -1506,9 +1506,9 @@ mod escape_warnings {
}
// F-string literals — tokenizer + string_parser paths
ast::Expr::FString(fstring_expr) => {
for part in fstring_expr.value.as_slice() {
for part in &fstring_expr.value {
match part {
ast::FStringPart::Literal(string_lit) => {
ast::FStringPartRef::Literal(string_lit) => {
// Plain string part in f-string concatenation
if !matches!(
string_lit.flags.prefix(),
@@ -1517,7 +1517,7 @@ mod escape_warnings {
self.check_quoted_literal(string_lit.range, false);
}
}
ast::FStringPart::FString(fstring) => {
ast::FStringPartRef::FString(fstring) => {
if matches!(
fstring.flags.prefix(),
ast::str_prefix::FStringPrefix::Raw { .. }
crates/vm/src/vm/vm_new.rs
+83 -511
View File
@@ -1,7 +1,5 @@
#[cfg(feature = "parser")]
use ruff_python_ast::token::TokenKind;
use ruff_python_parser::{InterpolatedStringErrorType, LexicalErrorType, ParseErrorType};
use ruff_python_parser::{LexicalErrorType, ParseErrorType};
use rustpython_common::wtf8::Wtf8Buf;
use rustpython_compiler_core::SourceLocation;
@@ -9,7 +7,7 @@ use rustpython_compiler_core::SourceLocation;
use core::ops::RangeInclusive;
#[cfg(feature = "parser")]
use rustpython_compiler::{CompileError, ParseError, is_blank_python_source};
use rustpython_compiler::is_blank_python_source;
use crate::{
AsObject, Py, PyObject, PyObjectRef, PyPayload, PyRef, PyResult,
@@ -49,262 +47,6 @@ macro_rules! define_exception_fn {
};
}
#[derive(Clone, Debug)]
struct SyntaxErrorInfo {
msg: String,
narrow_caret: bool,
}
impl SyntaxErrorInfo {
#[must_use]
const fn new(msg: String, narrow_caret: bool) -> Self {
Self { msg, narrow_caret }
}
#[cfg(feature = "parser")]
#[must_use]
const fn handle_expected_token(expected: TokenKind, found: TokenKind) -> &'static str {
match (expected, found) {
(TokenKind::Colon, TokenKind::Newline) => "expected ':'",
(TokenKind::Lpar, _) => "expected '('",
(TokenKind::Else, y) if !matches!(y, TokenKind::Colon) => {
"expected 'else' after 'if' expression"
}
_ => "invalid syntax",
}
}
#[cfg(feature = "parser")]
fn analyze_compile_error(&mut self, compile_error: &CompileError) {
let CompileError::Parse(ParseError {
error, location, ..
}) = compile_error
else {
return;
};
let msg = match error {
ParseErrorType::FStringError(InterpolatedStringErrorType::UnterminatedString)
| ParseErrorType::Lexical(LexicalErrorType::FStringError(
InterpolatedStringErrorType::UnterminatedString,
)) => "unterminated f-string literal".into(),
ParseErrorType::TStringError(InterpolatedStringErrorType::UnterminatedString)
| ParseErrorType::Lexical(LexicalErrorType::TStringError(
InterpolatedStringErrorType::UnterminatedString,
)) => "unterminated t-string literal".into(),
ParseErrorType::FStringError(
InterpolatedStringErrorType::UnterminatedTripleQuotedString,
)
| ParseErrorType::Lexical(LexicalErrorType::FStringError(
InterpolatedStringErrorType::UnterminatedTripleQuotedString,
)) => "unterminated triple-quoted f-string literal".into(),
ParseErrorType::TStringError(
InterpolatedStringErrorType::UnterminatedTripleQuotedString,
)
| ParseErrorType::Lexical(LexicalErrorType::TStringError(
InterpolatedStringErrorType::UnterminatedTripleQuotedString,
)) => "unterminated triple-quoted t-string literal".into(),
// ruff already prefixes these with `f-string: ` / `t-string: `, matching CPython.
// It quotes braces with backticks where CPython uses single quotes.
ParseErrorType::FStringError(_)
| ParseErrorType::TStringError(_)
| ParseErrorType::Lexical(
LexicalErrorType::FStringError(_) | LexicalErrorType::TStringError(_),
) => self.msg.replace('`', "'"),
ParseErrorType::UnexpectedExpressionToken => "invalid syntax".into(),
ParseErrorType::ExpectedToken { expected, found } => {
Self::handle_expected_token(*expected, *found).into()
}
ParseErrorType::InvalidStarredExpressionUsage => {
self.narrow_caret = true;
"invalid syntax".into()
}
ParseErrorType::InvalidDeleteTarget => "invalid syntax".into(),
ParseErrorType::Lexical(LexicalErrorType::LineContinuationError) => {
"unexpected character after line continuation character".into()
}
ParseErrorType::Lexical(LexicalErrorType::UnclosedStringError) => {
format!(
"unterminated string literal (detected at line {})",
location.line
)
}
ParseErrorType::EmptyTypeParams => "Type parameter list cannot be empty".into(),
ParseErrorType::InvalidStarPatternUsage => {
self.narrow_caret = true;
"cannot use starred expression here".into()
}
ParseErrorType::ExpectedKeywordParam => "named arguments must follow bare *".into(),
ParseErrorType::EmptyImportNames => "Expected one or more names after 'import'".into(),
ParseErrorType::UnparenthesizedGeneratorExpression => {
"Generator expression must be parenthesized".into()
}
ParseErrorType::NonDefaultParamAfterDefaultParam => {
"parameter without a default follows parameter with a default".into()
}
ParseErrorType::VarParameterWithDefault => {
"var-positional argument cannot have default value".into()
}
ParseErrorType::PositionalAfterKeywordArgument => {
"positional argument follows keyword argument".into()
}
ParseErrorType::PositionalAfterKeywordUnpacking => {
"positional argument follows keyword argument unpacking".into()
}
ParseErrorType::InvalidArgumentUnpackingOrder => {
"iterable argument unpacking follows keyword argument unpacking".into()
}
ParseErrorType::ParamAfterVarKeywordParam => {
"arguments cannot follow var-keyword argument".into()
}
ParseErrorType::InvalidAnnotatedAssignmentTarget => {
"illegal target for annotation".into()
}
ParseErrorType::Lexical(LexicalErrorType::UnrecognizedToken { .. })
| ParseErrorType::SimpleStatementsOnSameLine
| ParseErrorType::SimpleAndCompoundStatementOnSameLine
| ParseErrorType::ExpectedExpression => "invalid syntax".into(),
ParseErrorType::OtherError(s) if s.starts_with("Expected an identifier") => {
"invalid syntax".into()
}
// What the parser says when it cannot continue the list it is
// recovering; each of these situations is a plain "invalid syntax".
ParseErrorType::OtherError(s)
if matches!(
s.as_str(),
"Expected a statement"
| "Expected an `elif` or `else` clause, or the end of the `if` statement."
| "Expected an `except` or `finally` clause or the end of the `try` statement."
| "The keyword is not allowed as a variable declaration name"
| "Expected an assignment target"
| "Expected a type parameter or the end of the type parameter list"
| "Expected an import name or a ')'"
| "Expected an import name"
| "Expected an expression or the end of the slice list"
| "Expected an expression or a ']'"
| "Expected an expression or a '}'"
| "Expected an expression or a ')'"
| "Expected an expression"
| "Expected a pattern or the end of the sequence pattern"
| "Expected a mapping pattern or the end of the mapping pattern"
| "Expected a pattern or a ')'"
| "Expected a delete target"
| "Expected a parameter or the end of the parameter list"
| "Expected an expression or the end of the with item list"
) =>
{
"invalid syntax".into()
}
ParseErrorType::OtherError(s)
if s.eq_ignore_ascii_case(
"bytes literal cannot be mixed with non-bytes literals",
) =>
{
"cannot mix bytes and nonbytes literals".into()
}
ParseErrorType::OtherError(s)
if s.eq_ignore_ascii_case("positional patterns cannot follow keyword patterns") =>
{
"positional patterns follow keyword patterns".into()
}
ParseErrorType::OtherError(s)
if s.eq_ignore_ascii_case("boolean 'not' expression cannot be used here") =>
{
"'not' after an operator must be parenthesized".into()
}
ParseErrorType::OtherError(s)
if s.eq_ignore_ascii_case("trailing comma not allowed") =>
{
"trailing comma not allowed without surrounding parentheses".into()
}
ParseErrorType::OtherError(s)
if s.eq_ignore_ascii_case(
"multiple exception types must be parenthesized when using `as`",
) =>
{
"multiple exception types must be parenthesized when using 'as'".into()
}
ParseErrorType::OtherError(s)
if s.eq_ignore_ascii_case(
"position-only parameter separator not allowed as first parameter",
) =>
{
"at least one argument must precede /".into()
}
ParseErrorType::OtherError(s)
if s.eq_ignore_ascii_case("only one '/' separator allowed") =>
{
"/ may appear only once".into()
}
ParseErrorType::OtherError(s)
if s.eq_ignore_ascii_case("'/' parameter must appear before '*' parameter") =>
{
"/ must be ahead of *".into()
}
ParseErrorType::OtherError(s)
if s.eq_ignore_ascii_case("expected `except` or `finally` after `try` block") =>
{
"expected 'except' or 'finally' block".into()
}
ParseErrorType::OtherError(s)
if s.eq_ignore_ascii_case("only one '*' parameter allowed") =>
{
"* argument may appear only once".into()
}
ParseErrorType::OtherError(s)
if s.eq_ignore_ascii_case(
r#"cannot have both 'except' and 'except*' on the same 'try'"#,
) =>
{
r#"cannot have both 'except' and 'except*' on the same 'try'"#.into()
}
_ => return,
};
self.msg = msg;
}
}
/// Collection of object creation helpers
impl VirtualMachine {
/// Create a new python object
@@ -703,38 +445,6 @@ impl VirtualMachine {
.expect("UnicodeEncodeError constructor")
}
#[cfg(feature = "parser")]
fn source_has_mixed_tabs_and_spaces(source: Option<&str>, error_line: usize) -> bool {
source.is_some_and(|source| {
let mut has_space_indent = false;
let mut has_tab_indent = false;
for (i, line) in source.lines().enumerate() {
if i + 1 > error_line {
break;
}
let rest = line.trim_start_matches([' ', '\t']);
// Blank and comment-only lines do not participate in indent.
if rest.is_empty() || rest.starts_with('#') {
continue;
}
let indent = &line.as_bytes()[..line.len() - rest.len()];
if indent.is_empty() {
continue;
}
if indent.contains(&b' ') && indent.contains(&b'\t') {
return true;
}
if indent.contains(&b' ') {
has_space_indent = true;
}
if indent.contains(&b'\t') {
has_tab_indent = true;
}
}
has_space_indent && has_tab_indent
})
}
// TODO: don't take ownership should make the success path faster
pub fn new_key_error(&self, obj: PyObjectRef) -> PyBaseExceptionRef {
let key_error = self.ctx.exceptions.key_error.to_owned();
@@ -769,25 +479,27 @@ impl VirtualMachine {
let syntax_error_type = match &error {
#[cfg(feature = "parser")]
crate::compiler::CompileError::Parse(rustpython_compiler::ParseError {
error:
ruff_python_parser::ParseErrorType::Lexical(
ruff_python_parser::LexicalErrorType::IndentationError,
)
| ruff_python_parser::ParseErrorType::UnexpectedIndentation,
location,
error: ParseErrorType::Lexical(LexicalErrorType::TabError),
..
}) => {
if Self::source_has_mixed_tabs_and_spaces(source, location.line.get()) {
self.ctx.exceptions.tab_error
} else {
self.ctx.exceptions.indentation_error
}
}
}) => self.ctx.exceptions.tab_error,
#[cfg(feature = "parser")]
crate::compiler::CompileError::Parse(rustpython_compiler::ParseError {
error:
ParseErrorType::Lexical(
LexicalErrorType::IndentationError | LexicalErrorType::TooDeepIndentation,
)
| ParseErrorType::UnexpectedIndentation,
..
}) => self.ctx.exceptions.indentation_error,
#[cfg(feature = "parser")]
crate::compiler::CompileError::Parse(rustpython_compiler::ParseError {
error:
ruff_python_parser::ParseErrorType::Lexical(
ruff_python_parser::LexicalErrorType::Eof,
ruff_python_parser::LexicalErrorType::Eof
| ruff_python_parser::LexicalErrorType::UnclosedBracket {
incomplete: true,
..
},
),
..
}) => incomplete_or_syntax(allow_incomplete),
@@ -807,26 +519,31 @@ impl VirtualMachine {
error:
ruff_python_parser::ParseErrorType::Lexical(
ruff_python_parser::LexicalErrorType::FStringError(
ruff_python_parser::InterpolatedStringErrorType::UnterminatedTripleQuotedString,
ruff_python_parser::InterpolatedStringErrorType::UnterminatedTripleQuotedString { .. },
)
| ruff_python_parser::LexicalErrorType::TStringError(
ruff_python_parser::InterpolatedStringErrorType::UnterminatedTripleQuotedString,
ruff_python_parser::InterpolatedStringErrorType::UnterminatedTripleQuotedString { .. },
),
),
..
}) => incomplete_or_syntax(allow_incomplete),
#[cfg(feature = "parser")]
crate::compiler::CompileError::Parse(rustpython_compiler::ParseError {
error:
ruff_python_parser::ParseErrorType::Lexical(
ruff_python_parser::LexicalErrorType::UnclosedStringError,
),
error: ruff_python_parser::ParseErrorType::ExpectedIndentedBlock { .. },
raw_location,
..
}) => {
if allow_incomplete {
incomplete_or_syntax(source.is_some_and(unclosed_string_is_incomplete))
// The block can still follow when the error is found at whitespace, that is, at
// the end of the input.
let at_end = source.is_some_and(|source| {
source
.get(raw_location.start().to_usize()..raw_location.end().to_usize())
.is_some_and(|text| text.chars().all(|c| c.is_ascii_whitespace()))
});
if allow_incomplete && at_end {
self.ctx.exceptions.incomplete_input_error
} else {
self.ctx.exceptions.syntax_error
self.ctx.exceptions.indentation_error
}
}
#[cfg(feature = "parser")]
@@ -835,39 +552,7 @@ impl VirtualMachine {
raw_location,
..
}) => {
if s.starts_with("Expected an indented block after")
|| s.starts_with("expected an indented block after")
{
if allow_incomplete {
// Check that all chars in the error are whitespace, if so, the source is
// incomplete. Otherwise, we've found code that might violates
// indentation rules.
let mut is_incomplete = true;
if let Some(source) = source {
let start = raw_location.start().to_usize();
let end = raw_location.end().to_usize();
let mut iter = source.chars();
iter.nth(start);
for _ in start..end {
if let Some(c) = iter.next() {
if !c.is_ascii_whitespace() {
is_incomplete = false;
}
} else {
break;
}
}
}
if is_incomplete {
self.ctx.exceptions.incomplete_input_error
} else {
self.ctx.exceptions.indentation_error // not syntax_error
}
} else {
self.ctx.exceptions.indentation_error
}
} else if allow_incomplete
if allow_incomplete
&& (s == "incomplete input"
|| (s == "unexpected EOF while parsing"
&& source.is_some_and(|source| {
@@ -908,52 +593,12 @@ impl VirtualMachine {
source.and_then(|src| get_statement(src, error.location()))
};
let mut msg = error.to_string();
if !msg.starts_with("Exceeds the limit ")
&& !msg.starts_with("Did you mean ")
&& !msg.starts_with("Invalid star expression")
&& !msg.starts_with("Function parameters cannot be parenthesized")
&& !msg.starts_with("Lambda expression parameters cannot be parenthesized")
&& !msg.starts_with("Cannot have two type comments on def")
&& !msg.starts_with("Variable annotation syntax is")
&& !msg.starts_with("The '@' operator is")
&& !msg.starts_with("Async functions are")
&& !msg.starts_with("Async comprehensions are")
&& !msg.starts_with("Async for loops are")
&& !msg.starts_with("Async with statements are")
&& !msg.starts_with("Exception groups are")
&& !msg.starts_with("Positional-only parameters are")
&& !msg.starts_with("Pattern matching is")
&& !msg.starts_with("Type statement is")
&& !msg.starts_with("Type parameter lists are")
&& !msg.starts_with("Type parameter defaults are")
&& !msg.starts_with("Assignment expressions are")
&& !msg.starts_with("Await expressions are")
&& !msg.starts_with("Underscores in numeric literals are")
&& !msg.starts_with("Missing parentheses")
&& let Some(msg) = msg.get_mut(..1)
{
msg.make_ascii_lowercase();
}
cfg_select! {
feature = "parser" => {
let mut syntax_error_info = SyntaxErrorInfo::new(msg, false);
syntax_error_info.analyze_compile_error(error);
}
_ => {
let syntax_error_info = SyntaxErrorInfo::new(msg, false);
}
let msg = if syntax_error_type.is(self.ctx.exceptions.incomplete_input_error) {
String::from("incomplete input")
} else {
error.to_string()
};
if syntax_error_type.is(self.ctx.exceptions.tab_error) {
syntax_error_info.msg =
String::from("inconsistent use of tabs and spaces in indentation");
} else if syntax_error_type.is(self.ctx.exceptions.incomplete_input_error) {
syntax_error_info.msg = String::from("incomplete input");
}
let SyntaxErrorInfo { msg, narrow_caret } = syntax_error_info;
let unterminated_triple_quoted_string = msg.starts_with("unterminated triple-quoted");
let unexpected_eof_error = msg == "unexpected EOF while parsing";
if unterminated_triple_quoted_string
@@ -970,6 +615,50 @@ impl VirtualMachine {
|| msg.starts_with("except expressions without parentheses are")
|| msg.starts_with("Pattern matching is");
let line_end_binary_operator_error = msg.starts_with("The '@' operator is");
// Tokenizer errors other than unexpected EOF end where the tokenizer stopped, which is
// reported as end offset 0 or -1, or where they start.
let tokenizer_end_offset = cfg_select! {
feature = "parser" => {
match error {
crate::compiler::CompileError::Parse(rustpython_compiler::ParseError {
error: ParseErrorType::Lexical(
LexicalErrorType::TabError
| LexicalErrorType::TooDeepIndentation
| LexicalErrorType::LineContinuationError
| LexicalErrorType::UnclosedBracket { .. },
),
..
}) => Some(0),
crate::compiler::CompileError::Parse(rustpython_compiler::ParseError {
error: ParseErrorType::Lexical(LexicalErrorType::IndentationError)
| ParseErrorType::UnexpectedIndentation,
..
}) => Some(-1),
// The span runs from the last character of the key to the end of the line.
crate::compiler::CompileError::Parse(rustpython_compiler::ParseError {
error: ParseErrorType::ExpectedColonAfterDictionaryKey,
..
}) => Some(0),
// Other tokenizer errors end where they start, except those reported
// over a range.
crate::compiler::CompileError::Parse(rustpython_compiler::ParseError {
error: ParseErrorType::Lexical(lexical),
..
}) if lexical.is_tokenizer_error()
&& !matches!(
lexical,
LexicalErrorType::Eof
| LexicalErrorType::LeadingZerosInDecimalInteger
| LexicalErrorType::IncompatibleStringPrefixes { .. }
) =>
{
Some(error.python_location().1 as isize)
}
_ => None,
}
}
_ => None,
};
let unclosed_bracket_error = cfg_select! {
feature = "parser" => {
matches!(
@@ -1011,15 +700,14 @@ impl VirtualMachine {
.is_some_and(|ch| ch.is_ascii_whitespace()));
let (end_lineno, end_offset) = if no_end_offset {
(end_lineno, -1)
} else if let Some(end_offset) = tokenizer_end_offset {
(end_lineno, end_offset)
} else if unclosed_bracket_error {
// The bracket that was never closed is marked where it opened,
// and the span stops there.
(end_lineno, 0)
} else if line_end_binary_operator_error && end_offset == offset_raw {
(end_lineno, (end_offset + 1) as isize)
} else if narrow_caret {
let (l, o) = error.python_location();
(l, (o + 1) as isize)
} else {
(end_lineno, end_offset as isize)
};
@@ -1207,119 +895,3 @@ impl VirtualMachine {
.upcast()
}
}
#[cfg(feature = "parser")]
enum QuotedStringScan {
Closed(usize),
Unclosed {
triple: bool,
unescaped_newline: bool,
},
}
/// An unclosed string is incomplete when more input could still finish it.
/// A non-triple string that already hit an unescaped newline is a hard error.
#[cfg(feature = "parser")]
fn unclosed_string_is_incomplete(source: &str) -> bool {
let bytes = source.as_bytes();
let mut index = 0;
while index < bytes.len() {
match bytes[index] {
b'#' => {
while index < bytes.len() && bytes[index] != b'\n' {
index += 1;
}
}
b'\'' | b'"' => match scan_quoted_string_for_incomplete(bytes, index) {
QuotedStringScan::Closed(end) => index = end,
QuotedStringScan::Unclosed {
triple,
unescaped_newline,
} => {
// Single-quoted f/t-strings never set E_EOLS.
if !triple && interpolated_string_prefix_at(bytes, index) {
return false;
}
return triple || !unescaped_newline;
}
},
_ => index += 1,
}
}
false
}
#[cfg(feature = "parser")]
fn interpolated_string_prefix_at(bytes: &[u8], quote: usize) -> bool {
let Some(&prev) = quote.checked_sub(1).and_then(|index| bytes.get(index)) else {
return false;
};
let lower = prev.to_ascii_lowercase();
let marker_index = if matches!(lower, b'f' | b't') {
if quote >= 2 && bytes[quote - 2].eq_ignore_ascii_case(&b'r') {
quote - 2
} else {
quote - 1
}
} else if lower == b'r'
&& quote >= 2
&& matches!(bytes[quote - 2].to_ascii_lowercase(), b'f' | b't')
{
quote - 2
} else {
return false;
};
marker_index == 0 || !identifier_continue_before(bytes, marker_index)
}
#[cfg(feature = "parser")]
fn identifier_continue_before(bytes: &[u8], index: usize) -> bool {
if index == 0 {
return false;
}
if bytes[index - 1].is_ascii() {
return bytes[index - 1] == b'_' || bytes[index - 1].is_ascii_alphanumeric();
}
let mut start = index - 1;
while start > 0 && bytes[start] & 0b1100_0000 == 0b1000_0000 {
start -= 1;
}
::core::str::from_utf8(&bytes[start..index])
.ok()
.and_then(|text| text.chars().next_back())
.is_some_and(|ch| ch == '_' || ch.is_alphanumeric())
}
#[cfg(feature = "parser")]
fn scan_quoted_string_for_incomplete(bytes: &[u8], quote_index: usize) -> QuotedStringScan {
let quote = bytes[quote_index];
let triple =
bytes.get(quote_index + 1) == Some(&quote) && bytes.get(quote_index + 2) == Some(&quote);
let mut index = quote_index + if triple { 3 } else { 1 };
while index < bytes.len() {
if bytes[index] == b'\\' {
index = (index + 2).min(bytes.len());
continue;
}
if triple {
if bytes.get(index) == Some(&quote)
&& bytes.get(index + 1) == Some(&quote)
&& bytes.get(index + 2) == Some(&quote)
{
return QuotedStringScan::Closed(index + 3);
}
} else if bytes[index] == quote {
return QuotedStringScan::Closed(index + 1);
} else if bytes[index] == b'\n' {
return QuotedStringScan::Unclosed {
triple: false,
unescaped_newline: true,
};
}
index += 1;
}
QuotedStringScan::Unclosed {
triple,
unescaped_newline: false,
}
}
extra_tests/snippets/syntax_invalid.py
+43
View File
@@ -77,3 +77,46 @@ src = """
from __future__ import print_function
"""
compile(src, 'test.py', 'exec')
# CPython reports the unparenthesized `except ... as` error with a range that starts at the
# first exception type and stops just before the `:` closing the clause. `end_offset` is an
# exclusive 1-based *character* column, so that end lands on the column the `:` sits on.
#
# The cases below pin the parts that are easy to get wrong: the column counts characters
# rather than UTF-8 bytes, and an explicit line join may sit anywhere between the exception
# types and the `:`. Values verified against CPython 3.14.
except_as_ranges = [
# (clause, (lineno, offset), (end_lineno, end_offset))
("except A, B as e:", (3, 8), (3, 17)),
("except A, B as e :", (3, 8), (3, 18)),
("except A, B as e :", (3, 8), (3, 20)),
("except A, B as e\t:", (3, 8), (3, 18)),
("except A, B, C as blech:", (3, 8), (3, 24)),
("except* A, B, C as e:", (3, 9), (3, 21)),
# A non-ASCII name must not push the column right by its extra UTF-8 bytes.
("except Ä, B as e:", (3, 8), (3, 17)),
("except 사과, B as e:", (3, 8), (3, 18)),
("except A, B as 오:", (3, 8), (3, 17)),
# An explicit line join may precede or follow `as`, or close the clause on its own line.
("except A, B as exc\\\n :", (3, 8), (4, 5)),
("except A, B \\\nas exc:", (3, 8), (4, 7)),
("except A, \\\nB as exc:", (3, 8), (4, 9)),
("except A, B as \\\nexc:", (3, 8), (4, 4)),
]
for clause, start, end in except_as_ranges:
src = "try:\n pass\n%s\n pass\n" % clause
with assert_raises(SyntaxError) as ae:
compile(src, "test.py", "exec")
exc = ae.exception
assert exc.msg == "multiple exception types must be parenthesized when using 'as'", (
clause,
exc.msg,
)
assert (exc.lineno, exc.offset) == start, (clause, (exc.lineno, exc.offset), start)
assert (exc.end_lineno, exc.end_offset) == end, (
clause,
(exc.end_lineno, exc.end_offset),
end,
)
src/shell.rs
+5 -16
View File
@@ -60,29 +60,18 @@ fn shell_exec(
error: ParseErrorType::Lexical(
LexicalErrorType::Eof
| LexicalErrorType::FStringError(
InterpolatedStringErrorType::UnterminatedTripleQuotedString,
InterpolatedStringErrorType::UnterminatedTripleQuotedString { .. },
)
| LexicalErrorType::UnclosedStringError {
triple_quoted: true,
..
}
),
..
}))
) {
return ShellExecResult::ContinueLine;
}
if let VmCompileError::Compile(CompileError::Parse(ParseError {
error: ParseErrorType::Lexical(LexicalErrorType::UnclosedStringError),
raw_location,
..
})) = &err
{
let loc = raw_location.start().to_usize();
let mut iter = source.chars();
if let Some(quote) = iter.nth(loc)
&& iter.next() == Some(quote)
&& iter.next() == Some(quote)
{
return ShellExecResult::ContinueLine;
}
}
// An unfinished suite is _IncompleteInputError, not IndentationError.
let exc = err.into_pyexception_maybe_incomplete(vm, Some(source), !empty_line_given);