## Summary
`match` is a soft keyword. It can name a variable, a function, or the
target of an annotated assignment, including after a semicolon.
```python
first(); match = value
first(); match(value)
first(); match[0]: int
```
The parser reported a compound-statement error as soon as it saw the
`match` token. We now check the preceding separator when parsing a match
statement. If the token turns out to be an identifier, speculative
parsing discards the diagnostic. A genuine `match` statement on the same
logical line still produces the original error.
## Summary
A bare `raise` can end at a semicolon. For example, Python accepts this
handler and re-raises the current exception.
```python
try:
work()
except ValueError:
raise; report()
```
The parser tried to read the semicolon as the start of an exception
expression and reported a syntax error. We now recognize both a newline
and a semicolon as terminators for a bare `raise`.
Summary
--
While reviewing #28231, Codex found a preexisting bug shared with our
unary minus handling. Namely, CPython emits an error for code like this,
where the unary expression is part of a longer expression:
```pycon
>>> match foo:
... case -1**2: ...
...
File "<python-input-0>", line 2
case -1**2: ...
^^
SyntaxError: invalid syntax
```
but Ruff does not:
```console
$ ruff check --ignore ALL - <<EOF
match foo:
case -1**2: ...
EOF
All checks passed!
```
We now verify that the parsed unary expression is a number literal and
emit an error if not.
Test Plan
--
New inline snapshots
Summary
--
As added in https://github.com/python/cpython/pull/148566, CPython now
allows unary plus in match
patterns, mirroring the support for unary minus. We already had a
special case for this for improved
error reporting and recovery, so I just swapped out the `add_error` call
for
`add_unsupported_syntax_error` with a new error kind.
Part of https://github.com/astral-sh/ruff/issues/28206
Test Plan
--
New inline parser tests
## Summary
Reject unparenthesized `yield` expressions after the first element of an
expression list:
```python
def generator():
1, yield 2
```
The parser previously reused the yield-enabled expression context for
every tuple element. Python's grammar only allows the initial
simple-statement alternative to be a `yield` expression; later
comma-separated elements use the ordinary starred-expression grammar.
This clears the yield allowance for subsequent elements while preserving
valid parenthesized nested yields.
## Summary
Reject bare starred targets in comprehensions:
```python
[item for *items in source]
```
The semantic syntax checker already rejects a bare starred target in an
ordinary `for` statement. This applies the same validation to
comprehension clauses while preserving valid tuple and list unpacking
targets such as `for *items, in source`.
## Summary
Reject `__debug__` as a lambda parameter name:
```python
lambda __debug__: 0
```
Function parameters already use the semantic syntax check that prevents
assignments to `__debug__`, but lambda parameters only checked for
duplicate names. This applies the existing identifier validation
consistently to both forms.
## Summary
Reject parenthesized star imports such as:
```python
from module import (*)
```
Python only permits a bare star in `from` imports. The parser previously
accepted the parenthesized form because star aliases were parsed
independently of the surrounding import-list delimiters.
This adds the missing parser validation while preserving ordinary
parenthesized imports and bare star imports.
## Summary
Mapping-pattern duplicate detection and F601 compare keys through
`HashableExpr`, but numeric normalization previously only recognized
values equivalent to `True` or `False`. This meant that other values
Python treats as the same dictionary key were considered distinct:
```python
match value:
case {2: first, 2.0: second}: ...
```
This adds a dedicated numeric representation for integer, float,
boolean, and complex literals so that exactly integral floats compare
with equivalent integers, including signed forms and zero-imaginary
complex values. Tuple normalization is limited to statically known
values so dynamic tuple elements do not introduce false-positive F601
diagnostics.
Numeric normalization uses the existing `u64` integer representation.
Larger integers retain structural comparison, avoiding
arbitrary-precision conversion in parser and linter hot paths.
## Summary
Python reserves `_` for wildcard patterns and does not allow it as the
binding target in either of these forms:
```python
match value:
case 1 as _:
pass
case {**_}:
pass
```
We currently recover an AST for both forms without reporting a parser
error. Both are grammar errors in CPython, so this change validates the
capture target while parsing `as` and mapping patterns and reports
`cannot use '_' as a target` on the underscore.
We previously tried to address the mapping-pattern case in
[#6838](https://github.com/astral-sh/ruff/pull/6838) by representing the
rest target as a general `PatternMatchAs` node. This keeps the existing
`Identifier` representation and recovered AST, including statements
after the invalid match, while rejecting the invalid target at the
parser boundary.
## Summary
Class definitions do not allow a bare generator expression as a base,
even though the same syntax is valid as the sole argument in a function
call. We currently parse the class form without an error:
```python
class C(base for base in bases):
pass
```
This change reuses the existing unparenthesized-generator diagnostic for
class arguments while continuing to accept explicitly parenthesized or
unpacked generator expressions:
```python
class C((base for base in bases)): ...
class C(*(base for base in bases)): ...
```
## Summary
Mapping patterns reject duplicate literal keys, but duplicate detection
compared AST shapes instead of Python hash equality. As a result, values
that Python treats as the same key were accepted:
```python
match value:
case {1.0: first, True: second}: ...
```
This switches mapping-key detection to `HashableExpr` and extends
boolean-equivalent normalization through signed numeric expressions and
complex values with a zero imaginary part. We now report duplicate-key
syntax errors for `0`/`False`, `1.0`/`True`, `-0`/`False`, and `1 +
0j`/`True`, matching CPython.
## Summary
Function type parameter lists did not run the same default-order
validation as class and type-alias parameter lists. As a result, we
accepted functions with a required type parameter after one with a
default:
```python
def f[T = int, U](): ...
```
This reuses the existing type-parameter ordering check for function
definitions, so we now report the same semantic syntax error that we
already emit for equivalent classes and type aliases.
## Summary
Python permits at most one starred name in a sequence pattern, but our
parser currently accepts multiple:
```python
match value:
case [*head, middle, *tail]:
pass
```
This tracks whether a sequence pattern already contains a star and
reports `invalid-syntax` on each later starred name. The diagnostic
points to the additional starred pattern and uses CPython's error
message, while keeping the recovered AST available to downstream tools.
Fixes#25098.
This adds full support for PEP 798 through the parser, linter,
formatter, and type checker.
There are a lot of changes but they are mostly straightforward. Let me
know though if it's better to split this up.
## Summary
Before Python 3.12, a replacement field in an f-string can span multiple
lines only if the outer f-string is triple-quoted. This was relaxed in
Python 3.12, but we weren't rejecting these as syntactically invalid on
earlier versions.
Closes https://github.com/astral-sh/ruff/issues/24348.
## Summary
We now reject cases like:
- `lazy import ...` inside functions, including `async def`
- `lazy from ... import ...` inside functions, including `async def`
- `lazy import ...` and `lazy from ... import ...` inside class bodies
- `lazy import ...` and `lazy from ... import ...` anywhere inside a
`try` statement, including `except` / `except*`
- `lazy from ... import *` anywhere
- `lazy from __future__ import ...` anywhere
A follow-up to #23755.
See: https://github.com/astral-sh/ruff/issues/21305.
## Summary
Previously, `Parameters::visit_source_order` used `ParametersIterator`
which visits in type-based order. For well-formed Python code, this
matches source order; but error recovery can produce ASTs like `def
foo(**kwargs, a):`.
## Summary
Fixes#22528.
The parser incorrectly treats `match[0]: int` and `match [x, y, z]:
{dict}` as the start of a `match` statement, producing a false syntax
error. The issue is that the parser checks only for a colon after the
subject expression to confirm a match statement, but annotated
assignments with `match` as a variable name also have a colon.
This PR adds a `self.peek() == TokenKind::Newline` guard to the colon
check, so that `match` is only treated as a keyword when the colon is
followed by a newline (which is required for a match statement's body).
When the colon is followed by something else (like a type annotation
value), it falls through to the soft-keyword-as-identifier path.
## Test Plan
All 42 existing match-related parser tests pass. No snapshot changes
needed. Clippy clean.
Reproduction from the issue:
```python
# These should parse without error:
match[0]: int
match [x, y, z]: {0: "a", 1: "b"}
```
## Summary
Fixes a panic when parsing IPython escape commands with `Help` kind
(`?`) in expression contexts. The parser now reports an error instead of
panicking.
Fixes#21465.
## Problem
The parser panicked with `unreachable!()` in
`parse_ipython_escape_command_expression` when encountering escape
commands with `Help` kind (`?`) in expression contexts, where only
`Magic` (`%`) and `Shell` (`!`) are allowed.
## Approach
Replaced the `unreachable!()` panic with error handling that adds a
`ParseErrorType::OtherError` and continues parsing, returning a valid
AST node with the error attached.
## Test Plan
Added `test_ipython_escape_command_in_with_statement` and
`test_ipython_help_escape_command_as_expression` to verify the fix.
---------
Co-authored-by: Dhruv Manilawala <dhruvmanila@gmail.com>
Summary
--
Fixes#21360 by using the union of names instead of overwriting them, as
Micha suggested originally on #21104.
This avoids overwriting the `n` name in the `Subscript` by the empty set
of names visited in the nested OR pattern before visiting the other arm
of the outer OR pattern.
Test Plan
--
A new inline test case taken from the issue
## Summary
Fixes#21101 by storing the child visitor's names in the parent visitor.
This makes sure that `visitor.names` on line 1818 isn't empty after we
visit a nested OR pattern.
## Test Plan
New inline test cases derived from the issue,
[playground](https://play.ruff.rs/7b6439ac-ee8f-4593-9a3e-c2aa34a595d0)
Summary
--
This PR fixes the issue I added in #20867 and noticed in #20930. Cases
like this
cause an error on any Python version:
```py
f"{1:""}"
```
which gave me a false sense of security before. Cases like this are
still
invalid only before 3.12 and weren't flagged after the changes in
#20867:
```py
f'{1: abcd "{'aa'}" }'
# ^ reused quote
f'{1: abcd "{"\n"}" }'
# ^ backslash
```
I didn't recognize these as nested interpolations that also need to be
checked
for invalid expressions, so filtering out the whole format spec wasn't
quite
right. And `elements.interpolations()` only iterates over the outermost
interpolations, not the nested ones.
There's basically no code change in this PR, I just moved the existing
check
from `parse_interpolated_string`, which parses the entire string, to
`parse_interpolated_element`. This kind of seems more natural anyway and
avoids
having to try to recursively visit nested elements after the fact in
`parse_interpolated_string`. So viewing the diff with something like
```
git diff --color-moved --ignore-space-change --color-moved-ws=allow-indentation-change main
```
should make this more clear.
Test Plan
--
New tests
<!--
Thank you for contributing to Ruff/ty! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title? (Please prefix
with `[ty]` for ty pull
requests.)
- Does this pull request include references to any relevant issues?
-->
## Summary
<!-- What's the purpose of the change? What does it do, and why? -->
This PR implements semantic syntax error where alternative patterns bind
different names
## Test Plan
<!-- How was it tested? -->
I have written inline tests as directed in #17412
---------
Signed-off-by: 11happy <soni5happy@gmail.com>
Co-authored-by: Brent Westbrook <brentrwestbrook@gmail.com>
Summary
--
Fixes#20844 by refining the unsupported syntax error check for [PEP
701]
f-strings before Python 3.12 to allow backslash escapes and escaped
outer quotes
in the format spec part of f-strings. These are only disallowed within
the
f-string expression part on earlier versions. Using the examples from
the PR:
```pycon
>>> f"{1:\x64}"
'1'
>>> f"{1:\"d\"}"
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: Invalid format specifier '"d"' for object of type 'int'
```
Note that the second case is a runtime error, but this is actually
avoidable if
you override `__format__`, so despite being pretty weird, this could
actually be
a valid use case.
```pycon
>>> class C:
... def __format__(*args, **kwargs): return "<C>"
...
>>> f"{C():\"d\"}"
'<C>'
```
At first I thought narrowing the range we check to exclude the format
spec would
only work for escapes, but it turns out that cases like `f"{1:""}"` are
already
covered by an existing `ParseError`, so we can just narrow the range of
both our
escape and quote checks.
Our comment check also seems to be working correctly because it's based
on the
actual tokens. A case like
[this](https://play.ruff.rs/9f1c2ff2-cd8e-4ad7-9f40-56c0a524209f):
```python
f"""{1:# }"""
```
doesn't include a comment token, instead the `#` is part of an
`InterpolatedStringLiteralElement`.
Test Plan
--
New inline parser tests
[PEP 701]: https://peps.python.org/pep-0701/
This PR resolves the issue noticed in
https://github.com/astral-sh/ruff/pull/20777#discussion_r2417233227.
Namely, cases like this were being flagged as syntax errors despite
being perfectly valid on Python 3.8:
```pycon
Python 3.8.20 (default, Oct 2 2024, 16:34:12)
[Clang 18.1.8 ] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> with (open("foo.txt", "w")): ...
...
Ellipsis
>>> with (open("foo.txt", "w")) as f: print(f)
...
<_io.TextIOWrapper name='foo.txt' mode='w' encoding='UTF-8'>
```
The second of these was already allowed but not the first:
```shell
> ruff check --target-version py38 --ignore ALL - <<EOF
with (open("foo.txt", "w")): ...
with (open("foo.txt", "w")) as f: print(f)
EOF
invalid-syntax: Cannot use parentheses within a `with` statement on Python 3.8 (syntax was added in Python 3.9)
--> -:1:6
|
1 | with (open("foo.txt", "w")): ...
| ^
2 | with (open("foo.txt", "w")) as f: print(f)
|
Found 1 error.
```
There was some discussion of related cases in
https://github.com/astral-sh/ruff/pull/16523#discussion_r1984657793, but
it seems I overlooked the single-element case when flagging tuples. As
suggested in the other thread, we can just check if there's more than
one element or a trailing comma, which will cause the tuple parsing on
<=3.8 and avoid the false positives.
<!--
Thank you for contributing to Ruff/ty! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title? (Please prefix
with `[ty]` for ty pull
requests.)
- Does this pull request include references to any relevant issues?
-->
## Summary
<!-- What's the purpose of the change? What does it do, and why? -->
This PR implements
https://docs.astral.sh/ruff/rules/future-feature-not-defined/ (F407) as
a semantic syntax error.
## Test Plan
<!-- How was it tested? -->
I have written inline tests as directed in #17412
---------
Signed-off-by: 11happy <soni5happy@gmail.com>
<!--
Thank you for contributing to Ruff/ty! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title? (Please prefix
with `[ty]` for ty pull
requests.)
- Does this pull request include references to any relevant issues?
-->
## Summary
This PR implements
https://docs.astral.sh/ruff/rules/multiple-starred-expressions/ as a
semantic syntax error
## Test Plan
I have added inline tests as directed in #17412
---------
Signed-off-by: 11happy <soni5happy@gmail.com>
Co-authored-by: Brent Westbrook <36778786+ntBre@users.noreply.github.com>
<!--
Thank you for contributing to Ruff/ty! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title? (Please prefix
with `[ty]` for ty pull
requests.)
- Does this pull request include references to any relevant issues?
-->
## Summary
This PR implements F406
https://docs.astral.sh/ruff/rules/undefined-local-with-nested-import-star-usage/
as a semantic syntax error
## Test Plan
I have written inline tests as directed in #17412
---------
Signed-off-by: 11happy <soni5happy@gmail.com>
As of [this cpython PR](https://github.com/python/cpython/pull/135996),
it is not allowed to concatenate t-strings with non-t-strings,
implicitly or explicitly. Expressions such as `"foo" t"{bar}"` are now
syntax errors.
This PR updates some AST nodes and parsing to reflect this change.
The structural change is that `TStringPart` is no longer needed, since,
as in the case of `BytesStringLiteral`, the only possibilities are that
we have a single `TString` or a vector of such (representing an implicit
concatenation of t-strings). This removes a level of nesting from many
AST expressions (which is what all the snapshot changes reflect), and
simplifies some logic in the implementation of visitors, for example.
The other change of note is in the parser. When we meet an implicit
concatenation of string-like literals, we now count the number of
t-string literals. If these do not exhaust the total number of
implicitly concatenated pieces, then we emit a syntax error. To recover
from this syntax error, we encode any t-string pieces as _invalid_
string literals (which means we flag them as invalid, record their
range, and record the value as `""`). Note that if at least one of the
pieces is an f-string we prefer to parse the entire string as an
f-string; otherwise we parse it as a string.
This logic is exactly the same as how we currently treat
`BytesStringLiteral` parsing and error recovery - and carries with it
the same pros and cons.
Finally, note that I have not implemented any changes in the
implementation of the formatter. As far as I can tell, none are needed.
I did change a few of the fixtures so that we are always concatenating
t-strings with t-strings.
Closes#18671
Note that while this has, I believe, always been invalid syntax, it was
reported as a different syntax error until Python 3.12:
Python 3.11:
```pycon
>>> x = 1
>>> f"{x! s}"
File "<stdin>", line 1
f"{x! s}"
^
SyntaxError: f-string: invalid conversion character: expected 's', 'r', or 'a'
```
Python 3.12:
```pycon
>>> x = 1
>>> f"{x! s}"
File "<stdin>", line 1
f"{x! s}"
^^^
SyntaxError: f-string: conversion type must come right after the exclamanation mark
```
This PR implements template strings (t-strings) in the parser and
formatter for Ruff.
Minimal changes necessary to compile were made in other parts of the code (e.g. ty, the linter, etc.). These will be covered properly in follow-up PRs.
Fixes#17867
## Summary
The CPython parser does not allow generator expressions which are the
sole arguments in an argument list to have a trailing comma.
With this change, we start flagging such instances.
## Test Plan
Added new inline tests.
## Summary
Part of #17412
Starred expressions cannot be used as values in assignment expressions.
Add a new semantic syntax error to catch such instances.
Note that we already have
`ParseErrorType::InvalidStarredExpressionUsage` to catch some starred
expression errors during parsing, but that does not cover top level
assignment expressions.
## Test Plan
- Added new inline tests for the new rule
- Found some examples marked as "valid" in existing tests (`_ = *data`),
which are not really valid (per this new rule) and updated them
- There was an existing inline test - `assign_stmt_invalid_value_expr`
which had instances of `*` expression which would be deemed invalid by
this new rule. Converted these to tuples, so that they do not trigger
this new rule.
## Summary
Part of #17412
Add a new compile-time syntax error for detecting `nonlocal`
declarations at a module level.
## Test Plan
- Added new inline tests for the syntax error
- Updated existing tests for `nonlocal` statement parsing to be inside a
function scope
Co-authored-by: Brent Westbrook <36778786+ntBre@users.noreply.github.com>
## Summary
Based on the discussion in
https://github.com/astral-sh/ruff/pull/17298#discussion_r2033975460, we
decided to move the scope handling out of the `SemanticSyntaxChecker`
and into the `SemanticSyntaxContext` trait. This PR implements that
refactor by:
- Reverting all of the `Checkpoint` and `in_async_context` code in the
`SemanticSyntaxChecker`
- Adding four new methods to the `SemanticSyntaxContext` trait
- `in_async_context`: matches `SemanticModel::in_async_context` and only
detects the nearest enclosing function
- `in_sync_comprehension`: uses the new `is_async` tracking on
`Generator` scopes to detect any enclosing sync comprehension
- `in_module_scope`: reports whether we're at the top-level scope
- `in_notebook`: reports whether we're in a Jupyter notebook
- In-lining the `TestContext` directly into the
`SemanticSyntaxCheckerVisitor`
- This allows modifying the context as the visitor traverses the AST,
which wasn't possible before
One potential question here is "why not add a single method returning a
`Scope` or `Scopes` to the context?" The main reason is that the `Scope`
type is defined in the `ruff_python_semantic` crate, which is not
currently a dependency of the parser. It also doesn't appear to be used
in red-knot. So it seemed best to use these more granular methods
instead of trying to access `Scope` in `ruff_python_parser` (and
red-knot).
## Test Plan
Existing parser and linter tests.
Summary
--
Detect async comprehensions nested in sync comprehensions in async
functions before Python 3.11, when this was [changed].
The actual logic of this rule is very straightforward, but properly
tracking the async scopes took a bit of work. An alternative to the
current approach is to offload the `in_async_context` check into the
`SemanticSyntaxContext` trait, but that actually required much more
extensive changes to the `TestContext` and also to ruff's semantic
model, as you can see in the changes up to
31554b473507034735bd410760fde6341d54a050. This version has the benefit
of mostly centralizing the state tracking in `SemanticSyntaxChecker`,
although there was some subtlety around deferred function body traversal
that made the changes to `Checker` more intrusive too (hence the new
linter test).
The `Checkpoint` struct/system is obviously overkill for now since it's
only tracking a single `bool`, but I thought it might be more useful
later.
[changed]: https://github.com/python/cpython/issues/77527
Test Plan
--
New inline tests and a new linter integration test.
Summary
--
This PR extends the checks in #17101 and #17282 to annotated assignments
after Python 3.13.
Currently stacked on #17282 to include `await`.
Test Plan
--
New inline tests. These are simpler than the other cases because there's
no place to put generics.
Charlie Marsh