## Summary
`match` is a soft keyword. It can name a variable, a function, or the
target of an annotated assignment, including after a semicolon.
```python
first(); match = value
first(); match(value)
first(); match[0]: int
```
The parser reported a compound-statement error as soon as it saw the
`match` token. We now check the preceding separator when parsing a match
statement. If the token turns out to be an identifier, speculative
parsing discards the diagnostic. A genuine `match` statement on the same
logical line still produces the original error.
## Summary
A bare `raise` can end at a semicolon. For example, Python accepts this
handler and re-raises the current exception.
```python
try:
work()
except ValueError:
raise; report()
```
The parser tried to read the semicolon as the start of an exception
expression and reported a syntax error. We now recognize both a newline
and a semicolon as terminators for a bare `raise`.
## Summary
Changing the lexer's keyword set now takes one edit to the Rust list.
The compiler finds a collision-free multiplier and builds the 64-slot
table, so we can remove the Python generator and its lockfile.
A two-byte keyword like `if` packs to `0x6669_0000_0000_0000`. Only the
low two bytes of the multiplier affect its slot. We place keywords from
shortest to longest, choosing each new byte of the multiplier without
disturbing the shorter keywords. The lexer still selects a slot with one
multiply and shift, then compares the full word.
## Summary
We give ASCII identifiers a dedicated path, avoiding Unicode bookkeeping
and checking string prefixes only when a one- or two-letter name reaches
a quote. Both paths share the existing batched ASCII scanner, preserving
fast scanning for Unicode names with long ASCII suffixes.
Stacked on #29463 to measure the incremental improvement. [CodSpeed
reports](https://github.com/astral-sh/ruff/pull/29503/checks?check_run_id=118300735426)
6.6–9.8% greater efficiency across all five lexer benchmarks, with no
reported regressions. We preserve the existing scanner and add no
inlining attributes or unsafe code.
## Summary
We scan ASCII identifier continuations in sixteen-byte batches using a
byte lookup table, avoiding per-character decoding. Keyword lookup
filters by length and initial byte, then uses a compact hash table with
a full-word comparison to reject collisions. Non-ASCII suffixes retain
the existing Unicode path.
## Summary
We skip prefix-operator dispatch for identifiers and parse atoms together with their postfix chains. Keeping this work in `parse_primary_expression` makes the grouping-parenthesis flag local and avoids passing an intermediate `ParsedExpr` between atom and postfix parsing.
## Summary
This PR fixes a completion panic in a notebook with an empty last cell.
To analyze a notebook, we combine its Python cells into one source
string, adding a newline after each cell:
```text
First cell: "x = 1" at offsets 0..5
Empty last cell: "" at offsets 6..6
Combined source: "x = 1\n\n" at offsets 0..7
```
We parse the cells separately, but the resulting module describes the
combined source. Previously, we ended that module at offset 6, where the
empty cell's content ends. Position conversion uses the combined source,
so a request for character 1 of the empty cell can resolve to offset 7.
That is the end of the source, but it was outside the module's range.
Looking up the completion position then panicked with `Range is not
contained within root`.
We now give the module the range 0..7, as we would when parsing the same
source as a whole. This makes positions at the end of the combined
source safe to look up. #29307 separately corrects position conversion
so the same request resolves to offset 6, the end of the requested
cell's content.
Closes https://github.com/astral-sh/ty/issues/4687.
Use the vendored runtime and generator instead of the registry dependency,
providing Unicode 17 names with Python-compatible matching. Update workspace
dependencies and tool configuration, add parser regressions, and document how
to update Unicode data when supporting a new Python version.
## Summary
- Use the integer bit APIs stabilized in Rust 1.97 for packed node
indices, token capacity, rule iteration, and type-variable support sets.
- Use Cargo's warning policy in our native and WASM Clippy checks,
keeping warnings fatal without changing compiler flags and invalidating
the build cache. Update the contributing guides to match.
Preserves the Rust 1.97 MSRV. The warning policy change follows
https://github.com/astral-sh/uv/pull/22122.
## Summary
We reduce `Expr` from 64 to 56 bytes on 64-bit targets. The largest
variants determine the enum's size, so shrinking those variants saves
eight bytes in every `Expr` value, including expressions in the parsed
modules retained by ty.
The changes are:
- Store positional call arguments in `ThinVec`, reducing `Arguments` and
`ExprCall` by eight bytes.
- Store all comparison operands in one boxed slice. This removes the
separate left-operand allocation and reduces `ExprCompare` by eight
bytes while keeping operators in a boxed slice.
- Store dictionary-comprehension generators in a boxed slice, reducing
`ExprDictComp` by eight bytes.
- Store f-string and t-string elements in boxed slices, and represent a
single f-string directly as `FString`. This keeps single parts inline
and reduces `ExprFString` and `ExprTString` by eight bytes.
The parser, AST visitors, formatter, lint rules, and type inference use
the updated layouts. Size assertions cover the 56-byte enum and its
affected variants.
Use hashbrown's borrowed-entry insertion for long parser names so a
missing name reuses its lookup when it is inserted. Construct the
heap-backed `Name` only on a miss, while preserving the inline bypass
and shared allocations for repeated long names.
Summary
--
While reviewing #28231, Codex found a preexisting bug shared with our
unary minus handling. Namely, CPython emits an error for code like this,
where the unary expression is part of a longer expression:
```pycon
>>> match foo:
... case -1**2: ...
...
File "<python-input-0>", line 2
case -1**2: ...
^^
SyntaxError: invalid syntax
```
but Ruff does not:
```console
$ ruff check --ignore ALL - <<EOF
match foo:
case -1**2: ...
EOF
All checks passed!
```
We now verify that the parsed unary expression is a number literal and
emit an error if not.
Test Plan
--
New inline snapshots
Summary
--
As added in https://github.com/python/cpython/pull/148566, CPython now
allows unary plus in match
patterns, mirroring the support for unary minus. We already had a
special case for this for improved
error reporting and recovery, so I just swapped out the `add_error` call
for
`add_unsupported_syntax_error` with a new error kind.
Part of https://github.com/astral-sh/ruff/issues/28206
Test Plan
--
New inline parser tests
## Summary
- Use Rust 1.96's stabilized `assert_matches!` and
`debug_assert_matches!` macros throughout Ruff and ty.
- Include mismatched values in assertion failures and remove redundant
manual debug formatting.
- Preserve existing assertions for values that do not implement `Debug`.
Fixed a link to `frozen.discard` in the Python docs this was a valid
link in older Python docs but has been broken.
While I was there I noticed a few other uncanonical links so replaced
them with more canonical versions and removed some links to specific
Python versions that are unnecessary.
## Summary
This PR reduces the size of `Expr` from 72 to 64 bytes by storing only
the start offset on `ExprCall` (64 to 56 bytes) and boxing the uncommon
implicitly concatenated string representation in `ExprStringLiteral` (64
to 48 bytes). Both changes are required to reduce the size of the
containing `Expr` enum.
`ExprCall` derives its end offset from its argument list; parser and
error-recovery paths assert that their ranges share the same end.
---------
Co-authored-by: Charlie Marsh <charliemarsh@openai.com>
## Summary
This PR uses Hawk 0.1.11 to narrow internal declarations to the
visibility they actually need.
Here, I've intentionally _only_ made visibility changes; unused API
removals are split into follow-up PRs. In total, 1,201 declaration
visibility modifiers are narrowed. Public generated formatter APIs and
other declarations whose narrower visibility would activate dead-code or
API-shape lints remain unchanged.
## Summary
This is a first step towards #20411. After this, we would work to either
upstream changes or adjust things within Ruff.
Performance improvements have already been upstreamed
(https://github.com/rust-lang/annotate-snippets-rs/pull/437,
https://github.com/rust-lang/annotate-snippets-rs/pull/440)
Re-implemented the following from the fork:
- Hidden snippets
(https://github.com/rust-lang/annotate-snippets-rs/issues/448)
- Renderer-level disabling of hyperlinks
- Jupyter cell indices
(https://github.com/rust-lang/annotate-snippets-rs/issues/449)
- Fixable mark
(https://github.com/rust-lang/annotate-snippets-rs/issues/450)
- `lineno_offset` (maybe not needed if #20648 is taken care of)
- anonymized lines affects origin
(https://github.com/rust-lang/annotate-snippets-rs/issues/451)
- cut indicator (maybe not needed if we can switch to unicode renderer)
Additional hacks to minimize behavior change include
- Adjust upstream's hidden severity behavior to match Ruff's
(https://github.com/rust-lang/annotate-snippets-rs/issues/452)
- How newlines are annotated
(https://github.com/rust-lang/annotate-snippets-rs/issues/454)
- To not indent for line numbers when none will be present
(https://github.com/rust-lang/annotate-snippets-rs/issues/453)
- Enabling `simd` by default
All of the above represents work that needs to be resolved to address
#20411.
`lib.name` hack is used to simplify maintenance with the assumption that
we'll continue working towards #20411.
Behavior changes that were included
- Tweak when implicit padding lines are present
- Fix Origin's line/column is now derived from the primary annotation
- Make consistent the rendering of primary/context annotations
- Fix snippets to show up for empty source (but may degrade #24528)
- Fix handling of term width
- Fix rendering of annotations between newline and first character
(granted, maybe the caller is wrong)
- Tweak multiline annotation markers
New or upcoming features annotate-snippets features that we will be able
to leverage by moving this forward
- `AnnotationKind::Visible` to improve the quality of built-in
code-folding if Ruff were to switch to it
- `DecorStyle::Unicode`
- *(upcoming)* No-graphiscs mode for screen readers (may also be useful
for AIs)
Deferred:
- Moving related `Group`s into the same `Report`
- Using `secondary_title` for any `Group` after the first in a `Report`
- Using a `Snippet`, instead of titles, for "virtual code" like
https://github.com/astral-sh/ruff/pull/27033#discussion_r3622727236
- Switching hidden snippets to `Origin`
## Test Plan
Within cpython, I ran
```console
$ hyperfine -i "../../ruff/target-baseline/release/ruff check" "../../ruff/target-update/release/ruff check" -N -m20 --output=null
Benchmark 1: ../../ruff/target-baseline/release/ruff check
Time (mean ± σ): 27.5 ms ± 1.1 ms [User: 33.0 ms, System: 108.4 ms]
Range (min … max): 25.0 ms … 31.7 ms 94 runs
Warning: Ignoring non-zero exit code.
Benchmark 2: ../../ruff/target-update/release/ruff check
Time (mean ± σ): 28.1 ms ± 2.0 ms [User: 33.4 ms, System: 105.7 ms]
Range (min … max): 24.8 ms … 38.9 ms 102 runs
Warning: Ignoring non-zero exit code.
Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use t
he '--warmup' or '--prepare' options.
Summary
../../ruff/target-baseline/release/ruff check ran
1.03 ± 0.08 times faster than ../../ruff/target-update/release/ruff check
```
Before:
<img width="486" height="583" alt="image"
src="https://github.com/user-attachments/assets/2d94dabf-b28d-48c1-b17c-52aac59fd57e"
/>
After:
<img width="486" height="583" alt="image"
src="https://github.com/user-attachments/assets/b388ada3-e3d7-4ba0-b90b-1330eb81f4be"
/>
## Summary
This PR changes `ruff_python_ast::Name` from a 24-byte `CompactString`
to a 16-byte immutable `CharStr` from
[`astral-sh/char_str`](https://github.com/astral-sh/char_str).
`char_str` is an Astral-maintained fork of
[`lean_string`](https://github.com/ryota2357/lean_string) that provides
compact owned strings with mutable and immutable variants: growable
`CharString` for construction and immutable, exactly allocated `CharStr`
for storage. (It also provides Salsa and get-size features.)
Names up to 16 UTF-8 bytes remain inline, static names remain
allocation-free, and longer names use an exactly sized,
reference-counted allocation with no unused capacity.
Removing eight bytes from every `Name` also shrinks common AST nodes.
For example, `Identifier` and `ExprName` drop from 40 to 32 bytes, while
`ExprAttribute`, `StmtFunctionDef`, `StmtClassDef`, and several other
variants each drop by eight bytes.
The parser interns names that exceed the inline capacity, so repeated
longer names share one backing allocation within a parser invocation.
## Why 16 bytes?
Across the full `mypy_primer` corpus, there 20.8 million name
occurrences:
| UTF-8 length | Occurrences | Share | Unique name/file pairs | Share |
|---|---:|---:|---:|---:|
| 1–16 bytes | 18,955,757 | 91.02% | 3,192,424 | 79.42% |
| 17–24 bytes | 1,348,894 | 6.48% | 529,826 | 13.18% |
| 25+ bytes | 522,376 | 2.51% | 297,575 | 7.40% |
| Total | 20,827,027 | 100% | 4,019,825 | 100% |
The 16-byte cutoff keeps 91% of occurrences inline while saving eight
bytes for every occurrence. The 1–16-byte bucket alone removes 151.6 MB
of handle storage across the corpus.
The tradeoff is the 17–24-byte bucket: `CompactString` stored those
names inline, while `CharStr` allocates once per unique name within a
file. But the interning amortizes that cost when a name repeats. Names
longer than 24 bytes already allocated on `main`; for those names, this
change both reduces the handle size and deduplicates repeated backing
allocations.
## Memory impact
Accounting for the observed per-file multiplicities and structural
sharing, modeled `Name` storage across the corpus changes as follows:
| | `main` | This PR | Difference |
|---|---:|---:|---:|
| `Name` handles and backing storage | 526.18 MB | 366.27 MB | -159.91
MB (-30.4%) |
The current [instrumented memory
report](https://github.com/astral-sh/ruff/pull/26594#issuecomment-4906725175)
shows a consistent reduction in total retained memory:
| Project | `main` | This PR | Difference |
|---|---:|---:|---:|
| flake8 | 29.21 MB | 28.71 MB | -1.69% |
| trio | 71.01 MB | 70.01 MB | -1.41% |
| sphinx | 168.81 MB | 166.82 MB | -1.18% |
| prefect | 454.75 MB | 449.06 MB | -1.25% |
Within those totals, `parsed_module` decreases by roughly 2.1% and
`semantic_index` by 2.4–2.7%. Several name-heavy collections decrease by
considerably more.
## Summary
Use scratch buffers instead of allocating new `Vec`s and shrinking them
when constructing the AST node. While this doesn't remove the need for
copying the elements, it reduces the allocations within our parser
(`shrink_to_fit` often needs to allocate a new buffer and move all
elements). One shared scratch buffer per node type is sufficient,
because our parser is recursive (more nested parse rule always push to
the end, and parse function drain from the end as they complete).
## Performance
Very consistent perf improvement
## Codspeed memory regression
This can cause the parser's peak memory to increase, because we retain
the scratch buffer allocation after parsing a clause completed. However,
that memory is only temporary and released as soon as the parser
completes. Most project-level memory reports show a memory reduction.
It's only specific parser benchmarks that show a memory increase
## Summary
The lexer checks `pending_indentation` for every token. `Option::take`
writes `None` even on the common path where no indentation is pending,
so we now read the field first and clear it only when a value is
present. This avoids a redundant write in the hot lexer loop while
preserving the existing indentation behavior.
This PR is stacked on #26765 so that CodSpeed can measure the
pending-indentation optimization independently.
---------
Co-authored-by: Micha Reiser <micha@reiser.io>
Charlie Marsh