Enable `require-build-hashes` for the Python projects and inline scripts
so isolated builds require declared hashes for their build dependencies.
Pin the build backends and their dependencies with hashes.
---------
Co-authored-by: zaniebot <242828183+zaniebot@users.noreply.github.com>
Summary
--
This PR updates our default Python version to 3.11, in line with 3.10
reaching [EOL] next month, and
updates our latest Python version to 3.15. I followed last year's PR
(#20725) to find everywhere
that needed to change. Fortunately there were not as many test changes
needed as last year's
deferred annotation issues.
I also updated the language classifier in our pyproject.toml.
[EOL]: https://devguide.python.org/versions/#supported-versions
Test Plan
--
Existing tests with updated snapshots. Most of the updates were to
snapshots for TC001, TC002, and
TC003, which now default to the lazy-import fix.
After updating the snapshots, I tried to make sure we still had coverage
for whatever changed. That
meant adding a few more mdtests for the TC rules, and one small mdtest
for RUF017 to restore
coverage for the `functools.reduce` fix.
## Summary
- Use the integer bit APIs stabilized in Rust 1.97 for packed node
indices, token capacity, rule iteration, and type-variable support sets.
- Use Cargo's warning policy in our native and WASM Clippy checks,
keeping warnings fatal without changing compiler flags and invalidating
the build cache. Update the contributing guides to match.
Preserves the Rust 1.97 MSRV. The warning policy change follows
https://github.com/astral-sh/uv/pull/22122.
## Summary
We add accessors and iteration methods to `ExprCompare` and use them
throughout AST visitors, formatting, lint rules, and type inference. For
`a < b <= c`, `iter` yields `(a, Lt, b)` and `(b, LtE, c)`. `as_single`
returns a triple only when there is exactly one operator and two
operands, so callers can match an operator and both operands together
while rejecting comparison chains. Rules that only inspect operands use
the existing operand slice directly.
The `first_operand`, `first_operator`, and `second_operand` accessors
assert the parser's existing guarantee that a comparison has at least
one operator and two operands. Error recovery synthesizes missing
operands; manually constructed ASTs must uphold the same invariant.
## Summary
We reduce `Expr` from 64 to 56 bytes on 64-bit targets. The largest
variants determine the enum's size, so shrinking those variants saves
eight bytes in every `Expr` value, including expressions in the parsed
modules retained by ty.
The changes are:
- Store positional call arguments in `ThinVec`, reducing `Arguments` and
`ExprCall` by eight bytes.
- Store all comparison operands in one boxed slice. This removes the
separate left-operand allocation and reduces `ExprCompare` by eight
bytes while keeping operators in a boxed slice.
- Store dictionary-comprehension generators in a boxed slice, reducing
`ExprDictComp` by eight bytes.
- Store f-string and t-string elements in boxed slices, and represent a
single f-string directly as `FString`. This keeps single parts inline
and reduces `ExprFString` and `ExprTString` by eight bytes.
The parser, AST visitors, formatter, lint rules, and type inference use
the updated layouts. Size assertions cover the 56-byte enum and its
affected variants.
Delegate `Name` equality with borrowed strings to `CharStr`, including
Salsa lookups, so these callers use the compact string's comparison
implementation.
This connects Ruff's name comparisons to the short-string fast paths in
https://github.com/astral-sh/char_str/pull/40. Those fast paths require
a later `char_str` release; this change is compatible with the current
dependency.
In #28092 I fixed a bug where ty's redundant-cast removal fix could
create broken code if the original `cast()` call was multi-line. Doug
commented that this could perhaps be a general-purpose utility -- and it
turned out there were two existing similar Ruff rules with pre-existing
multi-line support that could benefit from such a shared utility.
This PR introduces a shared utility in `ruff_python_edits` for removing
a function call in favor of one of its arguments. This is the common
shape between redundant-cast removal, redundant `int()` removal, and
redundant `round()` removal.
The new utility is more robust than the previous code for any of the
three cases. Previously removing redundant `int`, `round`, and
`typing.cast` calls could discard parentheses needed for line
continuation or for token separation. For example, `int(1).to_bytes()`
could become `1.to_bytes()`, which is invalid.
The shared helper preserves optional argument parentheses and the
comments inside them, adds grouping where required, and avoids redundant
parentheses when the surrounding syntax already allows the expression.
Ruff and ty retain their own diagnostic eligibility and fix-safety
decisions.
## Test plan
- Extend RUF046 and RUF057 fixtures for split callees, assignment
expressions, integer attributes, adjacent keywords, and comments inside
optional parentheses. Existing snapshot cases retain their fix text and
safety classifications.
- Add compact helper tests in `ruff_python_edits` for conditional
callees, comprehension iterables, and dictionary expressions inside
f-strings.
- Existing ty cast mdtests cover positional and keyword arguments,
operator precedence, multiline values, and comment preservation.
Summary
--
This PR updates our ruff-pre-commit dependency to 0.16.5, which supports
the new category selectors,
and then enables `lint.preview` and the new default categories so that
we can dogfood them on our
code. There were 66 diagnostics initially, which I also fixed or
ignored:
```
28 FURB113 [ ] repeated-append
12 ASYNC240 [ ] blocking-path-method-in-async-function
6 RUF105 [*] noqa-comments
5 PLW1510 [ ] subprocess-run-without-check
3 PLW0108 [ ] unnecessary-lambda
2 EXE001 [ ] shebang-not-executable
2 RUF039 [*] unraw-re-pattern
1 B024 [ ] abstract-base-class-without-abstract-method
1 FURB142 [*] for-loop-set-mutations
1 PERF102 [ ] incorrect-dict-iterator
1 PLC0207 [*] missing-maxsplit-arg
1 PLW0127 [ ] self-assigning-variable
1 TRY004 [ ] type-check-without-type-error
1 PLW0120 [*] useless-else-on-loop
1 TRY201 [ ] verbose-raise
```
Of these, I ignored `repeated-append` and `blocking-path-method-in-async-function`. I think they are
still reasonable defaults, just less useful/important in our uses.
0.16.5 also includes https://github.com/astral-sh/ruff-pre-commit/pull/159, so I removed `markdown` from our main `ruff-format` hook, leaving it for the later mdtest formatting stage.
I also had to make a few changes related to config discovery in tests:
- move our cached ecosystem checkouts to `$RUNNER_TEMP` instead of inside of our checkout
- convert some CLI tests to use the `CliTest` helpers
- add `configurationPreference: editorOnly` to some notebook tests
In these cases, our own Ruff settings were being inherited by test code. This likely affects many other tests, but I only updated the obviously failing ones for now. I'm least confident about the notebook tests, but as far as I can tell from the history (https://github.com/astral-sh/ruff/pull/11944, https://github.com/astral-sh/ruff/pull/22810), the specific diagnostics are not the important parts being asserted.
Test Plan
--
CI on this PR
Removing a redundant `typing.cast` could discard parentheses needed for
multiline expressions, introducing syntax errors or silently changing
arithmetic results. Preserve the argument's existing parentheses and
enclosed comments, and add grouping when a multiline argument relies on
the call's parentheses for line continuation.
The shared parentheses lookup now checks for an actual closing token so
that constructing a fix for an unfinished call does not panic.
Fixes https://github.com/astral-sh/ty/issues/4399.
## Test plan
- Added mdtests for multiline conditional and arithmetic expressions,
including a line break before an operator that must not become a
separate statement.
- Covered reordered keyword arguments and comments inside argument
parentheses.
- Added unit coverage for unterminated casts with no final newline, with
and without a trailing comment.
## Summary
We now run Hawk 0.1.13 with Rust 1.98.0 in CI, following the pattern
used by uv. The configuration preserves intentional public APIs and
generated interfaces while checking all configured production binaries
and workspace non-production targets with `-D warnings`.
The accompanying cleanup narrows internal visibility, removes unused
benchmark classification state and accessors, and records intentional
public interfaces as audited Hawk expectations. The resulting
configuration reports zero Hawk findings across the configured targets.
## Summary
Skip checking scripts with invalid PEP 723 metadata blocks, that are,
blocks that ty can't parse successfully or configurations that don't
resolve successfully. Instead, emit one diagnostic that points to the
invalid metadata.
ty still initializes a `Program` for each of those scripts so that LSP
operations continue to work. But we skip those scripts in all code paths
where we perform semantic checks.
Fixes https://github.com/astral-sh/ty/issues/4181.
## Test plan
Added tests
<!--
Thank you for contributing to Ruff/ty! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title? (Please prefix
with `[ty]` for ty pull
requests.)
- Does this pull request include references to any relevant issues?
- Does this PR follow our AI policy
(https://github.com/astral-sh/.github/blob/main/AI_POLICY.md)?
-->
## Summary
Adds explicit 7d cooldowns to all of our PEP 723 scripts, as well as
similar cooldowns via `.npmrc`.
<!-- What's the purpose of the change? What does it do, and why? -->
## Test Plan
NFC.
<!-- How was it tested? -->
---
<sub>Stack created with <a
href="https://github.com/github/gh-stack">GitHub Stacks CLI</a> • <a
href="https://gh.io/stacks-feedback">Give Feedback 💬</a></sub>
## Summary
Preserve source ranges for PEP 723 script metadata by building a compact
source map during extraction and applying it when deserializing ranged
values.
Closes https://github.com/astral-sh/ty/issues/4180
## Summary
This PR reduces the size of `Expr` from 72 to 64 bytes by storing only
the start offset on `ExprCall` (64 to 56 bytes) and boxing the uncommon
implicitly concatenated string representation in `ExprStringLiteral` (64
to 48 bytes). Both changes are required to reduce the size of the
containing `Expr` enum.
`ExprCall` derives its end offset from its argument list; parser and
error-recovery paths assert that their ranges share the same end.
---------
Co-authored-by: Charlie Marsh <charliemarsh@openai.com>
## Summary
This PR uses Hawk 0.1.11 to narrow internal declarations to the
visibility they actually need.
Here, I've intentionally _only_ made visibility changes; unused API
removals are split into follow-up PRs. In total, 1,201 declaration
visibility modifiers are narrowed. Public generated formatter APIs and
other declarations whose narrower visibility would activate dead-code or
API-shape lints remain unchanged.
## Summary
Add support for configuring some (`rules` and `analysis` settings, same
as for `overrides`) from PEP 723 `tool.ty` script metadata. I do plan to
add support for `environment` settings and maybe even `terminal`, but
that requires more work, because different environments require
different dbs.
Given that we'll need to use different dbs. Why bother with integrating
this into `file_settings`, which only allows overriding a small sub-set
of options. The idea behind this is that I want to share the same db for
scripts with the same `environment` settings (and dependencies in the
future). This should reduce overall resource consumption because it
allows sharing parsed ASTs, semantic indices, for scripts that use the
same environment (and venv, which might not be that uncommon, thanks to
uv's canonical venvs). The integration into `file_settings` enables this
sharing by creating separate DBs only for the settings where this is
necessary, but relying on `file_settings` to handle differences in
enabled rules etc.
Using `file_settings` has one downside. It most likely means that the
script metadata is parsed twice. I think this is negligible. Most
metadata blocks are tiny.
https://github.com/user-attachments/assets/d2be2e67-2a6f-405a-9852-860377f8496c
## Configuration precedence
Scripts don't inherit any settings from the project (you got me, this is
not a 100% true today where we still use the project's python version,
but it will be). They're fully isolated. Each of them represent their
own project. I'm not a 100% sure yet if this is the ideal behavior or if
we need some mechanism that allows inheriting settings from the project.
But I'd like to defer this discussion to later.
Scripts respect configurations from other sources, the same as projects
do:
* CLI options or LSP configurations take highest precedence, overriding
configurations in the script metadata
* The scripts own configuration
* User configuration
* Fallback configurations. We only use this in the LSP today as a hint
of what the python version is. Specifically, we use the selected python
environment in VS Code, if the user didn't specify a python environment.
## Error handling
For now, I've intentionally decided not to deal with invalid script
metadata blocks. I expect that the validation should happen in
`check_file`, or even earlier, in the code constructing the script's
database. This is something I'll tackle next, and is why this PR assumes
that all script metadata blocks are valid, blocks with syntax errors are
simply ignored, unknown rules are not flagged.
## Memory regression
The main regression comes from the new `script_metadata` query. The
query already uses an `Option<Box<Metadata>>` to reduce the overhead for
files without script metadata.
The second cause of the regression is that `file_settings` now depends
on `script_metadata`. A dependency that Salsa now needs to track.
There's not much we can do about this. I considered moving the text scan
into `file_settings` but that only trades a dependency on
`script_metadata` for depending on `source_text` (and doing more work).
We do need to record the dependency because a user can add a script
metadata block to a regular file, which changes the file's settings.
I don't see a way (with proper incremental computation) to fully avoid
the memory regression. We may want to enable within-same-revision
garbage collection in the future, as we don't really need to retain the
metadata for a prolonged time.
## Test Plan
Added query, mdtest, and CLI coverage.
---------
Co-authored-by: Aria Desires <aria.desires@gmail.com>
## Summary
This PR changes `ruff_python_ast::Name` from a 24-byte `CompactString`
to a 16-byte immutable `CharStr` from
[`astral-sh/char_str`](https://github.com/astral-sh/char_str).
`char_str` is an Astral-maintained fork of
[`lean_string`](https://github.com/ryota2357/lean_string) that provides
compact owned strings with mutable and immutable variants: growable
`CharString` for construction and immutable, exactly allocated `CharStr`
for storage. (It also provides Salsa and get-size features.)
Names up to 16 UTF-8 bytes remain inline, static names remain
allocation-free, and longer names use an exactly sized,
reference-counted allocation with no unused capacity.
Removing eight bytes from every `Name` also shrinks common AST nodes.
For example, `Identifier` and `ExprName` drop from 40 to 32 bytes, while
`ExprAttribute`, `StmtFunctionDef`, `StmtClassDef`, and several other
variants each drop by eight bytes.
The parser interns names that exceed the inline capacity, so repeated
longer names share one backing allocation within a parser invocation.
## Why 16 bytes?
Across the full `mypy_primer` corpus, there 20.8 million name
occurrences:
| UTF-8 length | Occurrences | Share | Unique name/file pairs | Share |
|---|---:|---:|---:|---:|
| 1–16 bytes | 18,955,757 | 91.02% | 3,192,424 | 79.42% |
| 17–24 bytes | 1,348,894 | 6.48% | 529,826 | 13.18% |
| 25+ bytes | 522,376 | 2.51% | 297,575 | 7.40% |
| Total | 20,827,027 | 100% | 4,019,825 | 100% |
The 16-byte cutoff keeps 91% of occurrences inline while saving eight
bytes for every occurrence. The 1–16-byte bucket alone removes 151.6 MB
of handle storage across the corpus.
The tradeoff is the 17–24-byte bucket: `CompactString` stored those
names inline, while `CharStr` allocates once per unique name within a
file. But the interning amortizes that cost when a name repeats. Names
longer than 24 bytes already allocated on `main`; for those names, this
change both reduces the handle size and deduplicates repeated backing
allocations.
## Memory impact
Accounting for the observed per-file multiplicities and structural
sharing, modeled `Name` storage across the corpus changes as follows:
| | `main` | This PR | Difference |
|---|---:|---:|---:|
| `Name` handles and backing storage | 526.18 MB | 366.27 MB | -159.91
MB (-30.4%) |
The current [instrumented memory
report](https://github.com/astral-sh/ruff/pull/26594#issuecomment-4906725175)
shows a consistent reduction in total retained memory:
| Project | `main` | This PR | Difference |
|---|---:|---:|---:|
| flake8 | 29.21 MB | 28.71 MB | -1.69% |
| trio | 71.01 MB | 70.01 MB | -1.41% |
| sphinx | 168.81 MB | 166.82 MB | -1.18% |
| prefect | 454.75 MB | 449.06 MB | -1.25% |
Within those totals, `parsed_module` decreases by roughly 2.1% and
`semantic_index` by 2.4–2.7%. Several name-heavy collections decrease by
considerably more.
Summary
--
This PR adds a new lint rule for TOML config files that checks for the
use of rule codes as
selectors and offers a fix to replace them with human-readable names. At
a high-level, the
implementation just deserializes a TOML config into a stripped down
version of our normal
configuration types and then checks each of its `Spanned` selector
entries. Unless I oversimplified
this too much, it didn't end up being as tricky as I thought. The main
tricky part was adding basic
support for fixes in TOML files, but even this was pretty
straightforward.
Codex was concerned about calling this a safe fix because applying the
fix with `--preview`, or
another temporary way of enabling preview, would break the config if
preview isn't always enabled.
That seemed to go beyond our usual fix safety policy to me, but I'm also
happy to mark the fix as
unsafe if that's preferable.
Test Plan
--
New tests covering all of the expected selectors and new CLI tests to
make sure that `ruff.toml`
files get linted and respect various ways of suppressing fixes.
Previously we only linted
`pyproject.toml` files for `RUF200` and bypassed some of the checks in
`LintContext`. I left the
change to use `LintContext` as a separate commit since it also affects
`RUF200` and I briefly
considered splitting it off into a separate PR.
