Summary
--
As pointed out in [#29635], we still had this `Emitter` trait, despite
almost all output formats
moving to `ruff_db` and its `Renderer` trait. This PR cleans up the last
use of `Emitter` in Ruff's
`Grouped` output format, as well as some custom settings handling in the
`GroupedEmitter` that is
now addressed by the shared `DisplayDiagnosticConfig`.
If we want to move this output format to `ruff_db` or share it with ty
in the future, this should
also help with that.
Test Plan
--
Existing tests
[#29635]:
https://github.com/astral-sh/ruff/pull/29635#discussion_r4362259048
Summary
--
This PR fixesastral-sh/ty#2484 by moving our SARIF rendering from
`ruff_linter` into `ruff_db` and
wiring that up as an output format for ty.
The first commit introduces the metadata and makes some other
preparatory changes to the SARIF
emitter in Ruff. The second commit then moves the emitter to `ruff_db`
and removes the remaining
Ruff-specific calls. The third commit then wires that up to ty's CLI.
Test Plan
--
Existing tests, adapted to `ruff_db`, as well as a new ty CLI test
Summary
--
This addresses the suggestion from [#29635]. `DisplayDiagnosticConfig`
previously determined the
`DiagnosticFormat` for diagnostics coming from both Ruff and ty, which
meant that it had to have a
shared enum between the two, despite each program supporting a different
subset of the available
formats. This PR eliminates the shared `DiagnosticFormat` and dispatch
through `DisplayDiagnostics`,
leaving Ruff and ty to construct `Renderer`s directly and do their own
dispatch on the output
formats they actually support. This will be especially helpful in
#29635, where we want to provide
slightly different arguments to each new `SarifRenderer`.
The `Renderer` trait itself isn't strictly necessary, but it made the
calls on the Ruff side a lot
neater since the trait provides a default `write` method based on the
required `render` method.
Codex also threw in a default `display` method that helps in a couple of
places too.
Test Plan
--
Existing tests
[#29635]:
https://github.com/astral-sh/ruff/pull/29635#discussion_r4352505588
Summary
--
This updates `ruff_unicode_names2` and `ruff_unicode_names2_generator`
to use our workspace `rand` dependency and `rust-version`, prompted by
#29257 and #29359. Bumping the Rust version also introduces a handful of
Clippy diagnostic that Clippy was also able to fix automatically. I
originally pinned both of these to stay closer to the upstream version,
but we've already diverged a bit, and the required changes were quite
small anyway, so this seems like the better way to go.
This also allows us to drop the `std` feature on `rand`, a transitive
dependency on `zerocopy`, and the accompanying allowlist entry for its
build script.
While I was here, I also switched to our workspace edition, removed
unused feature-gated code and unnecessary suppressions, enabled
formatting and typo checks, and enabled doctests.
These are the changes from `rand` for reference:
- https://rust-random.github.io/book/update-0.10.html
- https://rust-random.github.io/book/update-0.9.html
Test Plan
--
Existing tests
---------
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
Summary
--
This PR updates our default Python version to 3.11, in line with 3.10
reaching [EOL] next month, and
updates our latest Python version to 3.15. I followed last year's PR
(#20725) to find everywhere
that needed to change. Fortunately there were not as many test changes
needed as last year's
deferred annotation issues.
I also updated the language classifier in our pyproject.toml.
[EOL]: https://devguide.python.org/versions/#supported-versions
Test Plan
--
Existing tests with updated snapshots. Most of the updates were to
snapshots for TC001, TC002, and
TC003, which now default to the lazy-import fix.
After updating the snapshots, I tried to make sure we still had coverage
for whatever changed. That
meant adding a few more mdtests for the TC rules, and one small mdtest
for RUF017 to restore
coverage for the `functools.reduce` fix.
Summary
--
I wasn't really planning to do a full stabilization pass, but I quickly skimmed through `preview.rs`
and remembered that the isort pragma change was kind a bug fix but technically breaking and thought
we might want to stabilize it.
This PR stabilizes the isort pragma comment width handling introduced in #27313, along with the
related pragma handling for `E501` and `W505` from #24071. There was a comment about also
stabilizing the related formatter preview feature on the function from #24071, so that is stabilized
here too.
I'm fine holding off on this, especially if the formatter change is a better fit for the 2027 style
guide. Like I said above, I mostly wanted to stabilize the isort change, but that snowballed a bit
since the helpers were shared.
Test Plan
--
Updated existing snapshots
Summary
--
This makes a few modifications to our default rule set for v0.17. As the
snapshots show, these had
already been incorporated into our preview defaults via rule
recategorization. This just ports them
back to the stable default rules.
This closes#27197, which brought the problems with the DTZ rules to our
attention. F406 being a
syntax error was raised in [this comment].
[this comment]:
https://github.com/astral-sh/ruff/issues/27199#issuecomment-5093309966
Test Plan
--
Updated snapshots
Summary
--
This is the CLI accompaniment to #27807. The main change here is showing
display-only fix diffs in
the CLI. However, display-only fixes don't have a corresponding flag
like `--unsafe-fixes`, and I
thought it would be weird to show them unconditionally when we don't do
the same for unsafe fixes,
so this PR also unconditionally renders unsafe fix diffs, even when that
flag isn't passed. We
already emit a note about the fix being unsafe, so I think it makes
sense always to show you the
fix, and make `--unsafe-fixes` enable actually applying the fix. We also
don't want to apply
display-only fixes, so enabling rendering of the fixes with
`--display-only-fixes` or similar would
also disagree with the semantics of the unsafe flag. I'm open to other
ideas here, though.
The sub-diagnostic for display-only fixes uses a higher severity than
the note for unsafe fixes and
renders in red, as shown below.
The change here also affects ty, which now has both unsafe and
display-only fixes.
I again checked what clippy and rustc do in these cases, and they don't
include any additional note or warning about the fix being unsafe:
<details><summary>Rust diagnostics</summary>
<p>
Here's a
[`clippy::needless_pass_by_value`](https://rust-lang.github.io/rust-clippy/rust-1.97.0/index.html#needless_pass_by_value)
diagnostic, which is `MaybeIncorrect` ( similar to our `Unsafe`):
```
warning: this argument is passed by value, but not consumed in the function body
--> src/main.rs:3:15
|
3 | fn size(data: Data) -> usize {
| ^^^^
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/rust-1.97.0/index.html#needless_pass_by_value
= note: requested on the command line with `-W clippy::needless-pass-by-value`
help: consider taking a reference instead
|
3 | fn size(data: &Data) -> usize {
| +
```
and here's a rustc
[`LetLike`](https://github.com/rust-lang/rust/blob/8bab26f4f68e0e26f0bb7960be334d5b520ea452/compiler/rustc_trait_selection/src/errors.rs#L268-L288)
diagnostic, which is `has-placeholders` (similar to our `DisplayOnly`
but definitely produces invalid code in this case):
```
error[E0282]: type annotations needed for `Vec<_>`
--> placeholder.rs:2:9
|
2 | let values = Vec::new();
| ^^^^^^ ---------- type must be known at this point
|
help: consider giving `values` an explicit type, where the type for type parameter `T` is specified
|
2 | let values: Vec<T> = Vec::new();
| ++++++++
```
</p>
</details>
Fixes https://github.com/astral-sh/ty/issues/1679
Test Plan
--
Updated existing snapshots, and some manual testing in the CLI:
<img width="965" height="400" alt="image"
src="https://github.com/user-attachments/assets/888ad79a-6853-48d4-8344-48da4af47d74"
/>
---------
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
Summary
--
As the title says, this PR extends our
[category](https://docs.astral.sh/ruff/linter/#rule-categories) docs to
include a migration guide explaining how to approach a migration to the
new categories. Initially I included a migration script and a
description of how it works, but if we're unlikely to remove linter
groups, it seems less helpful. Our recommendation is basically to
`extend-select` the new categories to round out your existing
configuration, and no dramatic migration is necessary.
I guess this is not really a "migration guide" anymore, as reflected in
the updated section title.
---------
Co-authored-by: Micha Reiser <micha@reiser.io>
Summary
--
While looking at #28887, I noticed a repeated ecosystem check failure
caused by a git error:
> Failed to commit: fatal: failed to encode
'tests/roots/test-root/wrongenc.inc' from iso-8859 to UTF-8
This is caused by a known upstream bug (sphinx-doc/sphinx#14364) with an
open PR (sphinx-doc/sphinx#14467), but the PR is waiting for the next
release milestone, I think, so it seemed easiest just to temporarily
remove the project to unblock CI checks.
Test Plan
--
CI on this PR
Summary
--
This PR adds the `--untrusted-workspace` flag to the server CLI, which
informs the server that it's running in an untrusted workspace. When
this is the case, we fall back on the internal formatting backend
instead of using `uv format` to avoid reading uv configuration and
potentially installing packages based on it.
Test Plan
--
New server tests
Summary
--
This PR fixes an issue I saw in the ecosystem report for the
stabilization PR
(https://github.com/astral-sh/ruff/pull/28761#issuecomment-5767098108).
In short, we were flagging annotations like this:
```py
__lazy_modules__ = ["pathlib"]
from pathlib import Path
class C:
path: Path
```
on Python versions before 3.14, even though they are not actually
resolved immediately when
`__lazy_modules__` makes the imports lazy (3.15 > 3.14, as it turns
out).
Because the semantic model checks the target version when setting
`TYPING_ONLY_ANNOTATION`, we
instead check the other two annotation flags:
`RUNTIME_EVALUATED_ANNOTATION` and
`RUNTIME_REQUIRED_ANNOTATION`. If an annotation is runtime-evaluated and
not runtime-required, and
we didn't already bail out at the
`semantic.execution_context().is_runtime()` check earlier in the
same function, then we know that we're in an annotation that _would_
have been typing-only on 3.14+.
Test Plan
--
New mdtests and the ecosystem check here. I expect -12 scikit-build-core
diagnostics compared to main.
Summary
--
This PR bumps our libcst version to 1.9 and incorporates the new support
for Python 3.15 into fixes
that involved imports and unpacking in comprehensions. The unpacking
fixes affect three C4 rules and are
in the first (non-empty) commit, while the lazy import fixes are a bit
more widespread and are in
the second commit.
I admittedly wasn't that familiar with our usage of libcst before this,
but it looks like we have
kind of a weird mix between using libcst and additional manual string
manipulation. My changes
preserve that with a couple of `format!("lazy {fix}")` calls. libcst
also itself makes this change a
bit awkward by using separate node types for lazy and non-lazy imports,
so I ended up returning the
parts of the imports instead of `Import` types directly in a couple of
cases.
Test Plan
--
Updated existing snapshots for the C4 rules and new mdtests for the
import helpers
Summary
--
This PR closes#25181 by recognizing and suggesting lazy imports in the
typing-only runtime imports
rules. As noted in [PEP 810], one common use for `TYPE_CHECKING` blocks
is to avoid importing code
that isn't actually needed at runtime:
> Type annotations frequently require imports that are never used at
runtime. The common workaround
> is to wrap them in `if TYPE_CHECKING:` blocks \[1\]. With lazy
imports, annotation-only imports
> impose no runtime penalty, eliminating the need for such guards and
making annotated codebases
> cleaner.
The footnote is even more relevant:
> Furthermore, there’s also external tooling, in the form of
flake8-type-checking, because it is
> common for developers to mislocate imports and accidentally introduce
a runtime dependency on an
> import only imported in such a block. Ironically, the static type
checker is of no help in these
> circumstances.
In line with this usage, this PR exempts existing lazy imports from
these rules, which otherwise
would still suggest moving them into `TYPE_CHECKING` blocks, and it also
adds a new fix to all three
rules that will add `lazy` in front of an import on 3.15+.
In line with our fixes for `TID254` and `TID255`, we fall back on the
existing `TYPE_CHECKING` fix
if the `lazy` addition is too complicated, i.e. when multiple names are
imported on the same line or
when we'd have to edit `__lazy_modules__`.
[PEP 810]: https://peps.python.org/pep-0810/
Test Plan
--
New mdtests
Summary
--
This is a follow-up to the review on #28533. We were previously visiting
all of the keyword
arguments to functional `TypedDict` constructor calls as type
definitions, in line with this usage:
```pycon
>>> from typing import TypedDict
>>> Point2D = TypedDict('Point2D', x=int, y=int, label=str)
>>> Point2D.__required_keys__
frozenset({'x', 'y', 'label'})
```
However, this usage was actually deprecated in 3.11 and fully removed in
3.13, paving the way for
the new `closed` keyword added in 3.15.
This PR updates our `TypedDict` keyword argument traversal to visit only
these legacy arguments and
the new `extra_items` argument as types and to visit other arguments as
non-types instead.
Test Plan
--
New mdtests and a couple of updates to existing snapshots
Summary
--
[PEP 728] introduced two new keyword arguments for `TypedDict`s: the
boolean `closed` argument and
`extra_items`, which allows specifying the type for any additional
entries in the `TypedDict`. See
this example from the PEP:
```py
class Movie(TypedDict, extra_items=bool):
name: str
a: Movie = {"name": "Blade Runner", "novel_adaptation": True} # OK
b: Movie = {
"name": "Blade Runner",
"year": 1982, # Not OK. 'int' is not assignable to 'bool'
}
```
Like some of my previous 3.15 PRs, this PR adds support for visiting the
value of the `extra_items`
keyword argument as a type expression, again using `F401` to demonstrate
that such quoted
expressions mark their corresponding imports as used. This requires a
little bit of special handling
when visiting class statements, which can also be deferred in stub
files, but overall wasn't too bad
I think. `closed` doesn't need any special handling in the semantic
model since it's just a flag,
not a type expression.
This PR also updates [`convert-typed-dict-functional-to-class`
(`UP013`)] to recognize the new
keywords and carry them along in its fix.
The first commit contains the visitor changes, and the second contains
the `UP013` update.
Once again, my Codex reviewers found an existing bug that this exposes
in a new way. We could
sidestep this, and substantially simplify the code at the same time, by
dropping the recursive base
class traversal, but it seemed more useful to me to keep the recursive
traversal and fix the other
bug later if needed. Either unhandled case (multiple levels of
inheritance in one file, or
inheritance + shadowing + unrelated class with `extra_items`) seems
quite niche, though.
<details>
<summary>`any_base_class` bug</summary>
Ancestor lookup can misclassify an ordinary class as a TypedDict — The
lookup resolves ancestor
names in the current scope, so an unrelated definition can shadow the
actual base:
```py
from typing import TypedDict
class Base:
def __init_subclass__(cls, **kwargs): pass
class Middle(Base): pass
class Namespace:
class Base(TypedDict): pass
class Record(Middle, extra_items="not a type"): pass
```
This valid code now produces a false F722. `Middle` inherits the
module-level `Base`, but the lookup
finds `Namespace.Base`. The inheritance walk should use each base
expression’s original binding.
</details>
[`convert-typed-dict-functional-to-class` (`UP013`)]:
https://docs.astral.sh/ruff/rules/convert-typed-dict-functional-to-class/
[PEP 728]: https://peps.python.org/pep-0728/
Test Plan
--
New mdtests
Summary
--
Python 3.15 adds the `parent_mode` argument to both [`os.makedirs`] and
[`pathlib.Path.mkdir`], which allows setting the mode for intermediate
directories separately from the main `mode` argument. This PR adds
support for the new argument in both [`non-octal-permissions`
(`RUF064`)], which now checks this argument's permissions too, and in
[`os-makedirs` (`PTH103`)], which just needs to recognize it for
converting calls from the `os` to the `pathlib` versions.
As a note to head off other Codexes, my Codex keeps complaining about
this fix being unsafe:
```diff
import os
+import pathlib
def create():
- os.makedirs("a/b", parent_mode=(mode := 0o700), mode=mode)
+ pathlib.Path("a/b").mkdir(mode=mode, parents=True, parent_mode=(mode := 0o700))
```
because it reorders the arguments and could change the order of side
effects. This is yet another preexisting issue with the way this rule
and a couple of other PTH rules rewrite argument lists that hasn't been
reported and isn't strictly related to the 3.15 changes. If we're really
worried about that, we could just mark the fix unsafe, I guess.
[`os-makedirs` (`PTH103`)]:
https://docs.astral.sh/ruff/rules/os-makedirs/
[`non-octal-permissions` (`RUF064`)]:
https://docs.astral.sh/ruff/rules/non-octal-permissions/
[`pathlib.Path.mkdir`]:
https://docs.python.org/3.15/library/pathlib.html#pathlib.Path.mkdir
[`os.makedirs`]:
https://docs.python.org/3.15/library/os.html#os.makedirs
Test Plan
--
New mdtests for both affected rules
Summary
--
This turned into a bit of a rabbit hole, and the very short summary is
that the fix for `UP040` is now always unsafe and documented as such.
Read on for more details.
`UP040`, `UP046`, and `UP047` all transform code using legacy type
variables (`typing.{TypeVar, TypeVarTuple, ParamSpec}`) to use PEP-695
syntax instead (`UP040` also applies to non-generic type aliases but
generics are the problem here). This means transformations like this:
```diff
from typing import Generic, TypeVar, TypeAlias, TypeAliasType
T = TypeVar("T")
# UP040
-Alias: TypeAlias = list[T]
+type Alias[T] = list[T]
# UP040
-AliasType = TypeAliasType("AliasType", set[T], type_params=(T,))
+type AliasType[T] = set[T]
# UP046
-class C(Generic[T]): ...
+class C[T]: ...
# UP047
-def f(x: T): ...
+def f[T](x: T): ...
```
This works very well if the type variable is defined in the same file.
However, if `T` is imported from another file, it could have bounds,
constraints, defaults, or a different type entirely that is mishandled
by the autofix.
As it turns out, `UP046` and `UP047` already handle this pretty
gracefully. Their fixes are _always_ unsafe, and the limitations around
imports are documented. `UP047` can't tell if an imported type is
generic at all, so it simply doesn't emit a diagnostic for imported
annotation types. `UP046` offers a diagnostic but not a fix. We _could_
consider an unsafe or display-only fix, perhaps, but I think we omitted
the fix as being even more unsafe than the normally unsafe fix.
That leaves `UP040` as the problematic case. Its fix for `TypeAlias`es
was also already unsafe in `.py` files but safe in `.pyi` files.
Unfortunately, the fix can still cause problems in stub files in a case
like this:
```py
from typing import TypeAlias
from somewhere import MyTypeVar
ListOfInt: TypeAlias = list[MyTypeVar]
```
The proposed safe fix is:
```diff
-ListOfInt: TypeAlias = list[MyTypeVar]
+type ListOfInt = list[MyTypeVar]
```
which is invalid:
https://play.ty.dev/6385facd-2a61-4f3e-8961-9124c645643a because it
mixes the old-style type variable with the new `type` statement instead
of making the `type` statement itself generic:
```py
type ListOfInt[MyTypeVar] = list[MyTypeVar]
```
The example here uses `.py` files and blocks for convenience, but the
same safe fix is offered for stub files. Similarly, the playground link
uses a bounded type variable, but the problem occurs even without bounds
or other restraints.
Second, the fix for `TypeAliasType` was also safe, even for imported
type variables. The problem here is very similar to that above. For
imported type variables, the safe fix could lose bounds and constraints
or even the kind of type variable. For example:
```diff
from typing import TypeAliasType
from somewhere import Ts
# somewhere.py
# from typing import TypeVarTuple
# Ts = TypeVarTuple("Ts", bound=int)
-MyAliasType = TypeAliasType("MyAliasType", tuple[*Ts], type_params=(Ts,))
+type MyAliasType[Ts] = tuple[*Ts]
```
This presents `Ts` as a regular `TypeVar` with no bound instead of a
bounded `TypeVarTuple`.
I tried following `UP046` and omitting the fix for imported type
variables, but it was a little bit awkward for the current structure of
`UP040`. I think it's sufficient and less intrusive into the current
implementation to mark it unsafe.
I also considered trying to retain safe fixes for `UP040` in stub files
when no imported types are involved, but making the fix always unsafe
again seemed like the more consistent and simpler approach. From what I
remember, none of these rules typically has much ecosystem impact
anyway.
[#28505]: https://github.com/astral-sh/ruff/pull/28505
[PEP 695]: https://peps.python.org/pep-0695/
Test Plan
--
New mdtests and also the ecosystem check here
Summary
--
The main 3.15-related change here is that `TypeVarTuple` now supports
the `bound` argument, which
was previously only allowed for `TypeVar`s (and `ParamSpec`s, see
below):
```py
from typing import TypeVar, TypeVarTuple
T = TypeVar("T", bound=int)
Ts = TypeVarTuple("Ts", bound=int) # new in 3.15
```
But PEP 695 doesn't include syntax for these bounds:
```py
def foo[T: int](): ... # valid TypeVar version
def foo[*Ts: int](): ... # analogous but invalid TypeVarTuple syntax
```
This was actually a preexisting bug, where we'd introduce a syntax error
if someone included an
invalid `bound` on their `TypeVarTuple`:
https://play.ruff.rs/f984fc13-8936-446c-bfde-35be960a448d
but this is a bigger issue on 3.15 now that `bound` is a supported
argument without corresponding
PEP-695 syntax.
While reviewing this, I realized that the same issue applies to
`ParamSpec`, which has actually
supported `bound` since 3.10, but again, there is no analogous PEP-695
syntax:
```py
def foo[**P: int](): ... # analogous but also invalid
```
The same issues also apply to constraints, thus the new
`has_unsupported_restriction` applies to both `TypeVarTuple` and
`ParamSpec` with either bounds or constraints.
Test Plan
--
New mdtests for the affected rules
Summary
--
`typing.TypeForm` was introduced in Python 3.15 and [PEP
747](https://peps.python.org/pep-0747/) as
a way to annotate the passing of type objects like `int | str`,
`list[int]`, or `MyTypeAlias`. It
can be used either as a subscripted generic, as in this example from the
PEP:
```py
from typing import TypeForm
def trycast[T](typx: TypeForm[T], value: object) -> T | None: ...
```
or as a callable, which returns its argument unchanged:
```py
from typing import TypeForm
TypeForm("list[int]")
```
This PR recognizes `typing.TypeForm` as a standard-library generic and
as a known `typing` callable
and visits its argument or subscript as a type definition.
While I was reading the [typing docs], I realized that a couple of
nearby types were also not
listed, so I've attempted to sneak them in here but could split them off
if needed. We already had
coverage for `typing.TypeGuard` but not `typing_extensions.TypeGuard`
(added in 3.10), and we were
missing both variants of `TypeIs` (added in 3.13). These are only valid
in the subscript form, so
they don't need new `Callable` entries.
[typing docs]:
https://docs.python.org/3.15/library/typing.html#typing.TypeForm
Test Plan
--
New mdtests, using `unused-import` (`F401`) to demonstrate that visiting
these arguments as type
definitions marks them as used, but other rules will be affected as
well.
Summary
--
This is a follow-up to #28459 adding simple support for conditionally
defining `__lazy_modules__`. We now track per-module laziness, which
allows us to merge definitions like:
```py
__lazy_modules__ = ["foo"]
if condition:
__lazy_modules__ = ["foo", "bar"]
```
into known laziness for `foo` and unknown laziness for `bar`. As in
#28459, that unknown laziness is still helpful and distinct from known
eagerness because it suppresses diagnostics.
The one quirk with the `branch_id` tracking used here is that it doesn't
handle exhaustive branches. For example, it would be nice if we noticed
that `foo` is always lazy here:
```py
if condition:
__lazy_modules__ = ["foo", "bar"]
else:
__lazy_modules__ = ["foo", "baz"]
```
but it's instead merged with the implicit known eagerness of all imports
before any `__lazy_modules__` assignment, producing `(foo, Unknown)`. So
this is effectively equivalent to the version without an `else` for
`foo`.
I think this is a decent tradeoff that biases us against false
positives, though, and doesn't involve any extra bookkeeping in `if`
statement traversal.
One of my three Codex reviewers also identified this as a false
negative, where the assignment and
use are in the same branch, but we've already marked the module as
`Unknown`:
```py
if condition:
__lazy_modules__ = ["json"]
import json
json.dumps({})
```
But again we're at least biased in the right direction, and we'd need
more bookkeeping to identify
this, so I was okay accepting that for now.
Test Plan
--
New mdtests for TID254 that exercise this behavior
Summary
--
For a bit of background, `__lazy_modules__` is a backwards compatibility
mechanism for lazy imports, introduced in [PEP
810](https://peps.python.org/pep-0810/) alongside the `lazy` keyword
itself. Instead of declaring an import `lazy` with the keyword, it can
also be listed in a module-level constant named `__lazy_modules__`:
```py
lazy import foo
# equivalently
__lazy_modules__ = ["foo"]
import foo
```
This can be used in projects that support older Python versions, as well
as 3.15.
It also supports more dynamic approaches (that this PR doesn't), as
noted in one of my inline comments, such as:
```py
class LazyImporter:
def __contains__(self, name): return True
__lazy_modules__ = LazyImporter()
```
As one example from the PEP for emulating wildcard lazy imports.
<hr>
The PR is pretty big in total, but it's intended to be reviewed
commit-by-commit. As usual, the very
first commit is empty for filling the PR summary, so the numbers below
refer to the remaining
non-empty commits.
The first commit adds `__lazy_modules__` tracking to the semantic model.
Because the declaration
affects later imports and can also be overwritten, we just store
whatever was assigned to it last
for later lookup when visiting other code.
The second commit expands [TID254] to recognize `__lazy_modules__` and
includes a few helper methods
on the semantic model that are used by later rules as well.
The third commit adds `__lazy_modules__` support to [TID255], which also
relies on a new
`BindingFlags::LAZY` variant. This marks bindings as originating from a
lazy import, which is easier
than trying to resolve a sequence of potentially shadowed
`__lazy_modules__` bindings to check if an
import was lazy. For example, we set the `LAZY` flag on `foo` here,
which would otherwise require
resolve multiple previous bindings:
```py
__lazy_modules__ = ["foo"]
import foo
__lazy_modules__ = []
class C(foo.Something): ...
```
The last four commits make smaller changes to four other rules. As with
TID254 and TID255, the
primary change is that we withhold a fix that involves
`__lazy_modules__` when it would change the
module's lazy status. Initially I had hoped to add fixes that modified
`__lazy_modules__` (adding to
it when we'd otherwise add `lazy`, removing from it when we'd remove
`lazy`), but this turned out to
be quite complicated. Unlike the `lazy` keyword, `__lazy_modules__` has
nonlocal effects that could
change the behavior of _any_ import from the same module in the file.
Trying to resolve all of that
to be sure the fix is reasaonable was a real mess, and reverting Codex's
attempt saved almost 700
lines for TID255 alone.
Similarly, in case this comes up in someone else's Codex review, my
Codex initially tried to handle
even more edge cases with `__lazy_modules__` definitions, such as
definitions in a loop or
redefining it with `*` imports and things like that. I tried to rein it
in toward something more
like our `__all__` and `from __future__ import annotations` handling,
which again seemed like the
better tradeoff to me.
Closes#25170.
[TID254]: https://docs.astral.sh/ruff/rules/lazy-import-mismatch/
[TID255]:
https://docs.astral.sh/ruff/rules/lazy-import-immediately-resolved/
Test Plan
--
New mdtests. The ecosystem results all look correct. They are cases
where `TID255` now fires in modules where `__lazy_modules__` is defined.
Summary
--
If I had realized how small this was, I would have just bundled it with
#28475, but this is another
tiny change to `UP035` to avoid replacing `typing.ByteString` with
`collections.abc.ByteString`, now
that both are deprecated. The `typing` variant has been deprecated since
3.9 and the
`collections.abc` version since 3.12, and they've now been removed from
their respective `__all__`s
in 3.15 in preparation for removal in 3.17.
The `collections.ByteString` to `collections.abc.ByteString`
transformation is retained since
`collections.ByteString` was fully removed back in 3.10.
Test Plan
--
Updated an existing snapshot
Summary
--
This PR removes `typing_extensions.no_type_check_decorator` from the
list of moved members in UP035. As the
[docs](https://docs.python.org/3/library/typing.html#typing.no_type_check_decorator)
indicate, this decorator has been deprecated since 3.13 and is removed
in 3.15, so the rule should just leave it alone.
The change is not version-gated because it seemed weird to me to
recommend making this change with a target version of 3.12, for example,
only for the user to have to change it back when they eventually bump
their Python version.
I initially thought that `typing_extensions` would continue to backport
this, but it appears that they removed it as well:
https://github.com/python/typing_extensions/pull/723. I don't think that
actually changes how we should handle it, though.
Test Plan
--
Added a couple of tests showing no diagnostic
Summary
--
This moves `__cached__` from the main list of `MAGIC_GLOBALS` to a
version-specific list because it will be
[removed](https://docs.python.org/3/reference/datamodel.html#module.__cached__)
in 3.15.
Test Plan
--
A new mdtest for `undefined-name` (`F821`), which is just one of the
rules affected but the easiest to test.
Summary
--
This should avoid requesting review from owners of these crates for the
automated version bumps,
assuming they don't want to be notified :)
Test Plan
--
Future releases
Summary
--
I initially wanted to avoid emitting `TID254` when an import that was
required to be lazy was
immediately resolved. However, it became clear that that approach was
much more complicated while
reviewing #26027. This PR implements the easier fix of skipping `TID255`
for imports that are
required to be lazy by the user's configuration, even if those imports
are immediately resolved. I
think this is also a sensible route.
A close alternative would be still emitting the `TID255` diagnostic and
only suppressing the fix. I
don't really feel strongly one way or the other. The diagnostic is
possibly still helpful to let you
know that you're defeating your desired lazy import, but especially in
an example like the one from
the issue, it would be pretty hard to avoid. This would just require
moving the new check down a few
lines to implement, though.
Fixes#25418
Test Plan
--
New mdtest based on the issue
Summary
--
While reviewing #28231, Codex found a preexisting bug shared with our
unary minus handling. Namely, CPython emits an error for code like this,
where the unary expression is part of a longer expression:
```pycon
>>> match foo:
... case -1**2: ...
...
File "<python-input-0>", line 2
case -1**2: ...
^^
SyntaxError: invalid syntax
```
but Ruff does not:
```console
$ ruff check --ignore ALL - <<EOF
match foo:
case -1**2: ...
EOF
All checks passed!
```
We now verify that the parsed unary expression is a number literal and
emit an error if not.
Test Plan
--
New inline snapshots
Summary
--
As added in https://github.com/python/cpython/pull/148566, CPython now
allows unary plus in match
patterns, mirroring the support for unary minus. We already had a
special case for this for improved
error reporting and recovery, so I just swapped out the `add_error` call
for
`add_unsupported_syntax_error` with a new error kind.
Part of https://github.com/astral-sh/ruff/issues/28206
Test Plan
--
New inline parser tests
Summary
--
I've noticed that some issues seem to attract a lot of (likely
automated) duplicate PRs. #28102 is a
recent example that elicited four PRs in the five days it has been open.
I'm not sure if the changes
here will actually help, but I've tried to add both contributor- and
agent-facing documentation to
suggest checking for linked PRs before opening another one.
Test Plan
--
Future issues
Summary
--
This PR updates our ruff-pre-commit dependency to 0.16.5, which supports
the new category selectors,
and then enables `lint.preview` and the new default categories so that
we can dogfood them on our
code. There were 66 diagnostics initially, which I also fixed or
ignored:
```
28 FURB113 [ ] repeated-append
12 ASYNC240 [ ] blocking-path-method-in-async-function
6 RUF105 [*] noqa-comments
5 PLW1510 [ ] subprocess-run-without-check
3 PLW0108 [ ] unnecessary-lambda
2 EXE001 [ ] shebang-not-executable
2 RUF039 [*] unraw-re-pattern
1 B024 [ ] abstract-base-class-without-abstract-method
1 FURB142 [*] for-loop-set-mutations
1 PERF102 [ ] incorrect-dict-iterator
1 PLC0207 [*] missing-maxsplit-arg
1 PLW0127 [ ] self-assigning-variable
1 TRY004 [ ] type-check-without-type-error
1 PLW0120 [*] useless-else-on-loop
1 TRY201 [ ] verbose-raise
```
Of these, I ignored `repeated-append` and `blocking-path-method-in-async-function`. I think they are
still reasonable defaults, just less useful/important in our uses.
0.16.5 also includes https://github.com/astral-sh/ruff-pre-commit/pull/159, so I removed `markdown` from our main `ruff-format` hook, leaving it for the later mdtest formatting stage.
I also had to make a few changes related to config discovery in tests:
- move our cached ecosystem checkouts to `$RUNNER_TEMP` instead of inside of our checkout
- convert some CLI tests to use the `CliTest` helpers
- add `configurationPreference: editorOnly` to some notebook tests
In these cases, our own Ruff settings were being inherited by test code. This likely affects many other tests, but I only updated the obviously failing ones for now. I'm least confident about the notebook tests, but as far as I can tell from the history (https://github.com/astral-sh/ruff/pull/11944, https://github.com/astral-sh/ruff/pull/22810), the specific diagnostics are not the important parts being asserted.
Test Plan
--
CI on this PR
Summary
--
This PR allows adding rules that only belong to categories and only have
names rather than requiring
a linter group and code.
The first non-empty commit uses `Rule::name` instead of
`Rule::noqa_code` throughout the linter tests, causing a huge number of
file renames without changes.
The second commit updates `FixTable` from a mapping of `SecondaryCode ->
(name, fix_count)` to
`DiagnosticId -> (Option<SecondaryCode>, fix_count)`. This doesn't
really change its usage but
involves minor diffs to a substantial number of lines. It also allows us
to drop some of the special
`hashbrown` handling and the `hashbrown` dependency in `ruff_linter`,
which is nice.
The third commit contains the bulk of the change, including the macro
changes needed to support
`map_codes` entries like that for the `pytest-fixture-autouse` rule in
the fourth commit:
```rust
() => rules::ruff::rules::PytestFixtureAutouse,
```
These changes are pretty mechanical overall, just handling cases where
the linter or noqa code could
be missing, which fortunately seemed pretty straightforward in every
case.
The fourth commit takes advantage of these changes to reinstate `RUF076`
as our first purely
`pedantic` rule. I'm happy to drop that if it's too controversial or
opens too many questions about
the previously-removed rule and its code, but I thought it was nice to
have a motivating example. We
moved this rule from `preview` to `removed` because it was too pedantic
for the `RUF` category, even
in preview, so it's exactly the kind of rule we can support now with the
categories.
Finally, the fifth commit updates the contributing docs to reflect that
linters and codes are
optional. This may need to be adjusted if we back out the `RUF076`
change.
Test Plan
--
Existing tests, plus a few new ones for `pytest-fixture-autouse`
## Summary
These are the last five commits from
https://github.com/astral-sh/ruff/pull/27666 that update the preview
default selectors based on the new categorization:
- add current preview rules to the default selectors
- add third-party rules to the default selectors
- add preview rules stabilized in 0.15 to the default selectors (oops)
- incorporate the DTZ and F406 changes from
https://github.com/astral-sh/ruff/issues/27213
- adjust 4 categories that genuinely differ between my previous and
current categorizations
## Test Plan
Updated default rules snapshot
Summary
--
This PR includes the contributor-facing documentation for the new
categories, as well as updated
rule acceptance guidelines built atop them.
I think we could also update our rule request template based on some of
these guidelines, but I'll save that for a follow-up once these are
solidified.
Test Plan
--
Future rule requests
---------
Co-authored-by: Micha Reiser <micha@reiser.io>
Summary
--
This includes the user-facing updates to our linter docs and selection
settings to accompany #27666.
Test Plan
--
A few existing snapshot updates but mostly users reading this
---------
Co-authored-by: Micha Reiser <micha@reiser.io>
## Summary
This PR introduces clippy-style rule categories that can be used
alongside existing linter-based categories in preview. The categories
have a lower `specificity` than linter groups, meaning that `select =
UP` and `ignore = correctness` will not ignore `UP` correctness rules.
In general this leads to more rules being enabled than the alternative
we considered where categories and linter groups had equal precedence,
which makes it seem like a safer default. We can revisit this if it
proves to be unintuitive, or we can add a Cargo-style `priority` field
in the worst case.
I've tried to organize the commits such that this can be reviewed
commit-by-commit and could also be split into multiple PRs. The first
two commits update the `ViolationMetadata` macro to allow providing a
`category` argument and add such arguments for all rules. This part
could be a standalone change without exposing the categories to users
but still depends on reaching consensus on all rule categories. The
second pair of commits allows using the new categories as selectors
alongside existing selectors, including some precedence tests with other
selector types.
This mostly addresses #1774, but there are some sub-issues there to
resolve before closing the issue.
## Test Plan
New CLI tests and a new mdtest suite around rule precedence
Summary
--
Before this, running `./scripts/release.sh` produced this error on my
machine:
```console
❯ ./scripts/release.sh
Updating metadata with rooster...
warning: `VIRTUAL_ENV=/Users/brw/.virtualenvs/openai` does not match the project environment path `.venv` and will be ig
nored; use `--active` to target the active environment instead
error: The lockfile at `uv.lock` needs to be updated, but `--locked` was provided.
hint: To update the lockfile, run `uv lock`.
```
Passing an additional `--default-index` flag to the initial `uv run`
also works, but it seems easier
to export the environment variable for the whole workflow.
Test Plan
--
Manual testing in today's release
Summary
--
Addresses
https://github.com/astral-sh/ruff/pull/27666#discussion_r3801704517,
where we want to
start using "group" to refer to secondary categories. This type is
already referred to as `status`
in the user-facing JSON output, so this change aligns the internal type
name with that usage.
Test Plan
--
Existing tests
Summary
--
I thought this was the case before
(https://github.com/astral-sh/ruff/issues/18686#issuecomment-2977459714),
but apparently that only applied to `ruff-lsp`, which I was somehow
still using at the time. This change exposes display-only fixes in the
native server, like in `ruff-lsp` and the [playground].
We append `(suggestion)` to the fix title for display-only fixes in an
effort to distinguish them from the more reliable safe and unsafe fixes.
To further differentiate these from safe fixes, this PR also sets the
`isPreferred` property to `true` for safe fixes. This means such fixes
can be applied automatically with the "Auto Fix" command (distinct from
the "Quick Fix" command, which always opens a menu), should sort first,
and get a slightly different lightbulb icon, at least in VS Code.
Before landing on this, I tried adding a [request for
confirmation](https://code.visualstudio.com/api/references/vscode-api#WorkspaceEditEntryMetadata)
to the fix, but the UI for this seems really annoying in VS Code. It
opens this separate panel, and you have to click the check box and also
click Apply:
<img width="597" height="236" alt="image"
src="https://github.com/user-attachments/assets/53c97e14-1f5d-49e7-8e6c-73d6c5552eb2"
/>
That seemed to go a bit farther than we need, especially relative to the
code change required and after I vetted our display-only fixes. Most of
them are only a bit less safe than our unsafe fixes. We don't have any
that seem obviously disastrous. Related to the scope of the code change,
this functionality may also not be available in some clients, at least
according to Codex. However, one cool thing about this approach is that
we could pass down rule-specific information about why the fix needs
confirmation. That would obviously increase the scope even more but was
an interesting finding.
These changes align pretty well with what rust-analyzer does. Its
applicability levels are a bit different from ours, but it marks
`MachineApplicable` fixes as preferred, `MaybeIncorrect` fixes as not
preferred, and `HasPlaceholders` suggestions don't have quick fixes. The
main difference is that our `DisplayOnly` fixes seem considerably safer
than
[`HasPlaceholders`](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_lint_defs/enum.Applicability.html#variant.HasPlaceholders),
so we can still set non-preferred fixes on them.
Test Plan
--
Updated existing snapshots and manual testing in VS Code. The quick fix
list previously only offered
"Disable for this line" but now also offers the display-only fix:
<img width="528" height="349" alt="image"
src="https://github.com/user-attachments/assets/adc9503d-446e-4360-a112-64df918bc1c4"
/>
I also tested the `Ruff: Fix all auto-fixable problems` and `Fix all`
commands. These still don't apply display-only fixes. Only inline quick
fixes include display-only fixes.
[playground]: https://play.ruff.rs/487aa90c-7809-4405-b9c5-ef9c5b8cf9c7
Summary
--
As uncovered in #17804, `F722` previously applied only to parse errors
and not to semantic syntax
errors. More confusingly, we were emitting semantic syntax errors, and
even other lint rule errors
that correspond to syntax errors, on string type definitions, even when
`F722` was disabled.
This PR adds a check to `Checker::report_semantic_error` for the
semantic model's
`in_string_type_definition` flag that converts these semantic errors
into regular `F722`
diagnostics, when the rule is enabled, and returns without reporting an
error when the rule is
disabled.
Test Plan
--
New mdtests covering semantic errors in forward annotations
Summary
--
I saw this suggestion on [Discord]:
> Would it make sense to put a mark in the full rule page for rules that
are selected by default? I
feel like that would be very useful in seeing for example what rules are
not selected from a group
like UP by default. The extra "default rules" page doesn't help with
that, for example.
> Like a blue dot, maybe in front of the rules in the table.
This PR implements this suggestion with a couple of modifications. I
included a check mark instead of a blue dot and as an additional
indicator at the end of each row, near the preview, fix, and other
indicators, rather than at the start.
Test Plan
--
Local build:
<img width="569" height="237" alt="image"
src="https://github.com/user-attachments/assets/1a6db243-ac2c-4b53-a378-541467a6018d"
/>
[Discord]:
https://discord.com/channels/1039017663004942429/1070132471699607623/1536942546972839957
Summary
--
Fixes#27333. The fixed annotation isn't actually equivalent in this
case.
```pycon
>>> import functools
...
... @functools.singledispatch
... def int_or_float(value: object) -> str:
... return False
...
... @int_or_float.register
... def _(value: int | float):
... return True
...
... assert int_or_float(3)
...
>>> import functools
...
... @functools.singledispatch
... def int_or_float(value: object) -> str:
... return False
...
... @int_or_float.register
... def _(value: float):
... return True
...
... assert int_or_float(3)
...
Traceback (most recent call last):
File "<python-input-1>", line 11, in <module>
assert int_or_float(3)
~~~~~~~~~~~~^^^
AssertionError
```
We had an existing helper for testing for `singledispatch` functions,
which we use to skip the first
parameter of such functions since this is the dispatching parameter.
Test Plan
--
New mdtests
Summary
--
Addresses
https://github.com/astral-sh/ruff-vscode/pull/1127#discussion_r3684426847,
which would
otherwise expose Ruff as a TOML formatter in VS Code. Instead of
registering formatting and range
formatting capabilities statically, we now register them dynamically
only for Python, Markdown, and
notebook files, excluding TOML files.
We still fall back on static registration if dynamic registration is
unavailable.
Test Plan
--
New e2e tests
Summary
--
This PR adds TOML linting and fixing support to the LSP following
https://github.com/astral-sh/ruff/pull/26772. It's organized as 3
initial refactoring commits followed by the two commits actually adding
TOML support.
After this lands, we'll also need to add TOML support in the VS Code
extension, like we did for Markdown files in
https://github.com/astral-sh/ruff-vscode/pull/950.
Test Plan
--
New e2e tests
Summary
--
This PR closes#26320 by adding the new
`lint.prefer-rule-codes-in-output` option that defaults to
`false` and disables the use of human-readable rule names in most output
formats, even in preview.
This does not affect formats like JSON that include the name and code as
separate fields.
Test Plan
--
New CLI test enabling the option, as well as manual testing in the
playground:
<img width="597" height="299" alt="image"
src="https://github.com/user-attachments/assets/e56e3681-5892-4dd6-ba66-1a91bc9529d3"
/>
and in VS Code:
<img width="738" height="396" alt="image"
src="https://github.com/user-attachments/assets/98ee3baa-c1fe-421c-a9a1-d63e5f4cc23b"
/>
Note the human-readable name in the `ruff: ignore` comment, which shows
that preview is enabled, while the `(F401)` in the output shows that the
setting is being used.
Summary
--
Fixes#27245.
Test Plan
--
Updated existing snapshots. These were previously emitting both B006 on
the outer comprehensions and
B008 on the inner `range` call, but now the inner diagnostic is omitted
because `range` is
immutable.
Summary
--
Now that we allow rule names in selectors in preview, we need to
normalize them back to codes to
allow running on stable. This PR implements this normalization as one of
our config overrides,
using the comparison executable to extract the rule mapping from `ruff
rule`.
Test Plan
--
CI on this PR and then #27154
Summary
--
This makes sense to accompany `ruff: ignore` comments being stabilized.
The preview gate has been pushed down to cover only the names vs codes
decision. I'll flag this as a comment, but I wasn't totally sure about
the LSP case. I left using `ruff: ignore` in its suppression edits
preview-gated for now.
Test Plan
--
Existing tests, dropping one that showed the preview error with
`--add-ignore`. I left `--preview` in many of the tests to continue
testing the human-readable names but removed it from one.
This PR stabilizes the expanded default rule set currently in preview
and adds a new generated `Default Rules` page to the docs to avoid
needing to include a lengthy selector block in multiple places.
Summary
--
This PR fixes#24528 by attaching more information to
`FormatCommandError::Parse` instances. I made
a couple of attempts at Micha's suggestion ([1]) to convert to
`Diagnostic`s much earlier, but doing
this only for errors was pretty awkward while still producing a much
larger diff. I think it would
be better to use diagnostics more deeply both for the error and
non-error cases eventually, which
should also help with #8232, but I considered the limited fix here a
better stopgap for the 0.16
release.
[1]:
https://github.com/astral-sh/ruff/issues/24528#issuecomment-4228856614
Test Plan
--
Updated snapshots for two existing tests, plus an additional notebook
test
Codex tried to sneak in bug fixes for
https://github.com/astral-sh/ruff/issues/26282, but I don't think these
have to block stabilization since they haven't come up in the wild, and
I plan to work on this issue after the minor release.
This one required a couple of updates to existing tests because the rule
was being used as a generic
preview rule. I switched these tests to use `RUF911`, which is our
`preview-test-rule`.
I had also previously opened a tracking issue for this rule's
stabilization in #19487, but as I
noted in
https://github.com/astral-sh/ruff/issues/19487#issuecomment-4454194086,
I no longer feel
like these issues really need to block stabilization. This rule is a
commonly requested reason to
enable specific preview rules and has been in preview for almost 3
years, so I think these other
issues can be handled separately.
I think we can close#19487 if we land this but keep the individual
issues open.
Summary
--
This PR stabilizes the Markdown formatting initially added in #22470. It
also reverts much of the
error plumbing in the LSP that was added in #24150. I considered keeping
the
`FormatResult::PreviewOnly` variant but went ahead and dropped it since
it was unused.
Closes#26866.
Test Plan
--
Updated existing snapshots, dropping `--preview` in some cases
Summary
--
Follow up to
https://github.com/astral-sh/ruff/pull/26958#discussion_r3607966160.
This setting was now always* set to true and only read in the `full`
output format, so we can just remove it.
The few cases where it wasn't already set to true were in tests, but it
seems fine or even
preferable to show the fixes there too.
Test Plan
--
Updated existing snapshots
Summary
--
This PR closes#19263 by stabilizing the part of the JSON output changes
that prompted me to open
the issue. The remaining uses of `DisplayDiagnosticConfig::preview`
relate to human-readable names
and severity.
I don't think it would really hurt in this case to stabilize all of the
differences, but I preserved
the separation between the earlier location/filename changes and
name/severity for now.
Test Plan
--
Updated snapshots
Summary
--
Closes#20755. I'll stack another PR on top of this to fix#24528, but
you can see the same issue in
one of the new snapshots.
Test Plan
--
Updated existing snapshots
Codex noticed a fix safety bug where comments could be deleted in a safe
fix branch, so I fixed that
too. Docs and tests looked good
```py
from decimal import Decimal
_ = Decimal.from_float(
# keep this comment
float("inf")
)
```
[Playground](https://play.ruff.rs/1016fd80-d170-4765-9856-c5b5f6b7fdeb)
Brent Westbrook