Enable `require-build-hashes` for the Python projects and inline scripts
so isolated builds require declared hashes for their build dependencies.
Pin the build backends and their dependencies with hashes.
---------
Co-authored-by: zaniebot <242828183+zaniebot@users.noreply.github.com>
Summary
--
While looking at #28887, I noticed a repeated ecosystem check failure
caused by a git error:
> Failed to commit: fatal: failed to encode
'tests/roots/test-root/wrongenc.inc' from iso-8859 to UTF-8
This is caused by a known upstream bug (sphinx-doc/sphinx#14364) with an
open PR (sphinx-doc/sphinx#14467), but the PR is waiting for the next
release milestone, I think, so it seemed easiest just to temporarily
remove the project to unblock CI checks.
Test Plan
--
CI on this PR
Summary
--
This PR updates our ruff-pre-commit dependency to 0.16.5, which supports
the new category selectors,
and then enables `lint.preview` and the new default categories so that
we can dogfood them on our
code. There were 66 diagnostics initially, which I also fixed or
ignored:
```
28 FURB113 [ ] repeated-append
12 ASYNC240 [ ] blocking-path-method-in-async-function
6 RUF105 [*] noqa-comments
5 PLW1510 [ ] subprocess-run-without-check
3 PLW0108 [ ] unnecessary-lambda
2 EXE001 [ ] shebang-not-executable
2 RUF039 [*] unraw-re-pattern
1 B024 [ ] abstract-base-class-without-abstract-method
1 FURB142 [*] for-loop-set-mutations
1 PERF102 [ ] incorrect-dict-iterator
1 PLC0207 [*] missing-maxsplit-arg
1 PLW0127 [ ] self-assigning-variable
1 TRY004 [ ] type-check-without-type-error
1 PLW0120 [*] useless-else-on-loop
1 TRY201 [ ] verbose-raise
```
Of these, I ignored `repeated-append` and `blocking-path-method-in-async-function`. I think they are
still reasonable defaults, just less useful/important in our uses.
0.16.5 also includes https://github.com/astral-sh/ruff-pre-commit/pull/159, so I removed `markdown` from our main `ruff-format` hook, leaving it for the later mdtest formatting stage.
I also had to make a few changes related to config discovery in tests:
- move our cached ecosystem checkouts to `$RUNNER_TEMP` instead of inside of our checkout
- convert some CLI tests to use the `CliTest` helpers
- add `configurationPreference: editorOnly` to some notebook tests
In these cases, our own Ruff settings were being inherited by test code. This likely affects many other tests, but I only updated the obviously failing ones for now. I'm least confident about the notebook tests, but as far as I can tell from the history (https://github.com/astral-sh/ruff/pull/11944, https://github.com/astral-sh/ruff/pull/22810), the specific diagnostics are not the important parts being asserted.
Test Plan
--
CI on this PR
Summary
--
This PR allows adding rules that only belong to categories and only have
names rather than requiring
a linter group and code.
The first non-empty commit uses `Rule::name` instead of
`Rule::noqa_code` throughout the linter tests, causing a huge number of
file renames without changes.
The second commit updates `FixTable` from a mapping of `SecondaryCode ->
(name, fix_count)` to
`DiagnosticId -> (Option<SecondaryCode>, fix_count)`. This doesn't
really change its usage but
involves minor diffs to a substantial number of lines. It also allows us
to drop some of the special
`hashbrown` handling and the `hashbrown` dependency in `ruff_linter`,
which is nice.
The third commit contains the bulk of the change, including the macro
changes needed to support
`map_codes` entries like that for the `pytest-fixture-autouse` rule in
the fourth commit:
```rust
() => rules::ruff::rules::PytestFixtureAutouse,
```
These changes are pretty mechanical overall, just handling cases where
the linter or noqa code could
be missing, which fortunately seemed pretty straightforward in every
case.
The fourth commit takes advantage of these changes to reinstate `RUF076`
as our first purely
`pedantic` rule. I'm happy to drop that if it's too controversial or
opens too many questions about
the previously-removed rule and its code, but I thought it was nice to
have a motivating example. We
moved this rule from `preview` to `removed` because it was too pedantic
for the `RUF` category, even
in preview, so it's exactly the kind of rule we can support now with the
categories.
Finally, the fifth commit updates the contributing docs to reflect that
linters and codes are
optional. This may need to be adjusted if we back out the `RUF076`
change.
Test Plan
--
Existing tests, plus a few new ones for `pytest-fixture-autouse`
Python tooling can currently build third-party source distributions. Set
`no-build` in the maintained Python projects and PEP 723 scripts,
retaining named exceptions for Git-pinned tooling and the local
ecosystem runner. Pin the isolated build dependencies for those
exceptions, including their transitive requirements.
Follows merged astral-sh/ruff#27856, which gives the legacy `ruff-lsp`
suite a dedicated, locked Python 3.12 test environment.
Follow-up to astral-sh/ruff#27843. Replaces astral-sh/ruff#27840, which
closed when the merged docs branch was deleted.
Co-authored-by: zaniebot <242828183+zaniebot@users.noreply.github.com>
<!--
Thank you for contributing to Ruff/ty! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title? (Please prefix
with `[ty]` for ty pull
requests.)
- Does this pull request include references to any relevant issues?
- Does this PR follow our AI policy
(https://github.com/astral-sh/.github/blob/main/AI_POLICY.md)?
-->
## Summary
As a pywin32 maintainer and Ruff user, I'd like to add pywin32 to the
ecosystem checks.
I think that pywin32 would be a good fit given its age, size,
popularity, and importance to the Python on Windows ecosystem. As well
as my efforts in the past 3 years modernizing its Python codebase.
Most of the Ruff issues / requests I opened have come from using it in
setuptools and pywin32.
## Test Plan
N/A ?
## Summary
Reintroducing `demisto/content` back to `ruff-ecosystem`, it's a huge
repo with 4.5K py file and 2.5M loc, so may have catch some cases for
ecosystem report. It was previously commented out in
https://github.com/astral-sh/ruff/pull/12129 due to the use of removed
`E999`
Issue with the use of removed rules upstream is resolved, though
apparently `E999` was dropped from selection a while ago and new issue
with the use of removed `UP038` occurred, but it's resolved now too -
https://github.com/demisto/content/pull/45227
Syntax error in still present in
https://github.com/demisto/content/blob/master/Packs/ThreatQ/Integrations/ThreatQ/ThreatQ.py,
so keeping `exclude`. Though apparently it doesn't break
`ruff-ecosystem` anymore, but keeping it safe for now. I've submitted a
fix upstream https://github.com/demisto/content/pull/45275, so `exclude`
can be removed later too.
## Test Plan
Tested running `ruff-ecosystem` locally, no issues found.
<!--
Thank you for contributing to Ruff/ty! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title? (Please prefix
with `[ty]` for ty pull
requests.)
- Does this pull request include references to any relevant issues?
- Does this PR follow our AI policy
(https://github.com/astral-sh/.github/blob/main/AI_POLICY.md)?
-->
## Summary
Previously disabled in https://github.com/astral-sh/ruff/pull/9854
Not sure which preview rule was the blocker exactly, but apparently it's
now stabilized and we can use `sphinx` in `ruff-ecosystem`.
## Test Plan
I ran `ruff-ecosystem` locally with and without `--force-preview` and it
worked without issues.
<!--
Thank you for contributing to Ruff/ty! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title? (Please prefix
with `[ty]` for ty pull
requests.)
- Does this pull request include references to any relevant issues?
- Does this PR follow our AI policy
(https://github.com/astral-sh/.github/blob/main/AI_POLICY.md)?
-->
## Summary
This reverts commit dd16d689ab (#23921),
since issue upstream is fixed in
https://github.com/apache/airflow/pull/64920 and workaround is no longer
needed.
https://patch-diff.githubusercontent.com/raw/apache/airflow/pull/64920.diff
```diff
diff --git a/task-sdk/src/airflow/sdk/_shared/AGENTS.md b/task-sdk/src/airflow/sdk/_shared/AGENTS.md
index 8d2c7cf9d3367..57a3a75046e45 120000
--- a/task-sdk/src/airflow/sdk/_shared/AGENTS.md
+++ b/task-sdk/src/airflow/sdk/_shared/AGENTS.md
@@ -1 +1 @@
-../../../../../airflow-core/src/airflow/_shared/AGENTS..md
\ No newline at end of file
+../../../../../airflow-core/src/airflow/_shared/AGENTS.md
\ No newline at end of file
```
So workaround is no longer needed.
## Test Plan
`ruff-ecosystem` runs without issues.
Summary
--
Now that we allow rule names in selectors in preview, we need to
normalize them back to codes to
allow running on stable. This PR implements this normalization as one of
our config overrides,
using the comparison executable to extract the rule mapping from `ruff
rule`.
Test Plan
--
CI on this PR and then #27154
Summary
--
This is the first in a series of migration rules to help users move from
`noqa` comments with rule
codes to `ruff:ignore` comments with names. This rule simply replaces
`noqa: codes` with
`ruff:ignore[codes]` (and the file-level variants), deferring the
transformation of codes to names
to a follow-up rule. This rule may be useful on its own without the
codes -> names rule, assuming we
stabilize `ruff:ignore` in the next minor release without stabilizing
human-readable names.
The rule skips any comment consisting entirely of known `external`
selectors, while still emitting diagnostics for totally unknown codes.
We also skip file-level `flake8: noqa` directives assuming they are
intended to be shared with flake8.
I initially wrote a small, standalone implementation in the rule file
itself, but it had to be
combined with the `RUF100` implementation to leave unused `noqa` codes
out of `RUF105` to be cleaned
up by `RUF100` instead, as I note in the mdtest for this case.
Test Plan
--
New mdtests
I also expect a huge number of ecosystem results on this PR, possibly so
many that the check times out.
---------
Co-authored-by: Micha Reiser <micha@reiser.io>
## Summary
Follow up to #26050 to clean up the ecosystem check script now that
diagnostics without severity are the baseline again. This reverts both
the temporary normalization code from that PR and the earlier regex
change from https://github.com/astral-sh/ruff/pull/23845.
## Test Plan
CI on this PR
Summary
--
This reverts #23845 since we're now prioritizing the human-readable
names and category work before getting back to severity.
I had to revert (the revert of) the changes to the ecosystem check
script and then add another normalization pass again to get CI to pass.
I can follow up immediately to revert these changes once this is the
baseline.
Test Plan
--
Existing tests updated following #25937
I said I didn't love diagnostics like this:
```
fib.py:1:8: unused-import: [*] `os` imported but unused
```
with both a human-readable name and a fix indicator on that PR, but I
think it's not actually that
bad, and I haven't thought of a better alternative, so I think it's fine
for now.
Summary
--
Reverts the name normalization part of #25937 now that human-readable
names are also present in the
baseline for future ecosystem check runs.
Test Plan
--
CI on this PR
## Summary
This PR updates all of our CLI output formats to use human-readable
names instead of rule codes in preview.
## Test Plan
The first commit adds a new CLI test and then each following commit
updates one output format and the associated snapshots.
The last commit also updates the ecosystem check to normalize rule names
back to codes to avoid a huge diff that causes the check to time out.
Alternatively, we may just want to accept the time out on this PR so
that we can see the un-normalized report in the future.
Summary
--
I noticed while extending the regex to match human-readable names that
it also didn't cover the
ASYNC rules, which have five letters in the rule code. For example,
#24644 revealed no new
diagnostics in the ecosystem, but a local run on my cached ecosystem
repos revealed many
diagnostics. This won't show up as a diff in CI on this PR but should
help with ecosystem checks on
the ASYNC rules in the future.
This is just the "aesthetic" part of implementing `warning` severities.
Configuration changes are in
https://github.com/astral-sh/ruff/pull/23846
It's expected that the `preview` ecosystem check times out here.
Summary
--
The huge number of changes in
https://github.com/astral-sh/ruff/pull/22205#issuecomment-3696660639
should have
obviously been a red flag, but I think it would be nice if CI failed
when new
ecosystem panics were introduced. This PR adds a check for diagnostic
lines that
start with `panic: Panicked at crates/`, raises a `ToolError` if any are
found
in the results from the comparison executable, and then ~~also exits
non-zero if
any errors are returned~~ fails the CI run if the corresponding error
message
was printed.
After trying this out in CI, I opted not to change the script's exit
code itself
because that suppressed the ecosystem comment. It feels a little hackier
this way but preserves the behavior I wanted of both failing CI and
still getting
the ecosystem comment to help with debugging.
Test Plan
--
Local testing on the 0.15.3 tag showing that ruff-ecosystem exited
non-zero and
some manual testing in CI, as you can see below.
Summary
--
I think Micha suggested this before, but I couldn't find the message.
This also
may have helped to catch #23125.
Test Plan
--
CI on this PR. I think this is a pretty large repo, so I want to make
sure it
doesn't cause too much of a slowdown, in addition to the job passing.
However,
it only takes ~4 seconds to check locally compared to ~22 for airflow,
despite
containing more files (16k vs 6.5k) and more lines of code (3 million vs
1.2
million).
## Summary
This PR is a spin-off from https://github.com/astral-sh/ruff/pull/19415.
It enables replacing the severity and lint name in a ty-style
diagnostic:
```
error[unused-import]: `os` imported but unused
```
with the noqa code and optional fix availability icon for a Ruff
diagnostic:
```
F401 [*] `os` imported but unused
F821 Undefined name `a`
```
or nothing at all for a Ruff syntax error:
```
SyntaxError: Expected one or more symbol names after import
```
Ruff adds the `SyntaxError` prefix to these messages manually.
Initially (d912458), I just passed a `hide_severity` flag through a
bunch of calls to get it into `annotate-snippets`, but after looking at
it again today, I think reusing the `None` severity/level gave a nicer
result. As I note in a lengthy code comment, I think all of this code
should be temporary and reverted when Ruff gets real severities, so
hopefully it's okay if it feels a little hacky.
I think the main visible downside of this approach is that we can't
style the asterisk in the fix availabilty icon in cyan, as in Ruff's
current output. It's part of the message in this PR and any styling gets
overwritten in `annotate-snippets`.
<img width="400" height="342" alt="image"
src="https://github.com/user-attachments/assets/57542ec9-a81c-4a01-91c7-bd6d7ec99f99"
/>
Hmm, I guess reusing `Level::None` also means the `F401` isn't red
anymore. Maybe my initial approach was better after all. In any case,
the rest of the PR should be basically the same, it just depends how we
want to toggle the severity.
## Test Plan
New `ruff_db` tests. These snapshots should be compared to the two tests
just above them (`hide_severity_output` vs `output` and
`hide_severity_syntax_errors` against `syntax_errors`).
Summary
--
I've been noticing this failure in the formatter ecosystem check and
decided to
look into it. We fail to parse the
[notebook](https://github.com/openai/openai-cookbook/blob/main/examples/mcp/databricks_mcp_cookbook.ipynb)
because some of the `code` cells
have non-Python code in them. `ruff format` only reports one of these,
corresponding to a shell snippet, but `ruff check` emits some additional
errors
about JS code later in the file too:
```
databricks_mcp_cookbook.ipynb:cell 21:1:11: SyntaxError: Simple statements must be separated by newlines or semicolons
databricks_mcp_cookbook.ipynb:cell 21:1:19: SyntaxError: Simple statements must be separated by newlines or semicolons
databricks_mcp_cookbook.ipynb:cell 21:1:50: SyntaxError: Simple statements must be separated by newlines or semicolons
databricks_mcp_cookbook.ipynb:cell 30:4:7: SyntaxError: Simple statements must be separated by newlines or semicolons
databricks_mcp_cookbook.ipynb:cell 30:4:41: E703 Statement ends with an unnecessary semicolon
databricks_mcp_cookbook.ipynb:cell 30:5:14: SyntaxError: Expected ':', found '{'
databricks_mcp_cookbook.ipynb:cell 30:6:9: SyntaxError: Expected ',', found '{'
databricks_mcp_cookbook.ipynb:cell 30:6:25: SyntaxError: Expected ',', found '='
databricks_mcp_cookbook.ipynb:cell 30:6:46: SyntaxError: Expected ',', found ';'
databricks_mcp_cookbook.ipynb:cell 30:6:47: SyntaxError: Expected '}', found newline
databricks_mcp_cookbook.ipynb:cell 30:7:1: SyntaxError: Unexpected indentation
databricks_mcp_cookbook.ipynb:cell 30:7:13: SyntaxError: Expected ':', found 'break'
databricks_mcp_cookbook.ipynb:cell 30:7:18: E703 Statement ends with an unnecessary semicolon
databricks_mcp_cookbook.ipynb:cell 30:8:28: SyntaxError: Simple statements must be separated by newlines or semicolons
databricks_mcp_cookbook.ipynb:cell 30:8:55: E703 Statement ends with an unnecessary semicolon
databricks_mcp_cookbook.ipynb:cell 30:9:18: SyntaxError: Expected an expression
databricks_mcp_cookbook.ipynb:cell 30:10:11: SyntaxError: Expected ',', found name
databricks_mcp_cookbook.ipynb:cell 30:10:16: SyntaxError: Expected ',', found '='
databricks_mcp_cookbook.ipynb:cell 30:10:22: SyntaxError: Expected ',', found name
databricks_mcp_cookbook.ipynb:cell 30:10:24: SyntaxError: Expected ',', found ';'
databricks_mcp_cookbook.ipynb:cell 30:11:27: SyntaxError: Expected ',', found '='
databricks_mcp_cookbook.ipynb:cell 30:11:34: SyntaxError: Expected ',', found name
databricks_mcp_cookbook.ipynb:cell 30:11:48: SyntaxError: Expected ',', found ';'
databricks_mcp_cookbook.ipynb:cell 30:11:49: SyntaxError: Expected '}', found NonLogicalNewline
databricks_mcp_cookbook.ipynb:cell 30:12:1: SyntaxError: Unexpected indentation
databricks_mcp_cookbook.ipynb:cell 30:12:16: E703 Statement ends with an unnecessary semicolon
databricks_mcp_cookbook.ipynb:cell 30:13:3: SyntaxError: Expected a statement
databricks_mcp_cookbook.ipynb:cell 30:13:4: SyntaxError: Expected a statement
databricks_mcp_cookbook.ipynb:cell 30:13:5: SyntaxError: Expected a statement
databricks_mcp_cookbook.ipynb:cell 30:13:5: E703 Statement ends with an unnecessary semicolon
databricks_mcp_cookbook.ipynb:cell 30:13:6: SyntaxError: Expected a statement
databricks_mcp_cookbook.ipynb:cell 30:14:1: SyntaxError: Expected a statement
databricks_mcp_cookbook.ipynb:cell 30:14:2: SyntaxError: Expected a statement
```
Test Plan
--
This PR
Summary
--
This should resolve the formatter ecosystem errors we've been seeing
lately. https://github.com/mesonbuild/meson-python/pull/728 added the
links, which I think are intentionally broken for testing purposes.
Test Plan
--
Ecosystem check on this PR
Summary
--
This updates the regex in `ruff-ecosystem` to catch syntax errors in an
effort to prevent bugs like #16874. This should catch `ParseError`s,
`UnsupportedSyntaxError`s, and the upcoming `SemanticSyntaxError`s.
Test Plan
--
I ran the ecosystem check locally comparing v0.11.0 and v0.11.1 and saw
a large number (2757!) of new syntax errors. I also manually tested the
regex on a few lines before that.
If we merge this before #16878, I'd expect to see that number decrease
substantially in that PR too, as another test.
## Summary
@zanieb noticed while we were discussing #12595 that this flag is now
unnecessary, so remove it and the flags which reference it.
## Test Plan
Question for maintainers: is there a test to add *or* remove here? (I’ve
opened this as a draft PR with that in view!)
renovate[bot]