mirror of
https://github.com/astral-sh/ruff.git
synced 2026-10-28 10:45:37 +08:00
Alex Waygood
2e34f5ba22
Our `py-fuzzer` script currently limits each failing seed to have a maximum of 60 seconds for minimization, even when the CI job has time remaining. For example, in [this fuzz job on #29025](https://github.com/astral-sh/ruff/actions/runs/37632264340/job/112829497328), minimization timed out for seed 57 even though there would have been another 15 minutes before GitHub cancelled the CI job. Running the script locally demonstrated that pysource_minimize was [easily able](https://github.com/astral-sh/ruff/pull/29025#issuecomment-6064567417) to minimize the panic if only given slightly more time to do so. This PR replaces the per-seed limit in CI with a global, shared budget for checking seeds and minimizing failures. Local runs have no overall budget by default. CI, however, records the job start time in its first step and passes it and the job timeout to the fuzzer, which subtracts elapsed setup and build time and reserves an additional 90 seconds for other job overhead and reporting. Individual calls to ty or Ruff still have a five-second timeout, allowing us to minimize hangs or pathological performance issues. By default, the fuzzer reports failing seeds as they are found and prints completed minimizations. When the deadline is reached, it stops starting new seed and minimizer checks, reports the original source for unfinished minimizations, and identifies unchecked seeds. Checks already in progress can finish after the deadline. The logs and, when requested, the GitHub issue body include a command to rerun failing and unchecked seeds; the script exits with status 2 if any seeds were left unchecked. Here are some mockups of what bot-opened issues could look like, using fictional data: <details> <summary>Issue with minimized, unminimized, and unchecked seeds</summary> Run listed here: https://github.com/astral-sh/ty/actions/runs/1234567890 Each snippet caused ty to exit with a status other than 0, 1, or 2, or time out after five seconds during fuzzing. Build ty from commit `0123456789abcdef0123456789abcdef01234567` with `cargo build --locked --bin=ty`. Save a snippet to `repro.py`, then run: ``` target/debug/ty check repro.py --python-version=3.10 --python-platform=linux ``` ### Seed: 78 ```python class Example: pass ``` <details> <summary>Seed: 93 (original source; minimization did not complete)</summary> ```python class Original: value = [1, 2, 3] def method(self): return self.value ``` </details> ### Unchecked seeds The time budget expired before 2 seeds could be checked. - `123` - `124` To check all seeds that failed or were left unchecked, check out Ruff commit `0123456789abcdef0123456789abcdef01234567` and run from the repository root: ```shell uv run --project=./python/py-fuzzer fuzz --bin=ty 78 93 123 124 ``` </details> <details> <summary>Issue with unchecked seeds and no failures</summary> Run listed here: https://github.com/astral-sh/ruff/actions/runs/1234567890 Test executable: ruff at commit `0123456789abcdef0123456789abcdef01234567`. ### Unchecked seeds The time budget expired before 3 seeds could be checked. - `10` - `11` - `12` To check all seeds that failed or were left unchecked, check out Ruff commit `0123456789abcdef0123456789abcdef01234567` and run from the repository root: ```shell uv run --project=./python/py-fuzzer fuzz --bin=ruff 10 11 12 ``` </details> ## Rejected alternatives 1. Keep a per-minimization budget, but just increase it (by a lot?). We could do this, and it fixes the theoretical problem that codex raised where a couple of seeds that take a very long time to minimize stop the script from even discovering bugs in later seeds (because it never even gets to those later seeds). I decided against this approach, however, because in practice I think it will only be hangs that take a _long_ time to minimize. We've seen in the past that a ty hang can cause the script as a whole to timeout in CI, but whenever it's done so in the past, it's never caused the script to fail to get onto non-hanging seeds. It's always been the case that the script has completed all seeds except the problematic ones that were causing it to hang. So I decided that per-minimization budgets were a solution looking for a problem. 2. Get rid of all budgets altogether, go back to something more similar to what the script was like prior to https://github.com/astral-sh/ruff/pull/29045. This is very attractive, because it avoids a lot of the new complexity added in this PR to do with calculating budgets, interrupting subprocesses that have already been started, etc. I decided against this, however, because I do think the fact that the script now minimizes hangs substantially increases the risk that the script might finish without having finished minimizing all seeds. In the event that this happens, I _do_ think it's quite useful (though not essential) for the script to report which seeds it didn't get to, before GitHub cancels the CI job altogether.
62 lines
1.5 KiB
Django/Jinja
62 lines
1.5 KiB
Django/Jinja
Run listed here: {{ run_url }}
|
|
|
|
{% if not reproducers %}
|
|
Test executable: {{ executable }} at commit `{{ executable_revision }}`.
|
|
|
|
{% else %}
|
|
{{ failure_description }}
|
|
|
|
Build {{ executable }} from commit `{{ executable_revision }}` with `cargo build --locked --bin={{ executable }}`. Save a snippet to `repro.py`, then run:
|
|
|
|
```
|
|
{{ reproduction_command }}
|
|
```
|
|
|
|
{% for reproducer in reproducers %}
|
|
{% if reproducer.bug.minimization_succeeded %}
|
|
|
|
### Seed: {{ reproducer.seed }}
|
|
|
|
{{ reproducer.fence }}python
|
|
{{ reproducer.bug.source }}
|
|
{{ reproducer.fence }}
|
|
{% else %}
|
|
|
|
<details>
|
|
<summary>Seed: {{ reproducer.seed }} (original source; minimization did not complete)</summary>
|
|
|
|
{{ reproducer.fence }}python
|
|
{{ reproducer.bug.source }}
|
|
{{ reproducer.fence }}
|
|
|
|
</details>
|
|
{% endif %}
|
|
{% endfor %}
|
|
{% endif %}
|
|
{% if omitted %}
|
|
|
|
Some reproducers were omitted to fit the issue body. The workflow logs list all failing seeds; run py-fuzzer from Ruff commit `{{ fuzzer_revision }}` with a seed to regenerate its input.
|
|
{% endif %}
|
|
{% if unchecked_count %}
|
|
|
|
### Unchecked seeds
|
|
|
|
The time budget expired before {{ unchecked_count }} seed{{ 's' if unchecked_count != 1 else '' }} could be checked.
|
|
|
|
{% for seed in unchecked_seeds %}
|
|
- `{{ seed }}`
|
|
{% endfor %}
|
|
{% if seeds_omitted %}
|
|
|
|
The full list of unchecked seeds is in the workflow logs.
|
|
{% endif %}
|
|
{% endif %}
|
|
{% if seed_rerun_command %}
|
|
|
|
To check all seeds that failed or were left unchecked, check out Ruff commit `{{ fuzzer_revision }}` and run from the repository root:
|
|
|
|
```shell
|
|
{{ seed_rerun_command }}
|
|
```
|
|
{% endif %}
|