mirror of
https://github.com/astral-sh/ruff.git
synced 2026-10-28 10:45:37 +08:00
Carl Meyer
b8c5e73abe
## Summary We've been seeing +/-5% false positives on many PRs all day today on one specific benchmark, `ty_micro[typevar_mapping_accumulation]`, across unrelated PRs that wouldn't plausibly impact that benchmark. The start of these false positives lines up very well with the rollout of CodSpeed v5 and its switch to "cycle estimation" rather than simple instruction counting by default. (Across 43 jobs before the CodSpeed v5 upgrade, we saw zero significant changes in this benchmark.) By examining all the various PRs on which this false positives has showed up, I ruled out other potential causes: - The false positive occurs on PRs where the benchmark ran v5 vs v5, so it's not an issue of mismatched CodSpeed version. - The false positive occurs on PRs where GitHub runner spec was identical from baseline to PR, so it's not a GitHub runner mismatch issue. This PR sets an option to disable the new cycle estimation mode, while staying on CodSpeed v5. This is easily reversible and seems like the most obvious thing to try to eliminate these false positives. - Disable CodSpeed v5's default cycle estimation for instrumented ty benchmarks while keeping the v5 action. - Leave Ruff benchmarks and wall-time benchmark jobs unchanged. The first comparison across this configuration change can be marked incomparable because its base and head use different runner settings. Once the updated workflow runs on `main`, subsequent runs will use the new matching baseline. ## Test plan - Existing instrumented ty simulation and memory benchmark matrix exercises the updated action configuration in CI. - Workflow validation, formatting, security, and typo hooks pass for the changed workflow.