[ty] Improve ecosystem summary reports (#27695)

.agents/skills/summarise-ecosystem-results/SKILL.md

@@ -8,7 +8,7 @@ description: Use when a user says "summarise ecosystem results", "summarize this
 ## Priorities
 
 1. Reproduce every retained behavior with the exact environment used by the Actions run.
-2. Lead the report with new or changed project failures, then cover meaningful flaky behavior, diagnostic changes, and clear minimized examples.
+2. Lead the report with new or meaningfully changed project failures, including intermittent severe failures, then cover stable diagnostic changes and clear minimized examples.
 3. Keep execution, audit, and traceability bookkeeping out of the report.
 
 ## Deliverable
@@ -19,13 +19,20 @@ Use the template's structure and omissions as the report contract. Remove all pl
 
 If summarising an ecosystem report is the only thing you're asked to do in a Codex App thread, you should rename that thread to "PR <number> ecosystem summary".
 
+## Reporting Policy
+
+- Focus on new or meaningfully changed behavior relative to the merge base. Evaluate individual diagnostics and failure outcomes, not a project's overall flaky or persistent status.
+- Omit flaky diagnostic changes, unchanged failures, and frequency fluctuations that leave the observed outcomes unchanged.
+- Report new or changed panics, crashes, overflows, and timeouts, including merge-base and PR run frequencies when intermittent behavior is involved.
+
 ## Workflow
 
 1. **Freeze the evidence.** Preserve any report URL or ecosystem-results comment explicitly supplied by the user before identifying the PR. For PR-only input, find its ecosystem-results comment and linked detailed report. Capture the matching Actions run and attempt as described in [references/evidence-acquisition.md](references/evidence-acquisition.md); never replace a supplied report with the PR's current report. Ignore later comment edits, PR updates, and workflow runs. Use the frozen detailed report as the authoritative change list and the comment for orientation when available.
-2. **Identify changed outcomes.** Check the detailed report for new, fixed, or changed project failures, panics, timeouts, abnormal exits, and meaningful flaky diagnostic or exit-status changes. Omit unchanged persistent failures. If neither project outcomes nor diagnostics changed, say explicitly that the run had no ecosystem impact and omit project-specific sections and reproduction details.
-3. **Reproduce from scratch.** Ignore retained memories and previous local artifacts. Load the `minimizing-ty-ecosystem-changes` skill, use its metadata helper and exact-run workflow, and reproduce each report entry before explaining or minimizing it. Reproduce flaky behavior with the reported run counts.
+2. **Identify changed outcomes.** Check the detailed report for new, fixed, or changed project failures, panics, overflows, timeouts, abnormal exits, and diagnostic changes, applying the reporting policy to each entry and outcome.
+3. **Reproduce from scratch.** Ignore retained memories and previous local artifacts. Load the `minimizing-ty-ecosystem-changes` skill, use its metadata helper and exact-run workflow, and reproduce each report entry before explaining or minimizing it. Reproduce intermittent severe failure changes with the reported run counts.
 4. **Minimize with provenance.** Include a standalone reproducer only when a verified reduction chain connects it to a cited ecosystem entry and preserves the same underlying trigger. If either cannot be verified, retain the original source excerpt and identify it as unminimized.
 5. **Group by cause.** Group entries only when the same base-to-PR behavior, underlying trigger, explanation, and reproducer account for every entry. Identical diagnostic text or displayed `@Todo` types do not establish equivalence.
-6. **Write and verify.** Fill the report template, record each affected project's strict or non-strict analysis mode, and include both strict-analysis flags in the comparison method when applicable. Check every link, diagnostic, reproducer's source provenance, and causal fingerprint when required, then run `uv run --only-group dev --locked prek run --files PR_<number>_ECOSYSTEM_SUMMARY.md`. Present the Markdown file as the finished product.
+6. **Find existing ty issues.** When a diagnostic change exposes a pre-existing shortcoming in ty, search the `astral-sh/ty` issue tracker for the precise underlying behavior. Link matching issues directly from the relevant report section; do not mistake incorrect or incomplete third-party annotations for ty shortcomings.
+7. **Write and verify.** Fill the report template, record each affected project's strict or non-strict analysis mode, and include both strict-analysis flags in the comparison method when applicable. Check every change number, link, diagnostic, reproducer's source provenance, and causal fingerprint when required, then run `uv run --only-group dev --locked prek run --files PR_<number>_ECOSYSTEM_SUMMARY.md`. Present the Markdown file as the finished product.
 
 When parallelizing reproduction or minimization, read [references/subagent-handoff.md](references/subagent-handoff.md). Otherwise, keep batches small and work through them sequentially.

.agents/skills/summarise-ecosystem-results/assets/report-template.md

@@ -1,22 +1,38 @@
-<!-- Replace every placeholder and remove all HTML comments before presenting the report. Keep each prose paragraph and list item on one source line. Omit project-specific reproduction details when there are no affected projects. Do not add change-count tables, bot-update timestamps, reproduction-completeness bookkeeping, import-audit details, exhaustive traceability appendices, raw URLs, or artifact hashes. -->
+<!-- Replace every placeholder and remove all HTML comments before presenting the report. Keep each prose paragraph and list item on one source line. Number retained change subsections consecutively within each section, restarting at 1 for each section. Do not mention the absence of new panics, overflows, or timeouts. Do not add change-count tables, bot-update timestamps, reproduction-completeness bookkeeping, import-audit details, exhaustive traceability appendices, raw URLs, or artifact hashes. -->
 
 # [PR #<number>](https://github.com/astral-sh/ruff/pull/<number>) ecosystem summary
 
-<Summarize the distinct failure, flakiness, and diagnostic behavior changes and their significance. Lead with the analysis readers need; do not describe how the report was generated.>
+<Summarize meaningful changes to project failures and diagnostic behavior, including changes involving intermittent severe failures, along with their significance. Lead with the analysis readers need; do not describe how the report was generated.>
 
-<!-- Omit this entire section if no project failures or meaningful flaky outcomes changed. -->
+<!-- Omit this entire section if no stable project failures changed. Repeat its numbered subsection for each distinct failure. -->
 
-## <New, fixed, or changed project failure or flaky outcome>
+## Project failures
+
+### 1. <New, fixed, or changed project failure>
 
 **Affected projects:**
 
 - [<project>](<project-url>): merge base: `<base outcome>`; PR: `<PR outcome>`.
 
-<Explain the crash, panic, timeout, abnormal exit, or flaky change, including relevant stderr and observed base/PR run frequencies where applicable.>
+<Explain the crash, panic, overflow, timeout, or abnormal exit, including relevant stderr where applicable.>
 
-<!-- Omit this entire section if no diagnostic behavior changed. -->
+<!-- Omit this entire section if no severe failure involving intermittent outcomes changed. Repeat its numbered subsection for each distinct change. -->
 
-## <Distinct behavior change>
+## Intermittent severe failures
+
+### 1. <New or changed intermittent panic, crash, overflow, or timeout>
+
+**Affected projects:**
+
+- [<project>](<project-url>): merge base: `<base outcome and count/runs, or not present>`; PR: `<PR outcome and count/runs, or not present>`.
+
+<Explain the change in failure behavior and relevant stderr. Do not include unchanged failures or frequency-only fluctuations.>
+
+<!-- Omit this entire section if no stable diagnostic behavior changed. Repeat its numbered subsection for each distinct behavior change. -->
+
+## Affected projects
+
+### 1. <Distinct behavior change>
 
 **Report entries:**
 
@@ -26,6 +42,10 @@
 
 <Explain the exact behavior on the merge base and PR. Group additional entries here only when the same explanation and minimized reproducer account for all of them.>
 
+<!-- If this diagnostic change exposes an existing ty shortcoming, search astral-sh/ty for issues covering that exact shortcoming. Include the following paragraph only when a matching issue exists. -->
+
+**Existing ty issues:** [ty#<issue-number>](https://github.com/astral-sh/ty/issues/<issue-number>)
+
 <!--
 A minimal reproducer should annotate every line with a new, changed, or removed diagnostic using comments immediately above that line. Include the full error messages and error codes from both revisions, including duplicates.