Files
robobun a5ca1cd2ac build: retry dep downloads for ~3 minutes and report the underlying failure (#37915)
### Problem
- Build lanes die in a dep fetch: `error: Failed to download after 5
attempts:
https://github.com/oven-sh/lol-html/archive/725ce499....tar.gz`, `cause:
fetch failed`. Build 93382 lost linux x64-asan on lolhtml. Build 93392
(a one-file PR) lost four lanes: x64-musl and x64-android on cares,
mimalloc and the WebKit tarball, freebsd aarch64 on lolhtml, windows
aarch64 on cares, libuv, mimalloc and WebKit.
- Those are the downloads that miss the image's prefetch cache
(everything else in the same logs says `using prefetch cache`): the deps
whose pins moved after the images were baked in #34782 (lolhtml #36733,
mimalloc #36431, c-ares #34007, libuv #36839, WebKit several times a
week), plus WebKit on every lane other than linux arm64, because
`prefetch-deps.ts` only enumerates the bake host's own target (handed
off separately). Each build makes on the order of a hundred live
github.com downloads.
- `downloadWithRetry` (`scripts/build/download.ts:156`) made 5 attempts
with 2+4+8+16s of backoff, about 30s in total. The logs show agent-wide
outages longer than that: on the x64-musl lane three downloads that
started together failed on all five attempts over roughly two minutes,
after lolhtml had succeeded on the same agent seconds earlier; on the
freebsd lane lolhtml failed five times in a row while cares and mimalloc
went through and WebKit succeeded on its second try.
- `BuildError.format()` (`scripts/build/error.ts:33`) prints one level
of cause. node's fetch throws `TypeError: fetch failed` and keeps the
real error (DNS, connect timeout, reset) in `.cause`, so every one of
these logs says only `cause: fetch failed`, and the retry lines say
nothing about what failed.

### Fix
- `downloadRetry`: 10 attempts, backoff doubling from 2s and capped at
30s, 180s of backoff in total instead of 30s. Exported as a
`RetryPolicy` (optional last parameter of `downloadWithRetry`) so the
test can run the production attempt count with the backoff zeroed.
- 408 and 429 are retried along with 5xx and network errors. Other 4xx
still fail on the first attempt and are still thrown unwrapped, which
`prefetch-deps.ts` relies on to tell a 404 (variant not published) from
a transient failure.
- Each retry line names the failure it is retrying, e.g. `retry 2/10 in
2000ms (fetch failed: other side closed)`, and `format()` prints the
whole cause chain through the new `describeError()`, so the next one of
these says what the network did.
- Why here: the prefetch cache goes stale by design as soon as a pin
moves, and WebKit moves faster than images get rebaked (the images were
rebaked on July 21 and this came back within two weeks; #30095 was an
earlier rebake for the same symptom), so the live path is permanently on
every build's critical path and has to outlast the outages CI actually
sees. The wider window only costs time while github.com is actually
down, when the lane would otherwise have failed; build-bun's step
timeout is 60 minutes.
- Verified with `test/internal/build-download-retry.test.ts` (`bun bd
test`, 7 pass). It drives the real `downloadWithRetry` against a local
server that drops connections or returns scripted statuses, checks the
retry lines and `format()` output, and pins the schedule's total backoff
at two minutes or more. Against the previous `download.ts`/`error.ts`
the file fails at import (`downloadRetry` and `describeError` did not
exist); each behavioral case is something the old loop did not do (10
attempts, 429 retried, reason on the retry line, cause chain in
`format()`).
- Also ran the loop under node, the runtime CI builds with, against a
dropping server: retry lines read `(fetch failed: other side closed)`
and the final report prints `cause: fetch failed: other side closed`.
`bunx tsc -p scripts/build/tsconfig.json` reports nothing for these
files.

### Background
- Dep fetching: configure emits one ninja `dep_fetch` edge per vendored
dep, which runs `scripts/build/fetch-cli.ts`; that calls
`downloadWithRetry` on
`https://github.com/<repo>/archive/<commit>.tar.gz`, and `fetchPrebuilt`
uses the same function for release tarballs such as WebKit. Both URL
kinds start with a 302 from github.com (to codeload.github.com and
objects.githubusercontent.com respectively), which is why one github.com
problem takes out both kinds at once.
- Prefetch cache: CI images run `scripts/prefetch-deps.ts` at bake time,
storing each tarball under `/opt/bun-prefetch/by-url/<sha256(url)>`.
`downloadWithRetry` looks there before touching the network, so a dep is
served from the image only while its pinned URL is the one that was
current at bake time; anything bumped later downloads live until the
next `[publish images]` rebake.
- `BuildError` is the build system's error type; `fetch-cli.ts` and
`build.ts` print failures through its `format()`, which produces the
`error:` / `hint:` / `cause:` lines seen in the build log.
2026-08-12 13:49:54 -07:00

80 lines
2.3 KiB
TypeScript

/**
* All build system errors go through this. Includes context for helpful messages.
*/
export class BuildError extends Error {
readonly hint: string | undefined;
readonly file: string | undefined;
constructor(
message: string,
context?: {
hint?: string;
file?: string;
cause?: unknown;
},
) {
super(message, context?.cause !== undefined ? { cause: context.cause } : undefined);
this.name = "BuildError";
this.hint = context?.hint;
this.file = context?.file;
}
/**
* Format for display to the user.
*/
format(): string {
let out = `error: ${this.message}\n`;
if (this.file !== undefined) {
out += ` at: ${this.file}\n`;
}
if (this.hint !== undefined) {
out += ` hint: ${this.hint}\n`;
}
if (this.cause !== undefined) {
out += ` cause: ${describeError(this.cause)}\n`;
}
return out;
}
}
/**
* One line naming an error and everything it wraps, outermost first:
* `fetch failed: getaddrinfo EAI_AGAIN github.com`. The outer message alone
* is often content-free: node's fetch reports every network failure as
* "fetch failed" and puts the reason (DNS, connect timeout, reset) in
* `cause`, and a connect that failed on every resolved address is an
* AggregateError whose own message is empty.
*/
export function describeError(err: unknown): string {
if (!(err instanceof Error)) return String(err);
const parts: string[] = [];
const seen = new Set<Error>();
let e: unknown = err;
while (e instanceof Error && !seen.has(e)) {
seen.add(e);
if (e instanceof AggregateError && e.errors.length > 0) {
const inner = e.errors.map(describeError).join("; ");
parts.push(e.message ? `${e.message} (${inner})` : inner);
} else {
parts.push(e.message || e.name);
}
e = e.cause;
}
// A non-Error cause (`{ cause: "ECONNRESET" }`) still carries the reason.
if (e !== undefined && e !== null && !(e instanceof Error)) parts.push(String(e));
return parts.join(": ");
}
/**
* Assert a condition, throwing BuildError if false.
*/
export function assert(
condition: unknown,
message: string,
context?: { hint?: string; file?: string },
): asserts condition {
if (!condition) {
throw new BuildError(message, context);
}
}