mirror of
https://github.com/oven-sh/bun.git
synced 2026-10-28 21:46:59 +08:00
### Problem - Build lanes die in a dep fetch: `error: Failed to download after 5 attempts: https://github.com/oven-sh/lol-html/archive/725ce499....tar.gz`, `cause: fetch failed`. Build 93382 lost linux x64-asan on lolhtml. Build 93392 (a one-file PR) lost four lanes: x64-musl and x64-android on cares, mimalloc and the WebKit tarball, freebsd aarch64 on lolhtml, windows aarch64 on cares, libuv, mimalloc and WebKit. - Those are the downloads that miss the image's prefetch cache (everything else in the same logs says `using prefetch cache`): the deps whose pins moved after the images were baked in #34782 (lolhtml #36733, mimalloc #36431, c-ares #34007, libuv #36839, WebKit several times a week), plus WebKit on every lane other than linux arm64, because `prefetch-deps.ts` only enumerates the bake host's own target (handed off separately). Each build makes on the order of a hundred live github.com downloads. - `downloadWithRetry` (`scripts/build/download.ts:156`) made 5 attempts with 2+4+8+16s of backoff, about 30s in total. The logs show agent-wide outages longer than that: on the x64-musl lane three downloads that started together failed on all five attempts over roughly two minutes, after lolhtml had succeeded on the same agent seconds earlier; on the freebsd lane lolhtml failed five times in a row while cares and mimalloc went through and WebKit succeeded on its second try. - `BuildError.format()` (`scripts/build/error.ts:33`) prints one level of cause. node's fetch throws `TypeError: fetch failed` and keeps the real error (DNS, connect timeout, reset) in `.cause`, so every one of these logs says only `cause: fetch failed`, and the retry lines say nothing about what failed. ### Fix - `downloadRetry`: 10 attempts, backoff doubling from 2s and capped at 30s, 180s of backoff in total instead of 30s. Exported as a `RetryPolicy` (optional last parameter of `downloadWithRetry`) so the test can run the production attempt count with the backoff zeroed. - 408 and 429 are retried along with 5xx and network errors. Other 4xx still fail on the first attempt and are still thrown unwrapped, which `prefetch-deps.ts` relies on to tell a 404 (variant not published) from a transient failure. - Each retry line names the failure it is retrying, e.g. `retry 2/10 in 2000ms (fetch failed: other side closed)`, and `format()` prints the whole cause chain through the new `describeError()`, so the next one of these says what the network did. - Why here: the prefetch cache goes stale by design as soon as a pin moves, and WebKit moves faster than images get rebaked (the images were rebaked on July 21 and this came back within two weeks; #30095 was an earlier rebake for the same symptom), so the live path is permanently on every build's critical path and has to outlast the outages CI actually sees. The wider window only costs time while github.com is actually down, when the lane would otherwise have failed; build-bun's step timeout is 60 minutes. - Verified with `test/internal/build-download-retry.test.ts` (`bun bd test`, 7 pass). It drives the real `downloadWithRetry` against a local server that drops connections or returns scripted statuses, checks the retry lines and `format()` output, and pins the schedule's total backoff at two minutes or more. Against the previous `download.ts`/`error.ts` the file fails at import (`downloadRetry` and `describeError` did not exist); each behavioral case is something the old loop did not do (10 attempts, 429 retried, reason on the retry line, cause chain in `format()`). - Also ran the loop under node, the runtime CI builds with, against a dropping server: retry lines read `(fetch failed: other side closed)` and the final report prints `cause: fetch failed: other side closed`. `bunx tsc -p scripts/build/tsconfig.json` reports nothing for these files. ### Background - Dep fetching: configure emits one ninja `dep_fetch` edge per vendored dep, which runs `scripts/build/fetch-cli.ts`; that calls `downloadWithRetry` on `https://github.com/<repo>/archive/<commit>.tar.gz`, and `fetchPrebuilt` uses the same function for release tarballs such as WebKit. Both URL kinds start with a 302 from github.com (to codeload.github.com and objects.githubusercontent.com respectively), which is why one github.com problem takes out both kinds at once. - Prefetch cache: CI images run `scripts/prefetch-deps.ts` at bake time, storing each tarball under `/opt/bun-prefetch/by-url/<sha256(url)>`. `downloadWithRetry` looks there before touching the network, so a dep is served from the image only while its pinned URL is the one that was current at bake time; anything bumped later downloads live until the next `[publish images]` rebake. - `BuildError` is the build system's error type; `fetch-cli.ts` and `build.ts` print failures through its `format()`, which produces the `error:` / `hint:` / `cause:` lines seen in the build log.
80 lines
2.3 KiB
TypeScript
80 lines
2.3 KiB
TypeScript
/**
|
|
* All build system errors go through this. Includes context for helpful messages.
|
|
*/
|
|
export class BuildError extends Error {
|
|
readonly hint: string | undefined;
|
|
readonly file: string | undefined;
|
|
|
|
constructor(
|
|
message: string,
|
|
context?: {
|
|
hint?: string;
|
|
file?: string;
|
|
cause?: unknown;
|
|
},
|
|
) {
|
|
super(message, context?.cause !== undefined ? { cause: context.cause } : undefined);
|
|
this.name = "BuildError";
|
|
this.hint = context?.hint;
|
|
this.file = context?.file;
|
|
}
|
|
|
|
/**
|
|
* Format for display to the user.
|
|
*/
|
|
format(): string {
|
|
let out = `error: ${this.message}\n`;
|
|
if (this.file !== undefined) {
|
|
out += ` at: ${this.file}\n`;
|
|
}
|
|
if (this.hint !== undefined) {
|
|
out += ` hint: ${this.hint}\n`;
|
|
}
|
|
if (this.cause !== undefined) {
|
|
out += ` cause: ${describeError(this.cause)}\n`;
|
|
}
|
|
return out;
|
|
}
|
|
}
|
|
|
|
/**
|
|
* One line naming an error and everything it wraps, outermost first:
|
|
* `fetch failed: getaddrinfo EAI_AGAIN github.com`. The outer message alone
|
|
* is often content-free: node's fetch reports every network failure as
|
|
* "fetch failed" and puts the reason (DNS, connect timeout, reset) in
|
|
* `cause`, and a connect that failed on every resolved address is an
|
|
* AggregateError whose own message is empty.
|
|
*/
|
|
export function describeError(err: unknown): string {
|
|
if (!(err instanceof Error)) return String(err);
|
|
const parts: string[] = [];
|
|
const seen = new Set<Error>();
|
|
let e: unknown = err;
|
|
while (e instanceof Error && !seen.has(e)) {
|
|
seen.add(e);
|
|
if (e instanceof AggregateError && e.errors.length > 0) {
|
|
const inner = e.errors.map(describeError).join("; ");
|
|
parts.push(e.message ? `${e.message} (${inner})` : inner);
|
|
} else {
|
|
parts.push(e.message || e.name);
|
|
}
|
|
e = e.cause;
|
|
}
|
|
// A non-Error cause (`{ cause: "ECONNRESET" }`) still carries the reason.
|
|
if (e !== undefined && e !== null && !(e instanceof Error)) parts.push(String(e));
|
|
return parts.join(": ");
|
|
}
|
|
|
|
/**
|
|
* Assert a condition, throwing BuildError if false.
|
|
*/
|
|
export function assert(
|
|
condition: unknown,
|
|
message: string,
|
|
context?: { hint?: string; file?: string },
|
|
): asserts condition {
|
|
if (!condition) {
|
|
throw new BuildError(message, context);
|
|
}
|
|
}
|