mirror of
https://github.com/oven-sh/bun.git
synced 2026-10-25 21:40:30 +08:00
Dylan Conway
fc297d4658
## What CI machine images are content-addressed. An image's name is `<key>-<hash>`: the key is the operating system or distro and the architecture (`linux-x64-debian`, `windows-aarch64`), and the hash covers everything its bake runs, the release and the base image included. A CI build asks whether each image exists and bakes the ones that do not before anything else runs; every later build (the PR's next push, `main` after the merge) finds them by name. The `[build images]` / `[publish images]` commit tags, the `# Version:` numbers and the `-build-<n>` names are gone: to change what is on a CI machine, edit one file. ## One file `scripts/build/ci-images/spec.ts` is everything about CI's machines: 1. **The data**: the files copied onto machines, every version (`pins`), where a bake puts things (`locations`), the six Linux and two Windows images each with its exact base image, the package lists, and how Packer bakes Windows. 2. **The tools**: one function per thing a bake sets up on a machine, and the order they run in. A tool does not contain shell. It returns steps that say what should be true (`download`, `unpack`, `installExecutable`, `directory`, `systemUser`, `service`, `registryValue`, `scheduledTaskAtStartup`, …), and one description serves every system the tool exists on: `bun` is one function for Linux, macOS and Windows. 3. **The machinery**: the vocabulary the tools are written in, its renderers (sh as root for Linux, sh with `sudo` where a system path is written for macOS, PowerShell for Windows), the Packer template, and the generator. No shell is kept in strings: the generated scripts have no helper functions, a step renders the lines themselves (a download is a `curl` line on every system), and in PowerShell every statement that ran a native program is followed by a check of its exit code. `bun run ci:images [key…]` writes `build/ci-images/<key>/` (`image.json`, `bootstrap.sh` or `bootstrap.ps1`, the files the script puts on the machine, and for Windows the Packer template) and prints each name, which is the sha256 of that directory. The generated script is what is read, linted and debugged: straight-line, with every value in it, looping only over what is known on the machine. The output depends only on committed files. Besides `spec.ts`, only `scripts/agent.ts` (copied whole onto every baked machine) and the vendored `scripts/build/xmac.mjs` (the build image) can rename an image; `spec.ts` lists both. `.buildkite/ci.ts` generates every image, asks the cloud for each name's state (`scripts/ci-image.ts`: AWS by name among the account's own images, Azure by gallery version), and for a missing or failed one uploads its directory and emits a bake step; for one another build is already baking, only a wait step. A fork's build cannot bake (a bake runs the branch's code on a machine that becomes everyone's image), so it does not ask the cloud anything and reads none of the cloud's credentials: it uses the images that exist, and fails with an explanation if its diff touches a bake input. A Linux bake runs `bootstrap.sh <commit> <image name>` as root on a machine started from the exact base image the spec names; a Windows bake runs Packer, pinned and checksummed, as the step's single command so that a cancel reaches it. The build system's LLVM, Node.js, xwin, Windows SDK, macOS SDK, Android API level and FreeBSD versions import from `pins`, and its sysroot and download-cache lookups from `locations`. On a Buildkite agent the build compares `bun`, `cmake`, `node`, `clang` and `ld.lld` with the pins; a bake installs and does not judge versions. `test/internal/source-lints/ci-image-pins.test.ts` keeps `rust-toolchain.toml` and the GitHub workflows (Bun, LLVM, the Rust nightly) in step with the spec, since other programs read those. Every bake writes `bun-image.json` on the machine and publishes it as the bake job's artifact. It is for keying caches of build outputs, so it holds what the name cannot: what arrived on the day of the bake. `Tool` requires an `identity`, so a tool that does not say how what it installs is known does not compile: `pinned` (the spec's value is written as it is), `observed` (a step rendered into the script, whose output is recorded: the glibc and musl cross sysroots as a digest of their trees, and on Windows the MSVC toolset and Windows SDK versions the Visual Studio installer chose), `packageDatabase` (the record lists apt's, apk's or Scoop's database with exact versions), `configuration`, or `notRecorded` with the reason (the prefetch). The file is deterministic: no times, hostnames or instance ids. ## What changes on the machines - The Rust toolchain CI builds with is installed by the bake, from the pinned channel with its components and targets. - Bun is 1.4.2 on the images and in the GitHub workflows (they were 1.3.13 and 1.3.14). - Pinned where they floated: the base images (exact AMI names, exact Marketplace versions), Bun and buildkite-agent on Windows, rustup-init, Packer, nssm. Floating, by name, as before: distro, Scoop and Homebrew packages, Docker, Tailscale, Chrome, Visual Studio, the patch release of LLVM from apt. - Bun's build of ninja (oven-sh/ninja) is baked into every machine, in a directory of its own that is not on PATH; the `ninja` the build runs is still the machine's own. - The prefetch (dependency sources, test Docker images, `bun install` cache) is one tool for Linux and Windows: it clones the commit being built, fetches, and removes the clone. Its contents stay outside the hash, so a dependency bump does not rename an image. Raising `prefetchTriggerVersion` in the spec is how the caches are refreshed: the number is written into that tool's section of the generated scripts, which renames every baked image, and the next build bakes them all. - The Windows image no longer carries one set of SSH host keys for every machine, and password logins are really off. - Smart App Control is turned off by the bake alone; the test runner's copy, for images baked before the bake did it, is removed (on machines from these images it found the policy already off). - Not carried over, because CI never ran it: the FreeBSD, dnf/yum/zypper and `--gcc-13` paths, the `qemu-user` packages, Strawberry Perl (the old script skipped it; `perl` on Windows is Git's). ## macOS The macOS test machines are in the spec too, with the same pinned Node.js, Bun, curl-h3, LLVM (through Homebrew) and Rust toolchain. CI does not bake or name them: `scripts/darwin-ci` generates `build/ci-images/darwin-<arch>/bootstrap.sh` on the host and runs it in the Tart guest image it builds, or on a bare host. `scripts/bootstrap.sh`, `scripts/bootstrap.ps1` and `scripts/packer/` are removed. The generated macOS script passes shellcheck and every URL in it exists, but it has not run on a Mac: `darwin-ci bake --ref <this branch>` on one Tart host is the test, and it leaves the host's image alone unless the toolchain check passes. ## Verified - An earlier revision of this branch, with hand-written scripts per tool, baked all eight images and ran the full build and test pipeline on them. The generated Linux scripts have since baked all six Linux images on real machines. - Four reviews of the generated output against those hand-written scripts (Linux; Windows and macOS; the generator's correctness, with probes; conventions) and one security review against main. What they found is fixed: `DEBIAN_FRONTEND` exported once so installers that call apt inherit it, files written from PowerShell without a byte-order mark (node-gyp read its `installVersion` as 0), `unzip` replacing on a Mac that is set up again, PowerShell literals in expressions always quoted. The security review found nothing looser than main except that the pipeline read cloud credentials in fork builds, which it no longer does. - For the generated scripts: shellcheck on all eight sh scripts, PowerShell's parser on both Windows scripts and the SSH key script, `packer validate` on both templates with exactly the pinned Packer; the Debian 13 and Alpine 3.23 x64 scripts run to completion in containers up to the prefetch; a simulated Buildkite run with every image missing uploads eight directories and emits eight bakes. - `tsc` is clean for both script projects; the source lints and the generator test pass.
104 lines
2.4 KiB
Nix
104 lines
2.4 KiB
Nix
# Simple shell.nix for users without flakes enabled
|
|
# For reproducible builds with locked dependencies, use: nix develop
|
|
# This uses unpinned <nixpkgs> for simplicity; flake.nix provides version pinning via flake.lock
|
|
{ pkgs ? import <nixpkgs> {} }:
|
|
|
|
pkgs.mkShell rec {
|
|
packages = with pkgs; [
|
|
# Core build tools (matching the CI images, scripts/build/ci-images/spec.ts)
|
|
cmake
|
|
ninja
|
|
llvmPackages_23.clang
|
|
llvmPackages_23.llvm
|
|
llvmPackages_23.lld
|
|
nodejs_24
|
|
bun
|
|
rustc
|
|
cargo
|
|
go
|
|
python3
|
|
ccache
|
|
pkg-config
|
|
gnumake
|
|
libtool
|
|
ruby
|
|
perl
|
|
|
|
# Libraries
|
|
openssl
|
|
zlib
|
|
libxml2
|
|
|
|
# Development tools
|
|
git
|
|
curl
|
|
wget
|
|
unzip
|
|
xz
|
|
|
|
# Linux-specific: gdb and Chromium deps for testing
|
|
] ++ pkgs.lib.optionals pkgs.stdenv.isLinux [
|
|
gdb
|
|
# Chromium dependencies for Puppeteer tests
|
|
xorg.libX11
|
|
xorg.libxcb
|
|
xorg.libXcomposite
|
|
xorg.libXcursor
|
|
xorg.libXdamage
|
|
xorg.libXext
|
|
xorg.libXfixes
|
|
xorg.libXi
|
|
xorg.libXrandr
|
|
xorg.libXrender
|
|
xorg.libXScrnSaver
|
|
xorg.libXtst
|
|
libxkbcommon
|
|
mesa
|
|
nspr
|
|
nss
|
|
cups
|
|
dbus
|
|
expat
|
|
fontconfig
|
|
freetype
|
|
glib
|
|
gtk3
|
|
pango
|
|
cairo
|
|
alsa-lib
|
|
at-spi2-atk
|
|
at-spi2-core
|
|
libgbm
|
|
liberation_ttf
|
|
atk
|
|
libdrm
|
|
xorg.libxshmfence
|
|
gdk-pixbuf
|
|
];
|
|
|
|
shellHook = ''
|
|
export CC="${pkgs.lib.getExe pkgs.llvmPackages_23.clang}"
|
|
export CXX="${pkgs.lib.getExe' pkgs.llvmPackages_23.clang "clang++"}"
|
|
export AR="${pkgs.llvmPackages_23.llvm}/bin/llvm-ar"
|
|
export RANLIB="${pkgs.llvmPackages_23.llvm}/bin/llvm-ranlib"
|
|
export CMAKE_C_COMPILER="$CC"
|
|
export CMAKE_CXX_COMPILER="$CXX"
|
|
export CMAKE_AR="$AR"
|
|
export CMAKE_RANLIB="$RANLIB"
|
|
export CMAKE_SYSTEM_PROCESSOR=$(uname -m)
|
|
export TMPDIR=''${TMPDIR:-/tmp}
|
|
'' + pkgs.lib.optionalString pkgs.stdenv.isLinux ''
|
|
export LD="${pkgs.lib.getExe' pkgs.llvmPackages_23.lld "ld.lld"}"
|
|
export NIX_CFLAGS_LINK="''${NIX_CFLAGS_LINK:+$NIX_CFLAGS_LINK }-fuse-ld=lld"
|
|
export LD_LIBRARY_PATH="${pkgs.lib.makeLibraryPath packages}''${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"
|
|
'' + ''
|
|
|
|
echo "====================================="
|
|
echo "Bun Development Environment (Nix)"
|
|
echo "====================================="
|
|
echo "To build: bun bd"
|
|
echo "To test: bun bd test <test-file>"
|
|
echo "====================================="
|
|
'';
|
|
}
|