mirror of
https://github.com/oven-sh/bun.git
synced 2026-10-25 21:40:30 +08:00
Dylan Conwayandautofix-ci[bot]
a2b69f7b06
## What
`bun_runtime` was a `staticlib`: rustc copied every crate and std into
one archive, `libbun_runtime.a` (1.2 GB in a debug tree, 220 MB in
release), and the link took that. It is now a library like every other
crate, and the link edge takes each crate's rlib
(`libbun_runtime-<hash>.rlib`, `libbun_core-<hash>.rlib`, … std's too)
beside the C/C++ objects, by names known at configure — the way it takes
the C/C++ objects. No crate is a final Rust artifact.
rustc decides a few things only when *it* links the final artifact. They
are decided here instead:
- **The allocator shim.** For bun it held one thing: the allocator
functions come from `#[global_allocator]` in
`src/runtime/bin_entry/mod.rs` and the allocation error handler from
std, which leaves `__rust_no_alloc_shim_is_unstable_v2`, an empty
function `alloc` calls before every allocation. The standard library
says what it is for (library/alloc/src/alloc.rs: "Make sure we don't
accidentally allow omitting the allocator shim in stable code until it
is actually stabilized"): a tripwire against linking rlibs any other way
than through rustc. It is defined next to the global allocator with
`#[rustc_std_internal_symbol]`, so rustc mangles it; checked on the
pinned nightly to be exactly the symbol rustc's shim defines. A
toolchain that renames it fails the link with an undefined symbol naming
the new one.
- **The panic runtime.** std depends on both `panic_abort` and
`panic_unwind`, and both define `__rust_start_panic`; rustc links the
one the panic strategy names and leaves the other out. `linkedRlibs()`
in `scripts/build/rust/units.ts` does the same (the duplicate-symbol
check found this on the first build). It also leaves out host code:
proc-macros and what they alone depend on.
- **Which std.** A debug build without ASan (Windows debug,
`--asan=off`) used the toolchain's prebuilt std, which is not part of
the unit graph; rustc bundled it into the archive. std is now compiled
from source in every build (release, ASan and Tier 3 builds already
did), so every rlib the link takes is the output of an edge. About 12 s
on a clean debug build here (`core` 6.3 s, `std` 2.1 s), nothing
afterwards; std in those builds gets the dev profile, as it already does
in Linux and macOS arm64 debug builds.
- **LTO.** rustc only runs LTO when it produces a final artifact, and it
produces none of bun's any more, so bun's link runs it. Every release
build but ASan's has one shape: `[profile.release] lto = "off"`, the
rlibs carry ThinLTO bitcode (`-C linker-plugin-lto`, added by
`rust.ts`), and lld optimises it.
- Cross-language LTO targets (Linux, macOS, Windows x64 as CI builds
it): unchanged. The Rust profile's LTO was already off there and lld
optimised the crates' bitcode together with the C++ and JSC; the archive
was only a container.
- Where the C/C++ has no LTO (Windows arm64, FreeBSD, Android,
`--lto=off`), Rust got rustc's fat LTO: the profile default from before
cross-language LTO existed (#30412), left in place when the LTO targets
moved to ThinLTO (e550f2c36a). It is now ThinLTO over the crates, run
by the link, like everywhere else. Linux x64 release `--lto=off`, thin
against fat from the same sources: link edge 77 s against 185 s;
stripped `bun` 197 KB smaller; bundling typescript.js +1.0 %
instructions and equal wall clock over 25 interleaved runs each (median
+0.5 %, min -1.8 %); host-call instruction counts between -3.0 % and
+2.0 %; startup equal. A relink of such a build redoes that LTO even
when only C++ changed; the prod-style build already redid its LTO on
every relink.
- FreeBSD was the one such target whose link had no identical-code
folding, though its objects already carry the table safe ICF reads. The
crates are one ThinLTO module each, so copies of a function several
crates instantiate are merged only if the linker folds them: its release
link now folds like the others (`--icf=safe`). FreeBSD x64 `bun` in CI's
size table: main 88.36 MiB, this branch **84.86 MiB** (aarch64: 89.75 to
85.89); cross-built here it is 89.08 MiB without the folding, and the
link edge takes 78 s against 185 s.
- A release built on a Windows host is now the same cross-language
ThinLTO link CI's cross-compiled one is: `bun.exe` 87,691,264 bytes
there, against 87,695,360 from CI for this branch; it runs a script
using `Bun.password`, `Bun.semver`, `Bun.Glob` and `HTMLRewriter`, and
the shim is still 8,192 bytes.
- Gone with the fat case: both `CARGO_PROFILE_RELEASE_LTO` overrides in
`rust.ts` (the profile says it), and the `release-dev` cargo profile,
which nothing could select and existed to avoid fat LTO's link time.
- **Every lld link states its LTO level and its ICF mode.** An ELF link
with C/C++ LTO carried `-O2` on the link line, which reaches lld only
through the clang driver, only on ELF and only beside `-flto` (as
`-plugin-opt=O2`). A link whose LTO covers the Rust crates alone passes
no `-flto`, so FreeBSD, Android and `--lto=off` ran at whatever lld
defaults to, as did lld-link and the macOS x64 cross link. Each lld link
that optimises bitcode now says `--lto-O<n> --lto-CGO<n>`
(`/opt:lldlto=<n> /opt:lldltocgo=<n>` on Windows) from one function: 3
on the macOS arm64 cross link, as before, and 2 elsewhere, which is what
each already ran at (lld ELF: `getInteger(OPT_lto_O, 2)`, CGO derived
from it; lld-link: `ltoo = 2`). ICF likewise comes from one function,
`safe` in release without ASan or valgrind and `none` otherwise, and
every lld link says it (`--icf=none`, `/OPT:NOICF` where it is off). A
native macOS link goes through Apple's ld, which takes neither option.
Relinked with the explicit flags, Linux x64 release (LTO), Linux x64
release `--lto=off` and FreeBSD x64 release are each the same size to
the byte, stripped binary and `.text`.
- **Code-generator settings rustc keeps outside the bitcode.** Where the
linker generates the Rust code for the first time (the targets without
C/C++ LTO), what rustc set on its own code generator has to be said to
the linker. Of the 19 such settings one differs for these targets: rustc
compiles Android's thread-locals as emulated TLS (bun targets API 28;
bionic has ELF TLS from 29), so that link passes `-mllvm -emulated-tls`.
Without it the Rust thread-locals become a PT_TLS segment and, on
x86_64, an import of `__tls_get_addr`. The ELF binary expectations now
record whether a PT_TLS segment is expected (absent on Android), checked
on every link.
- **The ELF link exports a list, not everything minus a script.** The
Linux, Android and FreeBSD links said `-rdynamic` and left the hiding to
`linker.lds`'s `local: *`. lld applies a version script before LTO
generates code, so a symbol first made there stayed exported: on
Android, the `__emutls_v.` / `__emutls_t.` pair of every thread-local
that ThinLTO keeps external (the exported-symbols check rejects them).
`--dynamic-list=src/symbols.dyn` beside it did nothing on ELF: its 684
names carry Mach-O's leading underscore and none matches. The links now
take `--export-dynamic-symbol-list` with the version script's `global:`
block, which configure writes to `<buildDir>/exports.list` with the
parser the binary check already uses; the script still assigns the
version node. `symbols.dyn` and the docs that said to update it are
gone. Linux x64 release and debug: the dynamic symbol table is identical
before and after (692 entries) and the release binary is the same size
to the byte. Reproduced with two C files and with two Rust crates linked
the way bun links Android. (`-Zdefault-visibility=hidden` also stops the
leak, and is not used: rustc does not share hidden generic
instantiations on a target with dynamic linking, so it turns
`-Zshare-generics` off, +1.1 MiB.)
- **macOS debug info.** Mach-O leaves DWARF in the input objects and
dsymutil follows the debug map to them; the linker's LTO object is a
temporary unless `-object_path_lto` says where to keep it. That flag now
applies whenever the link holds Rust bitcode, not only with C/C++ LTO on
(113 rustc compile units in the dSYM of a `--lto=off` build, 1 without).
The root crate no longer waits for every other crate's object code, only
their `.rmeta`s, like any library.
## The same code from a library root
As a `staticlib` the root crate got two things from rustc that nothing
in the source asked for; as a library it gets what the source says. Both
are fixed in the source, and the build system does nothing special for
the root.
- **`to_js_host_call` is `#[inline]`**, like every other adapter in
`src/jsc/host_fn.rs`. Without the attribute rustc instantiates a generic
function as a globally shared symbol (`instantiation_mode`,
`cross_crate_inlinable`), and a library built with `-Zshare-generics`
exports every instance. An exported instance cannot be made private at
compile time, so LLVM no longer folded it into the `extern "C"` wrapper
that is its only caller, and the closure's captures went through memory
across a real call. Out-of-line instances in the Linux x64 release
binary: 123 without the attribute, 4 with it (main: 8).
- **`bun_runtime` has no public API.** Nothing depends on it, but its
root modules were `pub`. For a library rustc takes everything reachable
from public items for API a dependent may instantiate (`rustc_passes`
reachable.rs) and exports the generic instances made from it. The root
modules are `pub(crate)`. What follows from that:
- `unreachable_pub`, which the workspace denies, now applies here: every
`pub` item in the crate becomes `pub(crate)` (rustc's own suggestions,
including inside macro definitions and under `cfg(windows)` /
`cfg(target_os = "macos")`), and re-exports become `pub(crate) use`.
This is most of the diff and it is mechanical.
- Re-exports nothing uses are deleted. Two aliases with a single
Windows-only user are removed and that user names the real path; one
re-export is `cfg(not(windows))`.
- `dead_code` now sees what `pub` hid, handled the way
`test/internal/source-lints/dead-code-escapes.test.ts` prescribes.
Unused on some platforms only (`Memfd`, `LinuxMemFdAllocator`, the
Windows IPC pieces, the POSIX half of `copy_file`, …):
`#[cfg_attr(<where unused>, allow(dead_code))]`, pinned in the limits
file. **Deleted, dead on every target:** `DNSRequestOwner::Prefetch`,
bake's `Mode::ProductionDynamic`, `SSRKind::Regular`, the
`ErrorKind::Js*` kinds the dev server never writes,
`BuiltinInput::ArrayBuf`'s `i` and `rm`'s `removed_count` (both only
ever 0), the payloads of the shell `Readable::Fd` / `Readable::Buffer`
and of `AnyRoute::FrameworkRouter` (stored, never read; the router count
is checked just above where that index was made). Held only for
ownership: the underscore name (`_arena`, `_bv2`, `_heap`,
`_ast_alloc_state`, `_allocations`, `_args`). Kept with the reason,
because the lint cannot see the use: `File.source` (read through the
`multi_array_columns!` accessor), `MessageId` (bake-codegen generates
the client's enum from it), the `Strong` in `Style::JavascriptDefined`
(a GC root), and `NapiStatus`, which C++ returns by value, so it
declares every `napi_status` value.
- The codegen scripts that scan the Rust source accept a visibility
qualifier (host exports, bake enums, class lookup), and the class and
sink generators emit `pub(crate) use`.
- clippy's `single_component_path_imports` does not look at modules of a
public API, so it now saw 15 lines that were already there: thirteen
`use <crate>;` are deleted, and two re-exports of a `define_scoped_log!`
macro are gone because neither was needed (`debug_log_dr` is defined in
the module that uses it; `source_map_store.rs` binds its own `map_log`).
- mordant, likewise, had never looked at this crate's items. One finding
was in code this PR edits and is fixed: the `Mode` to output-format
`match` existed twice and is `Mode::output_format`. Nine are recorded in
`mordant-baseline.toml`; all are true findings, each a type-level
refactor in a subsystem this PR does not otherwise touch:
`always_unwrapped_option` for `NewAsyncCpTask.shelltask` (node_fs.rs),
`PackageJSONFields.object` (init_command.rs), `NetworkSink.global_this`
(streams.rs) and `PerThread.all_server_files` (bake/production.rs);
`field_valid_only_when` for `gzip_id_bytes_read` (NativeZlib.rs),
`shared_fd` (udp_socket.rs) and `histogram` / `last_fire_ns`
(timer/mod.rs); `parallel_bools` for `Flags.nonblock` / `pollable`
(shell/IOWriter.rs).
What is left differs from main in shape only: on Linux x64, 1,824 of the
1,874 C-named entry points have main's shape (1,133 with a library root
alone). The rest are trait-impl methods (the `Blob` / `Request` /
`Response` extension traits): rustc "conservatively mark[s] all trait
impl items as reachable" in a library, which no annotation changes. Same
call count either way.
## Also removed
The four split build modes (`cpp-only`, `rust-only`, `link-only`,
`rust-and-link`). They existed to hand `libbun-*.a` and
`libbun_runtime.a` between CI agents; the pipeline only runs `ci-build`
(`archive-link`), which stays. Gone with them: their profiles, the three
extra emitters in `bun.ts`, the artifact upload / download /
step-polling code in `ci.ts`, the mode branches in `build.ts`,
`config.ts` and `configure.ts`, `computeDepLibs`, and the unused
`rustLinkFlags`. `tools/hawk`'s throwaway-root patch no longer needs to
add an `rlib` crate type. `orderFileEligible` no longer looks at the
mode, since every build links; main's `codegen` mode is unaffected
(`buildsRust` is `cfg.mode !== "codegen"`). Comments that described the
split modes say what is true now.
## Verification
Native builds on three machines, each configuration in its own build
directory. Every one links, reports 0 duplicate strong symbols, passes
the exported- and imported-symbol checks (and, in release, static
initializers and hardening), runs a script using `fetch`, `Bun.password`
(rust-argon2) and `HTMLRewriter` (lol-html), and passes
`test/js/bun/util/escapeHTML.test.js`:
| | Linux x64 | macOS arm64 | Windows x64 |
| --- | --- | --- | --- |
| debug | ✓ | ✓ | ✓ |
| debug `--asan=off` | ✓ | ✓ | (the default there) |
| release (LTO) | ✓ 262 s clean | ✓ | ✓ 421 s clean |
| release `--lto=off` | ✓ | ✓ | ✓ |
| `release-asan` | ✓ | | |
| FreeBSD x64 release, cross-built | links, checks pass | | |
- Linux release: no `panic_unwind` code in the binary; the build after
each of these is `ninja: no work to do`.
- Windows: the `.bin` shim is still 8,192 bytes and passes its checks.
It was the Windows debug build that showed the prebuilt std was missing
(275 undefined symbols from `core`, `std` and `alloc`).
- macOS: Apple's linker takes the `.rlib` files directly.
- **Before writing the change**, the same link was done by hand from an
unmodified debug tree: `bun_runtime`'s objects plus the 175 rlibs gave a
binary whose defined symbols were identical to the normal one's.
- The visibility change: `cargo check` in the dev and release profiles
for Linux gnu and musl, Android, macOS, FreeBSD and Windows; clippy on
the crate; release and debug builds here (0 duplicate strong symbols);
the dead-code inventory test, 44 pass with the built bun.
- `test/internal/build-rust-units.test.ts` gains a case on a synthetic
`-Zbuild-std` graph: a library root is named by its crate and hash, and
its link takes the target libraries with the one panic runtime the
profile uses (both `abort` and `unwind`) and no host code. The other
internal build-system tests pass; the two that said a plain debug build
links the prebuilt std now say it compiles std; two that used `mode:
"rust-only"` only to skip the macOS SDK lookup no longer need it.
## The binaries, against main
Every target's `bun` and `bun-profile` from a CI build of this branch
were compared with main's CI build of the commit it had merged: sizes
and sections, needed libraries, imported and exported symbols, every
symbol by demangled name, static initializers, embedded strings. That
build had a library root without the two source changes above.
- Stripped size within 0.1 % on all 12 targets; the code section is the
only one that moves by more than 1 KB; needed libraries, imports and
exports identical sets; static-initializer sections the same size; no
new paths or toolchain strings. It is what found the Android
thread-locals and the wrapper shapes.
- In that build the targets without C/C++ LTO still had a fat LTO over
the crates, and there the wrapper shapes already matched (FreeBSD x64:
3,360 of 3,389).
- That build's Linux x64 binary, run here against a pull-request build
of main's code: startup 2.434 ms vs 2.425 ms over 1,200 interleaved runs
each, `bun build` of typescript.js and a CPU-bound script within 1 %.
- Linux x64 with the two source changes, against main's CI binary:
stripped size +25 KB; instructions per call (`perf stat`, slope between
two iteration counts, five runs) for `fs.existsSync` +0.04 % and
`TextDecoder.decode` -0.6 %; `path.join` +0.6 %, where its wrapper is
806 instructions on main and 799 here and a build with main's exact
rustc semantics measured +0.2 % against the same baseline.
## Not verified here; CI will say
- Android's size with the export list in place of hidden visibility: the
builds with `-Zdefault-visibility=hidden` were +0.9 / +1.13 MiB against
main, and the flag alone accounts for that on a Linux build of the same
shape.
- The other 11 targets' binaries with the two source changes (the
comparison above has them for Linux x64 only).
- The macOS and Windows native builds in the table above were made
before the visibility change and before Rust became ThinLTO everywhere
(their `--lto=off` rows linked a fat LTO over the crates). With it,
those targets have `cargo check` in the dev and release profiles here,
and CI's builds.
- Windows arm64, FreeBSD, Android: that lld accepts and optimises the
Rust bitcode there. LTO is off for C++ on Windows arm64 because LLVM's
CodeView emitter aborts on JSC's NEON code under LTO; whether Rust's
bitcode hits the same is unknown.
- PGO builds: rustc used to put its profiler runtime into the archive;
the link now relies on the copy clang's `-fprofile-generate` brings,
which the comments in `rust.ts` say is the same LLVM runtime.
## Notes for review
- CI's `archive-link` mode still uploads `libbun-*.a` and the dependency
libraries as artifacts, and nothing downloads them now. Left as is.
- A build directory from before this change keeps its old
`libbun_runtime.a`; nothing reads it.
---------
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Cross-crate dead-code analysis (hawk)
hawk finds pub items no shipped root
reaches (hawk::dead_public) and over-visible declarations
(hawk::unnecessary_public, applied with --fix) across the whole Rust
workspace — the cross-crate cases rustc's per-crate dead_code cannot see.
The analysis config lives at the repo root: hawk.toml (release profile ×
the 11 shipped targets, plus overrides for enum variants whose discriminants
are external code tables).
Hawk needs a bin product to root the analysis, but Bun's product crate
(src/runtime) is a library whose rlib the native build links with the C++
objects. A [[bin]] there would be one more unit for every build to plan, so
the root is applied only for the duration of a run:
git apply tools/hawk/analysis-root.patch # adds a throwaway [[bin]] to src/runtime
printf 'fn main() {}\n' > src/runtime/hawk_root.rs
export BUN_CODEGEN_DIR="$PWD/build/debug/codegen" # run `bun bd --configure-only` + codegen first
cargo hawk check --jobs "$(nproc)" --output-format=json > hawk-report.json
git checkout -- src/runtime/Cargo.toml && rm -f src/runtime/hawk_root.rs # revert the scaffolding
The #[no_mangle] exports the C++ side calls are Hawk's real reachability
roots; the empty fn main() {} is sufficient because Hawk treats every
#[no_mangle] item as a root. // HOST_EXPORT(...)-marked functions are
kept via a [[root-marker]] (they are scraped by
src/codegen/generate-host-exports.ts and must stay pub fn).