Extends the module compile cache to WebAssembly modules loaded through
the ES module integration. A new kWasm entry type is keyed on the module
URL with the wire bytes as the hashed source, and stores the serialized
CompiledWasmModule. On load, cached code is deserialized through
v8::WasmModuleCompilation with the same compile options the translator
uses (js-string builtins and imported string constants), falling back
to compilation when V8 rejects it.
As for JavaScript, the cache entry is generated right after compilation.
V8 only serializes optimized tier code, so with the default lazy
baseline compilation there is nothing to cache and no entry is written;
a complete cache requires --no-liftoff --no-wasm-lazy-compilation, which
is documented as the way to use the compile cache for WebAssembly.
Signed-off-by: Guy Bedford <guybedford@gmail.com>
Assisted-by: Claude Code
PR-URL: https://github.com/nodejs/node/pull/66191
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Adds a "worker" package condition that is applied within Web Worker
threads created via the `Worker` global, for the worker entry point,
its dependencies, `import()`, `require()` and `import.meta.resolve()`.
Threads created via `worker_threads`, including from within a Web
Worker, do not apply the condition. Other conditions are unaffected.
Assisted-by: OpenCode
Signed-off-by: Guy Bedford <guybedford@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/66635
Reviewed-By: James M Snell <jasnell@gmail.com>
The relative resolve cache was keyed by a concatenated
${parent.path}\x00${request} string, allocating a new key for every
require() call including fully cached ones. Key the cache by the
parent directory first (a Map keyed by the already-retained
module.path string) and then by the request (a dictionary object,
whose property access internalizes dynamically-constructed
specifiers). Faster on every measured workload shape and slightly
smaller in memory, since the concatenated keys are no longer
retained.
Signed-off-by: Sam Attard <sattard@anthropic.com>
PR-URL: https://github.com/nodejs/node/pull/63884
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
Mounting a virtual file system requires calling vfs.mount() from inside
the program, so a program cannot itself be served from one: something
already running has to mount the VFS first.
Add two startup flags. --vfs-mount=<source> mounts a directory or an
archive as a virtual file system, and may be repeated.
--vfs-load=<source> mounts that source the same way and additionally
runs the entry point and all subsequent require()/import resolution
against it rather than against the real file system, so an application
can be run straight out of a directory or a ZIP archive:
node --experimental-vfs --vfs-load=my-app.zip
Both options append to one list, so mounts happen in the order written:
node --experimental-vfs --vfs-mount=a --vfs-load=b --vfs-mount=c
mounts a, b and c in that order and runs b. Mounting the same source
twice mounts it twice, at two mount points, and the entry point comes
from the mount --vfs-load contributed rather than the earlier one.
The provider backing a source is chosen from the source itself rather
than from its name: a directory is served by RealFSProvider and a file
whose bytes are a ZIP archive by ZipProvider, so an archive can carry
any extension. vfs.registerProvider() registers a provider for formats
there is no built-in for; selection is deferred until -r and --import
preloads have run, so a preloaded module can register one before its
source is claimed.
The entry point comes from the mount the way `node <directory>` takes
one, from package.json "main" or index.js. Nothing is consumed as an
entry point argument, so every positional reaches the program unchanged
from argv[2] on, and argv[1] reports the named source rather than the
reserved mount point, which is an opaque implementation detail.
--vfs-load may be given at most once, and is not permitted in
NODE_OPTIONS: which entry point runs is the command line's decision, and
NODE_OPTIONS is parsed first, so an environment variable could otherwise
redirect any invocation. --vfs-mount is permitted there. Workers inherit
the same mounts in the same order but not the loading: a worker mounts
what the parent mounted and runs its own entry point.
Signed-off-by: Philipp Dunkel <pip@pipobscure.com>
PR-URL: https://github.com/nodejs/node/pull/65748
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
A compile cache generated ahead of time and shipped inside an
application package should only ever be read: the package may be
immutable or covered by an integrity check, and a cache directory that
appears at run time would be a surprise. Add readOnly to
module.enableCompileCache() and NODE_COMPILE_CACHE_READONLY=1: existing
entries are loaded as before, nothing is serialized or persisted,
flushCompileCache() is a no-op, and the cache directory is used as found
rather than created, so enabling against a missing directory fails
instead of making one.
Signed-off-by: Shelley Vohr <shelley.vohr@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/65302
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: René <contact.9a5d6388@renegade334.me.uk>
The MIME regex used for data: URLs could backtrack super-linearly on a
malformed URL lacking a ',' separator. Make the optional parameter
group anchored on ';' so it cannot overlap with the media-type group.
Adds a benchmark (esm/get-data-protocol-format) parameterized over
path length, so a backtracking regression shows up as an ops/sec
cliff instead of a wall-clock assertion in a test, per review.
Fixes: https://github.com/nodejs/node/issues/61904
Signed-off-by: skdas20 <skdas5405@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/61951
Reviewed-By: Aviv Keller <me@aviv.sh>
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
Contexts that are not deserialized from the built-in snapshot -- worker
threads, and the main context of embedders that create their own
isolate or of `node --no-node-snapshot` -- compile (with the code cache
at best) every builtin the bootstrap touches, so each eagerly required
builtin is startup time (~0.15-0.4 ms apiece). A number of them are only
required eagerly so that they end up in the snapshot, or for features
the bootstrap path never uses.
Load lazily what those paths do not need:
- is_main_thread.js: preload util, url, the ESM loader (translators,
resolver, module_job/map, source maps, node:module, vm modules, mime,
data_url, the TypeScript stripper), internal/blob and
internal/dns/utils only while building a snapshot; they load on first
use otherwise.
- fs: internal/blob (+ internal/encoding and its tables) is only used
by fs.openAsBlob().
- internal/url: internal/data_url (+ internal/mime) is only used by the
Buffer-returning file URL helpers.
- internal/process/execution, the CommonJS loader, esm/translators and
esm/load: the TypeScript stripper and data: URL helpers are only
needed for TypeScript sources / data: URLs.
- pre_execution: internal/dns/utils (+ internal/net) is only needed up
front to validate an explicit --dns-result-order or to register the
resolver's snapshot serializer; the default order becomes the
variable's initializer.
- internal/worker: event_loop_utilization and error_serdes are only
needed once a sub-worker's ELU is read or it reports an error.
- worker_threads: `locks` is defined lazily, like util's lazy exports.
Main-thread startup with the snapshot is unchanged (the same modules
are preloaded into it; the bootstrap-modules test lists are adjusted).
A bare worker compiles 95 -> 83 builtins (cold start -5%); without the
snapshot an empty CommonJS entry point compiles 76 -> 59 builtins and an
empty ES module entry point 76 -> 69.
Signed-off-by: Shelley Vohr <shelley.vohr@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/65329
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
Reviewed-By: Aviv Keller <me@aviv.sh>
getNearestParentPackageJSON() memoized its answer per file, so every
module loaded still made one native call, and TraverseParent() builds
several std::filesystem::path temporaries per directory level and
serializes the whole package.json, which the JS side then usually
discarded because it already had that package.json deserialized.
The native traversal starts at the directory of the given path, so the
answer only depends on that directory: key the memo by it (following
NormalizePath()'s trailing-separator rule), so that all modules in a
directory share one native call. When the permission model is enabled
the traversal also depends on the read permissions in effect at call
time, so that configuration keeps the per-file cache.
Loading a 1000-module tree spread over ~240 directories goes from 1000
to 236 native calls.
Signed-off-by: Shelley Vohr <shelley.vohr@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/65326
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Jordan Harband <ljharb@gmail.com>
This brings back several improvements that were reverted by mistake when
landing https://redirect.github.com/nodejs/node/pull/64154
- Update the documentation about how the removal of side effects of
source collection
- Add non-enumerable `requireStack` and `topLevelAwaitLocations`
properties to `ERR_REQUIRE_ASYNC_MODULE`, the latter is only
populated when --experimental-print-required-tla is enabled
- Add "Required module: <url>" to the error message to identify the
required ESM entry point regardless of whether the flag is enabled
- Fix TLA caret column from 0-based to 1-based
- Store module source via a private symbol instead of a public property
- Use `hasAsyncGraph` (post-instantiation) in `throwIfAsyncGraph`
instead of walking the graph before instantiation
- Merge the require stack checking into the
`common.expectRequiredTLAError` helper.
- Removed tests that are made redundant by the snapshot tests
Signed-off-by: Joyee Cheung <joyeec9h3@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/64260
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
Previously in order to collect the locations of the TLA, we wait
until right before evalutation to ensure instantiation is
completed so that we can use
v8::Module::GetStalledTopLevelAwaitMessages(). Now we try to
add an additioanl shortcut to the source code in the module wraps
instead during compilation for modules that contain TLAs and
use acron to locate the TLAs when we need to throw
ERR_REQUIRE_AYNSC_MODULE, so we can do this as early as before
instantiation and do not need to run the module again to collect
the locations.
In addition, we now collect the require stack for
ERR_REQUIRE_ASYNC_MODULE too for better metadata in the errors.
Signed-off-by: Joyee Cheung <joyeec9h3@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/64154
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
The CommonJS loader was passing the bare filesystem path as the
`//# sourceURL` comment of type-stripped TypeScript, this leads to
two problems:
1. It reports hasSourceURL = false if the path contains any
whitespaces as this breaks V8's magic comment parser.
2. The inspector would incorrectly report a file path verbatim as
the script's URL.
Pass the module's file: URL as the sourceURL so the reported URL
is consistent across loaders and won't lead to an issue in
inspector clients that actually except a real URL for the
scripts.
Signed-off-by: Joyee Cheung <joyeec9h3@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/63705
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Marco Ippolito <marcoippolito54@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
Since the ESM loader is captured in the snapshot now, there's
no need to lazy load the helpers. Load them eagerly to capture
them into the snapshot. This also reduces the noise coming out
of --print-bytecode since we no longer compile the helper
functions at run time.
Signed-off-by: Joyee Cheung <joyeec9h3@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/63550
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>
Decided to take a short break from the work on QUIC
to implement a DTLS API. Very experimental at this
point but the basic API is there (inspired by the
QUIC API work).
The implementation is based on OpenSSL's built-in
DTLS support and no other dependencies are required.
DTLS is a datagram-based version of TLS that is used
for things like WebRTC and CoAP. It provides similar
security guarantees as TLS but is designed to work over
UDP instead of TCP.
This shouldn't be considered ready for production
but it is a good starting point for experimentation
and feedback.
```bash
./configure --experimental-dtls
make -j{nproc}
./node --experimental-dtls my-dtls-app.js
```
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode:Opus 4.6
PR-URL: https://github.com/nodejs/node/pull/63182
Fixes: https://github.com/nodejs/node/issues/61630
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Stephen Belanger <admin@stephenbelanger.com>
Reviewed-By: Rafael Gonzaga <rafael.nunu@hotmail.com>
- For imported CJS, if it's not customized by asynchronous hooks,
make sure it won't use the quirky re-invented require in all
cases.
- When the imported CJS module is customized by synchronous hooks,
in the synthetic module evalutation step, avoid calling the
respective default step again.
- Make the branching of loadCJSModuleWithModuleLoad() and
loadCJSModuleWithSpecialRequire() more explicit, and fold
the tentative fs read in the 'commonjs' translator into the
share createCJSModuleWrap() helper instead of checking it
twice in the same path.
Signed-off-by: Joyee Cheung <joyeec9h3@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/62920
Fixes: https://github.com/nodejs/node/issues/63060
Reviewed-By: Paolo Insogna <paolo@cowtech.it>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Gürgün Dayıoğlu <hey@gurgun.day>
Currently, --watch mode only tracks dependencies from the main
module graph (require/import). Worker thread entry points created
via new Worker() are not included, so changes to worker files do
not trigger restarts.
This change hooks into Worker initialization and registers the
worker entry file with watch mode, ensuring restarts when worker
files change.
Fixes: https://github.com/nodejs/node/issues/62275
Signed-off-by: SudhansuBandha <sudhansu9.vssut@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/62368
Reviewed-By: Moshe Atlow <moshe@atlow.co.il>
Temporarily restore fs patchability in ESM loader as a workaround
for helping downstream projects that depend on this undocumented
hidden contract transition into using hook proper APIs. This patch
intentionally avoids adding a test and instead adds warning
comments to hopefully steer new code away from depending on it.
PR-URL: https://github.com/nodejs/node/pull/62835
Refs: https://github.com/nodejs/node/issues/62012
Reviewed-By: Michaël Zasso <targos@protonmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Extends the Wasm ESM Integration for importing WebAssembly
modules in either the source phase or instance phase to support
importing static JS string constants from the special
import name `wasm:js/string-constants`.
PR-URL: https://github.com/nodejs/node/pull/62198
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Guy Bedford