build summary now tells why a cache miss occurred, if it was an uncommon
case:
* invalid cache manifest
* a specific file was invalidated
* directory contents changed
* metadata changed
Cache miss due to it being the first time a particular build step is
being executed remains silent.
Not all kinds of build steps have this extra data about which files
caused cache invalidation, most notably Compile steps.
- implement the logic for populating remaining input file digests when a
miss occurs while checking one of the input files
- make `Manifest.check` return not just "miss" but also explain why it
was a miss and which file caused it
- fix fundamental design flaw which clobbered input file digest, path,
and stat information that was needed to compare against on disk
manifest entries
- implement the logic for when the Manifest API user provides an already
available handle, stat, or contents, and when they request the handle
or contents.
- rename `Manifest.writeManifest` to `Manifest.finalize`
- rename `Manifest.manifest_dirty` to `Manifest.dirty`
would be nice if Io.Select.async had a variant where it returned the
eager result rather than trying to put it in the queue so that the queue
buffer doesn't need to be dynamically allocated
Io.Select.async has a fatal flaw: if the queue size isn't big enough to
hold the result of an eager async call, then it deadlocks.
The workaround in this thread is using concurrent() which should be
reverted because it will spawn too many threads.
- implement std.Build functions:
- `dependOnDirectoryMetadata`
- `dependOnDirectoryContents`
- `dependOnFileMetadata`
- `dependOnFileContents`
- switch cache manifest to binary format. saves about 25% of file size,
which is not a ton, but this is in the really hot path, so any amount
we can reduce file system cache pressure is going to help a lot.
- also means saving and loading is simpler - just copying bytes in and
out of memory plus a tiny bit of validation.
- batch cache manifest memory allocations together. one big allocation
for the entire manifest file contents, with the files hash map only
tracking starting offsets for each entry, and the input files as a
simple array list with the extra information needed for input files
(so they can be lazily loaded only on cache miss)
- remove the manifest file size limit
- input files that have requested content to be loaded is loaded
together into one big allocation rather than separately tracked.
- each file entry in the cache now tracks two additional bits:
- kind: file or directory
- tracking mode: contents or metadata
- when tracking a directory, "contents" are considered to be the
sorted list of file names contained in the directory (non recursive)
- rename "hit" to "check" and make it return enum instead of bool
- merge most "add...File" function variants together into one that has
an "options bag" parameter.
- take advantage of async I/O when checking for cache hit to compute
hashes concurrently. Care is taken to never cancel useful work, and
never wait for unnecessary work to complete.
bonus:
- build.zig: use std.log for printing log messages
When using Compilation.Path, we already know the path prefix, so just
use that directly instead of taking a detour through absolute paths.
Also, when using whole cache mode, don't call
`addModuleTableToCacheHash` because it is redundant with the logic
in `PerThread.update` which iterates over `zcu.alive_files` and adds
those files discovered via `@import` to the whole cache manifest.
That was causing files relative to cwd to be added to cache manifest
rather than being relative to build_root.
improves Path hygiene in the compiler in some places
also includes an assertion that will probably nede to be removed
regarding absolute paths making it into the cache manifest
Occasionally, two different Zig processes are waiting on the same
compilation artifacts. For example, if the user runs `zig build` for the
first time in two different terminals with the same version of Zig, one
of them will win the race on compiler_rt and start building it, while
the other process waits. Then they both use the same cache artifacts.
Now, the fact that this is happening is communicated to the user by
displaying it in a progress node during the checking for cache hit.
Matthew Lugg