Before the changes in this commit, the intention was to gather all pending lazy dependencies of all package scripts, and then only handle fetching after the root package script exits. However, that's not always feasible, since it's possible for a lazy dependency to depend on a lazy dependency, which means that the first fetch-and-retry can only know about the root's lazy dependencies. Then, during the second attempt at configuring, the root package script will see its lazy dependency as available, but in reality that dependency's package script may return error.LazyDependencyNeeded, which won't matter until *after* the root package script has finished running. Therefore, the root package script will continue running as if its lazy dependency is fully available, when in reality it has requested a fetch-and-retry.
This means that something like an `artifact` call in the root package script that depends on a lazy dependency of a lazy dependency will fail after the first fetch-and-retry, because the necessary remaining pending lazy dependencies would only get fetched once the root script finished.
In case it's helpful, the reproduction looks something like this:
root
└─ child1 (.lazy = true, gotten in root's build.zig via either `dependency` or `try dependencyLazy`)
└─ child2 (.lazy = true, gotten in child1's build.zig via `try dependencyLazy`)
To fix this, whenever an artifact/module/etc is looked up and not found while there are pending lazy dependencies, we now immediately exit, fetch, and retry, since that artifact/module/etc could only exist after one of those pending lazy dependencies is made available (i.e. the dependency's configuration stopped part-way due to a missing lazy dependency).
Before (fails):
> zig build
debug(fetch): FileNotFound: zig-pkg\zig_test_repo1-0.0.0-FuDjYawXAAAK0sFpnAa7IPMoTJpCs44sgfKP7ZBi3oAN
debug(fetch): FileNotFound: C:\Users\Ryan\AppData\Local\zig\p/zig_test_repo1-0.0.0-FuDjYawXAAAK0sFpnAa7IPMoTJpCs44sgfKP7ZBi3oAN.tar.gz
debug: configuration cache incomplete_manifest
info: fetching lazy dependency zig_test_repo1-0.0.0-FuDjYawXAAAK0sFpnAa7IPMoTJpCs44sgfKP7ZBi3oAN
debug(fetch): FileNotFound: zig-pkg\zig_test_repo1-0.0.0-FuDjYawXAAAK0sFpnAa7IPMoTJpCs44sgfKP7ZBi3oAN
debug(fetch): FileNotFound: C:\Users\Ryan\AppData\Local\zig\p/zig_test_repo1-0.0.0-FuDjYawXAAAK0sFpnAa7IPMoTJpCs44sgfKP7ZBi3oAN.tar.gz
debug(fetch): FileNotFound: zig-pkg\zig_test_repo2-0.0.0-eVW_ihsPAAAmwffTitFedoK7HGHiRfQTshvARXs0-qJx
debug(fetch): FileNotFound: C:\Users\Ryan\AppData\Local\zig\p/zig_test_repo2-0.0.0-eVW_ihsPAAAmwffTitFedoK7HGHiRfQTshvARXs0-qJx.tar.gz
debug: configuration cache incomplete_manifest
panic: unable to find artifact "zig_test_repo1_lib"
After (succeeds):
> zig build
debug(fetch): FileNotFound: zig-pkg\zig_test_repo1-0.0.0-FuDjYawXAAAK0sFpnAa7IPMoTJpCs44sgfKP7ZBi3oAN
debug(fetch): FileNotFound: C:\Users\Ryan\AppData\Local\zig\p/zig_test_repo1-0.0.0-FuDjYawXAAAK0sFpnAa7IPMoTJpCs44sgfKP7ZBi3oAN.tar.gz
debug: configuration cache incomplete_manifest
info: fetching lazy dependency zig_test_repo1-0.0.0-FuDjYawXAAAK0sFpnAa7IPMoTJpCs44sgfKP7ZBi3oAN
debug(fetch): FileNotFound: zig-pkg\zig_test_repo1-0.0.0-FuDjYawXAAAK0sFpnAa7IPMoTJpCs44sgfKP7ZBi3oAN
debug(fetch): FileNotFound: C:\Users\Ryan\AppData\Local\zig\p/zig_test_repo1-0.0.0-FuDjYawXAAAK0sFpnAa7IPMoTJpCs44sgfKP7ZBi3oAN.tar.gz
debug(fetch): FileNotFound: zig-pkg\zig_test_repo2-0.0.0-eVW_ihsPAAAmwffTitFedoK7HGHiRfQTshvARXs0-qJx
debug(fetch): FileNotFound: C:\Users\Ryan\AppData\Local\zig\p/zig_test_repo2-0.0.0-eVW_ihsPAAAmwffTitFedoK7HGHiRfQTshvARXs0-qJx.tar.gz
debug: configuration cache incomplete_manifest
info: fetching lazy dependency zig_test_repo2-0.0.0-eVW_ihsPAAAmwffTitFedoK7HGHiRfQTshvARXs0-qJx
debug(fetch): FileNotFound: zig-pkg\zig_test_repo2-0.0.0-eVW_ihsPAAAmwffTitFedoK7HGHiRfQTshvARXs0-qJx
debug(fetch): FileNotFound: C:\Users\Ryan\AppData\Local\zig\p/zig_test_repo2-0.0.0-eVW_ihsPAAAmwffTitFedoK7HGHiRfQTshvARXs0-qJx.tar.gz
debug: configuration cache incomplete_manifest
The logic in Maker expects unlazy_deps to only be populated for deps that need to be fetched. Without this check, declareDependency/declareDependencyStep could lead to an infinite fetch-and-retry loop.
The new deployed_prefix and related paths exist to support the use-case
of embedding absolute paths to other build artifacts in the build
artifacts themselves. For example pkg-config files are artifacts which
must contain absolute paths to the other library/header artifacts they
describe. Note that this feature is only supported to faciliate
interoperability with such legacy systems. In general, embedding
absolute paths in build artifacts should be avoided.
This patch also fixes a bug: currently passing an absolute path to
e.g. --prefix-lib-dir will cause $DESTDIR to be ignored for build
artifacts installed to that path.
I'm not entirely satisfied with the new "deployed" naming convention,
I want to avoid the breaking change of renaming "installed" to something
else and this is the best option I came up with.
Closes: https://codeberg.org/ziglang/zig/issues/37124
In order to allow the compiler to default to ReleaseSafe when compiling
for fuzzing, Build.standardOptimizeOption has been changed to return an
optional value.
- implement std.Build functions:
- `dependOnDirectoryMetadata`
- `dependOnDirectoryContents`
- `dependOnFileMetadata`
- `dependOnFileContents`
- switch cache manifest to binary format. saves about 25% of file size,
which is not a ton, but this is in the really hot path, so any amount
we can reduce file system cache pressure is going to help a lot.
- also means saving and loading is simpler - just copying bytes in and
out of memory plus a tiny bit of validation.
- batch cache manifest memory allocations together. one big allocation
for the entire manifest file contents, with the files hash map only
tracking starting offsets for each entry, and the input files as a
simple array list with the extra information needed for input files
(so they can be lazily loaded only on cache miss)
- remove the manifest file size limit
- input files that have requested content to be loaded is loaded
together into one big allocation rather than separately tracked.
- each file entry in the cache now tracks two additional bits:
- kind: file or directory
- tracking mode: contents or metadata
- when tracking a directory, "contents" are considered to be the
sorted list of file names contained in the directory (non recursive)
- rename "hit" to "check" and make it return enum instead of bool
- merge most "add...File" function variants together into one that has
an "options bag" parameter.
- take advantage of async I/O when checking for cache hit to compute
hashes concurrently. Care is taken to never cancel useful work, and
never wait for unnecessary work to complete.
bonus:
- build.zig: use std.log for printing log messages
followups from 8e48a5c524 along with
other related changes.
- user input options should not be serialized into Configuration.
- more helpful log messages when invalid -D arguments are provided
- use array hash maps in various places, simplifying iteration and
sorting.
- when creating a package instance, sort the user input options once,
before the lookup, rather than creating a fresh hash map and sorting
it on every call to hash (wtf!)
- when sorting, don't use ascii case insensitivity (wtf!!)
- simplify the detection of invalid arguments passed to zig build
- use package hash rather than file system path in package instance key
Technically there is one valid use case for `fieldNames` which is use
with an enum or union so that those types can be used interchangeably.
But in practice these functions are mainly abused, because the callsites
always know what kind of type it is.
This commit encourages Zig users to embrace using `@typeInfo` directly
when doing type reflection.
These were originally intended to tell the configure script whether or
not those third party integrations were enabled.
This is now a problem because we want to produce a configuration file
that is indendent of whether such external integrations will be enabled,
so that the logic does not need to be re-executed when those flags are
changed on the command line.
If we wish to make this feature interact with the configure script, let
us consider carefully how to add it in a future enhancement, and not
leave these dead fields sitting around in the meantime.
closes#35607
and remove "release" from the enum tag names.
no functional change, however, despite the addition of
backwards-compatibile declarations in this patch, it is breaking because
expressions that use `==` or `!=` operators will not able to use the
deprecated names.
I predict these names to survive until the 1.0 tag without any more
breakage.
in favor of dependencyLazy which has a different function signature and
will be eventually renamed to dependency
migrating to all dependencies being potentially lazy by setting the flag
in build.zig.zon and embracing error.LazyDependencyNeeded propagation as
the way to deal with discovered needed dependencies.
solves the problem that handling the `null` case from `lazyDependency`
was annoying.
This function can be used to prepend a list of arguments when running a
test step. Instead, the additional arguments should be manually added to
the run step which can handle a broader set of use cases.
This partially reverts commit af27e8d06f.
This broke two platforms:
* s390x-linux: It caused the `loadBits and storeBits` test to be analyzed and
run, and that test looks to me to be bogus on big endian.
* x86_64-windows: `tryFindProgram` is not fallible, yet the Windows code uses
`try`, so making it get analyzed exposed another compile error.
Reverting the analysis of `findProgram` for now to unbreak master and a number
of PRs, but we should still urgently fix both of these issues.
Repeated calls to `addModule` with the same name will override the
module that is visible to dependants which is likely unintentional.
Instead it should assert that no existing module has already been added
with the given name.
The same issue applies to `addNamedWriteFiles` and `addNamedLazyPath`.
On 32-bit systems usize is not u64 but it was assumed in multiple places
of the build script.
Sometimes using u64 instead of usize since available and used memory can
exceed usize on 32-bit systems (just like totalSystemMemory)
Example errors:
lib/compiler/build_runner.zig:508:26: error: expected type 'usize', found 'u64'
.available_rss = max_rss,
^~~~~~~
lib/compiler/build_runner.zig:508:26: note: unsigned 32-bit int cannot represent all possible unsigned 64-bit values
referenced by:
callMain [inlined]: lib/std/start.zig:699:88
callMainWithArgs [inlined]: lib/std/start.zig:638:20
posixCallMainAndExit: lib/std/start.zig:590:38
2 reference(s) hidden; use '-freference-trace=5' to see all references
lib/std/Build/WebServer.zig:849:38: error: expected type 'usize', found 'u64'
const buf = gpa.realloc(old_buf, new_len) catch @panic("out of memory");
^~~~~~~
lib/std/Build/WebServer.zig:849:38: note: unsigned 32-bit int cannot represent all possible unsigned 64-bit values
lib/std/mem/Allocator.zig:399:58: note: parameter type declared here
pub fn realloc(self: Allocator, old_mem: anytype, new_n: usize) Error!@TypeOf(old_mem) {
^~~~~
build.zig:548:10: error: type 'usize' cannot represent integer value '9300000000'
.max_rss = 9_300_000_000,
~^~~~~~~~~~~~~~~~~~~~~~~
build.zig:778:10: error: type 'usize' cannot represent integer value '8000000000'
.max_rss = 8_000_000_000,
~^~~~~~~~~~~~~~~~~~~~~~~
Ryan Liptak