Every agent shares one Docker daemon, and the coverage build used the
same `authelia:dist` tag that every suite's `pre-command` loads, retags
and removes. When another build's suites loaded their image between this
build's `docker build` and the `docker save` in `pre-artifact`, the tag
was gone (`reference does not exist`) or pointed at the other build's
image. Since `pre-artifact` had no `pipefail`, the failed save uploaded
an empty archive, the job passed and every suite in the build failed on
load.
The coverage image is now built under a per-job tag through a new
`--tag` flag on `authelia-scripts docker build`. `pre-artifact` takes
`authelia:dist` only to save the archive, inside the same lock the
suites hold around their load, and `pre-exit` removes the per-job tag.
With every write to `authelia:dist` under that lock, suites can no
longer pick up another build's image between their load and retag
either.
`pre-artifact` now fails the job when the save fails, and skips the save
when the build itself failed or was cancelled.
Signed-off-by: Amir Zarrinkafsh <3339418+nightah@users.noreply.github.com>
A pipeline-level `notify` webhook can only send to a URL, so it cannot
carry the `X-Buildkite-Token` header the Windmill HTTP route for
`buildkite_notification_webhook` now checks. The blocked build
notification moves to an organisation-level Notification Service webhook
scoped to this pipeline, and the hardcoded steps in the Buildkite
interface no longer include the `notify` block, so this copy drops it to
match.
Signed-off-by: Amir Zarrinkafsh <3339418+nightah@users.noreply.github.com>
Go fails every ancestor of a failing subtest and `gotestsum` reports
each of them, so a single unreliable test reaches Test Engine as a
separate failure and flaky test for every level of its name. Pass
`--junitfile-hide-parent-failures` so only the failure that accounts for
them is reported.
Signed-off-by: Amir Zarrinkafsh <3339418+nightah@users.noreply.github.com>
The published conformance suite image only offers Google and GitLab
logins, so a permanent conformance server showing the results the suites
write into the shared MongoDB would have to sit behind forward auth, and
fails to start at all without client ids for both providers. The image
is now rebuilt from the same release with `authelia-login.patch`
applied, which replaces both registrations with a single Authelia one
and the login page's two buttons with one carrying the Authelia logo.
Outside dev mode the server is configured with `OIDC_AUTHELIA_ISS`,
`OIDC_AUTHELIA_CLIENTID`, `OIDC_AUTHELIA_SECRET` and `OIDC_ADMIN_GROUP`.
The Authelia registration ships a placeholder issuer and credentials, so
the `OIDCConformance` suite no longer has to pass the Google and GitLab
placeholders, and the test asserting them is removed. The jar is built
once on the build platform and copied over the published image's, which
keeps its entrypoint, and the image is rebuilt when the patch changes.
The plans were described only by commit, or by a tag the suites never
run against, so the plans gathered on the shared server could not be
told apart by what they tested. They are now described as `branch
<branch> commit <commit>`, or `PR #<number> commit <commit>` for an
external contribution, whose Buildkite branch is the fork's
`owner:branch` form. Releases are never built from their tag, so a
commit on master whose header starts with `release: ` is described as
`release v<version> commit <commit>` instead. The branch and pull
request come from `BUILDKITE_BRANCH` and `BUILDKITE_PULL_REQUEST`,
falling back to the checked-out branch locally.
---------
Signed-off-by: Amir Zarrinkafsh <3339418+nightah@users.noreply.github.com>
A failed request shows up in the page only as a zero status. That does
not say whether the server never answered or Chrome gave up on it, and a
dropped request looks the same as a slow one.
Each tab now records the requests Chrome fails to complete, with the
reason Chrome gives, such as `net::ERR_NETWORK_CHANGED`. A failing test
writes them to `<test>.network.json` next to its other diagnostics, and
Buildkite uploads it with them. The recording ends when the tab closes.
---------
Signed-off-by: Amir Zarrinkafsh <3339418+nightah@users.noreply.github.com>
The `OIDCConformance` suite built the conformance server from
`Dockerfile.server` on every run, adding the build to each suite setup.
This moves it onto the same pipeline as the other integration images,
publishing it as `authelia/integration-oidcconformance` and pulling it
like `integration-haproxy`, `integration-duo` and `integration-samba`.
The Dockerfile moves to
`internal/suites/example/compose/oidc-conformance/Dockerfile`. It copies
the suite development CA from `common/pki`, so `integration.sh` gains an
optional `CONTEXT` for images whose build context is wider than their
own directory, and now passes the Dockerfile with `--file` rather than
changing into the directory. `pipeline.sh` triggers the new pipeline
when either the Dockerfile or `ca.public.crt` changes, and `pre-command`
points the suite at the branch tag when it does.
Signed-off-by: Amir Zarrinkafsh <3339418+nightah@users.noreply.github.com>
Organises the dynamic pipeline into four Buildkite group steps so the
build page reads as stages rather than one flat list:
`:hammer_and_wrench: Lint, Unit Test and Build` (Linting, Unit Test,
Build Image [coverage]), `:test_tube: Tests` (Integration Tests),
`:test_tube: External Tests` (External Tests) and `:rocket: Deploy`
(Deploy Manifest, Deploy Artifacts, Deploy AUR, Deploy APT).
`Vulnerability Scanning` and the `Trigger Pipeline` steps stay at the
top level.
Buildkite appends dynamically uploaded steps to the group containing the
uploading step only when the uploaded pipeline's first step is a group
whose name matches, so `e2etests.sh`, `e2epackages.sh` and
`aurpackages.sh` now emit their steps beneath a matching `group` key.
The suites therefore land inside `:test_tube: Tests`, `:test_tube:
External Tests` and `:rocket: Deploy` alongside the step that spawned
them instead of at the end of the pipeline. The group names, emoji
included, are byte-identical on both sides of the upload for that
reason.
Retires the `:selenium:` emoji, which has been wrong since the suites
moved to `github.com/go-rod/rod`: the suite steps are now `:chrome:`,
which is what go-rod actually drives over CDP. That emoji is a selector
rather than decoration, so the six `BUILDKITE_LABEL` conditions in
`post-checkout`, `pre-command` and `post-command` that gate `go mod
download`, the coverage image load, the codecov and Test Engine uploads,
the screenshot annotations and the container cleanup move with it.
`Integration Tests`, `External Tests` and `Deploy AUR` take
`:buildkite:`, both because it describes what those steps do — they only
run `buildkite-agent pipeline upload` — and because leaving the first
two on `:chrome:` would make them match the suite conditions. The AUR
package steps they upload move from `:linux:` to `:arch:`, which is the
distribution they actually publish to. The set of labels selected by
each condition is unchanged.
Also corrects the two documentation pages that still described the
suites as Selenium and chromedriver.
No step attribute changes: every `command`, `key`, `depends_on`,
`agents`, `retry`, `concurrency` and `if` is carried over untouched, and
keys remain globally unique so the existing cross-step dependencies
(`unit-test`, `build-docker-linux`, `build-docker-linux-coverage`,
`artifacts`) resolve across group boundaries.
Signed-off-by: Amir Zarrinkafsh <3339418+nightah@users.noreply.github.com>
We are moving branches in this repository to the `<type>/<summary>`
convention, which the container registries cannot take verbatim: a tag
admits only `[A-Za-z0-9_.-]`, so pushing
`authelia/authelia:fix/oidc-claims` fails with `invalid reference
format` on both DockerHub and GHCR. The same branch name reaches
`authelia/integration-duo` and friends through `resolve_tag_suffix`, and
the `sed` in the `INTEGRATION` hook additionally breaks on the `/` since
it is that expression's own delimiter.
Branch names are now folded into a tag by replacing anything outside the
permitted set with a `-`, so `fix/oidc-claims` publishes as
`fix-oidc-claims`. The mapping is implemented once per language that
needs it and each copy points at the other: `sanitize_tag` in
`.buildkite/libs/common.sh` covers `integration.sh`, `grypescans.sh` and
the `INTEGRATION` hook, and `sanitizeTag` in
`cmd/authelia-scripts/cmd/docker.go` covers the manifests we deploy. The
announcement in `.github/probot.js` keeps naming the tag it always did,
with a filter folding `head.ref` before the template renders it, since
the body is a mustache template and mustache cannot transform a value
itself.
The tag reaper needs the mapping too, and silently. It deletes every
published tag with no branch of the same name, so once the tags are
folded but the branch names it compares them against are not, every
branch carrying a `/` looks abandoned and loses its images on the next
`master` build. The branch list is now folded the same way before the
comparison, and sorted afterwards since folding changes the collation
order `comm` depends on.
While in that function, the GHCR lookup is matched with `index` rather
than `contains`. `contains` is a substring test, so reaping
`fix-oidc-claims` also deleted `fix-oidc-claims-policy`; prefixed branch
names make shared prefixes considerably more likely. It also emits a
version once per tag that matched, where `index` emits it once.
Renovate no longer needs `branchPrefix`, which only existed because
`renovate/` could not be used as a tag, so it returns to its default and
`branchPrefixOld` retires the branches left on the old prefix. The
collapse guard in `resolve_tag_suffix` moves with it.
Signed-off-by: Amir Zarrinkafsh <3339418+nightah@users.noreply.github.com>
Scorecard findings only showed up in the weekly workflow run against
`master`, well after the change had merged. The `lint.sh scorecard`
subcommand runs the four file based checks up front, so lefthook catches
them at commit time and reviewdog annotates them on the pull request.
The checks run against a `git checkout-index` export rather than the
working tree. Scanning the tree directly walks every `node_modules` and
does not finish; the export takes about 15 seconds. Only checks that
lost points are reported, as `scorecard` also warns about things it does
not penalize, such as the job level `contents` write the contributors
workflow needs to push the card.
`scorecard` and `jq` are now required tools. CI agents need `scorecard`
installed for the reviewdog runner.
---------
Signed-off-by: Amir Zarrinkafsh <3339418+nightah@users.noreply.github.com>
This adds the official OpenID Connect 1.0 Conformance Suites used by the
OpenID Foundation to assert conformance for certification to our test
suites. This allows us to offer the quality guarantee that every single
commit of Authelia conforms to the OpenID Connect 1.0 and related
specifications.
The tests allow the following results to indicate success automatically:
`PASSED`, `REVIEW` (result that manual review must occur), `SKIPPED`
(result that indicates the conformance test is not needed due to
reasonable criteria). In addition only the `FINISHED` status is
accepted.
If the result is `WARNING` which is just an informative result which
does not indicate lack of conformance it can be manually allowed on a
per module basis with an explanation. This allows both detecting new
`WARNING` results and necessitates documenting them for future
improvements.
All of the result and status checks align entirely with the standard
conformance certification process, ensuring that each commit would be
certifiable. The storage for the suites is able to be configured for
persistence, and every plan created carries the commit sha or tag;
allowing for historical evidence of conformance.
The linting step posts via `-reporter=github-check`, which hands the
results to the doghouse server, and the server fetches the raw pull
request diff whenever it is given a pull request number. GitHub answers
that request with `406 Sorry, the diff exceeded the maximum number of
files (300)` once a pull request is large enough, which aborts the post
with `fail to parse diff` and fails the whole step even though every
runner completed successfully.
The diff is dead weight here. The step runs with
`-filter-mode=nofilter`, so `FilterCheck` reports every diagnostic and
never consults the file diffs, and the check run itself is keyed on the
head SHA rather than the pull request, so the annotations land in
exactly the same place either way. Clearing `CI_PULL_REQUEST` for the
step skips the fetch entirely, which removes the 300 file cliff and one
API round trip per runner without losing any annotation. It is cleared
inline on the command rather than via step `env` because the node
environment hook exports it after the job environment is applied and
would otherwise put the value back. This must be revisited if the filter
mode ever becomes diff scoped.
`typos` aborts with `SIGABRT` when `std::fs::canonicalize` fails for an
entry the walker handed it: `report_result` reports the error and
substitutes `PathBuf::default()`, and `ConfigEngine::policy` then panics
on that empty path with `` `walk()` should be called first ``
(crate-ci/typos#1444, fix proposed in crate-ci/typos#1577). reviewdog
runs every runner in `.reviewdog.yml` concurrently in the same checkout,
so a file removed by another linter mid-walk is enough to trigger it, as
it was on #13100.
The wrapper prefixed every stderr line with a position so none of them
were discarded, which turned one crash into one finding per line: the
failing check carried 81 summary entries and 30 annotations, all of them
fragments of a single backtrace.
Prefix only the first line and let the rest match a `%C` continuation,
as the `golangci` runner already does. reviewdog puts the `%m` capture
of the `%E` line in `Diagnostic.Message` and the whole entry in
`Diagnostic.OriginalOutput`, which `service/github/check.go` sends as an
annotation's `raw_details`, so the crash becomes a single finding whose
reason is the first line and whose full backtrace stays intact one click
away. Ordinary typo findings are unaffected because an `%E` match
terminates the preceding entry.
The `%C%.%#` line is required rather than cosmetic: with `%E` alone the
entry is never terminated and reviewdog emits nothing at all.
Signed-off-by: Amir Zarrinkafsh <3339418+nightah@users.noreply.github.com>
The linting step has been unable to fail since `-fail-on-error` was
replaced with `-fail-level=error` when reviewdog was re-enabled, so a
failing runner has only ever shown up as a GitHub check while Buildkite
reported the step as passing. Build 58544 is an example: the `reuse`
runner reported `conclusion=failure` for two files missing SPDX headers
and the step still exited 0.
`-reporter=github-check` reports through the doghouse server, and that
path decides the exit code solely on `opt.failOnError`; `opt.failLevel`
is only plumbed into the reporters that comment directly. Pass both
flags so the deprecated one gates the step under `github-check` while
`-fail-level=error` keeps gating the `local` reporter used by the
private pipeline.
---------
Signed-off-by: Amir Zarrinkafsh <3339418+nightah@users.noreply.github.com>
The `typos` runner intermittently dies with `signal: aborted (core
dumped)` on the Buildkite agents, and reviewdog reports only `typos
failed with zero findings: The command itself failed`. reviewdog
discards a runner's stderr along with any stdout line that does not
match the `errorformat`, so whatever `typos` wrote on its way out never
reaches the build log, leaving nothing to diagnose.
Route the runner through `.buildkite/steps/lint.sh typos`, matching the
existing `shellcheck` subcommand. The wrapper captures stderr and, on
any exit code other than `0` (clean) or `2` (typos found), re-emits each
line as an `errorformat` matching finding against `.reviewdog.yml`. A
`SIGABRT` now surfaces as `typos failed (exit 134): <reason>` instead of
as an empty failure.
Cap `--threads` at `4` in both the wrapper and the lefthook job. `typos`
defaults to one thread per logical CPU and runs alongside five other
linters in the same step, which makes an in-process allocator abort the
most likely explanation for the crash. The whole repository still scans
in under a tenth of a second.
Adds `--only-fixed` to the `grype` invocation in `grypescans.sh` so both
the registry image scan and the CDX SBOM scans only fail on
vulnerabilities that have a fix available upstream. Unfixed CVEs are
noise for us: there is no action we can take to resolve them.
This change factorises common functions in the Buildkite pipeline and
fixes random missed triggers from a failed divergence check against a
`git merge-base --fork-point`.
Signed-off-by: Amir Zarrinkafsh <3339418+nightah@users.noreply.github.com>
An agent whose checkout was created with `git clone --depth` keeps every
tag ref but no tagged commit, because each tag points below the graft.
`git fetch --tags --force` restores the refs and leaves the truncation
in place, so `git describe --tags --abbrev=0` finds nothing to describe
`HEAD` with and GoReleaser fails with `git doesn't contain any tags`
after the frontend and swagger builds have already run. Two agents were
cloned this way, which is why the same commit failed on one and passed
when retried on another.
The hook now deepens a shallow checkout, checks the exit status of both
fetches, and asserts that a tag is reachable from `BUILDKITE_COMMIT`. A
checkout that cannot satisfy that fails the phase the agent retries and
wipes the directory for, rather than failing the job several minutes
later with an error that points at the wrong thing.
`getBuild` described from `git rev-list --tags --max-count=1`, which
finds a tag anywhere in the repository and then describes that commit
from itself, so it succeeded on exactly the checkouts GoReleaser
rejected. It now describes from `HEAD`, which is the question GoReleaser
asks, and takes `Tagged` from the described tag's own commit. Beyond
aligning the two, this stops `BuildTag` and the version string being
stamped from a tag the commit does not descend from.
Signed-off-by: Amir Zarrinkafsh <3339418+nightah@users.noreply.github.com>
Buildkite checks out by exact commit with `git fetch -v --prune --
origin <commit>`, and an explicit refspec disables git's tag
auto-following, so the agent's own fetch never brings a tag into the
checkout. That leaves this hook's bare `git fetch` as the only source of
tags, which picks them up as a side effect rather than a guarantee: it
restores tags that are missing entirely, but it silently leaves a tag
ref that already exists locally pointing at another commit, and its exit
status is never checked.
A checkout that ends up in that state fails late and confusingly.
`getBuild` describes from `git rev-list --tags --max-count=1`, so it
finds a tag anywhere in the repository and proceeds, and GoReleaser then
describes from `HEAD` and fails with `git doesn't contain any tags -
either add a tag or use --snapshot` after the frontend and swagger
builds have already run. The two disagree whenever the checkout holds
tags that are not ancestors of the commit being built, which this
repository has 35 of. Because the state belongs to the agent's
persistent working directory rather than the commit, the same commit
fails on one agent and passes when retried on another.
`--tags` fetches `refs/tags/*` explicitly instead of relying on
auto-following, and `--force` repairs stale tag refs, which a plain
fetch leaves untouched.
The `# shellcheck source=/dev/null` directive matches the one in
`healthcheck.sh` and silences the `SC1091` note for `bootstrap.sh`,
which the linter treats as blocking once the file is staged.
Co-authored-by: James Elliott <james-d-elliott@users.noreply.github.com>
pnpm resolves its store from the filesystem layout and never says which
branch it took: `${HOME}/.local/share/pnpm/store` when the project
shares a filesystem with the home directory, `<mountpoint>/.pnpm-store`
when it does not. On the agents `/buildkite` is an anonymous volume and
`/buildkite/builds` is the host bind mount, so every host side pnpm run
lands on `/buildkite/builds/.pnpm-store` and nothing has ever been
written to `/buildkite/.local`.
The coverage image copied that empty `/buildkite/.local` into the build
context and placed it at `/root/.local`, which is where the container's
pnpm does look, since both paths share the overlay there. The copy has
been empty since it was introduced, so its layer was permanently cached
and the image re-downloaded all 690 packages on every build.
Copying the real store in instead would cost more than it saves: it is
several gigabytes of small files that the build context would have to be
scanned through on every build. The store becomes a BuildKit cache
mount, which stays on the agent between builds and costs nothing to
carry, and `--store-dir` is passed explicitly because the guess pnpm
makes without it is what hid the original breakage.
`Dockerfile.dev` takes the same mount under the same cache id so the two
image builds warm each other, and both pin `pnpm@11` to match
`engines.pnpm` in `web/package.json`: a major bump rolls the store
version and would silently cold start the cache.
`docs/` ships a prettier configuration but nothing ever ran it, so
formatting drift was only caught when somebody thought to look. Wire it
into the two gates that already exist: lefthook fixes and re-stages the
docs files a commit touches, and reviewdog reports whatever is left on
the pull request.
Prettier has no machine readable output of its own, so it runs as the
only enabled rule of an ESLint configuration and reports through
`rdjson`, the same way the web package already does. That earns a
message naming the replacement, a range covering the offending word
rather than the whole line, and a suggestion the reviewer can commit,
none of which a diff of a rewritten tree can carry. It also leaves the
working tree alone, which matters because runners execute in parallel.
The `@eslint/*` packages are parsers rather than linters here: ESLint
has to build an AST before a rule can run, even one that works on the
raw source text, and that parse is the one thing prettier itself never
needs. It is also the only place the two disagree. SCSS gets through the
CSS parser only in tolerant mode, and the NDJSON test result file has to
be excluded because no JSON parser will accept it. Those exclusions live
in the ESLint configuration because prettier has no configuration file
option for ignores, and `.prettierignore` goes with them: every
remaining entry it carried named an extension ESLint never reaches.
Both lefthook jobs now lint `{staged_files}` rather than the whole
package. `stage_fixed` only ever stages the files lefthook filtered, so
a job that fixes everything leaves its repairs to files outside the
commit sitting in the working tree, surfacing only once the commit has
already finished. Scoping the command to the same set it can stage keeps
the two in step and stops a commit rewriting files it was never asked to
touch.
Correct the two integration guides the new configuration finds: emphasis
markers prettier normalises, which nothing had ever checked.
Rename `.prettierrc.yaml` to `prettier.config.mjs` so it matches the web
package, and drop the `format` script, which nothing calls now that
lefthook, reviewdog and the aggregate entrypoint all go through ESLint.
The linting step grows a `pnpm -C docs install` for the same reason it
already has one for `web`.
Both `lint` scripts take optional paths and fall back to the whole tree,
which is what lets lefthook hand them the staged set. Drop `--ext` while
passing, since flat configuration takes its file selection from the
configuration itself and the value given was a glob rather than the
extension list the flag expects. Replace `cd web &&` with `pnpm -C web
exec`, since `pnpm -C <dir> <binary>` resolves a binary only through
`exec` when it is not also a script of that package.
Signed-off-by: Amir Zarrinkafsh <3339418+nightah@users.noreply.github.com>
Six tests were reported flaky on master over seven days. Reading the
artifacts of every failing execution showed that most of what made them
hard to attribute was the harness discarding its own evidence.
Teardown runs after a setup that failed before its tab existed, so guard
the page it reaches for, and log a failed coverage collection rather
than calling `log.Fatal`, which ended the process and took the results
of the run with it. A run that exhausts `go test -timeout` panics from a
goroutine of its own, so no deferred capture runs and nothing is
uploaded at all: collect shortly before the deadline instead, and
discard it when the run finishes, since only a run that times out should
keep it.
Record what the page reports about itself, installed before the first
navigation because a document already loaded cannot be given the
recording afterwards, and name it in `artifact_paths`, which lists each
extension it uploads.
Bound element waits and report the state of the element with the
failure, rather than spending the rest of the test on one that never
appears. Raise the budgets while a test interrupts the datastore, and
settle the cluster before such a test returns, so the next test is not
measured against one still reconciling. Return the element on the page
the caller gave, since it hands that page to everything derived from it.
`TestShouldKeepSessionAfterRedisRestart` named the Traefik3 environment
although the suite it belongs to also runs as Traefik2.
Locally, wait on the log line the frontend actually emits when it
restarts, and drop the memory limit on that container: setup could not
otherwise get past its budget, and the dev server was being killed
inside the limit.
The import convention is four groups: the standard library, general
third party packages, the company packages under `authelia.com` and
`github.com/authelia`, and the project itself. `goimports` cannot
express this because `local-prefixes` describes a single trailing group
no matter how many prefixes are listed, so it tolerates the existing
layout but merges the company group into the third party group whenever
it formats a file, which `goimports-reviser` then has to undo.
`gci` is enabled to enforce it as its sections accept an ordered list,
with `prefix` covering both company prefixes and `localmodule` resolving
the project. It is bundled with `golangci-lint` so it replaces a
separately installed binary, and it runs after `goimports` so the two no
longer disagree about the result.
Equivalence was established by scrambling the import block of all six
hundred and thirteen files which have one and formatting the result with
each tool: the output is byte identical, including the twenty six
generated files both correctly decline to touch. `goimports-reviser` is
therefore removed from the reviewdog runners, the lint step, the git
hooks and the contributor documentation.
Signed-off-by: Amir Zarrinkafsh <3339418+nightah@users.noreply.github.com>
A `PathPrefix` run failed with the portal never rendering. The document
was served, 10 of the 27 chunks the entry module preloads returned
`200`, and the other 17 recorded a zero `responseStatus` against a zero
`encodedBodySize`, which is what a request that never received a
response looks like. The module graph therefore never resolved: `#root`
stayed empty, `#first-factor-stage` never appeared, and the test spent
its whole budget waiting for an element that could not arrive. No wait
recovers that page, because the document has loaded and nothing further
is ever fetched, so `doVisit` now fetches it once more when it finds an
empty root alongside a response that was lost. A page that is merely
slow, or one that was answered and still did not render, is left alone
so that no visit is made worse by this.
The proxy the suites run behind was starved at the time. It logged a
docker event stamped `22:43:54` at `22:44:11`, 17 seconds late, and the
configuration rebuild it was working through enumerated the 199
containers the shared daemon was holding for every other suite on the
machine. `--providers.providersthrottleduration` coalesces those events
so that work stays off the two cores the container has to proxy the
suite with.
Whether any of the 17 requests reached the proxy at all is a question
only its access log answers, and that log was not collectable. It shared
stdout with the debug log, where one rebuild emits a line per container
the daemon holds, so the 200 lines a failing test collected held no
access log line at all. Measured against a quiet daemon the same tail
also held none of the `Remote error` lines the `forwardAuth` middleware
logs and none of the routing configuration, so raising the level to
`INFO` would have traded one missing record for another. The access log
moves to a file in the suite directory instead, which a failing test
collects whole and `Up` discards from the run before, and the tail of
stdout carries the debug log alone at a depth one rebuild cannot
exhaust.
The suites are swept of the comments that restated the code they sat
above. The GoDoc of every unexported function goes, since the name says
what the header did, along with the narration in the scenarios, where
`// Logout.` sat above `doLogout` and `// Login one factor.` above
`doLoginOneFactor`. What a reader cannot get from the code stays, moved
to the line it explains: why the previous run's access log is removed,
why a missing one is not an error, why `rod` is asked for the text as
rendered, and how many failures it takes for the browser to give up on a
chunk.
Grype resolved the scan target through the Docker daemon, which returns
whatever is tagged locally. `authelia-scripts docker push-manifest`
builds with buildx and pushes straight to the registry, so the manifest
a build has just published is never loaded into the daemon and the scan
reads an image left over from an earlier build instead. Builds 57317 and
57344 both failed on Go stdlib advisories against `go1.26.5` while the
images they had actually pushed carried `go1.26.6` binaries and scan
clean.
`grypescans.sh` now scans `registry:${IMAGE}`. Nothing is written to the
image store, so the scan cannot go stale and cannot leave a tag behind.
Every run fetches a manifest rather than hitting a warm local tag, so
the Docker Hub login in `pre-command` is no longer gated on `CI_PRIVATE`
and those fetches are not counted against the anonymous pull rate limit
shared by every agent behind the one address; the matching `docker
logout` in `post-command` follows it.
Daemon resolution had already left one `authelia/authelia:<branch>` tag
behind per branch ever scanned on a host, and those accumulate on a
shared daemon that `cleanbuild=false` deliberately never prunes.
`pre-exit` drains them after the scan job. The filter names the two
repositories exactly so nothing the agents retain is reachable, and the
removal is by name without `-f` so an image also reached by another tag,
such as the `authelia:dist` a concurrent Build Image job is about to
save, is only untagged rather than deleted.
`grypescans.sh` also records failures instead of aborting on the first
one. A failing image scan previously skipped the `*.cdx.json` loop
entirely, which meant the SBOMs of the binaries the build actually
produced went unscanned in exactly the builds where that mattered most.
Signed-off-by: Amir Zarrinkafsh <3339418+nightah@users.noreply.github.com>
The coverage build image tarball is the largest artifact in the pipeline
and every `:selenium:` suite step has to download it before it can load
the image, so with the number of suites we run the fetch from the shared
Buildkite artifact store dominates the setup time of each of those
steps.
A `post-artifact` hook is added so that once the `:docker: Build Image
[coverage]` step has uploaded normally on a `tabris` agent, a `-tabris`
suffixed copy of `authelia-image-coverage.tar.zst` is also uploaded to a
node-local S3 bucket via the `BUILDKITE_ARTIFACT_UPLOAD_DESTINATION` and
`BUILDKITE_S3_*` overrides, placing the artifact alongside the agents
that will consume it.
The `pre-command` hook then prefers that localised copy on `tabris`
agents, renaming it back to the canonical name so the remainder of the
hook (the Kubernetes path, the slot aware `docker load`, and the
integration container rewrites) is untouched. If the localised download
fails, because the image was built on another node or the bucket is
unavailable, it falls back to the default artifact store, and agents
outside the pool keep the original single download path.
Signed-off-by: Amir Zarrinkafsh <3339418+nightah@users.noreply.github.com>
Co-authored-by: James Elliott <james-d-elliott@users.noreply.github.com>
Each Buildkite agent runs its own Docker daemon, so
`.buildkite/hooks/pre-exit` prunes the entire daemon after every job and
each of the ~50 agents on a host re-unpacks the same images from the
pull-through registry. Docker does not support two daemons sharing a
`data-root`, so storing each image once per host means one daemon per
host, and that only works once concurrent jobs stop colliding on the
resources they currently assume they own alone.
Every suite used the compose project `authelia` and the subnet
`192.168.240.0/24`. Both are now driven by `COMPOSE_PROJECT_NAME` and
`SUITE_SUBNET`, which `bootstrap.sh` derives from a per-agent
`SUITE_SLOT`, and the Traefik Docker provider is constrained to its own
project so it stops discovering other jobs' containers. That constraint
is given as command arguments and `static.yml` is retired, because
Traefik treats its three static configuration sources as mutually
exclusive and a `--configFile` makes it ignore every other argument. The
files compose cannot interpolate follow the same variable: the
`hostEntries` table, the redis `sentinel.conf` and `slave.conf`
templates via `entrypoint.sh`, and `NetworkACL/configuration.yml`
through Authelia's own `template` configuration filter. The matching in
`prepareHostsFile` also moves from a substring test to a field
comparison, since `mail.example.com` otherwise matched the
`mx1.mail.example.com` line and left the real entry pointing at the
previous subnet.
Chrome and the test process both run inside the agent container, and on
a shared daemon the suite network belongs to the host namespace rather
than the agent's, so `Up()` attaches the agent to
`${COMPOSE_PROJECT_NAME}_authelianet` and `Down()` detaches it before
teardown. `SUITE_TMP` is mounted so that the host, the agent and the
suite containers all resolve one path, which is what the CLI suite
relies on when a command writes a file in one and reads it from the
other. `SUITE_IMAGE` replaces the single mutable `authelia:dist` tag
with a per slot one whose lifetime matches its owner, the phpldapadmin
and envoy admin ports are offset per slot, and the crossbuild container
loses its fixed name and takes its toolchain from `/buildkite/.tools`
rather than `/usr`, because both the name and those mount sources belong
to the host now rather than to the agent that requested them.
Agents that opt in with `CLEANBUILD=false` no longer run `docker system
prune -af --volumes` or remove every container on the daemon; they
remove only what their own compose project created, including the images
it built. Because the daemon is no longer wiped between jobs, the
integration images gain a `pull_policy` so a mutable tag cannot freeze
at whatever a host first pulled.
Alongside the migration this fixes several failures it exposed. A suite
that dies during startup now has its container logs collected before
teardown, rather than leaving nothing but an exit code. Clicks locate
their element again when the document replaced it, which was failing the
change password scenario with a detached node. Incognito browser
contexts are tracked so they are disposed rather than accumulating for
the life of the shared browser. The reset password view guards against
initiating twice, and the development frontend pins pnpm and is given
enough memory to build. Every variable falls back to its current value,
so local development and the existing agents are unchanged and the
migration itself becomes a configuration change on the agent side.
---------
Signed-off-by: Amir Zarrinkafsh <3339418+nightah@users.noreply.github.com>
Co-authored-by: James Elliott <james-d-elliott@users.noreply.github.com>
Suite failures were absorbed by a blanket automatic retry, which meant
flakiness had no measurement and no record of what failed. Pulling 300
builds from the Buildkite API showed the scale of it: 3,666 suite jobs
carried 3,237 automatic retries, only 52% passed on the first attempt,
and the retry rate scaled with host contention from 10.5 per build when
nothing else was running to 67 when a dozen builds overlapped.
Emit `go test -json` and upload it so the flake rate can be measured,
bound the retries, and capture a screenshot, the page markup, the
resource timing and every container's logs whenever a test fails,
echoing the container logs through the test so they reach the failure
body rather than a separate process the log cannot be attributed from.
Fix the races this surfaced: an element helper that never waited, a
one-time password secret read before it was populated, a notification
asserted after the redirect that destroys it, and a navigation with no
load gate.
Address the contention the measurements pointed at. The containers
declared no limits, so every containerised Go runtime sized itself for
the whole host, and the six second default read timeout turned a
scheduling stall into a 408 and a blank login page. Gate the environment
on healthchecks under CI and declare the dependencies the backend needs,
since a failed startup check is fatal and a backend started before its
directory or database is ready crash loops until that dependency
appears.
Scope the remaining log based waits to entries produced after the action
being waited on, because `docker compose logs` is cumulative and a wait
could otherwise be satisfied by an entry from a previous test. Doing so
exposed a high availability test that had been passing for the wrong
reason, which now resolves the redis role from the sentinel rather than
assuming the order tests happen to run in.
The last three builds completed with no retries at all across all 28
suite jobs.
Closes#12720
---------
Signed-off-by: Amir Zarrinkafsh <3339418+nightah@users.noreply.github.com>
The buildkite release step previously published GitHub releases via the
legacy `hub` CLI and built its changelog body from
`conventional-changelog` output alone. This change replaces `hub release
create` / `hub release delete` with `gh release create` / `gh release
delete` and extends the notes pipeline so each bullet carries `by
@handle` attribution and a `New Contributors` section appears above the
existing Docker Container block.
The contributor attribution is bolted on as a bash `awk` post-process:
the step caches `gh api repos/authelia/authelia/compare/<prev>...<tag>`
to a tmp file, uses `jq` to render a flat `sha` & `login` map, then
pipes `conventional-changelog -p angular` output through an `awk` script
that extracts the 40-char commit sha from each bullet's `/commit/<sha>`
URL, looks it up in the map, and appends ` by @<login>`. The `New
Contributors` block comes from a separate `gh api -X POST
repos/.../releases/generate-notes` call (the side-effect-free preview
endpoint that backs GitHub's own auto-generated notes), filtered through
`awk` to pluck just that section out of the response body and downgrade
its heading from `## ` to `### ` so it matches the rest of the notes.
Every published release from `v4.26.1` through `v4.39.19` has been
backfilled with `by @handle` and `New Contributors`.
The `.github/workflows/pr-labels.yml` workflow gains automatic label
triage: it maps the Conventional Commit type to a matching `type/*`
label and the scope (when present) to an `area/*` label, using mappings
restricted to the type and scope sets enumerated in
`web/commitlint.config.mjs`. `status/needs-review` is applied once on
`opened` and left untouched on subsequent events so reviewers can
advance the status freely.
Several new `type/*` and `area/*` labels were created on the repository
so the workflow can apply them: `type/performance`, `type/refactor`,
`type/build`, `type/ci`, `type/test`, `type/release`, `type/revert`,
`type/i18n`, plus `area/authorization`, `area/server`, `area/totp`,
`area/duo`, `area/metrics`, `area/suites`, `area/buildkite`,
`area/utils`, `area/service`, `area/expression`, `area/ntp`.
---------
Signed-off-by: Amir Zarrinkafsh <3339418+nightah@users.noreply.github.com>
The git commit that publishes the apt repository update was previously
unsigned, even though the `Release.gpg` and `InRelease` files it
contains are GPG-signed with `security@authelia.com`. Sign the commit
with the same key so the deploy is verifiable end-to-end.
The apt `Release` signing on the preceding lines populates `gpg-agent`'s
passphrase cache for `security@authelia.com` via `--pinentry-mode
loopback --passphrase`, so `git commit -S` a few lines later hits the
cache and signs non-interactively without a wrapper, temp file, or
duplicated passphrase plumbing. Add `git verify-commit HEAD` as a gate
before `git push` so a bad signature aborts the deploy rather than
publishing an unverifiable commit.
Validated on Buildkite nodes with the production key: `git
verify-commit` exits zero, `%G?` returns `G` (good signature), `%GS`
resolves to the `security@authelia.com` UID, and the author/committer
remain `autheliabot`.
Signed-off-by: Amir Zarrinkafsh <3339418+nightah@users.noreply.github.com>
Add a new end-to-end suite that exercises the external pam_authelia
module (github.com/authelia/pam) against a real Authelia stack. The
suite container clones pam_authelia inside its Dockerfile and builds
both the Go binary and the PAM C shim, then runs sshd with
pam_authelia.so wired into pre-seeded /etc/pam.d entries so tests switch
modes per-case by `cp`-ing a config over /etc/pam.d/sshd.
Tests cover the four flows the module supports: 1FA only, 1FA+2FA with
TOTP, 2FA only (password sourced from pam_unix via PAM_AUTHTOK), and the
RFC 8628 OAuth2 Device Authorization grant backed by a pre-provisioned
confidential OIDC client. SSH logins use golang.org/x/crypto/ssh
keyboard-interactive so no external test tooling is required, and each
test asserts both the ssh exit code and the shim's debug log output.
Closes#497
---------
Signed-off-by: Amir Zarrinkafsh <3339418+nightah@users.noreply.github.com>
Replace the per-CVE stdlib pin block in `.grype.yaml` with a single
location-scoped ignore that only applies to Go stdlib reported from
inside the lefthook prebuilt binaries. The previous list churned every
time the grype DB rotated, since the `lefthook-linux-x64` npm
devDependency ships an embedded Go stdlib that surfaces every new Go
vulnerability against the `public_html` SBOM even though lefthook is a
developer-only git hook runner that never reaches a runtime artifact.
The `location:` field only gates matches when the SBOM carries
source-path metadata. SPDX-JSON records this as a free-text `sourceInfo`
string that grype does not promote into the package's `locations` array,
so `location:` globs cannot match. CycloneDX preserves the path under
`syft:location:0:path`, which grype reads, so a `**/lefthook*/**` glob
reliably scopes the ignore to the lefthook source path without
over-matching a hypothetical Authelia binary that happened to ship the
same stdlib version.
Switch the grype consumers accordingly: `.buildkite/hooks/pre-command`
downloads `*.cdx.json` for the `:grype: Vulnerability Scanning` label,
and `.buildkite/steps/grypescans.sh` iterates over `*.cdx.json`.
`.goreleaser.yml` still produces both `cdx-*` and `spdx-*` SBOM IDs, and
`.buildkite/annotations/artifacts.sh` still advertises both extensions,
so the SPDX SBOM continues to ship as a release artifact.
OpenVEX was evaluated and rejected. Grype's OpenVEX matcher only checks
`products[].@id` against the matched package purl; the `subcomponents`
array is parsed and ignored. A VEX statement with `product=lefthook +
subcomponent=stdlib` does not suppress the match, and inverting it is
functionally identical to a name+version+type ignore with no genuine
source scoping.
Also added `reason:` fields to the existing busybox 1.37.0 entries
explaining that the vulnerable code paths are backported in
`authelia/base`. The reported version string is unchanged from upstream
so grype still flags it.
Signed-off-by: Amir Zarrinkafsh <3339418+nightah@users.noreply.github.com>
After `hub release create` succeeds push `master` and all tags from
`authelia/authelia` to `authelia/authelia-cve` so the private CVE fork
stays aligned with upstream git metadata on every release. Without this
the CVE fork's history drifts, which makes Go's `-buildvcs` stamping and
Authelia's `BuildTag` resolution fall back to whatever stale tag the
fork happens to have reachable.
Feed `GITHUB_TOKEN` through a git credential helper rather than
embedding it in the remote URL so the token never lands in argv, git's
stderr, or on-disk config. Sync failures log a `:warning:` line but do
not fail the step, since the release itself has already been published
and should not be held hostage to a transient push error against the
mirror.
Signed-off-by: Amir Zarrinkafsh <3339418+nightah@users.noreply.github.com>
Grype failed with `MANIFEST_UNKNOWN` on the `authelia-cve` pipeline
because `grypescans.sh` hardcoded `authelia/authelia` while the private
pipeline publishes the image to `authelia/authelia-cve`. Append the
`-cve` suffix to the image name when `CI_PRIVATE` is set so grype scans
the tag that actually exists.
Since `authelia/authelia-cve` is a private repository grype also needs
registry credentials to pull the manifest. Add a `docker login` to the
grype `pre-command` hook and a matching `docker logout` to
`post-command`, both gated on `CI_PRIVATE`, reusing the
`DOCKER_USERNAME` and `DOCKER_PASSWORD` variables already wired up for
the public deploy step.
Signed-off-by: Amir Zarrinkafsh <3339418+nightah@users.noreply.github.com>
Wire goimports-reviser into the three existing lint paths (pre-commit
via lefthook, CI review comments via reviewdog, and the manual runner in
`.buildkite/steps/lint.sh`) so import ordering, grouping, and
unused-import removal are enforced consistently across local and CI
workflows.
The runner is configured with `-rm-unused -format -company-prefixes
authelia.com,github.com/authelia`, grouping first-party Authelia
packages into their own block after third-party imports. `-set-alias` is
intentionally omitted; imports work without an alias, and we do not want
one added automatically.
Signed-off-by: Amir Zarrinkafsh <3339418+nightah@users.noreply.github.com>
Several of the `.buildkite/` scripts still carried near-identical logic
after the earlier `pipeline.sh` cleanup: the DockerHub and GHCR
tag-cleanup loops in post-command repeated an identical inner flow,
`e2etests.sh` fanned out selenium agent selection across five
almost-identical heredoc arms, `pre-command` expanded the per-suite
compose rewrite into an elif chain, and `grypescans.sh` echoed and
invoked grype once per arm rather than resolving a single `IMAGE` first.
Collapse each concern behind one helper, lookup, or dispatch
(cleanup_tags, an associative array, a case statement, a single `IMAGE`
assignment) so a rule change only needs one edit.
Also guard the recently-introduced private authelia-cve pipeline against
leaking linting output to public GitHub: flip `LINT_REPORTER` to
reviewdog's local reporter when `CI_PRIVATE` is true, and skip the
`post-command` github-pr-review step entirely. Minor drive-by: reflow
the tab/space-mixed indentation in `annotations/artifacts.sh`.
Signed-off-by: Amir Zarrinkafsh <3339418+nightah@users.noreply.github.com>
The pipeline setup script repeated the full git-diff invocation and the
CI-bypass regex across every branch of the diverged/tag/branch
conditional, so a change to the bypass file list or the file-detection
logic required touching three near-identical copies.
Consolidate the base ref selection into a single BASE_REF variable,
extract the bypass regex to a BYPASS_REGEX constant, and add two small
helpers (changed, bypass_check) shared by the master, non-master and
merge-queue paths so future updates happen in one place.
Signed-off-by: Amir Zarrinkafsh <3339418+nightah@users.noreply.github.com>
The merge-queue bypass check diffed against `git merge-base
origin/master HEAD`, which relies on the Buildkite agent's cached
`origin/master` being current. When the agent's `origin/master` lagged
even one commit behind the actual queue base, the merge-base resolved to
an older ancestor and the diff picked up files from unrelated PRs that
had landed in between, flipping `CI_MERGE_QUEUE_BYPASS` to `false` for
docs-only PRs and triggering steps (e.g. grype vulnerability scanning)
that expect SBOM artifacts that were never produced upstream.
Since `gh-readonly-queue/*/pr-N-*` branches are always a single commit
on top of the queue base, diff against `HEAD^ HEAD` instead. This scopes
the bypass check to just the PR's own changes and removes the dependency
on `origin/master` freshness on the agent.
Signed-off-by: Amir Zarrinkafsh <3339418+nightah@users.noreply.github.com>
Introduce ShellCheck as an enforced lint step so shell scripts are held
to the same strictness as the Go, YAML, and JavaScript surfaces, and
take the opportunity to normalize the repository's shell hygiene while
the tooling is being put in place. A new .shellcheckrc keeps the default
ShellCheck rule set enabled and opts in to require-variable-braces
(SC2250) to enforce the ${VAR} style repo-wide. The tool is wired into
the local pre-commit path via lefthook (with file_types filtering on
text/x-shellscript and text/x-sh so only libmagic-detected shell files
are passed in, and --format=gcc so the output is clickable from the
IDE), into the Buildkite 🐕🦺 Linting step via a new shellcheck
subcommand on .buildkite/steps/lint.sh that discovers every tracked
shell script through git ls-files and a shebang scan, and into reviewdog
via .reviewdog.yml using the same lint.sh subcommand with --format=gcc
so findings annotate pull requests inline.
Along with the wiring this commit clears the ShellCheck baseline against
the current tree. Every bare $VAR reference was rewritten to ${VAR} so
SC2250 lands on an already-clean repository. Unquoted expansions and
command substitutions (SC2086 and SC2046) were quoted across the
Buildkite steps, hooks, pipeline.sh, setup.sh, redis and samba
entrypoints, and grypescans.sh; read calls gained -r (SC2162), cd calls
got || exit guards (SC2164), declare-and-assign in bootstrap.sh was
split to preserve return values (SC2155), indirect $? checks in setup.sh
were rewritten as if ! cmd (SC2181), repeated redirect blocks in samba
init.sh were collapsed into { ... } > file (SC2129), an unused LDOMAIN
variable was removed (SC2034), the runtime source in healthcheck.sh was
marked with a # shellcheck source=/dev/null directive (SC1091), and
framework-consumed variables in authelia-fb-rc.d gained a file-level
disable directive. Two suite entrypoints that used bash-only syntax
under #!/bin/sh had their shebangs bumped to #!/usr/bin/env bash, two
dev-host gen.sh scripts had their /bin/bash absolute paths swapped for
/usr/bin/env bash so they pick up Homebrew bash on developer machines,
and grypescans.sh had its split-string grypeCmd rewritten as a proper
bash array so the intentional word-splitting no longer needed an inline
disable.
---------
Signed-off-by: Amir Zarrinkafsh <3339418+nightah@users.noreply.github.com>
Co-authored-by: James Elliott <james-d-elliott@users.noreply.github.com>
Tighten the apt deploy script so it runs shellcheck-clean and lands a
less ambiguous commit in authelia/apt. Quote every expansion (SC2086),
guard the final cd with || exit (SC2164), and hoist the
/buildkite/.cache/apt path into an APT_CACHE_ROOT constant so the cache
location is defined once and easy to move. Also prefix the commit
summary written to authelia/apt with "authelia" so that repository's
history clearly
identifies which upstream project each release entry belongs to.
Signed-off-by: Amir Zarrinkafsh <3339418+nightah@users.noreply.github.com>
Replaces the `pkgvalidate.sh` build-only check with a new `suites
external` namespace that spins up a project-local dev server (Hugo for
docs, react-email for templates) and drives a headless rod browser
against it to assert the rendered output is correct.
Before, CI only confirmed that `pnpm -C docs build` and `pnpm -C
internal/templates/src render` exited cleanly — a regression in a
shortcode, a broken image path, a missing search initialiser, a Tailwind
class, or a template PreviewProps drift would pass silently.
Suites are gated behind a `//go:build externalsuites` tag and registered
in their own `ExternalGlobalRegistry`, so `go test ./internal/suites`
and `authelia-scripts suites list` remain unaffected. A new
`authelia-scripts suites external [list|test]` sub-namespace runs them
via a shared `DevServer` lifecycle helper that spawns `pnpm dev` in its
own process group, polls for HTTP readiness, and tears the whole tree
down on exit.
In Buildkite, `.buildkite/steps/pkgvalidate.sh` is deleted, and the two
old package-validation steps are replaced with a single `External Tests`
step that runs `e2epackages.sh | buildkite-agent pipeline upload`,
mirroring the `e2etests.sh` dynamic pattern so new external suites
register without pipeline edits.
---------
Signed-off-by: Amir Zarrinkafsh <3339418+nightah@users.noreply.github.com>
The docs and templates package validation steps were running
unnecessarily during merge queue builds. This adds the CI_MERGE_QUEUE
condition to their step filters, consistent with how other non-essential
steps are already skipped in the merge queue.
This adds two new Buildkite pipeline steps which validate that the pnpm
install and build/render commands complete successfully for the docs and
internal/templates/src packages respectively. A new pkgvalidate.sh step
script is introduced which accepts either 'docs' or 'templates' as an
argument to determine the target package.
Replace citizenstig/httpbin with mccutchen/go-httpbin and update all
reverse proxy configurations (HAProxy, Caddy, Nginx, Envoy) to use the
default port 8080. Replace sameersbn/squid:3.5.27-1 with
ubuntu/squid:6.6-24.04_edge and trim the squid configuration to only the
active directives. Add linux/arm64 to the Buildkite integration image
build platform targets for the authelia/integration-haproxy,
authelia/integration-duo and authelia/integration-samba images.
This runs the Grype vulnerability scanning step during merge queue
builds to scan SPDX SBOMs from the corresponding PR build. The step runs
without dependencies on other steps since those are bypassed in the
merge queue. Artifacts are retrieved from the last passed PR build via
the Buildkite API.
Amir Zarrinkafsh