Files
Alex Goodman 7c68081d9c chore: replace the integration tests with the quality gate (#3852)
* chore: replace the integration tests with the quality gate

- the Go integration tests in `test/integration` are gone, and the quality gate moves from `test/quality` into its place
- tasks are now `make integration`, `integration:score`, `integration:update-db` and `integration:clean`, matching vunnel's `integration:` tasks
- the race detector scan that used to ride along with the integration tests is now its own `make smoke-test` task and "Smoke tests" CI job
- labels live in `test/integration/additional-claims`, and the gate reads vunnel's from `tests/integration/additional-claims` (anchore/vunnel#1418)

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* test: check that each SBOM format matches the same as an image scan

- `make integration:sbom-formats` scans a few of the quality gate's images with grype from this checkout, then scans an SBOM of each image in syft-json, spdx-json and spdx-tag-value, and fails when the matches differ
- it uses the DB pinned in `test/integration/gates/pr.yaml`, so the comparison holds still between runs
- a format quirk that changes matches for a known reason is listed against that image and format, and a failure logs how the packages behind the differing matches differ
- runs as the new "Integration tests (SBOM formats)" CI job, in parallel with the gate

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* chore: split the integration tests into quality and api suites

- `make integration` runs both suites: `integration:quality` (the yardstick gate) and `integration:api` (the Go tests against grype's library)
- `integration:score` is now `integration:quality:score`
- CI runs the suites as a matrix in parallel, and one "Integration tests" check requires both

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* chore: log yardstick progress in CI

yardstick only shows progress on a terminal, so the CI job printed nothing until the verdict. In CI (`CI=true`) it now runs with `-v`.

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

---------

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
2026-10-27 17:38:24 -04:00

14 lines
582 B
YAML

# only pull in version updates that were released more than a week ago (low-pass filter for quickly-retracted releases)
cooldown: 7d
# note: we inherit tool definitions from anchore/go-make's embedded .binny.yaml, so we only
# need to specify tools here that are not already defined there (or that we want to override
# with different versions).
tools:
# used in the integration tests (quality gate) to compare grype from this checkout with the latest release
- name: yardstick
version:
want: v0.17.0
method: github-release
with:
repo: anchore/yardstick