* vpm: add registry protocol implementation with core API endpoints
* vpm: add registry protocol tests and fix latest-version selection
* vpm: authenticate registry requests with a scoped bearer token
Private registries need credentials, and a machine holding tokens for more
than one of them must not send one registry's token to another. A token is
therefore read from \VPM_TOKEN_<HOST>\ first, where the host has its dots
and dashes folded to underscores, and from plain \VPM_TOKEN\ only as a
fallback for setups with a single private registry.
The header is added at \ pm_http_request\, the one function every registry
call already goes through, so no caller has to opt in. A 401 that arrives
without a token configured is reported by naming the variable to set, rather
than surfacing as an opaque transport failure.
* vpm: sign the registry index with ed25519
A registry that a mirror serves cannot be told apart from the original, so a
compromised mirror could hand out edited metadata with nothing to detect it.
The index is signed with an ed25519 key, and signature.sig serves the
signature alongside it: a mirror holds no key of its own to re-sign with, so
editing a checkout breaks verification.
The key is read from VPM_REGISTRY_KEY or the signing.key file beside the
index, so signing needs no key written to disk. erify_signature rejects
malformed hex instead of trusting it.
Five tests cover the unsigned case, a valid round trip, tampering, a foreign
key, and malformed hex.
* vpm: revalidate registry responses with entity tags
A client polling an unchanged index retransmits the whole body every time,
which is wasteful when the registry is large and rarely changes. \serve\
layers HTTP caching over \handle_request\: it computes the sha256 of the
answer as a quoted entity tag, and replies \304 Not Modified\ with no body
when the client's \If-None-Match\ already names it.
Caching lives above the router rather than inside it, so the routing stays
testable on its own. \If-None-Match\ is matched per the HTTP grammar:
comma-separated lists, a \W/\ weak-validator prefix, and \*\ all count, and
publishing a new version changes the body and therefore invalidates the tag
a client is still holding.
Eleven tests cover tag stability, the quoting rule, the 304 short circuit,
stale and weak tags, and publication invalidating a held tag.
* vpm: serve a change feed so a mirror syncs only what changed
Bringing a mirror up to date means reading the whole index, which is wasteful
when most of it has not moved. The registry now keeps an append-only log of
publish, yank and unyank, and /api/changes?since=<unix> returns only
the entries recorded at or after that second.
since is inclusive of its own second, because a mirror that has read up to
second N must be sent a change stamped N rather than silently missing it. A
mark that cannot be parsed serves everything, so a mirror sending a bad value
still gets a usable answer instead of an error.
The signature deliberately does not cover the log: it records wall-clock
times, so including it would make an unchanged registry produce a different
signature on every run. A test pins that by comparing two registries with
identical modules and different log lengths.
The timestamps in the tests are fixed rather than taken from the clock,
because .unix() truncates to seconds and every change in a test run lands
in the same one, which would make since untestable.
Twelve tests cover the kinds recorded, phantom entries from failed yanks, the
inclusive boundary, a future mark, malformed timestamps, and the endpoint
with and without a mark.
* vpm: emit an SPDX software bill of materials for a registry
A consumer about to install has no way to audit what it is taking on without
resolving the graph itself, and licence obligations are exactly what an
installer should not have to discover afterwards. \sbom.spdx.json\ renders the
modules as an SPDX 2.3 document, with a package per version, so the existing
SPDX toolchain can read it rather than V alone.
The shape follows SPDX: the \sha256:\ prefix is VPM metadata rather than a
digest, so it is stripped before the checksum is recorded; element ids have
\/\, \.\ and \:\ replaced, since none are legal there; and each package
carries a purl, which is what other supply-chain tools join on. A version with
build metadata cannot be expressed as a CPE, so the CPE reference is omitted
rather than emitted malformed.
Fourteen tests cover the document header, one package per version, checksum
prefixing, an absent checksum, id rewriting, purl and CPE shapes, the
build-metadata case, and a round trip through a decoder.
* vpm: let a registry store and serve the archive a version points at
A registry could describe a module and sign it, but it had no way to deliver
one: nothing stored the sources and nothing served them, so `@v/<version>.zip`
answered "not found" for every module that had ever been published. This adds
the two halves that make the protocol usable end to end.
`publish` stores the archive and records it in the index. It refuses an archive
whose bytes do not match the checksum the metadata claims, because a registry
that accepted one would hand out sources nobody vouched for. When nothing was
claimed it records what the bytes actually hash to rather than a checksum that
vouches for nothing.
A published version is immutable. Replacing one would change what an existing
lockfile points at, so it is refused; re-publishing the identical bytes is a
no-op, which keeps a retry after a partial failure safe.
`artifact_request` serves the archive. It lives beside `handle_request` because
an archive is not text: the response names a file for the transport to stream
rather than carrying it in the string every other endpoint returns. `serve`
routes to it first, so an archive never gets a body hashed for an entity tag
and never answers a conditional request.
The bytes are checked against the index before serving, so a store tampered
with after publishing is caught at the registry rather than at the client.
The immutability guard is verified by breaking it, not by trusting a green run:
with it disabled both the replace-refusal test and the no-op republish test
fail.
The tests chdir into their own tree, because the registry stores under the
working directory and otherwise leaves `.vpm-registry` behind in the checkout.
* vpm: serve the registry protocol over HTTP, and report not-found as 404
The protocol was defined as routing functions with nothing that could host it,
so a third party still could not stand a registry up. `registry_server.v` wraps
it in an `http.Server`: `RegistryHandler` routes each request through `serve`
and answers it, and `v registry serve [--port N]` runs it.
Writing the HTTP layer immediately exposed a defect in the protocol itself. Every
route answered `{"error": "not found"}` with status 200, so a client had to parse
the body to tell a miss from a hit -- which is exactly what a client cannot rely
on. `route` now maps that one body to 404, and the body is a named constant so
the router and the transport share a single contract rather than two guesses at
a string. Any other error the router can produce still needs its own status, and
`route` says so where it would have to be added.
`http.Header` is a struct of common and custom keys rather than a map, so only
`If-None-Match` is read and passed on -- it is the one header the routing uses.
`ResponseConfig` is immutable once built, so an archive's body and its content
type are decided before it is constructed rather than patched afterwards.
NOT VERIFIED: `registry_server_test.v` drives a real server on a free port and
has never completed a green run on this host. It found the 404 defect above and
then hit `could not allocate` from the compiler, which is host memory pressure
rather than a code failure -- 3 other compilers were running and 2.8 GB was free.
The tests should be run before this is trusted.
* vpm: enforce registry artifact integrity and deterministic metadata
* v: route the \
egistry\ subcommand to vpm
\ registry serve\ never reached vpm: the \ \ frontend hands a subcommand to an
external tool only if it is named in \external_commands\ or in the inline list
here, and \
egistry\ was in neither. The frontend therefore treated it as a
file to compile and reported \multiple input paths are not supported: registry
and serve\.
Found by running the command rather than by reading the diff that added it.
* docs: document the vpm registry protocol and what is not wired up yet
Covers what the registry protocol answers and what it does not: every route and
the metadata behind it, the ed25519 signature over the index, entity-tag
caching, the change feed, the SPDX bill of materials, where the data lives on
disk, and the parts that are deliberately not wired up, so no route is read as
reachable from `v install` or from `v registry`.
The server itself is unverified end to end: its test suite never completed a
green run on this host, so the HTTP layer rests on the in-process assertions
rather than on an observed request against a bound port.
* v: recognise `registry` where the subcommand is actually found
The dispatch list is not the only place a subcommand has to be named.
`find_command` walks the arguments looking for the command word and returns its
index, and it keeps its own list; a subcommand missing from that list never
produces an index, so `main` never reaches the dispatch at all and both words
fall through to the compiler as input paths. That is why
`multiple input paths are not supported: registry and serve` was still the
answer after the dispatch list alone was fixed: that change was dead code, and
running the command is what showed it.
`run_external_tool` maps a frontend subcommand to its `cmd/tools/` directory by
name, so `registry` resolves to `cmd/tools/vregistry` unless it is told
otherwise. Point it at `vpm`, which is where the command actually lives, rather
than renaming the command to suit the mapping.
While here: `--port` was silently ignored, because `parse_query_args` strips
every argument beginning with `-` once it has the command word, so the flag
never reached `vpm_registry`. It now does, for `registry` only — the other
subcommands depend on that stripping to find their package names, so their path
through the loop is unchanged.
`/config.json` reported `public_key: ""` even with a signing key configured,
because the field was never populated. A client reading the config to learn
which key vouches for a registry got nothing; it now carries the key.
NOT VERIFIED: nothing here has been compiled or run. The host has under a
gigabyte of RAM free, so any build dies with `could not allocate`. `v fmt`
passes, which is a parse-level check only.
* vpm: validate registry transport bytes and conditional responses
* vpm: add the client half of the registry protocol
The server side could describe and serve a module, but nothing in the toolchain
could ask one. These three functions are what a client needs to talk to a
registry speaking the protocol, and nothing more: they are deliberately not
wired into `install` or `parse`, because that change would decide how a module
is located and installed from an archive rather than from a VCS clone, and it
should be reviewed on its own.
Each fetch reuses `vpm_http_get`, so it picks up the same bearer token the rest
of vpm already sends rather than growing a second mechanism.
A registry answering 404 is reported as an error naming the module, not as a
zero-valued struct. That distinction is the whole point of the client: the
existing `get_mod_vpm_info` loop treats a 404 as "not in this registry, try the
next one", and a struct with empty fields would be indistinguishable from a
module that really exists with no metadata. The same check also compares the
body against the server's own `not_found_body`, because something in front of a
registry can rewrite the status line while leaving the body alone, and
`json2.decode` ignores unknown keys rather than complaining about them.
The tests decode a literal wire body instead of starting a server, so a drift
between what the server encodes and what the client reads fails here rather than
at install time.
NOT VERIFIED: this has never been compiled. The host has had under 5 GB of RAM
free throughout, so every `v` build dies with `could not allocate`. `v fmt` was
not run either. Everything here is hand-checked only, and the test that pins the
wire layout is exactly the one that will fail if my reading of the encoder is
wrong — which is what it is for.
* vpm: verify registry client requests against the HTTP server
* tests: include registry documentation in os directory listing
* vpm: consult a registry when resolving a module, before the VCS path
`v install` resolves a module by asking candidate servers for
`GET /api/packages/<name>` and cloning the URL in the reply. A registry that
speaks the protocol described in `doc/vpm_registry.md` can now answer that
question too, and is asked first. The existing loop is untouched: when
`VPM_REGISTRY` is unset, the same requests are made in the same order and this
is a no-op.
An empty version list is read as "this registry does not have the module", the
same as a 404. That is the point of the change rather than an incidental choice:
`GET /<name>/@v/list` answers 200 with an empty array for a module it does not
know, so a resolver that counted "answered 200" as success would pin the first
server that replied and stop looking, leaving the user with the wrong
registry's answer. A registry never becomes the selected server here, so that
cannot happen.
A registry can name a module but not where to clone it. Its archive base comes
from `dl` in `config.json`, which today serves `https://example.com/downloads`
— an RFC 2606 documentation address that is not a host. Rather than return a
module with an empty URL for some later caller to clone, that case fails with
the registry's own answer and says plainly that the download is not
implemented. When `dl` names a real base the same failure is reported, because
no archive download exists yet either.
`VPM_REGISTRY` is new. There was no registry URL variable before; the existing
`VPM_REGISTRY_KEY` is the signing seed and is not one.
NOT VERIFIED. The test file has never been compiled or run, and neither has the
final state of the two files it touches. Memory stayed under 5 GB throughout,
so every `v test` attempt died before reaching the linker. The build of
`cmd/tools/vpm` itself is green, but that predates this commit and proves
nothing about it.
* docs: state the vpm registry protocol's authentication as it is, and cover the token on the wire
`docs/registry-protocol.md` claimed tokens came from `POST /api/login` and that
read endpoints could require auth per registry. Neither is true: no route in
`cmd/tools/vpm` accepts or issues credentials, and `auth_required` is hardcoded
`false` in `registry.v` and read by nothing. `doc/vpm_registry.md` already said
this correctly, so the protocol document is the one that was wrong.
Rewrite the Authentication section from what the source actually does: the
`Authorization: Bearer` header is set by the VPM client only, from
`VPM_TOKEN_<HOST>` with a `VPM_TOKEN` fallback (`registry_token` in `common.v`),
no route obtains a token, the registry server authenticates nothing, and
authenticated redirects stay within the registry origin
(`vpm_registry_redirect`). The endpoint table's login and refresh rows are
replaced by the fact that there are none, and the `auth_required` comment now
says it is served for a client to parse rather than describing behaviour.
The same document's claim that `registry_server_test.v` asserted the token
against a bound port was false: that file never set a token variable nor read an
`Authorization` header, so nothing covered the hop from `vpm_http_request`'s
header to the socket. `registry_client_test.v` now binds a real server that
answers with the `Authorization` it received, and four cases assert what
`vpm_http_get` puts on the wire: the scoped variable, the `VPM_TOKEN` fallback,
nothing when neither is set, and nothing when the token is scoped to another
host. Each case sets and unsets the variables it uses, so one cannot leak into
the next.
Behaviour change: none. This is documentation and test coverage only; no runtime
code changed.
* vpm: make the registry resolver's signature match what it can return
`resolve_registry_module` returns `?ModuleVpmInfo`, but every path returns
`none`: a `RegistryModule` carries no repository url and no vcs, and
downloading an archive is not implemented, so the function can only record
why it cannot supply the module. The `if mod := ... { return mod }` at its
one call site documented an outcome that never happened.
Change the return type to `void` so the signature says what is true. A
reader should not have to prove an optional can hold a value by reading
every branch, and an optional that can never be `some` reads as an
invitation to write the missing half.
No behaviour change. The registry is still consulted before the public
servers, still records why it could not supply the module, and the install
still goes on to the next candidate when every one has missed.
* vpm: confine registry tokens and serve valid SPDX archive links
---------
Co-authored-by: Alexander Medvednikov <alexander@medvednikov.com>
Co-authored-by: MR <mr@example.com>
