Neither doc previously said how much of a claimed transform chain is
actually honored; a reader could assume claiming any transform in a
chain was sufficient on its own. State the suffix rule plainly, with
the format/mkdir/overlay example the code itself now enforces, and
call out that format in particular can only ever be claimed as part
of a suffix that covers what follows it, never alone, since it
resolves mount points internal to the mount manager.
Signed-off-by: Derek McGowan <derek@mcg.dev>
Add a registry entry for the containerd.types.MountCapabilities extension,
and update the mount and bootstrap protocol docs for extensions replacing
the reserved-for-future-use capabilities field, and for the removal of the
runtime-allow-mounts annotation and the "/*" transform convention.
Signed-off-by: Derek McGowan <derek@mcg.dev>
The snapshot garbage collector holds the snapshotter write lock while
calling the underlying Snapshotter.Remove. cleanupSnapshotter also
detaches cancellation (context.WithoutCancel), so a Remove that never
returns - e.g. a proxy snapshotter whose RPC connection has wedged -
holds the write lock forever. Every subsequent snapshot operation on
the node then blocks behind the lock, including CRI RunPodSandbox,
turning the node into a permanent scheduling black hole while it still
reports Ready.
Observed in production with a proxy snapshotter (soci) that stopped
answering after a file-descriptor exhaustion storm: the GC goroutine
sat in WaitGroup.Wait for 4.5 hours with the Remove RPC parked in
waitOnHeader, and 17 CRI operations queued behind the metadata lock.
Bound each GC-path Remove with a timeout registered via pkg/timeout
(io.containerd.timeout.gc.snapshotter.remove, default 30m,
operator-tunable through config.toml like the other timeout keys). A
timed-out Remove returns DeadlineExceeded, which is not a
FailedPrecondition, so it propagates out of pruneBranch and ends the
pass: a globally wedged snapshotter costs one timeout per GC cycle
rather than one per orphaned snapshot, the deferred Cleanup is skipped
because the pass failed, and the snapshots left behind stay orphaned
for the next pass.
Signed-off-by: Nahum Litvin <nahuml@wix.com>
TaskService.Shutdown may be invoked multiple times. A grouped shim
itself does not terminate while it still has active tasks. It terminates only
after receiving a Shutdown request when no active tasks remain.
The shim manager should invoke TaskService.Shutdown after deleting every task,
regardless of whether the task represents sandbox or not. This ensures that the
shim receives Shutdown when the last active task is deleted.
In CRI, the pause container uses oom_score_adj=-998, making it unlikely to be
OOM-killed. However, it may still exit or be killed before the other containers
in the sandbox. SandboxService.Shutdown and TaskService.Shutdown are independent
APIs. Sandbox service should make sure there is no active containers
before shutdown. So, for the task service, we should invoke Shutdown for
each task.
Removing the supportSandboxAPIVersion check does not change behavior for shims
using task API versions below v3. Those shims were already forced down the
non-sandboxed path and received Shutdown after every task deletion. The behavior
change applies only to sandboxed tasks using task API v3, which now receive
Shutdown after deletion as well.
Signed-off-by: Wei Fu <fuweid89@gmail.com>
Describe layer_content_caches: what it replaces (a mkfs.erofs run per layer
on every node), how a hit is served, how multiple directories are searched,
and how to build a cache with ctr images build-erofs-cache. Call out that the
cached blob is symlinked rather than copied, so entries have to outlive the
snapshots referencing them, and that fsverity and IMMUTABLE_FL can't be used
alongside a cache.
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
Add proof-of-concept (PoC) execution requirements against a running
daemon to the triage guide and threat model.
Assisted-by: Antigravity
Signed-off-by: Samuel Karp <samuelkarp@google.com>
The debug server plugin returns ErrSkipPlugin when its address is empty
and the default configuration emitted by 'containerd config default'
sets the address to an empty string.
Assisted-by: Claude Code (Opus 4.8)
Signed-off-by: Samuel Karp <samuelkarp@google.com>
This commit introduces a comprehensive threat model, and security
triage guide, and operator guidelines based on the code, documentation
and a review of the published GitHub Security Advisories (GHSAs)
Assisted-by: gemini-cli
Assisted-by: Antigravity
Assisted-by: Claude Code (Opus 4.8)
Co-authored-by: Vinayak Goyal <vinayakankugoyal@gmail.com>
Signed-off-by: Samuel Karp <samuelkarp@google.com>
Add a new `enable_criu` configuration option under CRI plugin runtime
settings. When set to false, any checkpoint or restore request will fail
fast with an error indicating that CRIU support is disabled by
configuration. `enable_criu` currently defaults to true.
Add an integration test script to verify that setting `enable_criu` to
false in containerd configuration successfully disables checkpoint and
restore operations and fails fast.
Assisted-by: Antigravity
Signed-off-by: Samuel Karp <samuelkarp@google.com>
The RunPodSandbox unconditionally pre-pulls the pause container
image via ensurePauseImageExists() before starting any sandbox.
However, only the "podsandbox" controller actually uses the pause
image to create a pause container holding namespaces. Shim-based
sandbox controllers (e.g. Kata Containers) manage the sandbox
lifecycle entirely at the shim level and never reference the pause
image.
Add a DisablePauseImagePull flag to the Runtime config that gates
ensurePauseImageExists(). When a sandboxer is not "podsandbox", the
flag skips the unnecessary pre-pull, avoiding wasted network/storage
overhead and reducing sandbox startup latency.
The long-term direction is to offload image pulling entirely to the
controller implementation (shim level); this flag is an incremental
step toward that goal without introducing a breaking behavior change.
Also add unit tests to verify that ensurePauseImageExists is only
invoked for the "podsandbox" sandboxer and correctly skipped otherwise.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Document the new server plugin configuration blocks for GRPC, TTRPC,
debug, and metrics. Mark the legacy top-level sections as deprecated.
Note that in version 4, the TTRPC plugin is configured independently
from GRPC and uses its own defaults when its plugin block is omitted.
Signed-off-by: Derek McGowan <derek@mcg.dev>
This disables the slow_chown feature (nobody in their right mind
is going to be choosing erofs and want to slowly chown each file),
indicates that we support idmaps if the kernel supports it, and makes
sure to chown the upperdir.
This is more or less exactly how the overlay snapshotter does things,
minus the slow_chown part (which has discussions about dropping
altogether at some point anyways).
Signed-off-by: Andrew Halaney <ahalaney@netflix.com>
EROFS has supported a tiny metadata-only image to reference external
blobs since Linux 5.16. This eliminates the need to mount each EROFS
layer one by one and is also useful for VM-based containers (e.g.
nerdbox and Kata containers.)
Similar to LCOW/CimFS, `snapshots.UnpackKeyPrefix` is used to
trigger fsmerge generation (typically < 100 ms) on demand in Prepare().
In the future, we can also generate fsmeta in Commit() of the final
unpacking layer (by introducing an annotation to keep the chainID).
However, in the case of intermediate layer reuse, the Prepare() handling
will still be required.
```toml
[plugins."io.containerd.snapshotter.v1.erofs"]
max_unmerged_layers = 1 # enable fsmerge if image layers >= 2
```
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
adds a background stats collector that calculates `UsageNanoCores` for containers and pod sandboxes.
- run in the background every second to collect CPU metrics for all containers and sandboxes (similar to what cAdvisor does)
- keep a rolling buffer of CPU samples and calculates the instantaneous CPU usage rate from consecutive samples
- read pod-level CPU stats from the parent cgroup rather than the pause container
- add cgroupv2 Pressure Stall Information for CPU, memory, and IO
- add missing `Timestamp` and `Interfaces` fields
when Kubernetes runs with `PodAndContainerStatsFromCRI=true`, it expects `UsageNanoCores` to be set in stats responses.
This value represents how much CPU is being used right now (as opposed to `UsageCoreNanoSeconds` which is cumulative).
To calculate it, we need to compare CPU samples over time to replicate what is in cadvisor.
we can't yet really test this in CI as some changes in kubernetes has to land for `--feature-gates=PodAndContainerStatsFromCRI=true`
Signed-off-by: Davanum Srinivas <davanum@gmail.com>
The otelgrpc.UnaryClientInterceptor and otelgrpc.StreamClientInterceptor
options were deprecated and removed in favor of NewClientHandler.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Added v0.1.0 plugin support to the list of deprecated features in
RELEASES.md. Added a chapter about how to enable and configure the
default validator plugin in NRI.md
Signed-off-by: Krisztian Litkey <krisztian.litkey@intel.com>
Enabling the IMMUTABLE_FL file attribute causes dirty data to be
flushed synchronously at least on EXT4, which can greatly impact
container launch performance. In contrast, the overlayfs snapshotter
does not use syncfs by default.
Most users may not need IMMUTABLE_FL, let's make IMMUTABLE_FL optional
to align with the behavior of the overlayfs snapshotter and recover the
original performance.
1. tensorflow
Test commands:
$ nerdctl image pull --snapshotter=X --unpack="false" tensorflow/tensorflow:2.19.0
$ time nerdctl container --snapshotter=X run -d tensorflow/tensorflow:2.19.0 /bin/sh
Results:
overlayfs | 0m18.748s
erofs (no IMMUTABLE_FL) | 0m10.090s
erofs (with IMMUTABLE_FL) | 0m21.074s
2. ubuntu 22.04
Test commands:
$ nerdctl image pull --snapshotter=X --unpack="false" ubuntu:22.04
$ time nerdctl container --snapshotter=X run -d ubuntu:22.04 /bin/sh
Results:
overlayfs | 0m1.147s
erofs (no IMMUTABLE_FL) | 0m0.795s
erofs (with IMMUTABLE_FL) | 0m1.094s
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Derive filesystem UUIDs (`lsblk -o +UUID`) from the OCI layer digests
(although diffIDs are better in principle, but they're unavailable by
differs in advance) rather than generating a random one. This allows
EROFS to uniquely identify each layer using the content-addressable
filesystem UUID.
It can also be used for reproducible builds. To achieve this, configure
`mkfs_options` with `-T0 --mkfs-time` (However, `--mkfs-time` requires
erofs-utils 1.8+; Otherwise, all inode timestamps will be reset w/o it):
``` toml
[plugins."io.containerd.differ.v1.erofs"]
mkfs_options = ["-T0 --mkfs-time"]
```
Fixes: c73c8e5d52 ("Introduce EROFS differ")
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Derek McGowan