mirror of
https://github.com/containerd/containerd.git
synced 2026-10-29 01:55:16 +08:00
Nahum Litvin
a9d5caf7fc
The snapshot garbage collector holds the snapshotter write lock while calling the underlying Snapshotter.Remove. cleanupSnapshotter also detaches cancellation (context.WithoutCancel), so a Remove that never returns - e.g. a proxy snapshotter whose RPC connection has wedged - holds the write lock forever. Every subsequent snapshot operation on the node then blocks behind the lock, including CRI RunPodSandbox, turning the node into a permanent scheduling black hole while it still reports Ready. Observed in production with a proxy snapshotter (soci) that stopped answering after a file-descriptor exhaustion storm: the GC goroutine sat in WaitGroup.Wait for 4.5 hours with the Remove RPC parked in waitOnHeader, and 17 CRI operations queued behind the metadata lock. Bound each GC-path Remove with a timeout registered via pkg/timeout (io.containerd.timeout.gc.snapshotter.remove, default 30m, operator-tunable through config.toml like the other timeout keys). A timed-out Remove returns DeadlineExceeded, which is not a FailedPrecondition, so it propagates out of pruneBranch and ends the pass: a globally wedged snapshotter costs one timeout per GC cycle rather than one per orphaned snapshot, the deferred Cleanup is skipped because the pass failed, and the snapshots left behind stay orphaned for the next pass. Signed-off-by: Nahum Litvin <nahuml@wix.com>