Files
go-redis/csc_applyfast_test.go
Nedyalko Dyakov 58cc03e024 perf(csc)!: mark reads with a second-chance bit (#4034)
* feat(csc)!: always refresh invalidated entries

ClientSideCacheRefreshRecencyWindow bounded refresh to entries read
within a window. It was the only consumer that needed a per-READ
recency timestamp on the cache entry, and that timestamp is the
dominant cost of a cache hit.

Removing the option lets the read path stop stamping entries, which the
following commit does. Refresh now covers every invalidated entry.

BREAKING CHANGE: Options.ClientSideCacheRefreshRecencyWindow is removed.
Refresh-on-invalidate always refreshes every invalidated entry.

* perf(csc): mark reads with a second-chance bit

A cache hit stamped entry.lastAccessNs. The read already touches the
entry's cache line, but a store dirties it, so the line must be held
exclusive and costs a writeback when evicted from the CPU cache. With
many concurrent readers over a working set larger than L2, that store
dominates the cost of a hit.

A read now sets a readSinceSweep bit instead. A load that finds it set
leaves the line SHARED, so every core can hold it at once and only the
first read after an eviction sweep writes. Eviction consumes the bit as
a second chance; ordering still uses lastAccessNs, written on insert.

Measured at 256 concurrent callers, get=90/set=10, 99.94% hit rate:
263,801 -> 328,842 reads/s (+24.7%), CPU 27.0 -> 18.9 us/op, hit rate
unchanged. Removing the recency update entirely measured +40.5%, so
this recovers about three fifths of the headroom while keeping
eviction honest.

* perf(csc): decode simple cached replies without a reader

Serving a hit allocated a proto.Reader (and its bufio buffer) per hit
and re-parsed the cached bytes. applyCachedFast decodes the simple
single-frame replies -- nil, status, bulk string -- straight into the
Cmder, and getShared serves the bytes without the defensive copy where
the caller provably cannot retain them.

* fix(csc): do not evict the entry just fulfilled

fulfill publishes an entry with its second-chance bit clear and then
evicts. In a warm memory-capped shard, where every resident entry has
been read, that entry was the only clear-bit candidate and was evicted
on the spot; and since a clear candidate always existed, the sweep that
resets the others never ran, so the shard admitted nothing new. The
entry just fulfilled is now skipped in its own eviction pass.

A refresh republish also restores the invalidated entry's read bit with
its access token, so a refreshed hot key is not the first victim.

* fix(csc): decline negative bulk lengths other than -1

The fast cached-reply decoder took any negative bulk length as a RESP2
null. The reader rejects lengths below -1, and that rejection is what
makes processCached drop a bad entry and refetch; the fast path turned
it into a false miss. It now handles only -1 and declines the rest.

* fix(csc): reject bare LF in cached status replies

The fast cached-reply decoder rejected a status frame with an interior
CR but took one with a bare LF, such as "+OK\nMORE\r\n", as a valid
string. The reader stops at the first LF and rejects the frame, and
that rejection is what makes processCached drop the entry and refetch.
The fast path now declines an interior CR or LF.

* fix(csc): keep refresh recency in the publish lock

A refresh republish published the entry with its second-chance bit
clear and restored the bit in a second lock. On a warm shard the
republished entry was then the only clear-bit victim, so an insert in
that gap evicted the key the refresh had just fetched. The refresher
now stages the kept recency on its reservation (stageRefreshAccess),
and fulfill applies it in the same lock as the publish.

* docs(csc)!: refresh-on-invalidate refreshes all

ClientSideCacheRefreshOnInvalidate still said it re-fetches recently
read keys. Since the recency window was removed, it re-fetches every
cached entry of an invalidated key, so on a write-heavy keyspace the
refetch traffic covers the whole resident cache.

BREAKING CHANGE: Options.ClientSideCacheRefreshRecencyWindow and
UniversalOptions.ClientSideCacheRefreshRecencyWindow are removed. They
shipped only in v9.23.0-beta.1; delete the field where it is set.

* docs(csc): note reads are not monotonic per caller

A read that loses the reservation twice reads the server directly.
Another caller's fetch, reserved earlier and still running, can store
an older value afterwards, and a later hit returns it until the key's
invalidation arrives. This is within the staleness a hit already has,
so document it rather than drop the older fetch, which would cost its
waiters a miss each under the churn that leads here.
2026-10-01 14:36:58 +03:00

173 lines
5.2 KiB
Go

package redis
import (
"bytes"
"context"
"fmt"
"strings"
"testing"
"github.com/redis/go-redis/v9/internal/proto"
)
// applyCachedFast decodes the simple reply shapes without allocating a
// proto.Reader. It is only safe if it agrees with the generic path on every
// input, including the ones it declines: a fast path that silently decodes
// something differently from the reader would corrupt a cached read.
//
// This drives both paths over the same payloads and compares value and error.
func TestApplyCachedFastMatchesGenericReader(t *testing.T) {
ctx := context.Background()
big := strings.Repeat("x", 600)
cases := []struct {
name string
raw string
}{
{"empty bulk", "$0\r\n\r\n"},
{"short bulk", "$5\r\nhello\r\n"},
{"bulk with CR inside", "$4\r\na\rb\r\n"},
{"bulk with LF inside", "$4\r\na\nb\r\n"},
{"bulk with CRLF inside", "$6\r\na\r\nbc\r\n"},
{"binary bulk", "$3\r\n\x00\x01\x02\r\n"},
{"large bulk", fmt.Sprintf("$%d\r\n%s\r\n", len(big), big)},
{"resp2 null bulk", "$-1\r\n"},
{"resp3 null", "_\r\n"},
{"status ok", "+OK\r\n"},
{"status with space", "+string\r\n"},
{"empty status", "+\r\n"},
// Shapes the fast path must DECLINE rather than misread.
{"integer", ":42\r\n"},
{"array", "*2\r\n$1\r\na\r\n$1\r\nb\r\n"},
{"resp3 double", ",3.14\r\n"},
{"resp3 bool", "#t\r\n"},
{"verbatim", "=9\r\ntxt:hello\r\n"},
{"error", "-ERR nope\r\n"},
// Malformed / truncated: must not panic, must not disagree.
{"truncated bulk", "$5\r\nhel\r\n"},
{"bulk length lies long", "$99\r\nhello\r\n"},
{"no terminator", "$5\r\nhello"},
{"bad length", "$abc\r\nhello\r\n"},
// A bare LF ends the reader's line early, so these are invalid.
{"status with bare LF", "+OK\nMORE\r\n"},
{"status that is only a bare LF", "+\n\r\n"},
// Only -1 is the RESP2 null; the reader rejects any other negative.
{"negative bulk length", "$-2\r\n"},
{"very negative bulk length", "$-100\r\n"},
{"just crlf", "\r\n"},
{"empty", ""},
}
for _, tc := range cases {
for _, kind := range []string{"string", "status"} {
t.Run(tc.name+"/"+kind, func(t *testing.T) {
raw := []byte(tc.raw)
mk := func() Cmder {
if kind == "string" {
return NewStringCmd(ctx, "get", "k")
}
return NewStatusCmd(ctx, "type", "k")
}
// Generic path, verbatim from applyCachedReply's fallback.
gen := mk()
genErr := gen.readReply(proto.NewReaderSize(bytes.NewReader(raw), len(raw)+1))
// Fast path via applyCachedReply (falls back internally when it
// declines, so this is what production actually runs).
fast := mk()
fastErr := applyCachedReply(fast, raw)
sameErr := (genErr == nil) == (fastErr == nil)
if sameErr && genErr != nil {
// Both non-nil: Nil must match Nil exactly; other errors
// only have to agree on being errors, since the generic
// reader's messages are its own.
sameErr = (genErr == Nil) == (fastErr == Nil)
}
if !sameErr {
t.Fatalf("error mismatch: generic=%v fast=%v", genErr, fastErr)
}
var genVal, fastVal string
switch c := gen.(type) {
case *StringCmd:
genVal = c.Val()
fastVal = fast.(*StringCmd).Val()
case *StatusCmd:
genVal = c.Val()
fastVal = fast.(*StatusCmd).Val()
}
if genVal != fastVal {
t.Fatalf("value mismatch: generic=%q fast=%q", genVal, fastVal)
}
})
}
}
}
// The fast path must not alias the cache entry's bytes into the command: the
// slice it is handed belongs to the cache (see LocalCache.getShared), so a
// decoded value that shared its backing array would change under the caller
// the next time that entry was refetched.
func TestApplyCachedFastCopiesTheValue(t *testing.T) {
ctx := context.Background()
raw := []byte("$5\r\nhello\r\n")
cmd := NewStringCmd(ctx, "get", "k")
if err := applyCachedReply(cmd, raw); err != nil {
t.Fatalf("applyCachedReply: %v", err)
}
if got := cmd.Val(); got != "hello" {
t.Fatalf("value = %q, want hello", got)
}
// Scribble over the source the way a refetch writing into a reused buffer
// would. The decoded string must be unaffected.
for i := range raw {
raw[i] = 'Z'
}
if got := cmd.Val(); got != "hello" {
t.Fatalf("decoded value aliases the source: became %q after overwrite", got)
}
}
// getShared hands out the entry's own slice. Prove it really is shared (that
// is the point -- no copy) and that Get still copies, so the public contract
// is unchanged for third-party callers.
func TestGetSharedAliasesAndGetCopies(t *testing.T) {
ctx := context.Background()
lc := NewLocalCache(CacheConfig{MaxEntries: 8})
tok, fetch := lc.Reserve("ck", []string{"rk"})
if tok == 0 || !fetch {
t.Fatal("Reserve did not hand out a token")
}
if !lc.fulfill("ck", tok, 0, []byte("payload")) {
t.Fatal("fulfill failed")
}
a, ok := lc.getShared(ctx, "ck")
if !ok {
t.Fatal("getShared missed")
}
b, ok := lc.getShared(ctx, "ck")
if !ok {
t.Fatal("second getShared missed")
}
if &a[0] != &b[0] {
t.Fatal("getShared copied; it must alias the entry")
}
c, ok := lc.Get(ctx, "ck")
if !ok {
t.Fatal("Get missed")
}
if &c[0] == &a[0] {
t.Fatal("Get aliased the entry; the public contract must keep copying")
}
if string(c) != "payload" {
t.Fatalf("Get returned %q", c)
}
}