mirror of
https://github.com/redis/go-redis.git
synced 2026-10-28 16:35:06 +08:00
feat(autopipeline): automatic command pipelining (#3942)
* ci: run PR workflows for feature/** base branches PRs stacked against feature/autopipeline/integration (and any feature/** integration branch) didn't trigger build, e2e, govulncheck or codeql because their pull_request.branches filters only matched master/v9/ndyakov/**/etc. A pull_request event filter matches the BASE branch, so PRs targeting a feature branch were skipped. Add 'feature/**' to those filters. * feat(autopipeline): #1 base types and client wiring (#3864) * perf(proto): read payload+CRLF in one shot in ReadStringInto When the caller-supplied buffer has room for the payload plus the trailing CRLF, read both in a single io.ReadFull instead of a separate line read. This is the zero-copy fast path used by GetToBuffer. * feat(autopipeline): add AutoPipeliner with deferred typed results Introduce AutoPipeliner, a background batcher that coalesces concurrent commands from many goroutines into Redis pipelines for high throughput. Engine (autopipeline.go): - Sharded command queue (per-shard mutex + flusher + FIFO semaphore), round-robin enqueue to cut contention under concurrency. - Batch completion signalled by a single close(batch.done) instead of per-command channel sends. - AutoPipelineConfig{MaxBatchSize, MaxConcurrentBatches, Unordered, MaxFlushDelay, AdaptiveDelay}. Default is ordered (MaxConcurrentBatches=1). Validate() rejects MaxConcurrentBatches>1 without Unordered:true so callers can't silently lose ordering; NewAutoPipeliner panics on an invalid config. Deferred results (command.go): - baseCmd gains a ready channel + setReady/await/rawErr. Every exported value-accessor (Val/Result/Text/Int/Bytes/Bool/Float/Time/Scan/First/Item/ ...) awaits the result before reading cmd.val/cmd.err; delegating accessors inherit the await through the method they call. Execution-path reads use rawErr() to avoid awaiting the very batch they produce (self-deadlock). - Cmder interface extended with setReady/await/rawErr. Pooling (pipeline.go) reuses Pipeline/cmd-slice objects across batches. osscluster.go and redis.go use rawErr() on the pipeline execution path. * feat(autopipeline): return an error from constructors instead of panicking newAutoPipeliner / NewAutoPipeliner validated the config and panicked on a bad one. Since these run from the post-init AutoPipeline / AsyncAutoPipeline calls (not client construction), a panic is the wrong failure mode — a deterministic config mistake should surface as a returned error the caller can handle. Return (*AutoPipeliner, error); the config is rejected before any flusher goroutine starts, so the error path leaks nothing. * fix(autopipeline): don't cap the shutdown flush with a hardcoded 2s timeout flushBatchSliceShutdown wrapped the final drain in context.WithTimeout(2s). Because the effective wire deadline is min(ctx deadline, conn timeout), that 2s capped everything — including a relaxed read/write timeout set by maintnotifications during a failover/migration. In-flight commands on shutdown would then time out at 2s instead of honoring the relaxed window the relaxation was designed to protect. ap.ctx is already cancelled at this point, so use a plain context.Background() with no artificial deadline, matching the normal flush path and a plain client Exec: the connection's ReadTimeout/WriteTimeout (and any relaxed override) governs the wire deadline. A user who wants shutdown bounded sets those, as for any command. * fix(autopipeline): correct permit-wait timeout and error in flushBatchSlice The flush waited a hardcoded 5s for a concurrency permit and, on any non-nil result, errored every command in the batch with ErrClosed. Two problems: - ErrClosed is wrong when the wait simply timed out — the client is not closed. Callers saw a misleading closed error for a deadline. - 5s is below a maintnotifications relaxed window, so a legitimately slow in-flight batch (e.g. during a failover) holding the permits could make waiters fail spuriously. Distinguish a real shutdown (ap.ctx cancelled -> ErrClosed) from the backstop firing (-> the timeout error as-is), and raise the backstop to 30s — well above the default ReadTimeout and a relaxed window. It remains only a safety net: every permit holder releases via defer and each batch Exec is bounded by the conn timeout, and the wait still ends immediately when Close cancels ap.ctx. * docs(autopipeline): fix usage example to the error-returning API The AutoPipeliner type doc showed 'ap := client.AutoPipeline()', the old single-return form. AutoPipeline / AsyncAutoPipeline now return (*AutoPipeliner, error); update the example to assign and check err so the base reflects the final API surface. * refactor(autopipeline): drop orphan AutoPipeline() accessor stubs (*Pipeline).AutoPipeline() returned a silent nil and (*AutoPipeliner). AutoPipeline() returned itself, with a comment claiming they satisfy the Cmdable interface. They don't — neither Cmdable nor Pipeliner declares AutoPipeline(), so these methods satisfy nothing and the nil-returning Pipeline stub is a latent nil-deref footgun. The real entry points are Client/ClusterClient.AutoPipeline(config...) (*AutoPipeliner, error). Remove both stubs; the var _ Cmdable assertion still holds. * fix(autopipeline): Process honors mode; unexport constructor; doc fixes - Process() now routes through ap.cmdable like Do and the typed methods, so on a blocking autopipeliner it blocks instead of always running async. Previously it hardcoded processAsync, diverging from every other entry point. - Unexport NewAutoPipeliner -> newAutoPipeliner. The exported form took an unexported cmdableClient param and skipped cluster slot-sharding, silently degrading a *ClusterClient to cross-node batches. The public entry points (Client/ClusterClient.AutoPipeline/AsyncAutoPipeline) wire sharding correctly. - Fix stale docs: Do no longer claims it always 'returns immediately' (it blocks in blocking mode); TxPipeline/TxPipelined drop the false 'will panic for other clients' note (cmdableClient embeds Cmdable, so TxPipeline always exists). * ci: run PR workflows for feature/** base branches PRs stacked against a feature/** integration branch (e.g. feature/autopipeline/integration) didn't trigger build, e2e, govulncheck or codeql: a pull_request event filter matches the BASE branch, and those filters only listed master/v9/ndyakov/** etc. Add 'feature/**' so stacked PRs run CI. * fix(json): avoid autopipeline deadlock and stale reads in JSON commands JSONCmd/JSONSliceCmd readReply called baseCmd.Err(), which now awaits the autopipeline batch's done channel. readReply runs inside that batch's Exec, before done is closed, so a deferred JSON command would deadlock. Use rawErr() there instead. Also add the missing await() to JSONCmd.Expanded and JSONSliceCmd.Val/Result so deferred callers block for the result instead of reading empty/stale data. Reported by Cursor Bugbot on PR #3864. * ci: exempt autopipeline engine files from the unused linter The autopipeline stack defines its engine on lower branches and wires the callers in on top, so those symbols read as unused until the stack is assembled, failing golangci-lint on the base/pool PRs. Scope the unused exemption to autopipeline.go and pipeline.go only; unused stays enforced everywhere else. * docs(autopipeline): MaxBatchSize default is 200, not 100 DefaultAutoPipelineConfig and the zero-value fill both use 200; the field doc said 100. Reported by Copilot on PR #3865. * fix(search): await deferred FTHybrid cursor/raw accessors; clarify MaxBatchSize - FTHybridCmd.CursorResult/CursorVal/RawVal/RawResult read their fields without await(), unlike Val/Result. On a deferred autopipeline that returns empty/stale data before the command executes. Add await() to match. (readReply sets the fields directly, so this can't re-introduce the batch-Exec deadlock.) - Clarify the MaxBatchSize doc: it is a soft accumulation threshold, not a hard cap; under burst the batch can exceed it and execute as one larger (safe) pipeline. Reported by Cursor Bugbot on PR #3864. * fix(autopipeline): drain shards in Close after flushers exit A command could pass enqueue's under-lock closed-recheck moments before Close's CompareAndSwap, then append to a shard AFTER that shard's flusher had already drained and exited — leaving its batch.done unclosed, so the caller's blocking accessor hung forever (lost command). Close now drains every shard once more after wg.Wait() (flushers provably gone). The drain loop checks queue emptiness UNDER s.mu instead of via the unlocked s.Len(), so it cannot miss a command appended concurrently. This pairs with the closed-recheck in enqueue: s.mu serializes them, so either the late enqueue appends first and the sweep flushes it, or the sweep empties first and the enqueue then sees closed==true and rejects with ErrClosed. Fixes the hang in TestAutoPipelineCloseRace. * address pr comments * search needs to wait on raw result * feat(autopipeline): #2 add pipeline pool (#3865) * perf(autopipeline): flush once the queue stops growing on the default window accumulateBatch waited the full defaultAccumulateWindow (200µs) whenever the queue was below MaxBatchSize, even for a single blocking caller with nothing left to coalesce, so every low-concurrency command paid the whole window. The doc described a "stops growing" early exit that was never implemented. Implement it for the implicit default window only: a burst timer fires when no new command arrives for defaultAccumulateGap (20µs); each enqueue resets it, so concurrent load keeps coalescing up to the window cap while a lone caller flushes ~one gap after enqueuing. An explicit MaxFlushDelay / AdaptiveDelay is an intentional accumulation window and is still waited in full. defaultAccumulateWindow becomes a var so tests can enlarge it deterministically. * feat(autopipeline): default the blocking face to a single ordered batch stream DefaultBlockingAutoPipelineConfig used MaxConcurrentBatches:50 "for throughput", but that is pessimal. With many flush permits, each command finds a free permit and flushes on its own before others accumulate, collapsing batch size toward one command per round-trip — i.e. non-pipelined throughput at high latency. Backpressure from a single in-flight batch (MaxConcurrentBatches:1) instead lets callers whose commands return re-enqueue and flush together as the next batch, so batches stay deep (a near-continuous, double-buffered pipeline). Measured (get=80/set=20, 64B, localhost, 128 blocking workers): before (b=50): ~59k ops/s, p50 2.9ms, avg batch 1.6 after (b=1): ~420k ops/s, p50 0.30ms, avg batch ~94 Higher throughput AND ~10x lower latency, and it is naturally ordered. The async default (DefaultAutoPipelineConfig) is already MaxConcurrentBatches:1; for maximum throughput use AsyncAutoPipeline with a window of in-flight commands (inflight>1). * feat(pool): add optional dedicated pipeline connection pool Add an opt-in separate connection pool for pipeline/autopipeline execution (PipelinePoolSize), so heavy pipelining doesn't starve the regular command pool. withPipelineConn draws from it when configured and falls back to the main pool otherwise, draining push notifications before returning a connection. Attach the existing per-pool listeners to the dedicated pool as well, so its connections get the same lifecycle handling as the main pool: - streaming credentials re-auth hook - maintnotifications handoff hook (MOVING/MIGRATING). Because that hook's failed-handoff removal target is its own single pool field, the pipeline pool gets an independent hook (Manager.InitPoolHookForPool) bound to it, rather than sharing the main pool's hook — otherwise a pipeline-pool connection that failed handoff would be removed from the wrong pool. The extra hook shares the Manager, so MOVING/MIGRATING tracking stays central, and is shut down and detached on Manager.Close. * docs(options): correct pipeline buffer recommendation to 64-128 KiB The recommended 512 KiB overstated the benefit. Benchmarks show pipeline throughput climbs from the 32 KiB default to ~64 KiB and then plateaus; beyond ~128 KiB there is no further gain and very large buffers (>=512 KiB) can regress throughput and waste memory. The only effect of the buffer is holding one batch's wire bytes so it flushes in a single syscall, so size it to roughly MaxBatchSize x average-command-bytes. Update the recommendation, example, and memory-impact figures accordingly. * feat(pool): register the dedicated pipeline pool with OTel The optional pipeline connection pool was created, used and closed but never registered with the OTel recorder, so its connection-count / pending-request metrics were invisible. Add a pipelinePool param to otel.RegisterPools / UnregisterPools (registered as a regular pool under a _pipeline name suffix) and pass c.pipelinePool from NewClient and the client teardown. nil when no pipeline pool is configured, so default clients are unaffected. * fix(pool): limiter on pipeline-pool path, hook-attach race, test addr Address Copilot review on PR #3865: - withPipelineConn now honors opt.Limiter (Allow() before acquiring, ReportResult() on every exit incl. the early init/re-acquire failures), mirroring getConn/releaseConn. Enabling the pipeline pool no longer silently bypasses throttling and failure reporting. - InitPoolHookForPool now holds hooksMu across the closed-check, append and AddPoolHook, closing a (construction-only) window where Close could tear the hook down between append and attach, leaking it. AddPoolHook is lock-free and never re-enters the manager, so this is deadlock-safe. - pipeline_buffer_test.go uses the shared redisAddr instead of a hardcoded localhost:6379 so it respects the env/CI Redis address. * perf(autopipeline): decouple shard count from MaxConcurrentBatches Shard count was derived as min(GOMAXPROCS, MaxConcurrentBatches, 16), which silently fragmented the command queue as soon as a user raised MaxConcurrentBatches for throughput: 16 shards each accumulate a sliver of the load, so batches collapse toward one command per flush and pipelining disappears. Measured on a 14-core host (get=80/set=20, 64B, 128 goroutines, async window 8): MaxConcurrentBatches=50 meant 16 shards and 413k ops/s at 1.4ms p50, while the identical permit budget on ONE shard reaches 1.75M ops/s at 0.51ms p50 — the fragmentation, not the permit count, was the regression. Add AutoPipelineConfig.NumShards (0 = auto: one shard) and stop deriving the shard count from the permit budget. One deep queue plus 2-4 permits is the sweet spot: batch N+1 accumulates and executes while batch N's replies are in flight (async, window 8: 1.78M ops/s at 0.53ms p50 versus 1.0M at 1.0ms with a single permit). Every shard keeps at least one permit, so the effective global concurrency is max(NumShards, MaxConcurrentBatches). numAutoPipelineShards stays as the cluster-wiring default, where slot-routed shards keep each batch on one node. * perf(autopipeline): stripe the enqueue queue where ordering permits Profiling the windowed-GET throughput benchmark showed the shard enqueue mutex as the hottest lock in the engine: with 128 concurrent callers on one shard, lock/unlock slow paths (parking and waking contended waiters) accounted for roughly half of all CPU samples. Split the shard's queue into 8 stripes, each with its own mutex, queue, and batch-completion signal; enqueue picks a stripe round-robin, and the flusher drains every stripe into one merged pipeline so batches stay as deep as before. Striping applies where reordering is impossible or waived: unordered configs, and every blocking-face shard (a blocking caller waits for each command before issuing its next, so stripes cannot reorder its stream). The async ordered mode keeps a single stripe — with several stripes a caller's consecutive commands could land on opposite sides of an in-progress drain and execute out of order. Stripes are padded so neighbouring mutexes do not share a cache line. Measured (14-core, in-process throughput bench): windowed GET 5.2M -> 6.0M ops/sec; blocking mixed load 0.60M -> 0.77M ops/sec. External-process harness shows +2-7% with lower p99 (client CPU is not the bottleneck there). * fix(autopipeline): resolve pre-ship review findings in the engine Review of the feature (five independent passes; the two ordering defects were reproduced against a live server) surfaced correctness holes. Engine fixes: - Do executes on a NORMAL connection, outside the pipeline. Raw command names can carry connection state (SELECT/MULTI/SUBSCRIBE/CLIENT ...) or block the connection (BLPOP ...); batched onto a shared pipeline conn they silently poisoned it for every later batch, or stalled and retried unrelated commands. Do now carries plain Client.Do semantics (documented); both faces keep their call shape (blocking waits; async defers via a ready channel). Zero-arg Do reports a real error instead of ErrClosed. - Submit is rejected on the blocking face. Submit does not wait, so a windowed caller defeats the invariant the blocking face's enqueue striping relies on ("every caller waits per command") — reproduced as a deterministic reorder of 25% of adjacent submissions from a single goroutine. The deferred face is the intended home for windowed submission and is unchanged. - NumShards > 1 without Unordered fails construction on the deferred face: round-robin sharding flushes concurrently and does not preserve submit order (reproduced: 42% of windowed pairs reordered at NumShards=4). The blocking face is exempt (callers wait; Submit rejected), as is cluster slot sharding via the new unexported contentSharded marker (same-key commands share a shard, so per-key order holds). - Close no longer errors already-accepted commands: batch execution and permit waits run on a background context (wire timeouts still bound them; Close waits via wg/batchWg before pools tear down), and the shutdown drain now acquires the shard permit so a drained command cannot execute concurrently with — and be observed out of order against — a still-in-flight batch. - Validate rejects AdaptiveDelay with MaxFlushDelay <= 0, which silently disabled batch accumulation entirely while reading as a smart-tuning knob. - numAutoPipelineShards (the cluster wiring default) no longer derives from MaxConcurrentBatches: that coupling collapsed cluster slot routing to a single shard at the default budget, making per-node sharding dead code. - Minors: apStripe padded to a 128-byte stride (the 88-byte stride false-shared hot fields with the next stripe's mutex), 32-bit-safe unsigned modulo in the shard pick (int conversion panicked after counter wrap on GOARCH=386/arm), zero AutoFuture.Wait returns an error instead of nil-panicking, stale "maximizes latency" wording corrected. * fix(pool): pipeline pool survives clone; maintnotifications retires it too Two integration gaps from the pre-ship review: - baseClient.clone() dropped pipelinePool, so a WithTimeout clone silently fell back to the main pool: configured pipeline buffers and pool sizing evaporated with no error, and pipeline traffic polluted main-pool metrics. - The maintnotifications runtime downgrade retired tracked connections only through the primary pool; pipeline-pool connections (tracked identically via initConn) stayed in service with the feature enabled but no hook attached, silently dropping MOVING/MIGRATING pushes. Retire now offers the snapshot to every pool the manager attached a hook to — RetireConns skips connections a pool does not own, so this is safe by construction. * polish(autopipeline): pre-ship review polish — accessors, WaitContext, tidy From the deep review (improvements/docs/dead-code passes): - AutoFuture.WaitContext: ctx-aware wait (abandons the wait, never the command — per-command contexts are still not honored after enqueue, and now there is an escape hatch for callers that need one). - IsBlocking() and Config() accessors: code handed an *AutoPipeliner can tell which face it holds and what the effective (default-filled) config is, instead of probing Submit for an error. - Operator breadcrumb when the 30s permit backstop expires (both the flush and shutdown paths) — previously the one "should never happen" wedge case failed batches silently, with no internal.Logger call anywhere in the engine. - Static errors are errors.New sentinels (errSubmitBlockingFace, errZeroAutoFuture, errDoNoArgs) per house style. - Micro: putQueueSlice zeroes only the used prefix (tail invariant documented; was memclr'ing up to 8 KB per flush for small batches); flushBatchSlice skips provably-empty stripes without locking (safe only in that path — the wake protocol re-observes racing enqueues; the shutdown drain keeps locking). - Docs: the AutoPipeliner type doc's flush-trigger list rendered broken in godoc and implied a lone command never flushes — rewritten, plus a retry caveat (a batch is retried whole on a network error, so non-idempotent commands can execute twice on a dropped connection) and a pointer that Do runs outside the pipeline. Stale backstop comment (claimed Close cancels the permit wait; it deliberately does not) corrected. MaxFlushDelay doc no longer references an unexported identifier. - Dead code: commented-out process() stub removed; unused FIFOSemaphore methods removed (AcquireBlocking, Len, and Close — which would have panicked Release had anyone called it); defaultAccumulateGap is a const; closedBatch renamed completedBatch (it also signals successful direct execution, not just closure). * docs(autopipeline): WaitContext — result may become available later Spell out the post-deadline contract: a ctx error means the wait was abandoned, not the command — the batch is still in flight and may populate the command at any moment. Reading Cmd()'s value or error directly at that point races the executing batch; Wait again and only read after a non-context-error return. * perf(autopipeline): RTT-adaptive coalescing on the default window On a real network the fixed coalescing constants fragment batches. Measured against a Redis Enterprise QA endpoint (RTT ~52ms), 512 workers: when a batch's replies land, hundreds of goroutines wake over several milliseconds, but the flusher flushed 20µs after the first re-enqueues — the herd split into partial batches whose serialization costs a full RTT per split (blocking p50 174ms ≈ 3.3 RTT; manually setting MaxFlushDelay=5ms recovered 7.6k ops/s @ 67ms from 3.4k @ 174ms). On loopback the same constants are exactly right, so no fixed value serves both. The engine executes every batch, so it already knows the round trip: keep an EWMA (alpha 1/8) of batch execution time and derive the default-window timings from it — gap = clamp(RTT/64, 20µs, 3ms) window = clamp(RTT/4, 200µs, 15ms) Loopback RTTs clamp to the historical 20µs/200µs (behavior bit-identical); WAN RTTs widen both so reply-wakeup herds coalesce into one batch per round trip, automatically doing what a hand-tuned MaxFlushDelay did. Explicit MaxFlushDelay / AdaptiveDelay configurations are untouched. The derivation is a pure function (adaptiveCoalesce) for unit testing; EWMA updates are deliberately racy read-modify-writes (losing a sample is harmless, keeps the batch path CAS-free). * perf(autopipeline): merge reply cohorts after a blocked permit wait At MaxConcurrentBatches=1 the flusher takes the queue and THEN blocks on the permit for a full batch round trip. Callers whose replies land just after the take re-submit into the fresh queue during that wait; executing without them splits the herd into two alternating cohorts — each observing two round trips at half throughput — and the state is stable once entered. Measured on a 52ms link: p50 pinned at 120ms (2xRTT) for entire runs at 64 workers x inflight 10 while neighbouring worker counts ran at 1xRTT, and run-to-run bistability at other counts (whichever cohort structure forms at startup persists). After a blocked permit acquire on the default window, debounce the resubmission herd (accumulateBatch's RTT-adaptive gap) and fold the late arrivals into the already-taken batch before executing. The herd collapses back to one batch per round trip. TryAcquire successes (uncontended, or MaxConcurrentBatches>1 with free permits) skip the top-up entirely, and explicit MaxFlushDelay/AdaptiveDelay configs keep their own timing. * perf(autopipeline): replace debounce timer with cohort-tracked coalescing The default coalescing path armed a ~20us debounce timer on every flush. A Go timer that short fires ~1ms late on an idle or low-core host (wakeup latency dominates the requested delay), taxing every low-concurrency command ~5x its round trip: on an in-network endpoint (0.25ms RTT) a lone caller measured p50 1.31ms vs 0.24ms for a plain client. Replace wall-clock guessing with two facts the engine already has: - resubmitDebt: a completed batch of N commands wakes its N waiters together; in a closed loop they resubmit. Credit N at batch completion, debit 1 per enqueue, and hold the flusher until the debt drains - the herd flushes as one deep pipeline, exiting the moment it has landed. An exact per-herd count cannot ratchet into fragmentation the way a smoothed estimate does (an EWMA exit undershoots its own herd and locks into fragmented flushes; measured on a 57ms WAN link). - inFlight: executing batches mean more wakeups are coming. Hold stragglers while anything is in flight (gap-bounded, and below coalesceMinFlush hold them for the next herd rather than burning a pipeline connection on a near-empty flush); when nothing is in flight and no debt is outstanding, the shard is idle and flushes immediately with no timer armed. The silence-gap fallback (200us floor, execEWMA/8, 5ms ceil) only ends waits for callers that left; it never fires in steady state. Also dispatch the single-command fast path in its own goroutine: running Process inline blocked the flusher for a full round trip, and on a slow link a solo straggler then held a landed herd hostage for one RTT whose flush delayed the straggler again - a stable phase-lock at exactly 2x RTT (~25% of runs on a 57ms link). Measured (get=80/set=20, 64B, unordered, MaxConcurrentBatches 50): - loopback: w1 p50 0.156->0.130ms; w64 +16% tp; w256 +14%; w1024 even - in-network 2-vCPU (0.25ms RTT): w1 p50 1.31->0.235ms, 4.7x tp; w16 3.2x tp; w64 2.3x tp; w1024 +4%; w256 -20% (fast-link partial-batch overlap the old timer bought; accepted, fixing it re-fragments WAN) - WAN 57ms: w64/w256/w1024 within baseline variance, phase-lock gone The stops-growing/adaptive-window tests are superseded by autopipeline_coalesce_test.go; autopipeline_stops_growing_test.go and autopipeline_adaptive_rtt_test.go (pipeliner branch) test removed symbols and are replaced by it on rebase. * fix(autopipeline): bound the straggler hold; never erase a fresh herd's debt Two review findings on the cohort coalescing: - The gap-fire stale-debt reset was a blind Store(0) on the pipeliner-global counter. A batch completing on another shard (or racing the fire) between the flusher's last look and the reset would have its just-credited herd erased, fragmenting a herd that is really coming. Reset now requires the debt not to have grown since the timer was last armed, and uses CompareAndSwap so a credit racing the reset itself also survives. - The straggler hold (gap-fire with a near-empty queue while batches are in flight) re-armed unboundedly. With read timeouts disabled a wedged batch pins inFlight forever and up to coalesceMinFlush-1 queued commands would hang with it. The hold is now bounded by autoPipelinePermitBackstop, mirroring the permit wait's safety net. Also drop a stale test comment claiming Options.AutoPipelineConfig is not consulted by the blocking face (it is, when no config argument is given). * refactor(autopipeline): rename debt/herd vocabulary to expected arrivals/wave resubmitDebt reads as if something is owed; nothing is — the counter is an expectation: how many commands the engine knows are about to arrive because it just woke that many waiters. And 'herd' named the woken group by analogy rather than by mechanism. Rename for the next reader: resubmitDebt -> expectedArrivals (announce on completion, account coalesceCohort -> awaitExpectedArrivals per enqueue) cohortGap -> silenceGap (+Floor/Ceil consts) herdExpected -> expectingWave herd / cohort -> wave (comments; 'wave merge' at the permit site) Behavior unchanged; comments rewritten in the same vocabulary. * perf(autopipeline): pad stripes with x/sys/cpu.CacheLinePad The stripe pad was a hand-tuned [80]byte to reach a 128-byte stride. Use golang.org/x/sys/cpu.CacheLinePad instead: it is sized per GOARCH (64 B on x86-64/arm64, 128 B on ppc64, 256 B on s390x), so stripes land on their own cache line on every target rather than a constant that happens to fit the common ones and still false-shares on s390x. x/sys was already in the module graph (indirect); this promotes it to a direct dependency. A contended microbenchmark (8 goroutines, adjacent stripes) shows padded vs unpadded ~16x, and CacheLinePad vs the old [80]byte within noise on arm64 — so no regression, just arch-correct. * perf(autopipeline): lower silence-gap ceiling from 5ms to 2ms * chore(examples,extra): go mod tidy for golang.org/x/sys indirect dep * fix(autopipeline): filter the pipeline pool on Sentinel failover onFailover only filtered connPool, so after a Sentinel promotes a new master the dedicated pipeline pool kept idle connections dialed to the old master and pipelined traffic could keep using them. Filter pipelinePool with the same predicate. Also reword the TestPipelineBufferSizes comment: it verifies end-to-end pipelining with the buffer options set, not the actual socket buffer sizes (which are not observable here). * ci(autopipeline): govulncheck on go 1.26.5 (GO-2026-5856) The govulncheck workflow pinned go-version "1.26.x", which resolves to 1.26.4 and reports GO-2026-5856 (crypto/tls). Pin 1.26.5 (the fixed release), matching the pipeliner/benchmarks branches. * feat(autopipeline): #3 wire into Client and ClusterClient (#3867) * feat(autopipeline): wire AutoPipeliner into Client and ClusterClient Add Client.AutoPipeline() and ClusterClient.AutoPipeline() returning a deferred *AutoPipeliner, plus WithAutoPipeline() and propagation of the pipeline-pool options. Options gains AutoPipelineConfig. * test(autopipeline): cover correctness, ordering, config and cluster routing Standalone and cluster tests for the deferred AutoPipeline API: per-goroutine ordering, no cross-talk or lost commands, error isolation, config validation (panic on MaxConcurrentBatches>1 without Unordered), zero-copy GetToBuffer/SetFromBuffer, adaptive delay, and cluster cross-slot routing. * docs(autopipeline): add README section and 4-way decision-guide example README documents the deferred AutoPipeline API and the ordered-default / Unordered-for-parallel rule. example/autopipeline benchmarks four usage styles (normal blocking, ordered blocking-read, ordered read-later, unordered read-later) and prints a decision guide with throughput and ordering. * feat(autopipeline): AutoPipeline/AsyncAutoPipeline return an error The four AutoPipeline / AsyncAutoPipeline methods (Client and ClusterClient) now return (*AutoPipeliner, error) instead of *AutoPipeliner. These run from post-init calls, so an invalid config must surface as a returned error the caller can handle rather than a panic. On error no instance is cached, so a later call with a valid config still succeeds. Tests updated to the two-value form; the unsafe-config test now asserts an error instead of a panic. * test(autopipeline): drop test for removed AutoPipeline() self-accessor The (*AutoPipeliner).AutoPipeline() stub was removed (orphan, satisfied no interface); drop the test that asserted it returns itself. * test(autopipeline): use AsyncAutoPipeline after unexporting the constructor NewAutoPipeliner is now unexported; the unsafe-config test calls c.AsyncAutoPipeline (same error-on-bad-config contract). Update a stale doc comment reference too. * style: gofmt redis.go and osscluster.go Remove a stray double blank line after the AsyncAutoPipeline methods so golangci-lint's gofmt check passes. * ci(spellcheck): add autopipelining, goroutine(s) to the wordlist The README autopipeline section uses 'autopipelining' and 'goroutine(s)', which aren't in the spellcheck dictionary, failing check-spelling on the PR. Add them to .github/wordlist.txt. * test(autopipeline): fix CI failures on the pipeliner PR Three independent test-job failures, all in test code / the example module: - Blocking-face ginkgo specs panicked: ap.Do returns a generic *redis.Cmd, but the specs type-asserted *StringSliceCmd/*StringCmd. Use *redis.Cmd with its StringSlice()/Text() accessors. - Cluster autopipeline tests failed in the standalone test-ce job: the :7000 tests had no skip guard, and the :16600 tests guarded only on Ping (which passes against unjoined nodes that then return CLUSTERDOWN/MOVED). Add a shared skipIfClusterUnhealthy that gates on CLUSTER INFO cluster_state:ok, and use it in all cluster autopipeline tests. - example/autopipeline didn't compile (govulncheck 'all modules' load error): it called AutoPipeline()/AsyncAutoPipeline() in single-value context after they gained the error return. Assign (ap, err) and panic on err. * ci(spellcheck): add async, runnable to the wordlist The README autopipeline section uses 'async' and 'runnable' in prose; add them to .github/wordlist.txt. Verified the full README passes pyspelling via the exact CI image (jonasbn/github-action-spellcheck:0.62.0). * docs(readme): autopipeline examples use the (ap, err) return AutoPipeline()/AsyncAutoPipeline() return (*AutoPipeliner, error); the README snippets showed single-value calls that wouldn't compile. Capture and check err. Reported by Copilot on PR #3867. * test(autopipeline): cover stops-growing early exit and explicit-delay gating Two white-box tests for the default-window stops-growing behavior: - StopsGrowingDefaultWindow enlarges defaultAccumulateWindow to 1s and asserts a lone caller still flushes in well under it (deterministic, no sub-ms timing). - ExplicitDelayWaitsFullWindow guards the gating: an explicit MaxFlushDelay is still waited in full, so stops-growing never short-circuits a configured window. Config is passed as an argument to AutoPipeline (the blocking face does not read Options.AutoPipelineConfig). * docs(autopipeline): AutoPipeline default is a single ordered stream, not parallel batches * feat(autopipeline): cluster keeps multi-shard default; cover NumShards The engine now defaults to a single shard (deep queue) for standalone clients. Cluster clients still want several shards — commands are routed to shards by slot so each batch stays on one node — so the cluster wiring fills in the numAutoPipelineShards default when the user leaves NumShards at 0, copying the config rather than mutating the caller's struct. Tests cover: standalone defaulting to one shard regardless of the permit budget, an explicit NumShards override, negative-NumShards validation, and the cluster default (copy semantics + explicit passthrough). * test(autopipeline): adaptive-delay test pokes stripe 0 after enqueue striping The white-box adaptive-delay test set the shard's queueLen directly; that counter now lives on the shard's stripes. Point it at stripe 0 (these configs are ordered, so the shard has exactly one stripe). * fix(autopipeline): cluster review fixes — push drain, slot-shard default, docs Cluster-side resolutions from the pre-ship review: - ClusterClient.pipelineReadCmds drains buffered RESP3 push notifications before reading each reply, matching the standalone pipeline and cluster TxPipeline read loops. It was the only push-blind reader: a push frame (e.g. a maintnotifications MOVING notification) arriving mid-batch was consumed as a command's reply, shifting every subsequent reply in the pipeline by one command — and the autopipeliner routes all cluster traffic through this loop. - Cluster wiring uses the decoupled numAutoPipelineShards() default and marks the config contentSharded, restoring per-node slot sharding at the default configs (it had collapsed to one shard, making installAutoPipelineSharding dead code) while remaining exempt from the async NumShards ordering check — slot routing keeps every key on one shard, so per-key order holds. - Docs: ClusterClient.AutoPipeline's godoc was fused into the unexported clusterAutoPipelineConfig helper (the public method rendered undocumented); split and given a per-key ordering qualifier. Stale wording fixed in redis.go ("maximizes latency"), the example ("parallel-batch config"), and command.go (references to never-shipped FutureAutoPipeline/ChanAutoPipeline). - Tests: Submit rejected on the blocking face; NumShards>1 requires Unordered on the async face (blocking and cluster slot-sharded configs exempt); AdaptiveDelay without MaxFlushDelay rejected; Do bypasses batches (a MULTI via Do can no longer corrupt co-batched commands); cluster default config keeps multiple slot-routed shards; stops-growing tests use typed calls now that Do no longer enters the engine. * polish(autopipeline): unify getters, closed-client guard, config fallback, docs Wiring polish from the deep review: - The four AutoPipeline/AsyncAutoPipeline getters (Client and ClusterClient) shared a hand-rolled 15-line caching protocol each; they now delegate to one getOrCreateAutoPipeliner helper, which is also where the new closed-client guard lives: calling a getter after Close returns ErrClosed instead of silently building a fresh pipeliner against closed pools whose flusher goroutines nothing would ever stop. - Options.AutoPipelineConfig now feeds BOTH faces (it was silently ignored by the blocking AutoPipeline — a generically-named option honored by only one of two identically-shaped methods). Each face still falls back to its own default when the option is nil. FailoverOptions gains the AutoPipelineConfig field (failover clients could reach the feature but not configure it) and plumbs it to both the sentinel-backed client and the failover cluster client. - Docs: getter godocs state the Options fallback; ClusterClient.AsyncAutoPipeline documents per-key ordering across nodes; the README's stale "parallel pipelines" claim is corrected to the ordered-stream default, throughput figures are qualified, and the section gains the connection-count benefit plus a caveats paragraph (per-command contexts, blocking commands and Do bypass batching, whole-batch retry duplication, shared cached instance). Buffer-size option docs mention both pipeline entry points. - The scoped "unused" linter exemption for the stacked-PR period is removed: the wiring lands on this branch, so every engine symbol is referenced. Example build artifact gitignored. * docs(example): autopipeline example becomes a usage tour + benchmark The example was a throughput harness only — valuable, but it never showed how to actually use the API (nothing read a value back). It is now two acts: Act 1, a usage tour: the blocking face as a drop-in (goroutines Set+Get with plain-client call shape), the async face's submit-a-window-read-later pattern, Submit/AutoFuture for raw commands (async-only by design, with WaitContext noted), the Do escape hatch (runs on a normal connection outside the pipeline — plain Client.Do semantics), in-code tuning notes (Unordered + MaxConcurrentBatches 2-4, NumShards stays 0, first-call-config-wins caching, dedicated pipeline pool options), and an env-gated cluster tour (REDIS_CLUSTER_ADDRS) showing slot sharding needs no extra code. Act 2, the existing four-way throughput comparison, with review fixes: the unused rdb parameter dropped, the bench-tuned MaxConcurrentBatches: 80 magic number replaced by the recommended 2-4 (with a comment where big numbers belong), the baseline's PoolSize asymmetry explained where it stands, panic replaced by the file's print-and-exit convention, and the stale "parallel pipelines" package comment corrected to the ordered-stream default. Adds the README the example was missing (15 of 17 examples have one): run instructions, what each act shows, a sample results table, and the caveats users must know (per-command contexts, whole-batch retry duplication, per-key cluster ordering). * test(autopipeline): re-point tests that the Do change made vacuous AutoPipeliner.Do now executes on a normal connection outside the engine, so every test that drove ap.Do to exercise batching/flushing/queueing was passing vacuously against the plain-client path. Re-pointed at the engine: - ap.Do(ctx, "SET", ...) becomes the typed ap.Set(...) throughout the ginkgo suite, TestAutoPipelineBasic/MaxFlushDelay/Concurrency/SequentialSingleThread, the sequential tests, and the cluster tests (which now genuinely exercise slot sharding). - The error-isolation spec uses a typed WRONGTYPE failure inside a batch instead of Do("INVALID_COMMAND"), which no longer touches a batch at all. - The queue-length spec moves to the deferred face with a wide explicit window, making Len assertions deterministic and meaningful (they previously observed an untouched queue: always zero). - TestAutoPipelineSecondaryAccessors moves to the deferred face — the missing await() regression it guards is only expressible there (the blocking face waits inside the call, so it could never catch it). Stale reference to the removed autoPipelineCmd type dropped. - The sequential submit-then-read tests move to AsyncAutoPipeline (that is the deferred pattern they describe); TestOrderedModeWindowed likewise, making its "never blocks between submits" claim true. TestFutureFaceOrdering deleted — it duplicated TestBlockingFaceExecutesOnCall. Note: the ginkgo AutoPipeline specs require the full test rig (BeforeSuite spins up cluster/sentinel processes) and are validated in CI; everything reachable locally runs green under -race. * docs(readme): list Automatic pipelining under Features (experimental) Adds the feature-list entry linking to the section, marked experimental and scoped to its intended audience: high-throughput / high-load / scale use cases. The section itself now carries the same experimental note so readers landing via the anchor see it too. * docs(readme): point to the example app from the autopipelining intro The example pointer sat at the end of the caveats paragraph where nobody scrolling the section would find it; it now lives in the section's opening note (and the duplicate tail mention is removed). * test(autopipeline): pin the RTT-adaptive coalescing derivation Unit tests for adaptiveCoalesce (loopback clamps to the historical constants, WAN scales gap/window, ceilings bound the tax, enlarged-floor interaction with the stops-growing test hook) and the execEWMA smoothing behavior. * test(autopipeline): drop tests of removed coalescing internals autopipeline_stops_growing_test.go and autopipeline_adaptive_rtt_test.go exercised defaultAccumulateWindow / adaptiveCoalesce, which no longer exist; autopipeline_coalesce_test.go covers the replacement mechanism. * fix(autopipeline): non-pipelineable command fails itself, not the merged batch mapCmdsByNode returned a batch-wide error when any command's policy failed CanBeUsedInPipeline, so under the opt-in dynamic resolver one caller enqueueing e.g. a multi-shard MGET errored every unrelated caller co-batched in the same autopipeline flush (review finding M9). Set the error on the offending command and continue mapping the rest; the skipped command never joins a node pipeline, is not retried, and its waiter reads the policy error while everyone else proceeds. * docs(example): autopipeline example catches up with the engine - Do guidance: typed blocking commands (BLPop, XRead Block, Wait, ...) are diverted to a normal connection automatically — Do is not the escape hatch for them; and stop suggesting SELECT/SUBSCRIBE through Do (connection-state commands poison a pooled conn with plain Do semantics too — Options.DB and the PubSub API are the answers). - Tuning notes: extra MaxConcurrentBatches permits add overlapping batches bounded by the pipeline pool; with wave-boundary coalescing they no longer fragment batches. Reworded in main.go and the bench comment. - README: refresh sample numbers on the current engine (blocking face 552k -> 633k on the same loopback box) and note the pipeline-hook instrumentation shape in caveats. * ci(autopipeline): wordlist loopback/autopipeliner; go mod tidy example/autopipeline * chore: stop tracking .claude/worktrees gitlinks; gitignore them * perf(autopipeline): dispatch drained batches straight to processPipelineHook The flusher built a Pipeline per batch: one Pipeline alloc, a nil cmds slice growth-reallocating ~log2(N) times, and one interface Process call per command — then Exec handed the slice away, nothing reused. Extend cmdableClient with processPipelineHook (the exact hook-wrapped entry Pipeline.Exec wires, so hooks/OTel see the identical call) and hand the drained queue slices to it directly: zero-copy for single-stripe drains (every ordered shard), one pooled merge for striped drains. Shutdown flush uses the same path. Local A/B (M4 Max, redis 6379, count=6): B/op -12-14% on windowed workloads (p=0.002); blocking-face throughput +10% (1.09M -> 1.21M ops/s); windowed throughput unchanged (RTT-bound). Note: hooks now receive a pooled slice — retaining []Cmder past the hook call was never sanctioned; contents are zeroed on reuse. * perf(autopipeline): skip the shard round-robin atomic when there is one shard len(shards)==1 is the standalone default; every enqueue still bumped ap.next — a shared cache line — for a pick that is constant. Same guard the stripe pick already has. Also adds a goroutine-leak regression test (open/close many autopipelines, assert goroutines return to baseline). Local A/B: unordered dispatch ~-3% ns/op, windowed throughput +3-5%. * test(autopipeline): de-flake TestDoBypassesPipeline Force a real multi-command batch (MaxFlushDelay) so the batched SETs run on the dedicated pipeline pool, instead of racing the single-command fast path onto the MULTI-poisoned normal-pool connection. Fixes intermittent failures under -race and on loaded CI runners. * fix(autopipeline): surface retry-exhaustion errors and route streaming re-auth to the owning pool generalProcessPipeline only set command errors on the early-exit branch, not when the retry loop was exhausted. Manual Pipeline surfaces that via Exec's return value, but AutoPipeline discards the pipeline error and observes results only per-command, so a network failure that exhausted retries silently returned a nil error and zero values. Mirror the early-exit setCmdsErr on exhaustion, guarded by !isRedisError so a per-command redis error keeps its own reply. Streaming-credentials re-auth hardcoded c.connPool even for pipeline-pool connections, so a failed pipeline conn was closed and accounted against the main pool. Route the SingleConnPool base and the auth-error CloseConn to the owning pool via poolForConn (pool.Conn.PoolName). * test(autopipeline): retries, cluster MOVED, and pipeline-pool feature coverage - pipeline and autopipeline retry re-drive: success-after-retry, exhaustion surfaces the error, WRONGTYPE isolation, context-cancel not retried (scriptable bad-conn dialer, no external deps) - cluster MOVED redirect inherited by AutoPipeline via the shared execer, with a MaxRedirects=-1 anchor proving the redirect path is exercised - dedicated pipeline pool: ACL AUTH + SELECT handshake, StreamingCredentialsProvider re-auth, and maintnotifications all verified on pipeline-pool connections * feat(autopipeline): expose AutoPipeline on the UniversalClient interface Add AutoPipeline/AsyncAutoPipeline to the UniversalClient interface. *Client (including sentinel-backed failover clients) and *ClusterClient already implement them; *Ring returns errRingAutoPipelineUnsupported so it satisfies the interface and fails explicitly instead of being silently unavailable. * test(autopipeline): context contract, reply-shape demux, fuzz, backpressure, UniversalClient - AutoPipeline context contract: WaitContext abandons only the wait, caller ctx does not cancel execution, blocking commands honor ctx, submit-after-Close - heterogeneous reply-shape demux (bulk/array/int/map/nil/double/large/binary) under RESP3 and RESP2 - differential fuzz: identical results plain client vs autopipeline over random type-diverse command streams - blocking-command isolation, double-Close, windowed backpressure, bounded soak - AutoPipeline reachable via the UniversalClient interface (Ring returns unsupported) * chore(autopipeline): fix CI — remove unused putPipeliner, pin govulncheck to go 1.26.5 - lint: putPipeliner (pipeline.go) was documented but never wired to a caller, so golangci-lint flagged it (unused); remove it. - govulncheck: pin the scan's Go toolchain to 1.26.5, fixing the crypto/tls standard-library advisory GO-2026-5856 (was resolving to 1.26.4). * feat(autopipeline): AutoPipeliner implements UniversalClient (drop-in) Widen the internal cmdableClient to UniversalClient and delegate the non-batched surface (AddHook, Watch, Subscribe/PSubscribe/SSubscribe, PoolStats, Do, AutoPipeline/AsyncAutoPipeline) to the underlying client, so an *AutoPipeliner can be used anywhere a redis.UniversalClient is expected: data commands are batched, everything else runs on the underlying client. Narrow Do to return *Cmd to match the interface; drop the now-redundant .(*redis.Cmd) assertions in the tests. Close() still closes only the autopipeliner, not the underlying client. This lets tests (the command-suite matrix and maintnotifications e2e) parametrize a single UniversalClient subject that includes the autopipeliner. * refactor(autopipeline): take *AutoPipelineConfig (nil=default) not variadic Address review feedback (cxljs, #3867). The variadic `config ...*AutoPipelineConfig` reads like the functional-options pattern but isn't; a single `*AutoPipelineConfig` where nil selects the default is clearer. Changes AutoPipeline / AsyncAutoPipeline on *Client, *ClusterClient, *Ring, the UniversalClient interface and the AutoPipeliner drop-in, plus getOrCreateAutoPipeliner's override parameter. Call sites now pass nil instead of no argument. README, example and doc comments updated. * fix(autopipeline): PR #3867 review fixes - clone(): copy pipelinePoolName so timeout-derived clients keep the dedicated pipeline pool for re-auth routing (poolForConn) instead of falling back to the main pool. - clusterAutoPipelineConfig: mark contentSharded even when the caller sets an explicit NumShards. A cluster always routes by slot, so per-key order holds regardless of shard count; the async face was previously rejected. - TestFutureFaceTyped: build the deferred face with AsyncAutoPipeline so it actually exercises await()/deferred accessors. - Options docs (sentinel/ring/osscluster): correct the pipeline-pool enable condition — set by PipelineReadBufferSize/PipelineWriteBufferSize; PipelinePoolSize alone does not enable it. - README: define ctx in the AutoPipeline examples so they compile as-is. - Update the cluster shard-config test for the corrected contentSharded behavior. * refactor(autopipeline): AutoPipelineOptions + no-arg / WithOptions methods Rename the config type AutoPipelineConfig -> AutoPipelineOptions (matching the Options naming family: Options, ClusterOptions, ...) and split each face into a clean no-arg method plus an explicit variant: AutoPipeline() // Options.AutoPipelineOptions or default AutoPipelineWithOptions(*AutoPipelineOptions) AsyncAutoPipeline() AsyncAutoPipelineWithOptions(*AutoPipelineOptions) Drops the awkward AutoPipeline(nil) for the common default case while keeping an explicit per-instance override. Both faces remain independent (a client can hold a blocking and an async autopipeliner at once). Also renames Default{,Blocking}AutoPipelineConfig -> ...Options and the *Options AutoPipelineConfig fields -> AutoPipelineOptions. Applied across *Client, *ClusterClient, *Ring, the UniversalClient interface and the AutoPipeliner drop-in; call sites, README and example updated. * docs(autopipeline): reword no-arg -> no-argument (check-spelling) check-spelling flagged "arg" (tokenized from "no-arg") in the README. Use "no-argument" instead. * fix(autopipeline): hooks reading deferred results no longer deadlock A pipeline (or process) hook that reads a deferred command's result — cmd.Err()/Val()/Result() — blocked on the batch's done channel, which the dispatching goroutine closes only after the hook chain returns: a deterministic self-deadlock on the async face (#3867 review). Any hook that inspects per-command results inside the chain — a common pattern in custom logging/metrics instrumentation — triggers it. The blocking face was unaffected. Fix, leaving the execution path untouched: - Complete batches at the innermost exec seam. The async dispatch now runs the user-hook chain via withProcessPipelineHook around the base processPipeline, closing the batches (and announcing expected arrivals, preserving the wave-merge ordering) the moment replies are populated — still inside the chain, so a hook reading results after next() (the span-status/metrics pattern) finds them ready. The lone-command fast path and Do get the same treatment through withProcessHook. The flusher's deferred close stays as a panic backstop (apBatch.close is now idempotent). The blocking face keeps its previous flow unchanged. - Guard await() against self-deadlock for reads BEFORE next(): the batch records its dispatcher's goroutine id while inside the hook chain; a reader on that goroutine gets the not-yet-executed view — exactly what a plain pipeline hook sees before next() — instead of blocking on a channel only its own goroutine can close. The goroutine id is parsed only on the about-to-block slow path, never on the fast path. - Make cmd.ready an atomic pointer: processAsync wired it after the command was already published to the flusher, racing a hook's read. The atomic store keeps that race benign — a hook that wins sees nil and takes the non-blocking not-yet-executed view — with nothing added under the hot stripe enqueue lock. - Arm the goroutine-id guard only when user hooks are installed (hooksMixin.hookCount): hook-free clients never stamp the batch, so a blocked reader pays one atomic load, not a runtime.Stack parse. Benchmarked against the unfixed base (benchstat; sequential n=8 rounds plus drift-controlled interleaved runs for the sensitive benches): DispatchPath ordered/unordered/solo, Submit, FutureFace and Throughput are all statistically unchanged (~, p>0.05); allocs identical (4/op dispatch). An earlier iteration that put setReady under the hot stripe enqueue lock benched at -9% Submit ops/sec and was rejected; the shipped shape adds nothing under that lock and keeps the guard to one atomic load unless hooks are installed. Note: results become readable the moment execution completes, so a hook that mutates a command's result after next() is racy on the async face; hooks should treat results as read-only there. Regression tests cover before/after-next hooks across the batch, solo and Do dispatch paths plus the blocking face, race-enabled; the deadlock reproduces on the unfixed tree. * test(autopipeline): consolidate per-topic test files into two Collapse the 29 autopipeline_* / osscluster_autopipeline_* test files into: autopipeline_test.go all blackbox tests (package redis_test), including the cluster autopipeline tests autopipeline_internal_test.go whitebox tests (package redis): shard-count decoupling, coalescing, review-fix regressions Pure reorganization: no test bodies changed, only regrouped (matches the consolidation already done on the benchmarks branch). Counts preserved (68 Test funcs, 3 Describe suites). The pipeline_*_test.go files are left as-is: they cover the dedicated pipeline pool and plain-pipeline behavior, not the autopipeline engine. * fix(autopipeline): deep-review fixes — hook short-circuit, cluster guard, tests, docs Engine: - A hook that returns WITHOUT calling next (allowed by the Hook contract; circuit-breaker shape) used to wake every waiter with a silent zero value: the chain's error was discarded on all dispatch paths. The innermost now records execution; when a hook short-circuits, its error (or a dedicated sentinel) is set on every command before the batch completes. Covers both faces, the lone-command fast path, Do and the shutdown drain. - The async self-deadlock guard is now always armed on cluster clients: node clients carry their own hooks (OnNewNode + AddHook, the redisotel pattern) that hookCount() cannot see. Node-level hooks run on node-worker goroutines, which the goroutine-id guard cannot protect pre-next(); that limitation is documented (results are safe to read after next()). - AutoFuture.Wait/WaitContext get the same dispatcher-goroutine guard as the command accessors. - Lone-command dispatch releases its permit and inFlight slot before waking the caller, so a closed-loop lone caller's next command takes the immediate-flush path instead of arming the silence-gap wait. - AddHook mutates the hook slice under hooksMu, and the dispatch snapshots the slice under RLock (background flushers iterate it concurrently). Tests: - TestAutoPipelineConcurrency asserted a locally-incremented counter and could not fail; it now checks every command and verifies DBSize. - The autopipeline reply-demux test ran on the blocking face sequentially, so every command took the lone-command fast path and the batch demux it claims to cover never executed; it now uses the async face with a flush window so all nine heterogeneous commands land in one pipeline batch. - The three cluster autopipeline tests dialed :7000-:7002 and always skipped; they now use the test harness's :16600-:16602. - Plain go tests use apTestAddr() (REDIS_PORT-aware, :6379 default): the suite variable redisAddr defaults to :6380 and is only corrected inside the ginkgo BeforeSuite, which plain go tests may run without. - Cold-start latency tests warm up dial+handshake before timing; goroutine leak checks poll with a deadline instead of fixed post-GC sleeps; the flush-on-timer test waits on a completion signal instead of sleeping. - New coverage: hook short-circuit (both faces, batch+solo), hooks added after the autopipeliner exists (client and delegate AddHook), Close flushing pending deferred futures, client-level Options.AutoPipelineOptions taking effect on both faces, BLPOP on the async face (readTimeout diversion), and AdaptiveDelay end-to-end traffic. API/docs: - UniversalOptions gains AutoPipelineOptions, threaded through Simple/ Failover/Cluster (the other option structs already had it). - ErrRingAutoPipelineUnsupported is exported (it was named in public godocs while unexported and unmatchable with errors.Is). - ClusterOptions.AutoPipelineOptions doc corrected: it configures BOTH faces, not only AsyncAutoPipeline. Client/ClusterClient Close docs now mention both cached autopipeliners. - README: stale WithConfig name fixed, the parallel-batches example uses a realistic MaxConcurrentBatches (8) with a note that more permits don't deepen batches; godoc throughput claims hedged as local loopback measurements; stale setReady/await comments refreshed; MaxBatchSize doc notes the blocking face's 300 preset. - Remove three stray benchmark screenshots (513 KB) accidentally committed. * fix(osscluster): route cluster pipelines through the dedicated pipeline pool Cluster pipelines acquired connections via node.Client.getConn — the node's main pool — so the dedicated pipeline pool was never used on ClusterClient, even though node clients inherit PipelineReadBufferSize / PipelineWriteBufferSize / PipelinePoolSize through clientOptions (review finding by cxljs on #3867). processPipelineNode and processTxPipelineNode now acquire through withPipelineConn: the node's dedicated pipeline pool when configured, with the exact previous behavior as the fallback (withPipelineConn delegates to withConn when no pipeline pool exists). An entered flag distinguishes acquisition failures — which keep the old MarkAsFailing / mapCmdsByNode / setCmdsErr handling — from execution errors, which processPipelineNodeConn already handles. TestClusterPipelineUsesDedicatedPool covers both the manual pipeline and the autopipeline batch path against a real cluster and asserts the node pipeline pools actually carried the traffic. * fix(autopipeline): DoRaw escape hatch + node-hook short-circuit surfacing Address review feedback (ofekshenawa, #3867): - AutoPipeliner embeds cmdable, so DoRaw and DoRawWriteTo rode the batching engine even though Do deliberately runs outside it. Raw commands carry Do's caveats, and DoRawWriteTo streams to caller-provided io.Writers — user I/O that must not execute inside a shared batch's reply loop. Both now override with Do's exact escape-hatch shape through a shared runOutsidePipeline helper: synchronous Process on the blocking face; returns-immediately with hook-safe batch completion (and hook short-circuit surfacing) on the deferred face. Do is refactored onto the same helper. - processPipelineNode and processTxPipelineNode discarded the return of the node-level hook chain: a node hook returning without calling next (the circuit-breaker shape the Hook contract allows) left every command error-free, so a cluster pipeline could report success for commands that were never sent. The chain now records execution; when it never reached the transport, the chain's error — or the errHookShortCircuit sentinel — is set on every command. Deliberate aborts are not remapped for retry: a retry would re-run the same hook. Tests cover DoRaw/DoRawWriteTo on both faces under result-reading hooks (including the no-args guard) and the node-level short-circuit against a real cluster, using an armable pipeline-only breaker installed via OnNewNode — disarmed during warmup because connection-init handshakes are themselves pipelines through the hook chain. * fix(autopipeline): retry tests honor REDIS_PORT, doc nits - pipeline_retry_test.go hard-coded ":6379" four times, silently skipping (or hitting the wrong server) when tests run with REDIS_PORT set; use apTestAddr() like the rest of the suite - drop a duplicated sentence in the ClusterOptions doc - the example benchmark's comment promises a goroutine-sized pool for the plain-client baseline but set 50 for 500 goroutines; size the pool to the goroutine count so the baseline is not pool-starved * fix(autopipeline): honor a hook's post-next verdict; complete batches after the chain A hook that called next and then returned an error had that verdict discarded (review finding by ofekshenawa on #3867): batches completed at the innermost exec seam, so waiters were already awake and the executed flag suppressed the chain's error. Batches now complete at the caller's deferred closes, after the whole hook chain has returned. The dispatcher-goroutine-id guard in await()/Wait() carries the self-deadlock protection instead: a hook on the dispatch goroutine reading a command before next() gets the not-yet-executed view, and after next() the populated results — without blocking on a channel only its own goroutine completes. This restores plain-pipeline hook semantics across the board: - a post-next injected error (exec succeeded, hook returned an error) reaches every command that has no error of its own; per-command errors recorded by the exec itself always win, and a hook's rewrap of an exec failure is dropped rather than clobbering precise per-command results; - the lone-command fast path and Do record the chain's final verdict exactly like Client.Process — short-circuits, rewrites and suppressions are all honored; - hooks may adjust results before any waiter wakes, removing the previous 'results are racy to mutate after next()' caveat; - the expected-arrivals announce still precedes the closes on both faces, preserving the wave-merge ordering. Also: the differential fuzz passed a raw 1000 (nanoseconds) to Expire, which truncates to EXPIRE 0 (a delete) — now a real 100s TTL (review finding by cursor). New TestAutoPipelineHookPostNextError covers both faces across the batch, solo and Do paths; the full hook battery (deadlock before/after next, short-circuit, hooks-added-late, DoRaw) stays green through the guard, as do the AP race suite and the cluster tests against a live cluster. * chore(examples): tidy the new example modules against this branch's root The master merge added four example modules (ft-aliaslist, himport, search-aggregate-collect, ts-querylabels) tidied against master's root go.mod. This branch's root additionally requires golang.org/x/sys (cache line padding in the pipeline pool), and the examples replace the root module, so on this branch they need 'go mod tidy' — which made the govulncheck CI job fail while loading packages (no vulnerability involved). Tidy adds the x/sys indirect requirement to each. * fix(autopipeline): skip the dispGid stamp on the blocking face's solo path The lone-command fast path stamped the dispatcher goroutine id (a runtime.Stack header parse) whenever hooks were installed, without the !ap.blocking guard the multi-command and shutdown dispatch paths already have. On the blocking face the stamp is never consulted — blocking commands never carry a batch, so their result accessors don't block — making it pure overhead on every single-command flush (review finding by cursor on #3867). Also document, at the verdict guard in dispatchCmds, why a post-next hook error is applied only to an all-clean batch: hooks conventionally return next's error, so after a partial failure the chain error is presumed to be that echo, and stamping it on the commands that did succeed would overwrite valid replies with their batchmates' failure. Behavior unchanged. * fix(autopipeline): never read user command results on the execution path A full re-review of the PR found that the async autopipeliner deadlocked in the DEFAULT configuration. pipelineProcessCmds re-read the first command's error after the himport re-issue pass via Err(); on the async face Err() awaits the command's own batch — which this very goroutine is executing and only closes after the hook chain returns. With no hooks installed the dispatcher-goroutine-id guard is unarmed, so a multi-command batch whose first reply was any error (a plain redis.Nil cache miss is enough) blocked the dispatch goroutine on itself forever: shard permit never released, every waiter hung, Close() hung on the batch WaitGroup. Reproduced with 50 async GETs on missing keys — the runtime reports 'all goroutines are asleep'. The same execution-path Err() reads existed at five himport sites (himportAfterBatch, himportRetryFailedSets, the cluster re-queue), which additionally run on per-node worker goroutines where the guard can structurally never match — any async batch containing an HIMPORT command wedged even when every command succeeded. All six now use rawErr(), matching the already-swept neighboring sites, with comments stating the rule. Two regression tests pin the redis.Nil-first batch and the himport-in-batch cases on the hook-free default. Also from the re-review: - TSNRangePivotRowSliceCmd.Val()/Result() were the only command accessors missing await(): async-face TSNRange/TSNRevRange returned zero values while racing the flusher's readReply. Added; a mechanical sweep of every accessor confirms no other misses. - Client.WithTimeout copied the client struct while AutoPipeline()/Close() can concurrently mutate the mutex-guarded autopipeliner fields — a data race under -race; the snapshot is now taken under the guard. - NewFailoverClient created the dedicated pipeline pool but never set pipelinePoolName, so poolForConn would route failover pipeline conns to the main pool (latent until failover clients gain streaming credentials; wrong-pool CloseConn/accounting once they do). - AutoPipelineOptions coerced non-positive MaxBatchSize and MaxConcurrentBatches to defaults BEFORE Validate ran, so the documented negative-value construction errors were unreachable. Validate now runs first; zero still selects the default. Test-quality fixes (found by auditing what the suite actually pins): - The pipeline/AP 'retries on network error' tests were vacuous: the broken conn they seeded into the idle pool embeds a zero net.TCPConn, which the pool health check discards before use — both tests passed with retries disabled. They now use a dialer whose conn fails exactly one write at the wire after the handshake, and assert the failure was consumed and exactly one redial occurred, so the first attempt genuinely dies and the re-drive is exercised. - The differential fuzz drove the blocking face sequentially, so every command took the lone-command fast path: zero multi-command batches, meaning batch demux had no randomized coverage. Added an async windowed phase (submit all ops, then read — the ordered default preserves submit order, so the sequential oracle still applies) with a counting hook that fails the test if no real batch formed. Validated by mutation: swapping two replies in dispatchCmds now fails the fuzz. - The cluster NumShards default test asserted >= 2, which fails on single-vCPU runners (the default is clamp(GOMAXPROCS,1,16)); it now asserts the computed default. - README: the AutoPipeline getters also return ErrClosed on a closed client, not only invalid-config errors. * fix(autopipeline): async-face parity for metric-only result reads; CI and doc fixes Publish/SPublish and XRead read their command's result only to record OTel metrics. On the async autopipeline face a result read awaits execution, so with no recorder installed these silently turned fire-and-forget publishes into blocking calls. internal/otel gains Enabled() and the three sites are gated on it; TestAsyncAutoPipelinePublishDoesNotAwait pins submission completing inside the flush delay. The Eval family's post-issue read stays: it normalizes NOSCRIPT to ErrNoScript, which Script.Run/RunRO depend on — now documented as synchronous on the async face. The failover client now registers its pools (main, pubsub, pipeline) for OTel gauge metrics, matching NewClient; it previously registered none while still unregistering on close. example/client-side-caching (new with the csc master merge) is tidied against this branch's root module — the govulncheck job failed loading it (go mod tidy needed; no vulnerability involved), the same pattern as the four examples fixed before the rebase. Documented from the cross-cutting review: a hook must never issue a command on the same autopipeliner and synchronously wait for it (the nested command needs the dispatch slot the hook chain holds; the engine recovers only by failing that flush after the 30s permit backstops, taking co-drained bystanders with it); Options.Limiter is consulted once per batch dispatch like a manual pipeline, not once per command; an autopipeliner created on a WithTimeout clone is not stopped by the parent's Close and must be closed explicitly. README caveats plus the armSelfDeadlockGuard and WithTimeout doc comments carry the warnings. * chore(autopipeline): consolidate pipeline test files; mark the feature experimental Consolidate the five loose pipeline test files into two, mirroring the engine PR's test consolidation: the dedicated-pool battery (handshake, maintnotifications, streaming credentials) into pipeline_pool_test.go and the execution-behavior tests (heterogeneous reply-shape demux, the retry/re-drive loop) into pipeline_exec_test.go. Pure move — ten test functions before and after, with section markers keeping provenance. Mark autopipelining EXPERIMENTAL using the repo's existing marker line: the API may still change as feedback lands. The marker covers every public surface — the eight client getters (Client and ClusterClient, blocking and async, no-arg and WithOptions), the AutoPipeliner and AutoPipelineOptions types, both Default*Options constructors, the UniversalClient interface block and Options.AutoPipelineOptions — plus banners in the README autopipelining section and the example. * chore(autopipeline): assert AutoPipeliner's UniversalClient conformance in universal.go The compile-time assertion existed in autopipeline.go; move it into universal.go's assertion block next to Client, ClusterClient and Ring, where the interface and its implementors are declared. * chore: gofmt — drop the double blank line left by the assertion move * chore(autopipeline): skip the fallback options allocation when an override is supplied getOrCreateAutoPipeliner called the fallback constructor (which allocates a default AutoPipelineOptions) before checking for an explicit override, discarding the result whenever one was provided (review finding by cursor on #3867). Check the override first. * fix(autopipeline): run SHUTDOWN and MONITOR outside pipeline batches Both are on the typed command surface and rode pipeline batches. A batched SHUTDOWN kills the server before it replies, so every batchmate failed with EOF and the whole batch was retried against a dead server (late review finding by cxljs on #3865). A batched MONITOR rebinds the shared connection into a monitor stream, desyncing every batchmate reply behind it. Both now execute directly on their own pooled connection, exactly like blocking commands — one gate in submit(), shared by both faces. The rest of the typed surface audits clean: Quit is a deprecated panic stub; the connection-state commands (Auth, Select, Hello, SwapDB, ClientSetName, ClientSetInfo) live on statefulCmdable and are not reachable through the AutoPipeliner; ClientTracking is the same footgun on a plain pooled client (client-side caching is the supported path); CLIENT PAUSE and the BGSAVE/FLUSH family are server-wide and batch-safe; blocking commands were already routed out via their read timeout. TestAutoPipelineShutdownRunsOutsidePipeline pins both routings with path-recording hooks against a guaranteed-closed port, so nothing real is shut down by the test. * fix(osscluster): honor a node-level hook's post-next verdict on clean sub-batches processPipelineNode surfaced a node-level hook's error only for short-circuits (hook never called next). A hook that called next and then returned its own error — the node-level twin of the post-next case fixed in dispatchCmds — had that verdict dropped: on an all-clean sub-batch every command kept reporting success (suppressed Copilot finding on #3867). Apply the same rule as AutoPipeliner.dispatchCmds: a post-next error on a fully clean sub-batch can only be the hook's own verdict, so it is applied to the commands; on a mixed batch the exec-recorded outcomes win (hooks conventionally echo next's error, and stamping the echo would overwrite successful replies). No remap for retry — retrying would re-run the same hook. cmdsFirstErr reads rawErr, so the check cannot await on the node goroutine. TestClusterNodeHookPostNextError pins the behavior against a live cluster with an armable post-next hook (disarmed during warmup so conn-init handshake pipelines pass). Also: example/autopipeline/go.mod pinned v9.21.0 while every other example pins v9.22.0-beta.1 — aligned and tidied. * fix: deflake the release-time push peek and the handoff-removal test TestReleaseConnRemovesConnectionAfterPartialPushRead failed randomly across matrix cells, and the root cause is in product code: peekAndProcessPushNotifications set a 10µs read deadline after MaybeHasData() had already confirmed kernel-buffered bytes. On a loaded machine the deadline routinely expired before the first peek syscall ran; the push processor treats a zero-consumed timeout as 'no pending data', so a connection with buffered push bytes went back into the pool undrained instead of being drained or removed. The first read never waits on the network here — the deadline only needs to survive scheduler pauses — so raise it to 1ms. Reproduced on the base branch (300 plain / 100 -race iterations both fail); the same loops are fully green with the fix. TestConnectionHook/ConnectionRemovalOnHandoffFailure slept a fixed 100ms and then asserted the connection had been removed — racing the handoff worker's three retries with backoff on slow runners. Poll for the removal with a bounded deadline instead. * feat(autopipeline): #4 tests and benchmarks (#3892) * feat(autopipeline): #4 benchmarks and parametrized command-suite runs Rebuilt on the post-#3867 integration: the branch's history carried pre-rebase copies of pipeliner content that conflicted with the squashed merge, so this is the branch's actual payload as one commit. - autopipeline_bench_test.go: eight honest benchmarks — plain-client and manual-pipeline baselines, the dispatch path (per-command ns/op and allocs/op), the future face, Submit, zero-copy reads, cluster throughput, and the plain-vs-blocking-vs-windowed headline comparison — updated to the shipped API. Every fixed-duration driver divides by the timed region (the previous fixed divisor inflated cluster windowed ~2x and the plain baseline ~27%), zero-copy read buffers live outside the timed region so B/op reflects the command path, and every bench skips without a server. The earlier tuning sweeps (batch/flush-delay/buffer sizes) were deleted: at low parallelism they reported the configured MaxFlushDelay timer for every swept value — noise dressed as signal. autopipeline_bench_README.md rewritten multiplier-first: absolute numbers are machine/load-dependent, so the documented signal is the within-run ratio vs the plain-client baseline (plus environment-independent allocs/op); the main README's throughput claims got the same treatment. - iterator.go: ScanIterator.Next read the command's page/cursor fields without awaiting the fetch, which only enqueues on the deferred autopipeline face — iteration spun re-issuing SCANs with a stale cursor forever, flooding the engine with an unbounded batch and hanging the caller. Next now awaits the fetch before reading (a no-op on every other path); TestAsyncAutoPipelineScanIterator pins it. - Parametrized-suite fixes: BFScanDump/CFScanDump compared command structs (each async command carries its own batch pointer, so equality is structurally false there) — compare values; the SlowLog argument-count spec and the BLMoveM timeout spec now keep admin/conn calls on the raw client like their siblings; the commands/bitmap suites build the subject closer before their first assertion so a failed BeforeEach cannot leave AfterEach with a nil or stale closer. - CI: the subjects job focus covers all nine parametrized suites (the old case-sensitive focus silently ran one), each subject run carries an explicit -timeout, and the run-tests action refuses an empty make target. - Command-suite parametrization (#3885): the commands/array/bitmap/ iterator/json/probabilistic/search/timeseries/vectorset suites run against a subject selected by GOREDIS_TEST_SUBJECT (client, ap-blocking, ap-async) via the newUniversalSubject harness, so the whole surface can be replayed through either autopipeliner face without touching specs. - maintnotifications: a failed manager Close no longer drops the not-yet-processed additional pool hooks — they are put back so a retried Close still sees them. - pipeline_buffer_test.go: skip, rather than fail, when the docker test server is absent. - CI: a test-autopipeline-subjects job replays the command suite through both faces via GOREDIS_TEST_SUBJECT (run-tests action gains an injection-safe make-target input; Makefile gains the target). Dropped from the old branch: stale pre-rebase engine-file copies (superseded by #3867), its pre-consolidation AP test files (integration's are newer and complete), and three unreferenced PNGs. * chore(autopipeline): review polish — close leaked test clients, align the docs The ScanIterator regression test closes its autopipeliner explicitly and the RESP3 search suite closes client2 in AfterEach (it leaked one client per spec) — both flagged by Copilot. From the cold-read review: the main README attributed the parallel-batch ~10x multiplier to the blocking face's ordered default (the bench README says the default lands lower — the two now tell one story); pipeline_buffer_test.go gets its honest pre-existing comment back; the Makefile comment says the subjects focus 'covers' every parametrized suite rather than 'lists explicitly'. * fix(bench): feasible zero-copy load, exact b.N accounting; CI polish The zero-copy benchmark at 64KiB payloads drove 500 goroutines through a PoolSize-10 client — on a 4-vCPU CI runner that blows the default 3s write deadline, and the newly surfaced errors (previously swallowed into the ops/sec count) correctly failed the bench on two matrix cells. Large-payload sizes now run 100 goroutines with generous deadlines, so the benchmark measures the command path rather than write-deadline queueing; the B/op result is unchanged (~100x less at 64KiB). From Copilot's follow-up review: benchWindowed and the zero-copy driver partition b.N exactly across workers (the first b.N%goroutines take one extra op) instead of floor-and-min-1, so ns/op and allocs/op are per command at every -benchtime — the old split over-ran for small b.N and dropped the remainder for large; and the run-tests action invokes make -- "$MAKE_TARGET" so a target value starting with a dash cannot inject make options. Also backtick MaxConcurrentBatches/Unordered in the README prose the previous commit added — the spellchecker exempts code spans. * fix(autopipeline): full review round for the master merge Required fixes from the integration review: ReadStringInto's fast path gates on len instead of cap, so the trailing CRLF can never be written past the slice the caller handed over (a sub-slice of a packed buffer exposed trailing capacity belonging to the next segment — the pattern the zero-copy example itself teaches); cluster mapCmdsByNode is strictly all-or-nothing again for user pipelines, with fan-out-policy commands rejected at autopipeline submit through a new engine preflight hook (so a merged batch cannot be poisoned and the lone-command path agrees with batched dispatch); both pipeline read loops carry push-frame-mid-batch regression tests (the cluster one mutation-validated); and Submit() marks commands ready so result accessors before Wait() await instead of racing the dispatcher. Codex findings: a bad-conn push-drain error on the dedicated pipeline pool removes the connection instead of pooling a desynced reader; PipelineReadBufferSize gets the RESP3 minimum clamp on both construction paths; a CSC WithTimeout clone's Close delegates to the owner's Client.Close so cached autopipeliners stop and the closed flag flips. Engine: AutoPipelineOptions.MaxBatchBytes caps flushes by approximate payload volume and dispatch splits oversized drains into byte-bounded chunks (batches still complete once, after all chunks) — many large values flush as several bounded writes instead of one multi-megabyte burst that can stall a constrained link past its write deadline; dispatch goroutines recover panics into per-command errors instead of crashing the process (a plain client unwinds a hook panic into the caller; the engine's internal goroutines had nobody to recover); the outside-pipeline guard covers the full connection-hostile command set for raw Submit/Do; internal backstop timeouts surface ErrAutoPipelineTimeout rather than a misleading context.DeadlineExceeded; ErrHookShortCircuit and ErrSubmitBlockingFace are exported; the content-shard index is MinInt-safe. Docs and tests per the review: pending-command String() formatting documented as unsynchronized; AfterEach drains subjects before flushing in the module suites; the CI subjects job claims results parity rather than batch coverage; new tests cover RESP2 batched demux, blocking-face ctx cancel, dispatch-panic containment, byte-capped batching, Submit misuse under -race, push drains on both read loops, policy rejection on both pipeline kinds, and the never-past-len reader guarantee; assorted minors (govulncheck minor-series pin, apTestAddr in buffer tests, slotCache happens-before note, twin-path drift warning, Close worst-case and Validate caveats, EXPERIMENTAL markers on new exports). * test: flush through the subject before closing it in the module suites The previous commit ordered closeSubject() before rawClient.FlushDB — but the closer closes rawClient, so the flush silently failed on a closed client and state leaked across specs (SEARCH_INDEX_EXISTS on the next protocol iteration, red on every CI test cell). Flush through the SUBJECT instead, in the original order: on the async autopipeline face the flush is ordered after any queued writes and closeSubject's drain guarantees it executes, which is what the reorder was trying to achieve; on the plain and blocking paths it is identical to the old rawClient flush. * fix(autopipeline): deferred face returns immediately for diverted commands; timeseries AfterEach Typed blocking and connection-hostile commands (BLPOP, SHUTDOWN, ...) were executed inline on the deferred face, stalling the submitter for the full block duration — Do already honored the deferred contract via its background-goroutine escape hatch. The diverted submit path now routes through the same runOutsidePipeline, keeping each face's call shape: the blocking face runs synchronously, the deferred face returns immediately with a ready batch and the result accessors block (review finding by cursor on #3942). A regression test pins submission returning immediately while BLPOP still blocks server-side for its full window. Also: the timeseries suite gets the same subject-flush AfterEach the previous commit applied to json/probabilistic/vectorset — flushing via rawClient before closing the subject let a queued async write land after the flush. * test: count solo dispatches in the MaxBatchBytes test The counting hook only watched the pipeline hook, but a drain cycle that catches exactly one command takes the engine's lone-command fast path (ProcessHook, no pipeline) — on a loaded runner one of the ten commands split off that way and the total came up short. Record solo dispatches as size-1 entries, filter to the test's own SET commands so a fresh-conn handshake pipeline cannot pollute the counts, and allow the uncapped control one straggler split. * test: widen the sentinel resolution budget for loaded runners The regression test demanded the full sentinel resolution chain inside a 500ms context — routinely missed on loaded CI runners over docker networking. Three seconds keeps the signal (the retry-storm exhaustion it pins burned the whole context regardless of size) without the flake. * fix: legacy DisableIndentity mapping in cluster node options; align checkout action version The cluster node clientOptions mapped the deprecated DisableIndentity field from the NEW DisableIdentity option, silently ignoring a user-supplied legacy value — Ring and the failover client already copy the legacy field. Also bump the run-tests composite action's internal checkout to v7, matching the workflow-level version. Both pre-existing upstream, folded in on the way to master. * fix(autopipeline): delegate cluster-wide commands, abort chunks after failed prefix, share the closed flag across clones Three review findings (codex on #3942) plus a test seam: AutoPipeliner now delegates ClusterClient's cluster-wide method overrides (DBSize, ScriptLoad, ScriptFlush, ScriptExists) to the underlying client — the embedded cmdable shadowed them and sent each to a single random node. Chunked dispatch (MaxBatchBytes) stops dispatching remaining chunks once an earlier chunk dies on a transport-class failure, failing them with the same error: the unchunked path fails or retries the batch as a unit, so in the ordered stream later commands must not overtake a failed prefix. Per-command redis errors are normal chunk outcomes and do not abort. The pool-closed flag (baseClient.apClosed) becomes a shared *atomic.Bool copied by WithTimeout/clone: closing a clone closes the SHARED pools, so the parent's AutoPipeline getters now see ErrClosed instead of building a fresh flusher against dead pools. ClusterClient passes nil — its per-node clients own their pools. The maintnotifications Manager gets a shutdownTimeout field (default 10s, behavior unchanged) so the Close put-back path — a failed shutdown returns the pool hook and reopens the manager for a retry — is pinned by a test. Tests: TestAPClusterWideCommandsDelegate, TestDispatchChunkedAbortsAfterFailedPrefix (unit-level: through the public API the byte cap wakes the flusher mid-submission, so one logical batch may split into several drains and a single-dispatch assertion races), TestWithTimeoutCloneCloseBlocksParentResurrection, TestCloseRetriesAdditionalHookAfterShutdownError. * fix: String() awaits deferred commands; share pooled-conn helpers between main and pipeline pools Public String methods on commands read their result fields before awaiting, so formatting an in-flight deferred-autopipeline command from another goroutine raced with reply processing. Every String method now calls await() first — the same accessor contract as Err/Val/Result. The dispatch-gid guard in await keeps a hook that formats its own batch on the dispatch goroutine safe (it sees the not-yet-executed view instead of deadlocking), and cmdString remains the non-waiting internal formatter on rawErr. The hook-peek regression tests now format commands from inside hooks as well (review finding by ofek and cursor on #3942). withPipelineConn duplicated getConn/releaseConn by hand and had drifted again: the dedicated pipeline pool missed the client-side-cache post-read probe on release and the connection-create-time metric. Conn preparation and release now go through pool-parameterized helpers shared with the main path (initPooledConn, releaseConnToPool), closing the drift class. This also fixes a main-path slot leak — a conn that cannot be re-acquired after its handshake is now Removed from the pool instead of abandoned. The reported Acquire hang on Go 1.23+ timer channels is refuted empirically and pinned by regression hammers: under synchronous timer channels Stop returns TRUE for an expired-but-undelivered fire (the delivery is aborted), returning false only once the value was actually received — and the sole receiver is Acquire's own select, so a taken token implies an unreceived fire and the drain branch is unreachable. Under GODEBUG=asynctimerchan=1 (a consumer main module on go < 1.23) the fired value sits buffered and the drain consumes it before pooled Reset-reuse. The hammers race Release against timeout expiry under both semantics. * fix(autopipeline): register cluster node executors on batches; harden pooled semaphore timers A node-level hook reading a command result before next() — exactly what redisotel's tracing does on cluster node clients via OnNewNode plus rediscmd.CmdsString's Err() peek — permanently wedged the deferred cluster face: the read blocked on a batch that completes only after the node call returns, and the single dispatch-gid guard cannot vouch for the cluster fan-out goroutines. Batches now carry a registry of node executor goroutines, entered and left around each node call's hook chain, and the accessor guards (await, Wait, WaitContext) treat registered goroutines like the dispatcher: result reads get the not-yet-executed view instead of self-deadlocking. Reproduced against a live cluster and pinned by TestClusterNodeHookResultPeekNoDeadlock. The pooled semaphore timers get a portable drain: it moves to a single putSemTimer at pool put-back using select-with-default, correct under both timer-channel semantics. Under go >= 1.23 synchronous channels Stop discards a fired-but-undelivered value, so the drain branch is unreachable — the default arm is a safety net so no semantics shift can ever turn it into a blocking receive. Under GODEBUG=asynctimerchan=1 (a consumer main module on go < 1.23) the buffered fire is consumed before Reset-reuse. Stale-fire hammers pin both semantics against spurious instant timeouts. * test: make the search RESP2/RESP3 equivalence suite hermetic The subtests drop only the index between runs, so earlier subtests' hashes were re-indexed by the next one (the aggregate subtest saw the search subtest's third document), and waitForIndexing polled FT.INFO a hundred times without sleeping — returning in microseconds on a partial count. On a slow runner the index was still churning between the RESP2 and RESP3 queries, which surfaced as a transient extra empty row in the RESP3 aggregate reply. Each subtest now starts from a flushed keyspace, and the wait requires the exact document count with ingestion settled (Indexing == 0), sleeping between tries. * test: wait for search ingestion in the home_json doctests The search examples write three documents and FT.SEARCH them back to back, but indexing is asynchronous — on a loaded runner the hash example's query raced ingestion and returned zero results. Both examples now wait for the exact document count with ingestion settled before querying. The wait lives in a separate non-example file and is called inside REMOVE_START/REMOVE_END markers, so the generated documentation snippets are unchanged. * test: assert module-suite teardown; scale semaphore hammers under -short The json, probabilistic, timeseries and vectorset suites ignored the results of their AfterEach FlushDB and closeSubject calls. On the async autopipeline subject the flush is enqueued and its failure only observable through the command's Err(), while AutoPipeliner.Close reports nil — so a failed teardown could silently leak keys into later specs. Both steps are asserted now, and awaiting the flush also pins it to execute at teardown time rather than inside the close drain. The semaphore timer hammers scale down under -short (200k -> 2k race iterations, 200 -> 20 stale-fire rounds), following the repo's existing short-mode convention for stress tests; full runs keep full coverage. * perf(autopipeline): restore the accessor guards' no-executor fast path The node-executor guard replaced the dispatch-gid short-circuit in the accessor guards (await, Wait, WaitContext) with a check that parsed the goroutine id and took the batch mutex unconditionally. Every blocking-face command and every pre-done future passes that guard once per wait, so concurrent waiters serialized on one batch's mutex with a stack parse each: the blocking face dropped from ~830k to ~138k ops/sec (6x) on the loopback throughput benchmark, while the windowed face was unaffected (its reads mostly land after batch completion). isExecutorGoroutine now answers the common case — dispatch gid unset and no registered node executors — from two atomic loads before any goroutine id parse or lock; the mutex is reachable only inside a cluster batch's node fan-out window. Blocking-face throughput measured back at the pre-regression band, and the node-hook deadlock regression test still passes: the guard still vouches for cluster node executors. * fix(autopipeline): honor shared pool closure at submit; stamp ready under the stripe lock; divert conn-state cluster commands Three review findings (codex on #3942): A cached AutoPipeliner kept accepting enqueues after a WithTimeout clone closed the SHARED pools — the shared closed flag guarded only the getters. It now threads into the pipeliner and every submit path (batched and diverted) checks it, so work is refused with ErrClosed at submit instead of failing dispatch by dispatch against dead pools. The deferred face published a command to its shard before setReady stamped the gating batch, so a fast flush could take a command whose readyBatch was still nil — the cluster node-executor registration would skip that batch, and a node hook reading the command mid-dispatch could block on a batch its own call chain completes. enqueue now stamps ready under the stripe lock, before the command is visible to any drain; the ordering is guaranteed by the lock both sides take. readonly, readwrite and asking join the outside-pipeline divert set: they mutate connection-scoped cluster state (replica reads, a pending redirect) that must not leak onto shared pipeline connections — and ReadOnly/ReadWrite are reachable from the typed surface, not just Do. Tests: TestWithTimeoutCloneCloseStopsCachedAutoPipeliner, TestEnqueueStampsReadyUnderStripeLock, TestConnStateClusterCommandsRunOutsidePipeline. Throughput bands re-measured unchanged after the enqueue-path change. * fix(autopipeline): bound Close on tracked diverted dispatches; divert raw blocking commands; delegate HIMPORT Six review findings (codex, cxljs and ofek on #3942), all reproduced in the code before fixing. Close and diverted commands (one coupled change). Commands that run outside a batch — blocking and connection-hostile ones — executed on untracked goroutines, so Close could return while such a command still held a pooled connection. They are tracked now. Tracking alone would trade that leak for a worse bug: an accepted command is deliberately never cancelled by Close (the flush contract the batched path documents), so a zero-timeout blocking command, or a stalled read with ReadTimeout disabled, would hang Close forever. The wait is therefore bounded by the same backstop the permit wait uses, and on expiry Close returns an error naming what is still in flight rather than blocking its caller. Raw blocking commands. Typed helpers mark themselves with a per-command read timeout, which submit already diverts on; a raw Cmder built by hand carries no such marker, so BLPOP and friends could be queued onto a shared pipeline connection and hold it for the block duration. They are recognized by name now, with XREAD/XREADGROUP classified by their BLOCK argument so the common non-blocking form stays batched. Cluster-wide overrides. The HIMPORT admin trio joins the delegation set: like DBSize and the SCRIPT admin commands, ClusterClient fans them out to every master and updates the shared fieldset registry, so running them on a single routed node lets a later HImportSet for a key on another master fail. A reflection test now walks ClusterClient's method set and fails when an override that shadows the generic cmdable implementation has no delegate — this class has been found by review twice. Maintenance notifications. A delayed idle handoff for a connection from a dedicated pipeline pool queued on the primary pool hook, whose request carries the primary pool; a failed handoff then closed the connection without freeing its slot in the pipeline pool. Handoffs now route to the hook that owns the connection's pool, matched by the pool name already stamped on every connection. Close's bound is its own named constant rather than a borrowed one, with the reasoning recorded next to it: the bound is only reachable when a dispatch cannot end by itself, so a tighter value would not speed up healthy shutdowns — it would make Close report failure while a legitimate slow drain (a large final batch, a relaxed maintenance window during failover) is still finishing. Docs. The slot-to-shard mapping is described as what it is — a batch-depth heuristic. With non-contiguous slot ownership a shard's batch can still span nodes; correctness is unaffected because the node router resolves each command's own slot, and a key always lands on one shard so its relative order holds. Also lands the truth battery: thirteen end-to-end tests that check order, reply demultiplexing, exactly-once execution, at-most-once under retries, chunked-dispatch integrity, pipeline-pool isolation, cancellation honesty and the Close drain against an INDEPENDENT ground truth — a client-side model value, a server-side structure recording the real execution order, or a differential run of the same randomized workload through a plain client. * fix(autopipeline): close the divert registration race, honor shared close everywhere, order limiter reporting Five review findings on the previous commit (cursor Bugbot and codex on #3942), each reproduced in the code first. Shared close. Do, DoRaw and DoRawWriteTo guarded on this pipeliner's own closed flag rather than the combined check, so once a WithTimeout clone had closed the shared pool set they still accepted commands and dispatched against pools that were already gone. IsClosed had the same gap while its documentation promised that the owning client's close counts. Every guard that gates accepting new work now goes through the combined check, with the invariant recorded beside it: only Close's own compare-and-swap reads the instance flag directly, because it claims the shutdown rather than testing for it. Divert registration race. A diverted command could pass its closed check, Close could then observe an empty wait group and return, and only afterwards would the goroutine register — leaving an accepted command holding a pooled connection past Close, and racing Add against Wait. Registration now happens under a mutex with a closed re-check, and Close passes through that mutex once after marking itself closed, so a registration either completes before the wait or is rejected. The gate precedes publishing the command's batch: publishing first and then rejecting would leave the command gated on a batch nobody closes. Blocking detection. The BLOCK token is matched the way the encoder renders it — a raw Cmder may carry RESP keywords as []byte or *string, and a string-only type assertion let those XREAD forms be batched onto a shared connection. Cluster preflight. Diversion is decided before the preflight runs. The preflight keeps fan-out-policy commands out of shared pipelines, but a diverted command never rides one: it goes through Client/ClusterClient Process, where the cluster-wide aggregation works. Running the preflight first rejected typed WAIT/WAITAOF on a cluster whenever command policies were enabled. Limiter ordering. The dedicated pipeline-pool path registered the limiter report and the connection release as two defers, which run LIFO — the connection went back to the pool before the result was reported, so a circuit breaker could admit another pipelined operation before observing the failure. The main pool reports first; both paths now share one deferred exit that preserves that order and still reports on the acquire and handshake failure paths. * fix(autopipeline): bound the whole close drain; gate raw Do and telemetry on the divert contract Four review findings from codex plus two from Copilot, each reproduced in the code first. Raw Do on the blocking face. Do, DoRaw and DoRawWriteTo carried their own inline blocking-face branch that called Process directly, bypassing runOutsidePipeline and therefore the divert registration gate added in the previous commit. A raw blocking Do was untracked, so Close could observe no diverted work and tear down shared resources while it still held a pooled connection. Both faces now go through the one path that registers. Close's bound covered only the last wait. A dispatch that nothing can end sits just as easily in a flusher the wait group is waiting for, or in the shutdown sweep's own dispatch, as in the final wait — so Close hung before ever reaching the bound it documents. The whole drain tail (flusher exit, final shard sweep, batch and diverted waits) now runs under a single bound, and on expiry the error names the precise stage still outstanding, because a wedged flusher and a wedged blocking command call for different operator responses. Telemetry no longer changes the async call shape. The metric emissions after Publish, SPublish and XReadGroup read the command's outcome immediately, which AWAITS on the deferred face: with a recorder installed, an autopipelined publish blocked until Redis replied, and an XReadGroup with the default Block: 0 could block indefinitely in what should be a submit. They now consult a non-blocking readiness check and skip the metric for a submission that has not executed; emitting these from the execution path is a follow-up in the OTel wiring rather than something a command wrapper can do. Two Copilot findings: the ConnPool.Name doc block added last round swallowed Size()'s Godoc, leaving an exported method undocumented — restored; and the pipeline-buffer tests left fixed keys behind, one of which (test_key) is also used by another test file in the package, which is exactly the order-dependence that produced the search-suite flake earlier in this PR. * fix(autopipeline): stop Config leaking internal sharding; keep Eval non-blocking on the deferred face Two review findings from codex plus one from Copilot. Config leaked an internal invariant. The effective config returned to callers carried contentSharded, the bit cluster wiring sets to record that shards are slot-routed and therefore cannot reorder same-key commands. Copying that config into a standalone deferred autopipeliner silenced the check that requires Unordered for NumShards>1 — and round-robin shards genuinely do flush concurrently and break submit order. Internal-only fields are now cleared before the copy is handed out. Eval was synchronous on the deferred face. Normalizing NOSCRIPT to ErrNoScript read the command's error immediately after submitting, which awaits execution, so the whole Eval family lost the deferred call shape — a caller submitting a window of scripts, or running one slow script, blocked in the submit. The normalization now runs only when the result is already readable, and Script.Run/RunRO accept the server's raw NOSCRIPT error so the EVAL fallback keeps working when it was skipped. The synchronous and blocking faces are unchanged; on the deferred face the raw error is byte-identical to ErrNoScript, so errors.Is and MatchError assertions still hold. Copilot: the pipeline-buffer key cleanup added in the previous commit ran before the reachability check, issuing a DEL against a server that might not be there and turning a unit-only run into a failure instead of a skip. The four call sites skip first now, and the helper documents that ordering. * fix(autopipeline): keep a no-retry command's policy off its batchmates; complete the blocking set A command that forbids retries — today the zero-copy reads, whose reply decodes into a caller-owned buffer that a retry could not un-write — disables retries for the entire slice it is dispatched in. In a shared autopipeline batch that silently stripped retries from unrelated callers' ordinary commands, which would have been retried had they been flushed on their own. Mixed batches are now partitioned by retry policy and each group dispatched separately; uniform batches are returned untouched, so only the mixed case pays a second dispatch. Splitting rather than diverting the no-retry commands keeps zero-copy reads pipelined — measured after the change at 507k ops/sec and 497 B/op for a 4 KiB value, still ahead of the regular path. The raw-blocking allowlist was incomplete: blmovem, migrate, and the argument-driven ts.read BLOCK form could be queued onto a shared pipeline connection when submitted as raw Cmders, holding it for the block duration. The set is now derived from the typed helpers that call setReadTimeout rather than enumerated by hand, with that derivation recorded beside it and a test that pins every name and both arg-driven forms. Also rewords PipelineReadBufferSize's tuning guidance, which described the write side — sizing so a batch flushes without overflowing mid-write — and cross-references the option that guidance actually belongs to. codex's remaining finding, that ClusterClient.PoolStats does not fold in the per-node pipeline pools, is a monitoring gap rather than a correctness one and is left as a follow-up. * fix(otel): read the global recorder under its lock when registering pools RegisterPools and UnregisterPools read globalRecorder directly while SetGlobalRecorder writes it under recorderMu, so installing telemetry concurrently with creating or closing a client is a data race. A regression test reproduces it — two race reports under -race before the change, clean after — and both helpers now go through getRecorder. Also fixes the pipeline-buffer key cleanup added earlier in this PR: its post-test sweep ran in a t.Cleanup, which executes after the test function's defers, and every caller defers client.Close(). The DEL therefore landed on a closed client and silently did nothing, leaving exactly the keys the helper exists to remove. The sweep now uses its own short-lived client and reports a failure rather than swallowing it (both found by Copilot on #3942). * fix(autopipeline): isolate retry policy without reordering; honor successful hook short-circuits The no-retry isolation added in the previous commit split a mixed batch into two policy groups and dispatched the retryable one first, which REORDERED the stream: a zero-copy read submitted before a write to the same key executed after it, so the read observed the new value on a face that promises submit order. The batch is now sliced into contiguous runs and dispatched in order — each dispatched pipeline is still policy-uniform, so a no-retry command cannot strip retries from batchmates, and no command overtakes another. Uniform batches are still one dispatch with no allocation. Both new tests fail if the grouping is reintroduced, including the end-to-end one that reads the wrong value. A hook that returns nil WITHOUT calling next has short-circuited successfully: it served the batch itself and set the command values, which is exactly what a caching or mocking hook does and what plain Pipeline and Client hooks are permitted to do. The engine rewrote that success into an error, so such a hook worked on a pipeline but failed every autopipelined batch. All five sites now propagate only the hook's own error, and the sentinel this PR added for the synthesized case is retired. Cluster commands whose routing is not slot-derived — ReqSpecial, such as FT.CURSOR READ, which is sticky to the node holding the cursor — are now diverted instead of coalesced. A solo flush reached ClusterClient.process and its special routing, but inside a batch mapCmdsByNode routes by slot and can send the cursor read to the wrong shard, which only shows up under concurrent traffic. Diverting keeps them working on their own connection rather than rejecting them (all three found by codex on #3942). * fix(autopipeline): share the prefix-abort rule; report submit rejections from Process Breaking a batch into ordered pieces happens in two places now — the MaxBatchBytes chunker and the retry-policy runs — and each independently got the same rule wrong: once a piece dies on a transport-class failure, the remaining pieces must not be dispatched, because the unchunked path fails or retries the batch as a unit and later commands must never overtake a failed prefix. The runs loop dispatched them anyway. Both callers now go through one dispatchSequential helper that owns the rule, so a third variant cannot drift out of line; per-command redis errors remain normal outcomes that do not stop the sequence. The deferred face's Process discarded submit-time rejections. A closed pipeliner or a cluster preflight refusal sets the error on the command and queues nothing, so returning nil told the caller the command was accepted when it would never run — and code reaching the engine through UniversalClient.Process sees only that return value. Those rejections are returned now, identified by the shared completed batch rather than by probing for an error, so execution failures still stay off this path: the call must not wait. The type documentation still said String()/%v on a pending deferred command does not wait. It has waited since the accessor round, and a test pins exactly that, so the doc was steering users toward a race that no longer exists. It now describes the real contract, including the dispatch-goroutine exception, and points at Name()/Args() for logging a submission without waiting (all three found by codex on #3942).
This commit is contained in:
1 parent
228b463029
commit
c994cfc85f
91 files changed
+15148
-565
No files matched your search
@@ -16,11 +16,15 @@ inputs:
|
||||
client-libs-test-image-tag:
|
||||
description: 'Client libs test image tag'
|
||||
required: false
|
||||
make-target:
|
||||
description: 'Makefile target to run for the test step'
|
||||
required: false
|
||||
default: 'test.ci'
|
||||
runs:
|
||||
using: "composite"
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v6
|
||||
uses: actions/checkout@v7
|
||||
with:
|
||||
repository: ${{ inputs.repository }}
|
||||
ref: ${{ inputs.ref }}
|
||||
@@ -72,6 +76,9 @@ runs:
|
||||
env:
|
||||
RCE_DOCKER: "true"
|
||||
RE_CLUSTER: "false"
|
||||
# Passed through the environment (not ${{ }}-interpolated into the
|
||||
# script) so the input value cannot inject shell commands.
|
||||
MAKE_TARGET: ${{ inputs.make-target }}
|
||||
run: |
|
||||
make test.ci
|
||||
make -- "${MAKE_TARGET:?make-target must not be empty}"
|
||||
shell: bash
|
||||
@@ -1,10 +1,13 @@
|
||||
ACL
|
||||
ACLs
|
||||
APIs
|
||||
async
|
||||
autoload
|
||||
autoloader
|
||||
autoloading
|
||||
autopipelining
|
||||
analytics
|
||||
runnable
|
||||
Autoloading
|
||||
backend
|
||||
backends
|
||||
@@ -23,6 +26,8 @@ fieldset
|
||||
fieldsets
|
||||
FPM
|
||||
Golang
|
||||
goroutine
|
||||
goroutines
|
||||
IANA
|
||||
keyspace
|
||||
keyspaces
|
||||
@@ -122,3 +127,6 @@ TOCTOU
|
||||
wordlist
|
||||
worktree
|
||||
fetch's
|
||||
loopback
|
||||
autopipeliner
|
||||
autopipeliner's
|
||||
@@ -4,7 +4,7 @@ on:
|
||||
push:
|
||||
branches: [master, v9, 'v9.*']
|
||||
pull_request:
|
||||
branches: [master, v9, v9.7, v9.8, 'ndyakov/**', 'ofekshenawa/**', 'ce/**']
|
||||
branches: [master, v9, v9.7, v9.8, 'ndyakov/**', 'ofekshenawa/**', 'ce/**', 'feature/**']
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
@@ -105,3 +105,27 @@ jobs:
|
||||
files: coverage.txt
|
||||
token: ${{ secrets.CODECOV_TOKEN }}
|
||||
|
||||
test-autopipeline-subjects:
|
||||
name: test-autopipeline-subjects
|
||||
runs-on: ubuntu-latest
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
redis-version:
|
||||
- "8.10.x" # Redis CE 8.10
|
||||
go-version:
|
||||
- "stable"
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v7
|
||||
|
||||
# Replay the command integration suite through the AutoPipeliner Cmdable
|
||||
# faces (blocking + async) so every command is verified to behave the same
|
||||
# batched as on a plain client.
|
||||
- name: Run command suite through AutoPipeliner faces
|
||||
uses: ./.github/actions/run-tests
|
||||
with:
|
||||
go-version: ${{ matrix.go-version }}
|
||||
redis-version: ${{ matrix.redis-version }}
|
||||
make-target: test.autopipeline-subjects
|
||||
|
||||
@@ -15,7 +15,7 @@ on:
|
||||
push:
|
||||
branches: [master, v9, v9.7, v9.8]
|
||||
pull_request:
|
||||
branches: [master, v9, v9.7, v9.8]
|
||||
branches: [master, v9, v9.7, v9.8, 'feature/**']
|
||||
|
||||
jobs:
|
||||
analyze:
|
||||
|
||||
@@ -3,7 +3,7 @@ name: govulncheck
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
branches: [master, v9, v9.7, v9.8, "ndyakov/**", "ofekshenawa/**", "ce/**"]
|
||||
branches: [master, v9, v9.7, v9.8, "ndyakov/**", "ofekshenawa/**", "ce/**", "feature/**"]
|
||||
push:
|
||||
branches: [master, v9, "v9.*"]
|
||||
schedule:
|
||||
|
||||
@@ -4,7 +4,7 @@ on:
|
||||
push:
|
||||
branches: [master, v9, 'v9.*']
|
||||
pull_request:
|
||||
branches: [master, v9, v9.7, v9.8, 'ndyakov/**', 'ofekshenawa/**', 'ce/**']
|
||||
branches: [master, v9, v9.7, v9.8, 'ndyakov/**', 'ofekshenawa/**', 'ce/**', 'feature/**']
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
@@ -20,3 +20,9 @@ dockers/osscluster-tls/
|
||||
|
||||
# Per-user Claude Code settings (machine/personal overrides, not shared policy)
|
||||
.claude/settings.local.json
|
||||
|
||||
# example build artifacts
|
||||
example/autopipeline/autopipeline
|
||||
|
||||
# Claude Code worktrees (ephemeral, local)
|
||||
.claude/worktrees/
|
||||
@@ -71,6 +71,35 @@ test.ci.skip-vectorsets:
|
||||
cd internal/customvet && go build .
|
||||
go vet -vettool ./internal/customvet/customvet
|
||||
|
||||
# Replay the parametrized command integration suites through the AutoPipeliner
|
||||
# Cmdable faces (blocking + async) to prove the commands behave the same
|
||||
# batched as on a plain client. Selected via GOREDIS_TEST_SUBJECT (see
|
||||
# newUniversalSubject in main_test.go). Requires the docker env
|
||||
# (make docker.start).
|
||||
#
|
||||
# The focus covers every parametrized suite (some via the Commands/BitCount
|
||||
# substrings — Describe names differ in case). Skipped on purpose: DDL Commands (needs the cluster topology the
|
||||
# lightweight BeforeSuite doesn't build), HotKeys Commands (same), and
|
||||
# AutoPipeline Blocking Commands (the AP suite itself — running it through an
|
||||
# AP subject would nest engines).
|
||||
test.autopipeline-subjects:
|
||||
# RE_CLUSTER=true forces the lightweight BeforeSuite (no sentinel/ring/cluster
|
||||
# setup): the command suite only needs the standalone Redis, and running the
|
||||
# full stateful BeforeSuite twice (once per subject) against the same server
|
||||
# corrupts replication state and fails the second run. Both faces then run
|
||||
# cleanly against the same env. The explicit -timeout keeps a hang from
|
||||
# eating go test's 10m default per subject.
|
||||
set -e; for subj in ap-blocking ap-async; do \
|
||||
echo "=== command suite via GOREDIS_TEST_SUBJECT=$$subj ==="; \
|
||||
(export RE_CLUSTER=true && \
|
||||
export RCE_DOCKER=$(RCE_DOCKER) && \
|
||||
export REDIS_VERSION=$(REDIS_VERSION) && \
|
||||
export GOREDIS_TEST_SUBJECT=$$subj && \
|
||||
go test -v . -race -skip Example -run TestGinkgoSuite -timeout 8m \
|
||||
-ginkgo.focus='Commands|RediSearch commands|Probabilistic commands|RedisTimeseries commands|Redis VectorSet commands|BitCount|ScanIterator|Advanced JSON' \
|
||||
-ginkgo.skip='AutoPipeline Blocking Commands|DDL Commands|HotKeys Commands'); \
|
||||
done
|
||||
|
||||
bench:
|
||||
export RE_CLUSTER=$(RE_CLUSTER) && \
|
||||
export RCE_DOCKER=$(RCE_DOCKER) && \
|
||||
@@ -101,7 +130,7 @@ test.e2e.logic:
|
||||
go test -v -run "TestCreateTestFaultInjectorLogic|TestFaultInjectorClientCreation" ./maintnotifications/e2e/
|
||||
@echo "Logic tests completed!"
|
||||
|
||||
.PHONY: all test test.ci test.ci.skip-vectorsets bench fmt test.e2e test.e2e.logic docker.e2e.start docker.e2e.stop
|
||||
.PHONY: all test test.ci test.ci.skip-vectorsets test.autopipeline-subjects bench fmt test.e2e test.e2e.logic docker.e2e.start docker.e2e.stop
|
||||
|
||||
build:
|
||||
export RE_CLUSTER=$(RE_CLUSTER) && \
|
||||
|
||||
@@ -79,6 +79,9 @@ surface. The API is experimental and may change in a future release.
|
||||
- [StreamingCredentialsProvider (e.g. entra id, oauth)](#1-streaming-credentials-provider-highest-priority) (experimental)
|
||||
- [Pub/Sub](https://redis.uptrace.dev/guide/go-redis-pubsub.html).
|
||||
- [Pipelines and transactions](https://redis.uptrace.dev/guide/go-redis-pipelines.html).
|
||||
- [Automatic pipelining](#automatic-pipelining) (experimental) — batches concurrent
|
||||
commands into pipelines for you; meant for high-throughput / high-load / scale
|
||||
use cases.
|
||||
- [Scripting](https://redis.uptrace.dev/guide/lua-scripting.html).
|
||||
- [Redis Sentinel](https://redis.uptrace.dev/guide/go-redis-sentinel.html).
|
||||
- [Redis Cluster](https://redis.uptrace.dev/guide/go-redis-cluster.html).
|
||||
@@ -386,6 +389,118 @@ rdb := redis.NewClient(&redis.Options{
|
||||
})
|
||||
```
|
||||
|
||||
### Automatic pipelining
|
||||
|
||||
**Experimental** — the API may still change. Reach for autopipelining in
|
||||
high-throughput / high-load / scale scenarios; at low concurrency a plain
|
||||
client is simpler and just as fast. A runnable usage tour and throughput
|
||||
comparison live in [`example/autopipeline`](example/autopipeline).
|
||||
|
||||
> **EXPERIMENTAL:** the autopipelining API is subject to change in a future
|
||||
> release as we gather feedback — pin your go-redis version if you adopt it.
|
||||
|
||||
When many goroutines issue commands concurrently, autopipelining batches them
|
||||
into Redis pipelines automatically — without you writing any pipeline code. It
|
||||
comes in two faces:
|
||||
|
||||
- **`AutoPipeline()` — blocking, drop-in.** Each command call blocks until it
|
||||
executes and returns its own value/error, exactly like a normal client, so
|
||||
existing code keeps working unchanged. Under concurrency the engine coalesces
|
||||
commands from all goroutines into deep, back-to-back pipelines (a single
|
||||
ordered batch stream by default), reaching several times a plain client's
|
||||
executed commands per second in the same environment — roughly an order of
|
||||
magnitude with a parallel-batch config (`MaxConcurrentBatches` > 1 with
|
||||
`Unordered`). Per-goroutine ordering is preserved.
|
||||
- **`AsyncAutoPipeline()` — deferred, highest throughput.** Command calls return
|
||||
immediately; you submit a window of commands and read their results afterward,
|
||||
which keeps each pipeline deep — tens of times a plain client's throughput.
|
||||
Ordered by default. Absolute numbers depend heavily on the machine, network
|
||||
path and server; see `autopipeline_bench_README.md` for the benchmark
|
||||
methodology and multipliers.
|
||||
|
||||
```go
|
||||
rdb := redis.NewClient(&redis.Options{Addr: "localhost:6379"})
|
||||
defer rdb.Close()
|
||||
ctx := context.Background()
|
||||
|
||||
// Blocking face: drop-in for a normal client, batched under the hood.
|
||||
ap, err := rdb.AutoPipeline()
|
||||
if err != nil { // invalid AutoPipelineOptions, or the client is closed
|
||||
log.Fatal(err)
|
||||
}
|
||||
defer ap.Close()
|
||||
|
||||
var wg sync.WaitGroup
|
||||
for i := 0; i < 1000; i++ {
|
||||
wg.Add(1)
|
||||
go func(i int) {
|
||||
defer wg.Done()
|
||||
key := fmt.Sprintf("key:%d", i)
|
||||
if err := ap.Set(ctx, key, i, 0).Err(); err != nil { // blocks until executed
|
||||
log.Printf("set %s: %v", key, err)
|
||||
}
|
||||
}(i)
|
||||
}
|
||||
wg.Wait()
|
||||
```
|
||||
|
||||
For maximum throughput, submit a window on the async face and read later:
|
||||
|
||||
```go
|
||||
ctx := context.Background()
|
||||
ap, err := rdb.AsyncAutoPipeline() // ordered by default
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
defer ap.Close()
|
||||
|
||||
cmds := make([]*redis.StatusCmd, 0, 200)
|
||||
for i := 0; i < 200; i++ {
|
||||
cmds = append(cmds, ap.Set(ctx, fmt.Sprintf("key:%d", i), i, 0)) // returns immediately
|
||||
}
|
||||
for _, cmd := range cmds {
|
||||
if err := cmd.Err(); err != nil { // blocks until executed
|
||||
log.Printf("set: %v", err)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Each face has a no-argument form that uses `Options.AutoPipelineOptions` (or the
|
||||
built-in default) and a `WithOptions` form that takes an explicit
|
||||
`*AutoPipelineOptions`; both return `(*AutoPipeliner, error)` — the error is
|
||||
non-nil for an invalid config or a closed client (e.g.
|
||||
`ap, err := rdb.AsyncAutoPipelineWithOptions(&redis.AutoPipelineOptions{MaxConcurrentBatches: 8, Unordered: true})`);
|
||||
a handful of parallel batches saturates the link — more permits only add
|
||||
overlapping batches without deepening them.
|
||||
They work on `ClusterClient` too: commands are routed to the correct shard per
|
||||
key, so a single batch may span many slots; ordering across nodes is per key
|
||||
(same-key commands stay in order, different nodes' sub-pipelines run
|
||||
concurrently). Because batches share a few pipeline connections, autopipelining
|
||||
also needs far fewer connections than a plain client at the same concurrency
|
||||
(see `PipelinePoolSize`). Autopipelining is only a win under concurrency (or
|
||||
windowed submission) — a single goroutine issuing one blocking command at a
|
||||
time sees little benefit, and a hand-written `Pipeline()` is still fastest when
|
||||
you can batch by hand.
|
||||
|
||||
Caveats: a command's context is not honored once it is queued (batches execute
|
||||
on the autopipeliner's own context) — use a plain client for per-command
|
||||
deadlines. Blocking commands (`BLPOP`, `WAIT`, ...) are never batched and run
|
||||
directly on your context — as are `SHUTDOWN` and `MONITOR`, which would
|
||||
poison a shared pipeline connection — and `Do` also bypasses batching with plain
|
||||
`Client.Do` semantics — prefer the typed methods (`ap.Set`, `ap.Get`, ...). On
|
||||
a dropped connection a batch is retried whole (up to `MaxRetries`), so
|
||||
non-idempotent commands may execute twice. Both faces return a cached,
|
||||
client-shared instance: the first call's config wins and `Close` stops it for
|
||||
all callers. Hooks may read command results (the engine hands a hook running
|
||||
on the dispatch goroutine the same view a plain pipeline hook gets), but a
|
||||
hook must never issue a command on the same autopipeliner and wait for it —
|
||||
the nested command needs the very dispatch slot the hook is holding, and the
|
||||
engine only recovers by failing that flush after its 30s permit backstops.
|
||||
`Options.Limiter` is consulted once per batch dispatch (as with a manual
|
||||
pipeline), not once per command. An autopipeliner created on a
|
||||
`WithTimeout`/`WithReadTimeout` clone is not stopped by the parent's `Close` —
|
||||
close it explicitly.
|
||||
|
||||
### Advanced Configuration
|
||||
|
||||
go-redis supports extending the client identification phase to allow projects to send their own custom client identification.
|
||||
|
||||
@@ -0,0 +1,299 @@
|
||||
package redis
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// TestAdaptiveDelayCalculation tests the adaptive delay calculation logic
|
||||
func TestAdaptiveDelayCalculation(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
maxBatchSize int
|
||||
maxDelay time.Duration
|
||||
adaptive bool
|
||||
queueLen int
|
||||
expected time.Duration
|
||||
}{
|
||||
// Disabled adaptive delay
|
||||
{
|
||||
name: "adaptive disabled, returns fixed delay",
|
||||
maxBatchSize: 100,
|
||||
maxDelay: 100 * time.Microsecond,
|
||||
adaptive: false,
|
||||
queueLen: 50,
|
||||
expected: 100 * time.Microsecond,
|
||||
},
|
||||
{
|
||||
name: "adaptive disabled, zero delay",
|
||||
maxBatchSize: 100,
|
||||
maxDelay: 0,
|
||||
adaptive: false,
|
||||
queueLen: 50,
|
||||
expected: 0,
|
||||
},
|
||||
|
||||
// Enabled adaptive delay - 75% threshold
|
||||
{
|
||||
name: "75% full - no delay",
|
||||
maxBatchSize: 100,
|
||||
maxDelay: 100 * time.Microsecond,
|
||||
adaptive: true,
|
||||
queueLen: 75,
|
||||
expected: 0,
|
||||
},
|
||||
{
|
||||
name: "76% full - no delay",
|
||||
maxBatchSize: 100,
|
||||
maxDelay: 100 * time.Microsecond,
|
||||
adaptive: true,
|
||||
queueLen: 76,
|
||||
expected: 0,
|
||||
},
|
||||
{
|
||||
name: "100% full - no delay",
|
||||
maxBatchSize: 100,
|
||||
maxDelay: 100 * time.Microsecond,
|
||||
adaptive: true,
|
||||
queueLen: 100,
|
||||
expected: 0,
|
||||
},
|
||||
|
||||
// Enabled adaptive delay - 50% threshold
|
||||
{
|
||||
name: "50% full - 25% delay",
|
||||
maxBatchSize: 100,
|
||||
maxDelay: 100 * time.Microsecond,
|
||||
adaptive: true,
|
||||
queueLen: 50,
|
||||
expected: 25 * time.Microsecond,
|
||||
},
|
||||
{
|
||||
name: "60% full - 25% delay",
|
||||
maxBatchSize: 100,
|
||||
maxDelay: 100 * time.Microsecond,
|
||||
adaptive: true,
|
||||
queueLen: 60,
|
||||
expected: 25 * time.Microsecond,
|
||||
},
|
||||
{
|
||||
name: "74% full - 25% delay",
|
||||
maxBatchSize: 100,
|
||||
maxDelay: 100 * time.Microsecond,
|
||||
adaptive: true,
|
||||
queueLen: 74,
|
||||
expected: 25 * time.Microsecond,
|
||||
},
|
||||
|
||||
// Enabled adaptive delay - 25% threshold
|
||||
{
|
||||
name: "25% full - 50% delay",
|
||||
maxBatchSize: 100,
|
||||
maxDelay: 100 * time.Microsecond,
|
||||
adaptive: true,
|
||||
queueLen: 25,
|
||||
expected: 50 * time.Microsecond,
|
||||
},
|
||||
{
|
||||
name: "30% full - 50% delay",
|
||||
maxBatchSize: 100,
|
||||
maxDelay: 100 * time.Microsecond,
|
||||
adaptive: true,
|
||||
queueLen: 30,
|
||||
expected: 50 * time.Microsecond,
|
||||
},
|
||||
{
|
||||
name: "49% full - 50% delay",
|
||||
maxBatchSize: 100,
|
||||
maxDelay: 100 * time.Microsecond,
|
||||
adaptive: true,
|
||||
queueLen: 49,
|
||||
expected: 50 * time.Microsecond,
|
||||
},
|
||||
|
||||
// Enabled adaptive delay - <25% threshold
|
||||
{
|
||||
name: "24% full - 100% delay",
|
||||
maxBatchSize: 100,
|
||||
maxDelay: 100 * time.Microsecond,
|
||||
adaptive: true,
|
||||
queueLen: 24,
|
||||
expected: 100 * time.Microsecond,
|
||||
},
|
||||
{
|
||||
name: "10% full - 100% delay",
|
||||
maxBatchSize: 100,
|
||||
maxDelay: 100 * time.Microsecond,
|
||||
adaptive: true,
|
||||
queueLen: 10,
|
||||
expected: 100 * time.Microsecond,
|
||||
},
|
||||
{
|
||||
name: "1% full - 100% delay",
|
||||
maxBatchSize: 100,
|
||||
maxDelay: 100 * time.Microsecond,
|
||||
adaptive: true,
|
||||
queueLen: 1,
|
||||
expected: 100 * time.Microsecond,
|
||||
},
|
||||
|
||||
// Edge cases
|
||||
{
|
||||
name: "empty queue - no delay",
|
||||
maxBatchSize: 100,
|
||||
maxDelay: 100 * time.Microsecond,
|
||||
adaptive: true,
|
||||
queueLen: 0,
|
||||
expected: 0,
|
||||
},
|
||||
{
|
||||
name: "zero max delay - no delay",
|
||||
maxBatchSize: 100,
|
||||
maxDelay: 0,
|
||||
adaptive: true,
|
||||
queueLen: 50,
|
||||
expected: 0,
|
||||
},
|
||||
|
||||
// Different batch sizes
|
||||
{
|
||||
name: "small batch - 75% full",
|
||||
maxBatchSize: 10,
|
||||
maxDelay: 100 * time.Microsecond,
|
||||
adaptive: true,
|
||||
queueLen: 8,
|
||||
expected: 0,
|
||||
},
|
||||
{
|
||||
name: "small batch - 50% full",
|
||||
maxBatchSize: 10,
|
||||
maxDelay: 100 * time.Microsecond,
|
||||
adaptive: true,
|
||||
queueLen: 5,
|
||||
expected: 25 * time.Microsecond,
|
||||
},
|
||||
{
|
||||
name: "large batch - 75% full",
|
||||
maxBatchSize: 1000,
|
||||
maxDelay: 100 * time.Microsecond,
|
||||
adaptive: true,
|
||||
queueLen: 750,
|
||||
expected: 0,
|
||||
},
|
||||
{
|
||||
name: "large batch - 50% full",
|
||||
maxBatchSize: 1000,
|
||||
maxDelay: 100 * time.Microsecond,
|
||||
adaptive: true,
|
||||
queueLen: 500,
|
||||
expected: 25 * time.Microsecond,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
// Create autopipeliner with test config
|
||||
ap := &AutoPipeliner{
|
||||
config: &AutoPipelineOptions{
|
||||
MaxBatchSize: tt.maxBatchSize,
|
||||
MaxFlushDelay: tt.maxDelay,
|
||||
AdaptiveDelay: tt.adaptive,
|
||||
},
|
||||
}
|
||||
sh := &apShard{ap: ap, stripes: make([]apStripe, 1)}
|
||||
sh.stripes[0].queueLen.Store(int32(tt.queueLen))
|
||||
ap.shards = []*apShard{sh}
|
||||
|
||||
// Calculate delay
|
||||
result := ap.calculateDelay(tt.queueLen)
|
||||
|
||||
// Verify result
|
||||
if result != tt.expected {
|
||||
t.Errorf("calculateDelay() = %v, want %v", result, tt.expected)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestAdaptiveDelayIntegerArithmetic verifies integer arithmetic correctness
|
||||
func TestAdaptiveDelayIntegerArithmetic(t *testing.T) {
|
||||
maxBatch := 100
|
||||
maxDelay := 100 * time.Microsecond
|
||||
|
||||
ap := &AutoPipeliner{
|
||||
config: &AutoPipelineOptions{
|
||||
MaxBatchSize: maxBatch,
|
||||
MaxFlushDelay: maxDelay,
|
||||
AdaptiveDelay: true,
|
||||
},
|
||||
}
|
||||
sh := &apShard{ap: ap, stripes: make([]apStripe, 1)}
|
||||
ap.shards = []*apShard{sh}
|
||||
|
||||
for queueLen := 0; queueLen <= maxBatch; queueLen++ {
|
||||
sh.stripes[0].queueLen.Store(int32(queueLen))
|
||||
delay := ap.calculateDelay(queueLen)
|
||||
|
||||
// Verify delay is one of the expected values
|
||||
switch {
|
||||
case queueLen == 0:
|
||||
if delay != 0 {
|
||||
t.Errorf("queueLen=%d: expected 0, got %v", queueLen, delay)
|
||||
}
|
||||
case queueLen*4 >= maxBatch*3: // ≥75%
|
||||
if delay != 0 {
|
||||
t.Errorf("queueLen=%d (≥75%%): expected 0, got %v", queueLen, delay)
|
||||
}
|
||||
case queueLen*2 >= maxBatch: // ≥50%
|
||||
if delay != maxDelay/4 {
|
||||
t.Errorf("queueLen=%d (≥50%%): expected %v, got %v", queueLen, maxDelay/4, delay)
|
||||
}
|
||||
case queueLen*4 >= maxBatch: // ≥25%
|
||||
if delay != maxDelay/2 {
|
||||
t.Errorf("queueLen=%d (≥25%%): expected %v, got %v", queueLen, maxDelay/2, delay)
|
||||
}
|
||||
default: // <25%
|
||||
if delay != maxDelay {
|
||||
t.Errorf("queueLen=%d (<25%%): expected %v, got %v", queueLen, maxDelay, delay)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// BenchmarkCalculateDelay benchmarks the delay calculation
|
||||
func BenchmarkCalculateDelay(b *testing.B) {
|
||||
ap := &AutoPipeliner{
|
||||
config: &AutoPipelineOptions{
|
||||
MaxBatchSize: 100,
|
||||
MaxFlushDelay: 100 * time.Microsecond,
|
||||
AdaptiveDelay: true,
|
||||
},
|
||||
}
|
||||
sh := &apShard{ap: ap, stripes: make([]apStripe, 1)}
|
||||
sh.stripes[0].queueLen.Store(50)
|
||||
ap.shards = []*apShard{sh}
|
||||
|
||||
b.ResetTimer()
|
||||
for i := 0; i < b.N; i++ {
|
||||
_ = ap.calculateDelay(sh.Len())
|
||||
}
|
||||
}
|
||||
|
||||
// BenchmarkCalculateDelayDisabled benchmarks with adaptive delay disabled
|
||||
func BenchmarkCalculateDelayDisabled(b *testing.B) {
|
||||
ap := &AutoPipeliner{
|
||||
config: &AutoPipelineOptions{
|
||||
MaxBatchSize: 100,
|
||||
MaxFlushDelay: 100 * time.Microsecond,
|
||||
AdaptiveDelay: false,
|
||||
},
|
||||
}
|
||||
sh := &apShard{ap: ap, stripes: make([]apStripe, 1)}
|
||||
sh.stripes[0].queueLen.Store(50)
|
||||
ap.shards = []*apShard{sh}
|
||||
|
||||
b.ResetTimer()
|
||||
for i := 0; i < b.N; i++ {
|
||||
_ = ap.calculateDelay(sh.Len())
|
||||
}
|
||||
}
|
||||
@@ -14,16 +14,19 @@ import (
|
||||
|
||||
var _ = Describe("Array Commands", Label("array"), func() {
|
||||
ctx := context.TODO()
|
||||
var client *redis.Client
|
||||
var client redis.UniversalClient
|
||||
var rawClient *redis.Client
|
||||
var closeSubject func() error
|
||||
|
||||
BeforeEach(func() {
|
||||
SkipBeforeRedisVersion("8.8", "Redis 8.8.0 introduces support for Array")
|
||||
client = redis.NewClient(redisOptions())
|
||||
Expect(client.FlushDB(ctx).Err()).NotTo(HaveOccurred())
|
||||
rawClient = redis.NewClient(redisOptions())
|
||||
client, closeSubject = newUniversalSubject(rawClient)
|
||||
Expect(rawClient.FlushDB(ctx).Err()).NotTo(HaveOccurred())
|
||||
})
|
||||
|
||||
AfterEach(func() {
|
||||
Expect(client.Close()).NotTo(HaveOccurred())
|
||||
Expect(closeSubject()).NotTo(HaveOccurred())
|
||||
})
|
||||
|
||||
Describe("ARSet and ARGet", func() {
|
||||
|
||||
+2609
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,128 @@
|
||||
# Autopipelining benchmarks
|
||||
|
||||
These benchmarks validate the goal of autopipelining: batching concurrent
|
||||
commands into pipelines cuts network round-trips and raises throughput, without
|
||||
callers writing pipeline code.
|
||||
|
||||
Two measurement rules keep every number honest:
|
||||
|
||||
- **Throughput is measured on executed commands** — a command is counted only
|
||||
after its result has been read (`.Result()` / `.Err()`), never when it is
|
||||
merely queued. On the deferred face `ap.Set` returns immediately, so counting
|
||||
calls would measure enqueue speed, not throughput.
|
||||
- **Rates divide by the timed region, not the nominal window.** The
|
||||
fixed-duration drivers keep draining whatever was in flight when the deadline
|
||||
hit; that drain is part of the timed region, so `ops/sec` cannot be inflated
|
||||
by work that finished after the window closed.
|
||||
|
||||
## Running
|
||||
|
||||
The benchmarks talk to a real Redis on `:6379` (they skip when none answers).
|
||||
Start one first:
|
||||
|
||||
```sh
|
||||
make docker.start # or: docker run --rm -p 6379:6379 redis
|
||||
|
||||
# the headline three-way throughput comparison:
|
||||
go test -run '^$' -bench BenchmarkAutoPipelineThroughput -benchtime=1x .
|
||||
|
||||
# everything:
|
||||
go test -run '^$' -bench Benchmark -benchmem -benchtime=1x .
|
||||
```
|
||||
|
||||
The throughput benchmarks run for a fixed wall-clock duration (~3s each) and
|
||||
report `ops/sec`, so `-benchtime=1x` (one iteration) is correct for them.
|
||||
**Do not pass a time-based `-benchtime`** (e.g. `-benchtime=5s`): the
|
||||
fixed-duration drivers ignore `b.N`, so Go's framework would re-run the full
|
||||
window geometrically trying to fill the time budget.
|
||||
|
||||
The per-operation benchmarks (`BenchmarkDispatchPath`,
|
||||
`BenchmarkAutoPipelineZeroCopy`) are the opposite: their ns/op and allocs/op
|
||||
are only meaningful at the **default** `-benchtime` — at `-benchtime=1x` the
|
||||
workers still issue a minimum window each, all billed to a single iteration.
|
||||
(The "everything" command above also sweeps the repo's other root-package
|
||||
benchmarks; that is harmless, just broader than this file.)
|
||||
|
||||
## The headline benchmark: BenchmarkAutoPipelineThroughput
|
||||
|
||||
Three ways to issue the same workload (2000 goroutines), each counting only
|
||||
executed commands:
|
||||
|
||||
1. **Normal** — a plain client; each `Set` is a blocking round-trip. Bounded by
|
||||
Redis's non-pipelined ceiling (like `redis-benchmark` without `-P`).
|
||||
2. **AutoPipelineBlocking** — the blocking face with a parallel-batch config:
|
||||
`ap.Set(...)` blocks until executed, the same call shape as a normal client.
|
||||
Only one command per caller is in flight, but the flusher batches across the
|
||||
2000 callers into deep pipelines.
|
||||
3. **AutoPipelineWindowed** — the deferred face: each caller submits a window of
|
||||
200 commands, then reads the results. Keeps pipelines deepest.
|
||||
|
||||
The `WindowedGET` variant repeats (3) with GET instead of SET: SET throughput is
|
||||
server-bound (Redis's write processing), GET is cheaper on the server, so the
|
||||
GET number shows the client machinery itself is not the limit.
|
||||
|
||||
**Absolute numbers are machine- and load-dependent and vary a lot** — CPU
|
||||
count, Redis's own ceiling, network path (loopback vs docker veth vs real
|
||||
network), and noisy neighbors all move them by integer factors. The signal is
|
||||
the WITHIN-RUN multiplier against the `Normal` baseline measured in the same
|
||||
environment, plus `allocs/op` (which is exact and stable everywhere):
|
||||
|
||||
| variant | vs Normal (same run) |
|
||||
| ----------------------- | ---------------------- |
|
||||
| Normal | 1x (the baseline) |
|
||||
| AutoPipelineBlocking | ~10x |
|
||||
| AutoPipelineWindowed | ~25-30x |
|
||||
| AutoPipelineWindowedGET | above Windowed (reads) |
|
||||
|
||||
As one concrete example: an Apple Silicon laptop with a loopback Redis puts
|
||||
`Normal` around 80k ops/sec (so ~800k blocking, ~2.5M windowed); a 4-vCPU CI
|
||||
runner with dockerized Redis lands near half that on the CPU-bound variants —
|
||||
different absolutes, same multipliers and ordering.
|
||||
|
||||
The autopipeline variants use an explicit parallel-batch config
|
||||
(`MaxBatchSize: 300, MaxConcurrentBatches: 80, Unordered: true`) — **not the
|
||||
ordered default**. The default (`MaxConcurrentBatches: 1`,
|
||||
`DefaultAutoPipelineOptions` / `DefaultBlockingAutoPipelineOptions`) serializes
|
||||
batch execution: blocking usage lands at roughly half the parallel-batch
|
||||
multiplier, while windowed submission stays well into the tens-of-x even
|
||||
ordered.
|
||||
|
||||
## The other benchmarks
|
||||
|
||||
- **BenchmarkIndividualCommands** — plain-client baseline: one blocking
|
||||
round-trip per command across GOMAXPROCS workers. Its ns/op is your
|
||||
environment's RTT floor; every other number is best read against it.
|
||||
- **BenchmarkManualPipeline** — hand-built 100-deep `Pipeline().Exec()`,
|
||||
sequential: the per-command cost of explicit pipelining (roughly a tenth
|
||||
of a round-trip per command). The ceiling autopipelining approaches
|
||||
without anyone writing pipeline code.
|
||||
- **BenchmarkDispatchPath** — the engine's per-command dispatch cost with
|
||||
honest `b.N` accounting: ns/op and allocs/op per executed command
|
||||
(4 allocs/cmd on the submit path; unordered dispatch roughly halves the
|
||||
ordered ns/op), plus the lone-command blocking fast path (~1 RTT).
|
||||
- **BenchmarkFutureFace** — the typed future face on the ordered default
|
||||
config: per-command reads (`InOrder`) vs windowed reads (`Window200`,
|
||||
roughly 2x InOrder).
|
||||
- **BenchmarkAutoPipelineSubmit** — the non-blocking `Submit` entry point,
|
||||
windowed, on the ordered default; lands in the same band as
|
||||
`FutureFace/Window200`.
|
||||
- **BenchmarkAutoPipelineZeroCopy** — `GetToBuffer`/`SetFromBuffer` vs regular
|
||||
`Get`/`Set` (Set+Get pairs): B/op drops ~10x at 4KiB and ~90x at 64KiB
|
||||
(payloads decode into the caller's buffer instead of fresh strings), with
|
||||
throughput at parity or better; allocs/op 10 vs 11. The B/op and allocs/op
|
||||
ratios are environment-independent.
|
||||
- **BenchmarkClusterAutoPipelineThroughput** — the same blocking/windowed
|
||||
drivers against a local 3-master cluster (slot-routed shard batches keep
|
||||
per-node pipelines deep; scales past the standalone numbers in the same
|
||||
environment, with a wide run-order-dependent spread). Skips when no
|
||||
cluster answers on `:16600-16602`.
|
||||
|
||||
## What was deliberately removed
|
||||
|
||||
Earlier revisions carried "tuning sweep" benchmarks (batch sizes, flush
|
||||
delays, buffer sizes) whose numbers were dominated by the configured
|
||||
`MaxFlushDelay` timer at low parallelism — every swept value reported the same
|
||||
timer readout, which could only mislead someone tuning from them. They were
|
||||
removed rather than fixed: `BenchmarkDispatchPath` and the throughput drivers
|
||||
cover the engine's real knobs. Tune with your own workload shape; the engine's
|
||||
defaults need no tuning to hit the numbers above.
|
||||
@@ -0,0 +1,683 @@
|
||||
package redis_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"strconv"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/redis/go-redis/v9"
|
||||
)
|
||||
|
||||
// benchAddr is the standalone server the benchmarks run against — the same
|
||||
// server the docker-compose env exposes in CI (and `redis-benchmark`'s
|
||||
// default), so numbers are comparable with the reference bands in
|
||||
// autopipeline_bench_README.md.
|
||||
const benchAddr = ":6379"
|
||||
|
||||
// skipWithoutBenchServer skips the benchmark when no server answers, instead
|
||||
// of "passing" while measuring dial-failure latency.
|
||||
func skipWithoutBenchServer(b *testing.B) {
|
||||
b.Helper()
|
||||
probe := redis.NewClient(&redis.Options{Addr: benchAddr})
|
||||
defer probe.Close()
|
||||
if err := probe.Ping(context.Background()).Err(); err != nil {
|
||||
b.Skipf("no redis at %s: %v", benchAddr, err)
|
||||
}
|
||||
}
|
||||
|
||||
// BenchmarkIndividualCommands is the plain-client baseline: one blocking
|
||||
// round-trip per command across GOMAXPROCS workers.
|
||||
func BenchmarkIndividualCommands(b *testing.B) {
|
||||
skipWithoutBenchServer(b)
|
||||
ctx := context.Background()
|
||||
client := redis.NewClient(&redis.Options{
|
||||
Addr: benchAddr,
|
||||
})
|
||||
defer client.Close()
|
||||
|
||||
b.ResetTimer()
|
||||
b.RunParallel(func(pb *testing.PB) {
|
||||
i := 0
|
||||
for pb.Next() {
|
||||
key := fmt.Sprintf("key%d", i)
|
||||
if err := client.Set(ctx, key, i, 0).Err(); err != nil {
|
||||
b.Error(err)
|
||||
return
|
||||
}
|
||||
i++
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// BenchmarkManualPipeline benchmarks using manual pipelining. Deliberately
|
||||
// sequential (one goroutine, one pipeline at a time): it isolates the
|
||||
// per-batch cost of an explicit Pipeline().Exec() round-trip, so its numbers
|
||||
// are not directly comparable to the parallel BenchmarkIndividualCommands
|
||||
// above. For a like-for-like concurrent comparison of the client faces see
|
||||
// BenchmarkAutoPipelineThroughput, whose variants all share one harness.
|
||||
func BenchmarkManualPipeline(b *testing.B) {
|
||||
skipWithoutBenchServer(b)
|
||||
ctx := context.Background()
|
||||
client := redis.NewClient(&redis.Options{
|
||||
Addr: benchAddr,
|
||||
})
|
||||
defer client.Close()
|
||||
|
||||
const batchSize = 100
|
||||
|
||||
b.ResetTimer()
|
||||
|
||||
for i := 0; i < b.N; i += batchSize {
|
||||
pipe := client.Pipeline()
|
||||
|
||||
end := i + batchSize
|
||||
if end > b.N {
|
||||
end = b.N
|
||||
}
|
||||
|
||||
for j := i; j < end; j++ {
|
||||
key := fmt.Sprintf("key%d", j)
|
||||
pipe.Set(ctx, key, j, 0)
|
||||
}
|
||||
|
||||
if _, err := pipe.Exec(ctx); err != nil {
|
||||
b.Fatal(err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// BenchmarkFutureFace measures the uniform typed future face two ways:
|
||||
// - InOrder: fap.Set(...).Err() per command (blocking-like ergonomics).
|
||||
// - Window: submit a window of typed commands, then read them (max throughput).
|
||||
//
|
||||
// Both use the same fap.Set typed method; only when the result is read differs.
|
||||
// Fixed-duration driver: goroutines issue commands until the deadline and the
|
||||
// metric divides by the TIMED region (which includes draining the windows
|
||||
// already in flight at the deadline), so the reported ops/sec is honest.
|
||||
// NOTE: fixed-duration drivers ignore b.N — run them with the default
|
||||
// -benchtime or -benchtime=1x, not a time-based value (which would re-run the
|
||||
// full window geometrically).
|
||||
func BenchmarkFutureFace(b *testing.B) {
|
||||
run := func(b *testing.B, window int) {
|
||||
skipWithoutBenchServer(b)
|
||||
ctx := context.Background()
|
||||
c := redis.NewClient(&redis.Options{Addr: benchAddr, PoolSize: 250})
|
||||
defer c.Close()
|
||||
fap, err := c.AsyncAutoPipeline() // windowed submit-then-read is the deferred pattern
|
||||
if err != nil {
|
||||
b.Fatal(err)
|
||||
}
|
||||
defer fap.Close()
|
||||
|
||||
const duration = 3 * time.Second
|
||||
goroutines := 2000
|
||||
if window > 1 {
|
||||
goroutines = 500
|
||||
}
|
||||
var count int64
|
||||
deadline := time.Now().Add(duration)
|
||||
|
||||
b.ResetTimer()
|
||||
var wg sync.WaitGroup
|
||||
wg.Add(goroutines)
|
||||
for g := 0; g < goroutines; g++ {
|
||||
go func(id int) {
|
||||
defer wg.Done()
|
||||
key := "k" + strconv.Itoa(id)
|
||||
cs := make([]*redis.StatusCmd, 0, window)
|
||||
for time.Now().Before(deadline) {
|
||||
cs = cs[:0]
|
||||
for j := 0; j < window; j++ {
|
||||
cs = append(cs, fap.Set(ctx, key, j, 0))
|
||||
}
|
||||
for _, cc := range cs {
|
||||
if err := cc.Err(); err != nil {
|
||||
b.Error(err)
|
||||
return
|
||||
}
|
||||
}
|
||||
atomic.AddInt64(&count, int64(window))
|
||||
}
|
||||
}(g)
|
||||
}
|
||||
wg.Wait()
|
||||
b.StopTimer()
|
||||
b.ReportMetric(float64(count)/b.Elapsed().Seconds(), "ops/sec")
|
||||
}
|
||||
|
||||
b.Run("InOrder", func(b *testing.B) { run(b, 1) })
|
||||
b.Run("Window200", func(b *testing.B) { run(b, 200) })
|
||||
}
|
||||
|
||||
// BenchmarkAutoPipelineSubmit measures the non-blocking Submit path. Each
|
||||
// goroutine submits a window of commands without blocking, then waits on them,
|
||||
// so it pays one park/unpark round-trip per window instead of per command.
|
||||
// Fixed-duration driver; see the BenchmarkFutureFace note about -benchtime.
|
||||
func BenchmarkAutoPipelineSubmit(b *testing.B) {
|
||||
skipWithoutBenchServer(b)
|
||||
ctx := context.Background()
|
||||
client := redis.NewClient(&redis.Options{Addr: benchAddr, PoolSize: 250})
|
||||
defer client.Close()
|
||||
ap, err := client.AsyncAutoPipeline() // Submit is only allowed on the deferred face
|
||||
if err != nil {
|
||||
b.Fatal(err)
|
||||
}
|
||||
defer ap.Close()
|
||||
|
||||
const (
|
||||
duration = 3 * time.Second
|
||||
goroutines = 500
|
||||
window = 200
|
||||
)
|
||||
var count int64
|
||||
deadline := time.Now().Add(duration)
|
||||
|
||||
b.ResetTimer()
|
||||
var wg sync.WaitGroup
|
||||
wg.Add(goroutines)
|
||||
for g := 0; g < goroutines; g++ {
|
||||
go func(id int) {
|
||||
defer wg.Done()
|
||||
futs := make([]redis.AutoFuture, 0, window)
|
||||
key := fmt.Sprintf("submit:%d", id)
|
||||
for time.Now().Before(deadline) {
|
||||
futs = futs[:0]
|
||||
for j := 0; j < window; j++ {
|
||||
futs = append(futs, ap.Submit(ctx, redis.NewStatusCmd(ctx, "set", key, j)))
|
||||
}
|
||||
for i := range futs {
|
||||
if err := futs[i].Wait(); err != nil {
|
||||
b.Error(err)
|
||||
return
|
||||
}
|
||||
}
|
||||
atomic.AddInt64(&count, window)
|
||||
}
|
||||
}(g)
|
||||
}
|
||||
wg.Wait()
|
||||
b.StopTimer()
|
||||
b.ReportMetric(float64(count)/b.Elapsed().Seconds(), "ops/sec")
|
||||
}
|
||||
|
||||
// clusterBenchAddrs is the local osscluster (docker-compose `cluster` profile,
|
||||
// formed with `redis-cli --cluster create ... --cluster-replicas 1`).
|
||||
var clusterBenchAddrs = []string{"127.0.0.1:16600", "127.0.0.1:16601", "127.0.0.1:16602"}
|
||||
|
||||
func newClusterBenchClient(b *testing.B) *redis.ClusterClient {
|
||||
b.Helper()
|
||||
c := redis.NewClusterClient(&redis.ClusterOptions{Addrs: clusterBenchAddrs, PoolSize: 500})
|
||||
if err := c.Ping(context.Background()).Err(); err != nil {
|
||||
c.Close()
|
||||
b.Skipf("cluster not reachable on %v: %v", clusterBenchAddrs, err)
|
||||
}
|
||||
return c
|
||||
}
|
||||
|
||||
// BenchmarkClusterAutoPipelineThroughput measures executed throughput against a
|
||||
// 3-master cluster. ClusterClient.AutoPipeline routes commands to shards by slot
|
||||
// so each shard's batch stays on one node (keeping per-node pipelines deep), which
|
||||
// is what lets a cluster scale past a single instance. As with the standalone
|
||||
// benchmark, a command is counted only after its result is read, and the metric
|
||||
// divides by the timed region (including the post-deadline drain of windows
|
||||
// already in flight), so the reported ops/sec is honest.
|
||||
// Fixed-duration driver; see the BenchmarkFutureFace note about -benchtime.
|
||||
func BenchmarkClusterAutoPipelineThroughput(b *testing.B) {
|
||||
const duration = 3 * time.Second
|
||||
|
||||
b.Run("Blocking", func(b *testing.B) {
|
||||
c := newClusterBenchClient(b)
|
||||
defer c.Close()
|
||||
ap, err := c.AutoPipelineWithOptions(&redis.AutoPipelineOptions{MaxBatchSize: 512, MaxConcurrentBatches: 200, Unordered: true})
|
||||
if err != nil {
|
||||
b.Fatal(err)
|
||||
}
|
||||
defer ap.Close()
|
||||
const G = 2000
|
||||
var count int64
|
||||
dl := time.Now().Add(duration)
|
||||
ctx := context.Background()
|
||||
b.ResetTimer()
|
||||
var wg sync.WaitGroup
|
||||
wg.Add(G)
|
||||
for g := 0; g < G; g++ {
|
||||
go func(id int) {
|
||||
defer wg.Done()
|
||||
i, key := 0, fmt.Sprintf("b:%d", id)
|
||||
for time.Now().Before(dl) {
|
||||
const run = 50
|
||||
for r := 0; r < run; r++ {
|
||||
i++
|
||||
if err := ap.Set(ctx, key, i, 0).Err(); err != nil {
|
||||
b.Error(err)
|
||||
return
|
||||
}
|
||||
}
|
||||
atomic.AddInt64(&count, run)
|
||||
}
|
||||
}(g)
|
||||
}
|
||||
wg.Wait()
|
||||
b.StopTimer()
|
||||
b.ReportMetric(float64(count)/b.Elapsed().Seconds(), "ops/sec")
|
||||
})
|
||||
|
||||
b.Run("Windowed", func(b *testing.B) {
|
||||
c := newClusterBenchClient(b)
|
||||
defer c.Close()
|
||||
ap, err := c.AsyncAutoPipelineWithOptions(&redis.AutoPipelineOptions{MaxBatchSize: 300, MaxConcurrentBatches: 96, Unordered: true})
|
||||
if err != nil {
|
||||
b.Fatal(err)
|
||||
}
|
||||
defer ap.Close()
|
||||
const G, W = 1000, 300
|
||||
var count int64
|
||||
dl := time.Now().Add(duration)
|
||||
ctx := context.Background()
|
||||
b.ResetTimer()
|
||||
var wg sync.WaitGroup
|
||||
wg.Add(G)
|
||||
for g := 0; g < G; g++ {
|
||||
go func(id int) {
|
||||
defer wg.Done()
|
||||
cmds := make([]*redis.StatusCmd, 0, W)
|
||||
for time.Now().Before(dl) {
|
||||
cmds = cmds[:0]
|
||||
for j := 0; j < W; j++ {
|
||||
cmds = append(cmds, ap.Set(ctx, fmt.Sprintf("w:%d:%d", id, j), j, 0))
|
||||
}
|
||||
for _, cm := range cmds {
|
||||
if err := cm.Err(); err != nil {
|
||||
b.Error(err)
|
||||
return
|
||||
}
|
||||
}
|
||||
atomic.AddInt64(&count, int64(W))
|
||||
}
|
||||
}(g)
|
||||
}
|
||||
wg.Wait()
|
||||
b.StopTimer()
|
||||
b.ReportMetric(float64(count)/b.Elapsed().Seconds(), "ops/sec")
|
||||
})
|
||||
}
|
||||
|
||||
// benchWindowed measures the per-command dispatch cost of the deferred face
|
||||
// with honest b.N accounting: ns/op and allocs/op are per executed command.
|
||||
func benchWindowed(b *testing.B, cfg *redis.AutoPipelineOptions, goroutines, window int) {
|
||||
skipWithoutBenchServer(b)
|
||||
ctx := context.Background()
|
||||
client := redis.NewClient(&redis.Options{Addr: benchAddr, PoolSize: 100})
|
||||
defer client.Close()
|
||||
ap, err := client.AsyncAutoPipelineWithOptions(cfg)
|
||||
if err != nil {
|
||||
b.Fatal(err)
|
||||
}
|
||||
defer ap.Close()
|
||||
|
||||
// one warmup command so pools/conns exist
|
||||
if err := ap.Set(ctx, "dispatch:warm", 1, 0).Err(); err != nil {
|
||||
b.Fatal(err)
|
||||
}
|
||||
|
||||
// Exact b.N partitioning: the first b.N%goroutines workers run one extra
|
||||
// command, so the total is b.N and ns/op / allocs/op are per command at
|
||||
// any -benchtime (a floor-and-min-1 split would over-run for small b.N
|
||||
// and drop the remainder for large).
|
||||
base, extra := b.N/goroutines, b.N%goroutines
|
||||
b.ReportAllocs()
|
||||
b.ResetTimer()
|
||||
var wg sync.WaitGroup
|
||||
wg.Add(goroutines)
|
||||
for g := 0; g < goroutines; g++ {
|
||||
go func(id int) {
|
||||
defer wg.Done()
|
||||
per := base
|
||||
if id < extra {
|
||||
per++
|
||||
}
|
||||
futs := make([]redis.AutoFuture, 0, window)
|
||||
cmds := 0
|
||||
for cmds < per {
|
||||
futs = futs[:0]
|
||||
n := window
|
||||
if rem := per - cmds; rem < n {
|
||||
n = rem
|
||||
}
|
||||
for k := 0; k < n; k++ {
|
||||
futs = append(futs, ap.Submit(ctx, redis.NewCmd(ctx, "set", "dispatch:k", k)))
|
||||
}
|
||||
for i := range futs {
|
||||
if err := futs[i].Wait(); err != nil {
|
||||
b.Error(err)
|
||||
}
|
||||
}
|
||||
cmds += n
|
||||
}
|
||||
}(g)
|
||||
}
|
||||
wg.Wait()
|
||||
b.StopTimer()
|
||||
}
|
||||
|
||||
func BenchmarkDispatchPath(b *testing.B) {
|
||||
b.Run("ordered/g64_w100", func(b *testing.B) {
|
||||
benchWindowed(b, &redis.AutoPipelineOptions{MaxBatchSize: 300}, 64, 100)
|
||||
})
|
||||
b.Run("unordered/g64_w100", func(b *testing.B) {
|
||||
benchWindowed(b, &redis.AutoPipelineOptions{
|
||||
MaxBatchSize: 300, MaxConcurrentBatches: 8, Unordered: true,
|
||||
}, 64, 100)
|
||||
})
|
||||
b.Run("solo/blocking", func(b *testing.B) {
|
||||
skipWithoutBenchServer(b)
|
||||
ctx := context.Background()
|
||||
client := redis.NewClient(&redis.Options{Addr: benchAddr})
|
||||
defer client.Close()
|
||||
ap, err := client.AutoPipeline()
|
||||
if err != nil {
|
||||
b.Fatal(err)
|
||||
}
|
||||
defer ap.Close()
|
||||
if err := ap.Set(ctx, "dispatch:warm", 1, 0).Err(); err != nil {
|
||||
b.Fatal(err)
|
||||
}
|
||||
b.ReportAllocs()
|
||||
b.ResetTimer()
|
||||
for i := 0; i < b.N; i++ {
|
||||
_ = ap.Set(ctx, "dispatch:k", i, 0).Err()
|
||||
}
|
||||
b.StopTimer()
|
||||
})
|
||||
}
|
||||
|
||||
// BenchmarkAutoPipelineZeroCopy compares regular Get/Set against the zero-copy
|
||||
// GetToBuffer/SetFromBuffer commands, both issued through the autopipeliner.
|
||||
// Each iteration is a Set+Get pair (so throughput mixes both directions).
|
||||
//
|
||||
// What zero-copy buys, per this benchmark's own output: B/op drops ~10x at
|
||||
// 4KiB and ~90x at 64KiB (the payload is decoded into the caller's buffer
|
||||
// instead of a freshly allocated string; allocs/op only goes 11 -> 10), with
|
||||
// throughput at parity (the fast path reads payload+CRLF in a single socket
|
||||
// read when the buffer has len >= n+2, as allocated here). The per-goroutine
|
||||
// read buffers are allocated OUTSIDE the timed region so B/op reflects the
|
||||
// command path, not the harness.
|
||||
func BenchmarkAutoPipelineZeroCopy(b *testing.B) {
|
||||
sizes := []int{64, 4096, 65536}
|
||||
for _, sz := range sizes {
|
||||
for _, zc := range []bool{false, true} {
|
||||
name := fmt.Sprintf("size=%d/regular", sz)
|
||||
if zc {
|
||||
name = fmt.Sprintf("size=%d/zerocopy", sz)
|
||||
}
|
||||
b.Run(name, func(b *testing.B) {
|
||||
skipWithoutBenchServer(b)
|
||||
ctx := context.Background()
|
||||
c := redis.NewClient(&redis.Options{
|
||||
Addr: benchAddr,
|
||||
PoolSize: 10,
|
||||
// Generous deadlines: at 64KiB payloads a slow CI runner
|
||||
// pushes gigabytes through few conns; the default 3s
|
||||
// write deadline trips under that load and the bench
|
||||
// (correctly) fails on the surfaced error.
|
||||
ReadTimeout: 30 * time.Second,
|
||||
WriteTimeout: 30 * time.Second,
|
||||
AutoPipelineOptions: &redis.AutoPipelineOptions{
|
||||
MaxBatchSize: 300,
|
||||
MaxConcurrentBatches: 80,
|
||||
Unordered: true,
|
||||
// Bound each pipeline write burst: 300 x 64KiB in one
|
||||
// batch is ~19MB down a single conn before any reply
|
||||
// is read — enough to stall a constrained CI runner
|
||||
// past its write deadline.
|
||||
MaxBatchBytes: 2 << 20,
|
||||
},
|
||||
})
|
||||
defer c.Close()
|
||||
c.FlushDB(ctx)
|
||||
ap, err := c.AutoPipeline()
|
||||
if err != nil {
|
||||
b.Fatal(err)
|
||||
}
|
||||
defer ap.Close()
|
||||
|
||||
payload := make([]byte, sz)
|
||||
for i := range payload {
|
||||
payload[i] = 'x'
|
||||
}
|
||||
pstr := string(payload)
|
||||
|
||||
// Fewer workers at large payloads: 500 goroutines x 64KiB
|
||||
// saturates a CI runner's socket budget and only measures
|
||||
// queueing on the write deadline.
|
||||
goroutines := 500
|
||||
if sz >= 65536 {
|
||||
goroutines = 100
|
||||
}
|
||||
// Exact b.N partitioning (see benchWindowed): each unit is
|
||||
// one Set+Get pair.
|
||||
base, extra := b.N/goroutines, b.N%goroutines
|
||||
var count int64
|
||||
|
||||
// len >= sz+2 opts into GetToBuffer's fast path (payload+CRLF
|
||||
// in a single socket read; the 2 bytes after the payload are
|
||||
// scratch). Allocated before ResetTimer so the harness buffers
|
||||
// don't pollute B/op.
|
||||
rbufs := make([][]byte, goroutines)
|
||||
for i := range rbufs {
|
||||
rbufs[i] = make([]byte, sz+2)
|
||||
}
|
||||
|
||||
b.ResetTimer()
|
||||
b.ReportAllocs()
|
||||
var wg sync.WaitGroup
|
||||
wg.Add(goroutines)
|
||||
for g := 0; g < goroutines; g++ {
|
||||
go func(id int) {
|
||||
defer wg.Done()
|
||||
per := base
|
||||
if id < extra {
|
||||
per++
|
||||
}
|
||||
rbuf := rbufs[id]
|
||||
for i := 0; i < per; i++ {
|
||||
key := fmt.Sprintf("zc:%d:%d", id, i)
|
||||
if zc {
|
||||
if err := ap.SetFromBuffer(ctx, key, payload).Err(); err != nil {
|
||||
b.Error(err)
|
||||
return
|
||||
}
|
||||
if err := ap.GetToBuffer(ctx, key, rbuf).Err(); err != nil {
|
||||
b.Error(err)
|
||||
return
|
||||
}
|
||||
} else {
|
||||
if err := ap.Set(ctx, key, pstr, 0).Err(); err != nil {
|
||||
b.Error(err)
|
||||
return
|
||||
}
|
||||
if err := ap.Get(ctx, key).Err(); err != nil {
|
||||
b.Error(err)
|
||||
return
|
||||
}
|
||||
}
|
||||
atomic.AddInt64(&count, 2)
|
||||
}
|
||||
}(g)
|
||||
}
|
||||
wg.Wait()
|
||||
b.StopTimer()
|
||||
b.ReportMetric(float64(count)/b.Elapsed().Seconds(), "ops/sec")
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// BenchmarkAutoPipelineThroughput compares executed-command throughput three
|
||||
// ways. In every variant a command is counted ONLY after its result has been
|
||||
// read (the command actually executed on the server) — there is no counting of
|
||||
// merely-queued commands — and the metric divides by the TIMED region, which
|
||||
// includes draining whatever was in flight at the deadline, so the reported
|
||||
// ops/sec is real throughput.
|
||||
//
|
||||
// 1. Normal — a plain client. Each Set is a blocking round-trip; throughput is
|
||||
// bounded by the connection pool and Redis's non-pipelined ceiling
|
||||
// (matching redis-benchmark without -P).
|
||||
// 2. AutoPipelineBlocking — ap.Set(...).Err() read immediately, the way the
|
||||
// normal client is used (drop-in, one command in flight per caller). The
|
||||
// flusher batches across the many concurrent callers into deep pipelines.
|
||||
// 3. AutoPipelineWindowed — submit a window of commands per caller, then read
|
||||
// their results. Keeps each pipeline deepest; the high-throughput usage.
|
||||
//
|
||||
// The autopipeline variants use a parallel-batch config (MaxConcurrentBatches>1,
|
||||
// Unordered) — NOT the ordered default. The default (MaxConcurrentBatches=1)
|
||||
// serializes batch execution and reaches roughly half the blocking number;
|
||||
// windowed submission stays in the millions even ordered.
|
||||
//
|
||||
// Fixed-duration drivers; see the BenchmarkFutureFace note about -benchtime.
|
||||
// Run: go test -run '^$' -bench BenchmarkAutoPipelineThroughput -benchtime=1x
|
||||
func BenchmarkAutoPipelineThroughput(b *testing.B) {
|
||||
const (
|
||||
duration = 3 * time.Second
|
||||
goroutines = 2000
|
||||
window = 200 // commands submitted before reading results (windowed variant)
|
||||
)
|
||||
|
||||
// apConfig is a parallel-batch config: many batches execute concurrently so
|
||||
// blocking callers don't serialize behind a single flusher. Unordered is
|
||||
// required for MaxConcurrentBatches>1.
|
||||
apConfig := func() *redis.AutoPipelineOptions {
|
||||
return &redis.AutoPipelineOptions{MaxBatchSize: 300, MaxConcurrentBatches: 80, Unordered: true}
|
||||
}
|
||||
|
||||
// drive runs `fn` on `goroutines` goroutines until the deadline and reports
|
||||
// executed ops/sec over the timed region. fn returns how many executed
|
||||
// commands it performed in one iteration (after reading their results); it
|
||||
// must not count un-read commands.
|
||||
drive := func(b *testing.B, fn func(id int) int) {
|
||||
var count int64
|
||||
deadline := time.Now().Add(duration)
|
||||
b.ResetTimer()
|
||||
var wg sync.WaitGroup
|
||||
wg.Add(goroutines)
|
||||
for g := 0; g < goroutines; g++ {
|
||||
go func(id int) {
|
||||
defer wg.Done()
|
||||
for time.Now().Before(deadline) {
|
||||
atomic.AddInt64(&count, int64(fn(id)))
|
||||
}
|
||||
}(g)
|
||||
}
|
||||
wg.Wait()
|
||||
b.StopTimer()
|
||||
// Divide by the timed region, not the nominal window: goroutines check
|
||||
// the deadline once per run/window, so real elapsed exceeds `duration`
|
||||
// and dividing by the constant would inflate the number.
|
||||
b.ReportMetric(float64(count)/b.Elapsed().Seconds(), "ops/sec")
|
||||
}
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
b.Run("Normal", func(b *testing.B) {
|
||||
skipWithoutBenchServer(b)
|
||||
c := redis.NewClient(&redis.Options{Addr: benchAddr, PoolSize: 100})
|
||||
defer c.Close()
|
||||
drive(b, func(id int) int {
|
||||
// k is only the SET payload; a counter shared across drive's
|
||||
// goroutines would race.
|
||||
const run = 50 // amortize the harness per-step cost; each Set still blocks
|
||||
for k := 0; k < run; k++ {
|
||||
if err := c.Set(ctx, fmt.Sprintf("n:%d", id), k, 0).Err(); err != nil {
|
||||
b.Error(err)
|
||||
}
|
||||
}
|
||||
return run
|
||||
})
|
||||
})
|
||||
|
||||
b.Run("AutoPipelineBlocking", func(b *testing.B) {
|
||||
skipWithoutBenchServer(b)
|
||||
c := redis.NewClient(&redis.Options{Addr: benchAddr})
|
||||
defer c.Close()
|
||||
ap, err := c.AutoPipelineWithOptions(apConfig()) // blocking face, parallel batches
|
||||
if err != nil {
|
||||
b.Fatal(err)
|
||||
}
|
||||
defer ap.Close()
|
||||
drive(b, func(id int) int {
|
||||
// Each command call blocks until executed (drop-in shape, no .Result()
|
||||
// needed). We issue a small run per drive() step so the harness's
|
||||
// per-step atomic/closure cost is amortized and doesn't understate the
|
||||
// command rate — every command here still fully executes and is counted.
|
||||
// k is only the SET payload; a shared counter would race.
|
||||
const run = 50
|
||||
for k := 0; k < run; k++ {
|
||||
if err := ap.Set(ctx, fmt.Sprintf("b:%d", id), k, 0).Err(); err != nil {
|
||||
b.Error(err)
|
||||
}
|
||||
}
|
||||
return run
|
||||
})
|
||||
})
|
||||
|
||||
b.Run("AutoPipelineWindowed", func(b *testing.B) {
|
||||
skipWithoutBenchServer(b)
|
||||
c := redis.NewClient(&redis.Options{Addr: benchAddr})
|
||||
defer c.Close()
|
||||
ap, err := c.AsyncAutoPipelineWithOptions(apConfig()) // deferred face: submit a window, read later
|
||||
if err != nil {
|
||||
b.Fatal(err)
|
||||
}
|
||||
defer ap.Close()
|
||||
drive(b, func(id int) int {
|
||||
cmds := make([]*redis.StatusCmd, 0, window)
|
||||
for j := 0; j < window; j++ {
|
||||
cmds = append(cmds, ap.Set(ctx, fmt.Sprintf("w:%d", id), j, 0)) // does not block
|
||||
}
|
||||
n := 0
|
||||
for _, cmd := range cmds {
|
||||
if _, err := cmd.Result(); err != nil { // read result = executed
|
||||
b.Error(err)
|
||||
}
|
||||
n++
|
||||
}
|
||||
return n // only commands whose result was read
|
||||
})
|
||||
})
|
||||
|
||||
// Windowed GET: same windowed-async pattern but read-only. SET throughput is
|
||||
// capped by Redis's write processing, so the SET variants above are
|
||||
// server-bound, not client-bound. GET is cheaper on the server; this variant
|
||||
// shows the client machinery itself is not the limit for the SET numbers.
|
||||
b.Run("AutoPipelineWindowedGET", func(b *testing.B) {
|
||||
skipWithoutBenchServer(b)
|
||||
c := redis.NewClient(&redis.Options{Addr: benchAddr})
|
||||
defer c.Close()
|
||||
if err := c.Set(ctx, "bench:get", "v", 0).Err(); err != nil {
|
||||
b.Fatal(err)
|
||||
}
|
||||
ap, err := c.AsyncAutoPipelineWithOptions(apConfig())
|
||||
if err != nil {
|
||||
b.Fatal(err)
|
||||
}
|
||||
defer ap.Close()
|
||||
drive(b, func(id int) int {
|
||||
cmds := make([]*redis.StringCmd, 0, window)
|
||||
for j := 0; j < window; j++ {
|
||||
cmds = append(cmds, ap.Get(ctx, "bench:get"))
|
||||
}
|
||||
n := 0
|
||||
for _, cmd := range cmds {
|
||||
if _, err := cmd.Result(); err != nil {
|
||||
b.Error(err)
|
||||
}
|
||||
n++
|
||||
}
|
||||
return n
|
||||
})
|
||||
})
|
||||
}
|
||||
File diff suppressed because it is too large.
Load diff
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,826 @@
|
||||
package redis_test
|
||||
|
||||
// Truth battery: adversarial end-to-end validation of the AutoPipeliner's
|
||||
// core contracts against a live server — per-goroutine ORDER, exact RESPONSE
|
||||
// demultiplexing, no lost/duplicated executions — under both faces, ordered
|
||||
// and unordered configs, chunked dispatch, mid-flight cancellation and
|
||||
// close-drain. Written for the PR #3942 validation pass; every test derives
|
||||
// an independent ground truth (a model value computed client-side, or a
|
||||
// differential run against a plain client) rather than trusting the engine's
|
||||
// own accounting.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"math/rand"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/redis/go-redis/v9"
|
||||
)
|
||||
|
||||
func truthAddr() string { return apTestAddr() }
|
||||
|
||||
func newTruthClient(t *testing.T) *redis.Client {
|
||||
t.Helper()
|
||||
c := redis.NewClient(&redis.Options{Addr: truthAddr(), PoolSize: 64})
|
||||
if err := c.Ping(context.Background()).Err(); err != nil {
|
||||
c.Close()
|
||||
t.Skipf("no redis: %v", err)
|
||||
}
|
||||
return c
|
||||
}
|
||||
|
||||
// TestTruthBlockingPerGoroutineOrder pins the blocking face's ordering
|
||||
// contract with a server-side proof: each goroutine RPUSHes its own strictly
|
||||
// increasing sequence through a SHARED autopipeliner; the list on the server
|
||||
// is the ground truth and must come back exactly 0..M-1 for every goroutine.
|
||||
func TestTruthBlockingPerGoroutineOrder(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
client := newTruthClient(t)
|
||||
defer client.Close()
|
||||
client.FlushDB(ctx)
|
||||
ap, err := client.AutoPipelineWithOptions(&redis.AutoPipelineOptions{
|
||||
MaxBatchSize: 64, MaxConcurrentBatches: 8, Unordered: true, NumShards: 4,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
const G, M = 64, 200
|
||||
runWithWatchdog(t, 60*time.Second, func() {
|
||||
var wg sync.WaitGroup
|
||||
for g := 0; g < G; g++ {
|
||||
wg.Add(1)
|
||||
go func(g int) {
|
||||
defer wg.Done()
|
||||
key := "order:blocking:" + strconv.Itoa(g)
|
||||
for i := 0; i < M; i++ {
|
||||
if err := ap.RPush(ctx, key, i).Err(); err != nil {
|
||||
t.Errorf("g%d rpush %d: %v", g, i, err)
|
||||
return
|
||||
}
|
||||
}
|
||||
}(g)
|
||||
}
|
||||
wg.Wait()
|
||||
})
|
||||
for g := 0; g < G; g++ {
|
||||
vals, err := client.LRange(ctx, "order:blocking:"+strconv.Itoa(g), 0, -1).Result()
|
||||
if err != nil {
|
||||
t.Fatalf("lrange g%d: %v", g, err)
|
||||
}
|
||||
if len(vals) != M {
|
||||
t.Fatalf("g%d: %d elements, want %d (lost or duplicated commands)", g, len(vals), M)
|
||||
}
|
||||
for i, v := range vals {
|
||||
if v != strconv.Itoa(i) {
|
||||
t.Fatalf("g%d: position %d = %q, want %q (per-goroutine order violated)", g, i, v, i)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestTruthAsyncOrderedSingleGoroutine pins the deferred face's ordering on
|
||||
// the ORDERED default config: one goroutine submits an increasing sequence
|
||||
// without awaiting; the server-side list must be exactly in submit order,
|
||||
// and a read submitted after a write in the same window must observe it.
|
||||
func TestTruthAsyncOrderedSingleGoroutine(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
client := newTruthClient(t)
|
||||
defer client.Close()
|
||||
client.FlushDB(ctx)
|
||||
ap, err := client.AsyncAutoPipeline() // ordered default
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
const N = 3000
|
||||
runWithWatchdog(t, 60*time.Second, func() {
|
||||
cmds := make([]*redis.IntCmd, N)
|
||||
for i := 0; i < N; i++ {
|
||||
cmds[i] = ap.RPush(ctx, "order:async", i)
|
||||
}
|
||||
// read-your-write inside the window: SET then GET, same key, no await
|
||||
// between them.
|
||||
set := ap.Set(ctx, "order:ryw", "written", 0)
|
||||
get := ap.Get(ctx, "order:ryw")
|
||||
for i, c := range cmds {
|
||||
if err := c.Err(); err != nil {
|
||||
t.Fatalf("rpush %d: %v", i, err)
|
||||
}
|
||||
}
|
||||
if err := set.Err(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if v, err := get.Result(); err != nil || v != "written" {
|
||||
t.Fatalf("read-your-write: v=%q err=%v, want %q", v, err, "written")
|
||||
}
|
||||
})
|
||||
vals, err := client.LRange(ctx, "order:async", 0, -1).Result()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(vals) != N {
|
||||
t.Fatalf("%d elements, want %d", len(vals), N)
|
||||
}
|
||||
for i, v := range vals {
|
||||
if v != strconv.Itoa(i) {
|
||||
t.Fatalf("position %d = %q, want %q (submit order violated on ordered face)", i, v, i)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestTruthResponseDemux pins reply demultiplexing: every one of G*M GETs
|
||||
// must return exactly the unique payload its OWN goroutine wrote — any
|
||||
// cross-wiring of replies between commands or goroutines fails loudly.
|
||||
// Run on the most adversarial config: unordered, striped, parallel batches.
|
||||
func TestTruthResponseDemux(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
client := newTruthClient(t)
|
||||
defer client.Close()
|
||||
client.FlushDB(ctx)
|
||||
ap, err := client.AsyncAutoPipelineWithOptions(&redis.AutoPipelineOptions{
|
||||
MaxBatchSize: 128, MaxConcurrentBatches: 32, Unordered: true, NumShards: 8,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
const G, M = 100, 300
|
||||
// Unordered config: a GET may overtake its own SET by design, so the
|
||||
// phases are separated — all writes awaited, then all reads. The demux
|
||||
// property under test is untouched: every reply must match its OWN
|
||||
// command's unique payload.
|
||||
runWithWatchdog(t, 120*time.Second, func() {
|
||||
var wg sync.WaitGroup
|
||||
for g := 0; g < G; g++ {
|
||||
wg.Add(1)
|
||||
go func(g int) {
|
||||
defer wg.Done()
|
||||
sets := make([]*redis.StatusCmd, M)
|
||||
for i := 0; i < M; i++ {
|
||||
key := fmt.Sprintf("demux:%d:%d", g, i)
|
||||
want := fmt.Sprintf("payload|g=%d|i=%d|%x", g, i, g*1_000_003+i*7919)
|
||||
sets[i] = ap.Set(ctx, key, want, 0)
|
||||
}
|
||||
for i, s := range sets {
|
||||
if err := s.Err(); err != nil {
|
||||
t.Errorf("set g%d i%d: %v", g, i, err)
|
||||
return
|
||||
}
|
||||
}
|
||||
gets := make([]*redis.StringCmd, M)
|
||||
for i := 0; i < M; i++ {
|
||||
gets[i] = ap.Get(ctx, fmt.Sprintf("demux:%d:%d", g, i))
|
||||
}
|
||||
for i, gc := range gets {
|
||||
want := fmt.Sprintf("payload|g=%d|i=%d|%x", g, i, g*1_000_003+i*7919)
|
||||
got, err := gc.Result()
|
||||
if err != nil {
|
||||
t.Errorf("get g%d i%d: %v", g, i, err)
|
||||
return
|
||||
}
|
||||
if got != want {
|
||||
t.Errorf("DEMUX VIOLATION g%d i%d: got %q want %q", g, i, got, want)
|
||||
return
|
||||
}
|
||||
}
|
||||
}(g)
|
||||
}
|
||||
wg.Wait()
|
||||
})
|
||||
}
|
||||
|
||||
// TestTruthOrderedPairsAcrossGoroutines pins per-goroutine read-your-write
|
||||
// on the ORDERED default config with many goroutines sharing the pipeliner:
|
||||
// each goroutine interleaves SET/GET pairs on its own keys without any
|
||||
// barrier — the ordered face must preserve each goroutine's submit order.
|
||||
func TestTruthOrderedPairsAcrossGoroutines(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
client := newTruthClient(t)
|
||||
defer client.Close()
|
||||
client.FlushDB(ctx)
|
||||
ap, err := client.AsyncAutoPipeline() // ordered default
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
const G, M = 64, 150
|
||||
runWithWatchdog(t, 120*time.Second, func() {
|
||||
var wg sync.WaitGroup
|
||||
for g := 0; g < G; g++ {
|
||||
wg.Add(1)
|
||||
go func(g int) {
|
||||
defer wg.Done()
|
||||
for i := 0; i < M; i++ {
|
||||
key := fmt.Sprintf("pair:%d:%d", g, i)
|
||||
want := fmt.Sprintf("v|%d|%d", g, i)
|
||||
set := ap.Set(ctx, key, want, 0)
|
||||
get := ap.Get(ctx, key) // no await between: ordered face must sequence
|
||||
if err := set.Err(); err != nil {
|
||||
t.Errorf("set g%d i%d: %v", g, i, err)
|
||||
return
|
||||
}
|
||||
got, err := get.Result()
|
||||
if err != nil || got != want {
|
||||
t.Errorf("ORDER VIOLATION g%d i%d: got %q err=%v want %q", g, i, got, err, want)
|
||||
return
|
||||
}
|
||||
}
|
||||
}(g)
|
||||
}
|
||||
wg.Wait()
|
||||
})
|
||||
}
|
||||
|
||||
// TestTruthCountersExact pins no-lost-no-duplicated execution: G goroutines
|
||||
// INCRBY random deltas on one shared counter; the client-side sum is the
|
||||
// ground truth and the final server value must equal it exactly. A dropped
|
||||
// command, a duplicated retry, or a misrouted reply cannot pass.
|
||||
func TestTruthCountersExact(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
client := newTruthClient(t)
|
||||
defer client.Close()
|
||||
client.FlushDB(ctx)
|
||||
for _, cfg := range []struct {
|
||||
name string
|
||||
opts *redis.AutoPipelineOptions
|
||||
}{
|
||||
{"ordered-default", nil},
|
||||
{"unordered-parallel", &redis.AutoPipelineOptions{MaxBatchSize: 200, MaxConcurrentBatches: 40, Unordered: true, NumShards: 8}},
|
||||
{"tiny-batches", &redis.AutoPipelineOptions{MaxBatchSize: 2, MaxConcurrentBatches: 4, Unordered: true}},
|
||||
} {
|
||||
t.Run(cfg.name, func(t *testing.T) {
|
||||
var ap *redis.AutoPipeliner
|
||||
var err error
|
||||
if cfg.opts == nil {
|
||||
ap, err = client.AsyncAutoPipeline()
|
||||
} else {
|
||||
ap, err = client.AsyncAutoPipelineWithOptions(cfg.opts)
|
||||
}
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
key := "counter:" + cfg.name
|
||||
client.Del(ctx, key)
|
||||
const G, M = 50, 200
|
||||
var wantTotal atomic.Int64
|
||||
runWithWatchdog(t, 90*time.Second, func() {
|
||||
var wg sync.WaitGroup
|
||||
for g := 0; g < G; g++ {
|
||||
wg.Add(1)
|
||||
go func(g int) {
|
||||
defer wg.Done()
|
||||
rng := rand.New(rand.NewSource(int64(g)*7919 + 1))
|
||||
futs := make([]*redis.IntCmd, M)
|
||||
deltas := make([]int64, M)
|
||||
for i := 0; i < M; i++ {
|
||||
deltas[i] = rng.Int63n(1000) - 500
|
||||
futs[i] = ap.IncrBy(ctx, key, deltas[i])
|
||||
}
|
||||
for i, f := range futs {
|
||||
if err := f.Err(); err != nil {
|
||||
t.Errorf("incrby: %v", err)
|
||||
return
|
||||
}
|
||||
wantTotal.Add(deltas[i])
|
||||
}
|
||||
}(g)
|
||||
}
|
||||
wg.Wait()
|
||||
})
|
||||
got, err := client.Get(ctx, key).Int64()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got != wantTotal.Load() {
|
||||
t.Fatalf("counter drift: server=%d client-sum=%d (lost or duplicated executions)", got, wantTotal.Load())
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestTruthDifferentialVsPlainClient runs an identical randomized mixed
|
||||
// workload through a plain client (the model) and the autopipeliner (the
|
||||
// subject) on disjoint keyspaces, then diffs the full keyspace dumps. Any
|
||||
// divergence in values, hashes, or list contents is an engine correctness
|
||||
// bug by construction.
|
||||
func TestTruthDifferentialVsPlainClient(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
client := newTruthClient(t)
|
||||
defer client.Close()
|
||||
client.FlushDB(ctx)
|
||||
ap, err := client.AsyncAutoPipelineWithOptions(&redis.AutoPipelineOptions{
|
||||
MaxBatchSize: 100, MaxConcurrentBatches: 16, Unordered: true, NumShards: 4,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
type op struct {
|
||||
kind int
|
||||
key, field string
|
||||
val int64
|
||||
}
|
||||
const G, M = 32, 150
|
||||
plans := make([][]op, G)
|
||||
for g := 0; g < G; g++ {
|
||||
rng := rand.New(rand.NewSource(int64(g) + 42))
|
||||
plans[g] = make([]op, M)
|
||||
for i := range plans[g] {
|
||||
plans[g][i] = op{
|
||||
kind: rng.Intn(4),
|
||||
key: strconv.Itoa(rng.Intn(20)), // few keys -> heavy contention
|
||||
field: "f" + strconv.Itoa(rng.Intn(5)),
|
||||
val: rng.Int63n(100),
|
||||
}
|
||||
}
|
||||
}
|
||||
run := func(prefix string, exec redis.Cmdable) {
|
||||
var wg sync.WaitGroup
|
||||
for g := 0; g < G; g++ {
|
||||
wg.Add(1)
|
||||
go func(g int) {
|
||||
defer wg.Done()
|
||||
for _, o := range plans[g] {
|
||||
key := prefix + o.key
|
||||
var err error
|
||||
switch o.kind {
|
||||
case 0:
|
||||
err = exec.IncrBy(ctx, key+":c", o.val).Err()
|
||||
case 1:
|
||||
err = exec.HIncrBy(ctx, key+":h", o.field, o.val).Err()
|
||||
case 2:
|
||||
err = exec.SAdd(ctx, key+":s", o.val).Err()
|
||||
case 3:
|
||||
err = exec.RPush(ctx, key+":l:"+strconv.Itoa(g), o.val).Err()
|
||||
}
|
||||
if err != nil {
|
||||
t.Errorf("%s op: %v", prefix, err)
|
||||
return
|
||||
}
|
||||
}
|
||||
}(g)
|
||||
}
|
||||
wg.Wait()
|
||||
}
|
||||
runWithWatchdog(t, 120*time.Second, func() {
|
||||
run("model:", client) // plain client = the model
|
||||
run("subject:", ap) // autopipeliner = the subject
|
||||
})
|
||||
dump := func(prefix string) map[string]string {
|
||||
out := map[string]string{}
|
||||
keys, err := client.Keys(ctx, prefix+"*").Result()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, k := range keys {
|
||||
short := strings.TrimPrefix(k, prefix)
|
||||
switch typ := client.Type(ctx, k).Val(); typ {
|
||||
case "string":
|
||||
out[short] = client.Get(ctx, k).Val()
|
||||
case "hash":
|
||||
out[short] = fmt.Sprint(client.HGetAll(ctx, k).Val())
|
||||
case "set":
|
||||
vals, _ := client.SMembers(ctx, k).Result()
|
||||
m := map[string]bool{}
|
||||
for _, v := range vals {
|
||||
m[v] = true
|
||||
}
|
||||
out[short] = fmt.Sprint(len(m)) // set cardinality (order-free)
|
||||
case "list":
|
||||
out[short] = strings.Join(client.LRange(ctx, k, 0, -1).Val(), ",")
|
||||
default:
|
||||
out[short] = "type:" + typ
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
model, subject := dump("model:"), dump("subject:")
|
||||
if len(model) != len(subject) {
|
||||
t.Fatalf("keyspace size differs: model=%d subject=%d", len(model), len(subject))
|
||||
}
|
||||
for k, mv := range model {
|
||||
if sv, ok := subject[k]; !ok || sv != mv {
|
||||
t.Errorf("DIVERGENCE key %q: model=%q subject=%q", k, mv, sv)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestTruthChunkedDispatchIntegrity drives the MaxBatchBytes chunked path
|
||||
// with values large enough to split every drain into several chunks and
|
||||
// verifies exact payload echo — chunk boundaries must not tear, reorder, or
|
||||
// cross-wire replies.
|
||||
func TestTruthChunkedDispatchIntegrity(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
client := newTruthClient(t)
|
||||
defer client.Close()
|
||||
client.FlushDB(ctx)
|
||||
ap, err := client.AsyncAutoPipelineWithOptions(&redis.AutoPipelineOptions{
|
||||
MaxBatchSize: 500, MaxConcurrentBatches: 8, Unordered: true,
|
||||
MaxBatchBytes: 64 * 1024, // ~4 x 16KiB values per chunk
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
const G, M = 16, 40
|
||||
base := strings.Repeat("x", 16*1024)
|
||||
runWithWatchdog(t, 120*time.Second, func() {
|
||||
var wg sync.WaitGroup
|
||||
for g := 0; g < G; g++ {
|
||||
wg.Add(1)
|
||||
go func(g int) {
|
||||
defer wg.Done()
|
||||
sets := make([]*redis.StatusCmd, M)
|
||||
for i := 0; i < M; i++ {
|
||||
key := fmt.Sprintf("chunk:%d:%d", g, i)
|
||||
want := fmt.Sprintf("%s|tail=%d:%d", base, g, i)
|
||||
sets[i] = ap.Set(ctx, key, want, 0)
|
||||
}
|
||||
for i, s := range sets {
|
||||
if err := s.Err(); err != nil {
|
||||
t.Errorf("set g%d i%d: %v", g, i, err)
|
||||
return
|
||||
}
|
||||
}
|
||||
for i := 0; i < M; i++ {
|
||||
key := fmt.Sprintf("chunk:%d:%d", g, i)
|
||||
want := fmt.Sprintf("%s|tail=%d:%d", base, g, i)
|
||||
got, err := ap.Get(ctx, key).Result()
|
||||
if err != nil {
|
||||
t.Errorf("get %s: %v", key, err)
|
||||
return
|
||||
}
|
||||
if got != want {
|
||||
t.Errorf("CHUNK TEAR %s: got %d bytes tail %q, want %d bytes", key, len(got), got[len(got)-24:], len(want))
|
||||
return
|
||||
}
|
||||
}
|
||||
}(g)
|
||||
}
|
||||
wg.Wait()
|
||||
})
|
||||
}
|
||||
|
||||
// TestTruthCancelNeverLies pins cancellation semantics: a context canceled
|
||||
// mid-flight may fail the WAIT, but any command that reports success must
|
||||
// have really executed, and any that reports an error must not have half a
|
||||
// result. Verified by re-reading the keyspace with a fresh context.
|
||||
func TestTruthCancelNeverLies(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
client := newTruthClient(t)
|
||||
defer client.Close()
|
||||
client.FlushDB(ctx)
|
||||
ap, err := client.AsyncAutoPipeline()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
const N = 2000
|
||||
runWithWatchdog(t, 60*time.Second, func() {
|
||||
cctx, cancel := context.WithCancel(ctx)
|
||||
futs := make([]*redis.StatusCmd, N)
|
||||
for i := 0; i < N; i++ {
|
||||
futs[i] = ap.Set(cctx, "cancel:"+strconv.Itoa(i), "v"+strconv.Itoa(i), 0)
|
||||
if i == N/2 {
|
||||
cancel()
|
||||
}
|
||||
}
|
||||
okCount, errCount := 0, 0
|
||||
for i, f := range futs {
|
||||
if err := f.Err(); err == nil {
|
||||
okCount++
|
||||
// claimed success must be true on the server
|
||||
v, gerr := client.Get(ctx, "cancel:"+strconv.Itoa(i)).Result()
|
||||
if gerr != nil || v != "v"+strconv.Itoa(i) {
|
||||
t.Fatalf("cmd %d reported OK but server has %q/%v", i, v, gerr)
|
||||
}
|
||||
} else {
|
||||
errCount++
|
||||
}
|
||||
}
|
||||
t.Logf("cancel split: %d ok, %d error (both outcomes legal; lies are not)", okCount, errCount)
|
||||
cancel()
|
||||
})
|
||||
}
|
||||
|
||||
// TestTruthCloseDrains pins Close's drain contract under load: every future
|
||||
// submitted BEFORE Close returns a definite outcome (value or error, no
|
||||
// hang), and successes are really on the server.
|
||||
func TestTruthCloseDrains(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
client := newTruthClient(t)
|
||||
defer client.Close()
|
||||
client.FlushDB(ctx)
|
||||
ap, err := client.AsyncAutoPipelineWithOptions(&redis.AutoPipelineOptions{
|
||||
MaxBatchSize: 64, MaxConcurrentBatches: 8, Unordered: true,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
const N = 5000
|
||||
runWithWatchdog(t, 60*time.Second, func() {
|
||||
futs := make([]*redis.StatusCmd, N)
|
||||
for i := 0; i < N; i++ {
|
||||
futs[i] = ap.Set(ctx, "drain:"+strconv.Itoa(i), "v", 0)
|
||||
}
|
||||
if err := ap.Close(); err != nil {
|
||||
t.Fatalf("close: %v", err)
|
||||
}
|
||||
okCount := 0
|
||||
for i, f := range futs {
|
||||
if err := f.Err(); err == nil {
|
||||
okCount++
|
||||
} else if err != redis.ErrClosed {
|
||||
t.Fatalf("future %d: unexpected error %v", i, err)
|
||||
}
|
||||
}
|
||||
// Every claimed success must exist server-side.
|
||||
var present int64
|
||||
for i := 0; i < N; i++ {
|
||||
if client.Exists(ctx, "drain:"+strconv.Itoa(i)).Val() == 1 {
|
||||
present++
|
||||
}
|
||||
}
|
||||
if int(present) < okCount {
|
||||
t.Fatalf("%d futures claimed OK but only %d keys present", okCount, present)
|
||||
}
|
||||
t.Logf("close drain: %d/%d confirmed executed, rest cleanly rejected", okCount, N)
|
||||
})
|
||||
}
|
||||
|
||||
// TestTruthClusterOrderAndDemux repeats the order and demux proofs against a
|
||||
// live cluster through the sharded autopipeliner: per-goroutine order on
|
||||
// hashtagged keys (one slot per goroutine, so order is meaningful) and exact
|
||||
// payload echo across slot-spread keys.
|
||||
func TestTruthClusterOrderAndDemux(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
cc := redis.NewClusterClient(&redis.ClusterOptions{Addrs: []string{":16600", ":16601", ":16602"}})
|
||||
defer cc.Close()
|
||||
if err := cc.Ping(ctx).Err(); err != nil {
|
||||
t.Skipf("no cluster: %v", err)
|
||||
}
|
||||
for _, n := range []string{":16600", ":16601", ":16602"} {
|
||||
nc := redis.NewClient(&redis.Options{Addr: n})
|
||||
nc.FlushDB(ctx)
|
||||
nc.Close()
|
||||
}
|
||||
ap, err := cc.AsyncAutoPipeline()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
const G, M = 32, 100
|
||||
runWithWatchdog(t, 120*time.Second, func() {
|
||||
var wg sync.WaitGroup
|
||||
for g := 0; g < G; g++ {
|
||||
wg.Add(1)
|
||||
go func(g int) {
|
||||
defer wg.Done()
|
||||
// order proof: hashtag pins the goroutine's list to one slot
|
||||
key := fmt.Sprintf("{ord%d}:list", g)
|
||||
futs := make([]*redis.IntCmd, M)
|
||||
for i := 0; i < M; i++ {
|
||||
futs[i] = ap.RPush(ctx, key, i)
|
||||
}
|
||||
for i, f := range futs {
|
||||
if err := f.Err(); err != nil {
|
||||
t.Errorf("g%d rpush %d: %v", g, i, err)
|
||||
return
|
||||
}
|
||||
}
|
||||
// demux proof: slot-spread unique echoes, phase-separated
|
||||
sets := make([]*redis.StatusCmd, M)
|
||||
for i := 0; i < M; i++ {
|
||||
sets[i] = ap.Set(ctx, fmt.Sprintf("cdemux:%d:%d", g, i), fmt.Sprintf("p|%d|%d", g, i), 0)
|
||||
}
|
||||
for i, s := range sets {
|
||||
if err := s.Err(); err != nil {
|
||||
t.Errorf("g%d set %d: %v", g, i, err)
|
||||
return
|
||||
}
|
||||
}
|
||||
for i := 0; i < M; i++ {
|
||||
want := fmt.Sprintf("p|%d|%d", g, i)
|
||||
got, err := ap.Get(ctx, fmt.Sprintf("cdemux:%d:%d", g, i)).Result()
|
||||
if err != nil || got != want {
|
||||
t.Errorf("CLUSTER DEMUX g%d i%d: got %q err=%v want %q", g, i, got, err, want)
|
||||
return
|
||||
}
|
||||
}
|
||||
}(g)
|
||||
}
|
||||
wg.Wait()
|
||||
})
|
||||
for g := 0; g < G; g++ {
|
||||
vals, err := cc.LRange(ctx, fmt.Sprintf("{ord%d}:list", g), 0, -1).Result()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(vals) != M {
|
||||
t.Fatalf("cluster g%d: %d elements want %d", g, len(vals), M)
|
||||
}
|
||||
for i, v := range vals {
|
||||
if v != strconv.Itoa(i) {
|
||||
t.Fatalf("CLUSTER ORDER g%d: pos %d = %q want %q", g, i, v, i)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestTruthRESP2Demux repeats the demultiplexing proof on RESP2, where
|
||||
// replies carry no push-frame framing and a desynchronized stream shows up
|
||||
// as cross-wired values rather than a protocol error.
|
||||
func TestTruthRESP2Demux(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
client := redis.NewClient(&redis.Options{Addr: truthAddr(), Protocol: 2, PoolSize: 32})
|
||||
defer client.Close()
|
||||
if err := client.Ping(ctx).Err(); err != nil {
|
||||
t.Skipf("no redis: %v", err)
|
||||
}
|
||||
client.FlushDB(ctx)
|
||||
ap, err := client.AsyncAutoPipelineWithOptions(&redis.AutoPipelineOptions{
|
||||
MaxBatchSize: 100, MaxConcurrentBatches: 16, Unordered: true, NumShards: 4,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
const G, M = 40, 150
|
||||
runWithWatchdog(t, 90*time.Second, func() {
|
||||
var wg sync.WaitGroup
|
||||
for g := 0; g < G; g++ {
|
||||
wg.Add(1)
|
||||
go func(g int) {
|
||||
defer wg.Done()
|
||||
sets := make([]*redis.StatusCmd, M)
|
||||
for i := 0; i < M; i++ {
|
||||
sets[i] = ap.Set(ctx, fmt.Sprintf("r2:%d:%d", g, i), fmt.Sprintf("v|%d|%d", g, i), 0)
|
||||
}
|
||||
for _, s := range sets {
|
||||
if err := s.Err(); err != nil {
|
||||
t.Errorf("resp2 set: %v", err)
|
||||
return
|
||||
}
|
||||
}
|
||||
// mixed reply SHAPES in one batch: string, integer, array,
|
||||
// hash, nil — a demux slip shows up as a type/shape mismatch.
|
||||
for i := 0; i < M; i++ {
|
||||
key := fmt.Sprintf("r2:%d:%d", g, i)
|
||||
sv := ap.Get(ctx, key)
|
||||
iv := ap.StrLen(ctx, key)
|
||||
av := ap.LRange(ctx, "r2:missing:list", 0, -1)
|
||||
nv := ap.Get(ctx, "r2:missing:key")
|
||||
want := fmt.Sprintf("v|%d|%d", g, i)
|
||||
if got, err := sv.Result(); err != nil || got != want {
|
||||
t.Errorf("RESP2 string demux g%d i%d: %q %v", g, i, got, err)
|
||||
return
|
||||
}
|
||||
if got, err := iv.Result(); err != nil || got != int64(len(want)) {
|
||||
t.Errorf("RESP2 int demux g%d i%d: %d %v want %d", g, i, got, err, len(want))
|
||||
return
|
||||
}
|
||||
if got, err := av.Result(); err != nil || len(got) != 0 {
|
||||
t.Errorf("RESP2 array demux g%d i%d: %v %v", g, i, got, err)
|
||||
return
|
||||
}
|
||||
if _, err := nv.Result(); err != redis.Nil {
|
||||
t.Errorf("RESP2 nil demux g%d i%d: err=%v want redis.Nil", g, i, err)
|
||||
return
|
||||
}
|
||||
}
|
||||
}(g)
|
||||
}
|
||||
wg.Wait()
|
||||
})
|
||||
}
|
||||
|
||||
// TestTruthPipelinePoolInterleaving runs the autopipeliner against a client
|
||||
// configured with the DEDICATED pipeline pool while a plain client hammers
|
||||
// the same server through the regular pool: batched traffic and single
|
||||
// commands must not contaminate each other's connections.
|
||||
func TestTruthPipelinePoolInterleaving(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
client := redis.NewClient(&redis.Options{
|
||||
Addr: truthAddr(), PoolSize: 32,
|
||||
PipelineReadBufferSize: 128 << 10, PipelineWriteBufferSize: 128 << 10,
|
||||
PipelinePoolSize: 4,
|
||||
})
|
||||
defer client.Close()
|
||||
if err := client.Ping(ctx).Err(); err != nil {
|
||||
t.Skipf("no redis: %v", err)
|
||||
}
|
||||
client.FlushDB(ctx)
|
||||
ap, err := client.AsyncAutoPipelineWithOptions(&redis.AutoPipelineOptions{
|
||||
MaxBatchSize: 128, MaxConcurrentBatches: 16, Unordered: true,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
const G, M = 24, 200
|
||||
runWithWatchdog(t, 120*time.Second, func() {
|
||||
var wg sync.WaitGroup
|
||||
// plain-client noise on the regular pool
|
||||
stop := make(chan struct{})
|
||||
for n := 0; n < 8; n++ {
|
||||
wg.Add(1)
|
||||
go func(n int) {
|
||||
defer wg.Done()
|
||||
for i := 0; ; i++ {
|
||||
select {
|
||||
case <-stop:
|
||||
return
|
||||
default:
|
||||
}
|
||||
key := fmt.Sprintf("noise:%d", n)
|
||||
want := fmt.Sprintf("n|%d|%d", n, i)
|
||||
if err := client.Set(ctx, key, want, 0).Err(); err != nil {
|
||||
t.Errorf("noise set: %v", err)
|
||||
return
|
||||
}
|
||||
if got, err := client.Get(ctx, key).Result(); err != nil || got != want {
|
||||
t.Errorf("NOISE CONTAMINATION: got %q err=%v want %q", got, err, want)
|
||||
return
|
||||
}
|
||||
}
|
||||
}(n)
|
||||
}
|
||||
// autopipelined traffic on the pipeline pool
|
||||
var apWg sync.WaitGroup
|
||||
for g := 0; g < G; g++ {
|
||||
apWg.Add(1)
|
||||
go func(g int) {
|
||||
defer apWg.Done()
|
||||
sets := make([]*redis.StatusCmd, M)
|
||||
for i := 0; i < M; i++ {
|
||||
sets[i] = ap.Set(ctx, fmt.Sprintf("pp:%d:%d", g, i), fmt.Sprintf("p|%d|%d", g, i), 0)
|
||||
}
|
||||
for _, s := range sets {
|
||||
if err := s.Err(); err != nil {
|
||||
t.Errorf("pp set: %v", err)
|
||||
return
|
||||
}
|
||||
}
|
||||
for i := 0; i < M; i++ {
|
||||
want := fmt.Sprintf("p|%d|%d", g, i)
|
||||
got, err := ap.Get(ctx, fmt.Sprintf("pp:%d:%d", g, i)).Result()
|
||||
if err != nil || got != want {
|
||||
t.Errorf("PIPELINE POOL DEMUX g%d i%d: got %q err=%v want %q", g, i, got, err, want)
|
||||
return
|
||||
}
|
||||
}
|
||||
}(g)
|
||||
}
|
||||
apWg.Wait()
|
||||
close(stop)
|
||||
wg.Wait()
|
||||
})
|
||||
}
|
||||
|
||||
// TestTruthRetriesDoNotDuplicate pins at-most-once execution across the
|
||||
// engine's retry machinery: non-idempotent INCRs run through an
|
||||
// autopipeliner whose client has retries ENABLED while the server is
|
||||
// pounded concurrently; the final counter must equal the exact number of
|
||||
// successful futures — a retried batch that re-executed a committed command
|
||||
// would overshoot.
|
||||
func TestTruthRetriesDoNotDuplicate(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
client := redis.NewClient(&redis.Options{Addr: truthAddr(), PoolSize: 32, MaxRetries: 3})
|
||||
defer client.Close()
|
||||
if err := client.Ping(ctx).Err(); err != nil {
|
||||
t.Skipf("no redis: %v", err)
|
||||
}
|
||||
client.FlushDB(ctx)
|
||||
ap, err := client.AsyncAutoPipelineWithOptions(&redis.AutoPipelineOptions{
|
||||
MaxBatchSize: 64, MaxConcurrentBatches: 16, Unordered: true, NumShards: 4,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
const G, M = 40, 250
|
||||
var okCount atomic.Int64
|
||||
runWithWatchdog(t, 120*time.Second, func() {
|
||||
var wg sync.WaitGroup
|
||||
for g := 0; g < G; g++ {
|
||||
wg.Add(1)
|
||||
go func(g int) {
|
||||
defer wg.Done()
|
||||
futs := make([]*redis.IntCmd, M)
|
||||
for i := 0; i < M; i++ {
|
||||
futs[i] = ap.Incr(ctx, "retry:counter")
|
||||
}
|
||||
for _, f := range futs {
|
||||
if err := f.Err(); err == nil {
|
||||
okCount.Add(1)
|
||||
}
|
||||
}
|
||||
}(g)
|
||||
}
|
||||
wg.Wait()
|
||||
})
|
||||
got, err := client.Get(ctx, "retry:counter").Int64()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got != okCount.Load() {
|
||||
t.Fatalf("at-most-once violated: counter=%d successful futures=%d", got, okCount.Load())
|
||||
}
|
||||
t.Logf("at-most-once: %d increments == %d successful futures", got, okCount.Load())
|
||||
}
|
||||
+16
-8
@@ -13,12 +13,16 @@ type bitCountExpected struct {
|
||||
}
|
||||
|
||||
var _ = Describe("BitCountBite", func() {
|
||||
var client *redis.Client
|
||||
var client redis.UniversalClient
|
||||
var rawClient *redis.Client
|
||||
var closeSubject func() error
|
||||
key := "bit_count_test"
|
||||
|
||||
BeforeEach(func() {
|
||||
client = redis.NewClient(redisOptions())
|
||||
Expect(client.FlushDB(ctx).Err()).NotTo(HaveOccurred())
|
||||
rawClient = redis.NewClient(redisOptions())
|
||||
// Subject/closer before the first Expect (see commands_test.go).
|
||||
client, closeSubject = newUniversalSubject(rawClient)
|
||||
Expect(rawClient.FlushDB(ctx).Err()).NotTo(HaveOccurred())
|
||||
values := []int{0, 1, 0, 0, 1, 0, 1, 0, 1, 1}
|
||||
for i, v := range values {
|
||||
cmd := client.SetBit(ctx, key, int64(i), v)
|
||||
@@ -27,7 +31,7 @@ var _ = Describe("BitCountBite", func() {
|
||||
})
|
||||
|
||||
AfterEach(func() {
|
||||
Expect(client.Close()).NotTo(HaveOccurred())
|
||||
Expect(closeSubject()).NotTo(HaveOccurred())
|
||||
})
|
||||
|
||||
It("bit count bite", func() {
|
||||
@@ -53,12 +57,16 @@ var _ = Describe("BitCountBite", func() {
|
||||
})
|
||||
|
||||
var _ = Describe("BitCountByte", func() {
|
||||
var client *redis.Client
|
||||
var client redis.UniversalClient
|
||||
var rawClient *redis.Client
|
||||
var closeSubject func() error
|
||||
key := "bit_count_test"
|
||||
|
||||
BeforeEach(func() {
|
||||
client = redis.NewClient(redisOptions())
|
||||
Expect(client.FlushDB(ctx).Err()).NotTo(HaveOccurred())
|
||||
rawClient = redis.NewClient(redisOptions())
|
||||
// Subject/closer before the first Expect (see commands_test.go).
|
||||
client, closeSubject = newUniversalSubject(rawClient)
|
||||
Expect(rawClient.FlushDB(ctx).Err()).NotTo(HaveOccurred())
|
||||
values := []int{0, 0, 0, 0, 0, 0, 0, 1, 1, 1}
|
||||
for i, v := range values {
|
||||
cmd := client.SetBit(ctx, key, int64(i), v)
|
||||
@@ -67,7 +75,7 @@ var _ = Describe("BitCountByte", func() {
|
||||
})
|
||||
|
||||
AfterEach(func() {
|
||||
Expect(client.Close()).NotTo(HaveOccurred())
|
||||
Expect(closeSubject()).NotTo(HaveOccurred())
|
||||
})
|
||||
|
||||
It("bit count byte", func() {
|
||||
|
||||
+381
-12
File diff suppressed because it is too large.
Load diff
+176
-158
@@ -31,15 +31,26 @@ func (t *TimeValue) ScanRedis(s string) (err error) {
|
||||
|
||||
var _ = Describe("Commands", func() {
|
||||
ctx := context.TODO()
|
||||
var client *redis.Client
|
||||
// client is the UniversalClient under test (the raw client or an
|
||||
// autopipeliner, per GOREDIS_TEST_SUBJECT — see newUniversalSubject).
|
||||
// rawClient is always the underlying *redis.Client, used for setup/teardown
|
||||
// and the admin/connection-only calls (FlushDB, Config*, Ping, PoolStats,
|
||||
// Conn, ...) that are not part of the UniversalClient interface.
|
||||
var client redis.UniversalClient
|
||||
var rawClient *redis.Client
|
||||
var closeSubject func() error
|
||||
|
||||
BeforeEach(func() {
|
||||
client = redis.NewClient(redisOptions())
|
||||
Expect(client.FlushDB(ctx).Err()).NotTo(HaveOccurred())
|
||||
rawClient = redis.NewClient(redisOptions())
|
||||
// Build the subject (and its closer) BEFORE any assertion: if the
|
||||
// FlushDB Expect fails, AfterEach still runs and must have a valid
|
||||
// closer for THIS spec's clients, not nil or the previous spec's.
|
||||
client, closeSubject = newUniversalSubject(rawClient)
|
||||
Expect(rawClient.FlushDB(ctx).Err()).NotTo(HaveOccurred())
|
||||
})
|
||||
|
||||
AfterEach(func() {
|
||||
Expect(client.Close()).NotTo(HaveOccurred())
|
||||
Expect(closeSubject()).NotTo(HaveOccurred())
|
||||
})
|
||||
|
||||
Describe("server", func() {
|
||||
@@ -71,7 +82,7 @@ var _ = Describe("Commands", func() {
|
||||
Expect(err.Error()).To(authErr)
|
||||
Expect(cmds[0].Err().Error()).To(authErr)
|
||||
|
||||
stats := client.PoolStats()
|
||||
stats := rawClient.PoolStats()
|
||||
Expect(stats.Hits).To(Equal(uint32(2)))
|
||||
Expect(stats.Misses).To(Equal(uint32(1)))
|
||||
Expect(stats.Timeouts).To(Equal(uint32(0)))
|
||||
@@ -101,13 +112,13 @@ var _ = Describe("Commands", func() {
|
||||
})
|
||||
|
||||
It("should Ping", func() {
|
||||
ping := client.Ping(ctx)
|
||||
ping := rawClient.Ping(ctx)
|
||||
Expect(ping.Err()).NotTo(HaveOccurred())
|
||||
Expect(ping.Val()).To(Equal("PONG"))
|
||||
})
|
||||
|
||||
It("should Ping with Do method", func() {
|
||||
result := client.Conn().Do(ctx, "PING")
|
||||
result := rawClient.Conn().Do(ctx, "PING")
|
||||
Expect(result.Err()).NotTo(HaveOccurred())
|
||||
Expect(result.Val()).To(Equal("PONG"))
|
||||
})
|
||||
@@ -117,7 +128,7 @@ var _ = Describe("Commands", func() {
|
||||
|
||||
// assume testing on single redis instance
|
||||
start := time.Now()
|
||||
val, err := client.Wait(ctx, 1, wait).Result()
|
||||
val, err := rawClient.Wait(ctx, 1, wait).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(Equal(int64(0)))
|
||||
Expect(time.Now()).To(BeTemporally("~", start.Add(wait), 3*time.Second))
|
||||
@@ -129,7 +140,7 @@ var _ = Describe("Commands", func() {
|
||||
// No replicas here, so WAITAOF blocks until timeout and returns
|
||||
// the two counts [numlocal, numreplicas]. numLocal=0 skips AOF.
|
||||
start := time.Now()
|
||||
val, err := client.WaitAOF(ctx, 0, 1, waitAOF).Result()
|
||||
val, err := rawClient.WaitAOF(ctx, 0, 1, waitAOF).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(HaveLen(2))
|
||||
Expect(val[1]).To(Equal(int64(0)))
|
||||
@@ -159,7 +170,7 @@ var _ = Describe("Commands", func() {
|
||||
It("should BgRewriteAOF", func() {
|
||||
Skip("flaky test")
|
||||
|
||||
val, err := client.BgRewriteAOF(ctx).Result()
|
||||
val, err := rawClient.BgRewriteAOF(ctx).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(ContainSubstring("Background append only file rewriting"))
|
||||
})
|
||||
@@ -169,30 +180,30 @@ var _ = Describe("Commands", func() {
|
||||
|
||||
// workaround for "ERR Can't BGSAVE while AOF log rewriting is in progress"
|
||||
Eventually(func() string {
|
||||
return client.BgSave(ctx).Val()
|
||||
return rawClient.BgSave(ctx).Val()
|
||||
}, "30s").Should(Equal("Background saving started"))
|
||||
})
|
||||
|
||||
It("Should CommandGetKeys", func() {
|
||||
keys, err := client.CommandGetKeys(ctx, "MSET", "a", "b", "c", "d", "e", "f").Result()
|
||||
keys, err := rawClient.CommandGetKeys(ctx, "MSET", "a", "b", "c", "d", "e", "f").Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(keys).To(Equal([]string{"a", "c", "e"}))
|
||||
|
||||
keys, err = client.CommandGetKeys(ctx, "EVAL", "not consulted", "3", "key1", "key2", "key3", "arg1", "arg2", "arg3", "argN").Result()
|
||||
keys, err = rawClient.CommandGetKeys(ctx, "EVAL", "not consulted", "3", "key1", "key2", "key3", "arg1", "arg2", "arg3", "argN").Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(keys).To(Equal([]string{"key1", "key2", "key3"}))
|
||||
|
||||
keys, err = client.CommandGetKeys(ctx, "SORT", "mylist", "ALPHA", "STORE", "outlist").Result()
|
||||
keys, err = rawClient.CommandGetKeys(ctx, "SORT", "mylist", "ALPHA", "STORE", "outlist").Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(keys).To(Equal([]string{"mylist", "outlist"}))
|
||||
|
||||
_, err = client.CommandGetKeys(ctx, "FAKECOMMAND", "arg1", "arg2").Result()
|
||||
_, err = rawClient.CommandGetKeys(ctx, "FAKECOMMAND", "arg1", "arg2").Result()
|
||||
Expect(err).To(HaveOccurred())
|
||||
Expect(err).To(MatchError("ERR Invalid command specified"))
|
||||
})
|
||||
|
||||
It("should CommandGetKeysAndFlags", func() {
|
||||
keysAndFlags, err := client.CommandGetKeysAndFlags(ctx, "LMOVE", "mylist1", "mylist2", "left", "left").Result()
|
||||
keysAndFlags, err := rawClient.CommandGetKeysAndFlags(ctx, "LMOVE", "mylist1", "mylist2", "left", "left").Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(keysAndFlags).To(Equal([]redis.KeyFlags{
|
||||
{
|
||||
@@ -205,19 +216,19 @@ var _ = Describe("Commands", func() {
|
||||
},
|
||||
}))
|
||||
|
||||
_, err = client.CommandGetKeysAndFlags(ctx, "FAKECOMMAND", "arg1", "arg2").Result()
|
||||
_, err = rawClient.CommandGetKeysAndFlags(ctx, "FAKECOMMAND", "arg1", "arg2").Result()
|
||||
Expect(err).To(HaveOccurred())
|
||||
Expect(err).To(MatchError("ERR Invalid command specified"))
|
||||
})
|
||||
|
||||
It("should ClientKill", func() {
|
||||
r := client.ClientKill(ctx, "1.1.1.1:1111")
|
||||
r := rawClient.ClientKill(ctx, "1.1.1.1:1111")
|
||||
Expect(r.Err()).To(MatchError("ERR No such client"))
|
||||
Expect(r.Val()).To(Equal(""))
|
||||
})
|
||||
|
||||
It("should ClientKillByFilter", func() {
|
||||
r := client.ClientKillByFilter(ctx, "TYPE", "test")
|
||||
r := rawClient.ClientKillByFilter(ctx, "TYPE", "test")
|
||||
Expect(r.Err()).To(MatchError("ERR Unknown client type 'test'"))
|
||||
Expect(r.Val()).To(Equal(int64(0)))
|
||||
})
|
||||
@@ -231,16 +242,16 @@ var _ = Describe("Commands", func() {
|
||||
defer func() {
|
||||
Expect(db.Close()).NotTo(HaveOccurred())
|
||||
}()
|
||||
val, err := client.ClientList(ctx).Result()
|
||||
val, err := rawClient.ClientList(ctx).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).Should(ContainSubstring("name=killmyid"))
|
||||
|
||||
myid := db.ClientID(ctx).Val()
|
||||
killed := client.ClientKillByFilter(ctx, "ID", strconv.FormatInt(myid, 10))
|
||||
killed := rawClient.ClientKillByFilter(ctx, "ID", strconv.FormatInt(myid, 10))
|
||||
Expect(killed.Err()).NotTo(HaveOccurred())
|
||||
Expect(killed.Val()).To(BeNumerically("==", 1))
|
||||
|
||||
val, err = client.ClientList(ctx).Result()
|
||||
val, err = rawClient.ClientList(ctx).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).ShouldNot(ContainSubstring("name=killmyid"))
|
||||
})
|
||||
@@ -268,7 +279,7 @@ var _ = Describe("Commands", func() {
|
||||
// ok
|
||||
}
|
||||
|
||||
killed := client.ClientKillByFilter(ctx, "MAXAGE", "1")
|
||||
killed := rawClient.ClientKillByFilter(ctx, "MAXAGE", "1")
|
||||
Expect(killed.Err()).NotTo(HaveOccurred())
|
||||
Expect(killed.Val()).To(BeNumerically(">=", 1))
|
||||
|
||||
@@ -281,21 +292,21 @@ var _ = Describe("Commands", func() {
|
||||
})
|
||||
|
||||
It("should ClientID", func() {
|
||||
err := client.ClientID(ctx).Err()
|
||||
err := rawClient.ClientID(ctx).Err()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(client.ClientID(ctx).Val()).To(BeNumerically(">=", 0))
|
||||
Expect(rawClient.ClientID(ctx).Val()).To(BeNumerically(">=", 0))
|
||||
})
|
||||
|
||||
It("should ClientUnblock", func() {
|
||||
id := client.ClientID(ctx).Val()
|
||||
r, err := client.ClientUnblock(ctx, id).Result()
|
||||
id := rawClient.ClientID(ctx).Val()
|
||||
r, err := rawClient.ClientUnblock(ctx, id).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(r).To(Equal(int64(0)))
|
||||
})
|
||||
|
||||
It("should ClientUnblockWithError", func() {
|
||||
id := client.ClientID(ctx).Val()
|
||||
r, err := client.ClientUnblockWithError(ctx, id).Result()
|
||||
id := rawClient.ClientID(ctx).Val()
|
||||
r, err := rawClient.ClientUnblockWithError(ctx, id).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(r).To(Equal(int64(0)))
|
||||
})
|
||||
@@ -303,17 +314,17 @@ var _ = Describe("Commands", func() {
|
||||
// NonRedisEnterprise: Redis Enterprise CLIENT INFO advertises an extra
|
||||
// connection flag ("m") that the OSS client-info parser does not recognize.
|
||||
It("should ClientInfo", Label("NonRedisEnterprise"), func() {
|
||||
info, err := client.ClientInfo(ctx).Result()
|
||||
info, err := rawClient.ClientInfo(ctx).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(info).NotTo(BeNil())
|
||||
})
|
||||
|
||||
It("should ClientPause", Label("NonRedisEnterprise"), func() {
|
||||
err := client.ClientPause(ctx, time.Second).Err()
|
||||
err := rawClient.ClientPause(ctx, time.Second).Err()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
|
||||
start := time.Now()
|
||||
err = client.Ping(ctx).Err()
|
||||
err = rawClient.Ping(ctx).Err()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(time.Now()).To(BeTemporally("~", start.Add(time.Second), 800*time.Millisecond))
|
||||
})
|
||||
@@ -383,7 +394,7 @@ var _ = Describe("Commands", func() {
|
||||
pipe.ClientSetInfo(ctx, libInfo)
|
||||
}).To(Panic())
|
||||
// Test setting the default options for libName, libName suffix and libVer
|
||||
clientInfo := client.ClientInfo(ctx).Val()
|
||||
clientInfo := rawClient.ClientInfo(ctx).Val()
|
||||
Expect(clientInfo.LibName).To(ContainSubstring("go-redis(go-redis,"))
|
||||
// Test setting the libName suffix in options
|
||||
opt := redisOptions()
|
||||
@@ -395,9 +406,9 @@ var _ = Describe("Commands", func() {
|
||||
})
|
||||
|
||||
It("should ClientTracking ON/OFF", func() {
|
||||
skipIfClientTrackingUnavailable(ctx, client)
|
||||
skipIfClientTrackingUnavailable(ctx, rawClient)
|
||||
|
||||
conn := client.Conn()
|
||||
conn := rawClient.Conn()
|
||||
defer conn.Close()
|
||||
|
||||
status, err := conn.ClientTrackingOn(ctx, nil).Result()
|
||||
@@ -418,9 +429,9 @@ var _ = Describe("Commands", func() {
|
||||
})
|
||||
|
||||
It("should ClientTracking with BCAST and NOLOOP", func() {
|
||||
skipIfClientTrackingUnavailable(ctx, client)
|
||||
skipIfClientTrackingUnavailable(ctx, rawClient)
|
||||
|
||||
conn := client.Conn()
|
||||
conn := rawClient.Conn()
|
||||
defer conn.Close()
|
||||
|
||||
status, err := conn.ClientTracking(ctx, true, &redis.ClientTrackingOptions{
|
||||
@@ -461,7 +472,7 @@ var _ = Describe("Commands", func() {
|
||||
})
|
||||
|
||||
It("should ConfigGet", func() {
|
||||
val, err := client.ConfigGet(ctx, "*").Result()
|
||||
val, err := rawClient.ConfigGet(ctx, "*").Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).NotTo(BeEmpty())
|
||||
})
|
||||
@@ -478,7 +489,7 @@ var _ = Describe("Commands", func() {
|
||||
}
|
||||
|
||||
for prefix, lookup := range expected {
|
||||
val, err := client.ConfigGet(ctx, prefix).Result()
|
||||
val, err := rawClient.ConfigGet(ctx, prefix).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).NotTo(BeEmpty())
|
||||
Expect(val[lookup]).NotTo(BeEmpty())
|
||||
@@ -486,26 +497,26 @@ var _ = Describe("Commands", func() {
|
||||
})
|
||||
|
||||
It("should ConfigResetStat", Label("NonRedisEnterprise"), func() {
|
||||
r := client.ConfigResetStat(ctx)
|
||||
r := rawClient.ConfigResetStat(ctx)
|
||||
Expect(r.Err()).NotTo(HaveOccurred())
|
||||
Expect(r.Val()).To(Equal("OK"))
|
||||
})
|
||||
|
||||
It("should ConfigSet", Label("NonRedisEnterprise"), func() {
|
||||
configGet := client.ConfigGet(ctx, "maxmemory")
|
||||
configGet := rawClient.ConfigGet(ctx, "maxmemory")
|
||||
Expect(configGet.Err()).NotTo(HaveOccurred())
|
||||
Expect(configGet.Val()).To(HaveLen(1))
|
||||
_, ok := configGet.Val()["maxmemory"]
|
||||
Expect(ok).To(BeTrue())
|
||||
|
||||
configSet := client.ConfigSet(ctx, "maxmemory", configGet.Val()["maxmemory"])
|
||||
configSet := rawClient.ConfigSet(ctx, "maxmemory", configGet.Val()["maxmemory"])
|
||||
Expect(configSet.Err()).NotTo(HaveOccurred())
|
||||
Expect(configSet.Val()).To(Equal("OK"))
|
||||
})
|
||||
|
||||
It("should ConfigGet with Modules", Label("NonRedisEnterprise"), func() {
|
||||
SkipBeforeRedisVersion("8", "config get won't return modules configs before redis 8")
|
||||
configGet := client.ConfigGet(ctx, "*")
|
||||
configGet := rawClient.ConfigGet(ctx, "*")
|
||||
Expect(configGet.Err()).NotTo(HaveOccurred())
|
||||
Expect(configGet.Val()).To(HaveKey("maxmemory"))
|
||||
Expect(configGet.Val()).To(HaveKey("search-timeout"))
|
||||
@@ -518,11 +529,11 @@ var _ = Describe("Commands", func() {
|
||||
// (superseded by CONFIG GET/SET in Redis 8), so this OSS-only path errors.
|
||||
It("should ConfigSet FT DIALECT", Label("NonRedisEnterprise"), func() {
|
||||
SkipBeforeRedisVersion("8", "config doesn't include modules before Redis 8")
|
||||
defaultState, err := client.ConfigGet(ctx, "search-default-dialect").Result()
|
||||
defaultState, err := rawClient.ConfigGet(ctx, "search-default-dialect").Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
|
||||
// set to 3
|
||||
res, err := client.ConfigSet(ctx, "search-default-dialect", "3").Result()
|
||||
res, err := rawClient.ConfigSet(ctx, "search-default-dialect", "3").Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(res).To(BeEquivalentTo("OK"))
|
||||
|
||||
@@ -530,12 +541,12 @@ var _ = Describe("Commands", func() {
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(defDialect).To(BeEquivalentTo(map[string]interface{}{"DEFAULT_DIALECT": "3"}))
|
||||
|
||||
resGet, err := client.ConfigGet(ctx, "search-default-dialect").Result()
|
||||
resGet, err := rawClient.ConfigGet(ctx, "search-default-dialect").Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(resGet).To(BeEquivalentTo(map[string]string{"search-default-dialect": "3"}))
|
||||
|
||||
// set to 2
|
||||
res, err = client.ConfigSet(ctx, "search-default-dialect", "2").Result()
|
||||
res, err = rawClient.ConfigSet(ctx, "search-default-dialect", "2").Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(res).To(BeEquivalentTo("OK"))
|
||||
|
||||
@@ -544,7 +555,7 @@ var _ = Describe("Commands", func() {
|
||||
Expect(defDialect).To(BeEquivalentTo(map[string]interface{}{"DEFAULT_DIALECT": "2"}))
|
||||
|
||||
// set to 1
|
||||
res, err = client.ConfigSet(ctx, "search-default-dialect", "1").Result()
|
||||
res, err = rawClient.ConfigSet(ctx, "search-default-dialect", "1").Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(res).To(BeEquivalentTo("OK"))
|
||||
|
||||
@@ -552,19 +563,19 @@ var _ = Describe("Commands", func() {
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(defDialect).To(BeEquivalentTo(map[string]interface{}{"DEFAULT_DIALECT": "1"}))
|
||||
|
||||
resGet, err = client.ConfigGet(ctx, "search-default-dialect").Result()
|
||||
resGet, err = rawClient.ConfigGet(ctx, "search-default-dialect").Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(resGet).To(BeEquivalentTo(map[string]string{"search-default-dialect": "1"}))
|
||||
|
||||
// set to default
|
||||
res, err = client.ConfigSet(ctx, "search-default-dialect", defaultState["search-default-dialect"]).Result()
|
||||
res, err = rawClient.ConfigSet(ctx, "search-default-dialect", defaultState["search-default-dialect"]).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(res).To(BeEquivalentTo("OK"))
|
||||
})
|
||||
|
||||
It("should ConfigSet fail for ReadOnly", func() {
|
||||
SkipBeforeRedisVersion("8", "Config doesn't include modules before Redis 8")
|
||||
_, err := client.ConfigSet(ctx, "search-max-doctablesize", "100000").Result()
|
||||
_, err := rawClient.ConfigSet(ctx, "search-max-doctablesize", "100000").Result()
|
||||
Expect(err).To(HaveOccurred())
|
||||
})
|
||||
|
||||
@@ -582,21 +593,21 @@ var _ = Describe("Commands", func() {
|
||||
|
||||
// read the defaults to set them back later
|
||||
for setting := range expected {
|
||||
val, err := client.ConfigGet(ctx, setting).Result()
|
||||
val, err := rawClient.ConfigGet(ctx, setting).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
defaults[setting] = val[setting]
|
||||
}
|
||||
|
||||
// check if new values can be set
|
||||
for setting, value := range expected {
|
||||
val, err := client.ConfigSet(ctx, setting, value).Result()
|
||||
val, err := rawClient.ConfigSet(ctx, setting, value).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).NotTo(BeEmpty())
|
||||
Expect(val).To(Equal("OK"))
|
||||
}
|
||||
|
||||
for setting, value := range expected {
|
||||
val, err := client.ConfigGet(ctx, setting).Result()
|
||||
val, err := rawClient.ConfigGet(ctx, setting).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).NotTo(BeEmpty())
|
||||
Expect(val[setting]).To(Equal(value))
|
||||
@@ -604,7 +615,7 @@ var _ = Describe("Commands", func() {
|
||||
|
||||
// set back to the defaults
|
||||
for setting, value := range defaults {
|
||||
val, err := client.ConfigSet(ctx, setting, value).Result()
|
||||
val, err := rawClient.ConfigSet(ctx, setting, value).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).NotTo(BeEmpty())
|
||||
Expect(val).To(Equal("OK"))
|
||||
@@ -621,7 +632,7 @@ var _ = Describe("Commands", func() {
|
||||
}
|
||||
|
||||
for setting, value := range expected {
|
||||
val, err := client.ConfigSet(ctx, setting, value).Result()
|
||||
val, err := rawClient.ConfigSet(ctx, setting, value).Result()
|
||||
Expect(err).To(HaveOccurred())
|
||||
Expect(err).To(MatchError(ContainSubstring(setting)))
|
||||
Expect(val).To(BeEmpty())
|
||||
@@ -629,33 +640,33 @@ var _ = Describe("Commands", func() {
|
||||
})
|
||||
|
||||
It("should ConfigRewrite", Label("NonRedisEnterprise"), func() {
|
||||
configRewrite := client.ConfigRewrite(ctx)
|
||||
configRewrite := rawClient.ConfigRewrite(ctx)
|
||||
Expect(configRewrite.Err()).NotTo(HaveOccurred())
|
||||
Expect(configRewrite.Val()).To(Equal("OK"))
|
||||
})
|
||||
|
||||
It("should DBSize", func() {
|
||||
size, err := client.DBSize(ctx).Result()
|
||||
size, err := rawClient.DBSize(ctx).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(size).To(Equal(int64(0)))
|
||||
})
|
||||
|
||||
It("should Info", func() {
|
||||
info := client.Info(ctx)
|
||||
info := rawClient.Info(ctx)
|
||||
Expect(info.Err()).NotTo(HaveOccurred())
|
||||
Expect(info.Val()).NotTo(Equal(""))
|
||||
})
|
||||
|
||||
It("should InfoMap", Label("redis.info"), func() {
|
||||
info := client.InfoMap(ctx)
|
||||
info := rawClient.InfoMap(ctx)
|
||||
Expect(info.Err()).NotTo(HaveOccurred())
|
||||
Expect(info.Val()).NotTo(BeNil())
|
||||
|
||||
info = client.InfoMap(ctx, "dummy")
|
||||
info = rawClient.InfoMap(ctx, "dummy")
|
||||
Expect(info.Err()).NotTo(HaveOccurred())
|
||||
Expect(info.Val()).To(BeNil())
|
||||
|
||||
info = client.InfoMap(ctx, "server")
|
||||
info = rawClient.InfoMap(ctx, "server")
|
||||
Expect(info.Err()).NotTo(HaveOccurred())
|
||||
Expect(info.Val()).To(HaveLen(1))
|
||||
})
|
||||
@@ -664,22 +675,22 @@ var _ = Describe("Commands", func() {
|
||||
// INFO shape, so the OSS module-name assertions do not hold.
|
||||
It("should Info Modules", Label("redis.info", "NonRedisEnterprise"), func() {
|
||||
SkipBeforeRedisVersion("8", "modules are included in info for Redis Version >= 8")
|
||||
info := client.Info(ctx)
|
||||
info := rawClient.Info(ctx)
|
||||
Expect(info.Err()).NotTo(HaveOccurred())
|
||||
Expect(info.Val()).NotTo(BeNil())
|
||||
|
||||
info = client.Info(ctx, "search")
|
||||
info = rawClient.Info(ctx, "search")
|
||||
Expect(info.Err()).NotTo(HaveOccurred())
|
||||
Expect(info.Val()).To(ContainSubstring("search"))
|
||||
|
||||
info = client.Info(ctx, "modules")
|
||||
info = rawClient.Info(ctx, "modules")
|
||||
Expect(info.Err()).NotTo(HaveOccurred())
|
||||
Expect(info.Val()).To(ContainSubstring("search"))
|
||||
Expect(info.Val()).To(ContainSubstring("ReJSON"))
|
||||
Expect(info.Val()).To(ContainSubstring("timeseries"))
|
||||
Expect(info.Val()).To(ContainSubstring("bf"))
|
||||
|
||||
info = client.Info(ctx, "everything")
|
||||
info = rawClient.Info(ctx, "everything")
|
||||
Expect(info.Err()).NotTo(HaveOccurred())
|
||||
Expect(info.Val()).To(ContainSubstring("search"))
|
||||
Expect(info.Val()).To(ContainSubstring("ReJSON"))
|
||||
@@ -691,16 +702,16 @@ var _ = Describe("Commands", func() {
|
||||
// different INFO layout from OSS, so the parsed module map is empty here.
|
||||
It("should InfoMap Modules", Label("redis.info", "NonRedisEnterprise"), func() {
|
||||
SkipBeforeRedisVersion("8", "modules are included in info for Redis Version >= 8")
|
||||
info := client.InfoMap(ctx)
|
||||
info := rawClient.InfoMap(ctx)
|
||||
Expect(info.Err()).NotTo(HaveOccurred())
|
||||
Expect(info.Val()).NotTo(BeNil())
|
||||
|
||||
info = client.InfoMap(ctx, "search")
|
||||
info = rawClient.InfoMap(ctx, "search")
|
||||
Expect(info.Err()).NotTo(HaveOccurred())
|
||||
Expect(len(info.Val())).To(BeNumerically(">=", 2))
|
||||
Expect(info.Val()["search_version"]).ToNot(BeNil())
|
||||
|
||||
info = client.InfoMap(ctx, "modules")
|
||||
info = rawClient.InfoMap(ctx, "modules")
|
||||
Expect(info.Err()).NotTo(HaveOccurred())
|
||||
val := info.Val()
|
||||
modules, ok := val["Modules"]
|
||||
@@ -712,20 +723,20 @@ var _ = Describe("Commands", func() {
|
||||
Expect(modules["timeseries"]).ToNot(BeNil())
|
||||
Expect(modules["bf"]).ToNot(BeNil())
|
||||
|
||||
info = client.InfoMap(ctx, "everything")
|
||||
info = rawClient.InfoMap(ctx, "everything")
|
||||
Expect(info.Err()).NotTo(HaveOccurred())
|
||||
Expect(len(info.Val())).To(BeNumerically(">=", 10))
|
||||
})
|
||||
|
||||
It("should Info cpu", func() {
|
||||
info := client.Info(ctx, "cpu")
|
||||
info := rawClient.Info(ctx, "cpu")
|
||||
Expect(info.Err()).NotTo(HaveOccurred())
|
||||
Expect(info.Val()).NotTo(Equal(""))
|
||||
Expect(info.Val()).To(ContainSubstring(`used_cpu_sys`))
|
||||
})
|
||||
|
||||
It("should Info cpu and memory", func() {
|
||||
info := client.Info(ctx, "cpu", "memory")
|
||||
info := rawClient.Info(ctx, "cpu", "memory")
|
||||
Expect(info.Err()).NotTo(HaveOccurred())
|
||||
Expect(info.Val()).NotTo(Equal(""))
|
||||
Expect(info.Val()).To(ContainSubstring(`used_cpu_sys`))
|
||||
@@ -733,7 +744,7 @@ var _ = Describe("Commands", func() {
|
||||
})
|
||||
|
||||
It("should LastSave", Label("NonRedisEnterprise"), func() {
|
||||
lastSave := client.LastSave(ctx)
|
||||
lastSave := rawClient.LastSave(ctx)
|
||||
Expect(lastSave.Err()).NotTo(HaveOccurred())
|
||||
Expect(lastSave.Val()).NotTo(Equal(0))
|
||||
})
|
||||
@@ -741,38 +752,38 @@ var _ = Describe("Commands", func() {
|
||||
It("should Save", Label("NonRedisEnterprise"), func() {
|
||||
// workaround for "ERR Background save already in progress"
|
||||
Eventually(func() string {
|
||||
return client.Save(ctx).Val()
|
||||
return rawClient.Save(ctx).Val()
|
||||
}, "10s").Should(Equal("OK"))
|
||||
})
|
||||
|
||||
It("should SlaveOf", Label("NonRedisEnterprise"), func() {
|
||||
slaveOf := client.SlaveOf(ctx, "localhost", "8888")
|
||||
slaveOf := rawClient.SlaveOf(ctx, "localhost", "8888")
|
||||
Expect(slaveOf.Err()).NotTo(HaveOccurred())
|
||||
Expect(slaveOf.Val()).To(Equal("OK"))
|
||||
|
||||
slaveOf = client.SlaveOf(ctx, "NO", "ONE")
|
||||
slaveOf = rawClient.SlaveOf(ctx, "NO", "ONE")
|
||||
Expect(slaveOf.Err()).NotTo(HaveOccurred())
|
||||
Expect(slaveOf.Val()).To(Equal("OK"))
|
||||
})
|
||||
|
||||
It("should ReplicaOf", Label("NonRedisEnterprise"), func() {
|
||||
replicaOf := client.ReplicaOf(ctx, "localhost", "8888")
|
||||
replicaOf := rawClient.ReplicaOf(ctx, "localhost", "8888")
|
||||
Expect(replicaOf.Err()).NotTo(HaveOccurred())
|
||||
Expect(replicaOf.Val()).To(Equal("OK"))
|
||||
|
||||
replicaOf = client.ReplicaOf(ctx, "NO", "ONE")
|
||||
replicaOf = rawClient.ReplicaOf(ctx, "NO", "ONE")
|
||||
Expect(replicaOf.Err()).NotTo(HaveOccurred())
|
||||
Expect(replicaOf.Val()).To(Equal("OK"))
|
||||
})
|
||||
|
||||
It("should Time", func() {
|
||||
tm, err := client.Time(ctx).Result()
|
||||
tm, err := rawClient.Time(ctx).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(tm).To(BeTemporally("~", time.Now(), 3*time.Second))
|
||||
})
|
||||
|
||||
It("should Command", Label("NonRedisEnterprise"), func() {
|
||||
cmds, err := client.Command(ctx).Result()
|
||||
cmds, err := rawClient.Command(ctx).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
|
||||
cmd := cmds["mget"]
|
||||
@@ -794,7 +805,7 @@ var _ = Describe("Commands", func() {
|
||||
|
||||
It("should Command Tips", Label("NonRedisEnterprise"), func() {
|
||||
SkipAfterRedisVersion("7.9", "Redis 8 changed the COMMAND reply format")
|
||||
cmds, err := client.Command(ctx).Result()
|
||||
cmds, err := rawClient.Command(ctx).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
|
||||
cmd := cmds["touch"]
|
||||
@@ -809,7 +820,7 @@ var _ = Describe("Commands", func() {
|
||||
})
|
||||
|
||||
It("should return all command names", func() {
|
||||
cmdList := client.CommandList(ctx, nil)
|
||||
cmdList := rawClient.CommandList(ctx, nil)
|
||||
Expect(cmdList.Err()).NotTo(HaveOccurred())
|
||||
cmdNames := cmdList.Val()
|
||||
|
||||
@@ -825,7 +836,7 @@ var _ = Describe("Commands", func() {
|
||||
filter := &redis.FilterBy{
|
||||
Module: "JSON",
|
||||
}
|
||||
cmdList := client.CommandList(ctx, filter)
|
||||
cmdList := rawClient.CommandList(ctx, filter)
|
||||
Expect(cmdList.Err()).NotTo(HaveOccurred())
|
||||
Expect(cmdList.Val()).To(HaveLen(0))
|
||||
})
|
||||
@@ -835,7 +846,7 @@ var _ = Describe("Commands", func() {
|
||||
ACLCat: "admin",
|
||||
}
|
||||
|
||||
cmdList := client.CommandList(ctx, filter)
|
||||
cmdList := rawClient.CommandList(ctx, filter)
|
||||
Expect(cmdList.Err()).NotTo(HaveOccurred())
|
||||
cmdNames := cmdList.Val()
|
||||
|
||||
@@ -847,7 +858,7 @@ var _ = Describe("Commands", func() {
|
||||
filter := &redis.FilterBy{
|
||||
Pattern: "*GET*",
|
||||
}
|
||||
cmdList := client.CommandList(ctx, filter)
|
||||
cmdList := rawClient.CommandList(ctx, filter)
|
||||
Expect(cmdList.Err()).NotTo(HaveOccurred())
|
||||
cmdNames := cmdList.Val()
|
||||
|
||||
@@ -861,33 +872,33 @@ var _ = Describe("Commands", func() {
|
||||
|
||||
Describe("debugging", Label("NonRedisEnterprise"), func() {
|
||||
It("should DebugObject", func() {
|
||||
err := client.DebugObject(ctx, "foo").Err()
|
||||
err := rawClient.DebugObject(ctx, "foo").Err()
|
||||
Expect(err).To(MatchError("ERR no such key"))
|
||||
|
||||
err = client.Set(ctx, "foo", "bar", 0).Err()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
|
||||
s, err := client.DebugObject(ctx, "foo").Result()
|
||||
s, err := rawClient.DebugObject(ctx, "foo").Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(s).To(ContainSubstring("serializedlength:4"))
|
||||
})
|
||||
|
||||
It("should MemoryUsage", func() {
|
||||
err := client.MemoryUsage(ctx, "foo").Err()
|
||||
err := rawClient.MemoryUsage(ctx, "foo").Err()
|
||||
Expect(err).To(Equal(redis.Nil))
|
||||
|
||||
err = client.Set(ctx, "foo", "bar", 0).Err()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
|
||||
n, err := client.MemoryUsage(ctx, "foo").Result()
|
||||
n, err := rawClient.MemoryUsage(ctx, "foo").Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(n).NotTo(BeZero())
|
||||
|
||||
n, err = client.MemoryUsage(ctx, "foo", 0).Result()
|
||||
n, err = rawClient.MemoryUsage(ctx, "foo", 0).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(n).NotTo(BeZero())
|
||||
|
||||
_, err = client.MemoryUsage(ctx, "foo", 0, 1).Result()
|
||||
_, err = rawClient.MemoryUsage(ctx, "foo", 0, 1).Result()
|
||||
Expect(err).Should(MatchError("MemoryUsage expects single sample count"))
|
||||
})
|
||||
})
|
||||
@@ -1071,10 +1082,10 @@ var _ = Describe("Commands", func() {
|
||||
Expect(refCount.Err()).NotTo(HaveOccurred())
|
||||
Expect(refCount.Val()).To(Equal(int64(1)))
|
||||
|
||||
client.ConfigSet(ctx, "maxmemory-policy", "volatile-lfu")
|
||||
rawClient.ConfigSet(ctx, "maxmemory-policy", "volatile-lfu")
|
||||
freq := client.ObjectFreq(ctx, "key")
|
||||
Expect(freq.Err()).NotTo(HaveOccurred())
|
||||
client.ConfigSet(ctx, "maxmemory-policy", "noeviction") // default
|
||||
rawClient.ConfigSet(ctx, "maxmemory-policy", "noeviction") // default
|
||||
|
||||
err := client.ObjectEncoding(ctx, "key").Err()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
@@ -1986,7 +1997,7 @@ var _ = Describe("Commands", func() {
|
||||
Expect(string(orig.Bytes())).To(Equal("value"))
|
||||
|
||||
clone := orig.Clone()
|
||||
Expect(client.Process(ctx, clone)).To(MatchError(ContainSubstring("cannot be cloned")))
|
||||
Expect(rawClient.Process(ctx, clone)).To(MatchError(ContainSubstring("cannot be cloned")))
|
||||
|
||||
// Follow-up call on the same client must work: the clone's
|
||||
// readReply drained the network reply so the connection
|
||||
@@ -3472,7 +3483,7 @@ var _ = Describe("Commands", func() {
|
||||
})
|
||||
|
||||
It("should fail module loadex", Label("NonRedisEnterprise"), func() {
|
||||
dryRun := client.ModuleLoadex(ctx, &redis.ModuleLoadexConfig{
|
||||
dryRun := rawClient.ModuleLoadex(ctx, &redis.ModuleLoadexConfig{
|
||||
Path: "/path/to/non-existent-library.so",
|
||||
Conf: map[string]interface{}{
|
||||
"param1": "value1",
|
||||
@@ -4344,9 +4355,9 @@ var _ = Describe("Commands", func() {
|
||||
Expect(err).To(Equal(redis.Nil))
|
||||
Expect(val).To(BeNil())
|
||||
|
||||
Expect(client.Ping(ctx).Err()).NotTo(HaveOccurred())
|
||||
Expect(rawClient.Ping(ctx).Err()).NotTo(HaveOccurred())
|
||||
|
||||
stats := client.PoolStats()
|
||||
stats := rawClient.PoolStats()
|
||||
Expect(stats.Hits).To(Equal(uint32(2)))
|
||||
Expect(stats.Misses).To(Equal(uint32(1)))
|
||||
Expect(stats.Timeouts).To(Equal(uint32(0)))
|
||||
@@ -4626,9 +4637,9 @@ var _ = Describe("Commands", func() {
|
||||
Expect(err).To(Equal(redis.Nil))
|
||||
Expect(val).To(BeNil())
|
||||
|
||||
Expect(client.Ping(ctx).Err()).NotTo(HaveOccurred())
|
||||
Expect(rawClient.Ping(ctx).Err()).NotTo(HaveOccurred())
|
||||
|
||||
stats := client.PoolStats()
|
||||
stats := rawClient.PoolStats()
|
||||
Expect(stats.Hits).To(Equal(uint32(2)))
|
||||
Expect(stats.Misses).To(Equal(uint32(1)))
|
||||
Expect(stats.Timeouts).To(Equal(uint32(0)))
|
||||
@@ -5149,9 +5160,9 @@ var _ = Describe("Commands", func() {
|
||||
Expect(err).To(Equal(redis.Nil))
|
||||
Expect(val).To(BeNil())
|
||||
|
||||
Expect(client.Ping(ctx).Err()).NotTo(HaveOccurred())
|
||||
Expect(rawClient.Ping(ctx).Err()).NotTo(HaveOccurred())
|
||||
|
||||
stats := client.PoolStats()
|
||||
stats := rawClient.PoolStats()
|
||||
Expect(stats.Hits).To(Equal(uint32(2)))
|
||||
Expect(stats.Misses).To(Equal(uint32(1)))
|
||||
Expect(stats.Timeouts).To(Equal(uint32(0)))
|
||||
@@ -5652,9 +5663,9 @@ var _ = Describe("Commands", func() {
|
||||
Expect(err).To(Equal(redis.Nil))
|
||||
Expect(val).To(BeNil())
|
||||
|
||||
Expect(client.Ping(ctx).Err()).NotTo(HaveOccurred())
|
||||
Expect(rawClient.Ping(ctx).Err()).NotTo(HaveOccurred())
|
||||
|
||||
stats := client.PoolStats()
|
||||
stats := rawClient.PoolStats()
|
||||
Expect(stats.Hits).To(Equal(uint32(2)))
|
||||
Expect(stats.Misses).To(Equal(uint32(1)))
|
||||
Expect(stats.Timeouts).To(Equal(uint32(0)))
|
||||
@@ -5734,9 +5745,9 @@ var _ = Describe("Commands", func() {
|
||||
Expect(err).To(Equal(redis.Nil))
|
||||
Expect(val).To(BeNil())
|
||||
|
||||
Expect(client.Ping(ctx).Err()).NotTo(HaveOccurred())
|
||||
Expect(rawClient.Ping(ctx).Err()).NotTo(HaveOccurred())
|
||||
|
||||
stats := client.PoolStats()
|
||||
stats := rawClient.PoolStats()
|
||||
Expect(stats.Hits).To(Equal(uint32(2)))
|
||||
Expect(stats.Misses).To(Equal(uint32(1)))
|
||||
Expect(stats.Timeouts).To(Equal(uint32(0)))
|
||||
@@ -6472,9 +6483,9 @@ var _ = Describe("Commands", func() {
|
||||
Expect(err).To(Equal(redis.Nil))
|
||||
Expect(val).To(BeNil())
|
||||
|
||||
Expect(client.Ping(ctx).Err()).NotTo(HaveOccurred())
|
||||
Expect(rawClient.Ping(ctx).Err()).NotTo(HaveOccurred())
|
||||
|
||||
stats := client.PoolStats()
|
||||
stats := rawClient.PoolStats()
|
||||
Expect(stats.Hits).To(Equal(uint32(2)))
|
||||
Expect(stats.Misses).To(Equal(uint32(1)))
|
||||
Expect(stats.Timeouts).To(Equal(uint32(0)))
|
||||
@@ -9986,44 +9997,46 @@ var _ = Describe("Commands", func() {
|
||||
It("returns slow query result", func() {
|
||||
const key = "slowlog-log-slower-than"
|
||||
|
||||
old := client.ConfigGet(ctx, key).Val()
|
||||
client.ConfigSet(ctx, key, "0")
|
||||
defer client.ConfigSet(ctx, key, old[key])
|
||||
old := rawClient.ConfigGet(ctx, key).Val()
|
||||
rawClient.ConfigSet(ctx, key, "0")
|
||||
defer rawClient.ConfigSet(ctx, key, old[key])
|
||||
|
||||
err := client.Do(ctx, "slowlog", "reset").Err()
|
||||
err := rawClient.Do(ctx, "slowlog", "reset").Err()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
|
||||
client.Set(ctx, "test", "true", 0)
|
||||
Expect(client.Set(ctx, "test", "true", 0).Err()).NotTo(HaveOccurred())
|
||||
|
||||
result, err := client.SlowLogGet(ctx, -1).Result()
|
||||
result, err := rawClient.SlowLogGet(ctx, -1).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(len(result)).NotTo(BeZero())
|
||||
})
|
||||
|
||||
It("returns the number of slow queries", Label("NonRedisEnterprise"), func() {
|
||||
// Reset slowlog
|
||||
err := client.SlowLogReset(ctx).Err()
|
||||
err := rawClient.SlowLogReset(ctx).Err()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
|
||||
const key = "slowlog-log-slower-than"
|
||||
|
||||
old := client.ConfigGet(ctx, key).Val()
|
||||
old := rawClient.ConfigGet(ctx, key).Val()
|
||||
// first slowlog entry is the config set command itself
|
||||
client.ConfigSet(ctx, key, "0")
|
||||
defer client.ConfigSet(ctx, key, old[key])
|
||||
rawClient.ConfigSet(ctx, key, "0")
|
||||
defer rawClient.ConfigSet(ctx, key, old[key])
|
||||
|
||||
// Set a key to trigger a slow query, and this is the second slowlog entry
|
||||
client.Set(ctx, "test", "true", 0)
|
||||
result, err := client.SlowLogLen(ctx).Result()
|
||||
// Set a key to trigger a slow query, and this is the second slowlog entry.
|
||||
// Read the result so the command has actually executed before we count
|
||||
// (required on the deferred autopipeline face, harmless otherwise).
|
||||
Expect(client.Set(ctx, "test", "true", 0).Err()).NotTo(HaveOccurred())
|
||||
result, err := rawClient.SlowLogLen(ctx).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(result).Should(Equal(int64(2)))
|
||||
|
||||
// Reset slowlog
|
||||
err = client.SlowLogReset(ctx).Err()
|
||||
err = rawClient.SlowLogReset(ctx).Err()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
|
||||
// Check if slowlog is empty, this is the first slowlog entry after reset
|
||||
result, err = client.SlowLogLen(ctx).Result()
|
||||
result, err = rawClient.SlowLogLen(ctx).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(result).Should(Equal(int64(1)))
|
||||
})
|
||||
@@ -10031,16 +10044,21 @@ var _ = Describe("Commands", func() {
|
||||
It("returns the command argument count", Label("NonRedisEnterprise"), func() {
|
||||
SkipBeforeRedisVersion("8.10", "SLOWLOG GET reports the command argument count since Redis 8.10")
|
||||
|
||||
// Config*/SlowLog/Do stay on the raw client: on the deferred
|
||||
// autopipeline face a Do runs outside the pipeline and could
|
||||
// execute before a just-enqueued unread ConfigSet flushes.
|
||||
const key = "slowlog-log-slower-than"
|
||||
old := client.ConfigGet(ctx, key).Val()
|
||||
client.ConfigSet(ctx, key, "0")
|
||||
defer client.ConfigSet(ctx, key, old[key])
|
||||
old := rawClient.ConfigGet(ctx, key).Val()
|
||||
rawClient.ConfigSet(ctx, key, "0")
|
||||
defer rawClient.ConfigSet(ctx, key, old[key])
|
||||
|
||||
// A short command: CommandArgc matches the number of stored Args.
|
||||
Expect(client.Do(ctx, "slowlog", "reset").Err()).NotTo(HaveOccurred())
|
||||
client.Set(ctx, "test", "true", 0)
|
||||
Expect(rawClient.Do(ctx, "slowlog", "reset").Err()).NotTo(HaveOccurred())
|
||||
// The measured command goes through the subject; assert the write
|
||||
// so the deferred face has executed it before SlowLogGet reads.
|
||||
Expect(client.Set(ctx, "test", "true", 0).Err()).NotTo(HaveOccurred())
|
||||
|
||||
result, err := client.SlowLogGet(ctx, -1).Result()
|
||||
result, err := rawClient.SlowLogGet(ctx, -1).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
var setEntry *redis.SlowLog
|
||||
for i := range result {
|
||||
@@ -10053,18 +10071,18 @@ var _ = Describe("Commands", func() {
|
||||
|
||||
// A long command: Redis truncates the stored Args at slowlog-max-argc,
|
||||
// but CommandArgc still reports the full original argument count.
|
||||
maxArgc := client.ConfigGet(ctx, "slowlog-max-argc").Val()["slowlog-max-argc"]
|
||||
client.ConfigSet(ctx, "slowlog-max-argc", "32")
|
||||
defer client.ConfigSet(ctx, "slowlog-max-argc", maxArgc)
|
||||
Expect(client.Do(ctx, "slowlog", "reset").Err()).NotTo(HaveOccurred())
|
||||
maxArgc := rawClient.ConfigGet(ctx, "slowlog-max-argc").Val()["slowlog-max-argc"]
|
||||
rawClient.ConfigSet(ctx, "slowlog-max-argc", "32")
|
||||
defer rawClient.ConfigSet(ctx, "slowlog-max-argc", maxArgc)
|
||||
Expect(rawClient.Do(ctx, "slowlog", "reset").Err()).NotTo(HaveOccurred())
|
||||
|
||||
args := []interface{}{"rpush", "slowlog-big-list"}
|
||||
for i := 0; i < 50; i++ {
|
||||
args = append(args, fmt.Sprintf("v%d", i))
|
||||
}
|
||||
Expect(client.Do(ctx, args...).Err()).NotTo(HaveOccurred())
|
||||
Expect(rawClient.Do(ctx, args...).Err()).NotTo(HaveOccurred())
|
||||
|
||||
result, err = client.SlowLogGet(ctx, -1).Result()
|
||||
result, err = rawClient.SlowLogGet(ctx, -1).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
var pushEntry *redis.SlowLog
|
||||
for i := range result {
|
||||
@@ -10085,17 +10103,17 @@ var _ = Describe("Commands", func() {
|
||||
const key = "latency-monitor-threshold"
|
||||
|
||||
// reset all latencies first to ensure clean state
|
||||
err := client.LatencyReset(ctx).Err()
|
||||
err := rawClient.LatencyReset(ctx).Err()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
|
||||
old := client.ConfigGet(ctx, key).Val()
|
||||
client.ConfigSet(ctx, key, "1")
|
||||
defer client.ConfigSet(ctx, key, old[key])
|
||||
old := rawClient.ConfigGet(ctx, key).Val()
|
||||
rawClient.ConfigSet(ctx, key, "1")
|
||||
defer rawClient.ConfigSet(ctx, key, old[key])
|
||||
|
||||
err = client.Do(ctx, "DEBUG", "SLEEP", 0.01).Err()
|
||||
err = rawClient.Do(ctx, "DEBUG", "SLEEP", 0.01).Err()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
|
||||
result, err := client.Latency(ctx).Result()
|
||||
result, err := rawClient.Latency(ctx).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(len(result)).NotTo(BeZero())
|
||||
})
|
||||
@@ -10104,39 +10122,39 @@ var _ = Describe("Commands", func() {
|
||||
const key = "latency-monitor-threshold"
|
||||
|
||||
// reset all latencies
|
||||
err := client.LatencyReset(ctx).Err()
|
||||
err := rawClient.LatencyReset(ctx).Err()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
|
||||
old := client.ConfigGet(ctx, key).Val()
|
||||
old := rawClient.ConfigGet(ctx, key).Val()
|
||||
// Use a higher threshold (100ms) to avoid capturing normal operations
|
||||
// that could cause flakiness due to timing variations on busy CI runners.
|
||||
client.ConfigSet(ctx, key, "100")
|
||||
defer client.ConfigSet(ctx, key, old[key])
|
||||
rawClient.ConfigSet(ctx, key, "100")
|
||||
defer rawClient.ConfigSet(ctx, key, old[key])
|
||||
|
||||
// get latency after reset
|
||||
result, err := client.Latency(ctx).Result()
|
||||
result, err := rawClient.Latency(ctx).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(len(result)).Should(Equal(0))
|
||||
|
||||
// create a new latency event by sleeping past the threshold
|
||||
err = client.Do(ctx, "DEBUG", "SLEEP", 0.15).Err()
|
||||
err = rawClient.Do(ctx, "DEBUG", "SLEEP", 0.15).Err()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
|
||||
// get latency after create a new latency
|
||||
result, err = client.Latency(ctx).Result()
|
||||
result, err = rawClient.Latency(ctx).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(len(result)).Should(BeNumerically(">=", 1))
|
||||
triggered := result[0].Name
|
||||
|
||||
// reset all latencies again
|
||||
err = client.LatencyReset(ctx).Err()
|
||||
err = rawClient.LatencyReset(ctx).Err()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
|
||||
// get latency after reset again. Don't assert total length is 0:
|
||||
// other operations on the connection may briefly land in the
|
||||
// latency log on a slow CI runner. Instead verify the specific
|
||||
// event we triggered is gone.
|
||||
result, err = client.Latency(ctx).Result()
|
||||
result, err = rawClient.Latency(ctx).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
for _, event := range result {
|
||||
if event.Name == triggered {
|
||||
@@ -10149,35 +10167,35 @@ var _ = Describe("Commands", func() {
|
||||
const key = "latency-monitor-threshold"
|
||||
|
||||
// reset all latencies first to ensure clean state
|
||||
err := client.LatencyReset(ctx).Err()
|
||||
err := rawClient.LatencyReset(ctx).Err()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
|
||||
old := client.ConfigGet(ctx, key).Val()
|
||||
old := rawClient.ConfigGet(ctx, key).Val()
|
||||
// Use a higher threshold (100ms) to avoid capturing normal operations
|
||||
// that could cause flakiness due to timing variations
|
||||
client.ConfigSet(ctx, key, "100")
|
||||
defer client.ConfigSet(ctx, key, old[key])
|
||||
rawClient.ConfigSet(ctx, key, "100")
|
||||
defer rawClient.ConfigSet(ctx, key, old[key])
|
||||
|
||||
result, err := client.Latency(ctx).Result()
|
||||
result, err := rawClient.Latency(ctx).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(len(result)).Should(Equal(0))
|
||||
|
||||
// Use a longer sleep (150ms) to ensure it exceeds the 100ms threshold
|
||||
err = client.Do(ctx, "DEBUG", "SLEEP", 0.15).Err()
|
||||
err = rawClient.Do(ctx, "DEBUG", "SLEEP", 0.15).Err()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
|
||||
result, err = client.Latency(ctx).Result()
|
||||
result, err = rawClient.Latency(ctx).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(len(result)).Should(BeNumerically(">=", 1))
|
||||
|
||||
// reset latency by event name
|
||||
eventName := result[0].Name
|
||||
err = client.LatencyReset(ctx, eventName).Err()
|
||||
err = rawClient.LatencyReset(ctx, eventName).Err()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
|
||||
// Verify the specific event was reset (not that all events are gone)
|
||||
// This avoids flakiness from other operations triggering latency events
|
||||
result, err = client.Latency(ctx).Result()
|
||||
result, err = rawClient.Latency(ctx).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
for _, event := range result {
|
||||
if event.Name == eventName {
|
||||
|
||||
@@ -107,6 +107,10 @@ func ExampleClient_search_json() {
|
||||
}
|
||||
// STEP_END
|
||||
|
||||
// REMOVE_START
|
||||
waitForIndexedDocs(ctx, rdb, "idx:users", 3)
|
||||
// REMOVE_END
|
||||
|
||||
// STEP_START query1
|
||||
findPaulResult, err := rdb.FTSearch(
|
||||
ctx,
|
||||
@@ -313,6 +317,10 @@ func ExampleClient_search_hash() {
|
||||
}
|
||||
// STEP_END
|
||||
|
||||
// REMOVE_START
|
||||
waitForIndexedDocs(ctx, rdb, "hash-idx:users", 3)
|
||||
// REMOVE_END
|
||||
|
||||
// STEP_START query1_hash
|
||||
findPaulHashResult, err := rdb.FTSearch(
|
||||
ctx,
|
||||
|
||||
@@ -0,0 +1,24 @@
|
||||
package example_commands_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
|
||||
"github.com/redis/go-redis/v9"
|
||||
)
|
||||
|
||||
// waitForIndexedDocs blocks until the search index reports exactly n
|
||||
// documents with ingestion settled, or ~5s pass. The search examples HSET /
|
||||
// JSONSET and query back to back, but indexing is asynchronous — on a loaded
|
||||
// runner the query raced ingestion and returned zero results. Test-only:
|
||||
// example bodies call this inside REMOVE_START/REMOVE_END markers, so it
|
||||
// never appears in the documentation snippets.
|
||||
func waitForIndexedDocs(ctx context.Context, rdb *redis.Client, index string, n int) {
|
||||
for i := 0; i < 100; i++ {
|
||||
info, err := rdb.FTInfo(ctx, index).Result()
|
||||
if err == nil && info.NumDocs == n && info.Indexing == 0 {
|
||||
return
|
||||
}
|
||||
time.Sleep(50 * time.Millisecond)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,66 @@
|
||||
# Automatic pipelining
|
||||
|
||||
> **EXPERIMENTAL:** the autopipelining API is subject to change in a future release.
|
||||
|
||||
A tour of go-redis **autopipelining** plus a runnable throughput comparison.
|
||||
|
||||
Autopipelining batches commands from many goroutines into Redis pipelines
|
||||
automatically. It comes in two faces:
|
||||
|
||||
| Face | Call shape | Use when |
|
||||
|---|---|---|
|
||||
| `AutoPipeline()` (blocking) | each call blocks until executed — drop-in for a plain client | you want a speedup without changing code; ordering per goroutine |
|
||||
| `AsyncAutoPipeline()` (deferred) | calls return immediately; result accessors block | you can submit a window of commands and read results later — highest throughput |
|
||||
|
||||
## Run
|
||||
|
||||
```bash
|
||||
docker run --rm -p 6379:6379 redis
|
||||
go run .
|
||||
```
|
||||
|
||||
- `REDIS_ADDR` — point at a different server (default `localhost:6379`).
|
||||
- `REDIS_CLUSTER_ADDRS` — comma-separated cluster seed addresses; enables the
|
||||
cluster part of the tour (slot sharding is automatic, per-key order holds).
|
||||
|
||||
## What it shows
|
||||
|
||||
**Act 1 — usage tour**
|
||||
|
||||
1. Blocking face as a drop-in: goroutines `Set`+`Get` with plain-client call
|
||||
shape; the engine batches them under the hood on a handful of connections.
|
||||
2. Async face: submit a window of `Get`s, read the results afterwards — the
|
||||
throughput pattern.
|
||||
3. `Submit` + `AutoFuture` for raw `Cmder`s (async face only — `Submit` is
|
||||
rejected on the blocking face by design). `Wait`/`WaitContext` to collect.
|
||||
4. `Do` — the escape hatch. Runs on a **normal** connection outside the
|
||||
pipeline (plain `Client.Do` semantics); use it for raw commands the typed
|
||||
surface doesn't cover, never expect it to batch. (Typed blocking commands
|
||||
— `BLPop`, `XRead` with `Block`, ... — are diverted to a normal connection
|
||||
automatically.)
|
||||
5. Tuning notes: `Unordered` + `MaxConcurrentBatches: 2-4` for peak async
|
||||
throughput; leave `NumShards` at 0; the instance is cached per client
|
||||
(first call's config wins); optional dedicated pipeline pool via
|
||||
`PipelineReadBufferSize`/`PipelineWriteBufferSize`/`PipelinePoolSize`.
|
||||
|
||||
**Act 2 — throughput comparison** (sample, 500 goroutines, 3s, loopback —
|
||||
indicative, not a spec):
|
||||
|
||||
```
|
||||
approach ops/sec ordering vs normal
|
||||
1. normal blocking 57438 ordered 1.0x
|
||||
2. autopipeline ordered, blocking read 632742 ordered 11.0x
|
||||
3. autopipeline ordered, read later 2436067 ordered 42.4x
|
||||
4. autopipeline unordered, read later 2629600 UNORDERED 45.8x
|
||||
```
|
||||
|
||||
## Caveats worth knowing
|
||||
|
||||
- A command's context is not honored once queued; use a plain client for
|
||||
per-command deadlines (or `AutoFuture.WaitContext` to bound a wait).
|
||||
- A batch that fails on a network error is retried whole (up to `MaxRetries`),
|
||||
so non-idempotent commands may execute twice on a dropped connection.
|
||||
- On `ClusterClient`, ordering across nodes is per key.
|
||||
- Batched commands fire the client's *pipeline* hooks (one span per batch, not
|
||||
per command), so per-command instrumentation looks different from a plain
|
||||
client.
|
||||
@@ -0,0 +1,13 @@
|
||||
module github.com/redis/go-redis/example/autopipeline
|
||||
|
||||
go 1.24
|
||||
|
||||
replace github.com/redis/go-redis/v9 => ../..
|
||||
|
||||
require github.com/redis/go-redis/v9 v9.22.0-beta.1
|
||||
|
||||
require (
|
||||
github.com/cespare/xxhash/v2 v2.3.0 // indirect
|
||||
go.uber.org/atomic v1.11.0 // indirect
|
||||
golang.org/x/sys v0.30.0 // indirect
|
||||
)
|
||||
@@ -0,0 +1,20 @@
|
||||
github.com/bsm/ginkgo/v2 v2.12.0 h1:Ny8MWAHyOepLGlLKYmXG4IEkioBysk6GpaRTLC8zwWs=
|
||||
github.com/bsm/ginkgo/v2 v2.12.0/go.mod h1:SwYbGRRDovPVboqFv0tPTcG1sN61LM1Z4ARdbAV9g4c=
|
||||
github.com/bsm/gomega v1.27.10 h1:yeMWxP2pV2fG3FgAODIY8EiRE3dy0aeFYt4l7wh6yKA=
|
||||
github.com/bsm/gomega v1.27.10/go.mod h1:JyEr/xRbxbtgWNi8tIEVPUYZ5Dzef52k01W3YH0H+O0=
|
||||
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
|
||||
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
|
||||
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
|
||||
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/klauspost/cpuid/v2 v2.2.10 h1:tBs3QSyvjDyFTq3uoc/9xFpCuOsJQFNPiAhYdw2skhE=
|
||||
github.com/klauspost/cpuid/v2 v2.2.10/go.mod h1:hqwkgyIinND0mEev00jJYCxPNVRVXFQeu1XKlok6oO0=
|
||||
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
|
||||
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||
github.com/stretchr/testify v1.3.0 h1:TivCn/peBQ7UY8ooIcPgZFpTNSz0Q2U6UrFlUfqbe0Q=
|
||||
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
|
||||
github.com/zeebo/xxh3 v1.1.0 h1:s7DLGDK45Dyfg7++yxI0khrfwq9661w9EN78eP/UZVs=
|
||||
github.com/zeebo/xxh3 v1.1.0/go.mod h1:IisAie1LELR4xhVinxWS5+zf1lA4p0MW4T+w+W07F5s=
|
||||
go.uber.org/atomic v1.11.0 h1:ZvwS0R+56ePWxUNi+Atn9dWONBPp/AUETXlHW0DxSjE=
|
||||
go.uber.org/atomic v1.11.0/go.mod h1:LUxbIzbOniOlMKjJjyPfpl4v+PKK2cNJn91OQbhoJI0=
|
||||
golang.org/x/sys v0.30.0 h1:QjkSwP/36a20jFYWkSue1YwXzLmsV5Gfq7Eiy72C1uc=
|
||||
golang.org/x/sys v0.30.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
|
||||
@@ -0,0 +1,388 @@
|
||||
// Command autopipeline is a tour of go-redis automatic pipelining plus a
|
||||
// runnable throughput comparison.
|
||||
//
|
||||
// Act 1 — usage tour: the blocking face as a drop-in client, the async face
|
||||
// with a submission window, Submit/AutoFuture for raw commands, the Do escape
|
||||
// hatch, and (env-gated) cluster usage.
|
||||
//
|
||||
// Act 2 — throughput: the same workload (N goroutines issuing SET for a fixed
|
||||
// duration) run four ways:
|
||||
//
|
||||
// 1. normal blocking — plain client. Each command is a synchronous
|
||||
// round-trip. Simplest; ordered; throughput bounded by the pool size (~100k).
|
||||
//
|
||||
// 2. AutoPipeline() blocking — the blocking face: ap.Set(...) blocks until
|
||||
// executed, exactly like a normal client (drop-in). Per-goroutine ordered,
|
||||
// and the flusher coalesces all goroutines' commands into deep,
|
||||
// back-to-back pipelines (one ordered batch stream by default), so it
|
||||
// reaches ~1M+ ops/sec with no code change beyond the constructor.
|
||||
//
|
||||
// 3. AsyncAutoPipeline() ordered, read later — the deferred face: ap.Set(...)
|
||||
// returns immediately; submit a window of commands, then read their results.
|
||||
// Ordered (default) AND fastest: the window keeps each pipeline deep
|
||||
// (~2-3M). Best when you can defer reading results.
|
||||
//
|
||||
// 4. AsyncAutoPipeline() unordered, read later — same windowed style with a
|
||||
// parallel-batch config (MaxConcurrentBatches>1 + Unordered:true). Gives up
|
||||
// global ordering for peak throughput; use for independent / order-
|
||||
// insensitive bulk work.
|
||||
//
|
||||
// Decision guide:
|
||||
// - Need simplest code / low concurrency -> 1 (normal).
|
||||
// - Want a drop-in speedup, keep ordering -> 2 (AutoPipeline, blocking).
|
||||
// - High throughput, can read results later -> 3 (AsyncAutoPipeline, ordered).
|
||||
// - Max throughput, order does not matter -> 4 (AsyncAutoPipeline, unordered).
|
||||
//
|
||||
// Both faces return a CACHED, client-shared instance: the first call's config
|
||||
// wins, later configs are ignored, and Close stops the instance for every
|
||||
// caller. Close the autopipeliner (or the client) when done.
|
||||
//
|
||||
// Start a Redis first (e.g. `docker run --rm -p 6379:6379 redis`) then:
|
||||
//
|
||||
// go run .
|
||||
//
|
||||
// Set REDIS_ADDR to point at a different server; set REDIS_CLUSTER_ADDRS
|
||||
// (comma-separated) to also run the cluster part of the tour.
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"time"
|
||||
|
||||
"github.com/redis/go-redis/v9"
|
||||
)
|
||||
|
||||
const (
|
||||
duration = 3 * time.Second
|
||||
goroutines = 500
|
||||
window = 200 // commands submitted before reading results (read-later runs)
|
||||
)
|
||||
|
||||
func addr() string {
|
||||
if a := os.Getenv("REDIS_ADDR"); a != "" {
|
||||
return a
|
||||
}
|
||||
return "localhost:6379"
|
||||
}
|
||||
|
||||
func fatalf(format string, args ...any) {
|
||||
fmt.Printf(format+"\n", args...)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
type result struct {
|
||||
name string
|
||||
ordered string
|
||||
ops float64
|
||||
}
|
||||
|
||||
func main() {
|
||||
ctx := context.Background()
|
||||
|
||||
probe := redis.NewClient(&redis.Options{Addr: addr()})
|
||||
if err := probe.Ping(ctx).Err(); err != nil {
|
||||
fatalf("cannot reach redis at %s: %v", addr(), err)
|
||||
}
|
||||
probe.Close()
|
||||
|
||||
usageTour(ctx)
|
||||
throughputComparison(ctx)
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Act 1 — usage tour
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
func usageTour(ctx context.Context) {
|
||||
fmt.Println("== usage tour ==")
|
||||
|
||||
rdb := redis.NewClient(&redis.Options{
|
||||
Addr: addr(),
|
||||
// Optional: give pipelines their own connection pool with bigger
|
||||
// buffers. Enabled when either buffer size is set; batches then run on
|
||||
// dedicated connections instead of the main pool.
|
||||
PipelineReadBufferSize: 64 << 10,
|
||||
PipelineWriteBufferSize: 64 << 10,
|
||||
PipelinePoolSize: 10,
|
||||
})
|
||||
defer rdb.Close()
|
||||
rdb.FlushDB(ctx)
|
||||
|
||||
// --- a. Blocking face: a drop-in client -------------------------------
|
||||
// Each call blocks until its command executed, exactly like a plain
|
||||
// client; the engine batches concurrent callers under the hood. Don't
|
||||
// expect windowed-throughput wins here — each caller waits per command;
|
||||
// the win is batching + far fewer connections at high concurrency.
|
||||
ap, err := rdb.AutoPipeline()
|
||||
if err != nil {
|
||||
fatalf("AutoPipeline: %v", err)
|
||||
}
|
||||
defer ap.Close()
|
||||
|
||||
var wg sync.WaitGroup
|
||||
for i := 0; i < 4; i++ {
|
||||
wg.Add(1)
|
||||
go func(i int) {
|
||||
defer wg.Done()
|
||||
key := fmt.Sprintf("tour:blocking:%d", i)
|
||||
if err := ap.Set(ctx, key, i, 0).Err(); err != nil { // blocks until executed
|
||||
fatalf("set: %v", err)
|
||||
}
|
||||
val, err := ap.Get(ctx, key).Int() // reads exactly like a plain client
|
||||
if err != nil || val != i {
|
||||
fatalf("get %s = %d, %v", key, val, err)
|
||||
}
|
||||
}(i)
|
||||
}
|
||||
wg.Wait()
|
||||
fmt.Println(" blocking face: 4 goroutines set+get their keys (per-goroutine order held)")
|
||||
|
||||
// --- b. Async face: submit a window, read results later ---------------
|
||||
// Calls return immediately; the result accessors (Err/Val/Result) block
|
||||
// until the command's batch executed. The window is what keeps pipelines
|
||||
// deep — this is the throughput pattern.
|
||||
aap, err := rdb.AsyncAutoPipeline()
|
||||
if err != nil {
|
||||
fatalf("AsyncAutoPipeline: %v", err)
|
||||
}
|
||||
defer aap.Close()
|
||||
|
||||
gets := make([]*redis.StringCmd, 0, 4)
|
||||
for i := 0; i < 4; i++ {
|
||||
gets = append(gets, aap.Get(ctx, fmt.Sprintf("tour:blocking:%d", i))) // returns immediately
|
||||
}
|
||||
for i, g := range gets {
|
||||
if v, err := g.Int(); err != nil || v != i {
|
||||
fatalf("async get %d = %d, %v", i, v, err)
|
||||
}
|
||||
}
|
||||
fmt.Println(" async face: window of 4 GETs submitted, results read afterwards")
|
||||
|
||||
// --- c. Submit + AutoFuture: raw commands on the async face -----------
|
||||
// Submit is the explicit form for raw Cmders. It is REJECTED on the
|
||||
// blocking face by design (Submit doesn't wait, which would break the
|
||||
// blocking face's ordering guarantees) — use it on AsyncAutoPipeline.
|
||||
cmd := redis.NewStatusCmd(ctx, "set", "tour:future", "hello")
|
||||
fut := aap.Submit(ctx, cmd)
|
||||
if err := fut.Wait(); err != nil { // or fut.WaitContext(ctx) to bound the wait
|
||||
fatalf("submit: %v", err)
|
||||
}
|
||||
fmt.Printf(" submit/future: raw SET executed, reply %q\n", cmd.Val())
|
||||
|
||||
// --- d. Do: the escape hatch, NOT batched ------------------------------
|
||||
// Do runs on a normal connection outside the pipeline (plain Client.Do
|
||||
// semantics). It exists for arbitrary raw commands the typed surface does
|
||||
// not cover. Blocking commands need no escape hatch: the typed methods
|
||||
// (BLPop, XRead with Block, Wait, ...) are diverted to a normal connection
|
||||
// automatically and never ride a shared pipeline connection. Keep
|
||||
// connection-state commands away from Do just as with a plain client:
|
||||
// pick the DB via Options.DB, subscribe via the PubSub API.
|
||||
if err := aap.Do(ctx, "echo", "outside the pipeline").Err(); err != nil {
|
||||
fatalf("do: %v", err)
|
||||
}
|
||||
fmt.Println(" do: raw command ran on a normal connection (not batched)")
|
||||
|
||||
// --- e. Tuning notes (in code, so they stay honest) --------------------
|
||||
// For peak throughput on the async face give up global ordering:
|
||||
//
|
||||
// rdb.AsyncAutoPipelineWithOptions(&redis.AutoPipelineOptions{
|
||||
// MaxConcurrentBatches: 4, // small values suffice
|
||||
// Unordered: true,
|
||||
// })
|
||||
//
|
||||
// Extra permits add overlapping batches (each occupies a pipeline
|
||||
// connection for its round trip, so the pipeline pool size is the real
|
||||
// ceiling); they do not deepen batches — measure before raising. Leave
|
||||
// NumShards at 0 (one deep queue; cluster clients shard by slot
|
||||
// automatically). Remember the instance is cached per client: the FIRST
|
||||
// call's config wins.
|
||||
|
||||
// --- f. Cluster (env-gated) -------------------------------------------
|
||||
if addrs := os.Getenv("REDIS_CLUSTER_ADDRS"); addrs != "" {
|
||||
clusterTour(ctx, strings.Split(addrs, ","))
|
||||
} else {
|
||||
fmt.Println(" cluster: skipped (set REDIS_CLUSTER_ADDRS=host:port,... to run)")
|
||||
}
|
||||
fmt.Println()
|
||||
}
|
||||
|
||||
// clusterTour shows that autopipelining on a ClusterClient needs no extra
|
||||
// code: commands are routed to slot-affine shards automatically, so each
|
||||
// batch stays on one node and per-key ordering holds.
|
||||
func clusterTour(ctx context.Context, addrs []string) {
|
||||
cc := redis.NewClusterClient(&redis.ClusterOptions{Addrs: addrs})
|
||||
defer cc.Close()
|
||||
|
||||
ap, err := cc.AutoPipeline()
|
||||
if err != nil {
|
||||
fatalf("cluster AutoPipeline: %v", err)
|
||||
}
|
||||
defer ap.Close()
|
||||
|
||||
for i := 0; i < 8; i++ { // keys spread across slots/nodes
|
||||
if err := ap.Set(ctx, fmt.Sprintf("tour:cluster:%d", i), i, 0).Err(); err != nil {
|
||||
fatalf("cluster set: %v", err)
|
||||
}
|
||||
}
|
||||
fmt.Println(" cluster: 8 multi-slot SETs batched per node (slot sharding is automatic)")
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Act 2 — throughput comparison
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
func throughputComparison(ctx context.Context) {
|
||||
results := []result{
|
||||
{"1. normal blocking", "ordered", benchNormalBlocking(ctx)},
|
||||
{"2. autopipeline ordered, blocking read", "ordered", benchOrderedBlocking(ctx)},
|
||||
{"3. autopipeline ordered, read later", "ordered", benchOrderedReadLater(ctx)},
|
||||
{"4. autopipeline unordered, read later", "UNORDERED", benchUnorderedReadLater(ctx)},
|
||||
}
|
||||
|
||||
base := results[0].ops
|
||||
fmt.Printf("throughput (%d goroutines, %s):\n\n", goroutines, duration)
|
||||
fmt.Printf(" %-40s %12s %-9s %s\n", "approach", "ops/sec", "ordering", "vs normal")
|
||||
for _, r := range results {
|
||||
fmt.Printf(" %-40s %12.0f %-9s %.1fx\n", r.name, r.ops, r.ordered, r.ops/base)
|
||||
}
|
||||
fmt.Println(`
|
||||
Notes:
|
||||
- Read-later (3,4) is far faster than blocking-read (2) because it keeps the
|
||||
pipeline deep instead of one command in flight per goroutine.
|
||||
- Unordered (4) overlaps a few batches and is typically ~10-20% faster than
|
||||
ordered read-later (3); the gap widens with more concurrent callers. Both are
|
||||
bounded by the Go client's per-command overhead (object alloc + signalling),
|
||||
so they land somewhat below what the server alone can do, and the difference
|
||||
between 3 and 4 can look small at moderate load.
|
||||
- Prefer 3 (ordered, read later) unless you've measured that 4's extra
|
||||
throughput matters for your workload and you don't need ordering.`)
|
||||
}
|
||||
|
||||
// 1. Plain client: each Set is a synchronous round-trip. Ordered. PoolSize is
|
||||
// raised to goroutine-count territory so the baseline isn't artificially
|
||||
// pool-starved — that's the plain client's cost: connections scale with
|
||||
// concurrency (the autopipeline cases below use the default pool and still
|
||||
// hold only a handful of connections).
|
||||
func benchNormalBlocking(ctx context.Context) float64 {
|
||||
rdb := redis.NewClient(&redis.Options{Addr: addr(), PoolSize: goroutines})
|
||||
defer rdb.Close()
|
||||
rdb.FlushDB(ctx)
|
||||
|
||||
return run(func(g int, doOp func()) {
|
||||
key := fmt.Sprintf("normal:%d", g)
|
||||
rdb.Set(ctx, key, 1, 0)
|
||||
doOp()
|
||||
})
|
||||
}
|
||||
|
||||
// 2. Blocking autopipeline: ap.Set(...) blocks until executed, exactly like a
|
||||
// normal client. Drop-in; per-goroutine ordered. The flusher still batches this
|
||||
// caller's command with the other goroutines' commands, so throughput is far
|
||||
// above a plain client.
|
||||
func benchOrderedBlocking(ctx context.Context) float64 {
|
||||
rdb := redis.NewClient(&redis.Options{Addr: addr()})
|
||||
defer rdb.Close()
|
||||
rdb.FlushDB(ctx)
|
||||
ap, err := rdb.AutoPipeline() // blocking face (default: single ordered batch stream)
|
||||
if err != nil {
|
||||
fatalf("AutoPipeline: %v", err)
|
||||
}
|
||||
defer ap.Close()
|
||||
|
||||
return run(func(g int, doOp func()) {
|
||||
key := fmt.Sprintf("ob:%d", g)
|
||||
_ = ap.Set(ctx, key, 1, 0).Err() // the call itself blocks until executed
|
||||
doOp()
|
||||
})
|
||||
}
|
||||
|
||||
// 3. Async autopipeline, ordered, read later: submit a window of commands without
|
||||
// reading (non-blocking), then drain results. Ordered (default) and fast — the
|
||||
// window keeps the pipeline deep.
|
||||
func benchOrderedReadLater(ctx context.Context) float64 {
|
||||
rdb := redis.NewClient(&redis.Options{Addr: addr()})
|
||||
defer rdb.Close()
|
||||
rdb.FlushDB(ctx)
|
||||
// Default async config is ordered (MaxConcurrentBatches=1).
|
||||
ap, err := rdb.AsyncAutoPipeline()
|
||||
if err != nil {
|
||||
fatalf("AsyncAutoPipeline: %v", err)
|
||||
}
|
||||
return benchReadLater(ctx, ap)
|
||||
}
|
||||
|
||||
// 4. Async autopipeline, unordered, read later: a few overlapping batches for
|
||||
// peak throughput. MaxConcurrentBatches>1 gives up global ordering, so
|
||||
// Unordered must be true. A handful of concurrent batches is plenty — batch
|
||||
// N+1 accumulates and executes while batch N's replies are in flight; higher
|
||||
// values only add more overlapping batches (bounded by the pipeline pool
|
||||
// size), they don't deepen them.
|
||||
func benchUnorderedReadLater(ctx context.Context) float64 {
|
||||
rdb := redis.NewClient(&redis.Options{Addr: addr()})
|
||||
defer rdb.Close()
|
||||
rdb.FlushDB(ctx)
|
||||
ap, err := rdb.AsyncAutoPipelineWithOptions(&redis.AutoPipelineOptions{
|
||||
MaxBatchSize: 300,
|
||||
MaxConcurrentBatches: 4,
|
||||
Unordered: true, // required for MaxConcurrentBatches > 1
|
||||
})
|
||||
if err != nil {
|
||||
fatalf("AsyncAutoPipeline: %v", err)
|
||||
}
|
||||
return benchReadLater(ctx, ap)
|
||||
}
|
||||
|
||||
// benchReadLater submits a window of SET commands (non-blocking, returning the
|
||||
// usual *StatusCmd) on the given async autopipeliner and then reads their
|
||||
// results, which keeps the pipeline deep.
|
||||
func benchReadLater(ctx context.Context, ap *redis.AutoPipeliner) float64 {
|
||||
defer ap.Close()
|
||||
|
||||
var count int64
|
||||
deadline := time.Now().Add(duration)
|
||||
var wg sync.WaitGroup
|
||||
wg.Add(goroutines)
|
||||
for g := 0; g < goroutines; g++ {
|
||||
go func(id int) {
|
||||
defer wg.Done()
|
||||
key := fmt.Sprintf("rl:%d", id)
|
||||
cmds := make([]*redis.StatusCmd, 0, window)
|
||||
for time.Now().Before(deadline) {
|
||||
cmds = cmds[:0]
|
||||
for j := 0; j < window; j++ {
|
||||
cmds = append(cmds, ap.Set(ctx, key, j, 0)) // queued, non-blocking
|
||||
}
|
||||
for _, c := range cmds {
|
||||
_ = c.Err() // blocks until executed
|
||||
}
|
||||
atomic.AddInt64(&count, window)
|
||||
}
|
||||
}(g)
|
||||
}
|
||||
wg.Wait()
|
||||
return float64(count) / duration.Seconds()
|
||||
}
|
||||
|
||||
// run executes a per-command workload (op) on `goroutines` goroutines for the
|
||||
// fixed duration and returns ops/sec. op calls doOp() once per command executed.
|
||||
func run(op func(g int, doOp func())) float64 {
|
||||
var count int64
|
||||
deadline := time.Now().Add(duration)
|
||||
var wg sync.WaitGroup
|
||||
wg.Add(goroutines)
|
||||
for g := 0; g < goroutines; g++ {
|
||||
go func(id int) {
|
||||
defer wg.Done()
|
||||
for time.Now().Before(deadline) {
|
||||
op(id, func() { atomic.AddInt64(&count, 1) })
|
||||
}
|
||||
}(g)
|
||||
}
|
||||
wg.Wait()
|
||||
return float64(count) / duration.Seconds()
|
||||
}
|
||||
@@ -9,5 +9,5 @@ require github.com/redis/go-redis/v9 v9.21.0
|
||||
require (
|
||||
github.com/cespare/xxhash/v2 v2.3.0 // indirect
|
||||
go.uber.org/atomic v1.11.0 // indirect
|
||||
gopkg.in/yaml.v3 v3.0.1 // indirect
|
||||
golang.org/x/sys v0.30.0 // indirect
|
||||
)
|
||||
@@ -10,14 +10,11 @@ github.com/klauspost/cpuid/v2 v2.2.10 h1:tBs3QSyvjDyFTq3uoc/9xFpCuOsJQFNPiAhYdw2
|
||||
github.com/klauspost/cpuid/v2 v2.2.10/go.mod h1:hqwkgyIinND0mEev00jJYCxPNVRVXFQeu1XKlok6oO0=
|
||||
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
|
||||
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||
github.com/stretchr/testify v1.7.0 h1:nwc3DEeHmmLAfoZucVR881uASk0Mfjw8xYJ99tb5CcY=
|
||||
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
|
||||
github.com/stretchr/testify v1.3.0 h1:TivCn/peBQ7UY8ooIcPgZFpTNSz0Q2U6UrFlUfqbe0Q=
|
||||
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
|
||||
github.com/zeebo/xxh3 v1.1.0 h1:s7DLGDK45Dyfg7++yxI0khrfwq9661w9EN78eP/UZVs=
|
||||
github.com/zeebo/xxh3 v1.1.0/go.mod h1:IisAie1LELR4xhVinxWS5+zf1lA4p0MW4T+w+W07F5s=
|
||||
go.uber.org/atomic v1.11.0 h1:ZvwS0R+56ePWxUNi+Atn9dWONBPp/AUETXlHW0DxSjE=
|
||||
go.uber.org/atomic v1.11.0/go.mod h1:LUxbIzbOniOlMKjJjyPfpl4v+PKK2cNJn91OQbhoJI0=
|
||||
golang.org/x/sys v0.30.0 h1:QjkSwP/36a20jFYWkSue1YwXzLmsV5Gfq7Eiy72C1uc=
|
||||
golang.org/x/sys v0.30.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
||||
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
@@ -9,4 +9,5 @@ require github.com/redis/go-redis/v9 v9.22.0-beta.1
|
||||
require (
|
||||
github.com/cespare/xxhash/v2 v2.3.0 // indirect
|
||||
go.uber.org/atomic v1.11.0 // indirect
|
||||
golang.org/x/sys v0.30.0 // indirect
|
||||
)
|
||||
@@ -13,4 +13,5 @@ require (
|
||||
github.com/cespare/xxhash/v2 v2.3.0 // indirect
|
||||
go.uber.org/atomic v1.11.0 // indirect
|
||||
go.uber.org/multierr v1.9.0 // indirect
|
||||
golang.org/x/sys v0.30.0 // indirect
|
||||
)
|
||||
@@ -9,4 +9,5 @@ require github.com/redis/go-redis/v9 v9.22.0-beta.1
|
||||
require (
|
||||
github.com/cespare/xxhash/v2 v2.3.0 // indirect
|
||||
go.uber.org/atomic v1.11.0 // indirect
|
||||
golang.org/x/sys v0.30.0 // indirect
|
||||
)
|
||||
@@ -9,4 +9,5 @@ require github.com/redis/go-redis/v9 v9.22.0-beta.1
|
||||
require (
|
||||
github.com/cespare/xxhash/v2 v2.3.0 // indirect
|
||||
go.uber.org/atomic v1.11.0 // indirect
|
||||
golang.org/x/sys v0.30.0 // indirect
|
||||
)
|
||||
@@ -9,4 +9,5 @@ require github.com/redis/go-redis/v9 v9.22.0-beta.1
|
||||
require (
|
||||
github.com/cespare/xxhash/v2 v2.3.0 // indirect
|
||||
go.uber.org/atomic v1.11.0 // indirect
|
||||
golang.org/x/sys v0.30.0 // indirect
|
||||
)
|
||||
@@ -9,4 +9,5 @@ require github.com/redis/go-redis/v9 v9.22.0-beta.1
|
||||
require (
|
||||
github.com/cespare/xxhash/v2 v2.3.0 // indirect
|
||||
go.uber.org/atomic v1.11.0 // indirect
|
||||
golang.org/x/sys v0.30.0 // indirect
|
||||
)
|
||||
@@ -12,4 +12,5 @@ require (
|
||||
require (
|
||||
github.com/cespare/xxhash/v2 v2.3.0 // indirect
|
||||
go.uber.org/atomic v1.11.0 // indirect
|
||||
golang.org/x/sys v0.30.0 // indirect
|
||||
)
|
||||
@@ -9,4 +9,5 @@ require github.com/redis/go-redis/v9 v9.22.0-beta.1
|
||||
require (
|
||||
github.com/cespare/xxhash/v2 v2.3.0 // indirect
|
||||
go.uber.org/atomic v1.11.0 // indirect
|
||||
golang.org/x/sys v0.30.0 // indirect
|
||||
)
|
||||
@@ -9,4 +9,5 @@ require github.com/redis/go-redis/v9 v9.22.0-beta.1
|
||||
require (
|
||||
github.com/cespare/xxhash/v2 v2.3.0 // indirect
|
||||
go.uber.org/atomic v1.11.0 // indirect
|
||||
golang.org/x/sys v0.30.0 // indirect
|
||||
)
|
||||
@@ -9,4 +9,5 @@ require github.com/redis/go-redis/v9 v9.22.0-beta.1
|
||||
require (
|
||||
github.com/cespare/xxhash/v2 v2.3.0 // indirect
|
||||
go.uber.org/atomic v1.11.0 // indirect
|
||||
golang.org/x/sys v0.30.0 // indirect
|
||||
)
|
||||
@@ -12,4 +12,5 @@ require (
|
||||
require (
|
||||
github.com/cespare/xxhash/v2 v2.3.0 // indirect
|
||||
go.uber.org/atomic v1.11.0 // indirect
|
||||
golang.org/x/sys v0.30.0 // indirect
|
||||
)
|
||||
@@ -9,4 +9,5 @@ require github.com/redis/go-redis/v9 v9.22.0-beta.1
|
||||
require (
|
||||
github.com/cespare/xxhash/v2 v2.3.0 // indirect
|
||||
go.uber.org/atomic v1.11.0 // indirect
|
||||
golang.org/x/sys v0.30.0 // indirect
|
||||
)
|
||||
@@ -9,4 +9,5 @@ require github.com/redis/go-redis/v9 v9.22.0-beta.1
|
||||
require (
|
||||
github.com/cespare/xxhash/v2 v2.3.0 // indirect
|
||||
go.uber.org/atomic v1.11.0 // indirect
|
||||
golang.org/x/sys v0.30.0 // indirect
|
||||
)
|
||||
@@ -9,4 +9,5 @@ require github.com/redis/go-redis/v9 v9.22.0-beta.1
|
||||
require (
|
||||
github.com/cespare/xxhash/v2 v2.3.0 // indirect
|
||||
go.uber.org/atomic v1.11.0 // indirect
|
||||
golang.org/x/sys v0.30.0 // indirect
|
||||
)
|
||||
@@ -9,4 +9,5 @@ require github.com/redis/go-redis/v9 v9.22.0-beta.1
|
||||
require (
|
||||
github.com/cespare/xxhash/v2 v2.3.0 // indirect
|
||||
go.uber.org/atomic v1.11.0 // indirect
|
||||
golang.org/x/sys v0.30.0 // indirect
|
||||
)
|
||||
@@ -9,4 +9,5 @@ require github.com/redis/go-redis/v9 v9.22.0-beta.1
|
||||
require (
|
||||
github.com/cespare/xxhash/v2 v2.3.0 // indirect
|
||||
go.uber.org/atomic v1.11.0 // indirect
|
||||
golang.org/x/sys v0.30.0 // indirect
|
||||
)
|
||||
@@ -9,4 +9,5 @@ require github.com/redis/go-redis/v9 v9.22.0-beta.1
|
||||
require (
|
||||
github.com/cespare/xxhash/v2 v2.3.0 // indirect
|
||||
go.uber.org/atomic v1.11.0 // indirect
|
||||
golang.org/x/sys v0.30.0 // indirect
|
||||
)
|
||||
@@ -15,13 +15,16 @@ func runPackingExamples(ctx context.Context, rdb *redis.Client) {
|
||||
// --- Example 4: Packing multiple values into a single buffer ---
|
||||
fmt.Println("\n=== Example 4: Packing multiple values into one buffer ===")
|
||||
|
||||
// We have 4 values, each gets a 7-byte slot. Total buffer = 4 * 7 = 28 bytes.
|
||||
// The last value ("s!") is only 2 bytes — this demonstrates that GetToBuffer
|
||||
// works correctly when the value is smaller than the allocated segment.
|
||||
const slotSize = 7
|
||||
// We have 4 values of up to 7 bytes; each gets a 9-byte slot (value + 2
|
||||
// spare bytes). The 2 spare bytes opt each read into GetToBuffer's fast
|
||||
// path (payload + protocol CRLF in one socket read, the spare bytes used
|
||||
// as scratch INSIDE the slot) — nothing is ever written past the segment
|
||||
// you pass. The last value ("s!") is only 2 bytes — this demonstrates that
|
||||
// GetToBuffer works correctly when the value is smaller than the slot.
|
||||
const slotSize = 7 + 2
|
||||
values := []string{"hello!!", "world!!", "go-redi", "s!"}
|
||||
keys := make([]string, len(values))
|
||||
bigBuf := make([]byte, len(values)*slotSize) // 28 bytes
|
||||
bigBuf := make([]byte, len(values)*slotSize) // 36 bytes
|
||||
|
||||
// Write all keys.
|
||||
for i, v := range values {
|
||||
|
||||
@@ -16,6 +16,7 @@ require (
|
||||
github.com/cespare/xxhash/v2 v2.3.0 // indirect
|
||||
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
|
||||
go.uber.org/atomic v1.11.0 // indirect
|
||||
golang.org/x/sys v0.30.0 // indirect
|
||||
)
|
||||
|
||||
retract (
|
||||
|
||||
@@ -13,6 +13,7 @@ require (
|
||||
require (
|
||||
github.com/cespare/xxhash/v2 v2.3.0 // indirect
|
||||
go.uber.org/atomic v1.11.0 // indirect
|
||||
golang.org/x/sys v0.30.0 // indirect
|
||||
)
|
||||
|
||||
retract (
|
||||
|
||||
@@ -8,12 +8,10 @@ require (
|
||||
github.com/cespare/xxhash/v2 v2.3.0
|
||||
github.com/zeebo/xxh3 v1.1.0
|
||||
go.uber.org/atomic v1.11.0
|
||||
golang.org/x/sys v0.30.0
|
||||
)
|
||||
|
||||
require (
|
||||
github.com/klauspost/cpuid/v2 v2.2.10 // indirect
|
||||
golang.org/x/sys v0.30.0 // indirect
|
||||
)
|
||||
require github.com/klauspost/cpuid/v2 v2.2.10 // indirect
|
||||
|
||||
retract (
|
||||
v9.15.1 // This version is used to retract v9.15.0
|
||||
|
||||
+9
-4
@@ -349,6 +349,10 @@ func (c *baseClient) himportAfterCmd(cn *pool.Conn, hc himportCmder) {
|
||||
// error), invalidates stale prepared flags for SETs that found their
|
||||
// registered fieldset missing server-side, and applies registry updates for
|
||||
// user-issued HIMPORT commands that succeeded in the batch.
|
||||
// rawErr throughout: this runs on the execution path, before an async
|
||||
// autopipeline batch completes (its ready channel closes only after the
|
||||
// pipeline hook chain returns) — Err() on a user command would await and
|
||||
// self-deadlock the dispatcher.
|
||||
func (c *baseClient) himportAfterBatch(cn *pool.Conn, injected []Cmder, cmds []Cmder) {
|
||||
var failed map[string]error
|
||||
var refreshed map[string]struct{}
|
||||
@@ -368,7 +372,7 @@ func (c *baseClient) himportAfterBatch(cn *pool.Conn, injected []Cmder, cmds []C
|
||||
continue
|
||||
}
|
||||
if set, ok := hc.(*HImportSetCmd); ok {
|
||||
if rootCause, ok := failed[set.fieldsetName]; ok && himportNoSuchFieldset(set.Err()) {
|
||||
if rootCause, ok := failed[set.fieldsetName]; ok && himportNoSuchFieldset(set.rawErr()) {
|
||||
set.SetErr(rootCause)
|
||||
continue
|
||||
}
|
||||
@@ -378,7 +382,7 @@ func (c *baseClient) himportAfterBatch(cn *pool.Conn, injected []Cmder, cmds []C
|
||||
// version once so the SET's re-issue, the cluster re-queue on
|
||||
// whichever connection it lands, or the caller's transaction
|
||||
// retry replays the PREPARE.
|
||||
if himportNoSuchFieldset(set.Err()) {
|
||||
if himportNoSuchFieldset(set.rawErr()) {
|
||||
if _, done := refreshed[set.fieldsetName]; !done {
|
||||
if refreshed == nil {
|
||||
refreshed = make(map[string]struct{})
|
||||
@@ -391,7 +395,7 @@ func (c *baseClient) himportAfterBatch(cn *pool.Conn, injected []Cmder, cmds []C
|
||||
}
|
||||
continue
|
||||
}
|
||||
if hc.Err() == nil {
|
||||
if hc.rawErr() == nil {
|
||||
c.himportAfterCmd(cn, hc)
|
||||
}
|
||||
}
|
||||
@@ -416,7 +420,8 @@ func (c *baseClient) himportRetryFailedSets(ctx context.Context, cn *pool.Conn,
|
||||
}
|
||||
var retry []Cmder
|
||||
for _, cmd := range cmds {
|
||||
if set, ok := cmd.(*HImportSetCmd); ok && himportNoSuchFieldset(set.Err()) {
|
||||
// rawErr: execution path, same self-deadlock rule as himportAfterBatch.
|
||||
if set, ok := cmd.(*HImportSetCmd); ok && himportNoSuchFieldset(set.rawErr()) {
|
||||
if _, registered := c.himport.lookup(set.fieldsetName); registered {
|
||||
retry = append(retry, set)
|
||||
}
|
||||
|
||||
+3
-1
@@ -41,7 +41,9 @@ type himportForEach func(ctx context.Context, fn func(ctx context.Context, clien
|
||||
// Bounded by the cluster pipeline's attempt budget.
|
||||
func (c *ClusterClient) himportRequeueFailedSets(ctx context.Context, cmds []Cmder, failedCmds *cmdsMap) {
|
||||
for _, cmd := range cmds {
|
||||
if set, ok := cmd.(*HImportSetCmd); ok && himportNoSuchFieldset(set.Err()) {
|
||||
// rawErr: runs on the per-node execution goroutine, same
|
||||
// self-deadlock rule as himportAfterBatch.
|
||||
if set, ok := cmd.(*HImportSetCmd); ok && himportNoSuchFieldset(set.rawErr()) {
|
||||
if _, registered := c.himport.lookup(set.fieldsetName); registered {
|
||||
_ = c.mapCmdsByNode(ctx, failedCmds, []Cmder{set})
|
||||
}
|
||||
|
||||
@@ -155,6 +155,15 @@ func getRecorder() Recorder {
|
||||
return r
|
||||
}
|
||||
|
||||
// Enabled reports whether a real recorder is installed. Callers use it to
|
||||
// skip metric work whose INPUTS are expensive to obtain — e.g. reading a
|
||||
// command's result, which on the async autopipeline face blocks until the
|
||||
// command executes.
|
||||
func Enabled() bool {
|
||||
_, noop := getRecorder().(noopRecorder)
|
||||
return !noop
|
||||
}
|
||||
|
||||
// SetGlobalRecorder sets the global recorder (called by Init() in extra/redisotel-native)
|
||||
func SetGlobalRecorder(r Recorder) {
|
||||
recorderMu.Lock()
|
||||
@@ -266,10 +275,15 @@ func (noopRecorder) RecordStreamLag(context.Context, time.Duration, *pool.Conn,
|
||||
func (noopRecorder) RecordConnectionCount(context.Context, int, *pool.Conn, string, bool) {}
|
||||
func (noopRecorder) RecordPendingRequests(context.Context, int, *pool.Conn, string) {}
|
||||
|
||||
// RegisterPools registers connection pools with the global recorder.
|
||||
func RegisterPools(connPool pool.Pooler, pubSubPool PubSubPooler, addr string) {
|
||||
// Check if the global recorder implements PoolRegistrar
|
||||
if registrar, ok := globalRecorder.(PoolRegistrar); ok {
|
||||
// RegisterPools registers connection pools with the global recorder. pipelinePool
|
||||
// is the optional dedicated pipeline connection pool (nil when not configured);
|
||||
// it is registered as a regular pool under a "_pipeline" name suffix.
|
||||
func RegisterPools(connPool pool.Pooler, pubSubPool PubSubPooler, pipelinePool pool.Pooler, addr string) {
|
||||
// Check if the global recorder implements PoolRegistrar. Read it through
|
||||
// getRecorder: SetGlobalRecorder writes globalRecorder under recorderMu, and
|
||||
// clients are created (and closed) concurrently with telemetry being
|
||||
// installed, so an unlocked read here is a data race -race reports.
|
||||
if registrar, ok := getRecorder().(PoolRegistrar); ok {
|
||||
// Generate a unique ID for this client's pools
|
||||
uniqueID := generateUniqueID()
|
||||
|
||||
@@ -281,18 +295,27 @@ func RegisterPools(connPool pool.Pooler, pubSubPool PubSubPooler, addr string) {
|
||||
poolName := addr + "_" + uniqueID + "_pubsub"
|
||||
registrar.RegisterPubSubPool(poolName, pubSubPool)
|
||||
}
|
||||
if pipelinePool != nil {
|
||||
poolName := addr + "_" + uniqueID + "_pipeline"
|
||||
registrar.RegisterPool(poolName, pipelinePool)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// UnregisterPools removes connection pools from the global recorder
|
||||
func UnregisterPools(connPool pool.Pooler, pubSubPool PubSubPooler) {
|
||||
// Check if the global recorder implements PoolRegistrar
|
||||
if registrar, ok := globalRecorder.(PoolRegistrar); ok {
|
||||
// UnregisterPools removes connection pools from the global recorder. pipelinePool
|
||||
// is the optional dedicated pipeline connection pool (nil when not configured).
|
||||
func UnregisterPools(connPool pool.Pooler, pubSubPool PubSubPooler, pipelinePool pool.Pooler) {
|
||||
// Check if the global recorder implements PoolRegistrar (see RegisterPools
|
||||
// for why this goes through getRecorder rather than reading directly).
|
||||
if registrar, ok := getRecorder().(PoolRegistrar); ok {
|
||||
if connPool != nil {
|
||||
registrar.UnregisterPool(connPool)
|
||||
}
|
||||
if pubSubPool != nil {
|
||||
registrar.UnregisterPubSubPool(pubSubPool)
|
||||
}
|
||||
if pipelinePool != nil {
|
||||
registrar.UnregisterPool(pipelinePool)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,57 @@
|
||||
package otel
|
||||
|
||||
import (
|
||||
"sync"
|
||||
"testing"
|
||||
|
||||
"github.com/redis/go-redis/v9/internal/pool"
|
||||
)
|
||||
|
||||
type raceRecorder struct{ noopRecorder }
|
||||
|
||||
func (raceRecorder) RegisterPool(string, pool.Pooler) {}
|
||||
func (raceRecorder) UnregisterPool(pool.Pooler) {}
|
||||
func (raceRecorder) RegisterPubSubPool(string, PubSubPooler) {}
|
||||
func (raceRecorder) UnregisterPubSubPool(PubSubPooler) {}
|
||||
|
||||
// TestRegisterPoolsRaceWithSetGlobalRecorder pins that the pool registration
|
||||
// helpers read the global recorder under its lock. Installing telemetry while
|
||||
// another goroutine creates or closes clients is ordinary usage, and reading
|
||||
// globalRecorder without recorderMu while SetGlobalRecorder writes it under the
|
||||
// lock is a data race that -race reports (Copilot review on #3942).
|
||||
func TestRegisterPoolsRaceWithSetGlobalRecorder(t *testing.T) {
|
||||
t.Cleanup(func() { SetGlobalRecorder(noopRecorder{}) })
|
||||
|
||||
stop := make(chan struct{})
|
||||
writerDone := make(chan struct{})
|
||||
go func() {
|
||||
defer close(writerDone)
|
||||
for i := 0; ; i++ {
|
||||
select {
|
||||
case <-stop:
|
||||
return
|
||||
default:
|
||||
}
|
||||
if i%2 == 0 {
|
||||
SetGlobalRecorder(raceRecorder{})
|
||||
} else {
|
||||
SetGlobalRecorder(noopRecorder{})
|
||||
}
|
||||
}
|
||||
}()
|
||||
|
||||
var workers sync.WaitGroup
|
||||
for i := 0; i < 4; i++ {
|
||||
workers.Add(1)
|
||||
go func() {
|
||||
defer workers.Done()
|
||||
for j := 0; j < 2000; j++ {
|
||||
RegisterPools(nil, nil, nil, "127.0.0.1:6379")
|
||||
UnregisterPools(nil, nil, nil)
|
||||
}
|
||||
}()
|
||||
}
|
||||
workers.Wait()
|
||||
close(stop)
|
||||
<-writerDone
|
||||
}
|
||||
@@ -326,6 +326,11 @@ type Stats struct {
|
||||
PendingRequests uint32 // number of pending requests waiting for a connection
|
||||
|
||||
PubSubStats PubSubStats
|
||||
|
||||
// PipelineStats holds the stats of the separate pipeline connection pool
|
||||
// when one is configured (PipelineReadBufferSize/PipelineWriteBufferSize).
|
||||
// nil when pipelines share the main pool.
|
||||
PipelineStats *Stats
|
||||
}
|
||||
|
||||
type ConnRetirer interface {
|
||||
@@ -1666,6 +1671,13 @@ func (p *ConnPool) IdleLen() int {
|
||||
return int(n)
|
||||
}
|
||||
|
||||
// Name returns the pool's configured name, which is stamped on every
|
||||
// connection it creates (Conn.PoolName). Callers holding a Pooler can type
|
||||
// assert to interface{ Name() string } to find which pool owns a connection —
|
||||
// used by maintnotifications to route a handoff to the hook that owns the
|
||||
// conn's pool rather than always the primary one.
|
||||
func (p *ConnPool) Name() string { return p.cfg.Name }
|
||||
|
||||
// Size returns the maximum pool size (capacity).
|
||||
//
|
||||
// This is used by the streaming credentials manager to size the re-auth worker pool,
|
||||
|
||||
@@ -621,10 +621,33 @@ func (r *Reader) ReadStringInto(buf []byte) (int, error) {
|
||||
// bufio.Reader.Read first drains its internal buffer, then for
|
||||
// remaining data larger than its buffer size reads directly from the
|
||||
// underlying reader (socket) — effectively zero-copy.
|
||||
//
|
||||
// Fast path: when the caller VISIBLY hands over room for the trailing
|
||||
// CRLF too (len(buf) >= n+2), read the payload and the CRLF in a
|
||||
// single io.ReadFull. For large values this is one direct socket read
|
||||
// instead of a big read followed by a tiny separate Discard(2) read,
|
||||
// which is what makes GetToBuffer beat a regular Get (no payload
|
||||
// allocation and the same number of reads). The 2 trailing bytes land
|
||||
// past the returned length and are ignored.
|
||||
//
|
||||
// The gate is on len, NOT cap: a sub-slice of a larger buffer (e.g.
|
||||
// packed segments big[i*slot:(i+1)*slot]) exposes trailing capacity
|
||||
// that belongs to the caller's NEXT segment — writing the CRLF there
|
||||
// would silently corrupt caller-owned memory outside the slice they
|
||||
// passed. Callers who want the fast path pass len == payload+2 (the
|
||||
// returned length is still the payload length).
|
||||
if len(buf) >= n+2 {
|
||||
full := buf[:n+2]
|
||||
if _, err := io.ReadFull(r.rd, full); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return n, nil
|
||||
}
|
||||
// Slow path: buffer is exactly large enough for the payload only, so
|
||||
// read the payload into it and discard the CRLF separately.
|
||||
if _, err := io.ReadFull(r.rd, buf[:n]); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
// Discard trailing \r\n.
|
||||
if _, err := r.rd.Discard(2); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
|
||||
@@ -233,6 +233,78 @@ func TestReader_ReadStringInto_Int(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// When the destination buffer VISIBLY has room for the trailing CRLF
|
||||
// (len(buf) >= n+2), ReadStringInto takes the fast path that reads
|
||||
// payload+CRLF in a single io.ReadFull. Verify it returns the right payload
|
||||
// AND leaves the stream aligned so the following reply parses correctly.
|
||||
func TestReader_ReadStringInto_FastPathStreamAligned(t *testing.T) {
|
||||
// Two bulk strings back to back.
|
||||
r := proto.NewReader(bytes.NewReader([]byte("$5\r\nhello\r\n$5\r\nworld\r\n")))
|
||||
|
||||
buf := make([]byte, 7) // len 7 >= 5+2 -> fast path
|
||||
n, err := r.ReadStringInto(buf)
|
||||
if err != nil {
|
||||
t.Fatalf("ReadStringInto #1: %v", err)
|
||||
}
|
||||
if n != 5 || string(buf[:n]) != "hello" {
|
||||
t.Fatalf("#1 got %q (n=%d), want hello", buf[:n], n)
|
||||
}
|
||||
// Second reply must be intact (stream not corrupted by the CRLF handling).
|
||||
s, err := r.ReadString()
|
||||
if err != nil {
|
||||
t.Fatalf("ReadString #2: %v", err)
|
||||
}
|
||||
if s != "world" {
|
||||
t.Fatalf("#2 got %q, want world", s)
|
||||
}
|
||||
}
|
||||
|
||||
// Regression: a sub-slice of a larger buffer exposes trailing CAPACITY that
|
||||
// belongs to the caller's next packed segment. The fast path must gate on
|
||||
// len, not cap — the old cap gate wrote the trailing CRLF past len(buf) into
|
||||
// the neighboring segment, silently corrupting caller-owned memory.
|
||||
func TestReader_ReadStringInto_NeverWritesPastLen(t *testing.T) {
|
||||
r := proto.NewReader(bytes.NewReader([]byte("$5\r\nhello\r\n")))
|
||||
|
||||
big := make([]byte, 10)
|
||||
for i := range big {
|
||||
big[i] = 'Z'
|
||||
}
|
||||
segment := big[:5] // len 5, cap 10: spare capacity belongs to big[5:]
|
||||
n, err := r.ReadStringInto(segment)
|
||||
if err != nil {
|
||||
t.Fatalf("ReadStringInto: %v", err)
|
||||
}
|
||||
if string(segment[:n]) != "hello" {
|
||||
t.Fatalf("got %q, want hello", segment[:n])
|
||||
}
|
||||
if string(big[5:]) != "ZZZZZ" {
|
||||
t.Fatalf("bytes past len(buf) were modified: big[5:] = %q, want ZZZZZ", big[5:])
|
||||
}
|
||||
}
|
||||
|
||||
// When the buffer is exactly the payload size (cap == n), ReadStringInto uses
|
||||
// the slow path (read payload, discard CRLF). It must still align the stream.
|
||||
func TestReader_ReadStringInto_ExactBufStreamAligned(t *testing.T) {
|
||||
r := proto.NewReader(bytes.NewReader([]byte("$5\r\nhello\r\n$5\r\nworld\r\n")))
|
||||
|
||||
buf := make([]byte, 5) // cap == len == 5 -> slow path
|
||||
n, err := r.ReadStringInto(buf)
|
||||
if err != nil {
|
||||
t.Fatalf("ReadStringInto #1: %v", err)
|
||||
}
|
||||
if string(buf[:n]) != "hello" {
|
||||
t.Fatalf("#1 got %q, want hello", buf[:n])
|
||||
}
|
||||
s, err := r.ReadString()
|
||||
if err != nil {
|
||||
t.Fatalf("ReadString #2: %v", err)
|
||||
}
|
||||
if s != "world" {
|
||||
t.Fatalf("#2 got %q, want world", s)
|
||||
}
|
||||
}
|
||||
|
||||
func TestReader_ReadStringInto_Float(t *testing.T) {
|
||||
r := proto.NewReader(bytes.NewReader([]byte(",3.14\r\n")))
|
||||
buf := make([]byte, 16)
|
||||
|
||||
+24
-30
@@ -14,6 +14,28 @@ var semTimers = sync.Pool{
|
||||
},
|
||||
}
|
||||
|
||||
// putSemTimer stops a pooled timer and drains a stale fire before reuse,
|
||||
// portably across both timer-channel semantics (the drain must never block):
|
||||
//
|
||||
// - main module on go >= 1.23 (synchronous channels): Stop returns TRUE for
|
||||
// an expired-but-undelivered fire — the delivery is aborted — and false
|
||||
// only once the value was actually received. With the sole receiver
|
||||
// being Acquire's own select, the drain branch is unreachable; the
|
||||
// select-with-default is a safety net so a future semantics shift cannot
|
||||
// turn it into a blocking receive (reviewed on #3942).
|
||||
// - GODEBUG=asynctimerchan=1 (consumer main module on go < 1.23, old
|
||||
// buffered channels): Stop returns false and the fired value sits in the
|
||||
// buffer; the drain consumes it so the timer is clean for Reset-reuse.
|
||||
func putSemTimer(t *time.Timer) {
|
||||
if !t.Stop() {
|
||||
select {
|
||||
case <-t.C:
|
||||
default:
|
||||
}
|
||||
}
|
||||
semTimers.Put(t)
|
||||
}
|
||||
|
||||
// FastSemaphore is a channel-based semaphore optimized for performance.
|
||||
// It uses a fast path that avoids timer allocation when tokens are available.
|
||||
// The channel is pre-filled with tokens: Acquire = receive, Release = send.
|
||||
@@ -70,19 +92,13 @@ func (s *FastSemaphore) Acquire(ctx context.Context, timeout time.Duration, time
|
||||
|
||||
// Slow path: need to wait with timeout
|
||||
timer := semTimers.Get().(*time.Timer)
|
||||
defer semTimers.Put(timer)
|
||||
defer putSemTimer(timer)
|
||||
timer.Reset(timeout)
|
||||
|
||||
select {
|
||||
case <-s.tokens:
|
||||
if !timer.Stop() {
|
||||
<-timer.C
|
||||
}
|
||||
return nil
|
||||
case <-ctx.Done():
|
||||
if !timer.Stop() {
|
||||
<-timer.C
|
||||
}
|
||||
return ctx.Err()
|
||||
case <-timer.C:
|
||||
return timeoutErr
|
||||
@@ -152,42 +168,20 @@ func (s *FIFOSemaphore) TryAcquire() bool {
|
||||
func (s *FIFOSemaphore) Acquire(ctx context.Context, timeout time.Duration, timeoutErr error) error {
|
||||
// No fast path - always use timer to guarantee FIFO
|
||||
timer := semTimers.Get().(*time.Timer)
|
||||
defer semTimers.Put(timer)
|
||||
defer putSemTimer(timer)
|
||||
timer.Reset(timeout)
|
||||
|
||||
select {
|
||||
case <-s.tokens:
|
||||
if !timer.Stop() {
|
||||
<-timer.C
|
||||
}
|
||||
return nil
|
||||
case <-ctx.Done():
|
||||
if !timer.Stop() {
|
||||
<-timer.C
|
||||
}
|
||||
return ctx.Err()
|
||||
case <-timer.C:
|
||||
return timeoutErr
|
||||
}
|
||||
}
|
||||
|
||||
// AcquireBlocking acquires a token, blocking indefinitely until one is available.
|
||||
func (s *FIFOSemaphore) AcquireBlocking() {
|
||||
<-s.tokens
|
||||
}
|
||||
|
||||
// Release releases a token back to the semaphore.
|
||||
func (s *FIFOSemaphore) Release() {
|
||||
s.tokens <- struct{}{}
|
||||
}
|
||||
|
||||
// Close closes the semaphore, unblocking all waiting goroutines.
|
||||
// After close, all Acquire calls will receive a closed channel signal.
|
||||
func (s *FIFOSemaphore) Close() {
|
||||
close(s.tokens)
|
||||
}
|
||||
|
||||
// Len returns the current number of acquired tokens.
|
||||
func (s *FIFOSemaphore) Len() int32 {
|
||||
return s.max - int32(len(s.tokens))
|
||||
}
|
||||
@@ -0,0 +1,137 @@
|
||||
package internal
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
var errSemTestTimeout = errors.New("sem test timeout")
|
||||
|
||||
// hammerAcquireReleaseRace races Release against timeout expiry so the
|
||||
// Acquire select's token case regularly commits around the moment the pooled
|
||||
// timer fires, pinning the `if !timer.Stop() { <-timer.C }` drain against
|
||||
// BOTH timer-channel semantics (reviewed on #3942 as a suspected Go 1.23+
|
||||
// hang, refuted empirically):
|
||||
//
|
||||
// - go.mod >= 1.23 (synchronous channels): Stop returns TRUE for an
|
||||
// expired-but-undelivered fire — it aborts the delivery — and returns
|
||||
// false only once the value was actually received. With the receiving
|
||||
// select on the SAME goroutine, a taken token means the fire was never
|
||||
// received, so the drain branch is unreachable and cannot block.
|
||||
// - GODEBUG=asynctimerchan=1 (old buffered channels, e.g. a consumer main
|
||||
// module on go < 1.23): Stop returns false and the fired value sits
|
||||
// buffered; the drain consumes it so the pooled timer is clean for
|
||||
// Reset-reuse.
|
||||
func hammerAcquireReleaseRace(t *testing.T, tryAcquire func() bool, acquire func(context.Context, time.Duration, error) error, release func()) {
|
||||
t.Helper()
|
||||
ctx := context.Background()
|
||||
iters := 200000
|
||||
if testing.Short() {
|
||||
iters = 2000
|
||||
}
|
||||
done := make(chan struct{})
|
||||
go func() {
|
||||
defer close(done)
|
||||
for i := 0; i < iters; i++ {
|
||||
if !tryAcquire() {
|
||||
t.Error("token missing at loop start")
|
||||
return
|
||||
}
|
||||
var wg sync.WaitGroup
|
||||
wg.Add(1)
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
release()
|
||||
}()
|
||||
if err := acquire(ctx, time.Microsecond, errSemTestTimeout); err == nil {
|
||||
release()
|
||||
}
|
||||
wg.Wait()
|
||||
}
|
||||
}()
|
||||
select {
|
||||
case <-done:
|
||||
case <-time.After(60 * time.Second):
|
||||
t.Fatal("Acquire hung: timer drained with a blocking receive after Stop returned false " +
|
||||
"(Go 1.23+ synchronous timer channels discard undelivered fires)")
|
||||
}
|
||||
}
|
||||
|
||||
func TestFIFOSemaphoreAcquireTimerRace(t *testing.T) {
|
||||
s := NewFIFOSemaphore(1)
|
||||
hammerAcquireReleaseRace(t, s.TryAcquire, s.Acquire, s.Release)
|
||||
}
|
||||
|
||||
func TestFastSemaphoreAcquireTimerRace(t *testing.T) {
|
||||
s := NewFastSemaphore(1)
|
||||
hammerAcquireReleaseRace(t, s.TryAcquire, s.Acquire, s.Release)
|
||||
}
|
||||
|
||||
// The asynctimerchan=1 variants exercise the OLD buffered-channel semantics a
|
||||
// consumer main module on go < 1.23 gets (the GODEBUG default follows the
|
||||
// MAIN module's go directive, not this library's).
|
||||
func TestFIFOSemaphoreAcquireTimerRaceOldTimerChan(t *testing.T) {
|
||||
t.Setenv("GODEBUG", "asynctimerchan=1")
|
||||
s := NewFIFOSemaphore(1)
|
||||
hammerAcquireReleaseRace(t, s.TryAcquire, s.Acquire, s.Release)
|
||||
}
|
||||
|
||||
func TestFastSemaphoreAcquireTimerRaceOldTimerChan(t *testing.T) {
|
||||
t.Setenv("GODEBUG", "asynctimerchan=1")
|
||||
s := NewFastSemaphore(1)
|
||||
hammerAcquireReleaseRace(t, s.TryAcquire, s.Acquire, s.Release)
|
||||
}
|
||||
|
||||
// hammerStaleTimerFire pins pooled-timer hygiene: every iteration times an
|
||||
// Acquire out (firing the pooled timer), then immediately runs a
|
||||
// long-budget Acquire that must win via a released token — a stale fire
|
||||
// leaking through the pool would surface as an instant spurious timeout.
|
||||
func hammerStaleTimerFire(t *testing.T, tryAcquire func() bool, acquire func(context.Context, time.Duration, error) error, release func()) {
|
||||
t.Helper()
|
||||
ctx := context.Background()
|
||||
iters := 200
|
||||
if testing.Short() {
|
||||
iters = 20
|
||||
}
|
||||
for i := 0; i < iters; i++ {
|
||||
if !tryAcquire() {
|
||||
t.Fatal("token missing at loop start")
|
||||
}
|
||||
if err := acquire(ctx, time.Millisecond, errSemTestTimeout); err != errSemTestTimeout {
|
||||
t.Fatalf("iter %d: want timeout, got %v", i, err)
|
||||
}
|
||||
go func() {
|
||||
time.Sleep(2 * time.Millisecond)
|
||||
release()
|
||||
}()
|
||||
if err := acquire(ctx, 10*time.Second, errSemTestTimeout); err != nil {
|
||||
t.Fatalf("iter %d: spurious timeout from a stale pooled-timer fire: %v", i, err)
|
||||
}
|
||||
release()
|
||||
}
|
||||
}
|
||||
|
||||
func TestFIFOSemaphoreNoStaleTimerFire(t *testing.T) {
|
||||
s := NewFIFOSemaphore(1)
|
||||
hammerStaleTimerFire(t, s.TryAcquire, s.Acquire, s.Release)
|
||||
}
|
||||
|
||||
func TestFastSemaphoreNoStaleTimerFire(t *testing.T) {
|
||||
s := NewFastSemaphore(1)
|
||||
hammerStaleTimerFire(t, s.TryAcquire, s.Acquire, s.Release)
|
||||
}
|
||||
|
||||
func TestFIFOSemaphoreNoStaleTimerFireOldTimerChan(t *testing.T) {
|
||||
t.Setenv("GODEBUG", "asynctimerchan=1")
|
||||
s := NewFIFOSemaphore(1)
|
||||
hammerStaleTimerFire(t, s.TryAcquire, s.Acquire, s.Release)
|
||||
}
|
||||
|
||||
func TestFastSemaphoreNoStaleTimerFireOldTimerChan(t *testing.T) {
|
||||
t.Setenv("GODEBUG", "asynctimerchan=1")
|
||||
s := NewFastSemaphore(1)
|
||||
hammerStaleTimerFire(t, s.TryAcquire, s.Acquire, s.Release)
|
||||
}
|
||||
@@ -46,6 +46,14 @@ func (it *ScanIterator) Next(ctx context.Context) bool {
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
// Await the fetch before reading page/cursor: on the deferred
|
||||
// autopipeline face process() only enqueues, and reading the raw
|
||||
// fields of a not-yet-executed command would spin re-issuing SCANs
|
||||
// with a stale cursor forever. Err() blocks until executed there and
|
||||
// is a no-op read everywhere else.
|
||||
if err := it.cmd.Err(); err != nil {
|
||||
return false
|
||||
}
|
||||
|
||||
it.pos = 1
|
||||
|
||||
|
||||
+7
-4
@@ -10,7 +10,9 @@ import (
|
||||
)
|
||||
|
||||
var _ = Describe("ScanIterator", func() {
|
||||
var client *redis.Client
|
||||
var client redis.UniversalClient
|
||||
var rawClient *redis.Client
|
||||
var closeSubject func() error
|
||||
|
||||
seed := func(n int) error {
|
||||
pipe := client.Pipeline()
|
||||
@@ -44,12 +46,13 @@ var _ = Describe("ScanIterator", func() {
|
||||
}
|
||||
|
||||
BeforeEach(func() {
|
||||
client = redis.NewClient(redisOptions())
|
||||
Expect(client.FlushDB(ctx).Err()).NotTo(HaveOccurred())
|
||||
rawClient = redis.NewClient(redisOptions())
|
||||
client, closeSubject = newUniversalSubject(rawClient)
|
||||
Expect(rawClient.FlushDB(ctx).Err()).NotTo(HaveOccurred())
|
||||
})
|
||||
|
||||
AfterEach(func() {
|
||||
Expect(client.Close()).NotTo(HaveOccurred())
|
||||
Expect(closeSubject()).NotTo(HaveOccurred())
|
||||
})
|
||||
|
||||
It("should scan across empty DBs", func() {
|
||||
|
||||
@@ -96,6 +96,7 @@ func newJSONCmd(ctx context.Context, args ...interface{}) *JSONCmd {
|
||||
}
|
||||
|
||||
func (cmd *JSONCmd) String() string {
|
||||
cmd.await()
|
||||
return cmdString(cmd, cmd.val)
|
||||
}
|
||||
|
||||
@@ -105,6 +106,7 @@ func (cmd *JSONCmd) SetVal(val string) {
|
||||
|
||||
// Val returns the result of the JSON.GET command as a string.
|
||||
func (cmd *JSONCmd) Val() string {
|
||||
cmd.await()
|
||||
if len(cmd.val) == 0 && cmd.expanded != nil {
|
||||
val, err := json.Marshal(cmd.expanded)
|
||||
if err != nil {
|
||||
@@ -119,11 +121,13 @@ func (cmd *JSONCmd) Val() string {
|
||||
}
|
||||
|
||||
func (cmd *JSONCmd) Result() (string, error) {
|
||||
cmd.await()
|
||||
return cmd.Val(), cmd.Err()
|
||||
}
|
||||
|
||||
// Expanded returns the result of the JSON.GET command as unmarshalled JSON.
|
||||
func (cmd *JSONCmd) Expanded() (interface{}, error) {
|
||||
cmd.await()
|
||||
if len(cmd.val) != 0 && cmd.expanded == nil {
|
||||
err := json.Unmarshal([]byte(cmd.val), &cmd.expanded)
|
||||
if err != nil {
|
||||
@@ -136,15 +140,18 @@ func (cmd *JSONCmd) Expanded() (interface{}, error) {
|
||||
|
||||
func (cmd *JSONCmd) readReply(rd *proto.Reader) error {
|
||||
// nil response from JSON.(M)GET (cmd.baseCmd.err will be "redis: nil")
|
||||
// This happens when the key doesn't exist
|
||||
if cmd.baseCmd.Err() == Nil {
|
||||
// This happens when the key doesn't exist.
|
||||
// Use rawErr() (not Err()): readReply runs inside the batch's Exec, before
|
||||
// the autopipeline batch's done channel is closed, so Err()->await() would
|
||||
// deadlock on the very Exec that is calling readReply.
|
||||
if cmd.baseCmd.rawErr() == Nil {
|
||||
cmd.val = ""
|
||||
return Nil
|
||||
}
|
||||
|
||||
// Handle other base command errors
|
||||
if cmd.baseCmd.Err() != nil {
|
||||
return cmd.baseCmd.Err()
|
||||
if cmd.baseCmd.rawErr() != nil {
|
||||
return cmd.baseCmd.rawErr()
|
||||
}
|
||||
|
||||
if readType, err := rd.PeekReplyType(); err != nil {
|
||||
@@ -212,6 +219,7 @@ func NewJSONSliceCmd(ctx context.Context, args ...interface{}) *JSONSliceCmd {
|
||||
}
|
||||
|
||||
func (cmd *JSONSliceCmd) String() string {
|
||||
cmd.await()
|
||||
return cmdString(cmd, cmd.val)
|
||||
}
|
||||
|
||||
@@ -220,15 +228,19 @@ func (cmd *JSONSliceCmd) SetVal(val []interface{}) {
|
||||
}
|
||||
|
||||
func (cmd *JSONSliceCmd) Val() []interface{} {
|
||||
cmd.await()
|
||||
return cmd.val
|
||||
}
|
||||
|
||||
func (cmd *JSONSliceCmd) Result() ([]interface{}, error) {
|
||||
cmd.await()
|
||||
return cmd.val, cmd.err
|
||||
}
|
||||
|
||||
func (cmd *JSONSliceCmd) readReply(rd *proto.Reader) error {
|
||||
if cmd.baseCmd.Err() == Nil {
|
||||
// rawErr(), not Err(): readReply runs inside Exec before the batch's done
|
||||
// channel closes, so Err()->await() would deadlock (see JSONCmd.readReply).
|
||||
if cmd.baseCmd.rawErr() == Nil {
|
||||
cmd.val = nil
|
||||
return Nil
|
||||
}
|
||||
@@ -299,6 +311,7 @@ func NewIntPointerSliceCmd(ctx context.Context, args ...interface{}) *IntPointer
|
||||
}
|
||||
|
||||
func (cmd *IntPointerSliceCmd) String() string {
|
||||
cmd.await()
|
||||
return cmdString(cmd, cmd.val)
|
||||
}
|
||||
|
||||
@@ -307,10 +320,12 @@ func (cmd *IntPointerSliceCmd) SetVal(val []*int64) {
|
||||
}
|
||||
|
||||
func (cmd *IntPointerSliceCmd) Val() []*int64 {
|
||||
cmd.await()
|
||||
return cmd.val
|
||||
}
|
||||
|
||||
func (cmd *IntPointerSliceCmd) Result() ([]*int64, error) {
|
||||
cmd.await()
|
||||
return cmd.val, cmd.err
|
||||
}
|
||||
|
||||
|
||||
+27
-9
@@ -18,7 +18,9 @@ type JSONGetTestStruct struct {
|
||||
|
||||
var _ = Describe("JSON Commands", Label("json"), func() {
|
||||
ctx := context.TODO()
|
||||
var client *redis.Client
|
||||
var client redis.UniversalClient
|
||||
var rawClient *redis.Client
|
||||
var closeSubject func() error
|
||||
|
||||
setupRedisClient := func(protocolVersion int) *redis.Client {
|
||||
opt := &redis.Options{
|
||||
@@ -33,16 +35,23 @@ var _ = Describe("JSON Commands", Label("json"), func() {
|
||||
|
||||
AfterEach(func() {
|
||||
if client != nil {
|
||||
client.FlushDB(ctx)
|
||||
client.Close()
|
||||
// Flush through the SUBJECT: on the async face the flush is
|
||||
// ordered after any queued writes, and awaiting Err() forces it to
|
||||
// execute — so no late fire-and-forget command can survive the
|
||||
// flush, and the flush itself runs on a live client. Both steps
|
||||
// are asserted so a failed teardown cannot silently leak state
|
||||
// into later specs.
|
||||
Expect(client.FlushDB(ctx).Err()).NotTo(HaveOccurred())
|
||||
Expect(closeSubject()).NotTo(HaveOccurred())
|
||||
}
|
||||
})
|
||||
|
||||
protocols := []int{2, 3}
|
||||
for _, protocol := range protocols {
|
||||
BeforeEach(func() {
|
||||
client = setupRedisClient(protocol)
|
||||
Expect(client.FlushAll(ctx).Err()).NotTo(HaveOccurred())
|
||||
rawClient = setupRedisClient(protocol)
|
||||
client, closeSubject = newUniversalSubject(rawClient)
|
||||
Expect(rawClient.FlushAll(ctx).Err()).NotTo(HaveOccurred())
|
||||
})
|
||||
|
||||
Describe("arrays", Label("arrays"), func() {
|
||||
@@ -785,7 +794,9 @@ var _ = Describe("JSON Commands", Label("json"), func() {
|
||||
})
|
||||
|
||||
var _ = Describe("Go-Redis Advanced JSON and RediSearch Tests", func() {
|
||||
var client *redis.Client
|
||||
var client redis.UniversalClient
|
||||
var rawClient *redis.Client
|
||||
var closeSubject func() error
|
||||
var ctx = context.Background()
|
||||
|
||||
setupRedisClient := func(protocolVersion int) *redis.Client {
|
||||
@@ -801,8 +812,14 @@ var _ = Describe("Go-Redis Advanced JSON and RediSearch Tests", func() {
|
||||
|
||||
AfterEach(func() {
|
||||
if client != nil {
|
||||
client.FlushDB(ctx)
|
||||
client.Close()
|
||||
// Flush through the SUBJECT: on the async face the flush is
|
||||
// ordered after any queued writes, and awaiting Err() forces it to
|
||||
// execute — so no late fire-and-forget command can survive the
|
||||
// flush, and the flush itself runs on a live client. Both steps
|
||||
// are asserted so a failed teardown cannot silently leak state
|
||||
// into later specs.
|
||||
Expect(client.FlushDB(ctx).Err()).NotTo(HaveOccurred())
|
||||
Expect(closeSubject()).NotTo(HaveOccurred())
|
||||
}
|
||||
})
|
||||
|
||||
@@ -812,7 +829,8 @@ var _ = Describe("Go-Redis Advanced JSON and RediSearch Tests", func() {
|
||||
for _, protocol := range protocols {
|
||||
When("using protocol version", func() {
|
||||
BeforeEach(func() {
|
||||
client = setupRedisClient(protocol)
|
||||
rawClient = setupRedisClient(protocol)
|
||||
client, closeSubject = newUniversalSubject(rawClient)
|
||||
})
|
||||
|
||||
It("should perform complex JSON and RediSearch operations", func() {
|
||||
|
||||
@@ -6,12 +6,14 @@ import (
|
||||
"crypto/x509"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
@@ -460,6 +462,23 @@ type badConn struct {
|
||||
readErr, writeErr error
|
||||
}
|
||||
|
||||
// flakyWriteConn wraps a real, healthy connection and fails exactly one Write
|
||||
// when armed. Unlike seeding a broken conn into the idle pool (which the
|
||||
// pool's health check filters out before use), a wire-level failure on a
|
||||
// live conn genuinely reaches the pipeline write path — so retry tests using
|
||||
// it actually exercise the retry loop.
|
||||
type flakyWriteConn struct {
|
||||
net.Conn
|
||||
fail *atomic.Bool // arm with Store(true); consumed by the next Write
|
||||
}
|
||||
|
||||
func (c *flakyWriteConn) Write(b []byte) (int, error) {
|
||||
if c.fail.CompareAndSwap(true, false) {
|
||||
return 0, io.EOF
|
||||
}
|
||||
return c.Conn.Write(b)
|
||||
}
|
||||
|
||||
var _ net.Conn = &badConn{}
|
||||
|
||||
func (cn *badConn) SetReadDeadline(t time.Time) error {
|
||||
@@ -593,3 +612,51 @@ func initializeTLSCluster(ctx context.Context) error {
|
||||
func cleanupTLSCluster() {
|
||||
// TLS cluster is auto-managed by the container, no cleanup needed
|
||||
}
|
||||
|
||||
// newUniversalSubject returns the redis.UniversalClient that a parametrized
|
||||
// command suite runs against, selected by the GOREDIS_TEST_SUBJECT env var so a
|
||||
// whole suite can be replayed through the autopipeliner without touching each
|
||||
// spec:
|
||||
//
|
||||
// (unset)|client -> the *redis.Client itself (default; unchanged behavior)
|
||||
// ap-blocking -> client.AutoPipeline() (synchronous drop-in face)
|
||||
// ap-async -> client.AsyncAutoPipeline() (deferred face; result reads block)
|
||||
//
|
||||
// It returns the subject plus a single closer that closes the autopipeliner (if
|
||||
// any) and then the underlying *redis.Client — so callers just defer/AfterEach
|
||||
// one call. Admin/connection-only calls not on UniversalClient (FlushDB,
|
||||
// Config*, Ping, Info, Conn, ...) should use the underlying client directly.
|
||||
func newUniversalSubject(c *redis.Client) (redis.UniversalClient, func() error) {
|
||||
// closeWith closes the optional autopipeliner then the underlying client,
|
||||
// returning the first error.
|
||||
closeWith := func(ap *redis.AutoPipeliner) func() error {
|
||||
return func() error {
|
||||
var firstErr error
|
||||
if ap != nil {
|
||||
if err := ap.Close(); err != nil {
|
||||
firstErr = err
|
||||
}
|
||||
}
|
||||
if err := c.Close(); err != nil && firstErr == nil {
|
||||
firstErr = err
|
||||
}
|
||||
return firstErr
|
||||
}
|
||||
}
|
||||
switch os.Getenv("GOREDIS_TEST_SUBJECT") {
|
||||
case "ap-blocking":
|
||||
ap, err := c.AutoPipelineWithOptions(&redis.AutoPipelineOptions{MaxBatchSize: 300})
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
return ap, closeWith(ap)
|
||||
case "ap-async":
|
||||
ap, err := c.AsyncAutoPipelineWithOptions(&redis.AutoPipelineOptions{MaxBatchSize: 300})
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
return ap, closeWith(ap)
|
||||
default:
|
||||
return c, closeWith(nil)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,133 @@
|
||||
package maintnotifications
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/redis/go-redis/v9/internal/pool"
|
||||
)
|
||||
|
||||
// countingPool records how many pool hooks were added and removed so a test can
|
||||
// assert a maintnotifications hook is attached to (and detached from) a pool.
|
||||
type countingPool struct {
|
||||
pool.Pooler
|
||||
added int
|
||||
removed int
|
||||
}
|
||||
|
||||
func (p *countingPool) AddPoolHook(pool.PoolHook) { p.added++ }
|
||||
func (p *countingPool) RemovePoolHook(pool.PoolHook) { p.removed++ }
|
||||
|
||||
// TestInitPoolHookForPool verifies that a dedicated pool (e.g. a client's
|
||||
// pipeline connection pool) receives an independent maintnotifications pool hook,
|
||||
// and that the hook is detached on Close. This guards the regression where the
|
||||
// dedicated pipeline pool never got the MOVING/MIGRATING handoff hook, so
|
||||
// autopipelined/pipelined commands ran on connections without handoff handling.
|
||||
func TestInitPoolHookForPool(t *testing.T) {
|
||||
client := &MockClient{options: &MockOptions{}}
|
||||
primary := &countingPool{}
|
||||
pipelinePool := &countingPool{}
|
||||
|
||||
manager, err := NewManager(client, primary, DefaultConfig())
|
||||
if err != nil {
|
||||
t.Fatalf("failed to create manager: %v", err)
|
||||
}
|
||||
dialer := func(context.Context, string, string) (net.Conn, error) { return nil, nil }
|
||||
|
||||
manager.InitPoolHook(dialer)
|
||||
manager.InitPoolHookForPool(pipelinePool, dialer)
|
||||
|
||||
if primary.added != 1 {
|
||||
t.Fatalf("primary pool: expected 1 hook added, got %d", primary.added)
|
||||
}
|
||||
if pipelinePool.added != 1 {
|
||||
t.Fatalf("pipeline pool: expected 1 hook added, got %d", pipelinePool.added)
|
||||
}
|
||||
|
||||
// The additional hook must be a distinct instance bound to its own pool, so
|
||||
// failed-handoff removals target the pipeline pool, not the primary pool.
|
||||
if len(manager.additionalPoolHooks) != 1 {
|
||||
t.Fatalf("expected 1 additional pool hook tracked, got %d", len(manager.additionalPoolHooks))
|
||||
}
|
||||
if manager.additionalPoolHooks[0].hook == manager.poolHooksRef {
|
||||
t.Fatal("additional pool hook must be a separate instance from the primary hook")
|
||||
}
|
||||
if manager.additionalPoolHooks[0].pool != pipelinePool {
|
||||
t.Fatal("additional pool hook bound to the wrong pool")
|
||||
}
|
||||
|
||||
if err := manager.Close(); err != nil {
|
||||
t.Fatalf("manager close failed: %v", err)
|
||||
}
|
||||
if pipelinePool.removed != 1 {
|
||||
t.Fatalf("pipeline pool: expected hook removed on Close, got %d removals", pipelinePool.removed)
|
||||
}
|
||||
}
|
||||
|
||||
// TestInitPoolHookForPoolNil is a no-op guard: a nil pool must not panic or
|
||||
// register a hook.
|
||||
func TestInitPoolHookForPoolNil(t *testing.T) {
|
||||
client := &MockClient{options: &MockOptions{}}
|
||||
manager, err := NewManager(client, &countingPool{}, DefaultConfig())
|
||||
if err != nil {
|
||||
t.Fatalf("failed to create manager: %v", err)
|
||||
}
|
||||
manager.InitPoolHookForPool(nil, func(context.Context, string, string) (net.Conn, error) { return nil, nil })
|
||||
if len(manager.additionalPoolHooks) != 0 {
|
||||
t.Fatal("nil pool must not register an additional hook")
|
||||
}
|
||||
}
|
||||
|
||||
// TestCloseRetriesAdditionalHookAfterShutdownError pins Close's put-back
|
||||
// semantics: when an additional pool hook's Shutdown fails (a handoff worker
|
||||
// that won't drain within the budget), the failed hook — and any not yet
|
||||
// processed — must be restored so a RETRIED Close still shuts it down and
|
||||
// detaches it from its pool. The original code nilled the slice before the
|
||||
// loop, so a retried Close silently skipped teardown and leaked the hook.
|
||||
func TestCloseRetriesAdditionalHookAfterShutdownError(t *testing.T) {
|
||||
client := &MockClient{options: &MockOptions{}}
|
||||
primary := &countingPool{}
|
||||
pipelinePool := &countingPool{}
|
||||
|
||||
manager, err := NewManager(client, primary, DefaultConfig())
|
||||
if err != nil {
|
||||
t.Fatalf("failed to create manager: %v", err)
|
||||
}
|
||||
manager.shutdownTimeout = 50 * time.Millisecond // seam: fail fast, not in 10s
|
||||
|
||||
dialer := func(context.Context, string, string) (net.Conn, error) { return nil, nil }
|
||||
manager.InitPoolHook(dialer)
|
||||
manager.InitPoolHookForPool(pipelinePool, dialer)
|
||||
|
||||
hook := manager.additionalPoolHooks[0].hook
|
||||
// A handoff worker that won't drain: Shutdown waits on workerWg and must
|
||||
// time out against the shortened budget.
|
||||
hook.workerManager.workerWg.Add(1)
|
||||
|
||||
if err := manager.Close(); err == nil {
|
||||
t.Fatal("Close must fail while a hook's workers cannot drain")
|
||||
}
|
||||
if len(manager.additionalPoolHooks) != 1 {
|
||||
t.Fatalf("failed Close must put the hook back for retry, tracked=%d", len(manager.additionalPoolHooks))
|
||||
}
|
||||
if pipelinePool.removed != 0 {
|
||||
t.Fatalf("hook must stay attached after the failed Close, removals=%d", pipelinePool.removed)
|
||||
}
|
||||
if manager.closed.Load() {
|
||||
t.Fatal("manager must reopen after a failed Close so the caller can retry")
|
||||
}
|
||||
|
||||
// Worker drains; the retried Close finishes the job.
|
||||
hook.workerManager.workerWg.Done()
|
||||
if err := manager.Close(); err != nil {
|
||||
t.Fatalf("retried Close: %v", err)
|
||||
}
|
||||
if len(manager.additionalPoolHooks) != 0 {
|
||||
t.Fatalf("retried Close must consume the restored hook, tracked=%d", len(manager.additionalPoolHooks))
|
||||
}
|
||||
if pipelinePool.removed != 1 {
|
||||
t.Fatalf("retried Close must detach the hook from its pool, removals=%d", pipelinePool.removed)
|
||||
}
|
||||
}
|
||||
+149
-12
@@ -77,11 +77,24 @@ type Manager struct {
|
||||
activeOperationCount atomic.Int64 // Number of active operations
|
||||
closed atomic.Bool // Manager closed state
|
||||
|
||||
// shutdownTimeout bounds each pool hook's Shutdown during Close. A field
|
||||
// (not a constant) so tests can exercise the failed-Close-then-retry
|
||||
// path without waiting out the real budget.
|
||||
shutdownTimeout time.Duration
|
||||
|
||||
// Notification hooks for extensibility
|
||||
hooks []NotificationHook
|
||||
hooksMu sync.RWMutex // Protects hooks slice
|
||||
poolHooksRef *PoolHook
|
||||
|
||||
// additionalPoolHooks are pool hooks bound to pools other than the primary
|
||||
// one (e.g. a dedicated pipeline connection pool). Each is an independent
|
||||
// *PoolHook bound to its own pool because the hook's failed-handoff removal
|
||||
// target (HandoffRequest.Pool) is taken from the hook's single pool field,
|
||||
// so one hook cannot safely serve two pools. They share this Manager as
|
||||
// their operations manager, keeping MOVING/MIGRATING tracking centralized.
|
||||
additionalPoolHooks []additionalPoolHook
|
||||
|
||||
// Connections that successfully enabled maintnotifications. These need to be
|
||||
// retired before the pool-level listeners are removed.
|
||||
maintNotificationsConns sync.Map // connID -> *pool.Conn
|
||||
@@ -116,11 +129,12 @@ func NewManager(client interfaces.ClientInterface, pool pool.Pooler, config *Con
|
||||
}
|
||||
|
||||
hm := &Manager{
|
||||
client: client,
|
||||
pool: pool,
|
||||
options: client.GetOptions(),
|
||||
config: config.Clone(),
|
||||
hooks: make([]NotificationHook, 0),
|
||||
client: client,
|
||||
pool: pool,
|
||||
options: client.GetOptions(),
|
||||
config: config.Clone(),
|
||||
hooks: make([]NotificationHook, 0),
|
||||
shutdownTimeout: 10 * time.Second,
|
||||
}
|
||||
|
||||
// Set up push notification handling
|
||||
@@ -137,6 +151,78 @@ func (hm *Manager) InitPoolHook(baseDialer func(context.Context, string, string)
|
||||
hm.pool.AddPoolHook(poolHook)
|
||||
}
|
||||
|
||||
// additionalPoolHook pairs a pool hook with the pool it was attached to so the
|
||||
// manager can shut it down and detach it on Close.
|
||||
type additionalPoolHook struct {
|
||||
pool pool.Pooler
|
||||
hook *PoolHook
|
||||
}
|
||||
|
||||
// InitPoolHookForPool attaches a maintnotifications pool hook to an additional
|
||||
// pool (e.g. a client's dedicated pipeline connection pool). A fresh, independent
|
||||
// *PoolHook is created and bound to the given pool so that connections which fail
|
||||
// handoff are removed from the correct pool — the hook's removal target is its own
|
||||
// single pool field, so the primary hook cannot be reused for a second pool. The
|
||||
// new hook shares this Manager as its operations manager, so MOVING/MIGRATING
|
||||
// tracking and notification handling stay centralized across both pools.
|
||||
func (hm *Manager) InitPoolHookForPool(p pool.Pooler, baseDialer func(context.Context, string, string) (net.Conn, error)) {
|
||||
if p == nil {
|
||||
return
|
||||
}
|
||||
poolSize := 0
|
||||
network := ""
|
||||
if hm.options != nil {
|
||||
poolSize = hm.options.GetPoolSize()
|
||||
network = hm.options.GetNetwork()
|
||||
}
|
||||
hook := NewPoolHookWithPoolSize(baseDialer, network, hm.config, hm, poolSize)
|
||||
hook.SetPool(p)
|
||||
// The closed check, the append, AND the AddPoolHook must all be atomic with
|
||||
// respect to Close's snapshot: hold the lock across all three so either Close
|
||||
// runs first (closed==true here, so we neither register nor attach) or we
|
||||
// register+attach first (Close's snapshot then includes this hook and tears
|
||||
// it down). Attaching outside the lock left a window where Close could
|
||||
// snapshot/tear-down between the append and the attach, then AddPoolHook
|
||||
// would re-attach to a closed manager's pool — leaking an active hook.
|
||||
// p.AddPoolHook is lock-free (atomic swap on the pool's own hook manager) and
|
||||
// never calls back into this manager, so holding hooksMu across it is safe.
|
||||
hm.hooksMu.Lock()
|
||||
defer hm.hooksMu.Unlock()
|
||||
if hm.closed.Load() {
|
||||
return
|
||||
}
|
||||
hm.additionalPoolHooks = append(hm.additionalPoolHooks, additionalPoolHook{pool: p, hook: hook})
|
||||
p.AddPoolHook(hook)
|
||||
}
|
||||
|
||||
// hookForConn returns the pool hook that owns cn's pool: an additional hook
|
||||
// when the connection came from a secondary pool (e.g. a client's dedicated
|
||||
// pipeline pool), the primary hook otherwise. Handoffs must be queued through
|
||||
// the owning hook — the HandoffRequest carries that hook's pool, and a failed
|
||||
// handoff removes the connection from it, so queuing a pipeline-pool
|
||||
// connection on the primary hook would close the connection without freeing
|
||||
// its slot in the pipeline pool's bookkeeping.
|
||||
func (hm *Manager) hookForConn(cn *pool.Conn) *PoolHook {
|
||||
if cn == nil {
|
||||
return hm.poolHooksRef
|
||||
}
|
||||
name := cn.PoolName()
|
||||
if name == "" {
|
||||
return hm.poolHooksRef
|
||||
}
|
||||
hm.hooksMu.RLock()
|
||||
defer hm.hooksMu.RUnlock()
|
||||
for _, ah := range hm.additionalPoolHooks {
|
||||
// Pooler does not expose the name; the concrete pool does. A Pooler
|
||||
// implementation without it simply never matches and falls through to
|
||||
// the primary hook — the pre-existing behavior.
|
||||
if np, ok := ah.pool.(interface{ Name() string }); ok && np.Name() == name {
|
||||
return ah.hook
|
||||
}
|
||||
}
|
||||
return hm.poolHooksRef
|
||||
}
|
||||
|
||||
// setupPushNotifications sets up push notification handling by registering with the client's processor.
|
||||
func (hm *Manager) setupPushNotifications() error {
|
||||
processor := hm.client.GetPushProcessor()
|
||||
@@ -287,17 +373,38 @@ func (hm *Manager) maintNotificationsConnSnapshot() []*pool.Conn {
|
||||
|
||||
func (hm *Manager) retireMaintNotificationsConns(ctx context.Context) {
|
||||
conns := hm.maintNotificationsConnSnapshot()
|
||||
if len(conns) == 0 || hm.pool == nil {
|
||||
if len(conns) == 0 {
|
||||
return
|
||||
}
|
||||
|
||||
if retirer, ok := hm.pool.(pool.ConnRetirer); ok {
|
||||
retirer.RetireConns(ctx, conns, pool.CloseReasonMaintNotificationsDisabled)
|
||||
return
|
||||
// Tracked connections can live in the primary pool OR in any additional
|
||||
// pool this manager attached a hook to (e.g. a client's dedicated pipeline
|
||||
// connection pool — its conns run initConn and are tracked exactly like
|
||||
// primary ones). Retire through every pool: RetireConns skips connections
|
||||
// a pool does not own, so offering the full snapshot to each pool is safe.
|
||||
// Missing the additional pools left pipeline connections in service with
|
||||
// maintnotifications enabled but no hook attached after a runtime
|
||||
// downgrade — pushes on them were silently dropped.
|
||||
pools := make([]pool.Pooler, 0, 1+len(hm.additionalPoolHooks))
|
||||
if hm.pool != nil {
|
||||
pools = append(pools, hm.pool)
|
||||
}
|
||||
hm.hooksMu.RLock()
|
||||
for _, ah := range hm.additionalPoolHooks {
|
||||
if ah.pool != nil {
|
||||
pools = append(pools, ah.pool)
|
||||
}
|
||||
}
|
||||
hm.hooksMu.RUnlock()
|
||||
|
||||
for _, cn := range conns {
|
||||
_ = hm.pool.CloseConn(ctx, cn, pool.CloseReasonMaintNotificationsDisabled, pool.MetricStateIdle)
|
||||
for _, pl := range pools {
|
||||
if retirer, ok := pl.(pool.ConnRetirer); ok {
|
||||
retirer.RetireConns(ctx, conns, pool.CloseReasonMaintNotificationsDisabled)
|
||||
continue
|
||||
}
|
||||
for _, cn := range conns {
|
||||
_ = pl.CloseConn(ctx, cn, pool.CloseReasonMaintNotificationsDisabled, pool.MetricStateIdle)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -315,7 +422,7 @@ func (hm *Manager) Close() error {
|
||||
// Shutdown the pool hook if it exists
|
||||
if hm.poolHooksRef != nil {
|
||||
// Use a timeout to prevent hanging indefinitely
|
||||
shutdownCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||||
shutdownCtx, cancel := context.WithTimeout(context.Background(), hm.shutdownTimeout)
|
||||
defer cancel()
|
||||
|
||||
err := hm.poolHooksRef.Shutdown(shutdownCtx)
|
||||
@@ -330,6 +437,36 @@ func (hm *Manager) Close() error {
|
||||
}
|
||||
}
|
||||
|
||||
// Shutdown and detach any hooks bound to additional pools (e.g. a dedicated
|
||||
// pipeline pool). Snapshot under the lock so we don't iterate concurrently
|
||||
// with a registering InitPoolHookForPool; Shutdown itself runs unlocked.
|
||||
hm.hooksMu.Lock()
|
||||
additional := hm.additionalPoolHooks
|
||||
hm.additionalPoolHooks = nil
|
||||
hm.hooksMu.Unlock()
|
||||
for i, ah := range additional {
|
||||
shutdownCtx, cancel := context.WithTimeout(context.Background(), hm.shutdownTimeout)
|
||||
err := ah.hook.Shutdown(shutdownCtx)
|
||||
cancel()
|
||||
if err != nil {
|
||||
// Could not cleanly shut down this hook. Put it and the ones not
|
||||
// yet processed back so a retried Close still sees them, then stay
|
||||
// open so the caller can retry, matching the primary-hook behavior
|
||||
// above. Hooks before i already shut down and detached.
|
||||
hm.hooksMu.Lock()
|
||||
remaining := make([]additionalPoolHook, 0, len(additional)-i+len(hm.additionalPoolHooks))
|
||||
remaining = append(remaining, additional[i:]...)
|
||||
remaining = append(remaining, hm.additionalPoolHooks...)
|
||||
hm.additionalPoolHooks = remaining
|
||||
hm.hooksMu.Unlock()
|
||||
hm.closed.Store(false)
|
||||
return err
|
||||
}
|
||||
if ah.pool != nil {
|
||||
ah.pool.RemovePoolHook(ah.hook)
|
||||
}
|
||||
}
|
||||
|
||||
// Clear all active operations
|
||||
hm.activeMovingOps.Range(func(key, value interface{}) bool {
|
||||
hm.activeMovingOps.Delete(key)
|
||||
|
||||
@@ -833,10 +833,13 @@ func TestConnectionHook(t *testing.T) {
|
||||
t.Error("Connection should be pooled after failed handoff attempt")
|
||||
}
|
||||
|
||||
// Wait for handoff to be attempted and fail
|
||||
time.Sleep(100 * time.Millisecond)
|
||||
|
||||
// Verify that the connection was removed from the pool
|
||||
// Wait for the handoff to be attempted, retried (MaxHandoffRetries:3
|
||||
// with backoff), and finally fail: a fixed sleep loses that race on
|
||||
// loaded runners, so poll until the removal lands.
|
||||
deadline := time.Now().Add(5 * time.Second)
|
||||
for !mockPool.WasRemoved(conn.GetID()) && time.Now().Before(deadline) {
|
||||
time.Sleep(10 * time.Millisecond)
|
||||
}
|
||||
if !mockPool.WasRemoved(conn.GetID()) {
|
||||
t.Errorf("conn[%d] should have been removed from pool after handoff failure", conn.GetID())
|
||||
}
|
||||
|
||||
@@ -161,8 +161,15 @@ func (snh *NotificationHandler) handleMoving(ctx context.Context, handlerCtx pus
|
||||
// If the connection is in use (StateInUse), it will be queued when returned to the pool via OnPut.
|
||||
// This handles the case where the connection is idle and might never be retrieved again.
|
||||
if poolConn.GetStateMachine().GetState() == pool.StateIdle {
|
||||
if snh.manager.poolHooksRef != nil && snh.manager.poolHooksRef.workerManager != nil {
|
||||
if err := snh.manager.poolHooksRef.workerManager.queueHandoff(poolConn); err != nil {
|
||||
// Queue on the hook that owns this connection's pool, not
|
||||
// unconditionally on the primary one: the request carries the
|
||||
// hook's pool, and a failed handoff removes the connection
|
||||
// from it — for a dedicated pipeline-pool connection the
|
||||
// primary pool cannot do that, which would close the
|
||||
// connection while leaving a dead slot behind.
|
||||
owner := snh.manager.hookForConn(poolConn)
|
||||
if owner != nil && owner.workerManager != nil {
|
||||
if err := owner.workerManager.queueHandoff(poolConn); err != nil {
|
||||
internal.Logger.Printf(context.Background(), logs.FailedToQueueHandoff(poolConn.GetID(), err))
|
||||
} else {
|
||||
// Mark the connection as queued for handoff to prevent it from being retrieved
|
||||
|
||||
+80
@@ -188,6 +188,86 @@ type Options struct {
|
||||
// default: 32KiB (32768 bytes)
|
||||
WriteBufferSize int
|
||||
|
||||
// PipelineReadBufferSize is the size of the bufio.Reader buffer for pipeline connections.
|
||||
// If set to a value > 0, a separate connection pool will be created specifically for
|
||||
// pipelining operations (Pipeline, AutoPipeline and AsyncAutoPipeline) with
|
||||
// this buffer size.
|
||||
//
|
||||
// This allows you to use large buffers for pipelining (to reduce syscalls and improve
|
||||
// throughput) while keeping regular command buffers small (to save memory).
|
||||
//
|
||||
// If not set (0), pipeline operations will use the regular connection pool with
|
||||
// ReadBufferSize buffers.
|
||||
//
|
||||
// Recommended: 64–128 KiB for high-throughput pipelining. The benefit here is
|
||||
// on the READ side: a batch's replies arrive as one large stream, and a bigger
|
||||
// buffer consumes them in fewer syscalls instead of refilling repeatedly
|
||||
// mid-batch. Size it to roughly the reply volume of a typical batch — which
|
||||
// for read-heavy pipelines is dominated by value sizes, not command count.
|
||||
// (The write-side counterpart, sizing to the outgoing wire bytes so the batch
|
||||
// flushes without overflowing mid-write, belongs to PipelineWriteBufferSize.)
|
||||
// Benchmarks show throughput climbs from the 32 KiB default up to ~64 KiB and
|
||||
// then plateaus; going beyond ~128 KiB gives no further gain and very large
|
||||
// buffers (≥512 KiB) can regress throughput and waste memory. Bigger is not
|
||||
// better.
|
||||
//
|
||||
// Example:
|
||||
// client := redis.NewClient(&redis.Options{
|
||||
// Addr: "localhost:6379",
|
||||
// ReadBufferSize: 32 * 1024, // 32 KiB for regular commands
|
||||
// PipelineReadBufferSize: 128 * 1024, // 128 KiB for pipelining
|
||||
// PipelineWriteBufferSize: 128 * 1024,
|
||||
// })
|
||||
//
|
||||
// Memory impact: With PoolSize=100 and PipelinePoolSize=10:
|
||||
// - Without pipeline pool: 100 conns × 128 KiB = 12.8 MB (if all use 128 KiB buffers)
|
||||
// - With pipeline pool: (100 × 32 KiB) + (10 × 128 KiB) = 4.5 MB (~65% savings)
|
||||
//
|
||||
// default: 0 (use ReadBufferSize)
|
||||
PipelineReadBufferSize int
|
||||
|
||||
// PipelineWriteBufferSize is the size of the bufio.Writer buffer for pipeline connections.
|
||||
// If set to a value > 0, a separate connection pool will be created specifically for
|
||||
// pipelining operations (Pipeline, AutoPipeline and AsyncAutoPipeline) with
|
||||
// this buffer size.
|
||||
//
|
||||
// This allows you to use large buffers for pipelining (to reduce syscalls and improve
|
||||
// throughput) while keeping regular command buffers small (to save memory).
|
||||
//
|
||||
// If not set (0), pipeline operations will use the regular connection pool with
|
||||
// WriteBufferSize buffers.
|
||||
//
|
||||
// Recommended: 64–128 KiB for high-throughput pipelining (size to roughly
|
||||
// MaxBatchSize × average-command-bytes). Throughput plateaus past ~64 KiB and
|
||||
// gains nothing beyond ~128 KiB; very large buffers (≥512 KiB) can regress it.
|
||||
// See PipelineReadBufferSize for the full rationale.
|
||||
//
|
||||
// default: 0 (use WriteBufferSize)
|
||||
PipelineWriteBufferSize int
|
||||
|
||||
// PipelinePoolSize is the pool size for the separate pipeline connection pool.
|
||||
// Only used if PipelineReadBufferSize or PipelineWriteBufferSize is set.
|
||||
//
|
||||
// Pipelining typically needs fewer connections than regular operations because
|
||||
// batching reduces connection contention. A smaller pool saves memory while
|
||||
// maintaining high throughput.
|
||||
//
|
||||
// If not set (0), defaults to 10 connections.
|
||||
//
|
||||
// default: 10
|
||||
PipelinePoolSize int
|
||||
|
||||
// AutoPipelineOptions is the default config for BOTH autopipeliner faces:
|
||||
// AutoPipeline and AsyncAutoPipeline use it when called without an
|
||||
// explicit config, falling back to their per-face defaults
|
||||
// (DefaultBlockingAutoPipelineOptions / DefaultAutoPipelineOptions) when it
|
||||
// is nil. Pass a config to either method to override. Commands issued
|
||||
// through an autopipeliner are batched into pipelines to cut round-trips
|
||||
// and raise throughput.
|
||||
//
|
||||
// EXPERIMENTAL: this API is subject to change, use with caution.
|
||||
AutoPipelineOptions *AutoPipelineOptions
|
||||
|
||||
// PoolFIFO type of connection pool.
|
||||
//
|
||||
// - true for FIFO pool
|
||||
|
||||
+329
-41
@@ -142,6 +142,19 @@ type ClusterOptions struct {
|
||||
// default: 32KiB (32768 bytes)
|
||||
WriteBufferSize int
|
||||
|
||||
// PipelineReadBufferSize, PipelineWriteBufferSize and PipelinePoolSize
|
||||
// configure an optional separate connection pool used for pipelining on
|
||||
// each node, with its own (typically larger) buffers. See the same-named
|
||||
// fields on Options for details. The pool is created only when PipelineReadBufferSize or PipelineWriteBufferSize is set (PipelinePoolSize alone does not enable it).
|
||||
PipelineReadBufferSize int
|
||||
PipelineWriteBufferSize int
|
||||
PipelinePoolSize int
|
||||
|
||||
// AutoPipelineOptions is the default config for BOTH autopipeliner faces
|
||||
// (AutoPipeline and AsyncAutoPipeline), applied when they are called
|
||||
// without explicit options. See Options.AutoPipelineOptions.
|
||||
AutoPipelineOptions *AutoPipelineOptions
|
||||
|
||||
TLSConfig *tls.Config
|
||||
|
||||
// DisableRoutingPolicies disables the request/response policy routing system.
|
||||
@@ -457,11 +470,15 @@ func (opt *ClusterOptions) clientOptions() *Options {
|
||||
ConnMaxLifetimeJitter: opt.ConnMaxLifetimeJitter,
|
||||
ReadBufferSize: opt.ReadBufferSize,
|
||||
WriteBufferSize: opt.WriteBufferSize,
|
||||
DisableIdentity: opt.DisableIdentity,
|
||||
DisableIndentity: opt.DisableIdentity,
|
||||
IdentitySuffix: opt.IdentitySuffix,
|
||||
FailingTimeoutSeconds: opt.FailingTimeoutSeconds,
|
||||
TLSConfig: opt.TLSConfig,
|
||||
|
||||
PipelineReadBufferSize: opt.PipelineReadBufferSize,
|
||||
PipelineWriteBufferSize: opt.PipelineWriteBufferSize,
|
||||
PipelinePoolSize: opt.PipelinePoolSize,
|
||||
DisableIdentity: opt.DisableIdentity,
|
||||
DisableIndentity: opt.DisableIndentity,
|
||||
IdentitySuffix: opt.IdentitySuffix,
|
||||
FailingTimeoutSeconds: opt.FailingTimeoutSeconds,
|
||||
TLSConfig: opt.TLSConfig,
|
||||
// If ClusterSlots is populated, then we probably have an artificial
|
||||
// cluster whose nodes are not in clustering mode (otherwise there isn't
|
||||
// much use for ClusterSlots config). This means we cannot execute the
|
||||
@@ -1161,6 +1178,11 @@ type ClusterClient struct {
|
||||
// topology) so any connection serving an HIMPORT SET can lazily replay
|
||||
// the PREPARE (see himport.go, himport_cluster.go).
|
||||
himport *himportRegistry
|
||||
|
||||
autopipelinerMu *sync.Mutex // guards the autopipeliner fields against concurrent first-call creation
|
||||
autopipeliner *AutoPipeliner // blocking face (ClusterClient.AutoPipeline)
|
||||
asyncAutopipeliner *AutoPipeliner // deferred face (ClusterClient.AsyncAutoPipeline)
|
||||
autopipelinerClosed bool // set by Close: refuse to resurrect a pipeliner on a closed client
|
||||
}
|
||||
|
||||
// NewClusterClient returns a Redis Cluster client as described in
|
||||
@@ -1173,9 +1195,10 @@ func NewClusterClient(opt *ClusterOptions) *ClusterClient {
|
||||
opt.init()
|
||||
|
||||
c := &ClusterClient{
|
||||
opt: opt,
|
||||
nodes: newClusterNodes(opt),
|
||||
himport: newHImportRegistry(),
|
||||
opt: opt,
|
||||
nodes: newClusterNodes(opt),
|
||||
himport: newHImportRegistry(),
|
||||
autopipelinerMu: &sync.Mutex{},
|
||||
}
|
||||
|
||||
// Every node client shares the cluster-wide fieldset registry, replicas
|
||||
@@ -1239,7 +1262,26 @@ func (c *ClusterClient) ReloadState(ctx context.Context) {
|
||||
// It is rare to Close a ClusterClient, as the ClusterClient is meant
|
||||
// to be long-lived and shared between many goroutines.
|
||||
func (c *ClusterClient) Close() error {
|
||||
return c.nodes.Close()
|
||||
// Stop both cached autopipeliners (blocking and async faces) before
|
||||
// closing nodes, so its background flusher goroutines don't outlive the
|
||||
// client. AutoPipeliner.Close is idempotent and nil-safe here.
|
||||
c.autopipelinerMu.Lock()
|
||||
ap, async := c.autopipeliner, c.asyncAutopipeliner
|
||||
c.autopipeliner, c.asyncAutopipeliner = nil, nil
|
||||
c.autopipelinerClosed = true // getters refuse to resurrect on a closed client
|
||||
c.autopipelinerMu.Unlock()
|
||||
var firstErr error
|
||||
for _, p := range []*AutoPipeliner{ap, async} {
|
||||
if p != nil {
|
||||
if err := p.Close(); err != nil && firstErr == nil {
|
||||
firstErr = err
|
||||
}
|
||||
}
|
||||
}
|
||||
if err := c.nodes.Close(); err != nil && firstErr == nil {
|
||||
firstErr = err
|
||||
}
|
||||
return firstErr
|
||||
}
|
||||
|
||||
func (c *ClusterClient) Process(ctx context.Context, cmd Cmder) error {
|
||||
@@ -1571,6 +1613,155 @@ func (c *ClusterClient) Pipeline() Pipeliner {
|
||||
return &pipe
|
||||
}
|
||||
|
||||
// clusterAutoPipelineOptions applies the cluster shard-count default: commands
|
||||
// are routed to shards by slot (see installAutoPipelineSharding), so unlike a
|
||||
// standalone client — which defaults to a single deep queue — a cluster client
|
||||
// wants several shards to keep concurrent nodes' batches separate. The caller's
|
||||
// config is copied before the default is filled in, never mutated.
|
||||
func clusterAutoPipelineOptions(cfg *AutoPipelineOptions) *AutoPipelineOptions {
|
||||
c2 := *cfg
|
||||
if c2.NumShards == 0 {
|
||||
c2.NumShards = numAutoPipelineShards()
|
||||
}
|
||||
// A cluster always routes by slot, so per-key order holds regardless of shard
|
||||
// count; mark it so construction's NumShards ordering check (which targets
|
||||
// round-robin sharding) does not reject the cluster default or an explicit
|
||||
// NumShards on the deferred (async) face.
|
||||
c2.contentSharded = true
|
||||
return &c2
|
||||
}
|
||||
|
||||
// AutoPipeline returns the blocking autopipeliner for this cluster client: each
|
||||
// command call blocks until executed (drop-in shape) while the engine batches
|
||||
// concurrent callers into pipelines. Commands keep per-goroutine order; across
|
||||
// nodes, ordering is per key (slot routing keeps a key on one shard and node
|
||||
// sub-pipelines execute concurrently). Use AutoPipelineWithOptions to override
|
||||
// DefaultBlockingAutoPipelineOptions. Cached/shared; first call's config wins.
|
||||
// Close it (or the client) to release its goroutines.
|
||||
//
|
||||
// It returns an error if the supplied config is invalid (e.g. MaxConcurrentBatches>1
|
||||
// without Unordered, or a negative size); on error no instance is cached.
|
||||
//
|
||||
// EXPERIMENTAL: this API is subject to change, use with caution.
|
||||
func (c *ClusterClient) AutoPipeline() (*AutoPipeliner, error) {
|
||||
return c.AutoPipelineWithOptions(nil)
|
||||
}
|
||||
|
||||
// AutoPipelineWithOptions is AutoPipeline with explicit options instead of
|
||||
// ClusterOptions.AutoPipelineOptions / the default. Cached/shared; first call wins.
|
||||
//
|
||||
// EXPERIMENTAL: this API is subject to change, use with caution.
|
||||
func (c *ClusterClient) AutoPipelineWithOptions(config *AutoPipelineOptions) (*AutoPipeliner, error) {
|
||||
return getOrCreateAutoPipeliner(c.autopipelinerMu, &c.autopipeliner, &c.autopipelinerClosed, nil, config,
|
||||
func() *AutoPipelineOptions {
|
||||
if c.opt.AutoPipelineOptions != nil {
|
||||
return c.opt.AutoPipelineOptions
|
||||
}
|
||||
return DefaultBlockingAutoPipelineOptions()
|
||||
},
|
||||
func(cfg *AutoPipelineOptions) (*AutoPipeliner, error) {
|
||||
ap, err := newAutoPipeliner(c, clusterAutoPipelineOptions(cfg), true)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
c.installAutoPipelineSharding(ap)
|
||||
return ap, nil
|
||||
})
|
||||
}
|
||||
|
||||
// installAutoPipelineSharding routes commands to shards by cluster slot so each
|
||||
// shard's batch lands on a single master node, keeping per-node pipelines deep
|
||||
// instead of splitting every batch across all nodes at flush. Cluster slots are
|
||||
// contiguous per node, so bucketing by slot range (slot*shards/16384) keeps a
|
||||
// node's slots together. Keyless commands hash to slot -1 → bucket 0; multi-node
|
||||
// commands are already rejected from pipelines, so only single-node commands
|
||||
// reach here.
|
||||
func (c *ClusterClient) installAutoPipelineSharding(ap *AutoPipeliner) {
|
||||
// Reject commands whose request policy cannot ride a pipeline (ReqAllNodes/
|
||||
// ReqAllShards/ReqMultiShard) at submit, BEFORE they can join a merged
|
||||
// batch: mapCmdsByNode fails a whole mapping on such a command (user
|
||||
// pipelines are all-or-nothing), and one autopipeline caller must not be
|
||||
// able to poison unrelated callers' batches. Rejecting here also keeps the
|
||||
// lone-command fast path consistent with batched dispatch — the command is
|
||||
// refused regardless of what it happens to coalesce with.
|
||||
ap.setPreflight(func(ctx context.Context, cmd Cmder) error {
|
||||
if c.cmdInfoResolver == nil {
|
||||
return nil
|
||||
}
|
||||
if policy := c.cmdInfoResolver.GetCommandPolicy(ctx, cmd); policy != nil && !policy.CanBeUsedInPipeline() {
|
||||
return fmt.Errorf(
|
||||
"redis: cannot pipeline command %q with request policy ReqAllNodes/ReqAllShards/ReqMultiShard; Note: This behavior is subject to change in the future", cmd.Name(),
|
||||
)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
// Commands whose routing is not slot-derived must not be coalesced: a solo
|
||||
// flush reaches ClusterClient.process and its special handling (FT.CURSOR
|
||||
// READ/DEL are sticky to the node holding the cursor), but inside a batch
|
||||
// mapCmdsByNode routes by slot and can hit the wrong shard — visible only
|
||||
// under concurrent traffic, which is the worst way to find it. Divert them
|
||||
// instead of rejecting: they work fine on their own connection (review
|
||||
// finding by codex on #3942).
|
||||
ap.setMustDivert(func(ctx context.Context, cmd Cmder) bool {
|
||||
if c.cmdInfoResolver == nil {
|
||||
return false
|
||||
}
|
||||
policy := c.cmdInfoResolver.GetCommandPolicy(ctx, cmd)
|
||||
return policy != nil && policy.Request == routing.ReqSpecial
|
||||
})
|
||||
|
||||
const slots = 16384
|
||||
n := ap.numShards()
|
||||
ap.setShardFn(func(cmd Cmder) int {
|
||||
// Compute the exact slot once and cache it on the command; the flush
|
||||
// router (mapCmdsByNode) reuses the cached value, so the slot is resolved
|
||||
// once per command, not twice. Keyless (slot -1) buckets to shard 0.
|
||||
slot := c.cmdSlot(cmd, -1)
|
||||
if slot < 0 {
|
||||
return 0
|
||||
}
|
||||
return slot * n / slots
|
||||
})
|
||||
}
|
||||
|
||||
// AsyncAutoPipeline returns the deferred autopipeliner: command calls return
|
||||
// immediately and the result accessors block. Submit a window then read results
|
||||
// for the highest throughput. By default,
|
||||
// ClusterOptions.AutoPipelineOptions is used if set, otherwise
|
||||
// DefaultAutoPipelineOptions. Ordering across nodes is per key: slot routing
|
||||
// keeps a key on one shard, and node sub-pipelines execute concurrently. Use
|
||||
// AsyncAutoPipelineWithOptions to override. Cached/shared; first call's config wins.
|
||||
//
|
||||
// It returns an error if the supplied config is invalid (e.g. MaxConcurrentBatches>1
|
||||
// without Unordered, or a negative size); on error no instance is cached.
|
||||
//
|
||||
// EXPERIMENTAL: this API is subject to change, use with caution.
|
||||
func (c *ClusterClient) AsyncAutoPipeline() (*AutoPipeliner, error) {
|
||||
return c.AsyncAutoPipelineWithOptions(nil)
|
||||
}
|
||||
|
||||
// AsyncAutoPipelineWithOptions is AsyncAutoPipeline with an explicit config
|
||||
// instead of ClusterOptions.AutoPipelineOptions / the default. Cached/shared.
|
||||
//
|
||||
// EXPERIMENTAL: this API is subject to change, use with caution.
|
||||
func (c *ClusterClient) AsyncAutoPipelineWithOptions(config *AutoPipelineOptions) (*AutoPipeliner, error) {
|
||||
return getOrCreateAutoPipeliner(c.autopipelinerMu, &c.asyncAutopipeliner, &c.autopipelinerClosed, nil, config,
|
||||
func() *AutoPipelineOptions {
|
||||
if c.opt.AutoPipelineOptions != nil {
|
||||
return c.opt.AutoPipelineOptions
|
||||
}
|
||||
return DefaultAutoPipelineOptions()
|
||||
},
|
||||
func(cfg *AutoPipelineOptions) (*AutoPipeliner, error) {
|
||||
ap, err := newAutoPipeliner(c, clusterAutoPipelineOptions(cfg), false)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
c.installAutoPipelineSharding(ap)
|
||||
return ap, nil
|
||||
})
|
||||
}
|
||||
|
||||
func (c *ClusterClient) Pipelined(ctx context.Context, fn func(Pipeliner) error) ([]Cmder, error) {
|
||||
return c.Pipeline().Pipelined(ctx, fn)
|
||||
}
|
||||
@@ -1654,9 +1845,18 @@ func (c *ClusterClient) mapCmdsByNode(ctx context.Context, cmdsMap *cmdsMap, cmd
|
||||
policy = c.cmdInfoResolver.GetCommandPolicy(ctx, cmd)
|
||||
}
|
||||
if policy != nil && !policy.CanBeUsedInPipeline() {
|
||||
return fmt.Errorf(
|
||||
// All-or-nothing: a user Pipeline() relies on the whole batch
|
||||
// either dispatching or failing before anything executes, so a
|
||||
// non-pipelineable command fails the entire mapping pre-dispatch.
|
||||
// Autopipeline batches never reach here with such a command: the
|
||||
// cluster face rejects them at submit (see the preflight installed
|
||||
// by installAutoPipelineSharding), so one caller's bad command
|
||||
// cannot poison a merged batch.
|
||||
err := fmt.Errorf(
|
||||
"redis: cannot pipeline command %q with request policy ReqAllNodes/ReqAllShards/ReqMultiShard; Note: This behavior is subject to change in the future", cmd.Name(),
|
||||
)
|
||||
setCmdsErr(cmds, err)
|
||||
return err
|
||||
}
|
||||
slot := c.cmdSlot(cmd, -1)
|
||||
var node *clusterNode
|
||||
@@ -1691,9 +1891,18 @@ func (c *ClusterClient) mapCmdsByNode(ctx context.Context, cmdsMap *cmdsMap, cmd
|
||||
policy = c.cmdInfoResolver.GetCommandPolicy(ctx, cmd)
|
||||
}
|
||||
if policy != nil && !policy.CanBeUsedInPipeline() {
|
||||
return fmt.Errorf(
|
||||
// All-or-nothing: a user Pipeline() relies on the whole batch
|
||||
// either dispatching or failing before anything executes, so a
|
||||
// non-pipelineable command fails the entire mapping pre-dispatch.
|
||||
// Autopipeline batches never reach here with such a command: the
|
||||
// cluster face rejects them at submit (see the preflight installed
|
||||
// by installAutoPipelineSharding), so one caller's bad command
|
||||
// cannot poison a merged batch.
|
||||
err := fmt.Errorf(
|
||||
"redis: cannot pipeline command %q with request policy ReqAllNodes/ReqAllShards/ReqMultiShard; Note: This behavior is subject to change in the future", cmd.Name(),
|
||||
)
|
||||
setCmdsErr(cmds, err)
|
||||
return err
|
||||
}
|
||||
slot := c.cmdSlot(cmd, -1)
|
||||
var node *clusterNode
|
||||
@@ -1728,25 +1937,60 @@ func (c *ClusterClient) cmdsAreReadOnly(ctx context.Context, cmds []Cmder) bool
|
||||
func (c *ClusterClient) processPipelineNode(
|
||||
ctx context.Context, node *clusterNode, cmds []Cmder, failedCmds *cmdsMap,
|
||||
) {
|
||||
_ = node.Client.withProcessPipelineHook(ctx, cmds, func(ctx context.Context, cmds []Cmder) error {
|
||||
cn, err := node.Client.getConn(ctx)
|
||||
if err != nil {
|
||||
// This call runs on a per-node fan-out goroutine, so register it as an
|
||||
// executor of every deferred-face batch among cmds: a NODE-level hook
|
||||
// (OnNewNode — redisotel's tracing) reading a result before next() must
|
||||
// get the not-yet-executed view from the accessor guards instead of
|
||||
// blocking on a batch only this call chain completes (reproduced as a
|
||||
// permanent wedge with a rediscmd-shaped Err() peek).
|
||||
unregister := registerBatchExecutors(cmds)
|
||||
defer unregister()
|
||||
|
||||
// executed guards against a node-level hook short-circuiting (returning
|
||||
// without calling next): the inner callback then never runs, and without
|
||||
// surfacing the chain's error the cluster pipeline would report success
|
||||
// for commands that were never sent.
|
||||
executed := false
|
||||
err := node.Client.withProcessPipelineHook(ctx, cmds, func(ctx context.Context, cmds []Cmder) error {
|
||||
executed = true
|
||||
// Acquire through the node's dedicated pipeline pool when one is
|
||||
// configured (Pipeline*BufferSize propagate to node clients via
|
||||
// clientOptions); withPipelineConn falls back to the main pool
|
||||
// otherwise, preserving the previous behavior. entered distinguishes
|
||||
// an acquisition failure (fn never ran) from an execution error.
|
||||
entered := false
|
||||
err := node.Client.withPipelineConn(ctx, func(ctx context.Context, cn *pool.Conn) error {
|
||||
entered = true
|
||||
return c.processPipelineNodeConn(ctx, node, cn, cmds, failedCmds)
|
||||
})
|
||||
if err != nil && !entered {
|
||||
if !isContextError(err) {
|
||||
node.MarkAsFailing()
|
||||
}
|
||||
_ = c.mapCmdsByNode(ctx, failedCmds, cmds)
|
||||
setCmdsErr(cmds, err)
|
||||
return err
|
||||
}
|
||||
|
||||
var processErr error
|
||||
defer func() {
|
||||
node.Client.releaseConn(ctx, cn, processErr)
|
||||
}()
|
||||
processErr = c.processPipelineNodeConn(ctx, node, cn, cmds, failedCmds)
|
||||
|
||||
return processErr
|
||||
return err
|
||||
})
|
||||
if !executed {
|
||||
// A hook returned without calling next. If it supplied an error that is
|
||||
// a deliberate abort: set it and do not remap for retry (a retry would
|
||||
// re-run the same hook). If it returned nil it short-circuited
|
||||
// SUCCESSFULLY, having served the batch itself — the same thing a plain
|
||||
// Pipeline hook may do — so setCmdsErr(nil) leaves the values it set
|
||||
// intact (review finding by codex on #3942).
|
||||
setCmdsErr(cmds, err)
|
||||
return
|
||||
}
|
||||
if err != nil && cmdsFirstErr(cmds) == nil {
|
||||
// Post-next verdict from a node-level hook on an all-clean sub-batch:
|
||||
// the exec fully succeeded, so the error can only be the hook's own —
|
||||
// apply it, mirroring AutoPipeliner.dispatchCmds. On a mixed batch the
|
||||
// exec-recorded outcomes win (hooks conventionally echo next's error,
|
||||
// and stamping the echo would overwrite successful replies). No remap:
|
||||
// retrying would re-run the same hook.
|
||||
setCmdsErr(cmds, err)
|
||||
}
|
||||
}
|
||||
|
||||
func (c *ClusterClient) processPipelineNodeConn(
|
||||
@@ -1789,7 +2033,7 @@ func (c *ClusterClient) processPipelineNodeConn(
|
||||
setCmdsErr(cmds, err)
|
||||
return err
|
||||
}
|
||||
err := c.pipelineReadCmds(ctx, node, rd, cmds, failedCmds)
|
||||
err := c.pipelineReadCmds(ctx, node, cn, rd, cmds, failedCmds)
|
||||
if err == nil || isRedisError(err) {
|
||||
node.Client.himportAfterBatch(cn, injected, cmds)
|
||||
// SETs of registered fieldsets that lost their session state
|
||||
@@ -1810,11 +2054,22 @@ func (c *ClusterClient) processPipelineNodeConn(
|
||||
func (c *ClusterClient) pipelineReadCmds(
|
||||
ctx context.Context,
|
||||
node *clusterNode,
|
||||
cn *pool.Conn,
|
||||
rd *proto.Reader,
|
||||
cmds []Cmder,
|
||||
failedCmds *cmdsMap,
|
||||
) error {
|
||||
for i, cmd := range cmds {
|
||||
// Drain any buffered RESP3 push notifications before reading each
|
||||
// reply — otherwise a push frame (e.g. a maintnotifications MOVING
|
||||
// notification) is consumed AS the command's reply and every
|
||||
// subsequent reply in the pipeline shifts by one command. The
|
||||
// standalone pipeline and the cluster TxPipeline read loops already
|
||||
// do this; this loop was the only push-blind reader, and the
|
||||
// autopipeliner routes all cluster traffic through it.
|
||||
if err := node.Client.processPendingPushNotificationWithReader(ctx, cn, rd); err != nil {
|
||||
internal.Logger.Printf(ctx, "push: error processing pending notifications before reading reply: %v", err)
|
||||
}
|
||||
err := cmd.readReply(rd)
|
||||
cmd.SetErr(err)
|
||||
|
||||
@@ -1839,7 +2094,8 @@ func (c *ClusterClient) pipelineReadCmds(
|
||||
}
|
||||
}
|
||||
|
||||
if err := cmds[0].Err(); err != nil && shouldRetry(err, true) && !cmdsContainNoRetry(cmds) {
|
||||
// rawErr: execution path; never await an async command's batch here.
|
||||
if err := cmds[0].rawErr(); err != nil && shouldRetry(err, true) && !cmdsContainNoRetry(cmds) {
|
||||
_ = c.mapCmdsByNode(ctx, failedCmds, cmds)
|
||||
return err
|
||||
}
|
||||
@@ -2128,29 +2384,47 @@ func (c *ClusterClient) processTxPipelineNode(
|
||||
}
|
||||
|
||||
var outcome *txOutcome
|
||||
_ = node.Client.withProcessPipelineHook(ctx, wire, func(ctx context.Context, wire []Cmder) error {
|
||||
cn, err := node.Client.getConn(ctx)
|
||||
if err != nil {
|
||||
// executed guards against a node-level hook short-circuiting (returning
|
||||
// without calling next) — same treatment as processPipelineNode.
|
||||
executed := false
|
||||
chainErr := node.Client.withProcessPipelineHook(ctx, wire, func(ctx context.Context, wire []Cmder) error {
|
||||
executed = true
|
||||
// Acquire through the node's dedicated pipeline pool when configured
|
||||
// (same routing as processPipelineNode); withPipelineConn falls back
|
||||
// to the main pool otherwise. The inner fn's return value drives the
|
||||
// connection release exactly like the explicit releaseConn did:
|
||||
// redis errors keep the conn poolable, unread replies poison it.
|
||||
entered := false
|
||||
err := node.Client.withPipelineConn(ctx, func(ctx context.Context, cn *pool.Conn) error {
|
||||
entered = true
|
||||
outcome = c.processTxPipelineNodeConn(ctx, node, cn, wire, cmds, asking)
|
||||
connErr := outcome.err
|
||||
if isRedisError(outcome.err) {
|
||||
connErr = nil
|
||||
}
|
||||
if outcome.unreadReplies {
|
||||
connErr = errTxDirtyConn
|
||||
}
|
||||
return connErr
|
||||
})
|
||||
if !entered && err != nil {
|
||||
// Connection acquisition failed — fn never ran.
|
||||
if shouldRetry(err, true) && !cmdsContainNoRetry(cmds) {
|
||||
outcome = &txOutcome{kind: txRetryConn, err: err}
|
||||
} else {
|
||||
outcome = &txOutcome{kind: txFatal, err: err}
|
||||
}
|
||||
return err
|
||||
}
|
||||
|
||||
outcome = c.processTxPipelineNodeConn(ctx, node, cn, wire, cmds, asking)
|
||||
connErr := outcome.err
|
||||
if isRedisError(outcome.err) {
|
||||
connErr = nil
|
||||
}
|
||||
if outcome.unreadReplies {
|
||||
connErr = errTxDirtyConn
|
||||
}
|
||||
node.Client.releaseConn(ctx, cn, connErr)
|
||||
return connErr
|
||||
return err
|
||||
})
|
||||
|
||||
if !executed && chainErr != nil {
|
||||
// A node-level hook aborted with an error: surface its verdict. A hook
|
||||
// that returned nil short-circuited successfully (it served the batch),
|
||||
// which is legal for plain pipelines too, so it is not turned into a
|
||||
// fatal outcome (review finding by codex on #3942).
|
||||
outcome = &txOutcome{kind: txFatal, err: chainErr}
|
||||
}
|
||||
if outcome == nil {
|
||||
outcome = &txOutcome{kind: txFatal, err: fmt.Errorf("redis: tx pipeline produced no outcome")}
|
||||
}
|
||||
@@ -2597,8 +2871,22 @@ func (c *ClusterClient) cmdInfoPeek(name string) *CommandInfo {
|
||||
}
|
||||
|
||||
func (c *ClusterClient) cmdSlot(cmd Cmder, prefferedSlot int) int {
|
||||
// Serve/populate the per-command slot cache only on the natural-slot path
|
||||
// (prefferedSlot == -1). A forced prefferedSlot (retry re-routing) must not be
|
||||
// cached or served from cache. The cache lets the autopipeline shard router
|
||||
// and the pipeline-flush router (mapCmdsByNode) share one slot computation
|
||||
// instead of each recomputing it.
|
||||
if prefferedSlot == -1 {
|
||||
if slot, ok := cmd.cachedSlot(); ok {
|
||||
return slot
|
||||
}
|
||||
}
|
||||
info := c.cmdInfoPeek(cmd.Name())
|
||||
return c.cmdSlotWithPos(cmd, cmdFirstKeyPosWithInfo(cmd, info), prefferedSlot)
|
||||
slot := c.cmdSlotWithPos(cmd, cmdFirstKeyPosWithInfo(cmd, info), prefferedSlot)
|
||||
if prefferedSlot == -1 && slot >= 0 {
|
||||
cmd.setCachedSlot(slot)
|
||||
}
|
||||
return slot
|
||||
}
|
||||
|
||||
// cmdSlotWithPos computes the cluster slot for cmd given a pre-resolved first key
|
||||
|
||||
@@ -0,0 +1,88 @@
|
||||
package redis_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/redis/go-redis/v9"
|
||||
)
|
||||
|
||||
type nodePeekHook struct{ armed *atomic.Bool }
|
||||
|
||||
func (h nodePeekHook) DialHook(next redis.DialHook) redis.DialHook { return next }
|
||||
func (h nodePeekHook) ProcessHook(next redis.ProcessHook) redis.ProcessHook { return next }
|
||||
func (h nodePeekHook) ProcessPipelineHook(next redis.ProcessPipelineHook) redis.ProcessPipelineHook {
|
||||
return func(ctx context.Context, cmds []redis.Cmder) error {
|
||||
if h.armed.Load() {
|
||||
for _, c := range cmds {
|
||||
_ = c.Err() // redisotel via rediscmd.CmdsString does exactly this pre-next
|
||||
_ = c.String()
|
||||
}
|
||||
}
|
||||
err := next(ctx, cmds)
|
||||
if h.armed.Load() {
|
||||
for _, c := range cmds {
|
||||
_ = c.Err()
|
||||
_ = c.String()
|
||||
}
|
||||
}
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
// TestClusterNodeHookResultPeekNoDeadlock pins the batch-executor guard for
|
||||
// cluster per-node goroutines: redisotel installs tracing hooks on NODE
|
||||
// clients via OnNewNode, and rediscmd.CmdsString reads every command's Err()
|
||||
// BEFORE next() — on the deferred face those reads used to block on a batch
|
||||
// that completes only after the node call returns, wedging the dispatch
|
||||
// permanently (found while assessing the String review round on #3942).
|
||||
func TestClusterNodeHookResultPeekNoDeadlock(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
probe := redis.NewClusterClient(&redis.ClusterOptions{Addrs: []string{":16600", ":16601", ":16602"}})
|
||||
if err := probe.Ping(ctx).Err(); err != nil {
|
||||
probe.Close()
|
||||
t.Skipf("no cluster: %v", err)
|
||||
}
|
||||
probe.Close()
|
||||
|
||||
cc := redis.NewClusterClient(&redis.ClusterOptions{Addrs: []string{":16600", ":16601", ":16602"}})
|
||||
defer cc.Close()
|
||||
var armed atomic.Bool
|
||||
var installs atomic.Int32
|
||||
cc.OnNewNode(func(nc *redis.Client) {
|
||||
installs.Add(1)
|
||||
nc.AddHook(nodePeekHook{armed: &armed})
|
||||
})
|
||||
cc.Ping(ctx)
|
||||
t.Logf("node hooks installed: %d", installs.Load())
|
||||
|
||||
ap, err := cc.AsyncAutoPipeline()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
done := make(chan error, 1)
|
||||
go func() {
|
||||
armed.Store(true)
|
||||
defer armed.Store(false)
|
||||
cmds := make([]*redis.StatusCmd, 50)
|
||||
for i := range cmds {
|
||||
cmds[i] = ap.Set(ctx, "nodehook:"+string(rune('a'+i%26))+":"+string(rune('0'+i%10)), "v", 0)
|
||||
}
|
||||
var firstErr error
|
||||
for _, c := range cmds {
|
||||
if err := c.Err(); err != nil && firstErr == nil {
|
||||
firstErr = err
|
||||
}
|
||||
}
|
||||
done <- firstErr
|
||||
}()
|
||||
select {
|
||||
case err := <-done:
|
||||
t.Logf("no hang, err=%v", err)
|
||||
case <-time.After(10 * time.Second):
|
||||
t.Fatal("node-level hook result peek wedged the async cluster dispatch (batch executor registration regressed)")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,248 @@
|
||||
package redis_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
|
||||
"github.com/redis/go-redis/v9"
|
||||
)
|
||||
|
||||
// skipWithoutRedis skips the test when no server answers at apTestAddr(), so a
|
||||
// unit-only run doesn't turn missing integration infrastructure into a
|
||||
// failure. It probes with a DEFAULT-options client, not the test's configured
|
||||
// one: probing with the client under test would convert a regression in the
|
||||
// very options being exercised (buffer sizes, pipeline pool) into a silent
|
||||
// skip while the server is up.
|
||||
func skipWithoutRedis(t *testing.T, ctx context.Context, _ *redis.Client) {
|
||||
t.Helper()
|
||||
probe := redis.NewClient(&redis.Options{Addr: apTestAddr()})
|
||||
defer probe.Close()
|
||||
if err := probe.Ping(ctx).Err(); err != nil {
|
||||
t.Skipf("no redis at %s: %v", apTestAddr(), err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestPipelineBufferSizes verifies that enabling the pipeline buffer options keeps
|
||||
// pipelining working end-to-end. It does not assert the actual socket buffer sizes
|
||||
// (not observable here) - only that the configured client still runs pipelines.
|
||||
func TestPipelineBufferSizes(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
|
||||
// Create client with custom pipeline buffer sizes
|
||||
client := redis.NewClient(&redis.Options{
|
||||
Addr: apTestAddr(),
|
||||
ReadBufferSize: 64 * 1024, // 64 KiB for regular connections
|
||||
WriteBufferSize: 64 * 1024, // 64 KiB for regular connections
|
||||
PipelineReadBufferSize: 512 * 1024, // 512 KiB for pipeline connections
|
||||
PipelineWriteBufferSize: 512 * 1024, // 512 KiB for pipeline connections
|
||||
PipelinePoolSize: 5, // Small pool for pipelining
|
||||
})
|
||||
defer client.Close()
|
||||
skipWithoutRedis(t, ctx, client)
|
||||
cleanupBufferTestKeys(t, client)
|
||||
|
||||
// Test that regular commands work
|
||||
err := client.Set(ctx, "test_key", "test_value", 0).Err()
|
||||
if err != nil {
|
||||
t.Fatalf("Failed to set key: %v", err)
|
||||
}
|
||||
|
||||
val, err := client.Get(ctx, "test_key").Result()
|
||||
if err != nil {
|
||||
t.Fatalf("Failed to get key: %v", err)
|
||||
}
|
||||
if val != "test_value" {
|
||||
t.Fatalf("Expected 'test_value', got '%s'", val)
|
||||
}
|
||||
|
||||
// Test that pipeline works
|
||||
pipe := client.Pipeline()
|
||||
pipe.Set(ctx, "pipe_key1", "value1", 0)
|
||||
pipe.Set(ctx, "pipe_key2", "value2", 0)
|
||||
pipe.Get(ctx, "pipe_key1")
|
||||
pipe.Get(ctx, "pipe_key2")
|
||||
|
||||
cmds, err := pipe.Exec(ctx)
|
||||
if err != nil {
|
||||
t.Fatalf("Pipeline execution failed: %v", err)
|
||||
}
|
||||
|
||||
if len(cmds) != 4 {
|
||||
t.Fatalf("Expected 4 commands, got %d", len(cmds))
|
||||
}
|
||||
|
||||
// Verify results
|
||||
if cmds[2].(*redis.StringCmd).Val() != "value1" {
|
||||
t.Fatalf("Expected 'value1', got '%s'", cmds[2].(*redis.StringCmd).Val())
|
||||
}
|
||||
if cmds[3].(*redis.StringCmd).Val() != "value2" {
|
||||
t.Fatalf("Expected 'value2', got '%s'", cmds[3].(*redis.StringCmd).Val())
|
||||
}
|
||||
|
||||
t.Log("Pipeline with custom buffer sizes works correctly")
|
||||
}
|
||||
|
||||
// TestNoPipelinePool verifies that client works without pipeline pool (backward compatibility)
|
||||
func TestNoPipelinePool(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
|
||||
// Create client WITHOUT custom pipeline buffer sizes
|
||||
client := redis.NewClient(&redis.Options{
|
||||
Addr: apTestAddr(),
|
||||
ReadBufferSize: 64 * 1024, // 64 KiB for all connections
|
||||
WriteBufferSize: 64 * 1024, // 64 KiB for all connections
|
||||
// No PipelineReadBufferSize or PipelineWriteBufferSize
|
||||
})
|
||||
defer client.Close()
|
||||
skipWithoutRedis(t, ctx, client)
|
||||
cleanupBufferTestKeys(t, client)
|
||||
|
||||
// Test that pipeline still works (using regular pool)
|
||||
pipe := client.Pipeline()
|
||||
pipe.Set(ctx, "no_pipe_pool_key1", "value1", 0)
|
||||
pipe.Set(ctx, "no_pipe_pool_key2", "value2", 0)
|
||||
pipe.Get(ctx, "no_pipe_pool_key1")
|
||||
pipe.Get(ctx, "no_pipe_pool_key2")
|
||||
|
||||
cmds, err := pipe.Exec(ctx)
|
||||
if err != nil {
|
||||
t.Fatalf("Pipeline execution failed: %v", err)
|
||||
}
|
||||
|
||||
if len(cmds) != 4 {
|
||||
t.Fatalf("Expected 4 commands, got %d", len(cmds))
|
||||
}
|
||||
|
||||
// Verify results
|
||||
if cmds[2].(*redis.StringCmd).Val() != "value1" {
|
||||
t.Fatalf("Expected 'value1', got '%s'", cmds[2].(*redis.StringCmd).Val())
|
||||
}
|
||||
if cmds[3].(*redis.StringCmd).Val() != "value2" {
|
||||
t.Fatalf("Expected 'value2', got '%s'", cmds[3].(*redis.StringCmd).Val())
|
||||
}
|
||||
|
||||
t.Log("Pipeline without custom buffer sizes (backward compatibility) works correctly")
|
||||
}
|
||||
|
||||
// TestPipelinePoolStats verifies that PoolStats includes pipeline pool stats
|
||||
func TestPipelinePoolStats(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
|
||||
// Create client with custom pipeline buffer sizes
|
||||
client := redis.NewClient(&redis.Options{
|
||||
Addr: apTestAddr(),
|
||||
ReadBufferSize: 64 * 1024, // 64 KiB for regular connections
|
||||
WriteBufferSize: 64 * 1024, // 64 KiB for regular connections
|
||||
PipelineReadBufferSize: 512 * 1024, // 512 KiB for pipeline connections
|
||||
PipelineWriteBufferSize: 512 * 1024, // 512 KiB for pipeline connections
|
||||
PipelinePoolSize: 5, // Small pool for pipelining
|
||||
})
|
||||
defer client.Close()
|
||||
skipWithoutRedis(t, ctx, client)
|
||||
cleanupBufferTestKeys(t, client)
|
||||
|
||||
// Execute some pipeline commands
|
||||
pipe := client.Pipeline()
|
||||
for i := 0; i < 10; i++ {
|
||||
pipe.Set(ctx, "stats_key", "value", 0)
|
||||
}
|
||||
_, err := pipe.Exec(ctx)
|
||||
if err != nil {
|
||||
t.Fatalf("Pipeline execution failed: %v", err)
|
||||
}
|
||||
|
||||
// Get pool stats
|
||||
stats := client.PoolStats()
|
||||
if stats == nil {
|
||||
t.Fatal("PoolStats returned nil")
|
||||
}
|
||||
|
||||
// Verify pipeline stats are included
|
||||
if stats.PipelineStats == nil {
|
||||
t.Fatal("PipelineStats is nil - pipeline pool stats not included")
|
||||
}
|
||||
|
||||
t.Logf("Regular pool stats: TotalConns=%d, IdleConns=%d, Hits=%d, Misses=%d",
|
||||
stats.TotalConns, stats.IdleConns, stats.Hits, stats.Misses)
|
||||
t.Logf("Pipeline pool stats: TotalConns=%d, IdleConns=%d, Hits=%d, Misses=%d",
|
||||
stats.PipelineStats.TotalConns, stats.PipelineStats.IdleConns,
|
||||
stats.PipelineStats.Hits, stats.PipelineStats.Misses)
|
||||
|
||||
// Verify pipeline pool has connections
|
||||
if stats.PipelineStats.TotalConns == 0 {
|
||||
t.Error("Pipeline pool has no connections")
|
||||
}
|
||||
|
||||
t.Log("PoolStats includes pipeline pool stats correctly")
|
||||
}
|
||||
|
||||
// TestNoPipelinePoolStats verifies that PoolStats works without pipeline pool
|
||||
func TestNoPipelinePoolStats(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
|
||||
// Create client WITHOUT custom pipeline buffer sizes
|
||||
client := redis.NewClient(&redis.Options{
|
||||
Addr: apTestAddr(),
|
||||
ReadBufferSize: 64 * 1024, // 64 KiB for all connections
|
||||
WriteBufferSize: 64 * 1024, // 64 KiB for all connections
|
||||
})
|
||||
defer client.Close()
|
||||
skipWithoutRedis(t, ctx, client)
|
||||
cleanupBufferTestKeys(t, client)
|
||||
|
||||
// Execute some commands
|
||||
err := client.Set(ctx, "test_key", "test_value", 0).Err()
|
||||
if err != nil {
|
||||
t.Fatalf("Failed to set key: %v", err)
|
||||
}
|
||||
|
||||
// Get pool stats
|
||||
stats := client.PoolStats()
|
||||
if stats == nil {
|
||||
t.Fatal("PoolStats returned nil")
|
||||
}
|
||||
|
||||
// Verify pipeline stats are nil (no pipeline pool)
|
||||
if stats.PipelineStats != nil {
|
||||
t.Error("PipelineStats should be nil when no pipeline pool is configured")
|
||||
}
|
||||
|
||||
t.Logf("Regular pool stats: TotalConns=%d, IdleConns=%d, Hits=%d, Misses=%d",
|
||||
stats.TotalConns, stats.IdleConns, stats.Hits, stats.Misses)
|
||||
|
||||
t.Log("PoolStats works correctly without pipeline pool")
|
||||
}
|
||||
|
||||
// cleanupBufferTestKeys removes this file's fixed test keys before and after a
|
||||
// test. These tests share a Redis instance with the rest of the package (and
|
||||
// "test_key" is also used by autopipeline_test.go), so leaving state behind
|
||||
// makes results order-dependent — the same class of leak that produced a real
|
||||
// CI flake in the search suite (Copilot review on #3942).
|
||||
//
|
||||
// Call it AFTER skipWithoutRedis: it talks to the server, so running it first
|
||||
// turns a no-Redis environment into a hard failure instead of the intended
|
||||
// skip (also Copilot, on the first cut of this cleanup).
|
||||
func cleanupBufferTestKeys(t *testing.T, client *redis.Client) {
|
||||
t.Helper()
|
||||
keys := []string{
|
||||
"test_key", "stats_key",
|
||||
"pipe_key1", "pipe_key2",
|
||||
"no_pipe_pool_key1", "no_pipe_pool_key2",
|
||||
}
|
||||
ctx := context.Background()
|
||||
if err := client.Del(ctx, keys...).Err(); err != nil {
|
||||
t.Fatalf("cleanup keys: %v", err)
|
||||
}
|
||||
// The post-test sweep gets its OWN client: t.Cleanup runs after the test
|
||||
// function's defers, and every caller here defers client.Close(), so a
|
||||
// cleanup issued on the passed-in client would land on a closed one and
|
||||
// silently do nothing — leaving exactly the keys this helper exists to
|
||||
// remove (Copilot review on #3942).
|
||||
t.Cleanup(func() {
|
||||
c := redis.NewClient(&redis.Options{Addr: apTestAddr()})
|
||||
defer c.Close()
|
||||
if err := c.Del(context.Background(), keys...).Err(); err != nil {
|
||||
t.Errorf("post-test key cleanup failed, later tests may see stale keys: %v", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
@@ -0,0 +1,358 @@
|
||||
package redis_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
"net"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/redis/go-redis/v9"
|
||||
)
|
||||
|
||||
// Tests for pipeline execution behavior shared with the autopipeliner:
|
||||
// heterogeneous reply-shape demultiplexing and the retry/re-drive loop.
|
||||
|
||||
// ===== from pipeline_reply_shapes_test.go =====
|
||||
|
||||
// Reply-shape demux: a single batch mixing heterogeneous reply types must keep
|
||||
// the reader in sync so every command gets ITS OWN correctly-typed reply. Prior
|
||||
// correctness tests batched only homogeneous simple commands (SET/GET/INCR),
|
||||
// which proves key/value pairing but not that the parser stays aligned across
|
||||
// bulk + array + int + map + nil + double + large(multi-read) + binary-safe
|
||||
// values. One demux slip corrupts every later reply in the batch. AutoPipeline
|
||||
// dispatches through the same execer, so it inherits this.
|
||||
|
||||
func seedReplyShapeData(t *testing.T, ctx context.Context, c *redis.Client) (big, binary string) {
|
||||
t.Helper()
|
||||
big = strings.Repeat("A", 100_000) // > bufio buffer -> multi-read
|
||||
binary = "x\x00y\r\nz\x01\xffend" // embedded NUL, CRLF, high bytes
|
||||
c.Del(ctx, "rs:list", "rs:hash", "rs:big", "rs:bin", "rs:s", "rs:n", "rs:f")
|
||||
if err := c.RPush(ctx, "rs:list", "a", "b", "c").Err(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := c.HSet(ctx, "rs:hash", "f1", "v1", "f2", "v2").Err(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := c.Set(ctx, "rs:big", big, 0).Err(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := c.Set(ctx, "rs:bin", binary, 0).Err(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return big, binary
|
||||
}
|
||||
|
||||
func assertReplyShapes(t *testing.T, cSet *redis.StatusCmd, cGet *redis.StringCmd,
|
||||
cIncr *redis.IntCmd, cArr *redis.StringSliceCmd, cMap *redis.MapStringStringCmd,
|
||||
cNil *redis.StringCmd, cBig, cBin *redis.StringCmd, cFloat *redis.FloatCmd,
|
||||
big, binary string) {
|
||||
t.Helper()
|
||||
if cSet.Val() != "OK" {
|
||||
t.Errorf("status: %q, want OK", cSet.Val())
|
||||
}
|
||||
if cGet.Val() != "v" {
|
||||
t.Errorf("bulk: %q, want v", cGet.Val())
|
||||
}
|
||||
if cIncr.Val() != 1 {
|
||||
t.Errorf("int: %d, want 1", cIncr.Val())
|
||||
}
|
||||
if got := cArr.Val(); len(got) != 3 || got[0] != "a" || got[2] != "c" {
|
||||
t.Errorf("array: %v, want [a b c]", got)
|
||||
}
|
||||
if got := cMap.Val(); got["f1"] != "v1" || got["f2"] != "v2" {
|
||||
t.Errorf("map: %v", got)
|
||||
}
|
||||
if cNil.Err() != redis.Nil {
|
||||
t.Errorf("nil: err=%v, want redis.Nil", cNil.Err())
|
||||
}
|
||||
if cBig.Val() != big {
|
||||
t.Errorf("large: len=%d, want %d", len(cBig.Val()), len(big))
|
||||
}
|
||||
if cBin.Val() != binary {
|
||||
t.Errorf("binary-safe: %q, want %q", cBin.Val(), binary)
|
||||
}
|
||||
if cFloat.Val() != 1.5 {
|
||||
t.Errorf("double: %v, want 1.5", cFloat.Val())
|
||||
}
|
||||
}
|
||||
|
||||
func TestPipelineHeterogeneousReplyDemux(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
c := redis.NewClient(&redis.Options{Addr: apTestAddr()})
|
||||
defer c.Close()
|
||||
if err := c.Ping(ctx).Err(); err != nil {
|
||||
t.Skipf("no redis: %v", err)
|
||||
}
|
||||
big, binary := seedReplyShapeData(t, ctx, c)
|
||||
|
||||
pipe := c.Pipeline()
|
||||
cSet := pipe.Set(ctx, "rs:s", "v", 0)
|
||||
cGet := pipe.Get(ctx, "rs:s")
|
||||
cIncr := pipe.Incr(ctx, "rs:n")
|
||||
cArr := pipe.LRange(ctx, "rs:list", 0, -1)
|
||||
cMap := pipe.HGetAll(ctx, "rs:hash")
|
||||
cNil := pipe.Get(ctx, "rs:missing")
|
||||
cBig := pipe.Get(ctx, "rs:big")
|
||||
cBin := pipe.Get(ctx, "rs:bin")
|
||||
cFloat := pipe.IncrByFloat(ctx, "rs:f", 1.5)
|
||||
_, _ = pipe.Exec(ctx) // Exec returns the first error (the redis.Nil); check per-cmd
|
||||
|
||||
assertReplyShapes(t, cSet, cGet, cIncr, cArr, cMap, cNil, cBig, cBin, cFloat, big, binary)
|
||||
}
|
||||
|
||||
// TestPipelineHeterogeneousReplyDemuxRESP2 runs the same demux under RESP2, which
|
||||
// parses maps/doubles/verbatim differently on the wire — the reader must still
|
||||
// stay aligned across the mixed batch.
|
||||
func TestPipelineHeterogeneousReplyDemuxRESP2(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
c := redis.NewClient(&redis.Options{Addr: apTestAddr(), Protocol: 2})
|
||||
defer c.Close()
|
||||
if err := c.Ping(ctx).Err(); err != nil {
|
||||
t.Skipf("no redis: %v", err)
|
||||
}
|
||||
big, binary := seedReplyShapeData(t, ctx, c)
|
||||
|
||||
pipe := c.Pipeline()
|
||||
cSet := pipe.Set(ctx, "rs:s", "v", 0)
|
||||
cGet := pipe.Get(ctx, "rs:s")
|
||||
cIncr := pipe.Incr(ctx, "rs:n")
|
||||
cArr := pipe.LRange(ctx, "rs:list", 0, -1)
|
||||
cMap := pipe.HGetAll(ctx, "rs:hash")
|
||||
cNil := pipe.Get(ctx, "rs:missing")
|
||||
cBig := pipe.Get(ctx, "rs:big")
|
||||
cBin := pipe.Get(ctx, "rs:bin")
|
||||
cFloat := pipe.IncrByFloat(ctx, "rs:f", 1.5)
|
||||
_, _ = pipe.Exec(ctx)
|
||||
|
||||
assertReplyShapes(t, cSet, cGet, cIncr, cArr, cMap, cNil, cBig, cBin, cFloat, big, binary)
|
||||
}
|
||||
|
||||
func TestAutoPipelineHeterogeneousReplyDemux(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
c := redis.NewClient(&redis.Options{Addr: apTestAddr()})
|
||||
defer c.Close()
|
||||
if err := c.Ping(ctx).Err(); err != nil {
|
||||
t.Skipf("no redis: %v", err)
|
||||
}
|
||||
big, binary := seedReplyShapeData(t, ctx, c)
|
||||
|
||||
// Async face with a wide flush window: the 9 heterogeneous commands are
|
||||
// submitted without blocking and land in ONE pipeline batch, so this
|
||||
// actually exercises batch reply demux (the blocking face submits them
|
||||
// one at a time through the lone-command fast path, which never demuxes).
|
||||
ap, err := c.AsyncAutoPipelineWithOptions(&redis.AutoPipelineOptions{
|
||||
MaxBatchSize: 100,
|
||||
MaxFlushDelay: 50 * time.Millisecond,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer ap.Close()
|
||||
|
||||
cSet := ap.Set(ctx, "rs:s", "v", 0)
|
||||
cGet := ap.Get(ctx, "rs:s")
|
||||
cIncr := ap.Incr(ctx, "rs:n")
|
||||
cArr := ap.LRange(ctx, "rs:list", 0, -1)
|
||||
cMap := ap.HGetAll(ctx, "rs:hash")
|
||||
cNil := ap.Get(ctx, "rs:missing")
|
||||
cBig := ap.Get(ctx, "rs:big")
|
||||
cBin := ap.Get(ctx, "rs:bin")
|
||||
cFloat := ap.IncrByFloat(ctx, "rs:f", 1.5)
|
||||
|
||||
assertReplyShapes(t, cSet, cGet, cIncr, cArr, cMap, cNil, cBig, cBin, cFloat, big, binary)
|
||||
}
|
||||
|
||||
// ===== from pipeline_retry_test.go =====
|
||||
|
||||
// These tests exercise the pipeline retry/re-drive loop in generalProcessPipeline
|
||||
// (redis.go), which was previously untested: the CI harness runs with
|
||||
// MaxRetries=-1 (-> 0), so the retry loop never iterated. AutoPipeline dispatches
|
||||
// through this same execer, so it inherits whatever is proven here
|
||||
// (see autopipeline_retry_test.go for the AutoPipeline-specific coverage).
|
||||
|
||||
// TestPipelineRetriesOnNetworkError arms the pooled conn's next write to fail
|
||||
// at the wire (a seeded broken idle conn would be filtered by the pool health
|
||||
// check and never carry an attempt); the first pipeline attempt genuinely
|
||||
// dies, and the retry redials a healthy conn and re-drives the whole batch.
|
||||
// Verifies the replies are correct and in order after the re-drive.
|
||||
func TestPipelineRetriesOnNetworkError(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
var failNextWrite atomic.Bool
|
||||
var dials atomic.Int32
|
||||
client := redis.NewClient(&redis.Options{
|
||||
Addr: apTestAddr(), MaxRetries: 2, PoolSize: 1,
|
||||
Dialer: func(ctx context.Context, network, addr string) (net.Conn, error) {
|
||||
dials.Add(1)
|
||||
cn, err := (&net.Dialer{}).DialContext(ctx, network, addr)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &flakyWriteConn{Conn: cn, fail: &failNextWrite}, nil
|
||||
},
|
||||
})
|
||||
defer client.Close()
|
||||
if err := client.Ping(ctx).Err(); err != nil {
|
||||
t.Skipf("no redis: %v", err)
|
||||
}
|
||||
if err := client.Set(ctx, "pr:n", 0, 0).Err(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
dialsBefore := dials.Load()
|
||||
failNextWrite.Store(true)
|
||||
|
||||
pipe := client.Pipeline()
|
||||
set := pipe.Set(ctx, "pr:k", "v", 0)
|
||||
get := pipe.Get(ctx, "pr:k")
|
||||
incr := pipe.Incr(ctx, "pr:n")
|
||||
if _, err := pipe.Exec(ctx); err != nil {
|
||||
t.Fatalf("pipe.Exec after retry: %v", err)
|
||||
}
|
||||
if set.Val() != "OK" {
|
||||
t.Fatalf("set = %q, want OK", set.Val())
|
||||
}
|
||||
if get.Val() != "v" {
|
||||
t.Fatalf("get = %q, want v", get.Val())
|
||||
}
|
||||
if incr.Val() != 1 {
|
||||
t.Fatalf("incr = %d, want 1", incr.Val())
|
||||
}
|
||||
if failNextWrite.Load() {
|
||||
t.Fatal("armed write failure never consumed: the first attempt did not reach the wire")
|
||||
}
|
||||
if got := dials.Load(); got != dialsBefore+1 {
|
||||
t.Fatalf("dials after failure = %d, want %d (exactly one redial by the retry)", got, dialsBefore+1)
|
||||
}
|
||||
}
|
||||
|
||||
// TestPipelineRetryExhausted makes every dial hand back a broken conn, so all
|
||||
// MaxRetries+1 attempts fail. Verifies the pipeline returns an error (does not
|
||||
// hang), every command carries the error, and the loop ran the expected number
|
||||
// of attempts.
|
||||
func TestPipelineRetryExhausted(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
var dials int32
|
||||
client := redis.NewClient(&redis.Options{
|
||||
Addr: apTestAddr(),
|
||||
MaxRetries: 2, // -> 3 attempts total
|
||||
PoolSize: 1,
|
||||
Dialer: func(ctx context.Context, network, addr string) (net.Conn, error) {
|
||||
atomic.AddInt32(&dials, 1)
|
||||
return &badConn{writeErr: io.EOF}, nil // dial "succeeds", all I/O fails
|
||||
},
|
||||
})
|
||||
defer client.Close()
|
||||
|
||||
pipe := client.Pipeline()
|
||||
_ = pipe.Set(ctx, "k", "v", 0)
|
||||
_ = pipe.Get(ctx, "k")
|
||||
_, err := pipe.Exec(ctx)
|
||||
if err == nil {
|
||||
t.Fatal("expected error after retry exhaustion, got nil")
|
||||
}
|
||||
// MaxRetries=2 -> 3 attempts, each dials a fresh (bad) conn.
|
||||
if got := atomic.LoadInt32(&dials); got < 3 {
|
||||
t.Fatalf("dials = %d, want >= 3 (MaxRetries+1 attempts)", got)
|
||||
}
|
||||
// NOTE: manual Pipeline surfaces exhaustion via Exec's returned error. When
|
||||
// the failure is a conn-init failure (conn never runs pipelineProcessCmds),
|
||||
// the individual command objects are NOT populated with the error — see
|
||||
// TestAutoPipelineRetryExhaustionSurfacesError for why that matters for
|
||||
// AutoPipeline, which has no Exec return value to surface it.
|
||||
}
|
||||
|
||||
// TestPipelineWrongTypeNotRetriedIsolated verifies a server -ERR (WRONGTYPE) in
|
||||
// the middle of a pipeline is isolated to its own command — siblings still get
|
||||
// their correct replies — and is not retried (redis errors are non-retryable).
|
||||
func TestPipelineWrongTypeNotRetriedIsolated(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
client := redis.NewClient(&redis.Options{Addr: apTestAddr(), MaxRetries: 3})
|
||||
defer client.Close()
|
||||
if err := client.Ping(ctx).Err(); err != nil {
|
||||
t.Skipf("no redis: %v", err)
|
||||
}
|
||||
client.Del(ctx, "wt:list", "wt:k")
|
||||
if err := client.LPush(ctx, "wt:list", "x").Err(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
pipe := client.Pipeline()
|
||||
good1 := pipe.Set(ctx, "wt:k", "v", 0)
|
||||
bad := pipe.Incr(ctx, "wt:list") // WRONGTYPE against a list
|
||||
good2 := pipe.Get(ctx, "wt:k")
|
||||
_, _ = pipe.Exec(ctx) // Exec returns the first command error (the WRONGTYPE)
|
||||
|
||||
if bad.Err() == nil {
|
||||
t.Fatal("expected WRONGTYPE on the bad command")
|
||||
}
|
||||
if good1.Err() != nil {
|
||||
t.Fatalf("good1 inherited an error: %v", good1.Err())
|
||||
}
|
||||
if good2.Err() != nil || good2.Val() != "v" {
|
||||
t.Fatalf("good2 = %q, %v; want v, nil", good2.Val(), good2.Err())
|
||||
}
|
||||
}
|
||||
|
||||
// TestPipelineContextCanceledNotRetried verifies a cancelled context aborts the
|
||||
// pipeline without retrying (context errors are non-retryable) and every command
|
||||
// carries the context error.
|
||||
func TestPipelineContextCanceledNotRetried(t *testing.T) {
|
||||
client := redis.NewClient(&redis.Options{Addr: apTestAddr(), MaxRetries: 3})
|
||||
defer client.Close()
|
||||
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
cancel() // pre-cancelled
|
||||
|
||||
pipe := client.Pipeline()
|
||||
c := pipe.Get(ctx, "ctx:k")
|
||||
_, err := pipe.Exec(ctx)
|
||||
if err == nil {
|
||||
t.Fatal("expected context error, got nil")
|
||||
}
|
||||
if !errors.Is(err, context.Canceled) {
|
||||
t.Fatalf("err = %v, want context.Canceled", err)
|
||||
}
|
||||
if !errors.Is(c.Err(), context.Canceled) {
|
||||
t.Fatalf("cmd err = %v, want context.Canceled", c.Err())
|
||||
}
|
||||
}
|
||||
|
||||
// TestAutoPipelineHeterogeneousReplyDemuxRESP2Batch is the RESP2 twin of the
|
||||
// batched demux test above: the same nine heterogeneous replies land in one
|
||||
// real async batch over Protocol 2, pinning that batch reply demultiplexing
|
||||
// holds on both protocol encodings (the RESP2 variant was previously only
|
||||
// inferred from a comment, never asserted through a real batch).
|
||||
func TestAutoPipelineHeterogeneousReplyDemuxRESP2Batch(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
c := redis.NewClient(&redis.Options{Addr: apTestAddr(), Protocol: 2})
|
||||
defer c.Close()
|
||||
if err := c.Ping(ctx).Err(); err != nil {
|
||||
t.Skipf("no redis: %v", err)
|
||||
}
|
||||
big, binary := seedReplyShapeData(t, ctx, c)
|
||||
|
||||
ap, err := c.AsyncAutoPipelineWithOptions(&redis.AutoPipelineOptions{
|
||||
MaxBatchSize: 100,
|
||||
MaxFlushDelay: 50 * time.Millisecond,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer ap.Close()
|
||||
|
||||
cSet := ap.Set(ctx, "rs:s", "v", 0)
|
||||
cGet := ap.Get(ctx, "rs:s")
|
||||
cIncr := ap.Incr(ctx, "rs:n")
|
||||
cArr := ap.LRange(ctx, "rs:list", 0, -1)
|
||||
cMap := ap.HGetAll(ctx, "rs:hash")
|
||||
cNil := ap.Get(ctx, "rs:missing")
|
||||
cBig := ap.Get(ctx, "rs:big")
|
||||
cBin := ap.Get(ctx, "rs:bin")
|
||||
cFloat := ap.IncrByFloat(ctx, "rs:f", 1.5)
|
||||
|
||||
assertReplyShapes(t, cSet, cGet, cIncr, cArr, cMap, cNil, cBig, cBin, cFloat, big, binary)
|
||||
}
|
||||
@@ -0,0 +1,223 @@
|
||||
package redis_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/redis/go-redis/v9"
|
||||
"github.com/redis/go-redis/v9/auth"
|
||||
"github.com/redis/go-redis/v9/maintnotifications"
|
||||
)
|
||||
|
||||
// Tests for the DEDICATED pipeline connection pool (enabled by
|
||||
// PipelineReadBufferSize/PipelineWriteBufferSize): handshake parity,
|
||||
// maintnotifications wiring, and streaming-credentials re-auth on
|
||||
// pipeline-pool connections.
|
||||
|
||||
// ===== from pipeline_pool_handshake_test.go =====
|
||||
|
||||
// TestPipelinePoolHandshakeACLAndSelect verifies that connections in the
|
||||
// DEDICATED pipeline pool (enabled by PipelineReadBufferSize/WriteBufferSize)
|
||||
// perform the full handshake — ACL AUTH (username+password) and SELECT DB — the
|
||||
// same as normal-pool conns. A pipeline and an autopipeline both run on those
|
||||
// conns; if the handshake were skipped they would get NOAUTH or land in the
|
||||
// wrong DB.
|
||||
func TestPipelinePoolHandshakeACLAndSelect(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
admin := redis.NewClient(&redis.Options{Addr: apTestAddr()})
|
||||
defer admin.Close()
|
||||
if err := admin.Ping(ctx).Err(); err != nil {
|
||||
t.Skipf("no redis: %v", err)
|
||||
}
|
||||
|
||||
const user, pw, db = "ap_pipe_user", "ap_pipe_pw", 3
|
||||
_ = admin.Do(ctx, "ACL", "DELUSER", user).Err()
|
||||
if err := admin.Do(ctx, "ACL", "SETUSER", user, "on", ">"+pw, "~*", "+@all").Err(); err != nil {
|
||||
t.Skipf("ACL unsupported: %v", err)
|
||||
}
|
||||
defer admin.Do(ctx, "ACL", "DELUSER", user)
|
||||
|
||||
c := redis.NewClient(&redis.Options{
|
||||
Addr: apTestAddr(),
|
||||
Username: user,
|
||||
Password: pw,
|
||||
DB: db,
|
||||
PipelineReadBufferSize: 64 << 10,
|
||||
PipelineWriteBufferSize: 64 << 10,
|
||||
PipelinePoolSize: 3,
|
||||
})
|
||||
defer c.Close()
|
||||
|
||||
// Manual pipeline -> pipeline pool. Must AUTH (ACL) + SELECT db.
|
||||
pipe := c.Pipeline()
|
||||
pipe.Set(ctx, "hs:pipe", "v1", 0)
|
||||
g1 := pipe.Get(ctx, "hs:pipe")
|
||||
if _, err := pipe.Exec(ctx); err != nil {
|
||||
t.Fatalf("pipeline on ACL+DB pipeline pool: %v", err)
|
||||
}
|
||||
if g1.Val() != "v1" {
|
||||
t.Fatalf("pipeline get = %q, want v1", g1.Val())
|
||||
}
|
||||
if st := c.PoolStats(); st.PipelineStats == nil {
|
||||
t.Fatal("expected a dedicated pipeline pool (PipelineStats nil)")
|
||||
}
|
||||
|
||||
// AutoPipeline (batched) also runs on the pipeline pool.
|
||||
ap, err := c.AsyncAutoPipelineWithOptions(&redis.AutoPipelineOptions{MaxBatchSize: 300, MaxFlushDelay: 50 * time.Millisecond})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer ap.Close()
|
||||
s := ap.Set(ctx, "hs:ap1", "v2", 0)
|
||||
s2 := ap.Set(ctx, "hs:ap2", "v3", 0)
|
||||
if err := s.Err(); err != nil {
|
||||
t.Fatalf("autopipeline set1: %v", err)
|
||||
}
|
||||
if err := s2.Err(); err != nil {
|
||||
t.Fatalf("autopipeline set2: %v", err)
|
||||
}
|
||||
|
||||
// SELECT DB really took effect on the pipeline conns: keys exist in `db`,
|
||||
// not in DB 0.
|
||||
verify := redis.NewClient(&redis.Options{Addr: apTestAddr(), Username: user, Password: pw, DB: db})
|
||||
defer verify.Close()
|
||||
for _, k := range []string{"hs:pipe", "hs:ap1", "hs:ap2"} {
|
||||
if n, _ := verify.Exists(ctx, k).Result(); n != 1 {
|
||||
t.Fatalf("key %q not found in DB %d (SELECT not applied on pipeline conn)", k, db)
|
||||
}
|
||||
}
|
||||
db0 := redis.NewClient(&redis.Options{Addr: apTestAddr(), DB: 0})
|
||||
defer db0.Close()
|
||||
if n, _ := db0.Exists(ctx, "hs:pipe").Result(); n != 0 {
|
||||
t.Fatalf("hs:pipe leaked into DB 0 (wrong SELECT on pipeline conn)")
|
||||
}
|
||||
|
||||
// cleanup keys in db
|
||||
verify.Del(ctx, "hs:pipe", "hs:ap1", "hs:ap2")
|
||||
}
|
||||
|
||||
// ===== from pipeline_pool_maintnotif_test.go =====
|
||||
|
||||
// TestPipelinePoolMaintNotifications verifies maintnotifications is wired to the
|
||||
// DEDICATED pipeline pool: the CLIENT MAINT_NOTIFICATIONS handshake runs on
|
||||
// pipeline-pool conns during initConn and the pipeline read path drains push
|
||||
// frames, so pipelines and autopipelines keep working with the feature enabled.
|
||||
// (The manager registers the pipeline pool via InitPoolHookForPool and tracks
|
||||
// each conn via TrackMaintNotificationsConn.)
|
||||
func TestPipelinePoolMaintNotifications(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
c := redis.NewClient(&redis.Options{
|
||||
Addr: apTestAddr(),
|
||||
Protocol: 3, // maintnotifications requires RESP3
|
||||
MaintNotificationsConfig: &maintnotifications.Config{Mode: maintnotifications.ModeAuto},
|
||||
PipelineReadBufferSize: 64 << 10,
|
||||
PipelineWriteBufferSize: 64 << 10,
|
||||
PipelinePoolSize: 2,
|
||||
})
|
||||
defer c.Close()
|
||||
if err := c.Ping(ctx).Err(); err != nil {
|
||||
t.Skipf("no redis: %v", err)
|
||||
}
|
||||
|
||||
// Manual pipeline on the pipeline pool.
|
||||
pipe := c.Pipeline()
|
||||
pipe.Set(ctx, "mn:k", "v", 0)
|
||||
g := pipe.Get(ctx, "mn:k")
|
||||
if _, err := pipe.Exec(ctx); err != nil {
|
||||
t.Fatalf("pipeline with maintnotifications enabled: %v", err)
|
||||
}
|
||||
if g.Val() != "v" {
|
||||
t.Fatalf("pipeline get = %q, want v", g.Val())
|
||||
}
|
||||
if st := c.PoolStats(); st.PipelineStats == nil {
|
||||
t.Fatal("expected a dedicated pipeline pool (PipelineStats nil)")
|
||||
}
|
||||
|
||||
// AutoPipeline (batched) on the pipeline pool.
|
||||
ap, err := c.AsyncAutoPipelineWithOptions(&redis.AutoPipelineOptions{MaxBatchSize: 300, MaxFlushDelay: 50 * time.Millisecond})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer ap.Close()
|
||||
s1 := ap.Set(ctx, "mn:ap1", "v1", 0)
|
||||
s2 := ap.Incr(ctx, "mn:n")
|
||||
if err := s1.Err(); err != nil {
|
||||
t.Fatalf("autopipeline set with maintnotifications: %v", err)
|
||||
}
|
||||
if err := s2.Err(); err != nil {
|
||||
t.Fatalf("autopipeline incr with maintnotifications: %v", err)
|
||||
}
|
||||
if s2.Val() != 1 {
|
||||
t.Fatalf("autopipeline incr = %d, want 1", s2.Val())
|
||||
}
|
||||
|
||||
c.Del(ctx, "mn:k", "mn:ap1", "mn:n")
|
||||
}
|
||||
|
||||
// ===== from pipeline_pool_streamingcreds_test.go =====
|
||||
|
||||
// TestPipelinePoolStreamingCredsReauth verifies the StreamingCredentialsProvider
|
||||
// drives AUTH on DEDICATED pipeline-pool connections too, both at init and on a
|
||||
// credential rotation. It mirrors the main-pool streaming test but drives
|
||||
// pipeline traffic (which runs on the pipeline pool) and uses PipelinePoolSize=1
|
||||
// so the single pipeline conn must re-auth.
|
||||
//
|
||||
// Credentials are intentionally invalid on the server, so the pipeline itself
|
||||
// errors — the assertion is that the AUTH command was issued with the initial,
|
||||
// then the updated, credentials for the pipeline-pool connection (proving the
|
||||
// re-auth listener is wired on that pool).
|
||||
func TestPipelinePoolStreamingCredsReauth(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
recorder := newCommandRecorder(200)
|
||||
initialCreds := auth.NewBasicCredentials("initial_user", "initial_pass")
|
||||
updatedCreds := auth.NewBasicCredentials("updated_user", "updated_pass")
|
||||
updates := make(chan auth.Credentials, 1)
|
||||
|
||||
opt := &redis.Options{
|
||||
Addr: apTestAddr(),
|
||||
StreamingCredentialsProvider: &mockStreamingProvider{credentials: initialCreds, updates: updates},
|
||||
PipelineReadBufferSize: 64 << 10,
|
||||
PipelineWriteBufferSize: 64 << 10,
|
||||
PipelinePoolSize: 1,
|
||||
PoolSize: 1,
|
||||
}
|
||||
c := redis.NewClient(opt)
|
||||
defer c.Close()
|
||||
c.AddHook(recorder.Hook())
|
||||
|
||||
// probe with reachability: if there's no redis at all, skip.
|
||||
if err := c.Ping(ctx).Err(); err == nil {
|
||||
// creds are invalid, so a successful ping means auth isn't enforced here;
|
||||
// the test still exercises the AUTH-issuing path below.
|
||||
}
|
||||
|
||||
runPipe := func() {
|
||||
pipe := c.Pipeline()
|
||||
pipe.Set(ctx, "sc:k", "v", 0)
|
||||
pipe.Get(ctx, "sc:k")
|
||||
_, _ = pipe.Exec(ctx) // errors (bad creds); we only care that AUTH was issued
|
||||
}
|
||||
|
||||
runPipe()
|
||||
deadline := time.Now().Add(2 * time.Second)
|
||||
for !recorder.Contains("AUTH initial_user") && time.Now().Before(deadline) {
|
||||
time.Sleep(50 * time.Millisecond)
|
||||
runPipe()
|
||||
}
|
||||
if !recorder.Contains("AUTH initial_user") {
|
||||
t.Skipf("no AUTH observed (redis without auth enforcement or unreachable); last commands=%v", recorder.LastCommands())
|
||||
}
|
||||
|
||||
// rotate credentials; the pipeline-pool conn must re-auth with the new creds.
|
||||
updates <- updatedCreds
|
||||
deadline = time.Now().Add(2 * time.Second)
|
||||
for !recorder.Contains("AUTH updated_user") && time.Now().Before(deadline) {
|
||||
time.Sleep(50 * time.Millisecond)
|
||||
runPipe()
|
||||
}
|
||||
if !recorder.Contains("AUTH updated_user") {
|
||||
t.Fatalf("pipeline-pool conn did not re-auth after credential rotation; commands=%v", recorder.LastCommands())
|
||||
}
|
||||
close(updates)
|
||||
}
|
||||
@@ -233,6 +233,7 @@ func newScanDumpCmd(ctx context.Context, args ...interface{}) *ScanDumpCmd {
|
||||
}
|
||||
|
||||
func (cmd *ScanDumpCmd) String() string {
|
||||
cmd.await()
|
||||
return cmdString(cmd, cmd.val)
|
||||
}
|
||||
|
||||
@@ -241,10 +242,12 @@ func (cmd *ScanDumpCmd) SetVal(val ScanDump) {
|
||||
}
|
||||
|
||||
func (cmd *ScanDumpCmd) Result() (ScanDump, error) {
|
||||
cmd.await()
|
||||
return cmd.val, cmd.err
|
||||
}
|
||||
|
||||
func (cmd *ScanDumpCmd) Val() ScanDump {
|
||||
cmd.await()
|
||||
return cmd.val
|
||||
}
|
||||
|
||||
@@ -316,14 +319,17 @@ func (cmd *BFInfoCmd) SetVal(val BFInfo) {
|
||||
}
|
||||
|
||||
func (cmd *BFInfoCmd) String() string {
|
||||
cmd.await()
|
||||
return cmdString(cmd, cmd.val)
|
||||
}
|
||||
|
||||
func (cmd *BFInfoCmd) Val() BFInfo {
|
||||
cmd.await()
|
||||
return cmd.val
|
||||
}
|
||||
|
||||
func (cmd *BFInfoCmd) Result() (BFInfo, error) {
|
||||
cmd.await()
|
||||
return cmd.val, cmd.err
|
||||
}
|
||||
|
||||
@@ -653,14 +659,17 @@ func (cmd *CFInfoCmd) SetVal(val CFInfo) {
|
||||
}
|
||||
|
||||
func (cmd *CFInfoCmd) String() string {
|
||||
cmd.await()
|
||||
return cmdString(cmd, cmd.val)
|
||||
}
|
||||
|
||||
func (cmd *CFInfoCmd) Val() CFInfo {
|
||||
cmd.await()
|
||||
return cmd.val
|
||||
}
|
||||
|
||||
func (cmd *CFInfoCmd) Result() (CFInfo, error) {
|
||||
cmd.await()
|
||||
return cmd.val, cmd.err
|
||||
}
|
||||
|
||||
@@ -823,14 +832,17 @@ func (cmd *CMSInfoCmd) SetVal(val CMSInfo) {
|
||||
}
|
||||
|
||||
func (cmd *CMSInfoCmd) String() string {
|
||||
cmd.await()
|
||||
return cmdString(cmd, cmd.val)
|
||||
}
|
||||
|
||||
func (cmd *CMSInfoCmd) Val() CMSInfo {
|
||||
cmd.await()
|
||||
return cmd.val
|
||||
}
|
||||
|
||||
func (cmd *CMSInfoCmd) Result() (CMSInfo, error) {
|
||||
cmd.await()
|
||||
return cmd.val, cmd.err
|
||||
}
|
||||
|
||||
@@ -1024,14 +1036,17 @@ func (cmd *TopKInfoCmd) SetVal(val TopKInfo) {
|
||||
}
|
||||
|
||||
func (cmd *TopKInfoCmd) String() string {
|
||||
cmd.await()
|
||||
return cmdString(cmd, cmd.val)
|
||||
}
|
||||
|
||||
func (cmd *TopKInfoCmd) Val() TopKInfo {
|
||||
cmd.await()
|
||||
return cmd.val
|
||||
}
|
||||
|
||||
func (cmd *TopKInfoCmd) Result() (TopKInfo, error) {
|
||||
cmd.await()
|
||||
return cmd.val, cmd.err
|
||||
}
|
||||
|
||||
@@ -1279,14 +1294,17 @@ func (cmd *TDigestInfoCmd) SetVal(val TDigestInfo) {
|
||||
}
|
||||
|
||||
func (cmd *TDigestInfoCmd) String() string {
|
||||
cmd.await()
|
||||
return cmdString(cmd, cmd.val)
|
||||
}
|
||||
|
||||
func (cmd *TDigestInfoCmd) Val() TDigestInfo {
|
||||
cmd.await()
|
||||
return cmd.val
|
||||
}
|
||||
|
||||
func (cmd *TDigestInfoCmd) Result() (TDigestInfo, error) {
|
||||
cmd.await()
|
||||
return cmd.val, cmd.err
|
||||
}
|
||||
|
||||
|
||||
+19
-7
@@ -29,17 +29,24 @@ var _ = Describe("Probabilistic commands", Label("probabilistic"), func() {
|
||||
protocol := protocol // capture loop variable for each context
|
||||
|
||||
Context(fmt.Sprintf("with protocol version %d", protocol), func() {
|
||||
var client *redis.Client
|
||||
var client redis.UniversalClient
|
||||
var rawClient *redis.Client
|
||||
var closeSubject func() error
|
||||
|
||||
BeforeEach(func() {
|
||||
client = setupRedisClient(protocol)
|
||||
Expect(client.FlushAll(ctx).Err()).NotTo(HaveOccurred())
|
||||
rawClient = setupRedisClient(protocol)
|
||||
client, closeSubject = newUniversalSubject(rawClient)
|
||||
Expect(rawClient.FlushAll(ctx).Err()).NotTo(HaveOccurred())
|
||||
})
|
||||
|
||||
AfterEach(func() {
|
||||
if client != nil {
|
||||
client.FlushDB(ctx)
|
||||
client.Close()
|
||||
// Flush through the SUBJECT: ordered after queued writes
|
||||
// (see json_test.go); awaiting Err() forces it to execute, and
|
||||
// both steps are asserted so a failed teardown cannot silently
|
||||
// leak state into later specs.
|
||||
Expect(client.FlushDB(ctx).Err()).NotTo(HaveOccurred())
|
||||
Expect(closeSubject()).NotTo(HaveOccurred())
|
||||
}
|
||||
})
|
||||
|
||||
@@ -243,7 +250,11 @@ var _ = Describe("Probabilistic commands", Label("probabilistic"), func() {
|
||||
client.BFLoadChunk(ctx, "testbfsd1", e.Iter, e.Data)
|
||||
}
|
||||
infAfter := client.BFInfoSize(ctx, "testbfsd1")
|
||||
Expect(infBefore).To(BeEquivalentTo(infAfter))
|
||||
// Compare values, not command structs: on the async
|
||||
// autopipeline face each command carries its own batch
|
||||
// pointer, so struct equality is always false there (and
|
||||
// Val() awaits the result).
|
||||
Expect(infBefore.Val()).To(BeEquivalentTo(infAfter.Val()))
|
||||
})
|
||||
|
||||
It("should BFReserveWithArgs", Label("bloom", "bfreserveargs"), func() {
|
||||
@@ -368,7 +379,8 @@ var _ = Describe("Probabilistic commands", Label("probabilistic"), func() {
|
||||
client.CFLoadChunk(ctx, "testcfsd1", e.Iter, e.Data)
|
||||
}
|
||||
infAfter := client.CFInfo(ctx, "testcfsd1")
|
||||
Expect(infBefore).To(BeEquivalentTo(infAfter))
|
||||
// Compare values, not command structs (see BFScanDump above).
|
||||
Expect(infBefore.Val()).To(BeEquivalentTo(infAfter.Val()))
|
||||
})
|
||||
|
||||
It("should CFInfo and CFReserveWithArgs", Label("cuckoo", "cfinfo", "cfreserveargs"), func() {
|
||||
|
||||
+12
-4
@@ -20,8 +20,15 @@ type PubSubCmdable interface {
|
||||
func (c cmdable) Publish(ctx context.Context, channel string, message interface{}) *IntCmd {
|
||||
cmd := NewIntCmd(ctx, "publish", channel, message)
|
||||
_ = c(ctx, cmd)
|
||||
// Record PubSub message sent (if command succeeded)
|
||||
if cmd.Err() == nil {
|
||||
// Record PubSub message sent (if command succeeded). Gated on the result
|
||||
// being readable WITHOUT blocking: on the deferred autopipeline face the
|
||||
// call above only enqueues, so reading the outcome here would await the
|
||||
// batch and turn a fire-and-forget publish into a blocking call — i.e.
|
||||
// enabling telemetry would change the async call shape (review finding by
|
||||
// codex on #3942). The metric is therefore skipped for a submission that
|
||||
// has not executed yet; recording it from the execution path instead is a
|
||||
// follow-up in the OTel wiring, not something the command wrapper can do.
|
||||
if otel.Enabled() && cmd.resultReady() && cmd.rawErr() == nil {
|
||||
otel.RecordPubSubMessage(ctx, nil, "sent", channel, false)
|
||||
}
|
||||
return cmd
|
||||
@@ -30,8 +37,9 @@ func (c cmdable) Publish(ctx context.Context, channel string, message interface{
|
||||
func (c cmdable) SPublish(ctx context.Context, channel string, message interface{}) *IntCmd {
|
||||
cmd := NewIntCmd(ctx, "spublish", channel, message)
|
||||
_ = c(ctx, cmd)
|
||||
// Record PubSub message sent (if command succeeded)
|
||||
if cmd.Err() == nil {
|
||||
// Record PubSub message sent (if command succeeded). See Publish for why
|
||||
// this is gated on the result being readable without blocking.
|
||||
if otel.Enabled() && cmd.resultReady() && cmd.rawErr() == nil {
|
||||
otel.RecordPubSubMessage(ctx, nil, "sent", channel, true)
|
||||
}
|
||||
return cmd
|
||||
|
||||
@@ -239,6 +239,13 @@ func (hs *hooksMixin) dialHook(ctx context.Context, network, addr string) (net.C
|
||||
return hs.state.Load().current.dial(ctx, network, addr)
|
||||
}
|
||||
|
||||
// hookCount reports how many user hooks are installed. The autopipeliner
|
||||
// arms its await() self-deadlock guard only when hooks exist, keeping the
|
||||
// guard a single atomic load on hook-free clients.
|
||||
func (hs *hooksMixin) hookCount() int {
|
||||
return len(hs.state.Load().slice)
|
||||
}
|
||||
|
||||
func (hs *hooksMixin) processHook(ctx context.Context, cmd Cmder) error {
|
||||
return hs.state.Load().current.process(ctx, cmd)
|
||||
}
|
||||
@@ -344,10 +351,26 @@ func (h *onCloseHooks) run() error {
|
||||
}
|
||||
|
||||
type baseClient struct {
|
||||
// apClosed flips when the shared pools begin closing; every wrapper and
|
||||
// every clone SHARING those pools refuses to build a new autopipeliner
|
||||
// from then on. A pointer: withTimeout/clone copy it, so the flag is one
|
||||
// per pool-set, not one per wrapper. See baseClient.Close.
|
||||
apClosed *atomic.Bool
|
||||
|
||||
opt *Options
|
||||
optLock sync.RWMutex
|
||||
connPool pool.Pooler
|
||||
pubSubPool *pool.PubSubPool
|
||||
// pipelinePool is an optional separate connection pool for pipelining
|
||||
// operations, used when PipelineReadBufferSize/PipelineWriteBufferSize is
|
||||
// set so pipelines can use large buffers without bloating the main pool.
|
||||
// nil means pipelines use connPool.
|
||||
pipelinePool pool.Pooler
|
||||
// pipelinePoolName is the pool name assigned to pipelinePool's connections
|
||||
// (pool.Conn.PoolName()). It lets poolForConn route a connection back to the
|
||||
// pool that owns it — e.g. so streaming-credentials re-auth closes/accounts a
|
||||
// failed pipeline connection against pipelinePool, not connPool.
|
||||
pipelinePoolName string
|
||||
hooksMixin
|
||||
|
||||
// onClose holds named callbacks invoked when the client is closed.
|
||||
@@ -420,8 +443,11 @@ func (c *baseClient) clone() *baseClient {
|
||||
c.maintNotificationsManagerLock.RUnlock()
|
||||
|
||||
clone := &baseClient{
|
||||
apClosed: c.apClosed,
|
||||
opt: c.opt,
|
||||
connPool: c.connPool,
|
||||
pipelinePool: c.pipelinePool,
|
||||
pipelinePoolName: c.pipelinePoolName,
|
||||
pubSubPool: c.pubSubPool,
|
||||
onClose: c.onClose,
|
||||
pushProcessor: c.pushProcessor,
|
||||
@@ -488,16 +514,30 @@ func (c *baseClient) _getConn(ctx context.Context) (*pool.Conn, error) {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
if err := c.initPooledConn(ctx, c.connPool, cn); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return cn, nil
|
||||
}
|
||||
|
||||
// initPooledConn brings a conn freshly obtained from p to a usable state: it
|
||||
// runs the connection handshake if needed, records the connection-create-time
|
||||
// metric, and re-acquires the conn after initConn parks it IDLE. On failure
|
||||
// the conn is Removed from p (never leaked) and the error is unwrapped to the
|
||||
// caller-visible cause. Shared by the main-pool path (_getConn) and the
|
||||
// dedicated pipeline-pool path (withPipelineConn) so the two cannot drift.
|
||||
func (c *baseClient) initPooledConn(ctx context.Context, p pool.Pooler, cn *pool.Conn) error {
|
||||
if cn.IsInited() {
|
||||
return cn, nil
|
||||
return nil
|
||||
}
|
||||
|
||||
if err := c.initConn(ctx, cn); err != nil {
|
||||
c.connPool.Remove(ctx, cn, err)
|
||||
if err := errors.Unwrap(err); err != nil {
|
||||
return nil, err
|
||||
p.Remove(ctx, cn, err)
|
||||
if unwrapped := errors.Unwrap(err); unwrapped != nil {
|
||||
return unwrapped
|
||||
}
|
||||
return nil, err
|
||||
return err
|
||||
}
|
||||
|
||||
if dialStartNs := cn.GetDialStartNs(); dialStartNs > 0 {
|
||||
@@ -510,10 +550,25 @@ func (c *baseClient) _getConn(ctx context.Context) (*pool.Conn, error) {
|
||||
// initConn will transition to IDLE state, so we need to acquire it
|
||||
// before returning it to the user.
|
||||
if !cn.TryAcquire() {
|
||||
return nil, fmt.Errorf("redis: connection is not usable")
|
||||
err := fmt.Errorf("redis: connection is not usable")
|
||||
// Remove rather than abandon: an unacquirable conn left outside the
|
||||
// pool's accounting would leak its slot.
|
||||
p.Remove(ctx, cn, err)
|
||||
return err
|
||||
}
|
||||
|
||||
return cn, nil
|
||||
return nil
|
||||
}
|
||||
|
||||
// poolForConn returns the pool that owns cn — the dedicated pipeline pool when
|
||||
// cn was dialed there, otherwise the main pool. Re-auth close/accounting must
|
||||
// target the owning pool so a failed pipeline connection is removed from the
|
||||
// pipeline pool's books, not the main pool's.
|
||||
func (c *baseClient) poolForConn(cn *pool.Conn) pool.Pooler {
|
||||
if c.pipelinePool != nil && c.pipelinePoolName != "" && cn.PoolName() == c.pipelinePoolName {
|
||||
return c.pipelinePool
|
||||
}
|
||||
return c.connPool
|
||||
}
|
||||
|
||||
func (c *baseClient) reAuthConnection() func(poolCn *pool.Conn, credentials auth.Credentials) error {
|
||||
@@ -524,7 +579,7 @@ func (c *baseClient) reAuthConnection() func(poolCn *pool.Conn, credentials auth
|
||||
// Use background context - timeout is handled by ReadTimeout in WithReader/WithWriter
|
||||
ctx := context.Background()
|
||||
|
||||
connPool := pool.NewSingleConnPool(c.connPool, poolCn)
|
||||
connPool := pool.NewSingleConnPool(c.poolForConn(poolCn), poolCn)
|
||||
|
||||
// Pass hooks so that reauth commands are recorded/traced; share the
|
||||
// HIMPORT registry for the same reason as in initConn.
|
||||
@@ -539,6 +594,7 @@ func (c *baseClient) reAuthConnection() func(poolCn *pool.Conn, credentials auth
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
func (c *baseClient) onAuthenticationErr() func(poolCn *pool.Conn, err error) {
|
||||
return func(poolCn *pool.Conn, err error) {
|
||||
if err != nil {
|
||||
@@ -548,7 +604,7 @@ func (c *baseClient) onAuthenticationErr() func(poolCn *pool.Conn, err error) {
|
||||
// waits for IDLE state before transitioning to UNUSABLE for re-auth).
|
||||
// From metrics perspective, the connection was never "used" by a client.
|
||||
// Note: Using context.Background() as this callback doesn't have access to caller's context.
|
||||
err := c.connPool.CloseConn(context.Background(), poolCn, pool.CloseReasonAuthError, pool.MetricStateIdle)
|
||||
err := c.poolForConn(poolCn).CloseConn(context.Background(), poolCn, pool.CloseReasonAuthError, pool.MetricStateIdle)
|
||||
if err != nil {
|
||||
internal.Logger.Printf(context.Background(), "redis: failed to close connection: %v", err)
|
||||
// try to close the network connection directly
|
||||
@@ -980,29 +1036,38 @@ func (c *baseClient) releaseConn(ctx context.Context, cn *pool.Conn, err error)
|
||||
if c.opt.Limiter != nil {
|
||||
c.opt.Limiter.ReportResult(err)
|
||||
}
|
||||
c.releaseConnToPool(ctx, c.connPool, cn, err)
|
||||
}
|
||||
|
||||
// releaseConnToPool returns a conn to p after a command or pipeline ran on
|
||||
// it: bad conns are Removed, pending push notifications are drained (a
|
||||
// mid-frame drain failure also Removes — the reply stream may be
|
||||
// desynchronized), and a client-side-cache post-read probe is requested when
|
||||
// tracking is on. Limiter accounting stays with the callers, whose shapes
|
||||
// differ. Shared by releaseConn and withPipelineConn so the two cannot drift.
|
||||
func (c *baseClient) releaseConnToPool(ctx context.Context, p pool.Pooler, cn *pool.Conn, err error) {
|
||||
if isBadConn(err, false, c.opt.Addr) {
|
||||
c.connPool.Remove(ctx, cn, err)
|
||||
} else {
|
||||
// process any pending push notifications before returning the connection to the pool
|
||||
if err := c.processPushNotifications(ctx, cn); err != nil {
|
||||
internal.Logger.Printf(ctx, "push: error processing pending notifications before releasing connection: %v", err)
|
||||
if isBadConn(err, false, c.opt.Addr) {
|
||||
// A mid-frame read failure may leave the reply stream
|
||||
// desynchronized, so the connection cannot be reused.
|
||||
c.connPool.Remove(ctx, cn, err)
|
||||
return
|
||||
}
|
||||
}
|
||||
if c.cscTrackingRequested() {
|
||||
// A TLS-like wrapper can retain decrypted bytes after the command
|
||||
// reply even when its raw socket is empty. Ask the background
|
||||
// drainer for one bounded post-read probe before relying on raw
|
||||
// socket peeks again.
|
||||
cn.MarkCscReadPending()
|
||||
}
|
||||
c.connPool.Put(ctx, cn)
|
||||
p.Remove(ctx, cn, err)
|
||||
return
|
||||
}
|
||||
// process any pending push notifications before returning the connection to the pool
|
||||
if err := c.processPushNotifications(ctx, cn); err != nil {
|
||||
internal.Logger.Printf(ctx, "push: error processing pending notifications before releasing connection: %v", err)
|
||||
if isBadConn(err, false, c.opt.Addr) {
|
||||
// A mid-frame read failure may leave the reply stream
|
||||
// desynchronized, so the connection cannot be reused.
|
||||
p.Remove(ctx, cn, err)
|
||||
return
|
||||
}
|
||||
}
|
||||
if c.cscTrackingRequested() {
|
||||
// A TLS-like wrapper can retain decrypted bytes after the command
|
||||
// reply even when its raw socket is empty. Ask the background
|
||||
// drainer for one bounded post-read probe before relying on raw
|
||||
// socket peeks again.
|
||||
cn.MarkCscReadPending()
|
||||
}
|
||||
p.Put(ctx, cn)
|
||||
}
|
||||
|
||||
func (c *baseClient) withConn(
|
||||
@@ -1023,6 +1088,71 @@ func (c *baseClient) withConn(
|
||||
return fnErr
|
||||
}
|
||||
|
||||
// withPipelineConn executes fn with a connection from the pipeline pool when
|
||||
// one is configured (PipelineReadBufferSize/PipelineWriteBufferSize set),
|
||||
// otherwise it falls back to the regular pool via withConn.
|
||||
// withPipelineConn is withConn/releaseConn for the DEDICATED pipeline pool.
|
||||
// Conn preparation and release go through the shared pool-parameterized
|
||||
// helpers (initPooledConn, releaseConnToPool) — the paths used to mirror each
|
||||
// other by hand and drifted three times (a Limiter-ordering divergence, a
|
||||
// missed drain-error removal, a missed client-side-cache probe), so only the
|
||||
// Limiter shape is allowed to live here.
|
||||
func (c *baseClient) withPipelineConn(
|
||||
ctx context.Context, fn func(context.Context, *pool.Conn) error,
|
||||
) (retErr error) {
|
||||
// Use pipeline pool if available, otherwise fall back to regular pool.
|
||||
if c.pipelinePool == nil {
|
||||
return c.withConn(ctx, fn)
|
||||
}
|
||||
|
||||
// Honor the Limiter on the dedicated pipeline-pool path too, mirroring
|
||||
// getConn/releaseConn: Allow() before acquiring and ReportResult() on every
|
||||
// exit (including the early init/re-acquire failures below). Without this,
|
||||
// enabling the pipeline pool would silently bypass throttling and failure
|
||||
// reporting for callers that set a Limiter.
|
||||
if c.opt.Limiter != nil {
|
||||
if err := c.opt.Limiter.Allow(); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
// One deferred exit for both concerns, because their ORDER is part of the
|
||||
// contract: releaseConn reports the result BEFORE the connection becomes
|
||||
// available again, so a limiter or circuit breaker observes the failure
|
||||
// before it can admit the next operation. Two separate defers would run
|
||||
// LIFO and release first, letting another pipelined operation through
|
||||
// against a breaker that has not seen the failure yet (review finding by
|
||||
// codex on #3942). cn is nil on the acquire/init failure paths, which still
|
||||
// must report.
|
||||
var cn *pool.Conn
|
||||
var fnErr error
|
||||
defer func() {
|
||||
if c.opt.Limiter != nil {
|
||||
c.opt.Limiter.ReportResult(retErr)
|
||||
}
|
||||
if cn != nil {
|
||||
c.releaseConnToPool(ctx, c.pipelinePool, cn, fnErr)
|
||||
}
|
||||
}()
|
||||
|
||||
cn, retErr = c.pipelinePool.Get(ctx)
|
||||
if retErr != nil {
|
||||
cn = nil // nothing acquired: no release, but still report above
|
||||
return retErr
|
||||
}
|
||||
|
||||
if err := c.initPooledConn(ctx, c.pipelinePool, cn); err != nil {
|
||||
// initPooledConn already removed the conn from the pool on failure.
|
||||
cn = nil
|
||||
retErr = err
|
||||
return retErr
|
||||
}
|
||||
|
||||
fnErr = fn(ctx, cn)
|
||||
retErr = fnErr
|
||||
return retErr
|
||||
}
|
||||
|
||||
func (c *baseClient) dial(ctx context.Context, network, addr string) (net.Conn, error) {
|
||||
return c.opt.Dialer(ctx, network, addr)
|
||||
}
|
||||
@@ -1369,6 +1499,13 @@ func (c *baseClient) enableMaintNotificationsUpgrades() error {
|
||||
|
||||
// Initialize pool hook (safe to call without lock since manager is now set)
|
||||
manager.InitPoolHook(c.dialHook)
|
||||
// If a dedicated pipeline connection pool is in use, attach an independent
|
||||
// maintnotifications hook to it as well. Otherwise autopipelined/pipelined
|
||||
// commands run on pipeline-pool connections that never receive MOVING/
|
||||
// MIGRATING handoff handling.
|
||||
if c.pipelinePool != nil {
|
||||
manager.InitPoolHookForPool(c.pipelinePool, c.dialHook)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -1393,6 +1530,14 @@ func (c *baseClient) disableMaintNotificationsUpgrades() error {
|
||||
// It is rare to Close a Client, as the Client is meant to be
|
||||
// long-lived and shared between many goroutines.
|
||||
func (c *baseClient) Close() error {
|
||||
// The pools this baseClient owns are shared with every WithTimeout/
|
||||
// WithReadTimeout clone. Once ANY sharer closes them, no wrapper may
|
||||
// build a fresh autopipeliner against them — its flushers would run
|
||||
// against closed pools forever. The atomic is checked by the
|
||||
// AutoPipeline getters of every wrapper sharing this base.
|
||||
if c.apClosed != nil {
|
||||
c.apClosed.Store(true)
|
||||
}
|
||||
if h := c.cscDrainHandle; h != nil {
|
||||
h.closeOnce.Do(func() {
|
||||
h.closeErr = c.closeResources()
|
||||
@@ -1419,13 +1564,18 @@ func (c *baseClient) closeResources() error {
|
||||
}
|
||||
|
||||
// Unregister pools from OTel before closing them
|
||||
otel.UnregisterPools(c.connPool, c.pubSubPool)
|
||||
otel.UnregisterPools(c.connPool, c.pubSubPool, c.pipelinePool)
|
||||
|
||||
if c.connPool != nil {
|
||||
if err := c.connPool.Close(); err != nil && firstErr == nil {
|
||||
firstErr = err
|
||||
}
|
||||
}
|
||||
if c.pipelinePool != nil {
|
||||
if err := c.pipelinePool.Close(); err != nil && firstErr == nil {
|
||||
firstErr = err
|
||||
}
|
||||
}
|
||||
if c.pubSubPool != nil {
|
||||
if err := c.pubSubPool.Close(); err != nil && firstErr == nil {
|
||||
firstErr = err
|
||||
@@ -1490,7 +1640,9 @@ func (c *baseClient) generalProcessPipeline(
|
||||
|
||||
// Enable retries by default to retry dial errors returned by withConn.
|
||||
canRetry := true
|
||||
lastErr = c.withConn(ctx, func(ctx context.Context, cn *pool.Conn) error {
|
||||
// Route pipelines through the dedicated pipeline pool when configured;
|
||||
// withPipelineConn falls back to the regular pool when it is not.
|
||||
lastErr = c.withPipelineConn(ctx, func(ctx context.Context, cn *pool.Conn) error {
|
||||
lastConn = cn
|
||||
// Process any pending push notifications before executing the pipeline
|
||||
if err := c.processPushNotifications(ctx, cn); err != nil {
|
||||
@@ -1523,6 +1675,17 @@ func (c *baseClient) generalProcessPipeline(
|
||||
}
|
||||
}
|
||||
|
||||
// Retries exhausted on a retryable error: the loop fell through without the
|
||||
// early-exit branch running, so the commands were never populated with the
|
||||
// failure. Mirror that branch here so callers that observe results only
|
||||
// per-command — notably AutoPipeline, which discards this function's returned
|
||||
// error — see the error instead of a nil error and a zero value. Guard on
|
||||
// !isRedisError so a per-command redis error (e.g. LOADING) keeps its own
|
||||
// reply rather than being overwritten.
|
||||
if !isRedisError(lastErr) {
|
||||
setCmdsErr(cmds, lastErr)
|
||||
}
|
||||
|
||||
if pipelineOpDurationCallback != nil {
|
||||
operationDuration := time.Since(operationStart)
|
||||
pipelineOpDurationCallback(ctx, operationDuration, operationName, len(cmds), totalAttempts, lastErr, lastConn, c.opt.DB)
|
||||
@@ -1600,9 +1763,11 @@ func (c *baseClient) pipelineProcessCmds(
|
||||
}
|
||||
|
||||
// Preserve retryable first-command errors (e.g. LOADING) for the outer
|
||||
// loop; the re-issue above may have cleared it.
|
||||
// loop; the re-issue above may have cleared it. rawErr: this runs on the
|
||||
// execution path; never await here (an async autopipeline command's ready
|
||||
// channel is closed by this very batch — Err() would self-deadlock).
|
||||
if readErr != nil {
|
||||
readErr = cmds[0].Err()
|
||||
readErr = cmds[0].rawErr()
|
||||
}
|
||||
return readErr != nil, readErr
|
||||
}
|
||||
@@ -1621,7 +1786,9 @@ func (c *baseClient) pipelineReadCmds(ctx context.Context, cn *pool.Conn, rd *pr
|
||||
}
|
||||
}
|
||||
// Retry errors like "LOADING redis is loading the dataset in memory".
|
||||
return cmds[0].Err()
|
||||
// rawErr: this runs on the execution path; never await here (an async
|
||||
// autopipeline command's ready channel is closed by this very batch).
|
||||
return cmds[0].rawErr()
|
||||
}
|
||||
|
||||
func (c *baseClient) txPipelineProcessCmds(
|
||||
@@ -1740,6 +1907,11 @@ type Client struct {
|
||||
// cleanup owns the drainer) reachable while a WithTimeout clone can still
|
||||
// serve from its cache. Nil on the canonical wrapper and on non-CSC clones.
|
||||
cscLifecycleOwner *Client
|
||||
|
||||
autopipelinerMu *sync.Mutex // guards the autopipeliner fields against concurrent first-call creation
|
||||
autopipeliner *AutoPipeliner // blocking face (Client.AutoPipeline)
|
||||
asyncAutopipeliner *AutoPipeliner // deferred face (Client.AsyncAutoPipeline)
|
||||
autopipelinerClosed bool // set by Close: refuse to resurrect a pipeliner on a closed client
|
||||
}
|
||||
|
||||
// NewClient returns a client to the Redis Server specified by Options.
|
||||
@@ -1756,9 +1928,10 @@ func NewClient(opt *Options) *Client {
|
||||
|
||||
c := Client{
|
||||
baseClient: &baseClient{
|
||||
opt: opt,
|
||||
onClose: &onCloseHooks{},
|
||||
himport: newHImportRegistry(),
|
||||
apClosed: &atomic.Bool{},
|
||||
opt: opt,
|
||||
onClose: &onCloseHooks{},
|
||||
himport: newHImportRegistry(),
|
||||
},
|
||||
}
|
||||
c.init()
|
||||
@@ -1785,9 +1958,42 @@ func NewClient(opt *Options) *Client {
|
||||
panic(fmt.Errorf("redis: failed to create pubsub pool: %w", err))
|
||||
}
|
||||
|
||||
// Optionally create a separate connection pool for pipelining, with its own
|
||||
// (typically larger) buffers, so pipelines can use big buffers without
|
||||
// bloating the main pool. Enabled when either pipeline buffer size is set.
|
||||
if opt.PipelineReadBufferSize > 0 || opt.PipelineWriteBufferSize > 0 {
|
||||
pipelineOpt := opt.clone()
|
||||
if opt.PipelineReadBufferSize > 0 {
|
||||
pipelineOpt.ReadBufferSize = opt.PipelineReadBufferSize
|
||||
// Same clamp Options.init applies to the main pool: RESP3 push
|
||||
// parsing needs a minimum read buffer, and a tiny pipeline reader
|
||||
// would break push-notification handling on pipeline conns.
|
||||
if pipelineOpt.Protocol == 3 && pipelineOpt.ReadBufferSize < proto.MinRESP3ReadBufferSize {
|
||||
pipelineOpt.ReadBufferSize = proto.MinRESP3ReadBufferSize
|
||||
}
|
||||
}
|
||||
if opt.PipelineWriteBufferSize > 0 {
|
||||
pipelineOpt.WriteBufferSize = opt.PipelineWriteBufferSize
|
||||
}
|
||||
if opt.PipelinePoolSize > 0 {
|
||||
pipelineOpt.PoolSize = opt.PipelinePoolSize
|
||||
} else {
|
||||
pipelineOpt.PoolSize = 10 // default smaller pool for pipelining
|
||||
}
|
||||
pipelinePoolName := opt.Addr + "_" + uniqueID + "_pipeline"
|
||||
c.pipelinePoolName = pipelinePoolName
|
||||
c.pipelinePool, err = newConnPool(pipelineOpt, c.dialHook, pipelinePoolName)
|
||||
if err != nil {
|
||||
panic(fmt.Errorf("redis: failed to create pipeline connection pool: %w", err))
|
||||
}
|
||||
}
|
||||
|
||||
if opt.StreamingCredentialsProvider != nil {
|
||||
c.streamingCredentialsManager = streaming.NewManager(c.connPool, c.opt.PoolTimeout)
|
||||
c.connPool.AddPoolHook(c.streamingCredentialsManager.PoolHook())
|
||||
if c.pipelinePool != nil {
|
||||
c.pipelinePool.AddPoolHook(c.streamingCredentialsManager.PoolHook())
|
||||
}
|
||||
}
|
||||
|
||||
// CSC wiring (SharedTracking): shared cache + per-connection CLIENT TRACKING +
|
||||
@@ -1832,12 +2038,17 @@ func NewClient(opt *Options) *Client {
|
||||
|
||||
// Register pools with OTel recorder if it supports pool registration
|
||||
// This allows async gauge metrics to pull stats from pools periodically
|
||||
otel.RegisterPools(c.connPool, c.pubSubPool, opt.Addr)
|
||||
otel.RegisterPools(c.connPool, c.pubSubPool, c.pipelinePool, opt.Addr)
|
||||
|
||||
return &c
|
||||
}
|
||||
|
||||
func (c *Client) init() {
|
||||
// Fresh per-Client guard and no inherited autopipeliner: a WithTimeout clone
|
||||
// (clone := *c) must not share the parent's mutex or AutoPipeliner instance.
|
||||
c.autopipelinerMu = &sync.Mutex{}
|
||||
c.autopipeliner = nil
|
||||
c.asyncAutopipeliner = nil
|
||||
c.cmdable = c.Process
|
||||
c.initHooks(hooks{
|
||||
dial: c.baseClient.dial,
|
||||
@@ -1847,8 +2058,17 @@ func (c *Client) init() {
|
||||
})
|
||||
}
|
||||
|
||||
// WithTimeout returns a clone sharing the parent's connection pools with the
|
||||
// given read/write timeout. The clone caches its own autopipeliners: an
|
||||
// AutoPipeline()/AsyncAutoPipeline() created on the clone is NOT stopped by
|
||||
// the parent's Close — call Close on the clone's autopipeliner explicitly.
|
||||
func (c *Client) WithTimeout(timeout time.Duration) *Client {
|
||||
// Snapshot under the guard: AutoPipeline()/Close() mutate the
|
||||
// autopipeliner fields concurrently, so a bare struct copy of them is a
|
||||
// data race (init below discards the copied values either way).
|
||||
c.autopipelinerMu.Lock()
|
||||
clone := *c
|
||||
c.autopipelinerMu.Unlock()
|
||||
if c.cscLifecycleOwner != nil {
|
||||
clone.cscLifecycleOwner = c.cscLifecycleOwner
|
||||
} else if c.baseClient.cscDrainHandle != nil {
|
||||
@@ -1859,13 +2079,47 @@ func (c *Client) WithTimeout(timeout time.Duration) *Client {
|
||||
return &clone
|
||||
}
|
||||
|
||||
// Close closes the shared client resources. A WithTimeout clone delegates CSC
|
||||
// teardown to the canonical wrapper that owns the background drainer.
|
||||
// Close closes the client, stopping both cached autopipeliners (the blocking
|
||||
// AutoPipeline instance and the async AsyncAutoPipeline instance, if created)
|
||||
// before releasing the underlying resources, so their background flusher
|
||||
// goroutines don't outlive the client. AutoPipeliner.Close is idempotent and
|
||||
// safe to call here even if autopipelining was never used.
|
||||
// A WithTimeout clone delegates CSC teardown to the canonical wrapper that
|
||||
// owns the background drainer.
|
||||
func (c *Client) Close() error {
|
||||
if c.cscLifecycleOwner != nil {
|
||||
return c.cscLifecycleOwner.baseClient.Close()
|
||||
c.autopipelinerMu.Lock()
|
||||
ap, async := c.autopipeliner, c.asyncAutopipeliner
|
||||
c.autopipeliner, c.asyncAutopipeliner = nil, nil
|
||||
// A later AutoPipeline()/AsyncAutoPipeline() call must not build a fresh
|
||||
// pipeliner against the closed pools: nothing would ever close it and its
|
||||
// flusher goroutines would leak. The getters check this flag.
|
||||
c.autopipelinerClosed = true
|
||||
c.autopipelinerMu.Unlock()
|
||||
var firstErr error
|
||||
for _, p := range []*AutoPipeliner{ap, async} {
|
||||
if p != nil {
|
||||
if err := p.Close(); err != nil && firstErr == nil {
|
||||
firstErr = err
|
||||
}
|
||||
}
|
||||
}
|
||||
return c.baseClient.Close()
|
||||
if c.cscLifecycleOwner != nil {
|
||||
// Delegate through the OWNER's *Client.Close, not its baseClient:
|
||||
// the owner may hold cached autopipeliners of its own whose flusher
|
||||
// goroutines must stop with the shared pools, and its
|
||||
// autopipelinerClosed flag must flip so later owner getters cannot
|
||||
// resurrect a pipeliner against closed pools. Client.Close is
|
||||
// idempotent through baseClient.Close, so an owner also closed
|
||||
// directly is fine.
|
||||
if err := c.cscLifecycleOwner.Close(); err != nil && firstErr == nil {
|
||||
firstErr = err
|
||||
}
|
||||
return firstErr
|
||||
}
|
||||
if err := c.baseClient.Close(); err != nil && firstErr == nil {
|
||||
firstErr = err
|
||||
}
|
||||
return firstErr
|
||||
}
|
||||
|
||||
func (c *Client) Conn() *Conn {
|
||||
@@ -1955,7 +2209,10 @@ type PoolStats pool.Stats
|
||||
// PoolStats returns connection pool stats.
|
||||
func (c *Client) PoolStats() *PoolStats {
|
||||
stats := c.connPool.Stats()
|
||||
stats.PubSubStats = *(c.pubSubPool.Stats())
|
||||
stats.PubSubStats = *c.pubSubPool.Stats()
|
||||
if c.pipelinePool != nil {
|
||||
stats.PipelineStats = c.pipelinePool.Stats()
|
||||
}
|
||||
return (*PoolStats)(stats)
|
||||
}
|
||||
|
||||
@@ -1971,6 +2228,79 @@ func (c *Client) Pipeline() Pipeliner {
|
||||
return &pipe
|
||||
}
|
||||
|
||||
// AutoPipeline returns the blocking autopipeliner for this client: a drop-in
|
||||
// replacement for the normal command surface where each command call (ap.Set,
|
||||
// ap.Get, ...) blocks until executed, exactly like a plain client — but the
|
||||
// engine batches concurrent callers' commands into pipelines, so throughput is
|
||||
// far higher (measured locally over loopback: ~1M+ SET/sec vs ~100k; indicative, not a guarantee). Commands keep per-goroutine order.
|
||||
//
|
||||
// By default, Options.AutoPipelineOptions is used if set,
|
||||
// otherwise DefaultBlockingAutoPipelineOptions (a single ordered batch stream,
|
||||
// which maximizes throughput and minimizes latency for the blocking face — see
|
||||
// its doc). The instance is cached and shared; the first
|
||||
// call's config wins and later calls return the same instance until it is closed.
|
||||
// It must be closed (or close the client) to release its goroutines.
|
||||
//
|
||||
// It returns an error if the supplied config is invalid (e.g. MaxConcurrentBatches>1
|
||||
// without Unordered, or a negative size); on error no instance is cached.
|
||||
//
|
||||
// EXPERIMENTAL: this API is subject to change, use with caution.
|
||||
func (c *Client) AutoPipeline() (*AutoPipeliner, error) {
|
||||
return c.AutoPipelineWithOptions(nil)
|
||||
}
|
||||
|
||||
// AutoPipelineWithOptions is AutoPipeline with explicit options instead of
|
||||
// Options.AutoPipelineOptions / the default. The instance is cached and shared;
|
||||
// the first call's config wins.
|
||||
//
|
||||
// EXPERIMENTAL: this API is subject to change, use with caution.
|
||||
func (c *Client) AutoPipelineWithOptions(config *AutoPipelineOptions) (*AutoPipeliner, error) {
|
||||
return getOrCreateAutoPipeliner(c.autopipelinerMu, &c.autopipeliner, &c.autopipelinerClosed, c.baseClient.apClosed, config,
|
||||
func() *AutoPipelineOptions {
|
||||
if c.opt.AutoPipelineOptions != nil {
|
||||
return c.opt.AutoPipelineOptions
|
||||
}
|
||||
return DefaultBlockingAutoPipelineOptions()
|
||||
},
|
||||
func(cfg *AutoPipelineOptions) (*AutoPipeliner, error) { return newAutoPipeliner(c, cfg, true) })
|
||||
}
|
||||
|
||||
// AsyncAutoPipeline returns the deferred (async) autopipeliner: command calls
|
||||
// return immediately and the result accessors (Val/Result/Err) block until the
|
||||
// command has executed. Submit a window of commands, then read their results, to
|
||||
// keep each pipeline deep and reach the highest throughput (measured locally over loopback: ~2-3M SET/sec; indicative).
|
||||
//
|
||||
// By default, Options.AutoPipelineOptions is used if set,
|
||||
// otherwise DefaultAutoPipelineOptions (ordered, MaxConcurrentBatches: 1) — a
|
||||
// single goroutine's deferred commands execute in submit order. Use AsyncAutoPipelineWithOptions
|
||||
// to override (and, for parallel batches, set Unordered). The instance is
|
||||
// cached and shared; the first call's config wins. Close it (or the client) to
|
||||
// release its goroutines.
|
||||
//
|
||||
// It returns an error if the supplied config is invalid (e.g. MaxConcurrentBatches>1
|
||||
// without Unordered, or a negative size); on error no instance is cached.
|
||||
//
|
||||
// EXPERIMENTAL: this API is subject to change, use with caution.
|
||||
func (c *Client) AsyncAutoPipeline() (*AutoPipeliner, error) {
|
||||
return c.AsyncAutoPipelineWithOptions(nil)
|
||||
}
|
||||
|
||||
// AsyncAutoPipelineWithOptions is AsyncAutoPipeline with an explicit config
|
||||
// instead of Options.AutoPipelineOptions / the default. The instance is cached
|
||||
// and shared; the first call's config wins.
|
||||
//
|
||||
// EXPERIMENTAL: this API is subject to change, use with caution.
|
||||
func (c *Client) AsyncAutoPipelineWithOptions(config *AutoPipelineOptions) (*AutoPipeliner, error) {
|
||||
return getOrCreateAutoPipeliner(c.autopipelinerMu, &c.asyncAutopipeliner, &c.autopipelinerClosed, c.baseClient.apClosed, config,
|
||||
func() *AutoPipelineOptions {
|
||||
if c.opt.AutoPipelineOptions != nil {
|
||||
return c.opt.AutoPipelineOptions
|
||||
}
|
||||
return DefaultAutoPipelineOptions()
|
||||
},
|
||||
func(cfg *AutoPipelineOptions) (*AutoPipeliner, error) { return newAutoPipeliner(c, cfg, false) })
|
||||
}
|
||||
|
||||
func (c *Client) TxPipelined(ctx context.Context, fn func(Pipeliner) error) ([]Cmder, error) {
|
||||
return c.TxPipeline().Pipelined(ctx, fn)
|
||||
}
|
||||
@@ -2093,6 +2423,7 @@ type Conn struct {
|
||||
func newConn(opt *Options, connPool pool.Pooler, parentHooks *hooksMixin, himport *himportRegistry) *Conn {
|
||||
c := Conn{
|
||||
baseClient: baseClient{
|
||||
apClosed: &atomic.Bool{},
|
||||
opt: opt,
|
||||
connPool: connPool,
|
||||
onClose: &onCloseHooks{},
|
||||
@@ -2212,9 +2543,14 @@ func (c *baseClient) peekAndProcessPushNotifications(ctx context.Context, cn *po
|
||||
return nil
|
||||
}
|
||||
|
||||
// Short read timeout: 10us is enough to consume any already-buffered push
|
||||
// notifications without blocking the caller.
|
||||
return cn.WithReader(ctx, 10*time.Microsecond, func(rd *proto.Reader) error {
|
||||
// Short read timeout: MaybeHasData confirmed kernel-buffered bytes, so
|
||||
// the first read returns immediately — the deadline only needs to cover
|
||||
// scheduler pauses, not network waits. 10us was routinely lost to
|
||||
// scheduling on loaded machines: the peek then timed out with nothing
|
||||
// consumed, the processor treated that as "no pending data", and a
|
||||
// connection with buffered push bytes was returned to the pool instead
|
||||
// of being drained (or removed, when the frame turns out partial).
|
||||
return cn.WithReader(ctx, time.Millisecond, func(rd *proto.Reader) error {
|
||||
handlerCtx := c.pushNotificationHandlerContext(cn)
|
||||
return c.pushProcessor.ProcessPendingNotifications(ctx, handlerCtx, rd)
|
||||
})
|
||||
|
||||
@@ -147,6 +147,14 @@ type RingOptions struct {
|
||||
// default: 32KiB (32768 bytes)
|
||||
WriteBufferSize int
|
||||
|
||||
// PipelineReadBufferSize, PipelineWriteBufferSize and PipelinePoolSize
|
||||
// configure an optional separate connection pool used for pipelining on
|
||||
// each shard, with its own (typically larger) buffers. See the same-named
|
||||
// fields on Options for details. The pool is created only when PipelineReadBufferSize or PipelineWriteBufferSize is set (PipelinePoolSize alone does not enable it).
|
||||
PipelineReadBufferSize int
|
||||
PipelineWriteBufferSize int
|
||||
PipelinePoolSize int
|
||||
|
||||
TLSConfig *tls.Config
|
||||
Limiter Limiter
|
||||
|
||||
@@ -251,6 +259,10 @@ func (opt *RingOptions) clientOptions() *Options {
|
||||
ReadBufferSize: opt.ReadBufferSize,
|
||||
WriteBufferSize: opt.WriteBufferSize,
|
||||
|
||||
PipelineReadBufferSize: opt.PipelineReadBufferSize,
|
||||
PipelineWriteBufferSize: opt.PipelineWriteBufferSize,
|
||||
PipelinePoolSize: opt.PipelinePoolSize,
|
||||
|
||||
TLSConfig: opt.TLSConfig,
|
||||
Limiter: opt.Limiter,
|
||||
|
||||
@@ -851,6 +863,34 @@ func (c *Ring) Pipeline() Pipeliner {
|
||||
return &pipe
|
||||
}
|
||||
|
||||
// ErrRingAutoPipelineUnsupported is returned by Ring's AutoPipeline /
|
||||
// AsyncAutoPipeline (and their WithOptions forms); check for it with
|
||||
// errors.Is. Autopipelining is not implemented for Ring; use the per-shard
|
||||
// clients or a ClusterClient. Ring is part of the UniversalClient
|
||||
// interface, so these methods exist to satisfy it and fail explicitly rather
|
||||
// than being silently absent.
|
||||
var ErrRingAutoPipelineUnsupported = errors.New("redis: AutoPipeline is not supported by Ring")
|
||||
|
||||
// AutoPipeline is not supported by Ring; it returns ErrRingAutoPipelineUnsupported.
|
||||
func (c *Ring) AutoPipeline() (*AutoPipeliner, error) {
|
||||
return c.AutoPipelineWithOptions(nil)
|
||||
}
|
||||
|
||||
// AutoPipelineWithOptions is not supported by Ring; it returns ErrRingAutoPipelineUnsupported.
|
||||
func (c *Ring) AutoPipelineWithOptions(config *AutoPipelineOptions) (*AutoPipeliner, error) {
|
||||
return nil, ErrRingAutoPipelineUnsupported
|
||||
}
|
||||
|
||||
// AsyncAutoPipeline is not supported by Ring; it returns ErrRingAutoPipelineUnsupported.
|
||||
func (c *Ring) AsyncAutoPipeline() (*AutoPipeliner, error) {
|
||||
return c.AsyncAutoPipelineWithOptions(nil)
|
||||
}
|
||||
|
||||
// AsyncAutoPipelineWithOptions is not supported by Ring; it returns ErrRingAutoPipelineUnsupported.
|
||||
func (c *Ring) AsyncAutoPipelineWithOptions(config *AutoPipelineOptions) (*AutoPipeliner, error) {
|
||||
return nil, ErrRingAutoPipelineUnsupported
|
||||
}
|
||||
|
||||
func (c *Ring) TxPipelined(ctx context.Context, fn func(Pipeliner) error) ([]Cmder, error) {
|
||||
return c.TxPipeline().Pipelined(ctx, fn)
|
||||
}
|
||||
|
||||
@@ -192,7 +192,7 @@ func (s *Script) EvalShaRO(ctx context.Context, c Scripter, keys []string, args
|
||||
// it is retried using EVAL.
|
||||
func (s *Script) Run(ctx context.Context, c Scripter, keys []string, args ...interface{}) *Cmd {
|
||||
r := s.EvalSha(ctx, c, keys, args...)
|
||||
if errors.Is(r.Err(), ErrNoScript) {
|
||||
if isNoScriptErr(r.Err()) {
|
||||
return s.Eval(ctx, c, keys, args...)
|
||||
}
|
||||
return r
|
||||
@@ -202,8 +202,20 @@ func (s *Script) Run(ctx context.Context, c Scripter, keys []string, args ...int
|
||||
// it is retried using EVAL_RO.
|
||||
func (s *Script) RunRO(ctx context.Context, c Scripter, keys []string, args ...interface{}) *Cmd {
|
||||
r := s.EvalShaRO(ctx, c, keys, args...)
|
||||
if errors.Is(r.Err(), ErrNoScript) {
|
||||
if isNoScriptErr(r.Err()) {
|
||||
return s.EvalRO(ctx, c, keys, args...)
|
||||
}
|
||||
return r
|
||||
}
|
||||
|
||||
// isNoScriptErr reports whether err means "this digest is not cached", whether
|
||||
// it arrived already normalized to ErrNoScript or as the server's raw NOSCRIPT
|
||||
// error. Both are accepted because the Eval wrappers only normalize when the
|
||||
// result is readable without blocking — on the deferred autopipeline face the
|
||||
// raw error reaches here untouched (see cmdable.eval).
|
||||
func isNoScriptErr(err error) bool {
|
||||
if err == nil {
|
||||
return false
|
||||
}
|
||||
return errors.Is(err, ErrNoScript) || HasErrorPrefix(err, "NOSCRIPT")
|
||||
}
|
||||
+10
-2
@@ -60,8 +60,16 @@ func (c cmdable) eval(ctx context.Context, name, payload string, keys []string,
|
||||
cmd.SetFirstKeyPos(3)
|
||||
}
|
||||
_ = c(ctx, cmd)
|
||||
if err := cmd.Err(); err != nil {
|
||||
if HasErrorPrefix(err, "NOSCRIPT") {
|
||||
// Normalize NOSCRIPT to ErrNoScript for Script.Run/RunRO's EVAL fallback,
|
||||
// but only when the result is already readable: on the deferred
|
||||
// autopipeline face the call above merely enqueues, and reading the outcome
|
||||
// here would await execution — making the whole Eval family synchronous on
|
||||
// a face whose contract is to return immediately (review finding by codex
|
||||
// on #3942). When the result is still pending the normalization is skipped;
|
||||
// Script.Run/RunRO also match the raw NOSCRIPT prefix, so the fallback
|
||||
// keeps working on that face.
|
||||
if cmd.resultReady() {
|
||||
if err := cmd.rawErr(); err != nil && HasErrorPrefix(err, "NOSCRIPT") {
|
||||
cmd.SetErr(ErrNoScript)
|
||||
}
|
||||
}
|
||||
|
||||
+33
-1
@@ -963,22 +963,27 @@ func (cmd *AggregateCmd) SetVal(val *FTAggregateResult) {
|
||||
}
|
||||
|
||||
func (cmd *AggregateCmd) Val() *FTAggregateResult {
|
||||
cmd.await()
|
||||
return cmd.val
|
||||
}
|
||||
|
||||
func (cmd *AggregateCmd) Result() (*FTAggregateResult, error) {
|
||||
cmd.await()
|
||||
return cmd.val, cmd.err
|
||||
}
|
||||
|
||||
func (cmd *AggregateCmd) RawVal() interface{} {
|
||||
cmd.await()
|
||||
return cmd.rawVal
|
||||
}
|
||||
|
||||
func (cmd *AggregateCmd) RawResult() (interface{}, error) {
|
||||
cmd.await()
|
||||
return cmd.rawVal, cmd.err
|
||||
}
|
||||
|
||||
func (cmd *AggregateCmd) String() string {
|
||||
cmd.await()
|
||||
return cmdString(cmd, cmd.val)
|
||||
}
|
||||
|
||||
@@ -1612,7 +1617,6 @@ func (c cmdable) FTCreate(ctx context.Context, index string, options *FTCreateOp
|
||||
}
|
||||
if schema.IndexMissing {
|
||||
args = append(args, "INDEXMISSING")
|
||||
|
||||
}
|
||||
}
|
||||
cmd := NewStatusCmd(ctx, args...)
|
||||
@@ -2175,6 +2179,7 @@ func newFTInfoCmd(ctx context.Context, args ...interface{}) *FTInfoCmd {
|
||||
}
|
||||
|
||||
func (cmd *FTInfoCmd) String() string {
|
||||
cmd.await()
|
||||
return cmdString(cmd, cmd.val)
|
||||
}
|
||||
|
||||
@@ -2183,20 +2188,25 @@ func (cmd *FTInfoCmd) SetVal(val FTInfoResult) {
|
||||
}
|
||||
|
||||
func (cmd *FTInfoCmd) Result() (FTInfoResult, error) {
|
||||
cmd.await()
|
||||
return cmd.val, cmd.err
|
||||
}
|
||||
|
||||
func (cmd *FTInfoCmd) Val() FTInfoResult {
|
||||
cmd.await()
|
||||
return cmd.val
|
||||
}
|
||||
|
||||
func (cmd *FTInfoCmd) RawVal() interface{} {
|
||||
cmd.await()
|
||||
return cmd.rawVal
|
||||
}
|
||||
|
||||
func (cmd *FTInfoCmd) RawResult() (interface{}, error) {
|
||||
cmd.await()
|
||||
return cmd.rawVal, cmd.err
|
||||
}
|
||||
|
||||
func (cmd *FTInfoCmd) readReply(rd *proto.Reader) (err error) {
|
||||
readType, err := rd.PeekReplyType()
|
||||
if err != nil {
|
||||
@@ -2377,6 +2387,7 @@ func newFTSpellCheckCmd(ctx context.Context, args ...interface{}) *FTSpellCheckC
|
||||
}
|
||||
|
||||
func (cmd *FTSpellCheckCmd) String() string {
|
||||
cmd.await()
|
||||
return cmdString(cmd, cmd.val)
|
||||
}
|
||||
|
||||
@@ -2385,18 +2396,22 @@ func (cmd *FTSpellCheckCmd) SetVal(val []SpellCheckResult) {
|
||||
}
|
||||
|
||||
func (cmd *FTSpellCheckCmd) Result() ([]SpellCheckResult, error) {
|
||||
cmd.await()
|
||||
return cmd.val, cmd.err
|
||||
}
|
||||
|
||||
func (cmd *FTSpellCheckCmd) Val() []SpellCheckResult {
|
||||
cmd.await()
|
||||
return cmd.val
|
||||
}
|
||||
|
||||
func (cmd *FTSpellCheckCmd) RawVal() interface{} {
|
||||
cmd.await()
|
||||
return cmd.rawVal
|
||||
}
|
||||
|
||||
func (cmd *FTSpellCheckCmd) RawResult() (interface{}, error) {
|
||||
cmd.await()
|
||||
return cmd.rawVal, cmd.err
|
||||
}
|
||||
|
||||
@@ -2693,6 +2708,7 @@ func newFTSearchCmd(ctx context.Context, options *FTSearchOptions, args ...inter
|
||||
}
|
||||
|
||||
func (cmd *FTSearchCmd) String() string {
|
||||
cmd.await()
|
||||
return cmdString(cmd, cmd.val)
|
||||
}
|
||||
|
||||
@@ -2701,18 +2717,22 @@ func (cmd *FTSearchCmd) SetVal(val FTSearchResult) {
|
||||
}
|
||||
|
||||
func (cmd *FTSearchCmd) Result() (FTSearchResult, error) {
|
||||
cmd.await()
|
||||
return cmd.val, cmd.err
|
||||
}
|
||||
|
||||
func (cmd *FTSearchCmd) Val() FTSearchResult {
|
||||
cmd.await()
|
||||
return cmd.val
|
||||
}
|
||||
|
||||
func (cmd *FTSearchCmd) RawVal() interface{} {
|
||||
cmd.await()
|
||||
return cmd.rawVal
|
||||
}
|
||||
|
||||
func (cmd *FTSearchCmd) RawResult() (interface{}, error) {
|
||||
cmd.await()
|
||||
return cmd.rawVal, cmd.err
|
||||
}
|
||||
|
||||
@@ -2976,6 +2996,7 @@ func newFTHybridCmd(ctx context.Context, options *FTHybridOptions, args ...inter
|
||||
}
|
||||
|
||||
func (cmd *FTHybridCmd) String() string {
|
||||
cmd.await()
|
||||
return cmdString(cmd, cmd.val)
|
||||
}
|
||||
|
||||
@@ -2984,26 +3005,32 @@ func (cmd *FTHybridCmd) SetVal(val FTHybridResult) {
|
||||
}
|
||||
|
||||
func (cmd *FTHybridCmd) Result() (FTHybridResult, error) {
|
||||
cmd.await()
|
||||
return cmd.val, cmd.err
|
||||
}
|
||||
|
||||
func (cmd *FTHybridCmd) CursorResult() (*FTHybridCursorResult, error) {
|
||||
cmd.await()
|
||||
return cmd.cursorVal, cmd.err
|
||||
}
|
||||
|
||||
func (cmd *FTHybridCmd) Val() FTHybridResult {
|
||||
cmd.await()
|
||||
return cmd.val
|
||||
}
|
||||
|
||||
func (cmd *FTHybridCmd) CursorVal() *FTHybridCursorResult {
|
||||
cmd.await()
|
||||
return cmd.cursorVal
|
||||
}
|
||||
|
||||
func (cmd *FTHybridCmd) RawVal() interface{} {
|
||||
cmd.await()
|
||||
return cmd.rawVal
|
||||
}
|
||||
|
||||
func (cmd *FTHybridCmd) RawResult() (interface{}, error) {
|
||||
cmd.await()
|
||||
return cmd.rawVal, cmd.err
|
||||
}
|
||||
|
||||
@@ -3514,6 +3541,7 @@ func NewFTSynDumpCmd(ctx context.Context, args ...interface{}) *FTSynDumpCmd {
|
||||
}
|
||||
|
||||
func (cmd *FTSynDumpCmd) String() string {
|
||||
cmd.await()
|
||||
return cmdString(cmd, cmd.val)
|
||||
}
|
||||
|
||||
@@ -3522,18 +3550,22 @@ func (cmd *FTSynDumpCmd) SetVal(val []FTSynDumpResult) {
|
||||
}
|
||||
|
||||
func (cmd *FTSynDumpCmd) Val() []FTSynDumpResult {
|
||||
cmd.await()
|
||||
return cmd.val
|
||||
}
|
||||
|
||||
func (cmd *FTSynDumpCmd) Result() ([]FTSynDumpResult, error) {
|
||||
cmd.await()
|
||||
return cmd.val, cmd.err
|
||||
}
|
||||
|
||||
func (cmd *FTSynDumpCmd) RawVal() interface{} {
|
||||
cmd.await()
|
||||
return cmd.rawVal
|
||||
}
|
||||
|
||||
func (cmd *FTSynDumpCmd) RawResult() (interface{}, error) {
|
||||
cmd.await()
|
||||
return cmd.rawVal, cmd.err
|
||||
}
|
||||
|
||||
|
||||
+43
-5
@@ -3,6 +3,7 @@ package redis_test
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/redis/go-redis/v9"
|
||||
)
|
||||
@@ -37,6 +38,12 @@ func TestSearchCommandsRESP2AndRESP3Equivalence(t *testing.T) {
|
||||
|
||||
t.Run("FTInfo", func(t *testing.T) {
|
||||
// Create index
|
||||
// Hermetic keyspace: earlier subtests drop only the index, and their
|
||||
// leftover hashes would be re-indexed here (a third doc with its own
|
||||
// score made the aggregate row sets protocol-timing-dependent).
|
||||
if err := client2.FlushDB(ctx).Err(); err != nil {
|
||||
t.Fatalf("FlushDB failed: %v", err)
|
||||
}
|
||||
_, err := client2.FTCreate(ctx, "test-idx",
|
||||
&redis.FTCreateOptions{},
|
||||
&redis.FieldSchema{FieldName: "title", FieldType: redis.SearchFieldTypeText, Sortable: true},
|
||||
@@ -118,9 +125,9 @@ func TestSearchCommandsRESP2AndRESP3Equivalence(t *testing.T) {
|
||||
// the data is silently lost. Assert both that the RESP3 value is
|
||||
// non-empty and that it matches the RESP2 value.
|
||||
avgFields := []struct {
|
||||
name string
|
||||
resp2Val string
|
||||
resp3Val string
|
||||
name string
|
||||
resp2Val string
|
||||
resp3Val string
|
||||
}{
|
||||
{"BytesPerRecordAvg", info2.BytesPerRecordAvg, info3.BytesPerRecordAvg},
|
||||
{"RecordsPerDocAvg", info2.RecordsPerDocAvg, info3.RecordsPerDocAvg},
|
||||
@@ -148,7 +155,7 @@ func TestSearchCommandsRESP2AndRESP3Equivalence(t *testing.T) {
|
||||
// (see src/fork_gc/fork_gc.c statsCb) but typed as int in GCStats.
|
||||
// Compare each individually so a regression surfaces with a precise message.
|
||||
gcFields := []struct {
|
||||
name string
|
||||
name string
|
||||
resp2, resp3 int
|
||||
}{
|
||||
{"BytesCollected", info2.GCStats.BytesCollected, info3.GCStats.BytesCollected},
|
||||
@@ -167,6 +174,12 @@ func TestSearchCommandsRESP2AndRESP3Equivalence(t *testing.T) {
|
||||
|
||||
t.Run("FTSearch", func(t *testing.T) {
|
||||
// Create index
|
||||
// Hermetic keyspace: earlier subtests drop only the index, and their
|
||||
// leftover hashes would be re-indexed here (a third doc with its own
|
||||
// score made the aggregate row sets protocol-timing-dependent).
|
||||
if err := client2.FlushDB(ctx).Err(); err != nil {
|
||||
t.Fatalf("FlushDB failed: %v", err)
|
||||
}
|
||||
_, err := client2.FTCreate(ctx, "test-idx",
|
||||
&redis.FTCreateOptions{},
|
||||
&redis.FieldSchema{FieldName: "title", FieldType: redis.SearchFieldTypeText},
|
||||
@@ -217,6 +230,12 @@ func TestSearchCommandsRESP2AndRESP3Equivalence(t *testing.T) {
|
||||
|
||||
t.Run("FTAggregate", func(t *testing.T) {
|
||||
// Create index
|
||||
// Hermetic keyspace: earlier subtests drop only the index, and their
|
||||
// leftover hashes would be re-indexed here (a third doc with its own
|
||||
// score made the aggregate row sets protocol-timing-dependent).
|
||||
if err := client2.FlushDB(ctx).Err(); err != nil {
|
||||
t.Fatalf("FlushDB failed: %v", err)
|
||||
}
|
||||
_, err := client2.FTCreate(ctx, "test-idx",
|
||||
&redis.FTCreateOptions{},
|
||||
&redis.FieldSchema{FieldName: "title", FieldType: redis.SearchFieldTypeText},
|
||||
@@ -282,6 +301,12 @@ func TestSearchCommandsRESP2AndRESP3Equivalence(t *testing.T) {
|
||||
|
||||
t.Run("FTSpellCheck", func(t *testing.T) {
|
||||
// Create index
|
||||
// Hermetic keyspace: earlier subtests drop only the index, and their
|
||||
// leftover hashes would be re-indexed here (a third doc with its own
|
||||
// score made the aggregate row sets protocol-timing-dependent).
|
||||
if err := client2.FlushDB(ctx).Err(); err != nil {
|
||||
t.Fatalf("FlushDB failed: %v", err)
|
||||
}
|
||||
_, err := client2.FTCreate(ctx, "test-idx",
|
||||
&redis.FTCreateOptions{},
|
||||
&redis.FieldSchema{FieldName: "title", FieldType: redis.SearchFieldTypeText},
|
||||
@@ -320,6 +345,12 @@ func TestSearchCommandsRESP2AndRESP3Equivalence(t *testing.T) {
|
||||
|
||||
t.Run("FTSynDump", func(t *testing.T) {
|
||||
// Create index
|
||||
// Hermetic keyspace: earlier subtests drop only the index, and their
|
||||
// leftover hashes would be re-indexed here (a third doc with its own
|
||||
// score made the aggregate row sets protocol-timing-dependent).
|
||||
if err := client2.FlushDB(ctx).Err(); err != nil {
|
||||
t.Fatalf("FlushDB failed: %v", err)
|
||||
}
|
||||
_, err := client2.FTCreate(ctx, "test-idx",
|
||||
&redis.FTCreateOptions{},
|
||||
&redis.FieldSchema{FieldName: "title", FieldType: redis.SearchFieldTypeText},
|
||||
@@ -364,9 +395,16 @@ func waitForIndexing(t *testing.T, client *redis.Client, index string, expectedD
|
||||
if err != nil {
|
||||
t.Fatalf("FTInfo failed: %v", err)
|
||||
}
|
||||
if info.NumDocs >= expectedDocs {
|
||||
// Exact count AND ingestion settled: the subtests query RESP2 and
|
||||
// RESP3 back to back and compare, so a doc landing (or a reindex
|
||||
// finishing) between the two queries skews one side — seen on a slow
|
||||
// CI runner as an extra transient empty row in the RESP3 aggregate.
|
||||
// The loop also SLEPT nowhere before, burning its 100 tries in
|
||||
// microseconds while the indexer had not even started.
|
||||
if info.NumDocs == expectedDocs && info.Indexing == 0 {
|
||||
return
|
||||
}
|
||||
time.Sleep(50 * time.Millisecond)
|
||||
}
|
||||
t.Fatalf("Timeout waiting for indexing")
|
||||
}
|
||||
+130
-117
@@ -56,22 +56,25 @@ func encodeFloat16Vector(vec []float32) []byte {
|
||||
|
||||
var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
ctx := context.TODO()
|
||||
var client *redis.Client
|
||||
var client redis.UniversalClient
|
||||
var rawClient *redis.Client
|
||||
var closeSubject func() error
|
||||
|
||||
BeforeEach(func() {
|
||||
client = reNewClient(2, false)
|
||||
Expect(client.FlushDB(ctx).Err()).NotTo(HaveOccurred())
|
||||
rawClient = reNewClient(2, false)
|
||||
client, closeSubject = newUniversalSubject(rawClient)
|
||||
Expect(rawClient.FlushDB(ctx).Err()).NotTo(HaveOccurred())
|
||||
})
|
||||
|
||||
AfterEach(func() {
|
||||
Expect(client.Close()).NotTo(HaveOccurred())
|
||||
Expect(closeSubject()).NotTo(HaveOccurred())
|
||||
})
|
||||
|
||||
It("should FTCreate and FTSearch WithScores", Label("search", "ftcreate", "ftsearch"), func() {
|
||||
val, err := client.FTCreate(ctx, "txt", &redis.FTCreateOptions{}, &redis.FieldSchema{FieldName: "txt", FieldType: redis.SearchFieldTypeText}).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "txt")
|
||||
WaitForIndexing(rawClient, "txt")
|
||||
client.HSet(ctx, "doc1", "txt", "foo baz")
|
||||
client.HSet(ctx, "doc2", "txt", "foo bar")
|
||||
res, err := client.FTSearchWithArgs(ctx, "txt", "foo ~bar", &redis.FTSearchOptions{WithScores: true}).Result()
|
||||
@@ -88,7 +91,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
val, err := client.FTCreate(ctx, "txt", &redis.FTCreateOptions{StopWords: []interface{}{"foo", "bar", "baz"}}, &redis.FieldSchema{FieldName: "txt", FieldType: redis.SearchFieldTypeText}).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "txt")
|
||||
WaitForIndexing(rawClient, "txt")
|
||||
client.HSet(ctx, "doc1", "txt", "foo baz")
|
||||
client.HSet(ctx, "doc2", "txt", "hello world")
|
||||
res1, err := client.FTSearchWithArgs(ctx, "txt", "foo bar", &redis.FTSearchOptions{NoContent: true}).Result()
|
||||
@@ -103,7 +106,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
val, err := client.FTCreate(ctx, "txt", &redis.FTCreateOptions{}, &redis.FieldSchema{FieldName: "txt", FieldType: redis.SearchFieldTypeText}, &redis.FieldSchema{FieldName: "num", FieldType: redis.SearchFieldTypeNumeric}, &redis.FieldSchema{FieldName: "loc", FieldType: redis.SearchFieldTypeGeo}).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "txt")
|
||||
WaitForIndexing(rawClient, "txt")
|
||||
client.HSet(ctx, "doc1", "txt", "foo bar", "num", 3.141, "loc", "-0.441,51.458")
|
||||
client.HSet(ctx, "doc2", "txt", "foo baz", "num", 2, "loc", "-0.1,51.2")
|
||||
res1, err := client.FTSearchWithArgs(ctx, "txt", "foo", &redis.FTSearchOptions{Filters: []redis.FTSearchFilter{{FieldName: "num", Min: 0, Max: 2}}, NoContent: true}).Result()
|
||||
@@ -134,7 +137,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
val, err := client.FTCreate(ctx, "num", &redis.FTCreateOptions{}, &redis.FieldSchema{FieldName: "txt", FieldType: redis.SearchFieldTypeText}, &redis.FieldSchema{FieldName: "num", FieldType: redis.SearchFieldTypeNumeric, Sortable: true}).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "num")
|
||||
WaitForIndexing(rawClient, "num")
|
||||
client.HSet(ctx, "doc1", "txt", "foo bar", "num", 1)
|
||||
client.HSet(ctx, "doc2", "txt", "foo baz", "num", 2)
|
||||
client.HSet(ctx, "doc3", "txt", "foo qux", "num", 3)
|
||||
@@ -168,7 +171,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
val, err := client.FTCreate(ctx, "txt", &redis.FTCreateOptions{}, &redis.FieldSchema{FieldName: "title", FieldType: redis.SearchFieldTypeText, Weight: 5}, &redis.FieldSchema{FieldName: "body", FieldType: redis.SearchFieldTypeText}).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "txt")
|
||||
WaitForIndexing(rawClient, "txt")
|
||||
client.HSet(ctx, "doc1", "title", "RediSearch", "body", "Redisearch implements a search engine on top of redis")
|
||||
res1, err := client.FTSearchWithArgs(ctx, "txt", "search engine", &redis.FTSearchOptions{NoContent: true, Verbatim: true, LimitOffset: 0, Limit: 5}).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
@@ -185,7 +188,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
val, err := client.FTCreate(ctx, "idx", &redis.FTCreateOptions{}, text1, text2, num, geo, tag).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx")
|
||||
WaitForIndexing(rawClient, "idx")
|
||||
client.HSet(ctx, "doc1", "field", "aaa", "text", "1", "numeric", 1, "geo", "1,1", "tag", "1")
|
||||
client.HSet(ctx, "doc2", "field", "aab", "text", "2", "numeric", 2, "geo", "2,2", "tag", "2")
|
||||
res1, err := client.FTSearch(ctx, "idx", "@text:aa*").Result()
|
||||
@@ -221,7 +224,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
val, err := client.FTCreate(ctx, "txt", &redis.FTCreateOptions{}, text1, text2, text3).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "txt")
|
||||
WaitForIndexing(rawClient, "txt")
|
||||
res1, err := client.FTExplain(ctx, "txt", "@f3:f3_val @f2:f2_val @f1:f1_val").Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(res1).ToNot(BeEmpty())
|
||||
@@ -234,11 +237,11 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
val1, err := client.FTCreate(ctx, "testAlias", &redis.FTCreateOptions{Prefix: []interface{}{"index1:"}}, text1).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val1).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "testAlias")
|
||||
WaitForIndexing(rawClient, "testAlias")
|
||||
val2, err := client.FTCreate(ctx, "testAlias2", &redis.FTCreateOptions{Prefix: []interface{}{"index2:"}}, text2).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val2).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "testAlias2")
|
||||
WaitForIndexing(rawClient, "testAlias2")
|
||||
|
||||
client.HSet(ctx, "index1:lonestar", "name", "lonestar")
|
||||
client.HSet(ctx, "index2:yogurt", "name", "yogurt")
|
||||
@@ -273,7 +276,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
val, err := client.FTCreate(ctx, "idx1", &redis.FTCreateOptions{}, &redis.FieldSchema{FieldName: "txt", FieldType: redis.SearchFieldTypeText, Sortable: true, NoStem: true}).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx1")
|
||||
WaitForIndexing(rawClient, "idx1")
|
||||
|
||||
resInfo, err := client.FTInfo(ctx, "idx1").Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
@@ -286,7 +289,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
val, err := client.FTCreate(ctx, "idx1", &redis.FTCreateOptions{}, &redis.FieldSchema{FieldName: "txt", FieldType: redis.SearchFieldTypeText}).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx1")
|
||||
WaitForIndexing(rawClient, "idx1")
|
||||
|
||||
resAlter, err := client.FTAlter(ctx, "idx1", false, []interface{}{"body", redis.SearchFieldTypeText.String()}).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
@@ -307,7 +310,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
val, err := client.FTCreate(ctx, "idx1", &redis.FTCreateOptions{}, text1, text2).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx1")
|
||||
WaitForIndexing(rawClient, "idx1")
|
||||
|
||||
client.HSet(ctx, "doc1", "f1", "some valid content", "f2", "this is sample text")
|
||||
client.HSet(ctx, "doc2", "f1", "very important", "f2", "lorem ipsum")
|
||||
@@ -354,7 +357,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
val, err := client.FTCreate(ctx, "idx1", &redis.FTCreateOptions{}, text1, text2).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx1")
|
||||
WaitForIndexing(rawClient, "idx1")
|
||||
|
||||
resDictAdd, err := client.FTDictAdd(ctx, "custom_dict", "item1", "item2", "item3").Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
@@ -378,7 +381,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
val, err := client.FTCreate(ctx, "idx1", &redis.FTCreateOptions{}, text1).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx1")
|
||||
WaitForIndexing(rawClient, "idx1")
|
||||
|
||||
client.HSet(ctx, "doc1", "name", "Jon")
|
||||
client.HSet(ctx, "doc2", "name", "John")
|
||||
@@ -388,12 +391,12 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
Expect(res1.Total).To(BeEquivalentTo(int64(1)))
|
||||
Expect(res1.Docs[0].Fields["name"]).To(BeEquivalentTo("Jon"))
|
||||
|
||||
client.FlushDB(ctx)
|
||||
rawClient.FlushDB(ctx)
|
||||
text2 := &redis.FieldSchema{FieldName: "name", FieldType: redis.SearchFieldTypeText, PhoneticMatcher: "dm:en"}
|
||||
val2, err := client.FTCreate(ctx, "idx1", &redis.FTCreateOptions{}, text2).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val2).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx1")
|
||||
WaitForIndexing(rawClient, "idx1")
|
||||
|
||||
client.HSet(ctx, "doc1", "name", "Jon")
|
||||
client.HSet(ctx, "doc2", "name", "John")
|
||||
@@ -415,7 +418,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
val, err := client.FTCreate(ctx, "idx1", &redis.FTCreateOptions{}, text1).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx1")
|
||||
WaitForIndexing(rawClient, "idx1")
|
||||
|
||||
client.HSet(ctx, "doc1", "description", "The quick brown fox jumps over the lazy dog")
|
||||
client.HSet(ctx, "doc2", "description", "Quick alice was beginning to get very tired of sitting by her quick sister on the bank, and of having nothing to do.")
|
||||
@@ -457,7 +460,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
val, err := client.FTCreate(ctx, "idx1", &redis.FTCreateOptions{}, text1).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx1")
|
||||
WaitForIndexing(rawClient, "idx1")
|
||||
|
||||
client.HSet(ctx, "doc1", "description", "The quick brown fox jumps over the lazy dog")
|
||||
client.HSet(ctx, "doc2", "description", "Quick alice was beginning to get very tired of sitting by her quick sister on the bank, and of having nothing to do.")
|
||||
@@ -514,7 +517,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
val, err := client.FTCreate(ctx, "idx1", &redis.FTCreateOptions{}, text1, text2, text3, num).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx1")
|
||||
WaitForIndexing(rawClient, "idx1")
|
||||
|
||||
client.HSet(ctx, "search", "title", "RediSearch",
|
||||
"body", "Redisearch implements a search engine on top of redis",
|
||||
@@ -618,7 +621,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
val, err := client.FTCreate(ctx, "idx1", &redis.FTCreateOptions{}, text1, text2).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx1")
|
||||
WaitForIndexing(rawClient, "idx1")
|
||||
|
||||
client.HSet(ctx, "doc1", "t1", "a", "t2", "b")
|
||||
client.HSet(ctx, "doc2", "t1", "b", "t2", "a")
|
||||
@@ -659,7 +662,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
val, err := client.FTCreate(ctx, "idx1", &redis.FTCreateOptions{}, text1, text2).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx1")
|
||||
WaitForIndexing(rawClient, "idx1")
|
||||
|
||||
client.HSet(ctx, "doc1", "t1", "hello", "t2", "world")
|
||||
|
||||
@@ -701,7 +704,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
val, err := client.FTCreate(ctx, "idx1", &redis.FTCreateOptions{OnHash: true, Prefix: []interface{}{"product:"}}, title, description).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx1")
|
||||
WaitForIndexing(rawClient, "idx1")
|
||||
|
||||
client.HSet(ctx, "product:1", "title", "New Gaming Laptop", "description", "this is not a desktop")
|
||||
client.HSet(ctx, "product:2", "title", "Super Old Not Gaming Laptop", "description", "this laptop is not a new laptop but it is a laptop")
|
||||
@@ -757,7 +760,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
val, err := client.FTCreate(ctx, "idx1", &redis.FTCreateOptions{}, text1, num1).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx1")
|
||||
WaitForIndexing(rawClient, "idx1")
|
||||
|
||||
// 6 feb
|
||||
client.HSet(ctx, "doc1", "PrimaryKey", "9::362330", "CreatedDateTimeUTC", "1738823999")
|
||||
@@ -794,7 +797,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
val, err := client.FTCreate(ctx, "idx1", &redis.FTCreateOptions{}, text1, num1).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx1")
|
||||
WaitForIndexing(rawClient, "idx1")
|
||||
|
||||
client.HSet(ctx, "doc1", "PrimaryKey", "9::362330", "CreatedDateTimeUTC", "637387878524969984")
|
||||
client.HSet(ctx, "doc2", "PrimaryKey", "9::362329", "CreatedDateTimeUTC", "637387875859270016")
|
||||
@@ -813,7 +816,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
val, err := client.FTCreate(ctx, "idx1", &redis.FTCreateOptions{}, text1, num1).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx1")
|
||||
WaitForIndexing(rawClient, "idx1")
|
||||
|
||||
client.HSet(ctx, "doc1", "name", "bar", "age", "25")
|
||||
client.HSet(ctx, "doc2", "name", "foo", "age", "19")
|
||||
@@ -1065,7 +1068,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
val, err := client.FTCreate(ctx, "idx1", &redis.FTCreateOptions{SkipInitialScan: true}, text1).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx1")
|
||||
WaitForIndexing(rawClient, "idx1")
|
||||
|
||||
res, err := client.FTSearch(ctx, "idx1", "@foo:bar").Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
@@ -1078,7 +1081,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
val, err := client.FTCreate(ctx, "idx1", &redis.FTCreateOptions{OnJSON: true, Prefix: []interface{}{"king:"}}, text1).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx1")
|
||||
WaitForIndexing(rawClient, "idx1")
|
||||
|
||||
client.JSONSet(ctx, "king:1", "$", `{"name": "henry"}`)
|
||||
client.JSONSet(ctx, "king:2", "$", `{"name": "james"}`)
|
||||
@@ -1097,7 +1100,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
val, err := client.FTCreate(ctx, "idx1", &redis.FTCreateOptions{OnJSON: true}, text1, num1).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx1")
|
||||
WaitForIndexing(rawClient, "idx1")
|
||||
|
||||
client.JSONSet(ctx, "doc:1", "$", `{"name": "Jon", "age": 25}`)
|
||||
|
||||
@@ -1115,7 +1118,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
val, err := client.FTCreate(ctx, "idx1", &redis.FTCreateOptions{}, tag1).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx1")
|
||||
WaitForIndexing(rawClient, "idx1")
|
||||
|
||||
client.HSet(ctx, "1", "t", "HELLO")
|
||||
client.HSet(ctx, "2", "t", "hello")
|
||||
@@ -1134,7 +1137,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
val, err = client.FTCreate(ctx, "idx1", &redis.FTCreateOptions{}, tag2).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx1")
|
||||
WaitForIndexing(rawClient, "idx1")
|
||||
|
||||
res, err = client.FTSearch(ctx, "idx1", "@t:{HELLO}").Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
@@ -1153,7 +1156,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
val, err := client.FTCreate(ctx, "idx1", &redis.FTCreateOptions{OnJSON: true}, text1, num1).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx1")
|
||||
WaitForIndexing(rawClient, "idx1")
|
||||
|
||||
res, err := client.FTSearchWithArgs(ctx, "idx1", "*", &redis.FTSearchOptions{Return: []redis.FTSearchReturn{{FieldName: "$.t", As: "txt"}}}).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
@@ -1175,7 +1178,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
val, err := client.FTCreate(ctx, "idx1", &redis.FTCreateOptions{OnHash: true}, text1, text2).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx1")
|
||||
WaitForIndexing(rawClient, "idx1")
|
||||
|
||||
resSynUpdate, err := client.FTSynUpdateWithArgs(ctx, "idx1", "id1", &redis.FTSynUpdateOptions{SkipInitialScan: true}, []interface{}{"boy", "child", "offspring"}).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
@@ -1201,7 +1204,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
val, err := client.FTCreate(ctx, "idx1", &redis.FTCreateOptions{OnHash: true}, text1, text2).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx1")
|
||||
WaitForIndexing(rawClient, "idx1")
|
||||
|
||||
resSynUpdate, err := client.FTSynUpdate(ctx, "idx1", "id1", []interface{}{"boy", "child", "offspring"}).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
@@ -1239,7 +1242,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
val, err := client.FTCreate(ctx, "idx1", &redis.FTCreateOptions{OnJSON: true, Prefix: []interface{}{"king:"}}, text1, num1).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx1")
|
||||
WaitForIndexing(rawClient, "idx1")
|
||||
|
||||
client.JSONSet(ctx, "king:1", "$", `{"name": "henry", "num": 42}`)
|
||||
client.JSONSet(ctx, "king:2", "$", `{"name": "james", "num": 3.14}`)
|
||||
@@ -1263,7 +1266,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
val, err := client.FTCreate(ctx, "idx1", &redis.FTCreateOptions{OnJSON: true, Prefix: []interface{}{"king:"}}, tag1).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx1")
|
||||
WaitForIndexing(rawClient, "idx1")
|
||||
|
||||
client.JSONSet(ctx, "king:1", "$", `{"name": "henry", "country": {"name": "england"}}`)
|
||||
|
||||
@@ -1281,7 +1284,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
val, err := client.FTCreate(ctx, "idx1", &redis.FTCreateOptions{OnJSON: true}, text1, text2).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx1")
|
||||
WaitForIndexing(rawClient, "idx1")
|
||||
|
||||
client.JSONSet(ctx, "doc:1", "$", `{"prod:name": "RediSearch"}`)
|
||||
|
||||
@@ -1310,7 +1313,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
&redis.FieldSchema{FieldName: "v", FieldType: redis.SearchFieldTypeVector, VectorArgs: &redis.FTVectorArgs{HNSWOptions: hnswOptions}}).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx1")
|
||||
WaitForIndexing(rawClient, "idx1")
|
||||
|
||||
client.HSet(ctx, "a", "v", "aaaaaaaa")
|
||||
client.HSet(ctx, "b", "v", "aaaabaaa")
|
||||
@@ -1335,7 +1338,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
&redis.FieldSchema{FieldName: "v", FieldType: redis.SearchFieldTypeVector, VectorArgs: &redis.FTVectorArgs{HNSWOptions: hnswOptions}}).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx1")
|
||||
WaitForIndexing(rawClient, "idx1")
|
||||
|
||||
client.HSet(ctx, "a", "v", "aaaaaaaa")
|
||||
client.HSet(ctx, "b", "v", "aaaabaaa")
|
||||
@@ -1360,7 +1363,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
&redis.FieldSchema{FieldName: "v", FieldType: redis.SearchFieldTypeVector, VectorArgs: &redis.FTVectorArgs{HNSWOptions: hnswOptions}}).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx1")
|
||||
WaitForIndexing(rawClient, "idx1")
|
||||
|
||||
client.HSet(ctx, "a", "v", "aaaaaaaa")
|
||||
client.HSet(ctx, "b", "v", "aaaabaaa")
|
||||
@@ -1381,7 +1384,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
val, err := client.FTCreate(ctx, "idx1", &redis.FTCreateOptions{}, &redis.FieldSchema{FieldName: "name", FieldType: redis.SearchFieldTypeText}).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx1")
|
||||
WaitForIndexing(rawClient, "idx1")
|
||||
|
||||
client.HSet(ctx, "doc1", "name", "Alice")
|
||||
client.HSet(ctx, "doc2", "name", "Bob")
|
||||
@@ -1399,7 +1402,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
val, err := client.FTCreate(ctx, "idx1", &redis.FTCreateOptions{}, &redis.FieldSchema{FieldName: "numval", FieldType: redis.SearchFieldTypeNumeric}).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx1")
|
||||
WaitForIndexing(rawClient, "idx1")
|
||||
|
||||
client.HSet(ctx, "doc1", "numval", 101)
|
||||
client.HSet(ctx, "doc2", "numval", 102)
|
||||
@@ -1417,7 +1420,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
val, err := client.FTCreate(ctx, "idx1", &redis.FTCreateOptions{}, &redis.FieldSchema{FieldName: "g", FieldType: redis.SearchFieldTypeGeo}).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx1")
|
||||
WaitForIndexing(rawClient, "idx1")
|
||||
|
||||
client.HSet(ctx, "doc1", "g", "29.69465, 34.95126")
|
||||
client.HSet(ctx, "doc2", "g", "29.69350, 34.94737")
|
||||
@@ -1472,7 +1475,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
val, err := client.FTCreate(ctx, "idx1", &redis.FTCreateOptions{}, &redis.FieldSchema{FieldName: "txt", FieldType: redis.SearchFieldTypeText}).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx1")
|
||||
WaitForIndexing(rawClient, "idx1")
|
||||
|
||||
res, err := client.FTInfo(ctx, "idx1").Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
@@ -1486,7 +1489,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
val, err = client.FTCreate(ctx, "idx1", &redis.FTCreateOptions{}, &redis.FieldSchema{FieldName: "txt", FieldType: redis.SearchFieldTypeText, WithSuffixtrie: true}).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx1")
|
||||
WaitForIndexing(rawClient, "idx1")
|
||||
|
||||
res, err = client.FTInfo(ctx, "idx1").Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
@@ -1500,7 +1503,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
val, err = client.FTCreate(ctx, "idx1", &redis.FTCreateOptions{}, &redis.FieldSchema{FieldName: "t", FieldType: redis.SearchFieldTypeTag, WithSuffixtrie: true}).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx1")
|
||||
WaitForIndexing(rawClient, "idx1")
|
||||
|
||||
res, err = client.FTInfo(ctx, "idx1").Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
@@ -1586,7 +1589,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
val, err = client.FTCreate(ctx, "idx_hash", ftCreateOptions, schema...).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(Equal("OK"))
|
||||
WaitForIndexing(client, "idx_hash")
|
||||
WaitForIndexing(rawClient, "idx_hash")
|
||||
|
||||
ftSearchOptions := &redis.FTSearchOptions{
|
||||
DialectVersion: 4,
|
||||
@@ -1617,7 +1620,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
val, err := client.FTCreate(ctx, "idx1", &redis.FTCreateOptions{}, &redis.FieldSchema{FieldName: "geom", FieldType: redis.SearchFieldTypeGeoShape, GeoShapeFieldType: "FLAT"}).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx1")
|
||||
WaitForIndexing(rawClient, "idx1")
|
||||
|
||||
client.HSet(ctx, "small", "geom", "POLYGON((1 1, 1 100, 100 100, 100 1, 1 1))")
|
||||
client.HSet(ctx, "large", "geom", "POLYGON((1 1, 1 200, 200 200, 200 1, 1 1))")
|
||||
@@ -1648,7 +1651,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "index")
|
||||
WaitForIndexing(rawClient, "index")
|
||||
})
|
||||
|
||||
It("should test geoshapes query intersects and disjoint", Label("NonRedisEnterprise"), func() {
|
||||
@@ -1728,7 +1731,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
&redis.FieldSchema{FieldName: "description", FieldType: redis.SearchFieldTypeText, IndexMissing: true}).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx1")
|
||||
WaitForIndexing(rawClient, "idx1")
|
||||
|
||||
client.HSet(ctx, "property:1", map[string]interface{}{
|
||||
"title": "Luxury Villa in Malibu",
|
||||
@@ -1773,7 +1776,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
&redis.FieldSchema{FieldName: "description", FieldType: redis.SearchFieldTypeText, IndexEmpty: true}).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx1")
|
||||
WaitForIndexing(rawClient, "idx1")
|
||||
|
||||
client.HSet(ctx, "property:1", map[string]interface{}{
|
||||
"title": "Luxury Villa in Malibu",
|
||||
@@ -1837,7 +1840,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx1")
|
||||
WaitForIndexing(rawClient, "idx1")
|
||||
|
||||
// Insert vectors in int8 and uint8 format
|
||||
client.HSet(ctx, "doc1", "int8_vector", "\x01\x02", "uint8_vector", "\x01\x02")
|
||||
@@ -1880,7 +1883,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
&redis.FTCreateOptions{OnHash: true, Prefix: []interface{}{"doc:"}},
|
||||
&redis.FieldSchema{FieldName: "vector", FieldType: redis.SearchFieldTypeVector, VectorArgs: &redis.FTVectorArgs{FlatOptions: vecField}}).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
WaitForIndexing(client, "idx_vec")
|
||||
WaitForIndexing(rawClient, "idx_vec")
|
||||
|
||||
bigPos := []float32{1e38, 1e38}
|
||||
bigNeg := []float32{-1e38, -1e38}
|
||||
@@ -1930,7 +1933,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
&redis.FieldSchema{FieldName: "v", FieldType: redis.SearchFieldTypeVector, VectorArgs: &redis.FTVectorArgs{VamanaOptions: vamanaOptions}}).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx1")
|
||||
WaitForIndexing(rawClient, "idx1")
|
||||
|
||||
client.HSet(ctx, "a", "v", "aaaaaaaa")
|
||||
client.HSet(ctx, "b", "v", "aaaabaaa")
|
||||
@@ -1962,7 +1965,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
&redis.FieldSchema{FieldName: "v", FieldType: redis.SearchFieldTypeVector, VectorArgs: &redis.FTVectorArgs{VamanaOptions: vamanaOptions}}).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx1")
|
||||
WaitForIndexing(rawClient, "idx1")
|
||||
})
|
||||
|
||||
It("should FTCreate VECTOR with VAMANA algorithm - advanced parameters", Label("search", "ftcreate"), func() {
|
||||
@@ -1983,7 +1986,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
&redis.FieldSchema{FieldName: "v", FieldType: redis.SearchFieldTypeVector, VectorArgs: &redis.FTVectorArgs{VamanaOptions: vamanaOptions}}).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx1")
|
||||
WaitForIndexing(rawClient, "idx1")
|
||||
})
|
||||
|
||||
It("should fail FTCreate VECTOR with VAMANA - missing required parameters", Label("search", "ftcreate"), func() {
|
||||
@@ -2052,7 +2055,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
&redis.FieldSchema{FieldName: "v", FieldType: redis.SearchFieldTypeVector, VectorArgs: &redis.FTVectorArgs{VamanaOptions: vamanaOptions}}).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx1")
|
||||
WaitForIndexing(rawClient, "idx1")
|
||||
|
||||
// L2 distance test vectors
|
||||
vectors := [][]float32{
|
||||
@@ -2091,7 +2094,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
&redis.FieldSchema{FieldName: "v", FieldType: redis.SearchFieldTypeVector, VectorArgs: &redis.FTVectorArgs{VamanaOptions: vamanaOptions}}).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx1")
|
||||
WaitForIndexing(rawClient, "idx1")
|
||||
|
||||
vectors := [][]float32{
|
||||
{1.0, 0.0, 0.0},
|
||||
@@ -2129,7 +2132,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
&redis.FieldSchema{FieldName: "v", FieldType: redis.SearchFieldTypeVector, VectorArgs: &redis.FTVectorArgs{VamanaOptions: vamanaOptions}}).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx1")
|
||||
WaitForIndexing(rawClient, "idx1")
|
||||
|
||||
vectors := [][]float32{
|
||||
{1.0, 2.0, 3.0},
|
||||
@@ -2167,7 +2170,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
&redis.FieldSchema{FieldName: "v", FieldType: redis.SearchFieldTypeVector, VectorArgs: &redis.FTVectorArgs{VamanaOptions: vamanaOptions}}).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx1")
|
||||
WaitForIndexing(rawClient, "idx1")
|
||||
|
||||
vectors := [][]float32{
|
||||
{1.0, 2.0, 3.0, 4.0},
|
||||
@@ -2205,7 +2208,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
&redis.FieldSchema{FieldName: "v", FieldType: redis.SearchFieldTypeVector, VectorArgs: &redis.FTVectorArgs{VamanaOptions: vamanaOptions}}).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx1")
|
||||
WaitForIndexing(rawClient, "idx1")
|
||||
|
||||
vectors := [][]float32{
|
||||
{1.5, 2.5, 3.5, 4.5},
|
||||
@@ -2240,7 +2243,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
&redis.FieldSchema{FieldName: "v", FieldType: redis.SearchFieldTypeVector, VectorArgs: &redis.FTVectorArgs{VamanaOptions: vamanaOptions}}).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx1")
|
||||
WaitForIndexing(rawClient, "idx1")
|
||||
|
||||
vectors := [][]float32{
|
||||
{1.0, 2.0, 3.0, 4.0},
|
||||
@@ -2275,7 +2278,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
&redis.FieldSchema{FieldName: "v", FieldType: redis.SearchFieldTypeVector, VectorArgs: &redis.FTVectorArgs{VamanaOptions: vamanaOptions}}).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx1")
|
||||
WaitForIndexing(rawClient, "idx1")
|
||||
|
||||
client.HSet(ctx, "a", "v", "aaaaaaaa")
|
||||
client.HSet(ctx, "b", "v", "aaaabaaa")
|
||||
@@ -2305,7 +2308,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
&redis.FieldSchema{FieldName: "v", FieldType: redis.SearchFieldTypeVector, VectorArgs: &redis.FTVectorArgs{VamanaOptions: vamanaOptions}}).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx1")
|
||||
WaitForIndexing(rawClient, "idx1")
|
||||
|
||||
vectors := make([][]float32, 20)
|
||||
for i := 0; i < 20; i++ {
|
||||
@@ -2344,7 +2347,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
&redis.FieldSchema{FieldName: "v16", FieldType: redis.SearchFieldTypeVector, VectorArgs: &redis.FTVectorArgs{VamanaOptions: vamanaOptions16}}).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx16")
|
||||
WaitForIndexing(rawClient, "idx16")
|
||||
|
||||
// Test FLOAT32 with LVQ8
|
||||
vamanaOptions32 := &redis.FTVamanaOptions{
|
||||
@@ -2359,7 +2362,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
&redis.FieldSchema{FieldName: "v32", FieldType: redis.SearchFieldTypeVector, VectorArgs: &redis.FTVectorArgs{VamanaOptions: vamanaOptions32}}).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx32")
|
||||
WaitForIndexing(rawClient, "idx32")
|
||||
|
||||
// Add data to both indices
|
||||
for i := 0; i < 15; i++ {
|
||||
@@ -2407,7 +2410,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
&redis.FieldSchema{FieldName: "v", FieldType: redis.SearchFieldTypeVector, VectorArgs: &redis.FTVectorArgs{VamanaOptions: vamanaOptions}}).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx1")
|
||||
WaitForIndexing(rawClient, "idx1")
|
||||
|
||||
vectors := make([][]float32, 20)
|
||||
for i := 0; i < 20; i++ {
|
||||
@@ -2443,7 +2446,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
&redis.FieldSchema{FieldName: "v", FieldType: redis.SearchFieldTypeVector, VectorArgs: &redis.FTVectorArgs{VamanaOptions: vamanaOptions}}).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx1")
|
||||
WaitForIndexing(rawClient, "idx1")
|
||||
|
||||
vectors := make([][]float32, 25)
|
||||
for i := 0; i < 25; i++ {
|
||||
@@ -2479,7 +2482,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
&redis.FieldSchema{FieldName: "v", FieldType: redis.SearchFieldTypeVector, VectorArgs: &redis.FTVectorArgs{VamanaOptions: vamanaOptions}}).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx1")
|
||||
WaitForIndexing(rawClient, "idx1")
|
||||
|
||||
vectors := make([][]float32, 30)
|
||||
for i := 0; i < 30; i++ {
|
||||
@@ -2520,7 +2523,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
&redis.FieldSchema{FieldName: "v", FieldType: redis.SearchFieldTypeVector, VectorArgs: &redis.FTVectorArgs{VamanaOptions: vamanaOptions}}).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx1")
|
||||
WaitForIndexing(rawClient, "idx1")
|
||||
|
||||
vectors := make([][]float32, 15)
|
||||
for i := 0; i < 15; i++ {
|
||||
@@ -2551,7 +2554,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
}).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "testIdx")
|
||||
WaitForIndexing(rawClient, "testIdx")
|
||||
|
||||
client.HSet(ctx, "doc1", "txt", "hello world")
|
||||
|
||||
@@ -2571,7 +2574,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
}).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "txns")
|
||||
WaitForIndexing(rawClient, "txns")
|
||||
|
||||
client.HSet(ctx, "doc1", "dummy", "dummy")
|
||||
|
||||
@@ -2595,7 +2598,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
}).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx")
|
||||
WaitForIndexing(rawClient, "idx")
|
||||
|
||||
client.HSet(ctx, "doc1", "n", 0)
|
||||
|
||||
@@ -2618,7 +2621,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "aggTestAvg")
|
||||
WaitForIndexing(rawClient, "aggTestAvg")
|
||||
|
||||
client.HSet(ctx, "doc1", "grp", "g1")
|
||||
|
||||
@@ -2644,7 +2647,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "aggTestCount")
|
||||
WaitForIndexing(rawClient, "aggTestCount")
|
||||
|
||||
client.HSet(ctx, "doc1", "grp", "g1")
|
||||
|
||||
@@ -2670,7 +2673,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "aggTestSum")
|
||||
WaitForIndexing(rawClient, "aggTestSum")
|
||||
|
||||
client.HSet(ctx, "doc1", "grp", "g1")
|
||||
|
||||
@@ -2695,7 +2698,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "aggExpired")
|
||||
WaitForIndexing(rawClient, "aggExpired")
|
||||
|
||||
for i := 1; i <= 15; i++ {
|
||||
key := fmt.Sprintf("doc%d", i)
|
||||
@@ -2731,7 +2734,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "aggTimeoutHeavy")
|
||||
WaitForIndexing(rawClient, "aggTimeoutHeavy")
|
||||
|
||||
const totalDocs = 100000
|
||||
for i := 0; i < totalDocs; i++ {
|
||||
@@ -2740,7 +2743,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
}
|
||||
// default behaviour was changed in 8.0.1, set to fail to validate the timeout was triggered
|
||||
err = client.ConfigSet(ctx, "search-on-timeout", "fail").Err()
|
||||
err = rawClient.ConfigSet(ctx, "search-on-timeout", "fail").Err()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
|
||||
options := &redis.FTAggregateOptions{
|
||||
@@ -2762,7 +2765,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "aggTestCountDistinct")
|
||||
WaitForIndexing(rawClient, "aggTestCountDistinct")
|
||||
|
||||
client.HSet(ctx, "doc1", "grp", "g1")
|
||||
|
||||
@@ -2788,7 +2791,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "aggTestCountDistinctIsh")
|
||||
WaitForIndexing(rawClient, "aggTestCountDistinctIsh")
|
||||
|
||||
_, err = client.HSet(ctx, "doc1", "grp", "g1").Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
@@ -2813,7 +2816,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "scoringTest")
|
||||
WaitForIndexing(rawClient, "scoringTest")
|
||||
|
||||
_, err = client.HSet(ctx, "doc1", "description", "red apple").Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
@@ -2840,7 +2843,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "aggTestStddev")
|
||||
WaitForIndexing(rawClient, "aggTestStddev")
|
||||
|
||||
_, err = client.HSet(ctx, "doc1", "grp", "g1").Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
@@ -2867,7 +2870,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "aggTestQuantile")
|
||||
WaitForIndexing(rawClient, "aggTestQuantile")
|
||||
|
||||
_, err = client.HSet(ctx, "doc1", "grp", "g1").Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
@@ -2893,7 +2896,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "aggTestFirstValue")
|
||||
WaitForIndexing(rawClient, "aggTestFirstValue")
|
||||
|
||||
_, err = client.HSet(ctx, "doc1", "grp", "g1").Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
@@ -2918,14 +2921,14 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx1")
|
||||
WaitForIndexing(rawClient, "idx1")
|
||||
|
||||
val, err = client.FTCreate(ctx, "idx2", &redis.FTCreateOptions{},
|
||||
&redis.FieldSchema{FieldName: "name", FieldType: redis.SearchFieldTypeText},
|
||||
).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx2")
|
||||
WaitForIndexing(rawClient, "idx2")
|
||||
|
||||
_, err = client.FTAliasAdd(ctx, "idx2", "idx1").Result()
|
||||
Expect(err).To(HaveOccurred())
|
||||
@@ -2938,7 +2941,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "txtIndex")
|
||||
WaitForIndexing(rawClient, "txtIndex")
|
||||
|
||||
_, err = client.HSet(ctx, "doc1", "txt", "hello world").Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
@@ -3024,7 +3027,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
Expect(err).To(HaveOccurred())
|
||||
}
|
||||
|
||||
val, err := client.ConfigGet(ctx, "*").Result()
|
||||
val, err := rawClient.ConfigGet(ctx, "*").Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
// Since FT.CONFIG is deprecated since redis 8, use CONFIG instead with new search parameters.
|
||||
keys := make([]string, 0, len(val))
|
||||
@@ -3042,7 +3045,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "aggTestMinMax")
|
||||
WaitForIndexing(rawClient, "aggTestMinMax")
|
||||
|
||||
_, err = client.HSet(ctx, "doc1", "grp", "g1").Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
@@ -3077,7 +3080,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
&redis.FieldSchema{FieldName: "v", FieldType: redis.SearchFieldTypeVector, VectorArgs: &redis.FTVectorArgs{VamanaOptions: vamanaOptions}}).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx1")
|
||||
WaitForIndexing(rawClient, "idx1")
|
||||
|
||||
vectors := make([][]float32, 20)
|
||||
for i := 0; i < 20; i++ {
|
||||
@@ -3116,7 +3119,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
&redis.FieldSchema{FieldName: "v", FieldType: redis.SearchFieldTypeVector, VectorArgs: &redis.FTVectorArgs{VamanaOptions: vamanaOptions}}).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx1")
|
||||
WaitForIndexing(rawClient, "idx1")
|
||||
|
||||
vectors := make([][]float32, 20)
|
||||
for i := 0; i < 20; i++ {
|
||||
@@ -3165,7 +3168,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
&redis.FieldSchema{FieldName: "v", FieldType: redis.SearchFieldTypeVector, VectorArgs: &redis.FTVectorArgs{VamanaOptions: vamanaOptions}}).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, indexName)
|
||||
WaitForIndexing(rawClient, indexName)
|
||||
|
||||
for i := 0; i < 15; i++ {
|
||||
vec := make([]float32, 8)
|
||||
@@ -3212,7 +3215,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
&redis.FieldSchema{FieldName: "v", FieldType: redis.SearchFieldTypeVector, VectorArgs: &redis.FTVectorArgs{VamanaOptions: vamanaOptions}}).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, indexName)
|
||||
WaitForIndexing(rawClient, indexName)
|
||||
|
||||
for i := 0; i < 15; i++ {
|
||||
vec := make([]float32, 8)
|
||||
@@ -3261,7 +3264,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
&redis.FieldSchema{FieldName: "v", FieldType: redis.SearchFieldTypeVector, VectorArgs: &redis.FTVectorArgs{VamanaOptions: vamanaOptions}}).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, indexName)
|
||||
WaitForIndexing(rawClient, indexName)
|
||||
|
||||
for i := 0; i < 10; i++ {
|
||||
vec := make([]float32, 8)
|
||||
@@ -3313,7 +3316,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
&redis.FieldSchema{FieldName: "v", FieldType: redis.SearchFieldTypeVector, VectorArgs: &redis.FTVectorArgs{VamanaOptions: vamanaOptions}}).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, indexName)
|
||||
WaitForIndexing(rawClient, indexName)
|
||||
|
||||
for i := 0; i < 15; i++ {
|
||||
vec := make([]float32, 8)
|
||||
@@ -3365,7 +3368,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
&redis.FieldSchema{FieldName: "v", FieldType: redis.SearchFieldTypeVector, VectorArgs: &redis.FTVectorArgs{VamanaOptions: vamanaOptions}}).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx1")
|
||||
WaitForIndexing(rawClient, "idx1")
|
||||
|
||||
vectors := make([][]float32, 20)
|
||||
for i := 0; i < 20; i++ {
|
||||
@@ -3404,7 +3407,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
&redis.FieldSchema{FieldName: "v", FieldType: redis.SearchFieldTypeVector, VectorArgs: &redis.FTVectorArgs{VamanaOptions: vamanaOptions}}).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx1")
|
||||
WaitForIndexing(rawClient, "idx1")
|
||||
|
||||
vectors := make([][]float32, 20)
|
||||
for i := 0; i < 20; i++ {
|
||||
@@ -3453,7 +3456,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
&redis.FieldSchema{FieldName: "v", FieldType: redis.SearchFieldTypeVector, VectorArgs: &redis.FTVectorArgs{VamanaOptions: vamanaOptions}}).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, indexName)
|
||||
WaitForIndexing(rawClient, indexName)
|
||||
|
||||
// Add test data
|
||||
for i := 0; i < 15; i++ {
|
||||
@@ -3503,7 +3506,7 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx_vector")
|
||||
WaitForIndexing(rawClient, "idx_vector")
|
||||
|
||||
// Get FT.INFO
|
||||
resInfo, err := client.FTInfo(ctx, "idx_vector").Result()
|
||||
@@ -3609,10 +3612,13 @@ var _ = Describe("RediSearch commands Resp 2", Label("search"), func() {
|
||||
// Hybrid Search Tests
|
||||
var _ = Describe("FT.HYBRID Commands", func() {
|
||||
ctx := context.TODO()
|
||||
var client *redis.Client
|
||||
var client redis.UniversalClient
|
||||
var rawClient *redis.Client
|
||||
var closeSubject func() error
|
||||
|
||||
BeforeEach(func() {
|
||||
client = reNewClient(2, false)
|
||||
rawClient = reNewClient(2, false)
|
||||
client, closeSubject = newUniversalSubject(rawClient)
|
||||
// Create index with text, numeric, tag fields and vector fields
|
||||
err := client.FTCreate(ctx, "hybrid_idx", &redis.FTCreateOptions{},
|
||||
&redis.FieldSchema{FieldName: "description", FieldType: redis.SearchFieldTypeText},
|
||||
@@ -3643,7 +3649,7 @@ var _ = Describe("FT.HYBRID Commands", func() {
|
||||
},
|
||||
}).Err()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
WaitForIndexing(client, "hybrid_idx")
|
||||
WaitForIndexing(rawClient, "hybrid_idx")
|
||||
|
||||
// Add test data
|
||||
items := []struct {
|
||||
@@ -3678,6 +3684,7 @@ var _ = Describe("FT.HYBRID Commands", func() {
|
||||
AfterEach(func() {
|
||||
err := client.FTDropIndex(ctx, "hybrid_idx").Err()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(closeSubject()).NotTo(HaveOccurred())
|
||||
})
|
||||
|
||||
It("should perform basic hybrid search", Label("search", "fthybrid"), func() {
|
||||
@@ -4381,17 +4388,23 @@ var _ = Describe("RediSearch FT.Config with Resp2 and Resp3", Label("search", "N
|
||||
|
||||
var _ = Describe("RediSearch commands Resp 3", Label("search"), func() {
|
||||
ctx := context.TODO()
|
||||
var client *redis.Client
|
||||
var client redis.UniversalClient
|
||||
var rawClient *redis.Client
|
||||
var closeSubject func() error
|
||||
var client2 *redis.Client
|
||||
|
||||
BeforeEach(func() {
|
||||
client = reNewClient(3, true)
|
||||
rawClient = reNewClient(3, true)
|
||||
client, closeSubject = newUniversalSubject(rawClient)
|
||||
client2 = reNewClient(3, false)
|
||||
Expect(client.FlushDB(ctx).Err()).NotTo(HaveOccurred())
|
||||
Expect(rawClient.FlushDB(ctx).Err()).NotTo(HaveOccurred())
|
||||
})
|
||||
|
||||
AfterEach(func() {
|
||||
Expect(client.Close()).NotTo(HaveOccurred())
|
||||
Expect(closeSubject()).NotTo(HaveOccurred())
|
||||
// client2 is spec-local like the subject; leaking it across specs
|
||||
// accumulates pools and goroutines over the suite.
|
||||
Expect(client2.Close()).NotTo(HaveOccurred())
|
||||
})
|
||||
|
||||
It("should handle FTAggregate with RESP3", Label("search", "ftcreate", "ftaggregate"), func() {
|
||||
@@ -4400,11 +4413,11 @@ var _ = Describe("RediSearch commands Resp 3", Label("search"), func() {
|
||||
val, err := client.FTCreate(ctx, "idx1", &redis.FTCreateOptions{}, text1, num1).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx1")
|
||||
WaitForIndexing(rawClient, "idx1")
|
||||
|
||||
client.HSet(ctx, "doc1", "PrimaryKey", "9::362330", "CreatedDateTimeUTC", "637387878524969984")
|
||||
client.HSet(ctx, "doc2", "PrimaryKey", "9::362329", "CreatedDateTimeUTC", "637387875859270016")
|
||||
WaitForIndexing(client, "idx1")
|
||||
WaitForIndexing(rawClient, "idx1")
|
||||
|
||||
options := &redis.FTAggregateOptions{Apply: []redis.FTAggregateApply{{Field: "@CreatedDateTimeUTC * 10", As: "CreatedDateTimeUTC"}}}
|
||||
|
||||
@@ -4440,7 +4453,7 @@ var _ = Describe("RediSearch commands Resp 3", Label("search"), func() {
|
||||
val, err := client.FTCreate(ctx, "idx1", &redis.FTCreateOptions{}, &redis.FieldSchema{FieldName: "txt", FieldType: redis.SearchFieldTypeText, Sortable: true, NoStem: true}).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx1")
|
||||
WaitForIndexing(rawClient, "idx1")
|
||||
|
||||
// Test with UnstableResp3 true - both RawResult and Result should work
|
||||
resInfo, err := client.FTInfo(ctx, "idx1").RawResult()
|
||||
@@ -4475,7 +4488,7 @@ var _ = Describe("RediSearch commands Resp 3", Label("search"), func() {
|
||||
val, err := client.FTCreate(ctx, "idx1", &redis.FTCreateOptions{}, text1, text2).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx1")
|
||||
WaitForIndexing(rawClient, "idx1")
|
||||
|
||||
client.HSet(ctx, "doc1", "f1", "some valid content", "f2", "this is sample text")
|
||||
client.HSet(ctx, "doc2", "f1", "very important", "f2", "lorem ipsum")
|
||||
@@ -4511,7 +4524,7 @@ var _ = Describe("RediSearch commands Resp 3", Label("search"), func() {
|
||||
val, err := client.FTCreate(ctx, "txt", &redis.FTCreateOptions{StopWords: []interface{}{"foo", "bar", "baz"}}, &redis.FieldSchema{FieldName: "txt", FieldType: redis.SearchFieldTypeText}).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "txt")
|
||||
WaitForIndexing(rawClient, "txt")
|
||||
client.HSet(ctx, "doc1", "txt", "foo baz")
|
||||
client.HSet(ctx, "doc2", "txt", "hello world")
|
||||
|
||||
@@ -4555,7 +4568,7 @@ var _ = Describe("RediSearch commands Resp 3", Label("search"), func() {
|
||||
val, err := client.FTCreate(ctx, "idx1", &redis.FTCreateOptions{OnHash: true}, text1, text2).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "idx1")
|
||||
WaitForIndexing(rawClient, "idx1")
|
||||
|
||||
resSynUpdate, err := client.FTSynUpdate(ctx, "idx1", "id1", []interface{}{"boy", "child", "offspring"}).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
@@ -4612,7 +4625,7 @@ var _ = Describe("RediSearch commands Resp 3", Label("search"), func() {
|
||||
val, err := client.FTCreate(ctx, "txt", &redis.FTCreateOptions{}, text1, text2, text3).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(val).To(BeEquivalentTo("OK"))
|
||||
WaitForIndexing(client, "txt")
|
||||
WaitForIndexing(rawClient, "txt")
|
||||
res1, err := client.FTExplain(ctx, "txt", "@f3:f3_val @f2:f2_val @f1:f1_val").Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(res1).ToNot(BeEmpty())
|
||||
|
||||
+74
-6
@@ -12,11 +12,14 @@ import (
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"time"
|
||||
|
||||
"github.com/redis/go-redis/v9/auth"
|
||||
"github.com/redis/go-redis/v9/internal"
|
||||
"github.com/redis/go-redis/v9/internal/otel"
|
||||
"github.com/redis/go-redis/v9/internal/pool"
|
||||
"github.com/redis/go-redis/v9/internal/proto"
|
||||
"github.com/redis/go-redis/v9/maintnotifications"
|
||||
"github.com/redis/go-redis/v9/push"
|
||||
)
|
||||
@@ -123,6 +126,18 @@ type FailoverOptions struct {
|
||||
// default: 32KiB (32768 bytes)
|
||||
WriteBufferSize int
|
||||
|
||||
// PipelineReadBufferSize, PipelineWriteBufferSize and PipelinePoolSize
|
||||
// configure an optional separate connection pool used for pipelining, with
|
||||
// its own (typically larger) buffers. See the same-named fields on Options
|
||||
// for details. The pool is created only when PipelineReadBufferSize or PipelineWriteBufferSize is set (PipelinePoolSize alone does not enable it).
|
||||
PipelineReadBufferSize int
|
||||
PipelineWriteBufferSize int
|
||||
PipelinePoolSize int
|
||||
|
||||
// AutoPipelineOptions is the default config for the client's autopipeliner
|
||||
// faces. See Options.AutoPipelineOptions.
|
||||
AutoPipelineOptions *AutoPipelineOptions
|
||||
|
||||
PoolFIFO bool
|
||||
|
||||
PoolSize int
|
||||
@@ -176,7 +191,7 @@ type FailoverOptions struct {
|
||||
// seamlessly. Requires Protocol: 3 (RESP3) for push notifications.
|
||||
// If nil, maintnotifications upgrades are disabled.
|
||||
// (however if Mode is nil, it defaults to "auto" - enable if server supports it)
|
||||
//MaintNotificationsConfig *maintnotifications.Config
|
||||
// MaintNotificationsConfig *maintnotifications.Config
|
||||
}
|
||||
|
||||
func (opt *FailoverOptions) clientOptions() *Options {
|
||||
@@ -202,6 +217,11 @@ func (opt *FailoverOptions) clientOptions() *Options {
|
||||
ReadBufferSize: opt.ReadBufferSize,
|
||||
WriteBufferSize: opt.WriteBufferSize,
|
||||
|
||||
PipelineReadBufferSize: opt.PipelineReadBufferSize,
|
||||
PipelineWriteBufferSize: opt.PipelineWriteBufferSize,
|
||||
PipelinePoolSize: opt.PipelinePoolSize,
|
||||
AutoPipelineOptions: opt.AutoPipelineOptions,
|
||||
|
||||
DialTimeout: opt.DialTimeout,
|
||||
DialerRetries: opt.DialerRetries,
|
||||
DialerRetryTimeout: opt.DialerRetryTimeout,
|
||||
@@ -318,6 +338,11 @@ func (opt *FailoverOptions) clusterOptions() *ClusterOptions {
|
||||
ReadBufferSize: opt.ReadBufferSize,
|
||||
WriteBufferSize: opt.WriteBufferSize,
|
||||
|
||||
PipelineReadBufferSize: opt.PipelineReadBufferSize,
|
||||
PipelineWriteBufferSize: opt.PipelineWriteBufferSize,
|
||||
PipelinePoolSize: opt.PipelinePoolSize,
|
||||
AutoPipelineOptions: opt.AutoPipelineOptions,
|
||||
|
||||
DialTimeout: opt.DialTimeout,
|
||||
DialerRetries: opt.DialerRetries,
|
||||
DialerRetryTimeout: opt.DialerRetryTimeout,
|
||||
@@ -536,9 +561,10 @@ func NewFailoverClient(failoverOpt *FailoverOptions) *Client {
|
||||
|
||||
rdb := &Client{
|
||||
baseClient: &baseClient{
|
||||
opt: opt,
|
||||
onClose: &onCloseHooks{},
|
||||
himport: newHImportRegistry(),
|
||||
apClosed: &atomic.Bool{},
|
||||
opt: opt,
|
||||
onClose: &onCloseHooks{},
|
||||
himport: newHImportRegistry(),
|
||||
},
|
||||
}
|
||||
rdb.init()
|
||||
@@ -562,6 +588,40 @@ func NewFailoverClient(failoverOpt *FailoverOptions) *Client {
|
||||
panic(fmt.Errorf("redis: failed to create pubsub pool: %w", err))
|
||||
}
|
||||
|
||||
// Optionally create a separate connection pool for pipelining, with its own
|
||||
// (typically larger) buffers. Enabled when either pipeline buffer size is set.
|
||||
if opt.PipelineReadBufferSize > 0 || opt.PipelineWriteBufferSize > 0 {
|
||||
pipelineOpt := opt.clone()
|
||||
if opt.PipelineReadBufferSize > 0 {
|
||||
pipelineOpt.ReadBufferSize = opt.PipelineReadBufferSize
|
||||
// Same clamp Options.init applies to the main pool: RESP3 push
|
||||
// parsing needs a minimum read buffer, and a tiny pipeline reader
|
||||
// would break push-notification handling on pipeline conns.
|
||||
if pipelineOpt.Protocol == 3 && pipelineOpt.ReadBufferSize < proto.MinRESP3ReadBufferSize {
|
||||
pipelineOpt.ReadBufferSize = proto.MinRESP3ReadBufferSize
|
||||
}
|
||||
}
|
||||
if opt.PipelineWriteBufferSize > 0 {
|
||||
pipelineOpt.WriteBufferSize = opt.PipelineWriteBufferSize
|
||||
}
|
||||
if opt.PipelinePoolSize > 0 {
|
||||
pipelineOpt.PoolSize = opt.PipelinePoolSize
|
||||
} else {
|
||||
pipelineOpt.PoolSize = 10 // default smaller pool for pipelining
|
||||
}
|
||||
rdb.pipelinePoolName = mainPoolName + "_pipeline"
|
||||
rdb.pipelinePool, err = newConnPool(pipelineOpt, rdb.dialHook, rdb.pipelinePoolName)
|
||||
if err != nil {
|
||||
panic(fmt.Errorf("redis: failed to create pipeline connection pool: %w", err))
|
||||
}
|
||||
}
|
||||
|
||||
// Register pools for OTel async gauge metrics, matching NewClient (the
|
||||
// failover client previously registered none, so pool gauges were silent
|
||||
// for the identical standalone setup). The pipeline pool is nil when not
|
||||
// configured.
|
||||
otel.RegisterPools(rdb.connPool, rdb.pubSubPool, rdb.pipelinePool, opt.Addr)
|
||||
|
||||
rdb.onClose.register(onCloseHookIDSentinelFailover, failover.Close)
|
||||
|
||||
failover.mu.Lock()
|
||||
@@ -571,6 +631,13 @@ func NewFailoverClient(failoverOpt *FailoverOptions) *Client {
|
||||
return cn.RemoteAddr().String() != addr
|
||||
})
|
||||
}
|
||||
// Drop stale pipeline-pool connections dialed to the demoted master too;
|
||||
// otherwise pipelined traffic keeps using the old address after failover.
|
||||
if pipelinePool, ok := rdb.pipelinePool.(*pool.ConnPool); ok {
|
||||
_ = pipelinePool.Filter(func(cn *pool.Conn) bool {
|
||||
return cn.RemoteAddr().String() != addr
|
||||
})
|
||||
}
|
||||
}
|
||||
failover.mu.Unlock()
|
||||
|
||||
@@ -626,8 +693,9 @@ func NewSentinelClient(opt *Options) *SentinelClient {
|
||||
opt.init()
|
||||
c := &SentinelClient{
|
||||
baseClient: &baseClient{
|
||||
opt: opt,
|
||||
onClose: &onCloseHooks{},
|
||||
apClosed: &atomic.Bool{},
|
||||
opt: opt,
|
||||
onClose: &onCloseHooks{},
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
+4
-1
@@ -44,7 +44,10 @@ var _ = Describe("Sentinel PROTO 2", func() {
|
||||
|
||||
var _ = Describe("Sentinel resolution", func() {
|
||||
It("should resolve master without context exhaustion", func() {
|
||||
shortCtx, cancel := context.WithTimeout(ctx, 500*time.Millisecond)
|
||||
// Wide enough for a loaded CI runner (sentinel query + master dial +
|
||||
// handshake over docker networking routinely needs >500ms there),
|
||||
// while still far below the retry-storm exhaustion this test pins.
|
||||
shortCtx, cancel := context.WithTimeout(ctx, 3*time.Second)
|
||||
defer cancel()
|
||||
|
||||
client := redis.NewFailoverClient(&redis.FailoverOptions{
|
||||
|
||||
+9
-2
@@ -381,8 +381,15 @@ func (c cmdable) XReadGroup(ctx context.Context, a *XReadGroupArgs) *XStreamSlic
|
||||
cmd.SetFirstKeyPos(keyPos)
|
||||
_ = c(ctx, cmd)
|
||||
|
||||
// Record stream lag for each message (if command succeeded)
|
||||
if cmd.Err() == nil {
|
||||
// Record stream lag for each message (if command succeeded). Gated on the
|
||||
// result being readable WITHOUT blocking: this command carries a
|
||||
// read-timeout marker, so on the deferred autopipeline face it is diverted
|
||||
// and still running when we get here — and the default Block: 0 form can
|
||||
// wait indefinitely for messages, so reading the outcome would block the
|
||||
// submit call instead of returning a future (review finding by codex on
|
||||
// #3942). Skipped for a submission that has not executed yet; emitting
|
||||
// this from the execution path is a follow-up in the OTel wiring.
|
||||
if otel.Enabled() && cmd.resultReady() && cmd.rawErr() == nil {
|
||||
streams := cmd.Val()
|
||||
for _, stream := range streams {
|
||||
for _, msg := range stream.Messages {
|
||||
|
||||
@@ -194,6 +194,12 @@ func (c cmdable) GetDel(ctx context.Context, key string) *StringCmd {
|
||||
// (including redis.Nil when the key does not exist) via Err(). If buf is too
|
||||
// small to hold the value, Err() returns a "buffer too small" error.
|
||||
//
|
||||
// Nothing is ever written past len(buf). When len(buf) >= value length + 2,
|
||||
// the read takes a fast path that pulls the payload and the protocol's
|
||||
// trailing CRLF in a single socket read, using the two bytes after the
|
||||
// payload as scratch — size buffers with 2 spare bytes to opt in (see
|
||||
// example/zerocopy-buffer).
|
||||
//
|
||||
// This command opts out of automatic retries because partial data from a
|
||||
// failed attempt would already be sitting in the caller's buffer.
|
||||
func (c cmdable) GetToBuffer(ctx context.Context, key string, buf []byte) *ZeroCopyStringCmd {
|
||||
|
||||
@@ -622,6 +622,7 @@ func newTSTimestampValueCmd(ctx context.Context, args ...interface{}) *TSTimesta
|
||||
}
|
||||
|
||||
func (cmd *TSTimestampValueCmd) String() string {
|
||||
cmd.await()
|
||||
return cmdString(cmd, cmd.val)
|
||||
}
|
||||
|
||||
@@ -630,10 +631,12 @@ func (cmd *TSTimestampValueCmd) SetVal(val TSTimestampValue) {
|
||||
}
|
||||
|
||||
func (cmd *TSTimestampValueCmd) Result() (TSTimestampValue, error) {
|
||||
cmd.await()
|
||||
return cmd.val, cmd.err
|
||||
}
|
||||
|
||||
func (cmd *TSTimestampValueCmd) Val() TSTimestampValue {
|
||||
cmd.await()
|
||||
return cmd.val
|
||||
}
|
||||
|
||||
@@ -953,6 +956,7 @@ func newTSTimestampValueSliceCmd(ctx context.Context, args ...interface{}) *TSTi
|
||||
}
|
||||
|
||||
func (cmd *TSTimestampValueSliceCmd) String() string {
|
||||
cmd.await()
|
||||
return cmdString(cmd, cmd.val)
|
||||
}
|
||||
|
||||
@@ -961,10 +965,12 @@ func (cmd *TSTimestampValueSliceCmd) SetVal(val []TSTimestampValue) {
|
||||
}
|
||||
|
||||
func (cmd *TSTimestampValueSliceCmd) Result() ([]TSTimestampValue, error) {
|
||||
cmd.await()
|
||||
return cmd.val, cmd.err
|
||||
}
|
||||
|
||||
func (cmd *TSTimestampValueSliceCmd) Val() []TSTimestampValue {
|
||||
cmd.await()
|
||||
return cmd.val
|
||||
}
|
||||
|
||||
@@ -1290,6 +1296,7 @@ func newTSNRangePivotRowSliceCmd(ctx context.Context, args ...interface{}) *TSNR
|
||||
}
|
||||
|
||||
func (cmd *TSNRangePivotRowSliceCmd) String() string {
|
||||
cmd.await()
|
||||
return cmdString(cmd, cmd.val)
|
||||
}
|
||||
|
||||
@@ -1298,10 +1305,12 @@ func (cmd *TSNRangePivotRowSliceCmd) SetVal(val []TSNRangePivotRow) {
|
||||
}
|
||||
|
||||
func (cmd *TSNRangePivotRowSliceCmd) Result() ([]TSNRangePivotRow, error) {
|
||||
cmd.await()
|
||||
return cmd.val, cmd.err
|
||||
}
|
||||
|
||||
func (cmd *TSNRangePivotRowSliceCmd) Val() []TSNRangePivotRow {
|
||||
cmd.await()
|
||||
return cmd.val
|
||||
}
|
||||
|
||||
|
||||
+48
-41
@@ -33,17 +33,24 @@ var _ = Describe("RedisTimeseries commands", Label("timeseries"), func() {
|
||||
protocol := protocol // capture loop variable for each context
|
||||
|
||||
Context(fmt.Sprintf("with protocol version %d", protocol), func() {
|
||||
var client *redis.Client
|
||||
var client redis.UniversalClient
|
||||
var rawClient *redis.Client
|
||||
var closeSubject func() error
|
||||
|
||||
BeforeEach(func() {
|
||||
client = setupRedisClient(protocol)
|
||||
Expect(client.FlushAll(ctx).Err()).NotTo(HaveOccurred())
|
||||
rawClient = setupRedisClient(protocol)
|
||||
client, closeSubject = newUniversalSubject(rawClient)
|
||||
Expect(rawClient.FlushAll(ctx).Err()).NotTo(HaveOccurred())
|
||||
})
|
||||
|
||||
AfterEach(func() {
|
||||
if client != nil {
|
||||
client.FlushDB(ctx)
|
||||
client.Close()
|
||||
// Flush through the SUBJECT: ordered after queued writes
|
||||
// (see json_test.go); awaiting Err() forces it to execute, and
|
||||
// both steps are asserted so a failed teardown cannot silently
|
||||
// leak state into later specs.
|
||||
Expect(client.FlushDB(ctx).Err()).NotTo(HaveOccurred())
|
||||
Expect(closeSubject()).NotTo(HaveOccurred())
|
||||
}
|
||||
})
|
||||
|
||||
@@ -67,7 +74,7 @@ var _ = Describe("RedisTimeseries commands", Label("timeseries"), func() {
|
||||
Expect(result).To(BeEquivalentTo("OK"))
|
||||
resultInfo, err := client.TSInfo(ctx, "4").Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
if client.Options().Protocol == 2 {
|
||||
if rawClient.Options().Protocol == 2 {
|
||||
Expect(resultInfo["labels"].([]interface{})[0]).To(BeEquivalentTo([]interface{}{"Time", "Series"}))
|
||||
} else {
|
||||
Expect(resultInfo["labels"].(map[interface{}]interface{})["Time"]).To(BeEquivalentTo("Series"))
|
||||
@@ -164,7 +171,7 @@ var _ = Describe("RedisTimeseries commands", Label("timeseries"), func() {
|
||||
Expect(result).To(BeEquivalentTo(4))
|
||||
resultInfo, err := client.TSInfo(ctx, "4").Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
if client.Options().Protocol == 2 {
|
||||
if rawClient.Options().Protocol == 2 {
|
||||
Expect(resultInfo["labels"].([]interface{})).To(ContainElement([]interface{}{"Time", "Series"}))
|
||||
} else {
|
||||
Expect(resultInfo["labels"].(map[interface{}]interface{})["Time"]).To(BeEquivalentTo("Series"))
|
||||
@@ -258,7 +265,7 @@ var _ = Describe("RedisTimeseries commands", Label("timeseries"), func() {
|
||||
|
||||
resultInfo, err = client.TSInfo(ctx, "1").Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
if client.Options().Protocol == 2 {
|
||||
if rawClient.Options().Protocol == 2 {
|
||||
Expect(resultInfo["labels"]).To(BeEquivalentTo([]interface{}{}))
|
||||
} else {
|
||||
Expect(resultInfo["labels"]).To(BeEquivalentTo(map[interface{}]interface{}{}))
|
||||
@@ -271,7 +278,7 @@ var _ = Describe("RedisTimeseries commands", Label("timeseries"), func() {
|
||||
|
||||
resultInfo, err = client.TSInfo(ctx, "1").Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
if client.Options().Protocol == 2 {
|
||||
if rawClient.Options().Protocol == 2 {
|
||||
Expect(resultInfo["labels"].([]interface{})[0]).To(BeEquivalentTo([]interface{}{"Time", "Series"}))
|
||||
Expect(resultInfo["retentionTime"]).To(BeEquivalentTo(10))
|
||||
if redisVersionAtLeast("8") {
|
||||
@@ -361,7 +368,7 @@ var _ = Describe("RedisTimeseries commands", Label("timeseries"), func() {
|
||||
Expect(resultDeleteRule).To(BeEquivalentTo("OK"))
|
||||
resultInfo, err := client.TSInfo(ctx, "1").Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
if client.Options().Protocol == 2 {
|
||||
if rawClient.Options().Protocol == 2 {
|
||||
Expect(resultInfo["rules"]).To(BeEquivalentTo([]interface{}{}))
|
||||
} else {
|
||||
Expect(resultInfo["rules"]).To(BeEquivalentTo(map[interface{}]interface{}{}))
|
||||
@@ -563,7 +570,7 @@ var _ = Describe("RedisTimeseries commands", Label("timeseries"), func() {
|
||||
|
||||
result, err := client.TSMGet(ctx, []string{"Test=This"}).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
if client.Options().Protocol == 2 {
|
||||
if rawClient.Options().Protocol == 2 {
|
||||
Expect(result["a"][1].([]interface{})[1]).To(BeEquivalentTo("15"))
|
||||
Expect(result["b"][1].([]interface{})[1]).To(BeEquivalentTo("25"))
|
||||
} else {
|
||||
@@ -573,7 +580,7 @@ var _ = Describe("RedisTimeseries commands", Label("timeseries"), func() {
|
||||
mgetOpt := &redis.TSMGetOptions{WithLabels: true}
|
||||
result, err = client.TSMGetWithArgs(ctx, []string{"Test=This"}, mgetOpt).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
if client.Options().Protocol == 2 {
|
||||
if rawClient.Options().Protocol == 2 {
|
||||
Expect(result["b"][0]).To(ConsistOf([]interface{}{"Test", "This"}, []interface{}{"Taste", "That"}))
|
||||
} else {
|
||||
Expect(result["b"][0]).To(BeEquivalentTo(map[interface{}]interface{}{"Test": "This", "Taste": "That"}))
|
||||
@@ -599,7 +606,7 @@ var _ = Describe("RedisTimeseries commands", Label("timeseries"), func() {
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
result, err = client.TSMGet(ctx, []string{"is_compaction=true"}).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
if client.Options().Protocol == 2 {
|
||||
if rawClient.Options().Protocol == 2 {
|
||||
Expect(result["d"][1]).To(BeEquivalentTo([]interface{}{int64(0), "4"}))
|
||||
} else {
|
||||
Expect(result["d"][1]).To(BeEquivalentTo([]interface{}{int64(0), 4.0}))
|
||||
@@ -607,7 +614,7 @@ var _ = Describe("RedisTimeseries commands", Label("timeseries"), func() {
|
||||
mgetOpt = &redis.TSMGetOptions{Latest: true}
|
||||
result, err = client.TSMGetWithArgs(ctx, []string{"is_compaction=true"}, mgetOpt).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
if client.Options().Protocol == 2 {
|
||||
if rawClient.Options().Protocol == 2 {
|
||||
Expect(result["d"][1]).To(BeEquivalentTo([]interface{}{int64(10), "8"}))
|
||||
} else {
|
||||
Expect(result["d"][1]).To(BeEquivalentTo([]interface{}{int64(10), 8.0}))
|
||||
@@ -990,7 +997,7 @@ var _ = Describe("RedisTimeseries commands", Label("timeseries"), func() {
|
||||
result, err := client.TSMRange(ctx, 0, 200, []string{"Test=This"}).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(len(result)).To(BeEquivalentTo(2))
|
||||
if client.Options().Protocol == 2 {
|
||||
if rawClient.Options().Protocol == 2 {
|
||||
Expect(len(result["a"][1].([]interface{}))).To(BeEquivalentTo(100))
|
||||
} else {
|
||||
Expect(len(result["a"][2].([]interface{}))).To(BeEquivalentTo(100))
|
||||
@@ -999,7 +1006,7 @@ var _ = Describe("RedisTimeseries commands", Label("timeseries"), func() {
|
||||
mrangeOpt := &redis.TSMRangeOptions{Count: 10}
|
||||
result, err = client.TSMRangeWithArgs(ctx, 0, 200, []string{"Test=This"}, mrangeOpt).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
if client.Options().Protocol == 2 {
|
||||
if rawClient.Options().Protocol == 2 {
|
||||
Expect(len(result["a"][1].([]interface{}))).To(BeEquivalentTo(10))
|
||||
} else {
|
||||
Expect(len(result["a"][2].([]interface{}))).To(BeEquivalentTo(10))
|
||||
@@ -1013,13 +1020,13 @@ var _ = Describe("RedisTimeseries commands", Label("timeseries"), func() {
|
||||
result, err = client.TSMRangeWithArgs(ctx, 0, 500, []string{"Test=This"}, mrangeOpt).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(len(result)).To(BeEquivalentTo(2))
|
||||
if client.Options().Protocol == 2 {
|
||||
if rawClient.Options().Protocol == 2 {
|
||||
Expect(len(result["a"][1].([]interface{}))).To(BeEquivalentTo(20))
|
||||
} else {
|
||||
Expect(len(result["a"][2].([]interface{}))).To(BeEquivalentTo(20))
|
||||
}
|
||||
// Test WithLabels
|
||||
if client.Options().Protocol == 2 {
|
||||
if rawClient.Options().Protocol == 2 {
|
||||
Expect(result["a"][0]).To(BeEquivalentTo([]interface{}{}))
|
||||
} else {
|
||||
Expect(result["a"][0]).To(BeEquivalentTo(map[interface{}]interface{}{}))
|
||||
@@ -1027,7 +1034,7 @@ var _ = Describe("RedisTimeseries commands", Label("timeseries"), func() {
|
||||
mrangeOpt = &redis.TSMRangeOptions{WithLabels: true}
|
||||
result, err = client.TSMRangeWithArgs(ctx, 0, 200, []string{"Test=This"}, mrangeOpt).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
if client.Options().Protocol == 2 {
|
||||
if rawClient.Options().Protocol == 2 {
|
||||
Expect(result["a"][0]).To(ConsistOf([]interface{}{[]interface{}{"Test", "This"}, []interface{}{"team", "ny"}}))
|
||||
} else {
|
||||
Expect(result["a"][0]).To(BeEquivalentTo(map[interface{}]interface{}{"Test": "This", "team": "ny"}))
|
||||
@@ -1036,7 +1043,7 @@ var _ = Describe("RedisTimeseries commands", Label("timeseries"), func() {
|
||||
mrangeOpt = &redis.TSMRangeOptions{SelectedLabels: []interface{}{"team"}}
|
||||
result, err = client.TSMRangeWithArgs(ctx, 0, 200, []string{"Test=This"}, mrangeOpt).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
if client.Options().Protocol == 2 {
|
||||
if rawClient.Options().Protocol == 2 {
|
||||
Expect(result["a"][0].([]interface{})[0]).To(BeEquivalentTo([]interface{}{"team", "ny"}))
|
||||
Expect(result["b"][0].([]interface{})[0]).To(BeEquivalentTo([]interface{}{"team", "sf"}))
|
||||
} else {
|
||||
@@ -1051,7 +1058,7 @@ var _ = Describe("RedisTimeseries commands", Label("timeseries"), func() {
|
||||
mrangeOpt = &redis.TSMRangeOptions{FilterByTS: fts, FilterByValue: []int{1, 2}}
|
||||
result, err = client.TSMRangeWithArgs(ctx, 0, 200, []string{"Test=This"}, mrangeOpt).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
if client.Options().Protocol == 2 {
|
||||
if rawClient.Options().Protocol == 2 {
|
||||
Expect(result["a"][1].([]interface{})).To(BeEquivalentTo([]interface{}{[]interface{}{int64(15), "1"}, []interface{}{int64(16), "2"}}))
|
||||
} else {
|
||||
Expect(result["a"][2]).To(BeEquivalentTo([]interface{}{[]interface{}{int64(15), 1.0}, []interface{}{int64(16), 2.0}}))
|
||||
@@ -1060,7 +1067,7 @@ var _ = Describe("RedisTimeseries commands", Label("timeseries"), func() {
|
||||
mrangeOpt = &redis.TSMRangeOptions{GroupByLabel: "Test", Reducer: "sum"}
|
||||
result, err = client.TSMRangeWithArgs(ctx, 0, 3, []string{"Test=This"}, mrangeOpt).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
if client.Options().Protocol == 2 {
|
||||
if rawClient.Options().Protocol == 2 {
|
||||
Expect(result["Test=This"][1]).To(BeEquivalentTo([]interface{}{[]interface{}{int64(0), "0"}, []interface{}{int64(1), "2"}, []interface{}{int64(2), "4"}, []interface{}{int64(3), "6"}}))
|
||||
} else {
|
||||
Expect(result["Test=This"][3]).To(BeEquivalentTo([]interface{}{[]interface{}{int64(0), 0.0}, []interface{}{int64(1), 2.0}, []interface{}{int64(2), 4.0}, []interface{}{int64(3), 6.0}}))
|
||||
@@ -1068,7 +1075,7 @@ var _ = Describe("RedisTimeseries commands", Label("timeseries"), func() {
|
||||
mrangeOpt = &redis.TSMRangeOptions{GroupByLabel: "Test", Reducer: "max"}
|
||||
result, err = client.TSMRangeWithArgs(ctx, 0, 3, []string{"Test=This"}, mrangeOpt).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
if client.Options().Protocol == 2 {
|
||||
if rawClient.Options().Protocol == 2 {
|
||||
Expect(result["Test=This"][1]).To(BeEquivalentTo([]interface{}{[]interface{}{int64(0), "0"}, []interface{}{int64(1), "1"}, []interface{}{int64(2), "2"}, []interface{}{int64(3), "3"}}))
|
||||
} else {
|
||||
Expect(result["Test=This"][3]).To(BeEquivalentTo([]interface{}{[]interface{}{int64(0), 0.0}, []interface{}{int64(1), 1.0}, []interface{}{int64(2), 2.0}, []interface{}{int64(3), 3.0}}))
|
||||
@@ -1078,7 +1085,7 @@ var _ = Describe("RedisTimeseries commands", Label("timeseries"), func() {
|
||||
result, err = client.TSMRangeWithArgs(ctx, 0, 3, []string{"Test=This"}, mrangeOpt).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(len(result)).To(BeEquivalentTo(2))
|
||||
if client.Options().Protocol == 2 {
|
||||
if rawClient.Options().Protocol == 2 {
|
||||
Expect(result["team=ny"][1]).To(BeEquivalentTo([]interface{}{[]interface{}{int64(0), "0"}, []interface{}{int64(1), "1"}, []interface{}{int64(2), "2"}, []interface{}{int64(3), "3"}}))
|
||||
Expect(result["team=sf"][1]).To(BeEquivalentTo([]interface{}{[]interface{}{int64(0), "0"}, []interface{}{int64(1), "1"}, []interface{}{int64(2), "2"}, []interface{}{int64(3), "3"}}))
|
||||
} else {
|
||||
@@ -1089,7 +1096,7 @@ var _ = Describe("RedisTimeseries commands", Label("timeseries"), func() {
|
||||
mrangeOpt = &redis.TSMRangeOptions{Aggregator: redis.Count, BucketDuration: 10, Align: "-"}
|
||||
result, err = client.TSMRangeWithArgs(ctx, 0, 10, []string{"team=ny"}, mrangeOpt).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
if client.Options().Protocol == 2 {
|
||||
if rawClient.Options().Protocol == 2 {
|
||||
Expect(result["a"][1]).To(BeEquivalentTo([]interface{}{[]interface{}{int64(0), "10"}, []interface{}{int64(10), "1"}}))
|
||||
} else {
|
||||
Expect(result["a"][2]).To(BeEquivalentTo([]interface{}{[]interface{}{int64(0), 10.0}, []interface{}{int64(10), 1.0}}))
|
||||
@@ -1098,7 +1105,7 @@ var _ = Describe("RedisTimeseries commands", Label("timeseries"), func() {
|
||||
mrangeOpt = &redis.TSMRangeOptions{Aggregator: redis.Count, BucketDuration: 10, Align: 5}
|
||||
result, err = client.TSMRangeWithArgs(ctx, 0, 10, []string{"team=ny"}, mrangeOpt).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
if client.Options().Protocol == 2 {
|
||||
if rawClient.Options().Protocol == 2 {
|
||||
Expect(result["a"][1]).To(BeEquivalentTo([]interface{}{[]interface{}{int64(0), "5"}, []interface{}{int64(5), "6"}}))
|
||||
} else {
|
||||
Expect(result["a"][2]).To(BeEquivalentTo([]interface{}{[]interface{}{int64(0), 5.0}, []interface{}{int64(5), 6.0}}))
|
||||
@@ -1149,7 +1156,7 @@ var _ = Describe("RedisTimeseries commands", Label("timeseries"), func() {
|
||||
mrangeOpt := &redis.TSMRangeOptions{Latest: true}
|
||||
result, err := client.TSMRangeWithArgs(ctx, 0, 10, []string{"is_compaction=true"}, mrangeOpt).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
if client.Options().Protocol == 2 {
|
||||
if rawClient.Options().Protocol == 2 {
|
||||
Expect(result["b"][1]).To(ConsistOf([]interface{}{int64(0), "4"}, []interface{}{int64(10), "8"}))
|
||||
Expect(result["d"][1]).To(ConsistOf([]interface{}{int64(0), "4"}, []interface{}{int64(10), "8"}))
|
||||
} else {
|
||||
@@ -1188,7 +1195,7 @@ var _ = Describe("RedisTimeseries commands", Label("timeseries"), func() {
|
||||
}).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(len(result)).To(Equal(2))
|
||||
if client.Options().Protocol == 2 {
|
||||
if rawClient.Options().Protocol == 2 {
|
||||
Expect(result["multi-mrange-a"][1]).To(Equal([]interface{}{
|
||||
[]interface{}{int64(1000), "10", "20"},
|
||||
[]interface{}{int64(1020), "30", "40"},
|
||||
@@ -1235,7 +1242,7 @@ var _ = Describe("RedisTimeseries commands", Label("timeseries"), func() {
|
||||
result, err := client.TSMRevRange(ctx, 0, 200, []string{"Test=This"}).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(len(result)).To(BeEquivalentTo(2))
|
||||
if client.Options().Protocol == 2 {
|
||||
if rawClient.Options().Protocol == 2 {
|
||||
Expect(len(result["a"][1].([]interface{}))).To(BeEquivalentTo(100))
|
||||
} else {
|
||||
Expect(len(result["a"][2].([]interface{}))).To(BeEquivalentTo(100))
|
||||
@@ -1244,7 +1251,7 @@ var _ = Describe("RedisTimeseries commands", Label("timeseries"), func() {
|
||||
mrangeOpt := &redis.TSMRevRangeOptions{Count: 10}
|
||||
result, err = client.TSMRevRangeWithArgs(ctx, 0, 200, []string{"Test=This"}, mrangeOpt).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
if client.Options().Protocol == 2 {
|
||||
if rawClient.Options().Protocol == 2 {
|
||||
Expect(len(result["a"][1].([]interface{}))).To(BeEquivalentTo(10))
|
||||
} else {
|
||||
Expect(len(result["a"][2].([]interface{}))).To(BeEquivalentTo(10))
|
||||
@@ -1258,7 +1265,7 @@ var _ = Describe("RedisTimeseries commands", Label("timeseries"), func() {
|
||||
result, err = client.TSMRevRangeWithArgs(ctx, 0, 500, []string{"Test=This"}, mrangeOpt).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(len(result)).To(BeEquivalentTo(2))
|
||||
if client.Options().Protocol == 2 {
|
||||
if rawClient.Options().Protocol == 2 {
|
||||
Expect(len(result["a"][1].([]interface{}))).To(BeEquivalentTo(20))
|
||||
Expect(result["a"][0]).To(BeEquivalentTo([]interface{}{}))
|
||||
} else {
|
||||
@@ -1268,7 +1275,7 @@ var _ = Describe("RedisTimeseries commands", Label("timeseries"), func() {
|
||||
mrangeOpt = &redis.TSMRevRangeOptions{WithLabels: true}
|
||||
result, err = client.TSMRevRangeWithArgs(ctx, 0, 200, []string{"Test=This"}, mrangeOpt).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
if client.Options().Protocol == 2 {
|
||||
if rawClient.Options().Protocol == 2 {
|
||||
Expect(result["a"][0]).To(ConsistOf([]interface{}{[]interface{}{"Test", "This"}, []interface{}{"team", "ny"}}))
|
||||
} else {
|
||||
Expect(result["a"][0]).To(BeEquivalentTo(map[interface{}]interface{}{"Test": "This", "team": "ny"}))
|
||||
@@ -1277,7 +1284,7 @@ var _ = Describe("RedisTimeseries commands", Label("timeseries"), func() {
|
||||
mrangeOpt = &redis.TSMRevRangeOptions{SelectedLabels: []interface{}{"team"}}
|
||||
result, err = client.TSMRevRangeWithArgs(ctx, 0, 200, []string{"Test=This"}, mrangeOpt).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
if client.Options().Protocol == 2 {
|
||||
if rawClient.Options().Protocol == 2 {
|
||||
Expect(result["a"][0].([]interface{})[0]).To(BeEquivalentTo([]interface{}{"team", "ny"}))
|
||||
Expect(result["b"][0].([]interface{})[0]).To(BeEquivalentTo([]interface{}{"team", "sf"}))
|
||||
} else {
|
||||
@@ -1292,7 +1299,7 @@ var _ = Describe("RedisTimeseries commands", Label("timeseries"), func() {
|
||||
mrangeOpt = &redis.TSMRevRangeOptions{FilterByTS: fts, FilterByValue: []int{1, 2}}
|
||||
result, err = client.TSMRevRangeWithArgs(ctx, 0, 200, []string{"Test=This"}, mrangeOpt).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
if client.Options().Protocol == 2 {
|
||||
if rawClient.Options().Protocol == 2 {
|
||||
Expect(result["a"][1].([]interface{})).To(ConsistOf([]interface{}{int64(16), "2"}, []interface{}{int64(15), "1"}))
|
||||
} else {
|
||||
Expect(result["a"][2]).To(BeEquivalentTo([]interface{}{[]interface{}{int64(16), 2.0}, []interface{}{int64(15), 1.0}}))
|
||||
@@ -1301,7 +1308,7 @@ var _ = Describe("RedisTimeseries commands", Label("timeseries"), func() {
|
||||
mrangeOpt = &redis.TSMRevRangeOptions{GroupByLabel: "Test", Reducer: "sum"}
|
||||
result, err = client.TSMRevRangeWithArgs(ctx, 0, 3, []string{"Test=This"}, mrangeOpt).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
if client.Options().Protocol == 2 {
|
||||
if rawClient.Options().Protocol == 2 {
|
||||
Expect(result["Test=This"][1]).To(BeEquivalentTo([]interface{}{[]interface{}{int64(3), "6"}, []interface{}{int64(2), "4"}, []interface{}{int64(1), "2"}, []interface{}{int64(0), "0"}}))
|
||||
} else {
|
||||
Expect(result["Test=This"][3]).To(BeEquivalentTo([]interface{}{[]interface{}{int64(3), 6.0}, []interface{}{int64(2), 4.0}, []interface{}{int64(1), 2.0}, []interface{}{int64(0), 0.0}}))
|
||||
@@ -1309,7 +1316,7 @@ var _ = Describe("RedisTimeseries commands", Label("timeseries"), func() {
|
||||
mrangeOpt = &redis.TSMRevRangeOptions{GroupByLabel: "Test", Reducer: "max"}
|
||||
result, err = client.TSMRevRangeWithArgs(ctx, 0, 3, []string{"Test=This"}, mrangeOpt).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
if client.Options().Protocol == 2 {
|
||||
if rawClient.Options().Protocol == 2 {
|
||||
Expect(result["Test=This"][1]).To(BeEquivalentTo([]interface{}{[]interface{}{int64(3), "3"}, []interface{}{int64(2), "2"}, []interface{}{int64(1), "1"}, []interface{}{int64(0), "0"}}))
|
||||
} else {
|
||||
Expect(result["Test=This"][3]).To(BeEquivalentTo([]interface{}{[]interface{}{int64(3), 3.0}, []interface{}{int64(2), 2.0}, []interface{}{int64(1), 1.0}, []interface{}{int64(0), 0.0}}))
|
||||
@@ -1318,7 +1325,7 @@ var _ = Describe("RedisTimeseries commands", Label("timeseries"), func() {
|
||||
result, err = client.TSMRevRangeWithArgs(ctx, 0, 3, []string{"Test=This"}, mrangeOpt).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(len(result)).To(BeEquivalentTo(2))
|
||||
if client.Options().Protocol == 2 {
|
||||
if rawClient.Options().Protocol == 2 {
|
||||
Expect(result["team=ny"][1]).To(BeEquivalentTo([]interface{}{[]interface{}{int64(3), "3"}, []interface{}{int64(2), "2"}, []interface{}{int64(1), "1"}, []interface{}{int64(0), "0"}}))
|
||||
Expect(result["team=sf"][1]).To(BeEquivalentTo([]interface{}{[]interface{}{int64(3), "3"}, []interface{}{int64(2), "2"}, []interface{}{int64(1), "1"}, []interface{}{int64(0), "0"}}))
|
||||
} else {
|
||||
@@ -1329,7 +1336,7 @@ var _ = Describe("RedisTimeseries commands", Label("timeseries"), func() {
|
||||
mrangeOpt = &redis.TSMRevRangeOptions{Aggregator: redis.Count, BucketDuration: 10, Align: "-"}
|
||||
result, err = client.TSMRevRangeWithArgs(ctx, 0, 10, []string{"team=ny"}, mrangeOpt).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
if client.Options().Protocol == 2 {
|
||||
if rawClient.Options().Protocol == 2 {
|
||||
Expect(result["a"][1]).To(BeEquivalentTo([]interface{}{[]interface{}{int64(10), "1"}, []interface{}{int64(0), "10"}}))
|
||||
} else {
|
||||
Expect(result["a"][2]).To(BeEquivalentTo([]interface{}{[]interface{}{int64(10), 1.0}, []interface{}{int64(0), 10.0}}))
|
||||
@@ -1337,7 +1344,7 @@ var _ = Describe("RedisTimeseries commands", Label("timeseries"), func() {
|
||||
mrangeOpt = &redis.TSMRevRangeOptions{Aggregator: redis.Count, BucketDuration: 10, Align: 1}
|
||||
result, err = client.TSMRevRangeWithArgs(ctx, 0, 10, []string{"team=ny"}, mrangeOpt).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
if client.Options().Protocol == 2 {
|
||||
if rawClient.Options().Protocol == 2 {
|
||||
Expect(result["a"][1]).To(BeEquivalentTo([]interface{}{[]interface{}{int64(1), "10"}, []interface{}{int64(0), "1"}}))
|
||||
} else {
|
||||
Expect(result["a"][2]).To(BeEquivalentTo([]interface{}{[]interface{}{int64(1), 10.0}, []interface{}{int64(0), 1.0}}))
|
||||
@@ -1375,7 +1382,7 @@ var _ = Describe("RedisTimeseries commands", Label("timeseries"), func() {
|
||||
}).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(len(result)).To(Equal(2))
|
||||
if client.Options().Protocol == 2 {
|
||||
if rawClient.Options().Protocol == 2 {
|
||||
Expect(result["multi-mrevrange-a"][1]).To(Equal([]interface{}{
|
||||
[]interface{}{int64(1020), "30", "40"},
|
||||
[]interface{}{int64(1000), "10", "20"},
|
||||
@@ -1521,7 +1528,7 @@ var _ = Describe("RedisTimeseries commands", Label("timeseries"), func() {
|
||||
mrangeOpt := &redis.TSMRevRangeOptions{Latest: true}
|
||||
result, err := client.TSMRevRangeWithArgs(ctx, 0, 10, []string{"is_compaction=true"}, mrangeOpt).Result()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
if client.Options().Protocol == 2 {
|
||||
if rawClient.Options().Protocol == 2 {
|
||||
Expect(result["b"][1]).To(BeEquivalentTo([]interface{}{[]interface{}{int64(10), "8"}, []interface{}{int64(0), "4"}}))
|
||||
Expect(result["d"][1]).To(BeEquivalentTo([]interface{}{[]interface{}{int64(10), "8"}, []interface{}{int64(0), "4"}}))
|
||||
} else {
|
||||
|
||||
@@ -149,6 +149,11 @@ type UniversalOptions struct {
|
||||
// IsClusterMode can be used when only one Addrs is provided (e.g. Elasticache supports setting up cluster mode with configuration endpoint).
|
||||
IsClusterMode bool
|
||||
|
||||
// AutoPipelineOptions is the default config for the client's
|
||||
// autopipeliner faces (AutoPipeline / AsyncAutoPipeline), applied when
|
||||
// they are called without explicit options. See Options.AutoPipelineOptions.
|
||||
AutoPipelineOptions *AutoPipelineOptions
|
||||
|
||||
// MaintNotificationsConfig provides configuration for maintnotifications upgrades.
|
||||
MaintNotificationsConfig *maintnotifications.Config
|
||||
|
||||
@@ -226,6 +231,7 @@ func (o *UniversalOptions) Cluster() *ClusterOptions {
|
||||
DisableIdentity: o.DisableIdentity,
|
||||
DisableIndentity: o.DisableIndentity,
|
||||
IdentitySuffix: o.IdentitySuffix,
|
||||
AutoPipelineOptions: o.AutoPipelineOptions,
|
||||
FailingTimeoutSeconds: o.FailingTimeoutSeconds,
|
||||
UnstableResp3: o.UnstableResp3,
|
||||
PushNotificationProcessor: o.PushNotificationProcessor,
|
||||
@@ -294,6 +300,7 @@ func (o *UniversalOptions) Failover() *FailoverOptions {
|
||||
DisableIdentity: o.DisableIdentity,
|
||||
DisableIndentity: o.DisableIndentity,
|
||||
IdentitySuffix: o.IdentitySuffix,
|
||||
AutoPipelineOptions: o.AutoPipelineOptions,
|
||||
UnstableResp3: o.UnstableResp3,
|
||||
PushNotificationProcessor: o.PushNotificationProcessor,
|
||||
// Note: MaintNotificationsConfig not supported for FailoverOptions
|
||||
@@ -352,6 +359,7 @@ func (o *UniversalOptions) Simple() *Options {
|
||||
DisableIdentity: o.DisableIdentity,
|
||||
DisableIndentity: o.DisableIndentity,
|
||||
IdentitySuffix: o.IdentitySuffix,
|
||||
AutoPipelineOptions: o.AutoPipelineOptions,
|
||||
UnstableResp3: o.UnstableResp3,
|
||||
PushNotificationProcessor: o.PushNotificationProcessor,
|
||||
MaintNotificationsConfig: o.MaintNotificationsConfig,
|
||||
@@ -373,6 +381,15 @@ type UniversalClient interface {
|
||||
Watch(ctx context.Context, fn func(*Tx) error, keys ...string) error
|
||||
Do(ctx context.Context, args ...interface{}) *Cmd
|
||||
Process(ctx context.Context, cmd Cmder) error
|
||||
// AutoPipeline / AsyncAutoPipeline return an AutoPipeliner for the concrete
|
||||
// client. Supported on *Client (including sentinel-backed failover clients)
|
||||
// and *ClusterClient; *Ring returns an error (not supported).
|
||||
//
|
||||
// EXPERIMENTAL: this API is subject to change, use with caution.
|
||||
AutoPipeline() (*AutoPipeliner, error)
|
||||
AutoPipelineWithOptions(config *AutoPipelineOptions) (*AutoPipeliner, error)
|
||||
AsyncAutoPipeline() (*AutoPipeliner, error)
|
||||
AsyncAutoPipelineWithOptions(config *AutoPipelineOptions) (*AutoPipeliner, error)
|
||||
Subscribe(ctx context.Context, channels ...string) *PubSub
|
||||
PSubscribe(ctx context.Context, channels ...string) *PubSub
|
||||
SSubscribe(ctx context.Context, channels ...string) *PubSub
|
||||
@@ -384,6 +401,9 @@ var (
|
||||
_ UniversalClient = (*Client)(nil)
|
||||
_ UniversalClient = (*ClusterClient)(nil)
|
||||
_ UniversalClient = (*Ring)(nil)
|
||||
// AutoPipeliner is a drop-in for the real clients; non-data operations
|
||||
// delegate to the underlying client.
|
||||
_ UniversalClient = (*AutoPipeliner)(nil)
|
||||
)
|
||||
|
||||
// NewUniversalClient returns a new multi client. The type of the returned client depends
|
||||
|
||||
@@ -50,17 +50,24 @@ var _ = Describe("Redis VectorSet commands", Label("vectorset"), func() {
|
||||
protocol := protocol
|
||||
|
||||
Context(fmt.Sprintf("with protocol version %d", protocol), func() {
|
||||
var client *redis.Client
|
||||
var client redis.UniversalClient
|
||||
var rawClient *redis.Client
|
||||
var closeSubject func() error
|
||||
|
||||
BeforeEach(func() {
|
||||
client = setupRedisClient(protocol)
|
||||
Expect(client.FlushAll(ctx).Err()).NotTo(HaveOccurred())
|
||||
rawClient = setupRedisClient(protocol)
|
||||
client, closeSubject = newUniversalSubject(rawClient)
|
||||
Expect(rawClient.FlushAll(ctx).Err()).NotTo(HaveOccurred())
|
||||
})
|
||||
|
||||
AfterEach(func() {
|
||||
if client != nil {
|
||||
client.FlushDB(ctx)
|
||||
client.Close()
|
||||
// Flush through the SUBJECT: ordered after queued writes
|
||||
// (see json_test.go); awaiting Err() forces it to execute, and
|
||||
// both steps are asserted so a failed teardown cannot silently
|
||||
// leak state into later specs.
|
||||
Expect(client.FlushDB(ctx).Err()).NotTo(HaveOccurred())
|
||||
Expect(closeSubject()).NotTo(HaveOccurred())
|
||||
}
|
||||
})
|
||||
|
||||
|
||||
Reference in new issue
Block a user