milvus/docs/dev/error_handling_casebook.md

Error Handling Casebook

Real positive and negative examples for choosing, wrapping and classifying merr error codes — a companion to:

• error_handling_guide.md — the rules and the decision tree (read it first).
• error_sentinel_convention.md — sentinel layering and naming.

Every negative example below is real: it is what the first draft of the error-standardization PR #50221 (issue #47420) actually wrote, caught and corrected during its review — 60+ misclassifications in total. (The PR was squash-merged, so these intermediate states are not visible in master's history; before #50221 most of these sites returned bare fmt.Errorf.) They cluster into the seven patterns in this document. If you (human or coding agent) are about to originate, wrap or classify an error, check your change against these patterns — the linter catches bare errors, but it cannot catch a wrong code or a wrong classification. That is what this casebook is for.

Quick self-check before any error-handling change:

1. Adding context to an existing error? → merr.Wrap / merr.Wrapf only. WrapErrXxxErr overwrites the inner code (Pattern 3).
2. The check "looks like validation"? → ask who produced the value. Plans, task types and component outputs are not user input (Patterns 1–2).
3. Passing a cause? → as the error argument, never into the format string (Pattern 4).
4. Marking a sentinel InputError? → first grep for retry.Do users of that code; InputError aborts retries (Pattern 5).
5. Converting an errors.New sentinel to merr? → first grep errors.Is(..., thatSentinel); merr.Is matches by code, so identity guards silently widen (Pattern 6).
6. About to "fix" an odd-looking boundary conversion? → it may be a wire contract (Pattern 7). Check for a comment, then check SDK/e2e expectations.

---

Pattern 1 — "Looks like validation" is not user input

A switch/if that rejects an unexpected value looks like input validation. The classification question is not "is this a check?" but "who produced the value being checked?" A plan, task type, or request assembled by a coordinator is internal protocol, not user input. If the check fires, Milvus (or a mixed-version deployment) has a bug — the user did nothing wrong, and returning an InputError would disable the retry/failover machinery and blame the user on dashboards.

❌ As first drafted (caught in review): an unrecognized task type — assigned by the coordinator, never typed by a user — returned a parameter error:

// task type comes from the coordinator, not from any user request
default:
    return p[TypeKey], merr.WrapErrParameterInvalidMsg("unrecognized task type '%s', taskID=%s", p[TypeKey], p[TaskIDKey])

✅ Use a typed capability error (pkg/taskcommon/properties.go, GetTaskType):

default:
    // Task types are assigned by the coordinator. An unrecognized type means
    // this worker does not implement the coordinator's task protocol, which is
    // a system capability mismatch rather than invalid user input.
    return p[TypeKey], merr.Wrapf(merr.ErrServiceUnimplemented,
        "unrecognized task type '%s', taskID=%s", p[TypeKey], p[TaskIDKey])

DataNode CreateTask/QueryTask/DropTask use the same capability code. This lets a coordinator distinguish an unsupported task protocol with errors.Is without relying on an error string or the wire Retriable flag. By contrast, malformed plans for task types that the worker does recognize remain ErrServiceInternal.

Rule of thumb: if the worker lacks the requested coordinator-owned capability → ErrServiceUnimplemented; if both sides claim the capability but their internal contract is malformed → ErrServiceInternal. Neither is user input.

Pattern 2 — Internal component output is not user input

Data produced by segcore, a query node, or any internal stage is a contract between Milvus components. When proxy-side reduce code finds a shape violation in what a query node returned, that is a Milvus bug — not ParameterInvalid.

❌ As first drafted: proxy search-reduce blamed the user for a malformed querynode response:

if data.NumQueries != nq {
    return merr.WrapErrParameterInvalidMsg("search result's nq(%d) mis-match with %d", data.NumQueries, nq)
}

✅ As merged (internal/proxy/search_reduce_util.go, checkSearchResultData):

if data.NumQueries != nq {
    return merr.WrapErrServiceInternalMsg("search result's nq(%d) mis-match with %d", data.NumQueries, nq)
}

Same fix applied to: the queryutil merge helpers (13 sites), proxy ShowPartitions array-alignment checks in meta_cache.go, channels_mgr vchannel/pchannel count checks, querynode stats-key handling, and the function-pipeline runner output checks (46 sites — those use WrapErrFunctionFailedMsg, the typed code that subsystem promises).

A sub-case worth naming: recovered panics are never input errors.

// internal/datanode/external/manager.go — task executor:
if r := recover(); r != nil {
    ...
    reason := fmt.Sprintf("task panicked: %v", r)
    ...
    // A recovered panic is a server-side failure, never caller input.
    retErr = merr.WrapErrServiceInternalMsg("%s", reason)
}

Pattern 3 — WrapErrXxxErr is a relabel, not a context-adder

merr.WrapErrServiceInternalErr(err, ...) (and every WrapErrXxxErr) reports the outer sentinel's code and retriability — it masks the inner typed code. Reaching for it "to add context" silently downgrades a precise, possibly-retriable inner error into a generic non-retriable ServiceInternal.

// ❌ inner err is already typed (e.g. ErrIoTooManyRequests, retriable);
//    this collapses it to ServiceInternal(5), non-retriable:
return merr.WrapErrServiceInternalErr(err, "failed to load segment %d", segID)

// ✅ keep the code, add the breadcrumb:
return merr.Wrapf(err, "failed to load segment %d", segID)

WrapErrXxxErr is only for a deliberate relabel — when your interface promises to hide the inner detail behind a different code. The choice of helper is the statement of intent; see §3.2 of the guide.

Storage-layer corollary (a cluster of real fixes): errors from ChunkObjectStorage Read/Write/Remove/Copy are already typed by mapObjectStorageError (ErrIoKeyNotFound, ErrIoTooManyRequests retriable, …). Wrapping them with WrapErrIoFailedErr/WrapErrServiceInternalErr flattens that taxonomy. Use merr.Wrap; originate a new Io error only for errors born untyped (e.g. WalkWithObjects historically returned raw client errors).

Pattern 4 — The cause goes in the error argument, never the format string

// ❌ errors.Is chain destroyed; %w renders as "%!w(...)" because
//    WrapErr*Msg formats with fmt.Sprintf:
return merr.WrapErrServiceInternalMsg("compact failed: %s", err)
return merr.WrapErrServiceInternalMsg("compact failed: %w", err)

// ✅ cause in the chain (deliberate relabel to ServiceInternal):
return merr.WrapErrServiceInternalErr(err, "compact failed")
// ✅ or, if the inner code should survive (Pattern 3):
return merr.Wrap(err, "compact failed")

Symptoms of the broken form: errors.Is(outer, innerSentinel) returns false, typed inner codes vanish from the wire, retriability resets, and the audit trail in Status.Reason still looks fine — which is why this one survives review so often.

Pattern 5 — InputError aborts retry.Do: scan before you mark

retry.Do aborts immediately on any InputError (and the proxy lb_policy stops cross-replica failover). Before adding WithErrorType(InputError) to a sentinel — or marking a code at some boundary — grep for files that both produce/consume that code and sit inside a retry.Do loop. A transient "not ready yet" condition must never ride an InputError-marked code.

Real case: CheckAllQnReady is polled inside retry.Do during CreateCollection (waiting for query nodes to sync file resources — a transient condition). Had its error used an InputError-marked code, the retry loop would have aborted on the first attempt:

// internal/coordinator/file_resource_observer.go — transient wait condition,
// expressed with a retriable system code on purpose:
err = merr.WrapErrServiceUnavailableMsg("file resource not synced, node-%d", nodeID)

// internal/rootcoord/create_collection_task.go — the consumer (condensed):
err := retry.Do(ctx, func() error {
    ...
    return t.fileResourceObserver.CheckAllQnReady()
}, retry.Attempts(10), retry.Sleep(3*time.Second))

For transient conditions use ErrServiceUnavailable / ErrServiceNotReady / ErrServiceResourceInsufficient — never a code that is (or may become) InputError-marked.

The dual-identity solution. When the same code is user-blame at the API boundary but a transient internal condition elsewhere (ErrCollectionNotFound, ErrDatabaseNotFound, ErrPartitionNotFound, ErrAliasNotFound, ErrFieldNotFound), the sentinel stays SystemError and the boundary stamps it — only where the name came from the user:

// internal/proxy/meta_cache.go — the central chokepoint for user-supplied names:
return collection, merr.WrapErrAsInputErrorWhen(err,
    merr.ErrCollectionNotFound, merr.ErrDatabaseNotFound)

WrapErrAsInputError(When) relabels the classification only; code, message and errors.Is chain are untouched. Internal retry.Do paths (datacoord handler/recovery refreshing a possibly-stale cache) keep retrying through the unmarked sentinel.

Pattern 6 — Converting an errors.Is control-flow sentinel to merr

merr.Is matches by numeric code alone. A bare errors.New sentinel matches by pointer identity. Convert a control-flow sentinel to a merr and every errors.Is(err, thatSentinel) guard starts matching all errors that share the code — silently.

Real near-miss (review P0): the errIgnored* idempotency family (defined in internal/rootcoord/meta_table.go — errIgnoredCreateCollection, errIgnoredAlterCollection, errIgnoredDropPartition, ...) is caught by errors.Is guards in root_coord.go and translated to success. Had these become merr errors (say, code 5), any ServiceInternal error in those paths would have been swallowed into a fake success — a data-loss class bug.

// ✅ correct as-is (meta_table.go): pointer-identity sentinel, caught
//    in-process, translated before any wire boundary (see the convention doc):
errIgnoredCreateCollection = errors.New("ignored create collection") // create collection with same schema, so it can be ignored.

// root_coord.go — the guard whose semantics a merr conversion would widen:
if errors.Is(err, errIgnoredCreateCollection) { ... }

Before converting any sentinel to merr: grep -rn "errors.Is(.*thatName" — every hit is a guard whose semantics you are about to widen from "this exact signal" to "any error with this code".

Pattern 7 — Boundary conversions that look wrong but are contracts

Two standing examples; both carry code comments — read them before "fixing".

knowhere ConfigInvalid (2006) → ParameterInvalid (1100). At the index-param validation boundary, knowhere reports user-supplied index params as ConfigInvalid. Routing all C-status codes through the segcore system-error table here broke ~25 e2e cases: SDKs assert code 1100 / "invalid parameter" for bad index params. The boundary keeps the special case (internal/util/indexparamcheck/vector_index_checker.go, HandleCStatus):

const knowhereConfigInvalid = 2006
if int32(status.error_code) == knowhereConfigInvalid {
    return merr.WrapErrParameterInvalidMsg("%s", errorMsg)
}
return merr.SegcoreError(int32(status.error_code), errorMsg) // system blame, code preserved in message

This is the reverse of Pattern 1: here the C++ side really is validating user input, so collapsing it into the system table would have been the misclassification.

segcore codes pass through to the wire. An in-band C++ code (2000-2099) now reaches the client as its ORIGINAL value: 2028 stays 2028, 2024 stays 2024, and an in-band code added by a future C++ version passes through unchanged under the ErrSegcore family umbrella. Guardrails: an out-of-band (garbage) code still collapses to 2000; cross-family mappings keep their sentinel's wire code (2046 → ErrCollectionSchemaVersionNotReady, wire 110). Family identity for errors.Is is preserved via inner/Unwrap.

Compatibility (changed in the pass-through PR): previously most codes were projected to wire 2000 with the real code buried in Reason; additionally the sentinels were renumbered to the C++ values they represent — ErrSegcoreUnsupported 2001 → 2003, ErrSegcorePretendFinished 2002 → 2033 — because their old numbers squatted on C++ UnexpectedError / NotImplemented and would false-match under code-based errors.Is. An application matching wire codes 2000/2001/2002 must be updated: wire 2001 now means C++ UnexpectedError itself (an unclassified internal failure), wire 2002 means C++ NotImplemented, and the Unsupported / pretend-finished signals arrive as 2003 / 2033.

Retriability is unchanged by the wire projection: transient codes (2012-2015, 2018, 2027, 2034, 2036-2037, 2040, 2043, 2045-2046, ...) are marked retriable by classForCode; input codes (2025/2026/2028/2031/2032/2042) are InputError. Don't "improve" a call site by hand-picking a 20xx number — go through merr.SegcoreError(code, msg) and let the table decide retriability and projection. Guard tests: pkg/util/merr/segcore_test.go (wire_code_projection, named_sentinel_wire_transitions, TestSegcoreCodeTableCoverage).

---

Choosing a code: quick reference for commonly confused sentinels

Full list: pkg/util/merr/errors.go (the init-time registry panics on duplicate codes). The table covers the choices that actually get confused in practice. "Input" = baked-in WithErrorType(InputError); "boundary" = stamped via WrapErrAsInputError(When) at the proxy chokepoint only.

You want to express…	Use	Not	Why
request value is malformed / out of range	ErrParameterInvalid 1100 (Input)	ErrServiceInternal	user can fix it by changing the request
request lacks a required field	ErrParameterMissing 1101 (Input)	ErrParameterInvalid	both Input; Missing is the precise statement
an internal invariant / contract was violated	ErrServiceInternal 5	ErrParameterInvalid	Patterns 1–2: nobody's request caused it
a worker does not implement a coordinator task type	ErrServiceUnimplemented 10	ErrParameterInvalid / ErrServiceInternal	a typed capability mismatch lets the coordinator fail fast or fall back deliberately
transient "not ready, try again shortly"	ErrServiceUnavailable 2 / ErrServiceNotReady 1 (retriable)	any not-found code	Pattern 5: must survive retry.Do
name lookup failed, name typed by user	ErrCollectionNotFound 100 etc. + boundary stamp	marking the sentinel Input globally	dual identity, Pattern 5
name lookup failed from internal state (id-based, cache refresh)	ErrCollectionNotFound 100 etc., unmarked	ErrServiceInternal	keeps refresh/retry paths alive; stays system-blame
user's import file/data is bad	ErrImportFailed 2100 (Input)	ErrImportSysFailed	2101 is the server-side twin — pick by blame, not by stage
import broke for server-side reasons	ErrImportSysFailed 2101	ErrImportFailed	same, reversed
bad search/query expression or plan	ErrQueryPlan 2201 (Input)	ErrParameterInvalid	dedicated code, SDKs branch on it
object-storage op failed (already typed by mapObjectStorageError)	merr.Wrap(err, …)	WrapErrIoFailedErr	Pattern 3: don't flatten Io taxonomy
C++ status crossing cgo	merr.SegcoreError(code, msg)	hand-picked 20xx sentinel	Pattern 7: the table owns projection/retriability
persisted meta / binlog is corrupt	ErrDataIntegrity 1009	ErrParameterInvalid	stored state, not the current request
function/embedding pipeline contract broke	WrapErrFunctionFailedMsg 2400	ErrServiceInternal	that subsystem's promised code
in-process control-flow signal (idempotent no-op, queue empty)	package-level errors.New sentinel, caught + translated	any merr	Pattern 6; see convention doc

---

For coding agents (and humans in a hurry)

Minimum procedure when a task touches error handling:

1. Read error_handling_guide.md §"Decision tree" and §"Input vs System" — then this casebook's pattern list.
2. Apply the blame test: is the request content itself what forces this branch? → Input factory. A Milvus bug, or an internal/transient failure (e.g. the not-ready condition in Pattern 5, a TOCTOU race), → System factory — even when a correct Milvus does reach it on a valid request (those must stay SystemError so retry.Do keeps retrying).
3. Never use WrapErrXxxErr/WrapErrXxxMsg("%s", err) to add context — merr.Wrap(f) only (Patterns 3–4).
4. Before adding/marking InputError: grep retry.Do consumers (Pattern 5). Before converting a sentinel to merr: grep errors.Is guards (Pattern 6).
5. Don't invent codes, don't hand-edit boundary conversions with comments (Pattern 7), don't return bare errors.New/fmt.Errorf (the linter will reject it anyway).
6. Touched a wire projection, oldCode mapping, or metric label? Run the guard tests: pkg/util/merr/error_classification_test.go (closed-world Input set), segcore_test.go (wire projection), and a full make test-go — contract changes break packages you didn't touch.