Files
zhenshan.caoandClaude Opus 5 b67818e600 fix: [2.6] keep proxy metric status label stable, split cause into its own label (#51495) (#51909)
Cherry-pick of #51496
pr: #51495
issue: #51493

### Why this branch needs it

#51496 landed this fix on `hotfix-2.6.20` (shipped as v2.6.20) but
**never on `2.6`**. `2.6` still carries the v2.6.19 in-place status
split — `metrics.go` has `FailInputLabel` / `FailSystemLabel` /
`RejectedUserLabel` / `RejectedSystemLabel`, and `req_count` has no
`cause` label at all.

So the next release cut from `2.6` would emit `fail_input` /
`fail_system` again: a user upgrading **v2.6.20 -> 2.6.21** would see
the dashboards and alert rules that v2.6.20 had just repaired break a
second time — and this time as a regression from a working state, which
is harder to notice than the original break.

### What

#50221 split the `status` label values of `milvus_proxy_req_count` /
`milvus_proxy_grpc_latency` in place. It shipped in **v2.6.19**, so
upgrading from <= v2.6.18 silently zeroes every existing dashboard panel
and alert rule matching `status="fail"` / `status="rejected"` — an alert
that stops firing rather than erroring.

This keeps the classification but moves it onto its own dimension,
restoring the status domain to what <= v2.6.18 emitted:

```
status: success | fail | rejected | retry | total | abandon    (as before v2.6.19)
cause:  user | system | cancel                                 (new, only on fail/rejected)
```

| v2.6.19 / v2.6.20 | this PR |
|---|---|
| `status="fail_input"` | `status="fail", cause="user"` |
| `status="fail_system"` | `status="fail", cause="system"` |
| `status="rejected_user"` | `status="rejected", cause="user"` |
| `status="rejected_system"` | `status="rejected", cause="system"` |
| `status="cancel"` | `status="fail"` or `"rejected"`, `cause="cancel"`
|
| `success` / `retry` / `total` / `abandon` | unchanged, **no cause** |

Outcomes that have no responsible party emit `cause=""`. Prometheus
treats an empty label value as equivalent to the label being absent, so
those series stay byte-identical to what pre-2.6.19 emitted — the new
label materializes only where it carries information.

Old queries work unchanged and still count each request once:
`status="fail"` is again every hard failure, because Prometheus
aggregates over `cause` for free.

### Cherry-pick notes

`git cherry-pick` of #51496 applied with **no conflicts** (19 files).
Cross-checked file by file against `hotfix-2.6.20`: 16 files are
byte-identical, and the 3 that differ do so only in `2.6`-only content
unrelated to this change (`internal/proxy/util.go` —
`validateElementType`; `handler_v2.go` and `util_test.go` — surrounding
`2.6` code). The cause-related lines in all three were extracted and
compared separately: identical.

The emit surface was **re-verified on this branch rather than assumed**:
all 12 `WithLabelValues` sites for the two metrics were checked for
label arity by a paren-balanced scan, because adding a label makes arity
a runtime panic, not a compile error. (`2.6` has 12 sites where
master/3.0 have 62 — it lacks the `InterceptorImpl` / `snapshot_impl.go`
surface.) No stale `FailInputLabel` / `FailSystemLabel` /
`RejectedUserLabel` / `RejectedSystemLabel` references remain, and all
`ParseMetricLabel` / `failMetricLabel` call sites use the two-value
signature.

### Verification on this branch

- Compiles: `./internal/...` and the `pkg` module.
- Green: `internal/proxy`, `internal/distributed/proxy`,
`internal/distributed/proxy/httpserver`,
`internal/distributed/proxy/client`, `pkg/metrics`,
`pkg/util/requestutil`, `pkg/util/merr`.
- `make verifiers`: 0 issues.

### Related

- #51496 — `hotfix-2.6.20` (merged, the version this is taken from)
- #51526 — `3.0`
- #51907 — `master` (aligns `CauseNA` there from `"na"` to `""`)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-07-28 22:50:18 +08:00
..