Online MAD supports `decay`, but its model-specific arguments and
example omit it, which can make valid configurations look unsupported.
Document the `(0, 1]` range, default `1.0`, adaptation tradeoff and
example setting. Also include Online MAD in the shared decay section’s
model list.
Validation: checked the accepted parameter range/default and existing
section anchor; `git diff --check` passes. Documentation only: one file,
+3 / -1; no runtime changes or full documentation build.
## Motivation
Reduce the learning curve for VictoriaMetrics Anomaly Detection,
especially for engineers without an ML background. Give readers clearer
entry points and a practical path from the behavior of their metrics to
a model, a playground example, and a configuration they can adapt.
## Summary
Add **Choose a model for your metrics**, organized around three
questions: has one metric changed, have related signals changed
together, or is one entity different from its peers?
## Solution
Use concise explanations and illustrations to introduce each comparison,
then provide six expandable recipes covering Online Z-score, Online MAD,
Online Seasonal Quantile, Temporal Envelope, multivariate Temporal
Envelope, and Peer Outlier. Each recipe includes when to use it,
alternatives, a complete reader/model/scheduler example, a reproducible
playground state, expected results, and tuning hints.
## Changes
- Add the guide and three chart illustrations. Include steady CPU,
latency spikes and a level change, seasonal plateaus, evolving traffic,
job mix versus CPU, and an unusual replica among changing peers.
- Compare the multivariate recipe with independent envelopes, CPU per
total job rate, and a known-cost residual rule. Explain when the simpler
rule is enough.
- Link the guide from the main page, Quickstart, FAQ, model overview,
model-type introductions, UI playgrounds, and guides index. Place model
selection before alerting setup.
- Add green tips at the relevant entry points and matching descriptions
in the guides index. Move **Tune a model** after the UI example workflow
and link it from model configuration.
- Render UI availability badges as HTML shortcodes to preserve headings
and anchors in the current Hugo build.
## Validation
- PASS: full local vmdocs build with Hugo 0.166.0, 452 pages.
- PASS: all six recipe schemas, production periodic schedulers, one-fit
historical replays, and shared URL settings.
- PASS: deterministic public query/API checks. The new three-channel
recipe flags 5/6 incident observations with no healthy flags. Standard
independent envelopes and CPU/total-job rate flag none. The known-cost
residual flags 6/6, as documented.
- PASS: public UI restoration and execution for the new multivariate
recipe and ratio baseline. Desktop and 390-pixel mobile rendering
checked.
- PASS: 44 guide destinations/anchors, all six incoming page links, and
whitespace checks. The focused native Linkinator crawl of the new guide
also passes, including image and fragment checks.
- Full-site Linkinator crawl: 14 findings across 11 URLs outside the
guide's outgoing links, including existing Helm/operator assets,
local-service example URLs, and a malformed changelog URL. These are
separate from the passing guide link check.
- Backend/frontend runtime suites were not run because this PR changes
documentation only.
## Diff size
Full PR against VictoriaMetrics master at
`3fec2f72414d00d1acda807c8818d7af90cc966d`.
| Area | Core | Else/tests |
| --- | --- | --- |
| Backend | +0 / -0 | +0 / -0 |
| Frontend | +0 / -0 | +0 / -0 |
| Repository else | +0 / -0 | +1064 / -52 |
| Total | +0 / -0 | +1064 / -52 |
Net text change: +1012 lines across nine Markdown files. Three WebP
assets total 179,236 bytes.
## Changelog
[external] Documentation only. No runtime behavior or release changelog
change.
## Caveats
Playground examples use fixed historical dates and require network
access. Their results illustrate the selected data and settings rather
than guaranteeing detection quality for other workloads.
Previously any transient handshake error timeout can force vminsert into legacy mode,
which drops metric metadata.
This commit binds check only to the first handshake request part and io.EOF error.
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/pull/11762
Previously, the default time range was [0, now]. Such time range would
trigger the search across all indexDBs and all days within each indexDB
and resulted in an unnecessary load on a storage that at times lead to
OOMs.
The default time range is now changed to the current day and if a client
wants list of tenants for a specific date range the client is now
expected to specify that time range excplicitly.
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/11791
---------
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
Signed-off-by: Artem Fetishev <149964189+rtm0@users.noreply.github.com>
Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>
After ae97c11a77 added at
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/11466
LabelNames and LabelValues search request started to obtain metricIDs
directly from tag filters cache. Previous those methods received a copy
of uint64Set. After this change direct cache items. Since LabelNames
retries uint64Set items with AppendTo, which may modify cached object.
It could result into runtime panic.
This commit uses uint64set.Set directly and iterates over each element.
In addition, use Union instead of UnionMayOwn for DeleteSeries
requests. Because it also could mutate cached items.
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/11788
Scratch Docker images don't have `curl` or `wget`, so users can't set up
Docker `HEALTHCHECK` easily. This commit adds `-healthcheck` flag.
When it is set, the binary performs connection check for the running process on
localhost ( or defined listen address) and exits:
- 0 if target port accepted connection at one of `-httpListenAddr`
- 1 otherwise, with the error in the log
Example:
```yaml
healthcheck:
test: ["CMD", "/victoria-metrics-prod", "-healthcheck"]
```
Related issue:
https://github.com/VictoriaMetrics/VictoriaLogs/issues/1565
Prometheus doesn't support the previous `default_rollup()` function and
rejects the restore requests during vmalert restarts.
This commit restores compatibility with Prometheus by quering plain `ALERTS_FOR_STATE{...}` and set `lookback_delta`, `max_lookback` to `-remoteRead.lookback`. Also it sets `step` because VM
uses `step` as the lookback window and `max_lookback` as its default and
cap.
fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/11574
For kubernetes discovery if `role: endpoints` or `role: endpointslice` is set. When an endpoint's
targetRef points to a pod in a different namespace, the common pod
labels were incorrectly added after the endpoint labels and
`__meta_kubernetes_namespace` ended up as the pod's namespace instead of
the Endpoints/EndpointSlice one.
This commit preserves original `Endpoint` and `EndpointSlice` namespace name and namespace labels as Prometheus does.
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/11798
## Description
- Really short addition to the docs, might want to adapt the styling to
your preferences
- Derived playground link from links above, might not be desirable, LMK
## Background
I used to deploy Grafana for dashboarding, felt a bit cluttered for me.
Moreover,
[grafana-operator](https://github.com/grafana/grafana-operator) JSON
semantics for Grafana are heavy (yet powerful).
However, VMUI's dashboards are a perfect fit for my use case.
I spent a lot of time in Victoria's docs (very good quality BTW), but
didn't see mention of the dashboarding capabilities.
Having things like #11274 and configuration of default for `Table View`
& `Hide Common Labels` would be great (variables templating would be
even greater but more complicated) additions.
I truly think dashboard capabilities showcasing could be beneficial for
many people, as it is natively available, really easy to use (e.g. a
`ConfigMap` mounted via `vmselect.configMaps` for `VMCluster`), and
powerful enough for cases that do not need (maybe more like _"that do
not feel like needing"_) extensive features like ultra-customizable
graphs, ML, AI, ... I really appreciate the simplicity of it.
VictoriaTraces' recent UI upgrade for trace viewing is also awesome.
Thanks for building such a great product!
---------
Co-authored-by: Max Kotliar <mkotlyar@victoriametrics.com>
1. The instance filter was inconsistent across the panels. Pressure and
memory allocation panels now use the instance filter the same as the
others.
2. Disk I/O and syscall panels showed no data instead of a zero value.
This could lead to the wrong decision that metrics are unavailable.
Checked single, multiple and All instance selections, zero values and
missing metrics. Synced the VM copy.
---------
Co-authored-by: Max Kotliar <mkotlyar@victoriametrics.com>
Adds `vm_request_duration_seconds{path="/admin/tenants"}` for tracking
the time it takes to fetch tenant IDs from the database. With the switch
from global to per-day index this call can become slower than before.
Related PR https://github.com/VictoriaMetrics/VictoriaMetrics/pull/11785
The brute-force protection delays requests with invalid auth tokens by
2-3 seconds. Since v1.154.0 this also applies to requests proxied to
`unauthorized_user`. Introduced in
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/11680.
This breaks setups with two chained vmauth instances, where the first
one authenticates requests via `users` and the second one proxies them
via `unauthorized_user` (e.g. VMDistributed or VMCluster managed by the
operator). The second vmauth receives the forwarded `Authorization`
header, treats it as an invalid auth token and delays every request.
Skip the slowdown if no `users` are configured, since there is nothing
to brute-force in this case.
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/11774
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/pull/11777
---
It would also be good to patch chained vmauth in `VMDistributed`,
`VMCluster` and the like so they stop proxying auth headers. That's
what's recommended in [vmauth security
guides](https://docs.victoriametrics.com/victoriametrics/vmauth/#security)
(see p6)
cc @Haleygo @vrutkovs
Previously the tr was [0, MaxInt64] to indicate that the legacy indexDB
may contain any non-negative timestamp.
But the the time range can be narrowed down to [0, now+2d]. Say a
deployment that uses legacy indexDB (versions before v1.133.0). Then it
is restarted to with a version >= v1.133.0. It now uses partition
indexDBs and legacy indexDBs become read-only. Meaning that legacy
indexDBs now contain entries up until the day of restart. The day of
switching to partition indexDB is not recorded anywhere so the best we
can do is just use the day of the last restart. Also adding 2d because
vmstorage accepted timestamps with max 2d in the future to account for
time zone shifts.
The support of future timestamps (-futureRetention) was added after
v1.133.0 so it is guaranteed that versions that write to legacy indexDBs
(< v1.133.0) do not support them.
Extracted from https://github.com/VictoriaMetrics/VictoriaMetrics/pull/11196
---------
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
Adds a standalone guide for recovering data missing from one
VictoriaMetrics installation but still available in another. It explains
`vmagent`'s role in Multi-Cluster and Multi-AZ deployments, checks for
drops before recommending recovery, and selects a widened recovery
window from queue history.
Includes an all-tenant cluster example using `--vm-intercluster`, a
separate single-node example, and exact `1ms` deduplication guidance.
The cluster page links to the guide.
Validation: both examples pass `bash -n`, and the flags and metrics were
checked against current help and source. Local checks with real `vmctl`
and four single-node VictoriaMetrics processes verified recovery and
overlap deduplication. An HTTP routing harness verified that both tenant
IDs were preserved; a full cluster E2E was not run.
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/9854.
PR https://github.com/VictoriaMetrics/VictoriaMetrics/pull/11626
---------
Signed-off-by: Max Kotliar <kotlyar.maksim@gmail.com>
Co-authored-by: Max Kotliar <mkotlyar@victoriametrics.com>
the quick start CPU graph stays empty for ~100s, because `scrape.yaml`
has no `scrape_interval` so node_exporter is scraped once a minute, and
`rate()` needs two samples. this sets `scrape_interval: 10s` to match
`-selfScrapeInterval`, which brought it down to 43s on my machine.
the cluster section also gets the `foo:bar` vmauth login (vmui returns
401 without it), the current container list, and a fixed link to the
single-node monitoring page.
Previously, a target exposing many malformed lines (e.g. an HTML page)
logged an error for every line on every scrape, flooding the error log.
Add `-promscrape.maxParseErrorsPerScrape` command-line flag (20 by
default). Parse errors beyond the limit aren't logged; instead, a single
summary line with the number of suppressed errors and the target URL,
job and labels is logged after the scrape. Zero or negative value
disables the limit. It respects `-promscrape.suppressScrapeErrors`.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8856
PR https://github.com/VictoriaMetrics/VictoriaMetrics/pull/11727
Note: Prometheus stops parsing at the first error, discards the whole
scrape and sets `up=0`. It logs at most one parse error per scrape.
vmagent skips malformed lines, ingests the valid ones and keeps `up=1`,
so it can log many errors per scrape.
This PR removes the sharded hyperscale path [to satisfy the
comment](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/10201#discussion_r2690467397).
Each storage cell receives a full copy of the data, and the guide
compares two read paths for this setup. Also addressed several issues
with unclear configurations and explanations. I removed one more path
with stateless vmagent because it was odd and impractical.
---------
Co-authored-by: hagen1778 <roman@victoriametrics.com>
indexDB is now responsible for checking whether it might contain data
for the given search time range.
Previously this was done by Storage. I.e. it was adjusting the search
time range for every indexDB it was searching. This approach worked,
but indexDBs didn't know what time range they were accepting. One could
potentially pass any arbitrary time range to any indexDB and it would
search that time range. Such mistakes can never be caused by user input
but it still good to move this responsibility to indexDB to make it more
self-sufficient and independent from Storage and decrease the role of
special globalIndexTimeRange.
This change needs to be done before https://github.com/VictoriaMetrics/VictoriaMetrics/pull/11752.
---------
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
## Motivation
Document vmanomaly v1.31.1 and use the patch release in current
deployment examples.
## Summary
Add the backend changelog entry for periodic peer-model data fetching,
update current standard/DHI image references, and document
upgrade/rollback state compatibility.
## Changes
- `docs/anomaly-detection/CHANGELOG.md`: v1.31.1 periodic peer-model
data-fetch fix.
- `docs/anomaly-detection/Migration.md`: v1.31.0 → v1.31.1 compatibility
and rollback limitation.
- `docs/anomaly-detection/QuickStart.md`: standard and DHI image
examples.
- `docs/anomaly-detection/UI.md`: current Docker example and recommended
backend version (v1.31.1 or newer). No new UI release entry or UI
feature changes.
- `docs/anomaly-detection/components/models.md`: custom-model Docker
examples.
- `docs/anomaly-detection/guides/guide-vmanomaly-vmalert/README.md`:
deployment image.
- `deployment/docker/vmanomaly/vmanomaly-integration/compose.yml`:
integration image.
Historical feature-introduction versions and existing changelog entries
are preserved. No routine dependency-update notes are added.
## Validation
- `git diff --check`: passed.
- Integration Compose YAML parses successfully.
- Automated comparison confirms the existing UI release notes are
unchanged apart from the recommended backend version and its
explanation.
- Upgrade/rollback notes match the release compatibility graph.
- Full documentation site build not run; edits are Markdown text and
image tags.
## Diff size
Against VictoriaMetrics master `9f983ffe6`: seven files, +21/-14, net
+7. Backend/frontend implementation and tests: +0/-0. Repository
documentation/configuration: +21/-14.
## Publication
Draft for review. Confirm v1.31.1 image publication before merging.
Previously, sendBlock could block forever if kafka broker changed
certificates and was not able to reply with error.
This commit adds a check, if current version of producer was stopped and client must be re-created with new certificates.
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/11737
Previously, Kubernetes service-discovery created aggregated map for scrape objects and deleted entries from it
with empty scrape objects. Resulted map was assinged to api watcher. Since item deletion from map doesn't actually free memory allocated for it, it only removed objects. A lot of memory was used for map with a single element.
This commit reduce memory usage footprint by adding items to the map only when it's needed. It significantly dropped memory usage for large Kubernetes clusters with `VictoriaMetrics` and `Prometheus` operator installations.
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/pull/11751/
Fred Navruzov