mirror of
https://github.com/ClickHouse/ClickHouse.git
synced 2026-10-28 16:18:23 +08:00
Merge pull request #120054 from JTCunning/prometheus-docs-split
docs(promql): Split the Prometheus interface docs into two pages and improve readability of PromQL.
This commit is contained in:
3 files changed
+110
-90
No files matched your search
@@ -0,0 +1,85 @@
|
||||
---
|
||||
description: 'Prometheus metrics endpoint in ClickHouse: expose server metrics on a dedicated port for scraping.'
|
||||
sidebarTitle: 'Prometheus metrics endpoint'
|
||||
slug: /interfaces/prometheus-metrics
|
||||
title: 'Prometheus metrics endpoint'
|
||||
doc_type: 'reference'
|
||||
---
|
||||
|
||||
ClickHouse exposes its own metrics on a dedicated port in the Prometheus text format, so that a Prometheus server can scrape them.
|
||||
|
||||
To store Prometheus metrics in ClickHouse and query them with PromQL, see [Prometheus HTTP API and PromQL](/interfaces/prometheus) instead.
|
||||
|
||||
<Note>
|
||||
If you're using ClickHouse Cloud, you can expose metrics to Prometheus using the [Prometheus Integration](/products/cloud/features/monitoring/prometheus).
|
||||
</Note>
|
||||
|
||||
## Configure the metrics port {#configure-the-metrics-port}
|
||||
|
||||
Configure a dedicated port when a Prometheus server needs to scrape ClickHouse's own metrics:
|
||||
|
||||
```xml
|
||||
<prometheus>
|
||||
<port>9363</port>
|
||||
<endpoint>/metrics</endpoint>
|
||||
<metrics>true</metrics>
|
||||
<asynchronous_metrics>true</asynchronous_metrics>
|
||||
<events>true</events>
|
||||
<errors>true</errors>
|
||||
<histograms>true</histograms>
|
||||
<dimensional_metrics>true</dimensional_metrics>
|
||||
</prometheus>
|
||||
```
|
||||
|
||||
## Configure extended handlers {#configure-extended-handlers}
|
||||
|
||||
Section `<prometheus.handlers>` can be used to make more extended handlers on the same port.
|
||||
This section is similar to [`<http_handlers>`](/concepts/features/interfaces/http) but works for prometheus protocols:
|
||||
|
||||
```xml
|
||||
<prometheus>
|
||||
<port>9363</port>
|
||||
<handlers>
|
||||
<my_rule_1>
|
||||
<url>/metrics</url>
|
||||
<handler>
|
||||
<type>expose_metrics</type>
|
||||
<metrics>true</metrics>
|
||||
<asynchronous_metrics>true</asynchronous_metrics>
|
||||
<events>true</events>
|
||||
<errors>true</errors>
|
||||
<histograms>true</histograms>
|
||||
<dimensional_metrics>true</dimensional_metrics>
|
||||
<labels>
|
||||
<environment>production</environment>
|
||||
<shard from_env="SHARD_NAME"></shard>
|
||||
</labels>
|
||||
</handler>
|
||||
</my_rule_1>
|
||||
</handlers>
|
||||
</prometheus>
|
||||
```
|
||||
|
||||
## Settings {#settings}
|
||||
|
||||
For the complete behavior of these settings, see the [Prometheus section](/reference/settings/server-settings/settings/other#prometheus) of the server settings reference.
|
||||
|
||||
| Name | Default | Description |
|
||||
|---|---|---|
|
||||
| `port` | none | Port that serves ClickHouse metrics. |
|
||||
| `endpoint` | `/metrics` | HTTP endpoint for scraping metrics. It starts with `/`. Should not be used with the `<handlers>` section. |
|
||||
| `url` / `headers` / `method` | none | Filters used to find a matching handler for a request. Similar to the fields with the same names in the [`<http_handlers>`](/concepts/features/interfaces/http) section. |
|
||||
| `info` | true | Exposes the `ClickHouse_Info` gauge with server identity labels (`name`, `version`, `version_describe`, `version_major`, `version_minor`, `version_patch`). |
|
||||
| `metrics` | true | Exposes metrics from [`system.metrics`](/reference/system-tables/metrics). |
|
||||
| `asynchronous_metrics` | true | Exposes metrics from [`system.asynchronous_metrics`](/reference/system-tables/asynchronous_metrics). |
|
||||
| `events` | true | Exposes metrics from [`system.events`](/reference/system-tables/events). |
|
||||
| `errors` | true | Exposes error counts from [`system.errors`](/reference/system-tables/errors). |
|
||||
| `histograms` | true | Exposes metrics from [`system.histogram_metrics`](/reference/system-tables/histogram_metrics). |
|
||||
| `dimensional_metrics` | true | Exposes metrics from [`system.dimensional_metrics`](/reference/system-tables/dimensional_metrics). |
|
||||
| `labels` | none | Constant labels added to every exposed metric. Each child element defines one label: the element name is the label name (which must match `[a-zA-Z_][a-zA-Z0-9_]*`) and the element value is the label value. Label values support standard config substitutions such as the `from_env` attribute. A label name is rejected when it starts with `__` (reserved by Prometheus), or when it would collide with a label this endpoint already writes for one of its enabled sections. The reserved set therefore follows the endpoint's active export surface: `le` when `histograms` is enabled; the `ClickHouse_Info` labels (`name`, `version`, `version_describe`, `version_major`, `version_minor`, `version_patch`) when `info` is enabled; and any label used by an exposed histogram or dimensional metric family (for example, `group`, `direction`, or `operation_type`) when `histograms` or `dimensional_metrics` is enabled. Because it depends on what the endpoint actually exposes, a name can be valid on one endpoint but rejected on another. |
|
||||
|
||||
## Check the endpoint {#check-the-endpoint}
|
||||
|
||||
```bash
|
||||
curl http://127.0.0.1:9363/metrics
|
||||
```
|
||||
@@ -1,90 +1,22 @@
|
||||
---
|
||||
description: 'Prometheus HTTP API support in ClickHouse: remote write, remote read, PromQL queries, and server metrics.'
|
||||
sidebarTitle: 'Prometheus and PromQL'
|
||||
description: 'Prometheus HTTP API support in ClickHouse: remote write, remote read, and PromQL queries over a TimeSeries table.'
|
||||
sidebarTitle: 'Prometheus API and PromQL'
|
||||
slug: /interfaces/prometheus
|
||||
title: 'Prometheus protocols and PromQL'
|
||||
title: 'Prometheus HTTP API and PromQL'
|
||||
doc_type: 'reference'
|
||||
---
|
||||
|
||||
import { CloudNotSupportedBadge } from "/snippets/components/CloudNotSupportedBadge/CloudNotSupportedBadge.jsx";
|
||||
|
||||
## Expose ClickHouse server metrics {#expose}
|
||||
|
||||
<Note>
|
||||
If you're using ClickHouse Cloud, you can expose metrics to Prometheus using the [Prometheus Integration](/products/cloud/features/monitoring/prometheus).
|
||||
</Note>
|
||||
|
||||
Configure a dedicated port when a Prometheus server needs to scrape ClickHouse's own metrics:
|
||||
|
||||
```xml
|
||||
<prometheus>
|
||||
<port>9363</port>
|
||||
<endpoint>/metrics</endpoint>
|
||||
<metrics>true</metrics>
|
||||
<asynchronous_metrics>true</asynchronous_metrics>
|
||||
<events>true</events>
|
||||
<errors>true</errors>
|
||||
<histograms>true</histograms>
|
||||
<dimensional_metrics>true</dimensional_metrics>
|
||||
</prometheus>
|
||||
```
|
||||
|
||||
Section `<prometheus.handlers>` can be used to make more extended handlers on the same port.
|
||||
This section is similar to [`<http_handlers>`](/concepts/features/interfaces/http) but works for prometheus protocols:
|
||||
|
||||
```xml
|
||||
<prometheus>
|
||||
<port>9363</port>
|
||||
<handlers>
|
||||
<my_rule_1>
|
||||
<url>/metrics</url>
|
||||
<handler>
|
||||
<type>expose_metrics</type>
|
||||
<metrics>true</metrics>
|
||||
<asynchronous_metrics>true</asynchronous_metrics>
|
||||
<events>true</events>
|
||||
<errors>true</errors>
|
||||
<histograms>true</histograms>
|
||||
<dimensional_metrics>true</dimensional_metrics>
|
||||
<labels>
|
||||
<environment>production</environment>
|
||||
<shard from_env="SHARD_NAME"></shard>
|
||||
</labels>
|
||||
</handler>
|
||||
</my_rule_1>
|
||||
</handlers>
|
||||
</prometheus>
|
||||
```
|
||||
|
||||
Settings:
|
||||
|
||||
| Name | Default | Description |
|
||||
|---|---|---|
|
||||
| `port` | none | Port that serves ClickHouse metrics. |
|
||||
| `endpoint` | `/metrics` | HTTP endpoint for scraping metrics. It starts with `/`. Should not be used with the `<handlers>` section. |
|
||||
| `url` / `headers` / `method` | none | Filters used to find a matching handler for a request. Similar to the fields with the same names in the [`<http_handlers>`](/concepts/features/interfaces/http) section. |
|
||||
| `info` | true | Exposes the `ClickHouse_Info` gauge with server identity labels (`name`, `version`, `version_describe`, `version_major`, `version_minor`, `version_patch`). |
|
||||
| `metrics` | true | Exposes metrics from [`system.metrics`](/reference/system-tables/metrics). |
|
||||
| `asynchronous_metrics` | true | Exposes metrics from [`system.asynchronous_metrics`](/reference/system-tables/asynchronous_metrics). |
|
||||
| `events` | true | Exposes metrics from [`system.events`](/reference/system-tables/events). |
|
||||
| `errors` | true | Exposes error counts from [`system.errors`](/reference/system-tables/errors). |
|
||||
| `histograms` | true | Exposes metrics from [`system.histogram_metrics`](/reference/system-tables/histogram_metrics). |
|
||||
| `dimensional_metrics` | true | Exposes metrics from [`system.dimensional_metrics`](/reference/system-tables/dimensional_metrics). |
|
||||
| `labels` | none | Constant labels added to every exposed metric. Each child element defines one label: the element name is the label name (which must match `[a-zA-Z_][a-zA-Z0-9_]*`) and the element value is the label value. Label values support standard config substitutions such as the `from_env` attribute. A label name is rejected when it starts with `__` (reserved by Prometheus), or when it would collide with a label this endpoint already writes for one of its enabled sections. The reserved set therefore follows the endpoint's active export surface: `le` when `histograms` is enabled; the `ClickHouse_Info` labels (`name`, `version`, `version_describe`, `version_major`, `version_minor`, `version_patch`) when `info` is enabled; and any label used by an exposed histogram or dimensional metric family (for example, `group`, `direction`, or `operation_type`) when `histograms` or `dimensional_metrics` is enabled. Because it depends on what the endpoint actually exposes, a name can be valid on one endpoint but rejected on another. |
|
||||
|
||||
Check the endpoint:
|
||||
|
||||
```bash
|
||||
curl http://127.0.0.1:9363/metrics
|
||||
```
|
||||
|
||||
<CloudNotSupportedBadge/>
|
||||
|
||||
## Prometheus HTTP API and PromQL {#prometheus-http-api-and-promql}
|
||||
|
||||
ClickHouse implements the Prometheus HTTP API over a [`TimeSeries`](/reference/engines/table-engines/integrations/time-series) table. One handler serves remote write, remote read, instant PromQL queries, and range PromQL queries.
|
||||
|
||||
### Prerequisites {#prerequisites}
|
||||
To expose ClickHouse's own metrics for a Prometheus server to scrape, see the [Prometheus metrics endpoint](/concepts/features/interfaces/prometheus-metrics).
|
||||
|
||||
## Prerequisites {#prerequisites}
|
||||
|
||||
### Enable the TimeSeries setting {#enable-the-timeseries-setting}
|
||||
|
||||
Enable the `enable_time_series_table` setting for the user that creates and accesses the table:
|
||||
|
||||
@@ -92,16 +24,9 @@ Enable the `enable_time_series_table` setting for the user that creates and acce
|
||||
SET enable_time_series_table = 1;
|
||||
```
|
||||
|
||||
Create a database and a `TimeSeries` table:
|
||||
|
||||
```sql
|
||||
CREATE DATABASE prometheus;
|
||||
CREATE TABLE prometheus.metrics ENGINE = TimeSeries;
|
||||
```
|
||||
|
||||
For HTTP API requests, enable `enable_time_series_table` in the profile of the API user.
|
||||
|
||||
### Configure the Prometheus API {#configure-prometheus-api}
|
||||
### Configure the Prometheus API endpoints {#configure-prometheus-api}
|
||||
|
||||
Configure one prefix-routed handler on the main ClickHouse HTTP port:
|
||||
|
||||
@@ -151,7 +76,16 @@ Routing and handler settings:
|
||||
| `table` | none | The name of a `TimeSeries` table. When omitted, the request must provide the `table` query parameter. The configured name can include a database. |
|
||||
| `database` | none | The database containing the table. A request can provide it as a query parameter. When omitted, ClickHouse uses a database from a qualified `table` value or falls back to `default`. |
|
||||
|
||||
### Ingest metrics with remote write {#remote-write}
|
||||
### Create a TimeSeries table {#create-a-timeseries-table}
|
||||
|
||||
Create a database and a `TimeSeries` table:
|
||||
|
||||
```sql
|
||||
CREATE DATABASE prometheus;
|
||||
CREATE TABLE prometheus.metrics ENGINE = TimeSeries;
|
||||
```
|
||||
|
||||
## Ingest metrics with remote write {#remote-write}
|
||||
|
||||
ClickHouse supports the [Prometheus remote-write protocol](https://prometheus.io/docs/specs/remote_write_spec/). Configure Prometheus to write to the handler:
|
||||
|
||||
@@ -174,7 +108,7 @@ remote_write:
|
||||
|
||||
ClickHouse acknowledges an asynchronous remote-write request only after the data is flushed to all inner tables of the `TimeSeries` table, regardless of the [`wait_for_async_insert`](/reference/settings/session-settings/wait-for#wait_for_async_insert) setting: the remote-write protocol treats an acknowledged write as durable. If the flush fails, the request returns an error and Prometheus retries it.
|
||||
|
||||
### Query with PromQL {#promql-query-support}
|
||||
## Query with PromQL {#promql-query-support}
|
||||
|
||||
Use the instant-query endpoint to evaluate a PromQL expression at one point in time:
|
||||
|
||||
@@ -211,7 +145,7 @@ The expression is returned serialized from the parsed query, with the whitespace
|
||||
|
||||
See the [supported PromQL features](/reference/functions/table-functions/prometheusQueryRange#supported-promql-features) for the function and aggregation operator list used by the HTTP API, the `promql` dialect, and the table functions.
|
||||
|
||||
#### Grafana {#grafana}
|
||||
### Grafana {#grafana}
|
||||
|
||||
Configure a Prometheus data source with the base URL ending before `/api/v1`:
|
||||
|
||||
@@ -237,7 +171,7 @@ Grafana appends `/api/v1/query` or `/api/v1/query_range` to this base URL and ad
|
||||
Only the query endpoints `/api/v1/query`, `/api/v1/query_range`, and `/api/v1/format_query` and the metadata endpoints `/api/v1/series`, `/api/v1/labels`, `/api/v1/label/<name>/values`, and `/api/v1/metadata` are implemented. `/api/v1/series` requires at least one `match[]` series selector, supports the optional `start`, `end`, and `limit` parameters, and returns the union of the series matched by each selector. `/api/v1/labels` accepts the same parameters, with `match[]` being optional, and returns the sorted label names of the matched series (or of all series when no selectors are given). `/api/v1/label/<name>/values` accepts the same parameters as `/api/v1/labels` and returns the sorted values of one label, with `<name>` optionally using the Prometheus `U__...` escaping of label names that contain characters outside `[a-zA-Z0-9_]`. These endpoints cover what a Grafana Prometheus datasource uses for label browsing, template variables, and query-builder autocomplete.
|
||||
</Note>
|
||||
|
||||
#### SQL entry points {#sql-entry-points}
|
||||
### SQL entry points {#sql-entry-points}
|
||||
|
||||
ClickHouse uses the same PromQL converter for the HTTP API, the `promql` dialect, and the [`prometheusQuery`](/reference/functions/table-functions/prometheusQuery) and [`prometheusQueryRange`](/reference/functions/table-functions/prometheusQueryRange) table functions.
|
||||
|
||||
@@ -262,7 +196,7 @@ FROM prometheusQuery(
|
||||
);
|
||||
```
|
||||
|
||||
### Query metric metadata {#metadata}
|
||||
## Query metric metadata {#metadata}
|
||||
|
||||
The `/prometheus/api/v1/metadata` endpoint returns the metric metadata stored in the `Metrics` target table of the `TimeSeries` table: the type, help text, and unit of each metric family. It supports the following Prometheus parameters in the URL query string:
|
||||
|
||||
@@ -282,7 +216,7 @@ curl --user default:<password> --get \
|
||||
--data-urlencode "table=metrics"
|
||||
```
|
||||
|
||||
### Read metrics with remote read {#remote-read}
|
||||
## Read metrics with remote read {#remote-read}
|
||||
|
||||
ClickHouse supports the [Prometheus remote-read protocol](https://prometheus.io/docs/prometheus/latest/querying/remote_read_api/) at `/prometheus/api/v1/read`.
|
||||
|
||||
|
||||
@@ -227,6 +227,7 @@
|
||||
{ "$ref": "features/interfaces/odbc/navigation.json" },
|
||||
"concepts/features/interfaces/postgresql",
|
||||
"concepts/features/interfaces/prometheus",
|
||||
"concepts/features/interfaces/prometheus-metrics",
|
||||
"concepts/features/interfaces/ssh",
|
||||
"concepts/features/interfaces/grpc",
|
||||
"concepts/features/interfaces/arrowflight"
|
||||
|
||||
Reference in new issue
Block a user