mirror of
https://github.com/ClickHouse/ClickHouse.git
synced 2026-10-28 16:18:23 +08:00
Allow Map data type for asynchronous metrics
Per-CPU-core and per-device asynchronous metrics become single key-value
metrics: OS*TimeCPU<n>, CPUFrequencyMHz_<n>, Temperature*, EDAC*,
Block*_<device>, Network(Receive|Send)*_<interface>, Disk*_<disk>,
<disk>(Dead|Missing)BlobsQueueEstimate and AsyncLogging<Channel>QueueSize
are now published as a Map from the entity name to the value.
- system.asynchronous_metrics gets a key_values Map(String, Float64)
column; the value column is NaN for key-value metrics.
- system.asynchronous_metric_log logs them as one row per key via a new
key column; the default ORDER BY is now metric, key, event_date, event_time.
- The Prometheus endpoint exports them with a label, e.g.
ClickHouseAsyncMetrics_BlockReadBytes{device="sda"}.
- The Graphite MetricsTransmitter sends them as <prefix>.<Metric>.<key>.
Closes: https://github.com/ClickHouse/ClickHouse/issues/115321
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Alexey MilovidovandClaude Fable 5 committed
1 parent
3436feffff
commit
faa531eb0b
14 files changed
+693
-308
No files matched your search
@@ -15,6 +15,8 @@ import SystemTableCloud from '/snippets/_system_table_cloud.mdx';
|
||||
|
||||
Contains the historical values for `system.asynchronous_metrics`, which are saved once per time interval (one second by default). Enabled by default.
|
||||
|
||||
Key-value metrics of `system.asynchronous_metrics` (those broken down per CPU core, block device, network interface, or disk) are logged as one row per key, with the key in the `key` column. For scalar metrics the `key` column is empty.
|
||||
|
||||
## Columns {#columns}
|
||||
|
||||
{/*AUTOGENERATED_START*/}
|
||||
@@ -22,6 +24,7 @@ Contains the historical values for `system.asynchronous_metrics`, which are save
|
||||
- `event_date` ([Date](/reference/data-types/date)) — Event date.
|
||||
- `event_time` ([DateTime](/reference/data-types/datetime)) — Event time.
|
||||
- `metric` ([LowCardinality(String)](/reference/data-types/lowcardinality)) — Metric name.
|
||||
- `key` ([LowCardinality(String)](/reference/data-types/lowcardinality)) — The key of a key-value metric, e.g. the CPU core number or the block device name. Empty for scalar metrics.
|
||||
- `value` ([Float64](/reference/data-types/float)) — Metric value.
|
||||
{/*AUTOGENERATED_END*/}
|
||||
|
||||
@@ -38,6 +41,7 @@ hostname: clickhouse.eu-central1.internal
|
||||
event_date: 2023-11-14
|
||||
event_time: 2023-11-14 14:39:07
|
||||
metric: AsynchronousHeavyMetricsCalculationTimeSpent
|
||||
key:
|
||||
value: 0.001
|
||||
|
||||
Row 2:
|
||||
@@ -46,6 +50,7 @@ hostname: clickhouse.eu-central1.internal
|
||||
event_date: 2023-11-14
|
||||
event_time: 2023-11-14 14:39:08
|
||||
metric: AsynchronousHeavyMetricsCalculationTimeSpent
|
||||
key:
|
||||
value: 0
|
||||
|
||||
Row 3:
|
||||
@@ -54,6 +59,7 @@ hostname: clickhouse.eu-central1.internal
|
||||
event_date: 2023-11-14
|
||||
event_time: 2023-11-14 14:39:09
|
||||
metric: AsynchronousHeavyMetricsCalculationTimeSpent
|
||||
key:
|
||||
value: 0
|
||||
```
|
||||
|
||||
|
||||
@@ -15,49 +15,50 @@ import SystemTableCloud from '/snippets/_system_table_cloud.mdx';
|
||||
|
||||
Contains metrics that are calculated periodically in the background. For example, the amount of RAM in use.
|
||||
|
||||
A metric is either scalar (a single number in the `value` column) or key-value: broken down per entity, such as per CPU core, block device, network interface, or disk. Key-value metrics are represented as a single row with a [Map](/reference/data-types/map) in the `key_values` column (for example, `CPUFrequencyMHz` maps every CPU core number to its frequency), and their `value` column is `NaN`.
|
||||
|
||||
## Columns {#columns}
|
||||
|
||||
{/*AUTOGENERATED_START*/}
|
||||
- `metric` ([String](/reference/data-types/string)) — Metric name.
|
||||
- `value` ([Float64](/reference/data-types/float)) — Metric value.
|
||||
- `value` ([Float64](/reference/data-types/float)) — Metric value. For key-value metrics (broken down per CPU core, block device, disk, ...) it is NaN, and the values are in the `key_values` column.
|
||||
- `key_values` ([Map(String, Float64)](/reference/data-types/map)) — Values of a key-value metric, e.g. keyed by the CPU core number or the block device name. Empty for scalar metrics.
|
||||
- `description` ([String](/reference/data-types/string) - Metric description)
|
||||
{/*AUTOGENERATED_END*/}
|
||||
|
||||
## Example {#example}
|
||||
|
||||
```sql
|
||||
SELECT * FROM system.asynchronous_metrics LIMIT 10
|
||||
SELECT metric, value, key_values
|
||||
FROM system.asynchronous_metrics
|
||||
WHERE metric IN ('MemoryResident', 'OSUserTime', 'DiskTotal', 'BlockReadBytes', 'NetworkReceiveBytes')
|
||||
ORDER BY metric
|
||||
```
|
||||
|
||||
```text
|
||||
┌─metric──────────────────────────────────┬──────value─┬─description────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
|
||||
│ AsynchronousMetricsCalculationTimeSpent │ 0.00179053 │ Time in seconds spent for calculation of asynchronous metrics (this is the overhead of asynchronous metrics). │
|
||||
│ NumberOfDetachedByUserParts │ 0 │ The total number of parts detached from MergeTree tables by users with the `ALTER TABLE DETACH` query (as opposed to unexpected, broken or ignored parts). The server does not care about detached parts and they can be removed. │
|
||||
│ NumberOfDetachedParts │ 0 │ The total number of parts detached from MergeTree tables. A part can be detached by a user with the `ALTER TABLE DETACH` query or by the server itself it the part is broken, unexpected or unneeded. The server does not care about detached parts and they can be removed. │
|
||||
│ TotalRowsOfMergeTreeTables │ 2781309 │ Total amount of rows (records) stored in all tables of MergeTree family. │
|
||||
│ TotalBytesOfMergeTreeTables │ 7741926 │ Total amount of bytes (compressed, including data and indices) stored in all tables of MergeTree family. │
|
||||
│ NumberOfTables │ 93 │ Total number of tables summed across the databases on the server, excluding the databases that cannot contain MergeTree tables. The excluded database engines are those who generate the set of tables on the fly, like `Lazy`, `MySQL`, `PostgreSQL`, `SQlite`. │
|
||||
│ NumberOfDatabases │ 6 │ Total number of databases on the server. │
|
||||
│ MaxPartCountForPartition │ 6 │ Maximum number of parts per partition across all partitions of all tables of MergeTree family. Values larger than 300 indicates misconfiguration, overload, or massive data loading. │
|
||||
│ ReplicasSumMergesInQueue │ 0 │ Sum of merge operations in the queue (still to be applied) across Replicated tables. │
|
||||
│ ReplicasSumInsertsInQueue │ 0 │ Sum of INSERT operations in the queue (still to be replicated) across Replicated tables. │
|
||||
└─────────────────────────────────────────┴────────────┴────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
|
||||
┌─metric──────────────┬─────────────value─┬─key_values──────────────────────────────────────┐
|
||||
│ BlockReadBytes │ nan │ {'nvme0n1':67420160} │
|
||||
│ DiskTotal │ nan │ {'backups':34359738368,'default':5199475388416} │
|
||||
│ MemoryResident │ 970743808 │ {} │
|
||||
│ NetworkReceiveBytes │ nan │ {'docker0':0,'ens66':334554363} │
|
||||
│ OSUserTime │ 32.11807291562506 │ {} │
|
||||
└─────────────────────┴───────────────────┴─────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
A single entity of a key-value metric can be extracted with the map subscript:
|
||||
|
||||
```sql
|
||||
SELECT key_values['default'] AS default_disk_total
|
||||
FROM system.asynchronous_metrics
|
||||
WHERE metric = 'DiskTotal'
|
||||
```
|
||||
|
||||
## Metric descriptions {#metric-descriptions}
|
||||
|
||||
The descriptions below are generated from the C++ source by `utils/generate-async-metrics-docs`. The single source of truth is the string literal next to each metric registration in `src/Common/AsynchronousMetrics.cpp`, `src/Interpreters/ServerAsynchronousMetrics.cpp`, and `src/Coordination/KeeperAsynchronousMetrics.cpp`. Metric names that include a variable suffix (per-disk, per-CPU, per-interface, ...) are shown with a `*name*` placeholder; the running server reports them with the concrete suffix substituted in.
|
||||
The descriptions below are generated from the C++ source by `utils/generate-async-metrics-docs`. The single source of truth is the string literal next to each metric registration in `src/Common/AsynchronousMetrics.cpp`, `src/Interpreters/ServerAsynchronousMetrics.cpp`, and `src/Coordination/KeeperAsynchronousMetrics.cpp`. Metric names that include a variable part (currently only the HTTP connection pool group) are shown with a `*name*` placeholder; the running server reports them with the concrete name substituted in.
|
||||
|
||||
{/*AUTOGENERATED_METRICS_START*/}
|
||||
|
||||
### *name*DeadBlobsQueueEstimate {#namedeadblobsqueueestimate}
|
||||
|
||||
Estimated number of blobs enqueued for removal from the disk object storage (the blob manager dead queue). Disks without blob replication report 0.
|
||||
|
||||
### *name*MissingBlobsQueueEstimate {#namemissingblobsqueueestimate}
|
||||
|
||||
Estimated number of blobs awaiting replication to other locations of the disk (the blob manager missing queue). Disks without blob replication report 0.
|
||||
|
||||
### AsynchronousHeavyMetricsCalculationTimeSpent {#asynchronousheavymetricscalculationtimespent}
|
||||
|
||||
Time in seconds spent for calculation of asynchronous heavy (tables related) metrics (this is the overhead of asynchronous metrics).
|
||||
@@ -74,77 +75,77 @@ Time in seconds spent for calculation of asynchronous metrics (this is the overh
|
||||
|
||||
Metrics update interval
|
||||
|
||||
### AsyncLogging*metric_first*QueueSize {#asyncloggingmetric_firstqueuesize}
|
||||
### AsyncLoggingQueueSize {#asyncloggingqueuesize}
|
||||
|
||||
Number of async messages queued pending for logging in this channel
|
||||
Number of async messages queued pending for logging, keyed by the logging channel name.
|
||||
|
||||
### BlockActiveTime_*name* {#blockactivetime_name}
|
||||
### BlockActiveTime {#blockactivetime}
|
||||
|
||||
Time in seconds the block device had the IO requests queued. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. Source: `/sys/block`. See https://www.kernel.org/doc/Documentation/block/stat.txt
|
||||
Time in seconds every block device had the IO requests queued. The keys are the names of the block devices. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. Source: `/sys/block`. See https://www.kernel.org/doc/Documentation/block/stat.txt
|
||||
|
||||
### BlockActiveTimePerOp_*name* {#blockactivetimeperop_name}
|
||||
### BlockActiveTimePerOp {#blockactivetimeperop}
|
||||
|
||||
Similar to the `BlockActiveTime` metrics, but the value is divided to the number of IO operations to count the per-operation time.
|
||||
Similar to the `BlockActiveTime` metric, but the value is divided to the number of IO operations to count the per-operation time. The keys are the names of the block devices. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. Source: `/sys/block`. See https://www.kernel.org/doc/Documentation/block/stat.txt
|
||||
|
||||
### BlockDiscardBytes_*name* {#blockdiscardbytes_name}
|
||||
### BlockDiscardBytes {#blockdiscardbytes}
|
||||
|
||||
Number of discarded bytes on the block device. These operations are relevant for SSD. Discard operations are not used by ClickHouse, but can be used by other processes on the system. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. Source: `/sys/block`. See https://www.kernel.org/doc/Documentation/block/stat.txt
|
||||
Number of discarded bytes on every block device. These operations are relevant for SSD. Discard operations are not used by ClickHouse, but can be used by other processes on the system. The keys are the names of the block devices. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. Source: `/sys/block`. See https://www.kernel.org/doc/Documentation/block/stat.txt
|
||||
|
||||
### BlockDiscardMerges_*name* {#blockdiscardmerges_name}
|
||||
### BlockDiscardMerges {#blockdiscardmerges}
|
||||
|
||||
Number of discard operations requested from the block device and merged together by the OS IO scheduler. These operations are relevant for SSD. Discard operations are not used by ClickHouse, but can be used by other processes on the system. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. Source: `/sys/block`. See https://www.kernel.org/doc/Documentation/block/stat.txt
|
||||
Number of discard operations requested from every block device and merged together by the OS IO scheduler. These operations are relevant for SSD. Discard operations are not used by ClickHouse, but can be used by other processes on the system. The keys are the names of the block devices. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. Source: `/sys/block`. See https://www.kernel.org/doc/Documentation/block/stat.txt
|
||||
|
||||
### BlockDiscardOps_*name* {#blockdiscardops_name}
|
||||
### BlockDiscardOps {#blockdiscardops}
|
||||
|
||||
Number of discard operations requested from the block device. These operations are relevant for SSD. Discard operations are not used by ClickHouse, but can be used by other processes on the system. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. Source: `/sys/block`. See https://www.kernel.org/doc/Documentation/block/stat.txt
|
||||
Number of discard operations requested from every block device. These operations are relevant for SSD. Discard operations are not used by ClickHouse, but can be used by other processes on the system. The keys are the names of the block devices. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. Source: `/sys/block`. See https://www.kernel.org/doc/Documentation/block/stat.txt
|
||||
|
||||
### BlockDiscardTime_*name* {#blockdiscardtime_name}
|
||||
### BlockDiscardTime {#blockdiscardtime}
|
||||
|
||||
Time in seconds spend in discard operations requested from the block device, summed across all the operations. These operations are relevant for SSD. Discard operations are not used by ClickHouse, but can be used by other processes on the system. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. Source: `/sys/block`. See https://www.kernel.org/doc/Documentation/block/stat.txt
|
||||
Time in seconds spend in discard operations requested from every block device, summed across all the operations. These operations are relevant for SSD. Discard operations are not used by ClickHouse, but can be used by other processes on the system. The keys are the names of the block devices. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. Source: `/sys/block`. See https://www.kernel.org/doc/Documentation/block/stat.txt
|
||||
|
||||
### BlockInFlightOps_*name* {#blockinflightops_name}
|
||||
### BlockInFlightOps {#blockinflightops}
|
||||
|
||||
This value counts the number of I/O requests that have been issued to the device driver but have not yet completed. It does not include IO requests that are in the queue but not yet issued to the device driver. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. Source: `/sys/block`. See https://www.kernel.org/doc/Documentation/block/stat.txt
|
||||
This value counts the number of I/O requests that have been issued to the device driver but have not yet completed, for every block device. It does not include IO requests that are in the queue but not yet issued to the device driver. The keys are the names of the block devices. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. Source: `/sys/block`. See https://www.kernel.org/doc/Documentation/block/stat.txt
|
||||
|
||||
### BlockQueueTime_*name* {#blockqueuetime_name}
|
||||
### BlockQueueTime {#blockqueuetime}
|
||||
|
||||
This value counts the number of milliseconds that IO requests have waited on this block device. If there are multiple IO requests waiting, this value will increase as the product of the number of milliseconds times the number of requests waiting. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. Source: `/sys/block`. See https://www.kernel.org/doc/Documentation/block/stat.txt
|
||||
This value counts the number of milliseconds that IO requests have waited on every block device. If there are multiple IO requests waiting, this value will increase as the product of the number of milliseconds times the number of requests waiting. The keys are the names of the block devices. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. Source: `/sys/block`. See https://www.kernel.org/doc/Documentation/block/stat.txt
|
||||
|
||||
### BlockQueueTimePerOp_*name* {#blockqueuetimeperop_name}
|
||||
### BlockQueueTimePerOp {#blockqueuetimeperop}
|
||||
|
||||
Similar to the `BlockQueueTime` metrics, but the value is divided to the number of IO operations to count the per-operation time.
|
||||
Similar to the `BlockQueueTime` metric, but the value is divided to the number of IO operations to count the per-operation time. The keys are the names of the block devices. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. Source: `/sys/block`. See https://www.kernel.org/doc/Documentation/block/stat.txt
|
||||
|
||||
### BlockReadBytes_*name* {#blockreadbytes_name}
|
||||
### BlockReadBytes {#blockreadbytes}
|
||||
|
||||
Number of bytes read from the block device. It can be lower than the number of bytes read from the filesystem due to the usage of the OS page cache, that saves IO. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. Source: `/sys/block`. See https://www.kernel.org/doc/Documentation/block/stat.txt
|
||||
Number of bytes read from every block device. It can be lower than the number of bytes read from the filesystem due to the usage of the OS page cache, that saves IO. The keys are the names of the block devices. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. Source: `/sys/block`. See https://www.kernel.org/doc/Documentation/block/stat.txt
|
||||
|
||||
### BlockReadMerges_*name* {#blockreadmerges_name}
|
||||
### BlockReadMerges {#blockreadmerges}
|
||||
|
||||
Number of read operations requested from the block device and merged together by the OS IO scheduler. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. Source: `/sys/block`. See https://www.kernel.org/doc/Documentation/block/stat.txt
|
||||
Number of read operations requested from every block device and merged together by the OS IO scheduler. The keys are the names of the block devices. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. Source: `/sys/block`. See https://www.kernel.org/doc/Documentation/block/stat.txt
|
||||
|
||||
### BlockReadOps_*name* {#blockreadops_name}
|
||||
### BlockReadOps {#blockreadops}
|
||||
|
||||
Number of read operations requested from the block device. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. Source: `/sys/block`. See https://www.kernel.org/doc/Documentation/block/stat.txt
|
||||
Number of read operations requested from every block device. The keys are the names of the block devices. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. Source: `/sys/block`. See https://www.kernel.org/doc/Documentation/block/stat.txt
|
||||
|
||||
### BlockReadTime_*name* {#blockreadtime_name}
|
||||
### BlockReadTime {#blockreadtime}
|
||||
|
||||
Time in seconds spend in read operations requested from the block device, summed across all the operations. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. Source: `/sys/block`. See https://www.kernel.org/doc/Documentation/block/stat.txt
|
||||
Time in seconds spend in read operations requested from every block device, summed across all the operations. The keys are the names of the block devices. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. Source: `/sys/block`. See https://www.kernel.org/doc/Documentation/block/stat.txt
|
||||
|
||||
### BlockWriteBytes_*name* {#blockwritebytes_name}
|
||||
### BlockWriteBytes {#blockwritebytes}
|
||||
|
||||
Number of bytes written to the block device. It can be lower than the number of bytes written to the filesystem due to the usage of the OS page cache, that saves IO. A write to the block device may happen later than the corresponding write to the filesystem due to write-through caching. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. Source: `/sys/block`. See https://www.kernel.org/doc/Documentation/block/stat.txt
|
||||
Number of bytes written to every block device. It can be lower than the number of bytes written to the filesystem due to the usage of the OS page cache, that saves IO. A write to the block device may happen later than the corresponding write to the filesystem due to write-through caching. The keys are the names of the block devices. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. Source: `/sys/block`. See https://www.kernel.org/doc/Documentation/block/stat.txt
|
||||
|
||||
### BlockWriteMerges_*name* {#blockwritemerges_name}
|
||||
### BlockWriteMerges {#blockwritemerges}
|
||||
|
||||
Number of write operations requested from the block device and merged together by the OS IO scheduler. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. Source: `/sys/block`. See https://www.kernel.org/doc/Documentation/block/stat.txt
|
||||
Number of write operations requested from every block device and merged together by the OS IO scheduler. The keys are the names of the block devices. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. Source: `/sys/block`. See https://www.kernel.org/doc/Documentation/block/stat.txt
|
||||
|
||||
### BlockWriteOps_*name* {#blockwriteops_name}
|
||||
### BlockWriteOps {#blockwriteops}
|
||||
|
||||
Number of write operations requested from the block device. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. Source: `/sys/block`. See https://www.kernel.org/doc/Documentation/block/stat.txt
|
||||
Number of write operations requested from every block device. The keys are the names of the block devices. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. Source: `/sys/block`. See https://www.kernel.org/doc/Documentation/block/stat.txt
|
||||
|
||||
### BlockWriteTime_*name* {#blockwritetime_name}
|
||||
### BlockWriteTime {#blockwritetime}
|
||||
|
||||
Time in seconds spend in write operations requested from the block device, summed across all the operations. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. Source: `/sys/block`. See https://www.kernel.org/doc/Documentation/block/stat.txt
|
||||
Time in seconds spend in write operations requested from every block device, summed across all the operations. The keys are the names of the block devices. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. Source: `/sys/block`. See https://www.kernel.org/doc/Documentation/block/stat.txt
|
||||
|
||||
### CGroupMaxCPU {#cgroupmaxcpu}
|
||||
|
||||
@@ -186,9 +187,13 @@ Configured maximum bytes of the compiled-expression cache. Available headroom =
|
||||
|
||||
Configured maximum entries of the compiled-expression cache. Available headroom = this minus `CompiledExpressionCacheCount` (CurrentMetric).
|
||||
|
||||
### CPUFrequencyMHz_*core_id* {#cpufrequencymhz_core_id}
|
||||
### CPUFrequencyMHz {#cpufrequencymhz}
|
||||
|
||||
The current frequency of the CPU, in MHz. Most of the modern CPUs adjust the frequency dynamically for power saving and Turbo Boosting.
|
||||
The current frequency of every CPU core, in MHz, keyed by the CPU core number. Most of the modern CPUs adjust the frequency dynamically for power saving and Turbo Boosting.
|
||||
|
||||
### DeadBlobsQueueEstimate {#deadblobsqueueestimate}
|
||||
|
||||
Estimated number of blobs enqueued for removal from the disk object storage (the blob manager dead queue), keyed by the disk name. Disks without blob replication report 0.
|
||||
|
||||
### DictionaryMaxUpdateDelay {#dictionarymaxupdatedelay}
|
||||
|
||||
@@ -198,45 +203,45 @@ The maximum delay (in seconds) of dictionary update
|
||||
|
||||
Number of errors since last successful loading in all dictionaries.
|
||||
|
||||
### DiskAvailable_*name* {#diskavailable_name}
|
||||
### DiskAvailable {#diskavailable}
|
||||
|
||||
Available bytes on the disk (virtual filesystem). Remote filesystems may not provide this information and can show a large value like 16 EiB.
|
||||
Available bytes on every disk (virtual filesystem), keyed by the disk name. Remote filesystems may not provide this information and can show a large value like 16 EiB.
|
||||
|
||||
### DiskGetObjectThrottlerAvailable_*name* {#diskgetobjectthrottleravailable_name}
|
||||
### DiskGetObjectThrottlerAvailable {#diskgetobjectthrottleravailable}
|
||||
|
||||
Number of GetObject requests that can be currently issued without hitting throttling limit on the disk (virtual filesystem). Local filesystems may not provide this information.
|
||||
Number of GetObject requests that can be currently issued without hitting throttling limit on every disk (virtual filesystem), keyed by the disk name. Local filesystems may not provide this information.
|
||||
|
||||
### DiskGetObjectThrottlerRPS_*name* {#diskgetobjectthrottlerrps_name}
|
||||
### DiskGetObjectThrottlerRPS {#diskgetobjectthrottlerrps}
|
||||
|
||||
GetObject Request throttling limit on the disk in requests per second (virtual filesystem). Local filesystems may not provide this information.
|
||||
GetObject Request throttling limit on every disk in requests per second (virtual filesystem), keyed by the disk name. Local filesystems may not provide this information.
|
||||
|
||||
### DiskPutObjectThrottlerAvailable_*name* {#diskputobjectthrottleravailable_name}
|
||||
### DiskPutObjectThrottlerAvailable {#diskputobjectthrottleravailable}
|
||||
|
||||
Number of PutObject requests that can be currently issued without hitting throttling limit on the disk (virtual filesystem). Local filesystems may not provide this information.
|
||||
Number of PutObject requests that can be currently issued without hitting throttling limit on every disk (virtual filesystem), keyed by the disk name. Local filesystems may not provide this information.
|
||||
|
||||
### DiskPutObjectThrottlerRPS_*name* {#diskputobjectthrottlerrps_name}
|
||||
### DiskPutObjectThrottlerRPS {#diskputobjectthrottlerrps}
|
||||
|
||||
PutObject Request throttling limit on the disk in requests per second (virtual filesystem). Local filesystems may not provide this information.
|
||||
PutObject Request throttling limit on every disk in requests per second (virtual filesystem), keyed by the disk name. Local filesystems may not provide this information.
|
||||
|
||||
### DiskTotal_*name* {#disktotal_name}
|
||||
### DiskTotal {#disktotal}
|
||||
|
||||
The total size in bytes of the disk (virtual filesystem). Remote filesystems may not provide this information and can show a large value like 16 EiB.
|
||||
The total size in bytes of every disk (virtual filesystem), keyed by the disk name. Remote filesystems may not provide this information and can show a large value like 16 EiB.
|
||||
|
||||
### DiskUnreserved_*name* {#diskunreserved_name}
|
||||
### DiskUnreserved {#diskunreserved}
|
||||
|
||||
Available bytes on the disk (virtual filesystem) without the reservations for merges, fetches, and moves. Remote filesystems may not provide this information and can show a large value like 16 EiB.
|
||||
Available bytes on every disk (virtual filesystem) without the reservations for merges, fetches, and moves, keyed by the disk name. Remote filesystems may not provide this information and can show a large value like 16 EiB.
|
||||
|
||||
### DiskUsed_*name* {#diskused_name}
|
||||
### DiskUsed {#diskused}
|
||||
|
||||
Used bytes on the disk (virtual filesystem). Remote filesystems do not always provide this information.
|
||||
Used bytes on every disk (virtual filesystem), keyed by the disk name. Remote filesystems do not always provide this information.
|
||||
|
||||
### EDAC*i*_Correctable {#edaci_correctable}
|
||||
### EDACCorrectable {#edaccorrectable}
|
||||
|
||||
The number of correctable ECC memory errors. A high number of this value indicates bad RAM which has to be immediately replaced, because in presence of a high number of corrected errors, a number of silent errors may happen as well, leading to data corruption. Source: `/sys/devices/system/edac/mc/`
|
||||
The number of correctable ECC memory errors, keyed by the memory controller number. A high number of this value indicates bad RAM which has to be immediately replaced, because in presence of a high number of corrected errors, a number of silent errors may happen as well, leading to data corruption. Source: `/sys/devices/system/edac/mc/`
|
||||
|
||||
### EDAC*i*_Uncorrectable {#edaci_uncorrectable}
|
||||
### EDACUncorrectable {#edacuncorrectable}
|
||||
|
||||
The number of uncorrectable ECC memory errors. A non-zero number of this value indicates bad RAM which has to be immediately replaced, because it indicates potential data corruption. Source: `/sys/devices/system/edac/mc/`
|
||||
The number of uncorrectable ECC memory errors, keyed by the memory controller number. A non-zero number of this value indicates bad RAM which has to be immediately replaced, because it indicates potential data corruption. Source: `/sys/devices/system/edac/mc/`
|
||||
|
||||
### ExecutableUserDefinedFunctionMemoryResidentBytes {#executableuserdefinedfunctionmemoryresidentbytes}
|
||||
|
||||
@@ -716,6 +721,10 @@ Approximate virtual size of pthread stacks, summed from the sizes of the task's
|
||||
|
||||
The size of the virtual address space allocated by the server process, in bytes. The size of the virtual address space is usually much greater than the physical memory consumption, and should not be used as an estimate for the memory consumption. The large values of this metric are totally normal, and makes only technical sense.
|
||||
|
||||
### MissingBlobsQueueEstimate {#missingblobsqueueestimate}
|
||||
|
||||
Estimated number of blobs awaiting replication to other locations of the disk (the blob manager missing queue), keyed by the disk name. Disks without blob replication report 0.
|
||||
|
||||
### MySQLRejectedConnections {#mysqlrejectedconnections}
|
||||
|
||||
Number of rejected connections for the MySQL compatibility protocol.
|
||||
@@ -724,37 +733,37 @@ Number of rejected connections for the MySQL compatibility protocol.
|
||||
|
||||
Number of threads in the server of the MySQL compatibility protocol.
|
||||
|
||||
### NetworkReceiveBytes_*interface_name* {#networkreceivebytes_interface_name}
|
||||
### NetworkReceiveBytes {#networkreceivebytes}
|
||||
|
||||
Number of bytes received via the network interface. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server.
|
||||
Number of bytes received via every network interface. The keys are the names of the network interfaces. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server.
|
||||
|
||||
### NetworkReceiveDrop_*interface_name* {#networkreceivedrop_interface_name}
|
||||
### NetworkReceiveDrop {#networkreceivedrop}
|
||||
|
||||
Number of bytes a packet was dropped while received via the network interface. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server.
|
||||
Number of bytes a packet was dropped while received via every network interface. The keys are the names of the network interfaces. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server.
|
||||
|
||||
### NetworkReceiveErrors_*interface_name* {#networkreceiveerrors_interface_name}
|
||||
### NetworkReceiveErrors {#networkreceiveerrors}
|
||||
|
||||
Number of times error happened receiving via the network interface. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server.
|
||||
Number of times error happened receiving via every network interface. The keys are the names of the network interfaces. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server.
|
||||
|
||||
### NetworkReceivePackets_*interface_name* {#networkreceivepackets_interface_name}
|
||||
### NetworkReceivePackets {#networkreceivepackets}
|
||||
|
||||
Number of network packets received via the network interface. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server.
|
||||
Number of network packets received via every network interface. The keys are the names of the network interfaces. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server.
|
||||
|
||||
### NetworkSendBytes_*interface_name* {#networksendbytes_interface_name}
|
||||
### NetworkSendBytes {#networksendbytes}
|
||||
|
||||
Number of bytes sent via the network interface. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server.
|
||||
Number of bytes sent via every network interface. The keys are the names of the network interfaces. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server.
|
||||
|
||||
### NetworkSendDrop_*interface_name* {#networksenddrop_interface_name}
|
||||
### NetworkSendDrop {#networksenddrop}
|
||||
|
||||
Number of times a packed was dropped while sending via the network interface. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server.
|
||||
Number of times a packed was dropped while sending via every network interface. The keys are the names of the network interfaces. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server.
|
||||
|
||||
### NetworkSendErrors_*interface_name* {#networksenderrors_interface_name}
|
||||
### NetworkSendErrors {#networksenderrors}
|
||||
|
||||
Number of times error (e.g. TCP retransmit) happened while sending via the network interface. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server.
|
||||
Number of times error (e.g. TCP retransmit) happened while sending via every network interface. The keys are the names of the network interfaces. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server.
|
||||
|
||||
### NetworkSendPackets_*interface_name* {#networksendpackets_interface_name}
|
||||
### NetworkSendPackets {#networksendpackets}
|
||||
|
||||
Number of network packets sent via the network interface. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server.
|
||||
Number of network packets sent via every network interface. The keys are the names of the network interfaces. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server.
|
||||
|
||||
### NetworkTCPReceiveQueue {#networktcpreceivequeue}
|
||||
|
||||
@@ -816,26 +825,38 @@ The number of context switches that the system underwent on the host machine. Th
|
||||
|
||||
Relative CPU deficit, calculated as: how many threads are waiting for CPU relative to the number of threads, using CPU. If it is greater than zero, the server would benefit from more CPU. If it is significantly greater than zero, the server could become unresponsive. The metric is accumulated between the updates of asynchronous metrics.
|
||||
|
||||
### OSGuestNiceTime*cpu_suffix* {#osguestnicetimecpu_suffix}
|
||||
### OSGuestNiceTime {#osguestnicetime}
|
||||
|
||||
The ratio of time spent running a virtual CPU for guest operating systems under the control of the Linux kernel, when a guest was set to a higher priority (See `man procfs`). This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. This metric is irrelevant for ClickHouse, but still exists for completeness. The value for a single CPU core will be in the interval [0..1]. The value for all CPU cores is calculated as a sum across them [0..num cores].
|
||||
|
||||
### OSGuestNiceTimeCPU {#osguestnicetimecpu}
|
||||
|
||||
The ratio of time every CPU core spent running a virtual CPU for guest operating systems with higher priority, same as `OSGuestNiceTime` but not summed across the cores. The keys are the CPU core numbers, and the value for every core is in the interval [0..1]. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server.
|
||||
|
||||
### OSGuestNiceTimeNormalized {#osguestnicetimenormalized}
|
||||
|
||||
The value is similar to `OSGuestNiceTime` but divided to the number of CPU cores to be measured in the [0..1] interval regardless of the number of cores. This allows you to average the values of this metric across multiple servers in a cluster even if the number of cores is non-uniform, and still get the average resource utilization metric. If specified, the Cgroup CPU quota divided by its period can be used instead of the actual number of CPU cores, and in that case the value of this metric may exceed 1 at some moments.
|
||||
|
||||
### OSGuestTime*cpu_suffix* {#osguesttimecpu_suffix}
|
||||
### OSGuestTime {#osguesttime}
|
||||
|
||||
The ratio of time spent running a virtual CPU for guest operating systems under the control of the Linux kernel (See `man procfs`). This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. This metric is irrelevant for ClickHouse, but still exists for completeness. The value for a single CPU core will be in the interval [0..1]. The value for all CPU cores is calculated as a sum across them [0..num cores].
|
||||
|
||||
### OSGuestTimeCPU {#osguesttimecpu}
|
||||
|
||||
The ratio of time every CPU core spent running a virtual CPU for guest operating systems, same as `OSGuestTime` but not summed across the cores. The keys are the CPU core numbers, and the value for every core is in the interval [0..1]. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server.
|
||||
|
||||
### OSGuestTimeNormalized {#osguesttimenormalized}
|
||||
|
||||
The value is similar to `OSGuestTime` but divided to the number of CPU cores to be measured in the [0..1] interval regardless of the number of cores. This allows you to average the values of this metric across multiple servers in a cluster even if the number of cores is non-uniform, and still get the average resource utilization metric. If specified, the Cgroup CPU quota divided by its period can be used instead of the actual number of CPU cores, and in that case the value of this metric may exceed 1 at some moments.
|
||||
|
||||
### OSIdleTime*cpu_suffix* {#osidletimecpu_suffix}
|
||||
### OSIdleTime {#osidletime}
|
||||
|
||||
The ratio of time the CPU core was idle (not even ready to run a process waiting for IO) from the OS kernel standpoint. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. This does not include the time when the CPU was under-utilized due to the reasons internal to the CPU (memory loads, pipeline stalls, branch mispredictions, running another SMT core). The value for a single CPU core will be in the interval [0..1]. The value for all CPU cores is calculated as a sum across them [0..num cores].
|
||||
|
||||
### OSIdleTimeCPU {#osidletimecpu}
|
||||
|
||||
The ratio of time every CPU core was idle, same as `OSIdleTime` but not summed across the cores. The keys are the CPU core numbers, and the value for every core is in the interval [0..1]. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server.
|
||||
|
||||
### OSIdleTimeNormalized {#osidletimenormalized}
|
||||
|
||||
The value is similar to `OSIdleTime` but divided to the number of CPU cores to be measured in the [0..1] interval regardless of the number of cores. This allows you to average the values of this metric across multiple servers in a cluster even if the number of cores is non-uniform, and still get the average resource utilization metric. If specified, the Cgroup CPU quota divided by its period can be used instead of the actual number of CPU cores, and in that case the value of this metric may exceed 1 at some moments.
|
||||
@@ -844,18 +865,26 @@ The value is similar to `OSIdleTime` but divided to the number of CPU cores to b
|
||||
|
||||
The number of interrupts on the host machine. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server.
|
||||
|
||||
### OSIOWaitTime*cpu_suffix* {#osiowaittimecpu_suffix}
|
||||
### OSIOWaitTime {#osiowaittime}
|
||||
|
||||
The ratio of time the CPU core was not running the code but when the OS kernel did not run any other process on this CPU as the processes were waiting for IO. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. The value for a single CPU core will be in the interval [0..1]. The value for all CPU cores is calculated as a sum across them [0..num cores].
|
||||
|
||||
### OSIOWaitTimeCPU {#osiowaittimecpu}
|
||||
|
||||
The ratio of time every CPU core was waiting for IO, same as `OSIOWaitTime` but not summed across the cores. The keys are the CPU core numbers, and the value for every core is in the interval [0..1]. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server.
|
||||
|
||||
### OSIOWaitTimeNormalized {#osiowaittimenormalized}
|
||||
|
||||
The value is similar to `OSIOWaitTime` but divided to the number of CPU cores to be measured in the [0..1] interval regardless of the number of cores. This allows you to average the values of this metric across multiple servers in a cluster even if the number of cores is non-uniform, and still get the average resource utilization metric. If specified, the Cgroup CPU quota divided by its period can be used instead of the actual number of CPU cores, and in that case the value of this metric may exceed 1 at some moments.
|
||||
|
||||
### OSIrqTime*cpu_suffix* {#osirqtimecpu_suffix}
|
||||
### OSIrqTime {#osirqtime}
|
||||
|
||||
The ratio of time spent for running hardware interrupt requests on the CPU. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. A high number of this metric may indicate hardware misconfiguration or a very high network load. The value for a single CPU core will be in the interval [0..1]. The value for all CPU cores is calculated as a sum across them [0..num cores].
|
||||
|
||||
### OSIrqTimeCPU {#osirqtimecpu}
|
||||
|
||||
The ratio of time spent for running hardware interrupt requests on every CPU core, same as `OSIrqTime` but not summed across the cores. The keys are the CPU core numbers, and the value for every core is in the interval [0..1]. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server.
|
||||
|
||||
### OSIrqTimeNormalized {#osirqtimenormalized}
|
||||
|
||||
The value is similar to `OSIrqTime` but divided to the number of CPU cores to be measured in the [0..1] interval regardless of the number of cores. This allows you to average the values of this metric across multiple servers in a cluster even if the number of cores is non-uniform, and still get the average resource utilization metric. If specified, the Cgroup CPU quota divided by its period can be used instead of the actual number of CPU cores, and in that case the value of this metric may exceed 1 at some moments.
|
||||
@@ -888,10 +917,14 @@ The amount of memory in swap that was also loaded in RAM. Swap should be disable
|
||||
|
||||
The total amount of memory on the host system, in bytes.
|
||||
|
||||
### OSNiceTime*cpu_suffix* {#osnicetimecpu_suffix}
|
||||
### OSNiceTime {#osnicetime}
|
||||
|
||||
The ratio of time the CPU core was running userspace code with higher priority. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. The value for a single CPU core will be in the interval [0..1]. The value for all CPU cores is calculated as a sum across them [0..num cores].
|
||||
|
||||
### OSNiceTimeCPU {#osnicetimecpu}
|
||||
|
||||
The ratio of time every CPU core was running userspace code with higher priority, same as `OSNiceTime` but not summed across the cores. The keys are the CPU core numbers, and the value for every core is in the interval [0..1]. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server.
|
||||
|
||||
### OSNiceTimeNormalized {#osnicetimenormalized}
|
||||
|
||||
The value is similar to `OSNiceTime` but divided to the number of CPU cores to be measured in the [0..1] interval regardless of the number of cores. This allows you to average the values of this metric across multiple servers in a cluster even if the number of cores is non-uniform, and still get the average resource utilization metric. If specified, the Cgroup CPU quota divided by its period can be used instead of the actual number of CPU cores, and in that case the value of this metric may exceed 1 at some moments.
|
||||
@@ -912,26 +945,38 @@ The number of processes created. This is a system-wide metric, it includes all t
|
||||
|
||||
The number of runnable (running or ready to run) threads by the operating system. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server.
|
||||
|
||||
### OSSoftIrqTime*cpu_suffix* {#ossoftirqtimecpu_suffix}
|
||||
### OSSoftIrqTime {#ossoftirqtime}
|
||||
|
||||
The ratio of time spent for running software interrupt requests on the CPU. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. A high number of this metric may indicate inefficient software running on the system. The value for a single CPU core will be in the interval [0..1]. The value for all CPU cores is calculated as a sum across them [0..num cores].
|
||||
|
||||
### OSSoftIrqTimeCPU {#ossoftirqtimecpu}
|
||||
|
||||
The ratio of time spent for running software interrupt requests on every CPU core, same as `OSSoftIrqTime` but not summed across the cores. The keys are the CPU core numbers, and the value for every core is in the interval [0..1]. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server.
|
||||
|
||||
### OSSoftIrqTimeNormalized {#ossoftirqtimenormalized}
|
||||
|
||||
The value is similar to `OSSoftIrqTime` but divided to the number of CPU cores to be measured in the [0..1] interval regardless of the number of cores. This allows you to average the values of this metric across multiple servers in a cluster even if the number of cores is non-uniform, and still get the average resource utilization metric. If specified, the Cgroup CPU quota divided by its period can be used instead of the actual number of CPU cores, and in that case the value of this metric may exceed 1 at some moments.
|
||||
|
||||
### OSStealTime*cpu_suffix* {#osstealtimecpu_suffix}
|
||||
### OSStealTime {#osstealtime}
|
||||
|
||||
The ratio of time spent in other operating systems by the CPU when running in a virtualized environment. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. Not every virtualized environments present this metric, and most of them don't. The value for a single CPU core will be in the interval [0..1]. The value for all CPU cores is calculated as a sum across them [0..num cores].
|
||||
|
||||
### OSStealTimeCPU {#osstealtimecpu}
|
||||
|
||||
The ratio of time every CPU core spent in other operating systems when running in a virtualized environment, same as `OSStealTime` but not summed across the cores. The keys are the CPU core numbers, and the value for every core is in the interval [0..1]. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server.
|
||||
|
||||
### OSStealTimeNormalized {#osstealtimenormalized}
|
||||
|
||||
The value is similar to `OSStealTime` but divided to the number of CPU cores to be measured in the [0..1] interval regardless of the number of cores. This allows you to average the values of this metric across multiple servers in a cluster even if the number of cores is non-uniform, and still get the average resource utilization metric. If specified, the Cgroup CPU quota divided by its period can be used instead of the actual number of CPU cores, and in that case the value of this metric may exceed 1 at some moments.
|
||||
|
||||
### OSSystemTime*cpu_suffix* {#ossystemtimecpu_suffix}
|
||||
### OSSystemTime {#ossystemtime}
|
||||
|
||||
The ratio of time the CPU core was running OS kernel (system) code. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. The value for a single CPU core will be in the interval [0..1]. The value for all CPU cores is calculated as a sum across them [0..num cores].
|
||||
|
||||
### OSSystemTimeCPU {#ossystemtimecpu}
|
||||
|
||||
The ratio of time every CPU core was running OS kernel (system) code, same as `OSSystemTime` but not summed across the cores. The keys are the CPU core numbers, and the value for every core is in the interval [0..1]. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server.
|
||||
|
||||
### OSSystemTimeNormalized {#ossystemtimenormalized}
|
||||
|
||||
The value is similar to `OSSystemTime` but divided to the number of CPU cores to be measured in the [0..1] interval regardless of the number of cores. This allows you to average the values of this metric across multiple servers in a cluster even if the number of cores is non-uniform, and still get the average resource utilization metric. If specified, the Cgroup CPU quota divided by its period can be used instead of the actual number of CPU cores, and in that case the value of this metric may exceed 1 at some moments.
|
||||
@@ -948,10 +993,14 @@ The total number of threads, as the OS kernel scheduler seeing it.
|
||||
|
||||
The uptime of the host server (the machine where ClickHouse is running), in seconds.
|
||||
|
||||
### OSUserTime*cpu_suffix* {#osusertimecpu_suffix}
|
||||
### OSUserTime {#osusertime}
|
||||
|
||||
The ratio of time the CPU core was running userspace code. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server. This includes also the time when the CPU was under-utilized due to the reasons internal to the CPU (memory loads, pipeline stalls, branch mispredictions, running another SMT core). The value for a single CPU core will be in the interval [0..1]. The value for all CPU cores is calculated as a sum across them [0..num cores].
|
||||
|
||||
### OSUserTimeCPU {#osusertimecpu}
|
||||
|
||||
The ratio of time every CPU core was running userspace code, same as `OSUserTime` but not summed across the cores. The keys are the CPU core numbers, and the value for every core is in the interval [0..1]. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server.
|
||||
|
||||
### OSUserTimeNormalized {#osusertimenormalized}
|
||||
|
||||
The value is similar to `OSUserTime` but divided to the number of CPU cores to be measured in the [0..1] interval regardless of the number of cores. This allows you to average the values of this metric across multiple servers in a cluster even if the number of cores is non-uniform, and still get the average resource utilization metric. If specified, the Cgroup CPU quota divided by its period can be used instead of the actual number of CPU cores, and in that case the value of this metric may exceed 1 at some moments.
|
||||
@@ -1048,17 +1097,9 @@ Number of threads in the server of the TCP protocol (with TLS).
|
||||
|
||||
Number of threads in the server of the TCP protocol (without TLS).
|
||||
|
||||
### Temperature*i* {#temperaturei}
|
||||
### Temperature {#temperature}
|
||||
|
||||
The temperature of the corresponding device in ℃. A sensor can return an unrealistic value. Source: `/sys/class/thermal`
|
||||
|
||||
### Temperature_*hwmon_name* {#temperature_hwmon_name}
|
||||
|
||||
The temperature reported by the corresponding hardware monitor in ℃. A sensor can return an unrealistic value. Source: `/sys/class/hwmon`
|
||||
|
||||
### Temperature_*hwmon_name*_*sensor_name* {#temperature_hwmon_name_sensor_name}
|
||||
|
||||
The temperature reported by the corresponding hardware monitor and the corresponding sensor in ℃. A sensor can return an unrealistic value. Source: `/sys/class/hwmon`
|
||||
The temperature reported by every sensor, in ℃. A sensor can return an unrealistic value. The keys are the thermal zone numbers (source: `/sys/class/thermal`) or the names of the hardware monitors and their sensors (source: `/sys/class/hwmon`).
|
||||
|
||||
### TotalBytesOfMergeTreeTables {#totalbytesofmergetreetables}
|
||||
|
||||
|
||||
@@ -122,7 +122,14 @@ void MetricsTransmitter::transmit(std::vector<ProfileEvents::Count> & prev_count
|
||||
{
|
||||
for (const auto & name_value : async_metrics_values)
|
||||
{
|
||||
key_vals.emplace_back(asynchronous_metrics_path_prefix + name_value.first, name_value.second.value);
|
||||
if (name_value.second.isMap())
|
||||
{
|
||||
/// A key-value metric is sent as one path component deeper: `<prefix>.<metric>.<key>`.
|
||||
for (const auto & [key, value] : name_value.second.key_values)
|
||||
key_vals.emplace_back(asynchronous_metrics_path_prefix + name_value.first + "." + key, value);
|
||||
}
|
||||
else
|
||||
key_vals.emplace_back(asynchronous_metrics_path_prefix + name_value.first, name_value.second.value);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
+290
-124
@@ -1,6 +1,7 @@
|
||||
#include <Core/ServerSettings.h>
|
||||
#include <IO/MMappedFileCache.h>
|
||||
#include <IO/ReadHelpers.h>
|
||||
#include <IO/WriteHelpers.h>
|
||||
#include <IO/UncompressedCache.h>
|
||||
#include <Interpreters/Context.h>
|
||||
#include <base/cgroupsv2.h>
|
||||
@@ -685,9 +686,9 @@ void AsynchronousMetrics::applyCgroupCPUMetricsUpdate(
|
||||
}
|
||||
|
||||
void AsynchronousMetrics::applyCPUMetricsUpdate(
|
||||
AsynchronousMetricValues & new_values, const std::string & cpu_suffix, const ProcStatValuesCPU & delta_values, double multiplier)
|
||||
AsynchronousMetricValues & new_values, const ProcStatValuesCPU & delta_values, double multiplier)
|
||||
{
|
||||
new_values["OSUserTime" + cpu_suffix]
|
||||
new_values["OSUserTime"]
|
||||
= {static_cast<double>(delta_values.user) * multiplier,
|
||||
"The ratio of time the CPU core was running userspace code. This is a system-wide metric, it includes all the processes on the "
|
||||
"host machine, not just clickhouse-server."
|
||||
@@ -695,19 +696,19 @@ void AsynchronousMetrics::applyCPUMetricsUpdate(
|
||||
"stalls, branch mispredictions, running another SMT core)."
|
||||
" The value for a single CPU core will be in the interval [0..1]. The value for all CPU cores is calculated as a sum across "
|
||||
"them [0..num cores]."};
|
||||
new_values["OSNiceTime" + cpu_suffix]
|
||||
new_values["OSNiceTime"]
|
||||
= {static_cast<double>(delta_values.nice) * multiplier,
|
||||
"The ratio of time the CPU core was running userspace code with higher priority. This is a system-wide metric, it includes all "
|
||||
"the processes on the host machine, not just clickhouse-server."
|
||||
" The value for a single CPU core will be in the interval [0..1]. The value for all CPU cores is calculated as a sum across "
|
||||
"them [0..num cores]."};
|
||||
new_values["OSSystemTime" + cpu_suffix]
|
||||
new_values["OSSystemTime"]
|
||||
= {static_cast<double>(delta_values.system) * multiplier,
|
||||
"The ratio of time the CPU core was running OS kernel (system) code. This is a system-wide metric, it includes all the "
|
||||
"processes on the host machine, not just clickhouse-server."
|
||||
" The value for a single CPU core will be in the interval [0..1]. The value for all CPU cores is calculated as a sum across "
|
||||
"them [0..num cores]."};
|
||||
new_values["OSIdleTime" + cpu_suffix]
|
||||
new_values["OSIdleTime"]
|
||||
= {static_cast<double>(delta_values.idle) * multiplier,
|
||||
"The ratio of time the CPU core was idle (not even ready to run a process waiting for IO) from the OS kernel standpoint. This "
|
||||
"is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server."
|
||||
@@ -715,42 +716,42 @@ void AsynchronousMetrics::applyCPUMetricsUpdate(
|
||||
"stalls, branch mispredictions, running another SMT core)."
|
||||
" The value for a single CPU core will be in the interval [0..1]. The value for all CPU cores is calculated as a sum across "
|
||||
"them [0..num cores]."};
|
||||
new_values["OSIOWaitTime" + cpu_suffix]
|
||||
new_values["OSIOWaitTime"]
|
||||
= {static_cast<double>(delta_values.iowait) * multiplier,
|
||||
"The ratio of time the CPU core was not running the code but when the OS kernel did not run any other process on this CPU as "
|
||||
"the processes were waiting for IO. This is a system-wide metric, it includes all the processes on the host machine, not just "
|
||||
"clickhouse-server."
|
||||
" The value for a single CPU core will be in the interval [0..1]. The value for all CPU cores is calculated as a sum across "
|
||||
"them [0..num cores]."};
|
||||
new_values["OSIrqTime" + cpu_suffix]
|
||||
new_values["OSIrqTime"]
|
||||
= {static_cast<double>(delta_values.irq) * multiplier,
|
||||
"The ratio of time spent for running hardware interrupt requests on the CPU. This is a system-wide metric, it includes all the "
|
||||
"processes on the host machine, not just clickhouse-server."
|
||||
" A high number of this metric may indicate hardware misconfiguration or a very high network load."
|
||||
" The value for a single CPU core will be in the interval [0..1]. The value for all CPU cores is calculated as a sum across "
|
||||
"them [0..num cores]."};
|
||||
new_values["OSSoftIrqTime" + cpu_suffix]
|
||||
new_values["OSSoftIrqTime"]
|
||||
= {static_cast<double>(delta_values.softirq) * multiplier,
|
||||
"The ratio of time spent for running software interrupt requests on the CPU. This is a system-wide metric, it includes all the "
|
||||
"processes on the host machine, not just clickhouse-server."
|
||||
" A high number of this metric may indicate inefficient software running on the system."
|
||||
" The value for a single CPU core will be in the interval [0..1]. The value for all CPU cores is calculated as a sum across "
|
||||
"them [0..num cores]."};
|
||||
new_values["OSStealTime" + cpu_suffix]
|
||||
new_values["OSStealTime"]
|
||||
= {static_cast<double>(delta_values.steal) * multiplier,
|
||||
"The ratio of time spent in other operating systems by the CPU when running in a virtualized environment. This is a system-wide "
|
||||
"metric, it includes all the processes on the host machine, not just clickhouse-server."
|
||||
" Not every virtualized environments present this metric, and most of them don't."
|
||||
" The value for a single CPU core will be in the interval [0..1]. The value for all CPU cores is calculated as a sum across "
|
||||
"them [0..num cores]."};
|
||||
new_values["OSGuestTime" + cpu_suffix]
|
||||
new_values["OSGuestTime"]
|
||||
= {static_cast<double>(delta_values.guest) * multiplier,
|
||||
"The ratio of time spent running a virtual CPU for guest operating systems under the control of the Linux kernel (See `man "
|
||||
"procfs`). This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server."
|
||||
" This metric is irrelevant for ClickHouse, but still exists for completeness."
|
||||
" The value for a single CPU core will be in the interval [0..1]. The value for all CPU cores is calculated as a sum across "
|
||||
"them [0..num cores]."};
|
||||
new_values["OSGuestNiceTime" + cpu_suffix]
|
||||
new_values["OSGuestNiceTime"]
|
||||
= {static_cast<double>(delta_values.guest_nice) * multiplier,
|
||||
"The ratio of time spent running a virtual CPU for guest operating systems under the control of the Linux kernel, when a guest "
|
||||
"was set to a higher priority (See `man procfs`). This is a system-wide metric, it includes all the processes on the host "
|
||||
@@ -760,6 +761,59 @@ void AsynchronousMetrics::applyCPUMetricsUpdate(
|
||||
"them [0..num cores]."};
|
||||
}
|
||||
|
||||
void AsynchronousMetrics::applyPerCPUMetricsUpdate(AsynchronousMetricValues & new_values, ProcStatPerCPUMaps && per_cpu_maps)
|
||||
{
|
||||
if (per_cpu_maps.user.empty())
|
||||
return;
|
||||
|
||||
#define PER_CPU_EXPLANATION \
|
||||
" The keys are the CPU core numbers, and the value for every core is in the interval [0..1]." \
|
||||
" This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server."
|
||||
|
||||
new_values["OSUserTimeCPU"]
|
||||
= {"cpu", std::move(per_cpu_maps.user),
|
||||
"The ratio of time every CPU core was running userspace code, same as `OSUserTime` but not summed across the cores."
|
||||
PER_CPU_EXPLANATION};
|
||||
new_values["OSNiceTimeCPU"]
|
||||
= {"cpu", std::move(per_cpu_maps.nice),
|
||||
"The ratio of time every CPU core was running userspace code with higher priority, same as `OSNiceTime` but not summed across the cores."
|
||||
PER_CPU_EXPLANATION};
|
||||
new_values["OSSystemTimeCPU"]
|
||||
= {"cpu", std::move(per_cpu_maps.system),
|
||||
"The ratio of time every CPU core was running OS kernel (system) code, same as `OSSystemTime` but not summed across the cores."
|
||||
PER_CPU_EXPLANATION};
|
||||
new_values["OSIdleTimeCPU"]
|
||||
= {"cpu", std::move(per_cpu_maps.idle),
|
||||
"The ratio of time every CPU core was idle, same as `OSIdleTime` but not summed across the cores."
|
||||
PER_CPU_EXPLANATION};
|
||||
new_values["OSIOWaitTimeCPU"]
|
||||
= {"cpu", std::move(per_cpu_maps.iowait),
|
||||
"The ratio of time every CPU core was waiting for IO, same as `OSIOWaitTime` but not summed across the cores."
|
||||
PER_CPU_EXPLANATION};
|
||||
new_values["OSIrqTimeCPU"]
|
||||
= {"cpu", std::move(per_cpu_maps.irq),
|
||||
"The ratio of time spent for running hardware interrupt requests on every CPU core, same as `OSIrqTime` but not summed across the cores."
|
||||
PER_CPU_EXPLANATION};
|
||||
new_values["OSSoftIrqTimeCPU"]
|
||||
= {"cpu", std::move(per_cpu_maps.softirq),
|
||||
"The ratio of time spent for running software interrupt requests on every CPU core, same as `OSSoftIrqTime` but not summed across the cores."
|
||||
PER_CPU_EXPLANATION};
|
||||
new_values["OSStealTimeCPU"]
|
||||
= {"cpu", std::move(per_cpu_maps.steal),
|
||||
"The ratio of time every CPU core spent in other operating systems when running in a virtualized environment, same as `OSStealTime` but not summed across the cores."
|
||||
PER_CPU_EXPLANATION};
|
||||
new_values["OSGuestTimeCPU"]
|
||||
= {"cpu", std::move(per_cpu_maps.guest),
|
||||
"The ratio of time every CPU core spent running a virtual CPU for guest operating systems, same as `OSGuestTime` but not summed across the cores."
|
||||
PER_CPU_EXPLANATION};
|
||||
new_values["OSGuestNiceTimeCPU"]
|
||||
= {"cpu", std::move(per_cpu_maps.guest_nice),
|
||||
"The ratio of time every CPU core spent running a virtual CPU for guest operating systems with higher priority, same as `OSGuestNiceTime` but not summed across the cores."
|
||||
PER_CPU_EXPLANATION};
|
||||
|
||||
#undef PER_CPU_EXPLANATION
|
||||
}
|
||||
|
||||
void AsynchronousMetrics::applyCgroupNormalizedCPUMetricsUpdate(
|
||||
AsynchronousMetricValues & new_values, double num_cpus_to_normalize, const ProcStatValuesCPU & delta_values_all_cpus, double multiplier)
|
||||
{
|
||||
@@ -1580,6 +1634,7 @@ void AsynchronousMetrics::update(TimePoint update_time, bool force_update)
|
||||
|
||||
ProcStatValuesOther current_other_values{};
|
||||
ProcStatValuesCPU delta_values_all_cpus{};
|
||||
ProcStatPerCPUMaps per_cpu_maps;
|
||||
|
||||
while (!proc_stat->eof())
|
||||
{
|
||||
@@ -1611,16 +1666,25 @@ void AsynchronousMetrics::update(TimePoint update_time, bool force_update)
|
||||
{
|
||||
ProcStatValuesCPU delta_values = current_values - prev_values;
|
||||
|
||||
String cpu_suffix;
|
||||
if (!cpu_num_str.empty())
|
||||
{
|
||||
cpu_suffix = "CPU" + cpu_num_str;
|
||||
++num_cpus;
|
||||
per_cpu_maps.user[cpu_num_str] = static_cast<double>(delta_values.user) * multiplier;
|
||||
per_cpu_maps.nice[cpu_num_str] = static_cast<double>(delta_values.nice) * multiplier;
|
||||
per_cpu_maps.system[cpu_num_str] = static_cast<double>(delta_values.system) * multiplier;
|
||||
per_cpu_maps.idle[cpu_num_str] = static_cast<double>(delta_values.idle) * multiplier;
|
||||
per_cpu_maps.iowait[cpu_num_str] = static_cast<double>(delta_values.iowait) * multiplier;
|
||||
per_cpu_maps.irq[cpu_num_str] = static_cast<double>(delta_values.irq) * multiplier;
|
||||
per_cpu_maps.softirq[cpu_num_str] = static_cast<double>(delta_values.softirq) * multiplier;
|
||||
per_cpu_maps.steal[cpu_num_str] = static_cast<double>(delta_values.steal) * multiplier;
|
||||
per_cpu_maps.guest[cpu_num_str] = static_cast<double>(delta_values.guest) * multiplier;
|
||||
per_cpu_maps.guest_nice[cpu_num_str] = static_cast<double>(delta_values.guest_nice) * multiplier;
|
||||
}
|
||||
else
|
||||
{
|
||||
delta_values_all_cpus = delta_values;
|
||||
|
||||
applyCPUMetricsUpdate(new_values, cpu_suffix, delta_values, multiplier);
|
||||
applyCPUMetricsUpdate(new_values, delta_values, multiplier);
|
||||
}
|
||||
}
|
||||
|
||||
prev_values = current_values;
|
||||
@@ -1664,6 +1728,8 @@ void AsynchronousMetrics::update(TimePoint update_time, bool force_update)
|
||||
|
||||
if (!first_run)
|
||||
{
|
||||
applyPerCPUMetricsUpdate(new_values, std::move(per_cpu_maps));
|
||||
|
||||
ProcStatValuesOther delta_values = current_other_values - proc_stat_values_other;
|
||||
|
||||
new_values["OSInterrupts"] = { delta_values.interrupts, "The number of interrupts on the host machine. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server." };
|
||||
@@ -1828,6 +1894,7 @@ void AsynchronousMetrics::update(TimePoint update_time, bool force_update)
|
||||
// They contain tabs and are interspersed with other info.
|
||||
|
||||
int core_id = 0;
|
||||
AsynchronousMetricKeyValues cpu_frequencies;
|
||||
while (!cpuinfo->eof())
|
||||
{
|
||||
std::string s;
|
||||
@@ -1854,10 +1921,14 @@ void AsynchronousMetrics::update(TimePoint update_time, bool force_update)
|
||||
if (auto colon = s.find_first_of(':'))
|
||||
{
|
||||
auto mhz = std::stod(s.substr(colon + 2));
|
||||
new_values[fmt::format("CPUFrequencyMHz_{}", core_id)] = { mhz, "The current frequency of the CPU, in MHz. Most of the modern CPUs adjust the frequency dynamically for power saving and Turbo Boosting." };
|
||||
cpu_frequencies[toString(core_id)] = mhz;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (!cpu_frequencies.empty())
|
||||
new_values["CPUFrequencyMHz"] = { "cpu", std::move(cpu_frequencies),
|
||||
"The current frequency of every CPU core, in MHz, keyed by the CPU core number. Most of the modern CPUs adjust the frequency dynamically for power saving and Turbo Boosting." };
|
||||
}
|
||||
catch (...)
|
||||
{
|
||||
@@ -1931,6 +2002,24 @@ void AsynchronousMetrics::update(TimePoint update_time, bool force_update)
|
||||
|
||||
try
|
||||
{
|
||||
AsynchronousMetricKeyValues block_read_ops;
|
||||
AsynchronousMetricKeyValues block_write_ops;
|
||||
AsynchronousMetricKeyValues block_discard_ops;
|
||||
AsynchronousMetricKeyValues block_read_merges;
|
||||
AsynchronousMetricKeyValues block_write_merges;
|
||||
AsynchronousMetricKeyValues block_discard_merges;
|
||||
AsynchronousMetricKeyValues block_read_bytes;
|
||||
AsynchronousMetricKeyValues block_write_bytes;
|
||||
AsynchronousMetricKeyValues block_discard_bytes;
|
||||
AsynchronousMetricKeyValues block_read_time;
|
||||
AsynchronousMetricKeyValues block_write_time;
|
||||
AsynchronousMetricKeyValues block_discard_time;
|
||||
AsynchronousMetricKeyValues block_in_flight_ops;
|
||||
AsynchronousMetricKeyValues block_active_time;
|
||||
AsynchronousMetricKeyValues block_queue_time;
|
||||
AsynchronousMetricKeyValues block_active_time_per_op;
|
||||
AsynchronousMetricKeyValues block_queue_time_per_op;
|
||||
|
||||
for (auto & [name, device] : block_devs)
|
||||
{
|
||||
device->rewind();
|
||||
@@ -1961,82 +2050,114 @@ void AsynchronousMetrics::update(TimePoint update_time, bool force_update)
|
||||
/// Always in milliseconds according to the docs.
|
||||
static constexpr double time_multiplier = 1e-3;
|
||||
|
||||
#define BLOCK_DEVICE_EXPLANATION \
|
||||
" This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server." \
|
||||
" Source: `/sys/block`. See https://www.kernel.org/doc/Documentation/block/stat.txt"
|
||||
block_read_ops[name] = static_cast<double>(delta_values.read_ios);
|
||||
block_write_ops[name] = static_cast<double>(delta_values.write_ios);
|
||||
block_discard_ops[name] = static_cast<double>(delta_values.discard_ops);
|
||||
|
||||
new_values["BlockReadOps_" + name] = { delta_values.read_ios,
|
||||
"Number of read operations requested from the block device."
|
||||
BLOCK_DEVICE_EXPLANATION };
|
||||
new_values["BlockWriteOps_" + name] = { delta_values.write_ios,
|
||||
"Number of write operations requested from the block device."
|
||||
BLOCK_DEVICE_EXPLANATION };
|
||||
new_values["BlockDiscardOps_" + name] = { delta_values.discard_ops,
|
||||
"Number of discard operations requested from the block device. These operations are relevant for SSD."
|
||||
" Discard operations are not used by ClickHouse, but can be used by other processes on the system."
|
||||
BLOCK_DEVICE_EXPLANATION };
|
||||
block_read_merges[name] = static_cast<double>(delta_values.read_merges);
|
||||
block_write_merges[name] = static_cast<double>(delta_values.write_merges);
|
||||
block_discard_merges[name] = static_cast<double>(delta_values.discard_merges);
|
||||
|
||||
new_values["BlockReadMerges_" + name] = { delta_values.read_merges,
|
||||
"Number of read operations requested from the block device and merged together by the OS IO scheduler."
|
||||
BLOCK_DEVICE_EXPLANATION };
|
||||
new_values["BlockWriteMerges_" + name] = { delta_values.write_merges,
|
||||
"Number of write operations requested from the block device and merged together by the OS IO scheduler."
|
||||
BLOCK_DEVICE_EXPLANATION };
|
||||
new_values["BlockDiscardMerges_" + name] = { delta_values.discard_merges,
|
||||
"Number of discard operations requested from the block device and merged together by the OS IO scheduler."
|
||||
" These operations are relevant for SSD. Discard operations are not used by ClickHouse, but can be used by other processes on the system."
|
||||
BLOCK_DEVICE_EXPLANATION };
|
||||
block_read_bytes[name] = static_cast<double>(delta_values.read_sectors * sector_size);
|
||||
block_write_bytes[name] = static_cast<double>(delta_values.write_sectors * sector_size);
|
||||
block_discard_bytes[name] = static_cast<double>(delta_values.discard_sectors * sector_size);
|
||||
|
||||
new_values["BlockReadBytes_" + name] = { delta_values.read_sectors * sector_size,
|
||||
"Number of bytes read from the block device."
|
||||
" It can be lower than the number of bytes read from the filesystem due to the usage of the OS page cache, that saves IO."
|
||||
BLOCK_DEVICE_EXPLANATION };
|
||||
new_values["BlockWriteBytes_" + name] = { delta_values.write_sectors * sector_size,
|
||||
"Number of bytes written to the block device."
|
||||
" It can be lower than the number of bytes written to the filesystem due to the usage of the OS page cache, that saves IO."
|
||||
" A write to the block device may happen later than the corresponding write to the filesystem due to write-through caching."
|
||||
BLOCK_DEVICE_EXPLANATION };
|
||||
new_values["BlockDiscardBytes_" + name] = { delta_values.discard_sectors * sector_size,
|
||||
"Number of discarded bytes on the block device."
|
||||
" These operations are relevant for SSD. Discard operations are not used by ClickHouse, but can be used by other processes on the system."
|
||||
BLOCK_DEVICE_EXPLANATION };
|
||||
block_read_time[name] = static_cast<double>(delta_values.read_ticks) * time_multiplier;
|
||||
block_write_time[name] = static_cast<double>(delta_values.write_ticks) * time_multiplier;
|
||||
block_discard_time[name] = static_cast<double>(delta_values.discard_ticks) * time_multiplier;
|
||||
|
||||
new_values["BlockReadTime_" + name] = { static_cast<double>(delta_values.read_ticks) * time_multiplier,
|
||||
"Time in seconds spend in read operations requested from the block device, summed across all the operations."
|
||||
BLOCK_DEVICE_EXPLANATION };
|
||||
new_values["BlockWriteTime_" + name] = { static_cast<double>(delta_values.write_ticks) * time_multiplier,
|
||||
"Time in seconds spend in write operations requested from the block device, summed across all the operations."
|
||||
BLOCK_DEVICE_EXPLANATION };
|
||||
new_values["BlockDiscardTime_" + name] = { static_cast<double>(delta_values.discard_ticks) * time_multiplier,
|
||||
"Time in seconds spend in discard operations requested from the block device, summed across all the operations."
|
||||
" These operations are relevant for SSD. Discard operations are not used by ClickHouse, but can be used by other processes on the system."
|
||||
BLOCK_DEVICE_EXPLANATION };
|
||||
|
||||
new_values["BlockInFlightOps_" + name] = { delta_values.in_flight_ios,
|
||||
"This value counts the number of I/O requests that have been issued to"
|
||||
" the device driver but have not yet completed. It does not include IO"
|
||||
" requests that are in the queue but not yet issued to the device driver."
|
||||
BLOCK_DEVICE_EXPLANATION };
|
||||
new_values["BlockActiveTime_" + name] = { static_cast<double>(delta_values.io_ticks) * time_multiplier,
|
||||
"Time in seconds the block device had the IO requests queued."
|
||||
BLOCK_DEVICE_EXPLANATION };
|
||||
new_values["BlockQueueTime_" + name] = { static_cast<double>(delta_values.time_in_queue) * time_multiplier,
|
||||
"This value counts the number of milliseconds that IO requests have waited"
|
||||
" on this block device. If there are multiple IO requests waiting, this"
|
||||
" value will increase as the product of the number of milliseconds times the"
|
||||
" number of requests waiting."
|
||||
BLOCK_DEVICE_EXPLANATION };
|
||||
block_in_flight_ops[name] = static_cast<double>(delta_values.in_flight_ios);
|
||||
block_active_time[name] = static_cast<double>(delta_values.io_ticks) * time_multiplier;
|
||||
block_queue_time[name] = static_cast<double>(delta_values.time_in_queue) * time_multiplier;
|
||||
|
||||
if (delta_values.in_flight_ios)
|
||||
{
|
||||
/// TODO Check if these values are meaningful.
|
||||
|
||||
new_values["BlockActiveTimePerOp_" + name] = { static_cast<double>(delta_values.io_ticks) * time_multiplier / static_cast<double>(delta_values.in_flight_ios),
|
||||
"Similar to the `BlockActiveTime` metrics, but the value is divided to the number of IO operations to count the per-operation time." };
|
||||
new_values["BlockQueueTimePerOp_" + name] = { static_cast<double>(delta_values.time_in_queue) * time_multiplier / static_cast<double>(delta_values.in_flight_ios),
|
||||
"Similar to the `BlockQueueTime` metrics, but the value is divided to the number of IO operations to count the per-operation time." };
|
||||
block_active_time_per_op[name] = static_cast<double>(delta_values.io_ticks) * time_multiplier / static_cast<double>(delta_values.in_flight_ios);
|
||||
block_queue_time_per_op[name] = static_cast<double>(delta_values.time_in_queue) * time_multiplier / static_cast<double>(delta_values.in_flight_ios);
|
||||
}
|
||||
}
|
||||
|
||||
#define BLOCK_DEVICE_EXPLANATION \
|
||||
" The keys are the names of the block devices." \
|
||||
" This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server." \
|
||||
" Source: `/sys/block`. See https://www.kernel.org/doc/Documentation/block/stat.txt"
|
||||
|
||||
if (!block_read_ops.empty())
|
||||
{
|
||||
new_values["BlockReadOps"] = { "device", std::move(block_read_ops),
|
||||
"Number of read operations requested from every block device."
|
||||
BLOCK_DEVICE_EXPLANATION };
|
||||
new_values["BlockWriteOps"] = { "device", std::move(block_write_ops),
|
||||
"Number of write operations requested from every block device."
|
||||
BLOCK_DEVICE_EXPLANATION };
|
||||
new_values["BlockDiscardOps"] = { "device", std::move(block_discard_ops),
|
||||
"Number of discard operations requested from every block device. These operations are relevant for SSD."
|
||||
" Discard operations are not used by ClickHouse, but can be used by other processes on the system."
|
||||
BLOCK_DEVICE_EXPLANATION };
|
||||
|
||||
new_values["BlockReadMerges"] = { "device", std::move(block_read_merges),
|
||||
"Number of read operations requested from every block device and merged together by the OS IO scheduler."
|
||||
BLOCK_DEVICE_EXPLANATION };
|
||||
new_values["BlockWriteMerges"] = { "device", std::move(block_write_merges),
|
||||
"Number of write operations requested from every block device and merged together by the OS IO scheduler."
|
||||
BLOCK_DEVICE_EXPLANATION };
|
||||
new_values["BlockDiscardMerges"] = { "device", std::move(block_discard_merges),
|
||||
"Number of discard operations requested from every block device and merged together by the OS IO scheduler."
|
||||
" These operations are relevant for SSD. Discard operations are not used by ClickHouse, but can be used by other processes on the system."
|
||||
BLOCK_DEVICE_EXPLANATION };
|
||||
|
||||
new_values["BlockReadBytes"] = { "device", std::move(block_read_bytes),
|
||||
"Number of bytes read from every block device."
|
||||
" It can be lower than the number of bytes read from the filesystem due to the usage of the OS page cache, that saves IO."
|
||||
BLOCK_DEVICE_EXPLANATION };
|
||||
new_values["BlockWriteBytes"] = { "device", std::move(block_write_bytes),
|
||||
"Number of bytes written to every block device."
|
||||
" It can be lower than the number of bytes written to the filesystem due to the usage of the OS page cache, that saves IO."
|
||||
" A write to the block device may happen later than the corresponding write to the filesystem due to write-through caching."
|
||||
BLOCK_DEVICE_EXPLANATION };
|
||||
new_values["BlockDiscardBytes"] = { "device", std::move(block_discard_bytes),
|
||||
"Number of discarded bytes on every block device."
|
||||
" These operations are relevant for SSD. Discard operations are not used by ClickHouse, but can be used by other processes on the system."
|
||||
BLOCK_DEVICE_EXPLANATION };
|
||||
|
||||
new_values["BlockReadTime"] = { "device", std::move(block_read_time),
|
||||
"Time in seconds spend in read operations requested from every block device, summed across all the operations."
|
||||
BLOCK_DEVICE_EXPLANATION };
|
||||
new_values["BlockWriteTime"] = { "device", std::move(block_write_time),
|
||||
"Time in seconds spend in write operations requested from every block device, summed across all the operations."
|
||||
BLOCK_DEVICE_EXPLANATION };
|
||||
new_values["BlockDiscardTime"] = { "device", std::move(block_discard_time),
|
||||
"Time in seconds spend in discard operations requested from every block device, summed across all the operations."
|
||||
" These operations are relevant for SSD. Discard operations are not used by ClickHouse, but can be used by other processes on the system."
|
||||
BLOCK_DEVICE_EXPLANATION };
|
||||
|
||||
new_values["BlockInFlightOps"] = { "device", std::move(block_in_flight_ops),
|
||||
"This value counts the number of I/O requests that have been issued to"
|
||||
" the device driver but have not yet completed, for every block device. It does not include IO"
|
||||
" requests that are in the queue but not yet issued to the device driver."
|
||||
BLOCK_DEVICE_EXPLANATION };
|
||||
new_values["BlockActiveTime"] = { "device", std::move(block_active_time),
|
||||
"Time in seconds every block device had the IO requests queued."
|
||||
BLOCK_DEVICE_EXPLANATION };
|
||||
new_values["BlockQueueTime"] = { "device", std::move(block_queue_time),
|
||||
"This value counts the number of milliseconds that IO requests have waited"
|
||||
" on every block device. If there are multiple IO requests waiting, this"
|
||||
" value will increase as the product of the number of milliseconds times the"
|
||||
" number of requests waiting."
|
||||
BLOCK_DEVICE_EXPLANATION };
|
||||
}
|
||||
|
||||
if (!block_active_time_per_op.empty())
|
||||
{
|
||||
new_values["BlockActiveTimePerOp"] = { "device", std::move(block_active_time_per_op),
|
||||
"Similar to the `BlockActiveTime` metric, but the value is divided to the number of IO operations to count the per-operation time."
|
||||
BLOCK_DEVICE_EXPLANATION };
|
||||
new_values["BlockQueueTimePerOp"] = { "device", std::move(block_queue_time_per_op),
|
||||
"Similar to the `BlockQueueTime` metric, but the value is divided to the number of IO operations to count the per-operation time."
|
||||
BLOCK_DEVICE_EXPLANATION };
|
||||
}
|
||||
}
|
||||
catch (...)
|
||||
{
|
||||
@@ -2067,6 +2188,15 @@ void AsynchronousMetrics::update(TimePoint update_time, bool force_update)
|
||||
skipToNextLineOrEOF(*net_dev);
|
||||
skipToNextLineOrEOF(*net_dev);
|
||||
|
||||
AsynchronousMetricKeyValues network_receive_bytes;
|
||||
AsynchronousMetricKeyValues network_receive_packets;
|
||||
AsynchronousMetricKeyValues network_receive_errors;
|
||||
AsynchronousMetricKeyValues network_receive_drop;
|
||||
AsynchronousMetricKeyValues network_send_bytes;
|
||||
AsynchronousMetricKeyValues network_send_packets;
|
||||
AsynchronousMetricKeyValues network_send_errors;
|
||||
AsynchronousMetricKeyValues network_send_drop;
|
||||
|
||||
while (!net_dev->eof())
|
||||
{
|
||||
skipWhitespaceIfAny(*net_dev, true);
|
||||
@@ -2122,33 +2252,50 @@ void AsynchronousMetrics::update(TimePoint update_time, bool force_update)
|
||||
|
||||
if (!first_run)
|
||||
{
|
||||
new_values["NetworkReceiveBytes_" + interface_name] = { delta_values.recv_bytes,
|
||||
" Number of bytes received via the network interface."
|
||||
" This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server." };
|
||||
new_values["NetworkReceivePackets_" + interface_name] = { delta_values.recv_packets,
|
||||
" Number of network packets received via the network interface."
|
||||
" This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server." };
|
||||
new_values["NetworkReceiveErrors_" + interface_name] = { delta_values.recv_errors,
|
||||
" Number of times error happened receiving via the network interface."
|
||||
" This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server." };
|
||||
new_values["NetworkReceiveDrop_" + interface_name] = { delta_values.recv_drop,
|
||||
" Number of bytes a packet was dropped while received via the network interface."
|
||||
" This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server." };
|
||||
network_receive_bytes[interface_name] = static_cast<double>(delta_values.recv_bytes);
|
||||
network_receive_packets[interface_name] = static_cast<double>(delta_values.recv_packets);
|
||||
network_receive_errors[interface_name] = static_cast<double>(delta_values.recv_errors);
|
||||
network_receive_drop[interface_name] = static_cast<double>(delta_values.recv_drop);
|
||||
|
||||
new_values["NetworkSendBytes_" + interface_name] = { delta_values.send_bytes,
|
||||
" Number of bytes sent via the network interface."
|
||||
" This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server." };
|
||||
new_values["NetworkSendPackets_" + interface_name] = { delta_values.send_packets,
|
||||
" Number of network packets sent via the network interface."
|
||||
" This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server." };
|
||||
new_values["NetworkSendErrors_" + interface_name] = { delta_values.send_errors,
|
||||
" Number of times error (e.g. TCP retransmit) happened while sending via the network interface."
|
||||
" This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server." };
|
||||
new_values["NetworkSendDrop_" + interface_name] = { delta_values.send_drop,
|
||||
" Number of times a packed was dropped while sending via the network interface."
|
||||
" This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server." };
|
||||
network_send_bytes[interface_name] = static_cast<double>(delta_values.send_bytes);
|
||||
network_send_packets[interface_name] = static_cast<double>(delta_values.send_packets);
|
||||
network_send_errors[interface_name] = static_cast<double>(delta_values.send_errors);
|
||||
network_send_drop[interface_name] = static_cast<double>(delta_values.send_drop);
|
||||
}
|
||||
}
|
||||
|
||||
#define NETWORK_INTERFACE_EXPLANATION \
|
||||
" The keys are the names of the network interfaces." \
|
||||
" This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server."
|
||||
|
||||
if (!network_receive_bytes.empty())
|
||||
{
|
||||
new_values["NetworkReceiveBytes"] = { "interface", std::move(network_receive_bytes),
|
||||
"Number of bytes received via every network interface."
|
||||
NETWORK_INTERFACE_EXPLANATION };
|
||||
new_values["NetworkReceivePackets"] = { "interface", std::move(network_receive_packets),
|
||||
"Number of network packets received via every network interface."
|
||||
NETWORK_INTERFACE_EXPLANATION };
|
||||
new_values["NetworkReceiveErrors"] = { "interface", std::move(network_receive_errors),
|
||||
"Number of times error happened receiving via every network interface."
|
||||
NETWORK_INTERFACE_EXPLANATION };
|
||||
new_values["NetworkReceiveDrop"] = { "interface", std::move(network_receive_drop),
|
||||
"Number of bytes a packet was dropped while received via every network interface."
|
||||
NETWORK_INTERFACE_EXPLANATION };
|
||||
|
||||
new_values["NetworkSendBytes"] = { "interface", std::move(network_send_bytes),
|
||||
"Number of bytes sent via every network interface."
|
||||
NETWORK_INTERFACE_EXPLANATION };
|
||||
new_values["NetworkSendPackets"] = { "interface", std::move(network_send_packets),
|
||||
"Number of network packets sent via every network interface."
|
||||
NETWORK_INTERFACE_EXPLANATION };
|
||||
new_values["NetworkSendErrors"] = { "interface", std::move(network_send_errors),
|
||||
"Number of times error (e.g. TCP retransmit) happened while sending via every network interface."
|
||||
NETWORK_INTERFACE_EXPLANATION };
|
||||
new_values["NetworkSendDrop"] = { "interface", std::move(network_send_drop),
|
||||
"Number of times a packed was dropped while sending via every network interface."
|
||||
NETWORK_INTERFACE_EXPLANATION };
|
||||
}
|
||||
}
|
||||
catch (...)
|
||||
{
|
||||
@@ -2386,6 +2533,8 @@ void AsynchronousMetrics::update(TimePoint update_time, bool force_update)
|
||||
}
|
||||
}
|
||||
|
||||
AsynchronousMetricKeyValues temperatures;
|
||||
|
||||
try
|
||||
{
|
||||
for (size_t i = 0, size = thermal.size(); i < size; ++i)
|
||||
@@ -2395,8 +2544,7 @@ void AsynchronousMetrics::update(TimePoint update_time, bool force_update)
|
||||
in.rewind();
|
||||
Int64 temperature = 0;
|
||||
readText(temperature, in);
|
||||
new_values[fmt::format("Temperature{}", i)] = { static_cast<double>(temperature) * 0.001,
|
||||
"The temperature of the corresponding device in ℃. A sensor can return an unrealistic value. Source: `/sys/class/thermal`" };
|
||||
temperatures[toString(i)] = static_cast<double>(temperature) * 0.001;
|
||||
}
|
||||
}
|
||||
catch (...)
|
||||
@@ -2435,11 +2583,9 @@ void AsynchronousMetrics::update(TimePoint update_time, bool force_update)
|
||||
}
|
||||
|
||||
if (sensor_name.empty())
|
||||
new_values[fmt::format("Temperature_{}", hwmon_name)] = { static_cast<double>(temperature) * 0.001,
|
||||
"The temperature reported by the corresponding hardware monitor in ℃. A sensor can return an unrealistic value. Source: `/sys/class/hwmon`" };
|
||||
temperatures[hwmon_name] = static_cast<double>(temperature) * 0.001;
|
||||
else
|
||||
new_values[fmt::format("Temperature_{}_{}", hwmon_name, sensor_name)] = { static_cast<double>(temperature) * 0.001,
|
||||
"The temperature reported by the corresponding hardware monitor and the corresponding sensor in ℃. A sensor can return an unrealistic value. Source: `/sys/class/hwmon`" };
|
||||
temperatures[fmt::format("{}_{}", hwmon_name, sensor_name)] = static_cast<double>(temperature) * 0.001;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -2462,8 +2608,17 @@ void AsynchronousMetrics::update(TimePoint update_time, bool force_update)
|
||||
}
|
||||
}
|
||||
|
||||
if (!temperatures.empty())
|
||||
new_values["Temperature"] = { "sensor", std::move(temperatures),
|
||||
"The temperature reported by every sensor, in ℃. A sensor can return an unrealistic value."
|
||||
" The keys are the thermal zone numbers (source: `/sys/class/thermal`)"
|
||||
" or the names of the hardware monitors and their sensors (source: `/sys/class/hwmon`)." };
|
||||
|
||||
try
|
||||
{
|
||||
AsynchronousMetricKeyValues edac_correctable_errors;
|
||||
AsynchronousMetricKeyValues edac_uncorrectable_errors;
|
||||
|
||||
for (size_t i = 0, size = edac.size(); i < size; ++i)
|
||||
{
|
||||
/// NOTE maybe we need to take difference with previous values.
|
||||
@@ -2475,11 +2630,7 @@ void AsynchronousMetrics::update(TimePoint update_time, bool force_update)
|
||||
in.rewind();
|
||||
uint64_t errors = 0;
|
||||
readText(errors, in);
|
||||
new_values[fmt::format("EDAC{}_Correctable", i)] = { errors,
|
||||
"The number of correctable ECC memory errors."
|
||||
" A high number of this value indicates bad RAM which has to be immediately replaced,"
|
||||
" because in presence of a high number of corrected errors, a number of silent errors may happen as well, leading to data corruption."
|
||||
" Source: `/sys/devices/system/edac/mc/`" };
|
||||
edac_correctable_errors[toString(i)] = static_cast<double>(errors);
|
||||
}
|
||||
|
||||
if (edac[i].second)
|
||||
@@ -2488,13 +2639,23 @@ void AsynchronousMetrics::update(TimePoint update_time, bool force_update)
|
||||
in.rewind();
|
||||
uint64_t errors = 0;
|
||||
readText(errors, in);
|
||||
new_values[fmt::format("EDAC{}_Uncorrectable", i)] = { errors,
|
||||
"The number of uncorrectable ECC memory errors."
|
||||
" A non-zero number of this value indicates bad RAM which has to be immediately replaced,"
|
||||
" because it indicates potential data corruption."
|
||||
" Source: `/sys/devices/system/edac/mc/`" };
|
||||
edac_uncorrectable_errors[toString(i)] = static_cast<double>(errors);
|
||||
}
|
||||
}
|
||||
|
||||
if (!edac_correctable_errors.empty())
|
||||
new_values["EDACCorrectable"] = { "mc", std::move(edac_correctable_errors),
|
||||
"The number of correctable ECC memory errors, keyed by the memory controller number."
|
||||
" A high number of this value indicates bad RAM which has to be immediately replaced,"
|
||||
" because in presence of a high number of corrected errors, a number of silent errors may happen as well, leading to data corruption."
|
||||
" Source: `/sys/devices/system/edac/mc/`" };
|
||||
|
||||
if (!edac_uncorrectable_errors.empty())
|
||||
new_values["EDACUncorrectable"] = { "mc", std::move(edac_uncorrectable_errors),
|
||||
"The number of uncorrectable ECC memory errors, keyed by the memory controller number."
|
||||
" A non-zero number of this value indicates bad RAM which has to be immediately replaced,"
|
||||
" because it indicates potential data corruption."
|
||||
" Source: `/sys/devices/system/edac/mc/`" };
|
||||
}
|
||||
catch (...)
|
||||
{
|
||||
@@ -2572,10 +2733,15 @@ void AsynchronousMetrics::update(TimePoint update_time, bool force_update)
|
||||
|
||||
new_values["OSCPUOverload"] = { ProfileEvents::global_counters.getCPUOverload(context->getServerSettings()[ServerSetting::os_cpu_busy_time_threshold], /*reset*/ true), "Relative CPU deficit, calculated as: how many threads are waiting for CPU relative to the number of threads, using CPU. If it is greater than zero, the server would benefit from more CPU. If it is significantly greater than zero, the server could become unresponsive. The metric is accumulated between the updates of asynchronous metrics." };
|
||||
|
||||
for (const auto & metric : BaseDaemon::instance().getAsynchronousMetricsFromAsyncLogs())
|
||||
{
|
||||
new_values[fmt::format("AsyncLogging{}QueueSize", metric.first)]
|
||||
= {static_cast<double>(metric.second), "Number of async messages queued pending for logging in this channel"};
|
||||
AsynchronousMetricKeyValues async_logging_queue_sizes;
|
||||
for (const auto & metric : BaseDaemon::instance().getAsynchronousMetricsFromAsyncLogs())
|
||||
async_logging_queue_sizes[metric.first] = static_cast<double>(metric.second);
|
||||
|
||||
if (!async_logging_queue_sizes.empty())
|
||||
new_values["AsyncLoggingQueueSize"]
|
||||
= {"channel", std::move(async_logging_queue_sizes),
|
||||
"Number of async messages queued pending for logging, keyed by the logging channel name."};
|
||||
}
|
||||
|
||||
/// Add more metrics as you wish.
|
||||
|
||||
@@ -9,6 +9,8 @@
|
||||
#include <IO/ReadBufferFromFile.h>
|
||||
|
||||
#include <condition_variable>
|
||||
#include <limits>
|
||||
#include <map>
|
||||
#include <source_location>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
@@ -26,9 +28,21 @@ namespace DB
|
||||
|
||||
class ReadBuffer;
|
||||
|
||||
/// Per-entity values of a "key-value" metric: one value per CPU core, block device, network interface, etc.
|
||||
/// Ordered by key for deterministic output.
|
||||
using AsynchronousMetricKeyValues = std::map<String, double>;
|
||||
|
||||
struct AsynchronousMetricValue
|
||||
{
|
||||
/// The value of a scalar metric. For key-value metrics it is NaN: an aggregate across entities
|
||||
/// (sum, average, ...) is not meaningful for every metric, so none is provided.
|
||||
double value = 0;
|
||||
/// The values of a key-value metric (exposed as `Map(String, Float64)` in `system.asynchronous_metrics`).
|
||||
/// Empty for scalar metrics.
|
||||
AsynchronousMetricKeyValues key_values;
|
||||
/// For key-value metrics: what the key means, e.g. "cpu", "device", "disk". It is used as the label name
|
||||
/// for the Prometheus endpoint. Non-null if and only if the metric is a key-value metric.
|
||||
const char * key_label = nullptr;
|
||||
const char * documentation = nullptr;
|
||||
/// The source file where this metric and its documentation are produced. Asynchronous metrics are defined across
|
||||
/// several files (`AsynchronousMetrics.cpp`, `ServerAsynchronousMetrics.cpp`, `KeeperAsynchronousMetrics.cpp`, ...),
|
||||
@@ -39,7 +53,13 @@ struct AsynchronousMetricValue
|
||||
template <typename T>
|
||||
AsynchronousMetricValue(T value_, const char * documentation_, std::source_location source_ = std::source_location::current())
|
||||
: value(static_cast<double>(value_)), documentation(documentation_), source(source_.file_name()) {}
|
||||
AsynchronousMetricValue(const char * key_label_, AsynchronousMetricKeyValues key_values_, const char * documentation_,
|
||||
std::source_location source_ = std::source_location::current())
|
||||
: value(std::numeric_limits<double>::quiet_NaN()), key_values(std::move(key_values_)), key_label(key_label_)
|
||||
, documentation(documentation_), source(source_.file_name()) {}
|
||||
AsynchronousMetricValue() = default; /// For std::unordered_map::operator[].
|
||||
|
||||
bool isMap() const { return key_label != nullptr; }
|
||||
};
|
||||
|
||||
using AsynchronousMetricValues = std::unordered_map<std::string, AsynchronousMetricValue>;
|
||||
@@ -198,6 +218,22 @@ private:
|
||||
ProcStatValuesCPU operator-(const ProcStatValuesCPU & other) const;
|
||||
};
|
||||
|
||||
/// Accumulators for the per-CPU-core breakdown of the `/proc/stat` metrics,
|
||||
/// published as key-value metrics (`OSUserTimeCPU` and friends) keyed by the CPU core number.
|
||||
struct ProcStatPerCPUMaps
|
||||
{
|
||||
AsynchronousMetricKeyValues user;
|
||||
AsynchronousMetricKeyValues nice;
|
||||
AsynchronousMetricKeyValues system;
|
||||
AsynchronousMetricKeyValues idle;
|
||||
AsynchronousMetricKeyValues iowait;
|
||||
AsynchronousMetricKeyValues irq;
|
||||
AsynchronousMetricKeyValues softirq;
|
||||
AsynchronousMetricKeyValues steal;
|
||||
AsynchronousMetricKeyValues guest;
|
||||
AsynchronousMetricKeyValues guest_nice;
|
||||
};
|
||||
|
||||
struct ProcStatValuesOther
|
||||
{
|
||||
uint64_t interrupts;
|
||||
@@ -273,7 +309,9 @@ private:
|
||||
double multiplier);
|
||||
|
||||
void applyCPUMetricsUpdate(
|
||||
AsynchronousMetricValues & new_values, const std::string & cpu_suffix, const ProcStatValuesCPU & delta_values, double multiplier);
|
||||
AsynchronousMetricValues & new_values, const ProcStatValuesCPU & delta_values, double multiplier);
|
||||
|
||||
void applyPerCPUMetricsUpdate(AsynchronousMetricValues & new_values, ProcStatPerCPUMaps && per_cpu_maps);
|
||||
|
||||
void applyNormalizedCPUMetricsUpdate(
|
||||
AsynchronousMetricValues & new_values,
|
||||
|
||||
@@ -43,6 +43,12 @@ ColumnsDescription AsynchronousMetricLogElement::getColumnsDescription()
|
||||
parseQuery(codec_parser, "(ZSTD(1))", 0, DBMS_DEFAULT_MAX_PARSER_DEPTH, DBMS_DEFAULT_MAX_PARSER_BACKTRACKS),
|
||||
"Metric name."
|
||||
},
|
||||
{
|
||||
"key",
|
||||
std::make_shared<DataTypeLowCardinality>(std::make_shared<DataTypeString>()),
|
||||
parseQuery(codec_parser, "(ZSTD(1))", 0, DBMS_DEFAULT_MAX_PARSER_DEPTH, DBMS_DEFAULT_MAX_PARSER_BACKTRACKS),
|
||||
"The key of a key-value metric, e.g. the CPU core number or the block device name. Empty for scalar metrics."
|
||||
},
|
||||
{
|
||||
"value",
|
||||
std::make_shared<DataTypeFloat64>(),
|
||||
@@ -60,6 +66,7 @@ void AsynchronousMetricLogElement::appendToBlock(MutableColumns & columns) const
|
||||
columns[column_idx++]->insert(event_date);
|
||||
columns[column_idx++]->insert(event_time);
|
||||
columns[column_idx++]->insert(metric_name);
|
||||
columns[column_idx++]->insert(key);
|
||||
columns[column_idx++]->insert(value);
|
||||
}
|
||||
|
||||
@@ -75,12 +82,30 @@ void AsynchronousMetricLog::addValues(const AsynchronousMetricValues & values)
|
||||
/// but we need to store up to UINT64_MAX sometimes.
|
||||
static constexpr double precision = 1000.0;
|
||||
|
||||
for (const auto & [key, value] : values)
|
||||
{
|
||||
element.metric_name = key;
|
||||
element.value = round(value.value * precision) / precision;
|
||||
auto round_value = [](double value) { return round(value * precision) / precision; };
|
||||
|
||||
add([&](AsynchronousMetricLogElement & log_element) { log_element = element; });
|
||||
for (const auto & [name, value] : values)
|
||||
{
|
||||
element.metric_name = name;
|
||||
|
||||
if (value.isMap())
|
||||
{
|
||||
/// A key-value metric is logged as one row per key.
|
||||
for (const auto & [key, key_value] : value.key_values)
|
||||
{
|
||||
element.key = key;
|
||||
element.value = round_value(key_value);
|
||||
|
||||
add([&](AsynchronousMetricLogElement & log_element) { log_element = element; });
|
||||
}
|
||||
}
|
||||
else
|
||||
{
|
||||
element.key.clear();
|
||||
element.value = round_value(value.value);
|
||||
|
||||
add([&](AsynchronousMetricLogElement & log_element) { log_element = element; });
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -16,6 +16,8 @@ struct AsynchronousMetricLogElement
|
||||
UInt16 event_date{};
|
||||
time_t event_time{};
|
||||
std::string metric_name;
|
||||
/// The key of a key-value metric (e.g. the CPU core number or the block device name). Empty for scalar metrics.
|
||||
std::string key;
|
||||
double value{};
|
||||
|
||||
static std::string name() { return "AsynchronousMetricLog"; }
|
||||
@@ -31,8 +33,8 @@ public:
|
||||
|
||||
void addValues(const AsynchronousMetricValues &);
|
||||
|
||||
/// This table is usually queried for fixed metric name.
|
||||
static const char * getDefaultOrderBy() { return "metric, event_date, event_time"; }
|
||||
/// This table is usually queried for fixed metric name (and, for key-value metrics, a fixed key).
|
||||
static const char * getDefaultOrderBy() { return "metric, key, event_date, event_time"; }
|
||||
};
|
||||
|
||||
}
|
||||
@@ -333,6 +333,20 @@ void ServerAsynchronousMetrics::updateImpl(TimePoint update_time, TimePoint curr
|
||||
/// Free and total space on every configured disk.
|
||||
{
|
||||
DisksMap disks_map = getContext()->getDisksMap();
|
||||
|
||||
AsynchronousMetricKeyValues disk_total;
|
||||
AsynchronousMetricKeyValues disk_used;
|
||||
AsynchronousMetricKeyValues disk_available;
|
||||
AsynchronousMetricKeyValues disk_unreserved;
|
||||
#if USE_AWS_S3
|
||||
AsynchronousMetricKeyValues disk_put_object_throttler_rps;
|
||||
AsynchronousMetricKeyValues disk_put_object_throttler_available;
|
||||
AsynchronousMetricKeyValues disk_get_object_throttler_rps;
|
||||
AsynchronousMetricKeyValues disk_get_object_throttler_available;
|
||||
#endif
|
||||
AsynchronousMetricKeyValues dead_blobs_queue_estimate;
|
||||
AsynchronousMetricKeyValues missing_blobs_queue_estimate;
|
||||
|
||||
for (const auto & [name, disk] : disks_map)
|
||||
{
|
||||
auto total = disk->getTotalSpace();
|
||||
@@ -343,21 +357,16 @@ void ServerAsynchronousMetrics::updateImpl(TimePoint update_time, TimePoint curr
|
||||
auto available = disk->getAvailableSpace();
|
||||
auto unreserved = disk->getUnreservedSpace();
|
||||
|
||||
new_values[fmt::format("DiskTotal_{}", name)] = { *total,
|
||||
"The total size in bytes of the disk (virtual filesystem). Remote filesystems may not provide this information and can show a large value like 16 EiB." };
|
||||
disk_total[name] = static_cast<double>(*total);
|
||||
|
||||
if (available)
|
||||
{
|
||||
new_values[fmt::format("DiskUsed_{}", name)] = { *total - *available,
|
||||
"Used bytes on the disk (virtual filesystem). Remote filesystems do not always provide this information." };
|
||||
|
||||
new_values[fmt::format("DiskAvailable_{}", name)] = { *available,
|
||||
"Available bytes on the disk (virtual filesystem). Remote filesystems may not provide this information and can show a large value like 16 EiB." };
|
||||
disk_used[name] = static_cast<double>(*total - *available);
|
||||
disk_available[name] = static_cast<double>(*available);
|
||||
}
|
||||
|
||||
if (unreserved)
|
||||
new_values[fmt::format("DiskUnreserved_{}", name)] = { *unreserved,
|
||||
"Available bytes on the disk (virtual filesystem) without the reservations for merges, fetches, and moves. Remote filesystems may not provide this information and can show a large value like 16 EiB." };
|
||||
disk_unreserved[name] = static_cast<double>(*unreserved);
|
||||
}
|
||||
|
||||
#if USE_AWS_S3
|
||||
@@ -365,29 +374,65 @@ void ServerAsynchronousMetrics::updateImpl(TimePoint update_time, TimePoint curr
|
||||
{
|
||||
if (auto put_throttler = s3_client->getPutRequestThrottler())
|
||||
{
|
||||
new_values[fmt::format("DiskPutObjectThrottlerRPS_{}", name)] = { put_throttler->getMaxSpeed(),
|
||||
"PutObject Request throttling limit on the disk in requests per second (virtual filesystem). Local filesystems may not provide this information." };
|
||||
new_values[fmt::format("DiskPutObjectThrottlerAvailable_{}", name)] = { put_throttler->getAvailable(),
|
||||
"Number of PutObject requests that can be currently issued without hitting throttling limit on the disk (virtual filesystem). Local filesystems may not provide this information." };
|
||||
disk_put_object_throttler_rps[name] = static_cast<double>(put_throttler->getMaxSpeed());
|
||||
disk_put_object_throttler_available[name] = static_cast<double>(put_throttler->getAvailable());
|
||||
}
|
||||
if (auto get_throttler = s3_client->getGetRequestThrottler())
|
||||
{
|
||||
new_values[fmt::format("DiskGetObjectThrottlerRPS_{}", name)] = { get_throttler->getMaxSpeed(),
|
||||
"GetObject Request throttling limit on the disk in requests per second (virtual filesystem). Local filesystems may not provide this information." };
|
||||
new_values[fmt::format("DiskGetObjectThrottlerAvailable_{}", name)] = { get_throttler->getAvailable(),
|
||||
"Number of GetObject requests that can be currently issued without hitting throttling limit on the disk (virtual filesystem). Local filesystems may not provide this information." };
|
||||
disk_get_object_throttler_rps[name] = static_cast<double>(get_throttler->getMaxSpeed());
|
||||
disk_get_object_throttler_available[name] = static_cast<double>(get_throttler->getAvailable());
|
||||
}
|
||||
}
|
||||
#endif
|
||||
|
||||
if (auto object_storage_disk = std::dynamic_pointer_cast<DiskObjectStorage>(disk))
|
||||
{
|
||||
new_values[fmt::format("{}DeadBlobsQueueEstimate", name)] = { object_storage_disk->getDeadBlobsQueueEstimate(),
|
||||
"Estimated number of blobs enqueued for removal from the disk object storage (the blob manager dead queue). Disks without blob replication report 0." };
|
||||
new_values[fmt::format("{}MissingBlobsQueueEstimate", name)] = { object_storage_disk->getMissingBlobsQueueEstimate(),
|
||||
"Estimated number of blobs awaiting replication to other locations of the disk (the blob manager missing queue). Disks without blob replication report 0." };
|
||||
dead_blobs_queue_estimate[name] = static_cast<double>(object_storage_disk->getDeadBlobsQueueEstimate());
|
||||
missing_blobs_queue_estimate[name] = static_cast<double>(object_storage_disk->getMissingBlobsQueueEstimate());
|
||||
}
|
||||
}
|
||||
|
||||
if (!disk_total.empty())
|
||||
new_values["DiskTotal"] = { "disk", std::move(disk_total),
|
||||
"The total size in bytes of every disk (virtual filesystem), keyed by the disk name. Remote filesystems may not provide this information and can show a large value like 16 EiB." };
|
||||
|
||||
if (!disk_used.empty())
|
||||
new_values["DiskUsed"] = { "disk", std::move(disk_used),
|
||||
"Used bytes on every disk (virtual filesystem), keyed by the disk name. Remote filesystems do not always provide this information." };
|
||||
|
||||
if (!disk_available.empty())
|
||||
new_values["DiskAvailable"] = { "disk", std::move(disk_available),
|
||||
"Available bytes on every disk (virtual filesystem), keyed by the disk name. Remote filesystems may not provide this information and can show a large value like 16 EiB." };
|
||||
|
||||
if (!disk_unreserved.empty())
|
||||
new_values["DiskUnreserved"] = { "disk", std::move(disk_unreserved),
|
||||
"Available bytes on every disk (virtual filesystem) without the reservations for merges, fetches, and moves, keyed by the disk name. Remote filesystems may not provide this information and can show a large value like 16 EiB." };
|
||||
|
||||
#if USE_AWS_S3
|
||||
if (!disk_put_object_throttler_rps.empty())
|
||||
{
|
||||
new_values["DiskPutObjectThrottlerRPS"] = { "disk", std::move(disk_put_object_throttler_rps),
|
||||
"PutObject Request throttling limit on every disk in requests per second (virtual filesystem), keyed by the disk name. Local filesystems may not provide this information." };
|
||||
new_values["DiskPutObjectThrottlerAvailable"] = { "disk", std::move(disk_put_object_throttler_available),
|
||||
"Number of PutObject requests that can be currently issued without hitting throttling limit on every disk (virtual filesystem), keyed by the disk name. Local filesystems may not provide this information." };
|
||||
}
|
||||
|
||||
if (!disk_get_object_throttler_rps.empty())
|
||||
{
|
||||
new_values["DiskGetObjectThrottlerRPS"] = { "disk", std::move(disk_get_object_throttler_rps),
|
||||
"GetObject Request throttling limit on every disk in requests per second (virtual filesystem), keyed by the disk name. Local filesystems may not provide this information." };
|
||||
new_values["DiskGetObjectThrottlerAvailable"] = { "disk", std::move(disk_get_object_throttler_available),
|
||||
"Number of GetObject requests that can be currently issued without hitting throttling limit on every disk (virtual filesystem), keyed by the disk name. Local filesystems may not provide this information." };
|
||||
}
|
||||
#endif
|
||||
|
||||
if (!dead_blobs_queue_estimate.empty())
|
||||
{
|
||||
new_values["DeadBlobsQueueEstimate"] = { "disk", std::move(dead_blobs_queue_estimate),
|
||||
"Estimated number of blobs enqueued for removal from the disk object storage (the blob manager dead queue), keyed by the disk name. Disks without blob replication report 0." };
|
||||
new_values["MissingBlobsQueueEstimate"] = { "disk", std::move(missing_blobs_queue_estimate),
|
||||
"Estimated number of blobs awaiting replication to other locations of the disk (the blob manager missing queue), keyed by the disk name. Disks without blob replication report 0." };
|
||||
}
|
||||
}
|
||||
|
||||
{
|
||||
|
||||
@@ -126,7 +126,27 @@ void writeMetric(DB::WriteBuffer & wb, size_t metric, const std::string & labels
|
||||
writeMetricLine(wb, key, labels_suffix, value);
|
||||
}
|
||||
|
||||
void writeAsyncMetrics(DB::WriteBuffer & wb, const DB::AsynchronousMetricValues & values, const std::string & labels_suffix)
|
||||
void writeLabelValueEscaped(DB::WriteBuffer & wb, const std::string & value)
|
||||
{
|
||||
for (char c : value)
|
||||
{
|
||||
if (c == '\\' || c == '"')
|
||||
{
|
||||
DB::writeChar('\\', wb);
|
||||
DB::writeChar(c, wb);
|
||||
}
|
||||
else if (c == '\n')
|
||||
{
|
||||
DB::writeChar('\\', wb);
|
||||
DB::writeChar('n', wb);
|
||||
}
|
||||
else
|
||||
DB::writeChar(c, wb);
|
||||
}
|
||||
}
|
||||
|
||||
void writeAsyncMetrics(DB::WriteBuffer & wb, const DB::AsynchronousMetricValues & values,
|
||||
const std::string & constant_labels, const std::string & constant_labels_suffix)
|
||||
{
|
||||
for (const auto & name_value : values)
|
||||
{
|
||||
@@ -135,14 +155,39 @@ void writeAsyncMetrics(DB::WriteBuffer & wb, const DB::AsynchronousMetricValues
|
||||
if (!replaceInvalidChars(key))
|
||||
continue;
|
||||
|
||||
auto value = name_value.second;
|
||||
const auto & value = name_value.second;
|
||||
|
||||
std::string metric_doc{value.documentation};
|
||||
convertHelpToSingleLine(metric_doc);
|
||||
|
||||
writeOutLine(wb, "# HELP", key, metric_doc);
|
||||
writeOutLine(wb, "# TYPE", key, "gauge");
|
||||
writeMetricLine(wb, key, labels_suffix, value.value);
|
||||
|
||||
if (value.isMap())
|
||||
{
|
||||
/// A key-value metric is exported as one line per key, with the key as a label,
|
||||
/// e.g. `ClickHouseAsyncMetrics_BlockReadBytes{device="sda"} 123`.
|
||||
std::string label_name{value.key_label};
|
||||
if (!replaceInvalidChars(label_name))
|
||||
continue;
|
||||
|
||||
for (const auto & [map_key, map_value] : value.key_values)
|
||||
{
|
||||
DB::WriteBufferFromOwnString labels_wb;
|
||||
DB::writeChar('{', labels_wb);
|
||||
DB::writeText(constant_labels, labels_wb);
|
||||
if (!constant_labels.empty())
|
||||
DB::writeChar(',', labels_wb);
|
||||
DB::writeText(label_name, labels_wb);
|
||||
DB::writeText("=\"", labels_wb);
|
||||
writeLabelValueEscaped(labels_wb, map_key);
|
||||
DB::writeText("\"}", labels_wb);
|
||||
|
||||
writeMetricLine(wb, key, labels_wb.str(), map_value);
|
||||
}
|
||||
}
|
||||
else
|
||||
writeMetricLine(wb, key, constant_labels_suffix, value.value);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -185,7 +230,7 @@ void PrometheusMetricsWriter::writeMetrics(WriteBuffer & wb) const
|
||||
|
||||
void PrometheusMetricsWriter::writeAsynchronousMetrics(WriteBuffer & wb, const AsynchronousMetrics & async_metrics) const
|
||||
{
|
||||
writeAsyncMetrics(wb, async_metrics.getValues(), constant_labels_suffix);
|
||||
writeAsyncMetrics(wb, async_metrics.getValues(), constant_labels, constant_labels_suffix);
|
||||
}
|
||||
|
||||
void PrometheusMetricsWriter::writeErrors(WriteBuffer & wb) const
|
||||
@@ -422,7 +467,7 @@ void KeeperPrometheusMetricsWriter::writeAsynchronousMetrics([[maybe_unused]] Wr
|
||||
[[maybe_unused]] const AsynchronousMetrics & async_metrics) const
|
||||
{
|
||||
#if USE_NURAFT
|
||||
writeAsyncMetrics(wb, async_metrics.getValues(), constant_labels_suffix);
|
||||
writeAsyncMetrics(wb, async_metrics.getValues(), constant_labels, constant_labels_suffix);
|
||||
#endif
|
||||
}
|
||||
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
#include <Columns/IColumn.h>
|
||||
#include <Storages/System/SystemTableSourceRegistry.h>
|
||||
#include <DataTypes/DataTypeMap.h>
|
||||
#include <DataTypes/DataTypeString.h>
|
||||
#include <DataTypes/DataTypesNumber.h>
|
||||
#include <Common/AsynchronousMetrics.h>
|
||||
@@ -14,7 +15,8 @@ ColumnsDescription StorageSystemAsynchronousMetrics::getColumnsDescription()
|
||||
auto description = ColumnsDescription
|
||||
{
|
||||
{"metric", std::make_shared<DataTypeString>(), "Metric name."},
|
||||
{"value", std::make_shared<DataTypeFloat64>(), "Metric value."},
|
||||
{"value", std::make_shared<DataTypeFloat64>(), "Metric value. For key-value metrics (broken down per CPU core, block device, disk, ...) it is NaN, and the values are in the `key_values` column."},
|
||||
{"key_values", std::make_shared<DataTypeMap>(std::make_shared<DataTypeString>(), std::make_shared<DataTypeFloat64>()), "Values of a key-value metric, e.g. keyed by the CPU core number or the block device name. Empty for scalar metrics."},
|
||||
{"description", std::make_shared<DataTypeString>(), "Metric description."},
|
||||
};
|
||||
|
||||
@@ -38,7 +40,14 @@ void StorageSystemAsynchronousMetrics::fillData(MutableColumns & res_columns, Co
|
||||
{
|
||||
res_columns[0]->insert(name_value.first);
|
||||
res_columns[1]->insert(name_value.second.value);
|
||||
res_columns[2]->insert(name_value.second.documentation);
|
||||
|
||||
Map key_values;
|
||||
key_values.reserve(name_value.second.key_values.size());
|
||||
for (const auto & [key, value] : name_value.second.key_values)
|
||||
key_values.push_back(Tuple{key, value});
|
||||
res_columns[2]->insert(key_values);
|
||||
|
||||
res_columns[3]->insert(name_value.second.documentation);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -71,40 +71,21 @@ def test_async_logger_profile_events(start_cluster):
|
||||
|
||||
|
||||
def test_async_logger_asynchronous_metrics(start_cluster):
|
||||
metrics = json.loads(
|
||||
# The per-channel queue sizes are published as a single key-value metric,
|
||||
# keyed by the logging channel name.
|
||||
channels = json.loads(
|
||||
node.query(
|
||||
"""
|
||||
SELECT metric, value
|
||||
SELECT mapKeys(key_values) AS channels
|
||||
FROM system.asynchronous_metrics
|
||||
WHERE name ILIKE '%asynclog%'
|
||||
WHERE metric = 'AsyncLoggingQueueSize'
|
||||
FORMAT JSON
|
||||
"""
|
||||
)
|
||||
)["data"]
|
||||
)["data"][0]["channels"]
|
||||
|
||||
# Check that the metric exists
|
||||
assert (
|
||||
next(
|
||||
(x for x in metrics if x["metric"] == "AsyncLoggingErrorFileLogQueueSize"),
|
||||
None,
|
||||
)
|
||||
is not None
|
||||
), metrics
|
||||
assert (
|
||||
next(
|
||||
(x for x in metrics if x["metric"] == "AsyncLoggingFileLogQueueSize"), None
|
||||
)
|
||||
is not None
|
||||
), metrics
|
||||
assert (
|
||||
next(
|
||||
(x for x in metrics if x["metric"] == "AsyncLoggingConsoleQueueSize"), None
|
||||
)
|
||||
is not None
|
||||
), metrics
|
||||
assert (
|
||||
next(
|
||||
(x for x in metrics if x["metric"] == "AsyncLoggingTextLogQueueSize"), None
|
||||
)
|
||||
is not None
|
||||
), metrics
|
||||
# Check that all the channels are reported
|
||||
assert "ErrorFileLog" in channels, channels
|
||||
assert "FileLog" in channels, channels
|
||||
assert "Console" in channels, channels
|
||||
assert "TextLog" in channels, channels
|
||||
@@ -27,6 +27,7 @@ CREATE TABLE system.asynchronous_metrics
|
||||
(
|
||||
`metric` String,
|
||||
`value` Float64,
|
||||
`key_values` Map(String, Float64),
|
||||
`description` String,
|
||||
`name` String ALIAS metric
|
||||
)
|
||||
|
||||
@@ -0,0 +1,5 @@
|
||||
1 1
|
||||
0
|
||||
1 1
|
||||
1
|
||||
0
|
||||
@@ -0,0 +1,14 @@
|
||||
-- `DiskTotal` is a key-value metric: a single row with a Map from the disk name to the total size,
|
||||
-- and NaN in the `value` column.
|
||||
SELECT isNaN(value), key_values['default'] > 0 FROM system.asynchronous_metrics WHERE metric = 'DiskTotal';
|
||||
|
||||
-- There are no flattened per-disk metrics (like `DiskTotal_default`) anymore.
|
||||
SELECT count() FROM system.asynchronous_metrics WHERE metric LIKE 'DiskTotal\_%';
|
||||
|
||||
-- Scalar metrics have an empty map in the `key_values` column.
|
||||
SELECT empty(key_values), NOT isNaN(value) FROM system.asynchronous_metrics WHERE metric = 'AsynchronousMetricsUpdateInterval';
|
||||
|
||||
-- In the log, a key-value metric is written as one row per key.
|
||||
SYSTEM FLUSH LOGS asynchronous_metric_log;
|
||||
SELECT count() > 0 FROM system.asynchronous_metric_log WHERE metric = 'DiskTotal' AND key = 'default';
|
||||
SELECT count() FROM system.asynchronous_metric_log WHERE metric LIKE 'DiskTotal\_%';
|
||||
Reference in new issue
Block a user