# Generated workload from config
clickhouse keeper-bench \
    --config <config_file>

# Replay workload from a request log
clickhouse keeper-bench \
    -h localhost:9181 \
    --input-request-log <request_log_file>

An example config is available at programs/keeper-bench/example.yaml.

Command-line options

Flag	Short	Description
`--help`		Print help and exit
`--config`		YAML/XML config file
`--input-request-log`		Replay requests from a request log file (enables replay mode)
`--setup-nodes-snapshot-path`		Directory containing Keeper snapshots used to build initial node state for replay
`--concurrency`	`-c`	Number of parallel worker threads
`--report-delay`	`-d`	Delay between periodic reports in seconds (`0` disables periodic reports)
`--iterations`	`-i`	Total number of requests to execute (`0` means unlimited)
`--time-limit`	`-t`	Stop producing new requests after this many seconds
`--hosts`	`-h`	Host list, e.g. `-h host1:9181 host2:9181`
`--continue_on_errors`		Continue running after request exceptions

Rules:

--config or --input-request-log must be provided.
Command-line values override config values for overlapping fields.
Hosts can come from --hosts or from connections in config.
If --input-request-log is provided, replay mode is used, even if the config defines generator (so the same config can serve both modes).
Otherwise, generated mode is used; the config must define generator.

Modes

Generated mode (`generator`)

Use when you want synthetic workload generation.

Required:

generator.requests in config.
At least one host from --hosts or connections.

Optional:

setup for creating initial Keeper tree.
output for JSON output.

Replay mode (`--input-request-log`)

Use when you want to replay previously recorded Keeper traffic.

Required:

--input-request-log.
At least one host from --hosts or connections.

Optional:

--setup-nodes-snapshot-path to build/update initial snapshot state inferred from expected replay outcomes.

Storage mode (`--storage`)

Instead of a full Keeper cluster, runs only the KeeperStorage component, inside the keeper-bench process. Measures performance of just the storage and its request processing, without Raft, network, and request dispatch. connections configuration is not needed. Workload is generated the same way as in Generated mode.

Configuration file

Config can be YAML or XML.

Top-level keys
Special types
General settings
Connections
Setup
Generator
Replay request log
Output
Troubleshooting

Top-level keys

Key	Type	Default	Notes
`concurrency`	integer	`1`	Worker threads
`iterations`	integer	`0`	`0` means unlimited
`report_delay`	float	`1.0`	Seconds between periodic reports; `0` disables
`timelimit`	float	`0`	Seconds to stop producing new requests; `0` disables
`continue_on_error`	bool	`false`	Continue after request exceptions
`pipeline_depth`	integer	`1`	In-flight async requests per worker (`>= 1`)
`warmup_seconds`	float	`0`	Measurement warmup window
`enable_tracing`	bool	`false`	Attach OpenTelemetry trace context
`use_remove_recursive`	bool	`true`	Use the native `RemoveRecursive` Keeper request for cleanup (limited to 1M nodes per request to avoid blocking raft for too long); falls back to a manual recursive traversal if the subtree is larger, if the server does not advertise the `REMOVE_RECURSIVE` feature flag, or if this is set to `false` (needed for ZooKeeper or older Keeper versions)
`connections`	object	required if `--hosts` absent	Keeper endpoints and connection settings
`setup`	object	optional	Data tree created before run
`generator`	object	optional	Request generator (enables generated mode)
`output`	object	optional	JSON output controls

Special types

These reusable types are used in setup and generator sections.

`IntegerGetter`

A constant integer, or a random value drawn uniformly from a range on each use.

# constant
key: 42

# random from [10, 20]
key:
    min_value: 10
    max_value: 20

`StringGetter`

A constant string, or a random string whose length is an IntegerGetter.

# constant
key: "hello"

# random string with length drawn from [10, 20]
key:
    random_string:
        size:
            min_value: 10
            max_value: 20

`PathGetter`

A set of ZooKeeper paths that requests draw from. Sources can be combined freely: explicit path(s), the children of parent node(s), and tagged sets of paths created during setup.

# explicit paths
path:
    - "/path1"
    - "/path2"

# children of a parent node
path:
    children_of: "/path3"

# paths collected by tag during setup (see Setup section)
path:
    tagged: "my_tag"

# union of several sources
path:
    - "/path1"
    - tagged: "my_tag"
    - tagged: "other_tag"
    - children_of: "/path3"

Notes:

Paths must start with /.
children_of is resolved at startup by listing the parent; if it has no children (and no create generator adds to it), an exception is raised.
tagged references a tag name assigned to setup nodes via the tag field. All paths created with that tag are included. If the tag is not found (and no create generator outputs to it), an exception is raised.
With several sources, paths are drawn uniformly from their union: a set is picked with probability proportional the set's current size.

Dynamic path sets

A PathSet that some create generator outputs to (see the create section) is dynamic: its contents change while the benchmark runs. Created nodes are added to the set, and remove_factor removes take nodes out of it. This makes workloads like "set random nodes produced by create" possible: point the create's output and the set's path at the same tag (or at children_of of the same parent).

Behavior details:

Dynamic sets are sharded by worker thread: each thread reads and updates only its own shard, so different threads operate on disjoint subsets of paths. Initial contents (from setup or listing) are distributed across shards round-robin.
A dynamic set may transiently disagree with the real state of the tree (e.g. a remove reported as timed out actually succeeded). Requests drawing paths from a dynamic set therefore treat "node doesn't exist" and "node already exists" results as expected: they are counted separately as ignored errors (reported in stats and in the JSON output as ignored_errors) and don't stop the benchmark even without --continue_on_errors.
A dynamic set may start empty (e.g. a tag only a create outputs to). Generators whose input set is currently empty are skipped in favor of other generators; if all generators decline, an exception is raised.

General settings

# number of worker threads (default: 1)
concurrency: 20

# total requests to execute; 0 = unlimited (default: 0)
iterations: 10000

# periodic report interval in seconds; 0 = disable (default: 1.0)
report_delay: 4

# stop producing new requests after this many seconds; 0 = no limit (default: 0)
timelimit: 300

# continue on request exceptions (default: false)
continue_on_error: true

# max in-flight requests per worker; must be >= 1 (default: 1)
pipeline_depth: 8

# ignore stats during first N seconds, then reset counters (default: 0)
warmup_seconds: 5

# attach OpenTelemetry tracing context to requests (default: false)
enable_tracing: false

# use the native `RemoveRecursive` Keeper request to wipe setup roots; set to
# false for ZooKeeper or older Keeper servers that do not support it (default: true)
use_remove_recursive: true

Connections

Connections are defined under top-level connections.

Values directly under connections are defaults.
connections.host and connections.connection entries define concrete endpoints.
Each endpoint can open multiple sessions via sessions.

Per-connection settings

secure: boolean                  # use TLS (default: false)
operation_timeout_ms: integer    # operation timeout (default: Keeper client default)
session_timeout_ms: integer      # session timeout (default: Keeper client default)
connection_timeout_ms: integer   # connect timeout (default: Keeper client default)
use_compression: boolean         # protocol compression (default: false)
use_xid_64: boolean              # use 64-bit xid (default: false)
sessions: integer                # sessions per endpoint (default: 1)

Example

connections:
    # defaults for all endpoints
    secure: true
    operation_timeout_ms: 3000

    # one session
    host: "localhost:9181"

    # two sessions with per-endpoint overrides
    connection:
        host: "localhost:9182"
        sessions: 2
        operation_timeout_ms: 2000
        session_timeout_ms: 2000

Setup

setup defines nodes created before benchmarking.

Important behavior:

Before setup creation, the tool recursively removes each configured root node path.
On normal shutdown, it attempts to clean these root paths again.
repeat requires random name; otherwise an exception is raised.

Structure:

node:
    name: StringGetter
    data: StringGetter           # optional
    repeat: integer              # optional, requires random name
    tag: string                  # optional, collects created paths under this tag
    node: ...                    # nested children

The tag field labels a node so that all paths created from it (including repeated copies) are collected into a named set. Generators can reference this set with tagged in their PathGetter config, allowing requests to target specific nodes from the setup tree.

Example:

setup:
    node:
        name: "tree"
        node:
            repeat: 10
            name:
                random_string:
                    size: 20
            tag: "inner"
            node:
                repeat: 3
                name:
                    random_string:
                        size: 10
                tag: "leaves"

generator:
    requests:
        set:
            path:
                tagged: "leaves"
            data: "updated"

In this example, set requests target the 30 leaf nodes (3 per each of 10 inner nodes), selected randomly from all paths tagged "leaves" during setup.

Generator

generator controls synthetic workload generation.

generator:
    # fixed seed for reproducibility; if omitted, random seed is generated and printed
    seed: 12345

Requests are defined in generator.requests.

Supported request types: create, set, get, list, multi.
Each request entry can define weight (default 1, must be >= 1).
Nested multi is not allowed.

`create`

create:
    path: "/bench/creates"           # PathGetter (parent for created nodes)
    name_length: 10                    # IntegerGetter, default: 5
    data: "payload"                   # StringGetter, default: empty
    remove_factor: 0.5                 # in [0.0, 1.0], default: 0
    tag: "created_nodes"              # optional output tag

Created paths are recorded in an output PathSet, which other generators can read (see Dynamic path sets):

If tag is set, the created paths are added to that tagged set.
Otherwise, if path is a single fixed path, they are added to the children_of set of that parent (so e.g. a get with children_of pointing at the same parent automatically sees the created nodes).
Otherwise, if remove_factor is set, they are tracked in an anonymous set only this generator uses.

Setting tag while another generator reads children_of of the same fixed parent is an error: reference the tag consistently instead. The output set is only actually populated if some generator reads it (including this generator's own remove_factor).

When remove_factor is enabled, some operations become random removes of nodes from the output set. Node names are random and not checked for uniqueness; with small name_length creates can fail with "node exists" errors.

`set`

set:
    path: PathGetter
    data: StringGetter

`get`

get:
    path: PathGetter
    watch_probability: 0.5           # in [0.0, 1.0], default: 0 (no watches)

When watch_probability is set, each request has that probability of setting a watch on the node. Watch fire events are counted and reported in stats.

`list`

list:
    path: PathGetter
    watch_probability: 0.5           # in [0.0, 1.0], default: 0 (no watches)

Requests are sent as getChildren, the only list operation that ZooKeeper-compatible servers accept both standalone and as a multi sub-request; responses carry child names without a Stat.

When watch_probability is set, each request has that probability of setting a watch on the node. Watch fire events are counted and reported in stats.

`multi`

multi:
    size: IntegerGetter              # optional
    # nested request generators (`create`/`set`/`get`/`list`) with optional `weight`

Behavior:

If size is set, that many subrequests are sampled.
If size is omitted, one subrequest from each nested generator is included.

Example

generator:
    seed: 42
    requests:
        create:
            path: "/test_create"
            name_length:
                min_value: 10
                max_value: 20
            remove_factor: 0.5

        multi:
            weight: 20
            size: 10
            get:
                path:
                    children_of: "/test_get1"
            get:
                weight: 2
                path:
                    children_of: "/test_get2"

In this example, multi is selected about 20 times more often than create. Inside multi, gets from /test_get2 are selected about 2 times more often than gets from /test_get1.

Replay request log

Replay mode reads requests from --input-request-log.

Behavior details:

Input format and schema are auto-detected.
Compressed files are supported through ClickHouse format/compression detection.
Replay preserves per-session request ordering via executor queues.

Supported operation kinds in logs include Create, Set, Remove, Check, CheckNotExists, Sync, Get, List, Exists, Multi, and MultiRead.

Expected log columns:

hostname
request_event_time
thread_id
session_id
xid
has_watch
op_num
path
data
is_ephemeral
is_sequential
response_event_time
error
requests_size
version

If --setup-nodes-snapshot-path is provided during replay, the tool can infer required initial nodes from expected outcomes and write an updated snapshot.

Output

Stderr progress reports

Periodic stderr reports (controlled by report_delay) include:

Total read/write request counts (cumulative), errors, and ignored errors (see Dynamic path sets).
Read/write RPS and throughput (for the last reporting period).
Read/write latency percentiles (0, 10, ..., 90, 95, 99, 99.9, 99.99).
Watches fired (when watch_probability is configured).

JSON output

Configure JSON output with:

output:
    file: "output.json"
    # or:
    file:
        path: "output.json"
        with_timestamp: true
    stdout: true

JSON fields:

timestamp (epoch milliseconds).
errors.
ignored_errors (present only if nonzero; see Dynamic path sets).
read_results (present only if read requests exist).
write_results (present only if write requests exist).
watches_fired (present only when watches are used).

Each result object contains:

total_requests
requests_per_second
bytes_per_second
percentiles (array of { "<percent>": <latency_ms> } objects)

Troubleshooting

Common configuration exceptions:

No config file or hosts defined: provide --hosts or connections.
--config is required: generated mode needs a config file; pass --input-request-log for replay mode.
Config file must contain a generator section: define generator in the config, or pass --input-request-log for replay mode.
Invalid path for request generator: all paths must start with /.
... is referenced twice in path: the same tag or children_of parent appears more than once in one path section.
... is empty: check that the children_of target has children or that setup nodes carry the tag: the referenced path set ended up empty after setup.
Generator weight must be >= 1: use positive weights only.
remove_factor must be in [0.0, 1.0]: keep probability in range.
watch_probability must be in [0.0, 1.0]: keep probability in range.
Nested multi requests are not allowed: only one multi level is supported.
... is used by a request generator, but no setup node has this tag and no create generator outputs to it: a tagged path reference names an unknown tag. Check spelling and ensure the setup section includes nodes with matching tag values (or that a create generator outputs to the tag).
... outputs to tag ..., but another generator reads children_of ...: reference the created nodes via the tag everywhere instead of mixing tag with children_of.
Repeating node creation ..., but name is not randomly generated: use random name when repeat is set.

README.md

Keeper Bench

Quick start

Command-line options

Modes

Generated mode (generator)

Replay mode (--input-request-log)

Storage mode (--storage)

Configuration file

Table of contents

Top-level keys

Special types

IntegerGetter

StringGetter

PathGetter

Dynamic path sets

General settings

Connections

Per-connection settings

Example

Setup

Generator

create

set

get

list

multi

Example

Replay request log

Output

Stderr progress reports

JSON output

Troubleshooting

Generated mode (`generator`)

Replay mode (`--input-request-log`)

Storage mode (`--storage`)

`IntegerGetter`

`StringGetter`

`PathGetter`

`create`

`set`

`get`

`list`

`multi`