`ListRequestGenerator::generateImpl` built a `ZooKeeperListRequest` and set
`list_request_type` to `ListRequestType::ALL`. `ZooKeeperListRequest::getOpNum`
returns `OpNum::FilteredList` (500) whenever that `std::optional` is engaged,
without inspecting the value, so a semantically plain "list the children of a
path" request was encoded with a ClickHouse-Keeper-private op code.
Apache ZooKeeper has no op code above `whoAmI` (107) and rejects it two
different ways, both measured against `zookeeper:3.6.2`:
- standalone, `Request.isValid` falls through to `default: return false` and the
request is answered `Unimplemented` (-6); the server logs
`Received packet at server of unknown type 500`.
- inside `multi`/`multiRead`, `MultiOperationRecord.deserialize` throws
`IOException("Invalid type of op")`, which surfaces as `MARSHALLINGERROR` (-5).
Either one aborts the documented invocation
`clickhouse keeper-bench --config programs/keeper-bench/example.yaml`, whose
generator holds both a standalone `list` and a `multi` with `get` + `list`, on
any ZooKeeper-compatible server - a configuration
`programs/keeper-bench/README.md` advertises as supported. The setup phase still
succeeds because it goes through `ZooKeeper::list`, which normalizes `ALL` away.
`ALL` is semantically a plain `getChildren`, and the `list` generator exposes no
config knob for a filter, so `ALL` is the only value the tool can express. Emit
`ZooKeeperSimpleListRequest` instead: `OpNum::SimpleList` is 8, `getChildren`,
the only list op ZooKeeper accepts both standalone and as a `multi` sub-request.
Dropping `list_request_type` to get `OpNum::List` (12, `getChildren2`) would not
be enough, because ZooKeeper's multi whitelist does not contain `getChildren2`.
No capability detection is needed in either direction:
`KeeperContext::isOperationSupported` lists `OpNum::SimpleList` in its
unconditional group, so the generator now uses no Keeper-specific capability at
all.
Against ClickHouse Keeper the server-side work is unchanged: `KeeperStorageImpl`
reads `list_request_type.value_or(ALL)` and takes the same fast path for op 500
carrying `ALL` as for a plain list. Two observable deltas, both intended: the
response no longer carries a `Stat`, measured at exactly 72 bytes
(`sizeof(Coordination::Stat)`) per list response, and the run is reported under
the `SimpleList` stats bucket rather than `FilteredList`.
Closes: https://github.com/ClickHouse/ClickHouse/issues/72629
Literal paths, `children_of` and `tagged` entries can be combined freely in
one `path` section, as on master. Sets are weighted by an estimate of their
total size: the calling thread's shard size scaled by the number of shards,
so a per-thread sharded dynamic set is not underweighted against an
unsharded one.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A `PathGetter` can now reference several `tagged` sets; paths are drawn
uniformly from their union by first picking a set with probability
proportional to its size and then a path within it. Sizes are read from a
new per-shard atomic `cached_size` (relaxed loads) instead of locking the
shard mutex, so the weights can lag slightly behind concurrent updates of
dynamic sets, which is acceptable.
`PathGetter::takePath` was unused and is removed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`Generator.cpp` uses `std::views::keys` and relied on `<ranges>` arriving transitively, which
`_LIBCPP_REMOVE_TRANSITIVE_INCLUDES` does not allow once the includes it came through are trimmed.
It is compiled as part of `ninja clickhouse`.
Two forms:
- `remove_recursive: true` (with `remove_nodes_limit`) on a create
generator makes its `remove_factor`/`keep_count` removes use
`RemoveRecursive` requests, removing whole subtrees. Needed when other
generators create children under the removable nodes, where a plain
`Remove` would fail with `ZNOTEMPTY`.
- a standalone `remove_recursive` request generator that recursively
removes random nodes drawn from a path set (making it dynamic). Only
the removed root is taken out of the set; descendants tracked in other
sets go stale and surface as ignored errors. Since it competes with
creates purely by weights, the node count random-walks; `keep_count`
with `remove_recursive: true` is the count-stable alternative.
Example workload this enables: one create maintains ~N tagged nodes
(removing recursively), a second create makes children under the tagged
nodes, gets read them — total node count stays stable in steady state.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`keep_count: <n>` (or `auto`) on a create generator keeps its output path
set at a roughly constant size, so the dataset doesn't drift over a long
benchmark: each operation chooses between Create and Remove with a
probability based on the current shard size — 0.5 at the target,
approaching 0/1 as the size deviates (logistic curve, 5% of the target as
the scale). The last path in a shard is never removed, so a set can only
be empty if it started empty. `auto` snapshots the size at the end of
setup as the target. Mutually exclusive with `remove_factor`.
Sizes and targets of dynamic path sets are now printed with each periodic
report in both the generated and storage modes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Path sets that a `create` generator outputs to are now dynamic: created
nodes are added to the set on success and `remove_factor` removes take
nodes out of it (before sending; put back unless the outcome shows the
node is gone). The output set is the explicit `tag` from the create
generator's config, the `children_of` set of its fixed parent, or an
anonymous set when only `remove_factor` needs one; it is only populated
if some generator reads it. This allows pointing e.g. a `set` generator
at nodes produced by a `create` generator, and replaces the old
per-generator `paths_created` bookkeeping (along with the name
uniqueness enforcement, which never worked across threads anyway).
Dynamic sets are sharded by worker thread, so each thread reads and
updates only its own shard (shard mutexes still guard against response
callbacks running on client/commit threads). Since a dynamic set may lag
behind the real state, requests drawing paths from one report `ZNONODE`/
`ZNODEEXISTS` results as a separate `ignored_errors` counter instead of
failing, even without `continue_on_errors`. Generators whose input set
is currently empty decline, and another generator is picked instead.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Each `PathGetter` now draws from exactly one `PathSet` — a literal path
list, one `tagged` reference, or one `children_of` reference (mixing
sources in one `path` is no longer supported). Sets are registered in
`NodesSetup` while parsing the setup and generator configs, so generators
referencing the same tag or parent share one in-memory copy of the paths
instead of each generator merging its own copy.
The initialization sequence is reordered accordingly: parse generators
(registering sets), create the setup tree (populating tag sets on the
fly), resolve `children_of` sets by listing each distinct parent once,
then validate. Tag sets nobody reads are not populated at all.
The setup tree now stores `repeat` counts unexpanded, mirroring the
config; instances are expanded during the creation traversal (root names
are recorded for cleanup). For repeated subtrees this removes the
per-instance child pointers from memory.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Generators used to be constructed per worker thread, each re-parsing the
config and building its own copy of every path list (including the results
of `children_of` listings), multiplying memory usage by the concurrency.
Now a single `Generator` is built once on the main thread and shared; the
only per-call mutable state is the rng, passed in via `GenerateContext`
(`{rng, thread_idx}`) by each thread.
This also deletes the `setSeed`/`setSeedImpl` cascade with its per-component
seed offsets: with a fixed `generator.seed`, each thread's rng is simply
seeded with `seed + thread_idx`. `children_of` parents are now listed once
instead of once per thread, and the workload description is printed once.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Previously, when a write multi failed, all sub-request callbacks received
nullptr, losing the per-op error codes. Now each sub-op callback receives
its own sub-response. Sub-ops of a failed transaction whose own check
passed (reported as `ZOK` even though nothing was applied) receive a
synthesized response with `ZRUNTIMEINCONSISTENCY` so they are not taken
for applied ones.
This is preparation for dynamic path sets, where Create/Remove callbacks
need to distinguish "node is really gone" (`ZNONODE`) from "operation was
not applied" to keep the set contents correct.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New `--storage` flag switches keeper-bench into an in-process mode that exercises
`KeeperStorage` directly, without the rest of the keeper server (no network, no
raft, no state machine).
Threading model mirrors a simplified Keeper commit path:
* N generator threads (`--concurrency`, default 4) produce requests using the
existing `Generator` (so `setup` + `generator` config is reused).
* One "preprocess" thread calls `preprocessRequest` for every write (stands in
for the path that, in real keeper, runs under `raft_server::lock_`).
* One "commit" thread calls `processRequest` for writes. After each committed
write, with probability `1/writes_per_read_batch` it drains the pending read
queue and processes it in one `processLocalRequests` batch.
Inter-thread queues are `NonblockingBoundedQueue` (copied from commit
6a91c30115) with 100µs polling — no condition
variables. The preprocess->commit queue has a small configurable bound
(`storage.preprocess_commit_queue_size`, default 32).
Periodic reports include (for the period): committed writes/s and reads/s,
preprocess / commit-write / commit-read thread utilization, znode count, RSS,
snapshot-mode flag, and per-op-num average preprocess/process ns and ns-per-
list-name.
Every `storage.snapshot_toggle_periods` periods the runner toggles
`enableSnapshotMode`/`disableSnapshotMode` + `clearGarbageAfterSnapshot`, so the
snapshot overhead shows up in the timings.
Refactoring: `Generator`/`PathGetter` previously took a `Coordination::ZooKeeper
&` to resolve `children_of` paths. They now take a `ListChildrenFn` callback;
the existing Runner passes a lambda wrapping the ZooKeeper client, the new
StorageRunner passes a lambda that calls `KeeperStorage::processLocalRequests`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add `// Ok` comments to `catch (...)` blocks in `Runner.cpp` that handle
exceptions via `handle_request_exception` (which logs and counts errors)
but were flagged by the `catch_all` style check.
Fix seed collision in `MultiRequestGenerator::setSeedImpl`: the inner
`RequestGetter::setSeed` distributed sub-seeds as `seed + i + 1`, which
collided with siblings assigned by the outer `RequestGetter` using the
same scheme. Use a large prime offset (500009) to separate the inner
seed namespace.
https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=98906&sha=latest&name_0=PR
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Previously only `CreateRequestGenerator::rng` was seeded from the
configured seed. Four other RNGs — in `NumberGetter::getNumber`,
`StringGetter::getString` (via `generateRandomString`),
`PathGetter::getPath`, and `RequestGetter::getRequestGenerator` —
used independent `thread_local pcg64(randomSeed())` instances,
making runs non-deterministic even with a fixed seed.
Replace all `thread_local` RNGs with per-instance `mutable pcg64`
members and propagate the seed through the full generator hierarchy:
`Generator` → `RequestGetter` → each `RequestGenerator` subclass →
their `PathGetter`, `StringGetter`, and `NumberGetter` members.
Each component uses a distinct sub-seed offset to avoid correlation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add `pipeline_depth` and `queue_depth` settings to allow asynchronous
in-flight requests per consumer thread, hiding round-trip latency
- Add `warmup_seconds` to discard early stats while caches warm up
- Add `generator.seed` for reproducible workloads
- Track per-operation-type stats (Create, Remove, Get, Set, List, Multi)
in both stderr reports and JSON output
- Add `on_failure_callbacks` to clean up pending paths on failed creates
- Replace `paths_created` set with O(1) swap-and-pop vector+index structure
for random path removal
- Fix `static` → `thread_local` for RNG instances to avoid data races
- Fix `continue_on_errors` → `continue_on_error` (match actual config key)
- Validate `remove_factor` range, zero weights, and empty path getters
- Add retry limit for unique path name generation
- Simplify `removeRecursive` and deduplicate default ACL construction
- Document all new settings in README
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
