A manifest walk over an Iceberg table with many manifests was bounded by
`iceberg_manifest_decode_concurrency`: fetching a manifest (a remote round trip)
and parsing it (CPU, about 2 ms per manifest) were one task on a dedicated pool,
so only that many round trips overlapped (issue #127061: 671 manifests, 9.6 s at
4, 4.0 s at 32).
Manifest files are now read ahead on the IO thread pool, 64 at a time and at most
64 MiB of manifests not in the metadata files cache, and nothing is parsed there.
The read goes through `getOrSetManifestFile` as before, so concurrent queries
still share one read per manifest and `SYSTEM CLEAR ICEBERG METADATA CACHE` keeps
its semantics; the cached cell holds the bytes and parses them on the first
`getDeserializer()` call (`std::call_once`).
The threads that call `IcebergIterator::next` parse the manifests: each one
continues an idle partially read manifest or parses the next one, so parsing
keeps the parallelism of the reading threads; with one reading thread the files
come in the manifests' order as before. Delete manifests are parsed by the
threads that need them, and the last one sorts the deletes. `totalRows` reads
ahead and parses on the calling thread.
Removed: the manifest decode thread pool and its metrics, the producer thread,
the bounded queue and the stop flag of `ManifestFileIterator`,
`ParallelManifestDecode.h`. The query settings `iceberg_manifest_decode_concurrency`
(and its old name `iceberg_delete_manifest_decode_concurrency`) and
`iceberg_file_entries_queue_size`, and the server settings
`max_iceberg_manifest_decode_thread_pool_size`,
`max_iceberg_manifest_decode_thread_pool_free_size` and
`iceberg_manifest_decode_thread_pool_queue_size` are obsolete: accepted and ignored.
The integration tests of the decode concurrency now vary `max_threads` instead;
the test of the removed pool and the timing test of the removed setting are
deleted.
Closes#127061
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Resolve conflicts in the `dlopen` guard in `programs/main.cpp` and
`programs/keeper/keeper_main.cpp`: keep the guards identical (as master now
requires) and keep `USE_GPU` from this branch in both.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The previous approach was unsafe:
- Swallowing `EPIPE` in `WriteBufferFromFileDescriptor::nextImpl` affected every
pipe, also on the server: data written to a child process that exited early was
silently dropped, and `UserDefinedExecutableFunctionDriverInvoker` and
`clickhouse-disks sed`, which catch the `EPIPE` `ErrnoException`, would get a
`LOGICAL_ERROR` exception from `finalize` of the silently canceled buffer.
- Ignoring `SIGPIPE` in the client was redundant: Poco already blocks it in all
threads, which is why the write fails with `EPIPE` at all. And an ignored signal
is inherited through `exec`, while `ShellCommand` resets only the signal mask, so
the pager and executable UDFs started by `clickhouse-local` would inherit it.
Instead, the console output buffer of the client remembers that its write failed
with `EPIPE`. Then the client does not print the error, does not run further
queries (even with `--ignore-error`), and exits with code 0.
Closes: https://github.com/ClickHouse/ClickHouse/issues/126443
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Conflicts were only in the AI function integration tests: the cancellation
tests from master run alongside this branch's concurrency tests, pinned to one
request per thread.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
They feed only the progress bar and table, the interactive statistics,
`--print-profile-events` and `--memory-usage`. Producing them loads the time
zone for the sample block, which `SELECT 1` otherwise never needs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`argsToConfig` stores the arguments after `--` as flat keys of a
`MapConfiguration` layer (`merge_tree.max_suspicious_broken_parts` -> `7`).
Such a layer has no key for the parent node, so `has("merge_tree")` was
false on the merged `LayeredConfiguration` while `keys("merge_tree")`
already returned the command-line children. Every reader of the form
`if (!config.has(section)) return; config.keys(section, ...)` therefore
silently dropped a section that existed only on the command line:
`MergeTreeSettings`, `DistributedSettings`, `DatabaseReplicatedSettings`,
Keeper coordination settings, `macros` in clickhouse-local,
`remote_url_allow_hosts`, `http_options_response`, system logs,
`zookeeper`, and `filesystem_caches` (which threw at startup instead).
`LayeredConfiguration::getRaw` now reports a node that has children in
some layer as present with an empty value, after the exact-key lookup
over all layers has failed, so a real value is never shadowed whatever
the layer order. The per-site workaround in `LocalServer` that
materialized `http_options_response` for the same reason is removed.
The ZooKeeper reload on a configuration change is gated on the parsed
file it then reads, so a ZooKeeper section given only on the command
line keeps its startup session instead of failing every later reload.
Closes#124747
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Conflicts with https://github.com/ClickHouse/ClickHouse/pull/121671 resolved by
keeping its startup/shutdown race fix in `StorageMergeTree` while dropping the
statistics refresh task (`startStatisticsCache`, `stopStatisticsCache`,
`refresh_stats_task_mutex`), which this PR removes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Resolve conflicts in `MergeTreeData.cpp` with #124420: keep the per-part
`StatisticsCache` and the removal of `refreshStatistics`, take master's
removal of `readLockParts` around `loadStatistics` and the new
`LoadedStatistics` profile event. With the cache, `LoadedStatistics`
counts the parts whose statistics are collected for the estimator, from
the cache or from the part, so its description and the test comments are
updated accordingly.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Conflicts: master reworked the `startup`/`shutdown` race handling and guarded `refresh_stats_task`
with a mutex; this PR removes the periodic statistics refresh task entirely, so master's
`startup`/`shutdown` structure is kept without `startStatisticsCache`/`stopStatisticsCache`.
Conflicts were only in the generated session settings route map and explorer:
master's versions plus `ai_function_max_concurrent_requests_per_thread`.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
`DROP TABLE|VIEW|DATABASE ... SYNC` waits in `DatabaseCatalog::waitTableFinallyDropped`
until the background drop task removes the table. `LocalServer::processConfig` creates and
starts that task in both branches without `--path` (#81705), but the `--path` branch did it
only inside the metadata-loading block, i.e. only when `metadata/` exists and
`--only-system-tables` is not set. With a new directory (released versions) or with
`--only-system-tables` (master, new and existing directory) the task never existed, so the
SYNC drop waited forever. On master the plain new-directory case stopped hanging only as a
side effect of #123124, which writes `metadata/default.sql` before this check.
Start the tasks in the `--path` branch whenever the metadata block did not. The metadata
block itself is unchanged, so `createBackgroundTasks` still runs before `loadMetadata` and
`startupBackgroundTasks` after it. In `--only-system-tables` mode this does not touch the
inspected directory: `cleanup_task` is created only for the server, and clickhouse-local
never calls `loadMarkedAsDroppedTables`, so the drop task only handles drops issued in the
session. The cost is the general schedule pool, which every invocation without `--path`
already creates.
Tests: new 05321 covers `--only-system-tables` with a new and with an existing directory
plus a plain new directory. 04545 asserted that this mode creates no general schedule pool;
clickhouse-local now always creates it, so the test keeps checking only the five lazily
created pools. The rollback comment in `InterpreterCreateQuery` no longer claims that
clickhouse-local never finishes `waitTableFinallyDropped`.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A maximum entry size of 0 (`query_cache.max_entry_size_in_bytes` or
`query_cache.max_entry_size_in_rows`) now means "nothing may be cached" for
both backends, as it always did for the in-memory one. `clickhouse-local`
no longer overloads these limits to disable the in-memory cache: it
creates it with a zero size and takes the entry size limits from its
configuration, so the on-disk cache is usable there.
- `canWriteToQueryResultCacheOnDisk` and `QueryResultCache::canStoreEntries`
take the row limit into account, so the checks against storing wrong
results (non-deterministic functions, system tables, overflow modes) agree
with what the write path actually stores.
- The on-disk entry format (version 5) stores `query_cache_tag`, and
`SYSTEM DROP QUERY CACHE [TAG 'tag']` also removes the matching on-disk
entries from the filesystem cache selected by setting
`query_cache_on_disk_cache_name`, leaving other data of the filesystem
cache in place.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Conflicts:
- ci/jobs/scripts/docs/autogenerate/test_system_table_pages.py: removed on master, accept the deletion.
- src/Storages/MergeTree/MergeTreeData.cpp: keep includes from both sides.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
clickhouse-local set no Keeper component anywhere. Since #123124,
processConfig probes the database disk directly at startup:
tryReadAtomicDatabaseUUID calls existsFile, and
writeMissingDatabaseMetadataFile calls existsFile, createDirectories,
removeFileIfExists, writeMetadataFile and moveFile. On a database_disk
with Keeper-backed metadata each of these is a Keeper request, and with
enforce_keeper_component_tracking on, ZooKeeper::pushRequest throws
LOGICAL_ERROR "Current component is empty", which aborts debug and
sanitizer builds before any query runs.
Set one component as the first statement of processConfig, the same
shape as DisksApp::main. It dominates both helpers and any startup read
added later; ComponentGuard is a thread-local save/restore, so inner
guards (parseQueryFromMetadata, loadMetadata, ...) keep their own names.
Queries run after processConfig and stay at parity with the server.
No test: the Keeper metadata type is registered only under
CLICKHOUSE_CLOUD, so no community configuration reaches the throw.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Conflicts were only in the generated settings route maps and explorers: master's
versions plus this branch's `ai_function_*` settings.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Secret masking for logging, `SHOW CREATE` and `EXPLAIN` lived in
`src/Parsers/FunctionSecretArgumentsFinder.cpp`, a switch on the name of
every table function, engine and backup engine that re-implemented their
argument grammar by hand and drifted from it.
Each registration now declares what is secret in its arguments with a
`SecretArgumentsSpec`, a mandatory parameter of `TableFunctionFactory`,
`StorageFactory`, `DatabaseFactory`, `BackupFactory` and
`DictionarySourceFactory` (and an extra `FunctionFactory` overload for
`encrypt`, `HMAC` and friends). Shapes the fields cannot express are
`custom` callbacks living in the translation unit of the engine's argument
parser. The spec also carries the engine `SETTINGS` to hide (replacing the
`SETTINGS_TO_HIDE` includes in `ASTSetQuery`) and the dictionary `SOURCE`
keys (replacing the hardcoded list in `ASTFunctionWithKeyValueArguments`).
The parser keeps only `AbstractFunction`, the result type and the
`ISecretArgumentsFinder` interface. `SecretArgumentsRegistry` implements it
by looking the spec up in the factories and is installed by `server`,
`local` and `format`; an unregistered engine has every argument hidden.
The WebAssembly parser no longer carries the finder.
The masking cases of `gtest_Parser.cpp` move to a stateless test.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`ai_function_max_concurrent_requests_per_stream` becomes `..._per_thread`, and the
pool settings become `ai_function_max_thread_pool_size`,
`ai_function_max_thread_pool_free_size` and `ai_function_thread_pool_queue_size`.
Regenerated reference docs for these settings.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Alexey Milovidov