Resolve conflicts in the `dlopen` guard in `programs/main.cpp` and
`programs/keeper/keeper_main.cpp`: keep the guards identical (as master now
requires) and keep `USE_GPU` from this branch in both.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adds the memory layer of the GPU engine on its own, split out of `gpu-sum-aggregation`:
`PinnedBuffer`, `DeviceBuffer`, `DeviceStream`, `DeviceEvent`, the shared streams of
`StreamRegistry`, and `initializeDevice`/`checkCuda`/`onDevice` in `GPUDevice`.
With `ENABLE_GPU`, the `src/GPU` sources are built into `dbms` and linked with `ch_gpu::cudart`,
`USE_GPU` is set and shown in `system.build_options`, the `GPU_ERROR` error code is added, and the
`dlopen` override is lifted, as the CUDA runtime reaches the driver only through `dlopen`.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With `USE_OPENSSL_FIPS=ON`, `main.cpp` does not define the `dlopen` stubs because
the FIPS provider is loaded dynamically, but `keeper_main.cpp` still did. The
linker then pulled `keeper_main.cpp.o` out of `clickhouse-keeper-lib` to resolve
`dlopen`, which also defines `main` and `inside_main`, giving duplicate symbol
errors when linking `clickhouse`.
Use the same guard in both files and cross-reference them in comments.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The product name is written `ClickHouse` - one word, with a capital `C`
and a capital `H`. Two case variants are accepted, because a mixed case
is unusable there: `clickhouse` for identifiers, package, binary and host
names, and `CLICKHOUSE` for macros and environment variables. Everything
else - `Clickhouse`, `clickHouse`, `click_house`, `CLICK_HOUSE`,
`click-house`, `Click House` - is a misspelling, and the new
`clickhouse_spelling` check of the style job reports it, in code as well
as in comments, messages, documentation and file names.
The check searches every file tracked by git, and the file names
themselves. It skips:
* `contrib` - third-party sources;
* the generated locale directories of the documentation - a machine
translation of the English pages, which are checked instead;
* release notes - a record of what was published;
* `docs/_specs` - API specifications pinned from their generator;
* regions between `AUTOGENERATED_START` and `AUTOGENERATED_END` markers -
their source of truth (a `FunctionDocumentation` string, for instance)
is checked instead;
* URLs - a link whose path spells the name differently is not a mistake
we can fix, and rewriting it would only break the link;
* the exceptions listed in `clickhouse_spelling_ignore.txt` - text that
is not ours to spell: names defined by other projects (the
`ClickhouseClientOptions` of our Azure SDK fork, the `clickHouse*`
parameters of Google's Dataflow templates), verbatim quotations (Stack
Overflow post titles shown as query output), and the misspellings that
tests feed to the server on purpose (`LIKE` / `ILIKE` case variants, an
unknown URI scheme).
The rest of the change fixes the 191 misspellings this found. Most are
prose and comments; the renamed identifiers are `isClickhouseApp`,
`getClickhouseAttemptNumber` / `setClickhouseAttemptNumber`,
`getClickhouseTableSchemaById` / `hasClickhouseTableSchemaById`, the
`Clickhouse*` classes of `clickhouse-diagnostics`, the `tabClickhouse` /
`btnQueryClickhouse` element ids of the `utils` web tools, and
`clickHousePartsInserter.js`. Three of them are visible to tests: the
`X-Clickhouse-Format` HTTP header, the `SOURCE(Clickhouse(...))`
dictionary source and a directory named `Clickhouse`; the first two are
matched case-insensitively by the server (Poco header collections are
case-insensitive, and the parser lowercases the source name), which I
confirmed against a running server.
The rule is also written down in the style guide.
The whole multicall `clickhouse` binary now compiles (11343 edges) and links
(a 236 MB wasm64 module) with Emscripten, and `clickhouse local` executes
queries correctly under Node.js >= 24 and in Chromium 149: MergeTree tables
on the in-memory filesystem, aggregations over 100 million rows of `numbers`
in ~2 seconds including engine startup, 1847 SQL functions, JSONEachRow from
stdin, Pretty formats. The CI job's pinned target becomes `clickhouse` and
the job provides the module as an artifact.
What it took, beyond the `dbms` step:
* `ENABLE_CAPNP` off for `OS_WASM`: its `kj` library calls `fallocate`
unconditionally in its POSIX branch, which the Emscripten libc lacks.
* Executable link options for `OS_WASM` in cmake/target.cmake: an 8 MiB shadow
stack and 256 MiB of growable memory (the defaults are 64 KiB and a fixed
16 MiB, while `clickhouse` needs ~50 MiB for static data alone); `-g0` at
the link, because binaryen's `wasm-opt` asserts in its DWARF bookkeeping
(`AddrExprMap::add`) on a module with this much debug info;
`-sPROXY_TO_PTHREAD` so `main`, which blocks its thread freely, runs on a
pthread while the browser main thread keeps servicing the event loop that
spawns the Web Workers backing new pthreads - without it the first blocking
`pthread_create` deadlocks the page; and `-sEXIT_RUNTIME` so a finished
query tears the runtime down.
* The `dlopen`-blocking override in programs/main.cpp yields on `OS_WASM`:
Emscripten's libc unconditionally defines `dlerror`, so the override was a
duplicate symbol - and a sandbox cannot load libraries in the first place.
* `initgroups` for `clickhouse-su`: the one libc symbol the entire binary
found missing. Emscripten has the other identity stubs; this one fails
closed with ENOSYS.
* Explicit `Field::safeGet<unsigned long>` and `ColumnVector::indexImpl<size_t>`
instantiations extend from `OS_DARWIN` to `OS_WASM`: both are LP64 platforms
where `uint64_t` is `unsigned long long`, so `size_t` matches no `UInt*` type.
* The `SignalListener` thread in `ClientApplicationBase` follows
`OS_HAS_SIGNAL_HANDLERS`: without signals nothing ever writes to the signal
pipe, and the blocking read of that pipe - all the thread does - fails with
EAGAIN on Emscripten, which took the whole process down through
`KillingErrorHandler`.
Known wart, upstream: after the runtime reaches `exit(0)` (verified with
`-sPTHREADS_DEBUG`), one Web Worker survives teardown and keeps the Node.js
process alive; the browser does not care.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Bring the packed archive tooling to public builds: the `packed-io`
clickhouse-disks subcommand and the standalone `clickhouse-packed-io`
program, which list, extract and create ClickHouse packed-format
archives (`data.packed`). The implementation (`PackedFilesOperations`,
`CommandPackedIO`, `programs/packed-io`) only depends on the already
public `PackedFilesReader`/`PackedFilesWriter`, so un-gate it from
`#if CLICKHOUSE_CLOUD`.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The `unit_tests_dbms` gtest binary did not define
`__lsan_default_suppressions`, so OpenSSL EVP method-store allocations
(cached in the provider method store and never freed at exit) were
reported as leaks. This surfaced through the `FunctionsStress` fuzzer,
which registers the `HMAC` function and calls `EVP_MD_fetch` during
registration.
Move the sanitizer default options and LSan suppressions into a shared
`base/base/sanitizer_options.h` so `programs/main.cpp`, the gtest binary,
and the clickhouse fuzzer reuse a single definition. Broaden the
suppressions to anchor on `evp_generic_fetch`, `EVP_MD_fetch`, and
`ossl_method_construct` so the whole fetch path is covered (including
`ossl_provider_set_operation_bit`, which the previous list missed).
CI report:
https://s3.amazonaws.com/clickhouse-test-reports/PRs/106096/69616e065ac177580a0b55b49d448450b117b6b0/unit_tests_asan_ubsan_function_prop_fuzzer/unit_tests_asan_ubsan_function_prop_fuzzer.log
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Without this carve-out, `clickhouse --help` matches the
`startsWith(argv[i], "-h")` rule below it and gets routed into
`clickhouse-client`, which treats anything starting with `-h` as a
`--host` specification. This produces confusing behavior: either a
"Host argument requires value" error or an unintended host parse of
the string `--help`.
Explicit early routing to `mainEntryHelp` ensures that when the sole
argument is `--help`, `-h`, or `-?` the dispatcher prints its help
listing all subcommands and exits with code 0.
Extend the `03823_clickhouse_help_return_codes` test to cover the
three short forms.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- `printHelp` now takes an `std::ostream &` parameter: explicit `help`
writes to stdout, while `printHelpOnError` (mistyped commands) writes
to stderr, preserving the error-output contract for scripts.
- Shell-quote binary path and user:group in the `--no-sudo` branch of
`clickhouse start` to prevent command injection via crafted CLI args.
- Make the keeper availability check in the help return codes test more
precise: match `clickhouse keeper [args]` exactly instead of bare
`keeper` which could false-match `keeper-converter` etc.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Resolve merge conflicts:
- `base/base/coverage.cpp`: take master's version which removed the old
sanitizer coverage guard/trace functions (replaced by LLVM profile-based
coverage)
- `src/Client/BuzzHouse/AST/SQLProtoStr.cpp`: take master's unified
`CONV_FN(SQLIdentifier, ident)` replacing the individual per-type
converter functions
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The condition on the "unknown file path" error path compared `main_func`
(type `MainFunc`, i.e. `int (*)(int, char **)`) against `printHelp`
(type `void ()`), which is an incompatible function-pointer comparison
and made the branch unreachable.
Compare against `printHelpOnError` instead, which has the correct type
and is the actual default value assigned to `main_func`.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The `__asan_default_options`, `__lsan_default_options`,
`__lsan_default_suppressions`, `__msan_default_options`,
`__tsan_default_options`, and `__ubsan_default_options` functions need
forward declarations to satisfy `-Wmissing-prototypes`.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add `static` to file-local functions and forward declarations
for cross-TU functions to satisfy `-Wmissing-prototypes`.
This ensures all functions have visible prototypes before their
definitions, improving code quality and catching accidental
linkage issues.
Reference: https://github.com/ClickHouse/ClickHouse/issues/76944
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Implements self-contained, per-test source coverage collection using LLVM's
profiling runtime APIs and the binary's own `__llvm_covfun`/`__llvm_covmap`
ELF sections. No external tools (llvm-cov, llvm-profdata) or build artifacts
are required at runtime.
**How it works:**
At server startup (lazily on first `SYSTEM SET COVERAGE TEST` call):
- `readLLVMCoverageMapping("/proc/self/exe")` parses `__llvm_covfun` and
`__llvm_covmap` ELF sections, building a map of `NameRef → (file, line_start, line_end)`.
- `__llvm_covmap` filename tables are matched to `__llvm_covfun` records via
`FilenamesRef = MD5Hash(raw_filename_blob)`, matching LLVM's own scheme.
- The inline coverage mapping per function is decoded (ULEB128 virtual file
mapping, expression table, region sub-arrays) to extract each function's
entry region file and line range.
At each `SYSTEM SET COVERAGE TEST 'name'` call:
- `getCurrentCoveredNameRefs()` iterates `__llvm_profile_data` records,
uses the `CounterPtr` relative offset to find the function's entry counter,
and returns `NameRef` values for all functions with a non-zero entry counter.
- The `CoverageFlushCallback` resolves each `NameRef` to `(file, line_start, line_end)`
via the pre-built map, deduplicates, and inserts a row into `system.coverage_log`.
- `__llvm_profile_reset_counters()` resets all counters for the next test.
**Validated locally with 100 stateless tests:**
- 100 rows in `system.coverage_log`, one per test
- Line ranges validated: 0 inverted ranges, min_line=5, max_line=10900, avg_span≈14
- Test discrimination: `SELECT 1` covers 0 files unique to it; `ALTER TABLE`
covers 1372 files not in SELECT 1; hot-path files (ProfileEvents.cpp,
MergeTreeBackgroundExecutor.cpp) appear in all 100 tests as expected
**New files:**
- `src/Common/LLVMCoverageMapping.h/cpp`: ELF+covmap parser
- `src/Common/CoverageCollection.h/cpp`: lazy map loading + DB insertion
**Changed:**
- `base/base/coverage.h/cpp`: `CoverageFlushCallback` uses `vector<uint64_t>`
(NameRefs) instead of function addresses; `getCurrentCoveredNameRefs()` replaces
`getCurrentCoveredFunctionAddresses()` (FunctionPointer is null in LLVM 21)
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Two issues remained after the previous fix:
1. The ASan stack-size warning block (4 lines) was only partially
filtered in `clickhouse_proc.py`. The last line:
"For details see https://github.com/google/sanitizers/issues/189"
was not excluded, leaving `sanitizer_hits` non-empty and triggering
a spurious "Unknown error / Lost connection to server" result for
jobs that had no real sanitizer errors. Add the missing exclusion.
2. OpenSSL's EVP method cache (`evp_md_new`, `construct_evp_method`,
`CRYPTO_THREAD_lock_new`) is never freed at process exit. These
objects are allocated when the S3 client initializes HMAC via
`AWS SDK -> HMAC_Init_ex -> EVP_MD_fetch`. LSan reports them as
leaks, causing the server to abort with SIGABRT (`halt_on_error=1`),
which `clickhouse-test` records as "Unknown error / Lost connection
to server". Add suppressions to `__lsan_default_suppressions`.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
`clickhouse local` (used for post-test log collection) was aborting at
exit under ASan+LSan with 45 KB of leaks from OpenSSL provider
initialization (`ossl_provider_new`, `OSSL_PROVIDER_try_load_ex`, etc.).
OpenSSL 3.x intentionally does not free all global state at exit, so
these are false positives.
Added `__lsan_default_suppressions` alongside the existing
`__lsan_default_options` in `programs/main.cpp` to suppress the five
known OpenSSL leak patterns. This fixes the "Unknown error / Lost
connection to server" failures in:
- Stateless tests (amd_asan_ubsan, distributed plan, parallel, 1/2)
- Stateless tests (amd_asan_ubsan, distributed plan, parallel, 2/2)
- Stateless tests (arm_asan_ubsan, azure, sequential)
CI report: https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=99657&sha=24b9445c3f9c8dfc420b55f1a2fb369ae17220be&name_0=PR
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Simplify jemalloc integration by replacing the three-mechanism approach
(`JEMALLOC_IS_MALLOC` + `JEMALLOC_OVERRIDE___LIBC_*` + linker `--wrap`)
with a cleaner two-step design:
1. Compile jemalloc with `je_` prefix so it only exports `je_malloc`,
`je_free`, etc. (matching what Darwin already does).
2. Define our own `malloc`/`free` in `malloc.cpp` that perform memory
tracking and delegate to `je_malloc`/`je_free`. Provide `__libc_*`
aliases on glibc to intercept internal calls.
Changes:
- Enable `JEMALLOC_PREFIX "je_"` in all non-Darwin `jemalloc_internal_defs.h.in`
- Undef `JEMALLOC_IS_MALLOC` and all `JEMALLOC_OVERRIDE___LIBC_*` defines
- Remove `JEMALLOC_OVERRIDE_MEMALIGN`/`VALLOC`/`PVALLOC` from `jemalloc_defs.h`
- Add `JEMALLOC_NO_RENAME` interface define so consumer code sees `je_*` names
- Prefix all jemalloc API calls (`mallctl` -> `je_mallctl`, etc.) in ClickHouse
- Rewrite `memory.h` to call `je_malloc`/`je_free` directly instead of `__real_*`
- Rewrite `malloc.cpp` with tracking wrappers calling `je_*` underneath
- Rewrite `AllocationInterceptors.h` to map `__real_*` -> `je_*` or `::*`
- Remove `__wrap_*` functions from `AllocationInterceptors.cpp`
- Remove `--wrap` linker flags from `CMakeLists.txt`
- Add jemalloc + clickhouse_common_io deps to `clickhouse_malloc` OBJECT library
- And also fix JEMALLOC_CPREFIX for Darwin, that was the only platform that uses JE_MALLOC_CONF (over MALLOC_CONF)
Co-Authored-By: Claude <noreply@anthropic.com>
- Add `clickhouse docker-init` C++ subcommand that replaces `entrypoint.sh`
for shell-free distroless environments (fork/exec, POSIX-only, no shell deps)
- Add `docker/server/Dockerfile.distroless` and `docker/keeper/Dockerfile.distroless`
as parallel build targets alongside existing ubuntu/alpine variants
- Add `--no-distroless` CI flag and fix package selector to use `.deb` for distroless
- Add structural smoke-test harness under `docker/tests/distroless/`
The distroless images use `gcr.io/distroless/cc-debian12:nonroot` as base,
expose the same ports as ubuntu, run as uid/gid 101, and include a debug
variant (`distroless-debug`) with a busybox shell for troubleshooting.
Commit 66aeef7f59 introduced a reference to `printHelpOnError` which
doesn't exist — the function is called `printHelp`. This caused a
build failure.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When running `clickhouse path/to/nonexistent.sql` or
`clickhouse local /tmp/aaa.rep`, the error messages were confusing:
the main binary showed the generic "Use one of the following commands"
help, and `clickhouse-local` showed "Positional option is not supported".
Now, if the argument looks like a file path (contains `/` or `.`) but
the file doesn't exist, a clear "no such file" error is shown instead.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit improves the help output for all ClickHouse applications:
- Add `clickhouse help` command that lists all available subcommands
- Return exit code 0 for all help messages (previously some returned 1)
- Output help messages to stdout instead of stderr
- Use "clickhouse <cmd>" format instead of argv[0] in help messages
- Add dynamic `app_name` detection for `clickhouse client` and `clickhouse local`
to show correct invocation name based on how the binary was called
- Add `--no-sudo` option for `clickhouse start` and `clickhouse restart`
for environments without sudo (e.g., Docker containers)
- Add detailed help headers and footers for client and local with usage examples
Closes#58244
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Previously it was only shipped as part of the client package, now it
will be bundled into the binary, and will be available as:
- clickhouse chdig
- clickhouse-chdig
- chdig
Note, that since now chdig will not be linked, we can remove all
quirks for this, that has been done in #79666
