When the unwinder returns no frames, jemalloc still records the sampled
allocation and dumps it as a bare `@` line followed by its `t*:`, `tN:` and
`f:` records. Two readers of the dump mishandled it:
- `system.jemalloc_sampled_allocations` used an empty frame list as "no
backtrace header seen yet", so the first `f:` record under a bare `@` threw
`CANNOT_PARSE_TEXT` ("Allocation record without a preceding backtrace") and
the whole table was unreadable while such an allocation was alive. This is
the `05238_jemalloc_tables_system_jemalloc_grant` failure on
`Stateless tests (arm_binary, parallel)`:
https://s3.amazonaws.com/clickhouse-test-reports/praktika.html?PR=123624&sha=43f27878f78f7744848c3c2f7c7eca658a19a2d4&name_0=PRhttps://s3.amazonaws.com/clickhouse-test-reports/praktika.html?PR=123514&sha=82eabe3927c9dae9dd9e64e1a0d5949b27c545a4&name_0=PR&name_1=Stateless%20tests%20%28arm_binary%2C%20parallel%29
- The collapsed (FlameGraph) output of `system.jemalloc_profile_text`, the
default format and the one the Keeper jemalloc handler serves, consumed the
counters only after a non-empty stack, so the live bytes of such samples
were silently left out of the flame graph.
Track the header separately from its frames in both readers: the allocation
is returned with an empty `trace` (an `f:` record before any `@` line still
throws), and the collapsed output reports it under an `[unknown]` frame. The
fragmentation flame graph of the jemalloc web UI, which builds its stacks from
that `trace`, labels an empty one `[unknown]` too instead of drawing a blank
frame, and so does the same example in the allocation profiling docs. The
`trace` column description now says that it can be empty.
`05023_jemalloc_sampled_allocations` no longer asserts that no row has an
empty `trace`.
`JemallocProfileSource.CollapsedEmptyStack` feeds a bare `@` block to the
collapsed reader.
Related: https://github.com/ClickHouse/ClickHouse/pull/123624#issuecomment-5974953202
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The query may fail with:
Code: 386. DB::Exception: There is no supertype for types UInt64, Float64 because some of them are integers and some are floating point, but there is no floating point type, that can exactly represent all required integers: In scope SELECT format('{} {}', if(s.est_old_objects > 0, s.stack, format('[unattributed];arena_{}_class_{}', toString(b.arena), toString(b.size))), toString(toUInt64(if(s.est_old_objects > 0, (b.waste * s.est_old_objects) / sum(s.est_old_objects) OVER (PARTITION BY b.arena, b.index), b.waste)))) FROM system.jemalloc_arena_bins AS b LEFT JOIN (SELECT arena, size_class, sum(weight) AS est_old_objects, arrayStringConcat(arrayMap(a -> demangle(addressToSymbol(a)), arrayReverse(trace)), ';') AS stack FROM system.jemalloc_sampled_allocations WHERE age_ns > 60000000000 GROUP BY arena, size_class, trace) AS s ON (s.size_class = b.index) AND (s.arena = b.arena) WHERE (b.large = 0) AND (b.waste >= 1048576) AND (purpose = '') SETTINGS allow_introspection_functions = 1. (NO_COMMON_TYPE) (version 26.9.1.1)
v2: Apply Min class waste filter to the fragmentation flamegraph query
v3: Fragmentation tab: waste breakdown, [unattributed] flame graph rows, sortable tables
v4: Estimate profiler sampling overhead in the fragmentation breakdown
v5: Fragmentation tab: account cache_oblivious one-page padding of large allocations
v6: Build the fragmentation flame graph from a single sampled-allocations scan
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Route the four `innerHTML` error-message renderers in `jemalloc.html`
through the existing `escapeHtml` helper. The interpolated values
(`error.message`, user-typed `queryId`) are trust-anchored so this is
not a security boundary, but escaping is hygienic and matches the rest
of the file.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The "user" input on `play.html` and on the other web pages
(`dashboard.html`, `jemalloc.html`, `merges.html`, `binary.html`,
`webterminal.html`) previously came pre-filled with the literal text
`default`. Replace that with an empty value and a `user` placeholder
hint. When the value is empty, the page does not send a `user=` query
parameter (or, in the WebTerminal protocol, omits the `user` field
from the auth JSON), letting the server fall back to its configured
default user — which is not necessarily named `default`.
For the WebTerminal, `parseAuthMessage` now treats the `user` field as
optional: when omitted or empty, the caller-provided default is
preserved. The call site seeds it with `default`, mirroring the HTTP
fallback in `authenticateUserByHTTP`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Add `OBJECT_DEPENDS` for `KeeperJemallocHandler.cpp` in
`src/CMakeLists.txt` so that changes to the `#embed`-ed
`jemalloc.html` trigger recompilation, matching the existing pattern
for other handlers.
- Replace the 130-line `heapProfileToCollapsedStacks` in
`Common/Jemalloc.cpp` with a call to the existing
`JemallocProfileSource` collapsed-stack pipeline via the new
`symbolizeJemallocHeapProfileToString` helper. This eliminates
duplicated heap-dump parsing and address symbolization logic.
- Remove the unused `instanceName` JavaScript variable from
`jemalloc.html` (assigned but never referenced in any template).
https: //github.com/ClickHouse/ClickHouse/pull/100606
Made-with: Cursor
- Fix `attachStrictPath` not matching URIs with query parameters: replace
with a custom filter that matches the path portion, ignoring `?...`
suffixes. This caused `/jemalloc/profile?format=bad` to 404 in
non-jemalloc builds instead of returning 501.
- Fix HEAD test assertions: HEAD responses carry no body, so checking
`response.text` for "not available" fails on 501. Only assert the
status code for HEAD.
- Add HEAD short-circuit to `KeeperJemallocStatsHandler` to avoid the
expensive `je_malloc_stats_print` call when no body is needed.
- Handle `status.errors` in the web UI's `checkProfilingEnabled`: when
the `/jemalloc/status` response includes errors, treat profiling state
as unknown rather than silently defaulting to disabled.
Made-with: Cursor
- `handleStatus` no longer silently swallows `mallctl` exceptions.
Failures are logged via `tryLogCurrentException` and surfaced in the
JSON response as an `errors` array so operators can distinguish
"profiling disabled" from "status unknown".
- Remove triple-duplicated "Substitute user name" comment in
`jemalloc.html`.
Made-with: Cursor
Expose jemalloc heap profiling and statistics through Keeper's HTTP
control interface, matching the functionality already available in the
ClickHouse Server web UI.
New REST API endpoints on the Keeper HTTP control port:
- GET /jemalloc — serves the interactive web UI (shared with server via
`isKeeper` mode flag injected at serve time)
- GET /jemalloc/stats — returns raw `malloc_stats_print` output
- GET /jemalloc/status — returns profiling state as JSON
- GET /jemalloc/profile?format={collapsed|raw} — flushes a heap profile,
symbolizes it, and returns collapsed stacks for flame graph rendering
The web UI reuses the server's `jemalloc.html` with a thin `isKeeper`
abstraction that swaps SQL-based data fetching for REST API calls and
hides server-specific features (Query Profiler, connection params, output
format options). This avoids duplicating ~3000 lines of HTML/CSS/JS.
New shared helpers in `Jemalloc::heapProfileToCollapsedStacks` (parses
raw jemalloc heap dump, symbolizes addresses via `StackTrace::forEachFrame`,
aggregates into FlameGraph-compatible collapsed stacks) and
`Jemalloc::getStats` (captures `malloc_stats_print` output as a string).
Closes https://github.com/ClickHouse/ClickHouse/issues/100544
Made-with: Cursor
Minimize memory usage when querying `system.jemalloc_profile_text` in
collapsed format by streaming results in small blocks instead of
buffering up to 65K rows.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove duplicate `const queryId` declaration inside the `try` block of
`getQueryCollapsedProfile` that shadowed the outer one, causing a
"Cannot access 'queryId' before initialization" ReferenceError every
time the function was called.
Rename the function from `getQueryFlameGraphProfile` to
`getQueryCollapsedProfile` to better reflect what it does: it fetches
collapsed stack data and renders it as text with action buttons,
rather than generating the flame graph itself.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Change the Query Profiler's Flame Graph section to show the collapsed
profile data in a text box (matching the Global Profiler output style)
instead of directly generating and downloading an SVG. The button is
now "Get Profile" and the output card includes Copy All, Save,
Generate Flame Graph, and Close buttons.
Extract shared helper functions parameterized by result div ID to
deduplicate profile output code between Global and Query Profiler:
`getFullProfileText`, `copyProfileOutput`, `saveProfileOutput`,
`closeProfileOutput`, `showMoreProfileLines`,
`downloadFlameGraphFromProfile`, and `renderProfileOutputCard`.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add `SETTINGS allow_introspection_functions = 1` to the `flameGraph`
query so it works without requiring the setting globally.
- Handle `SYSTEM FLUSH LOGS` failure gracefully: show a warning instead
of aborting, since the user may lack privileges (e.g. readonly mode).
- Preserve the flush warning when no trace log data is found, so the
user knows the data may appear after a delay.
- Fix empty result detection: `min`/`max` return default values (0)
on empty sets, not empty strings.
- Darken light theme `--warning-color` for better readability.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Query `jemalloc.prof.thread_active_init` from `system.asynchronous_metrics`
instead of the config value from `system.server_settings` to reflect
whether profiling is truly active at the jemalloc level.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add URL, user, and password input fields to the header, matching
the pattern used in `dashboard.html`
- Extract `buildUrl` helper and shared `fetchOptions` to deduplicate
URL building across all fetch call sites
- Add `add_http_cors_header` and `enable_http_compression` to requests
- Add `Authorization: never` header to suppress browser auth popups
- Change profiling check to query `system.server_settings` for
`jemalloc_enable_global_profiler` instead of `system.asynchronous_metrics`
- Wrap initialization in `start()` function for compatibility with
dataplanectl credential injection
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Rename `system.jemalloc_profile` to `system.jemalloc_profile_text` and
the corresponding settings (`jemalloc_profile_output_format`,
`jemalloc_profile_symbolize_with_inline`, `jemalloc_profile_collapsed_use_count`)
to their `jemalloc_profile_text_*` equivalents, following the server-side rename.
Also change the default auto-refresh interval from 10s to 60s.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
