A program that formats an AST hiding secrets without installing a finder would show them. Programs that don't register the engines now install `NoSecretArgumentsFinder` explicitly.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A document nested 4520 to 7999 levels deep, given to `ch_format_json`, ended in `RangeError:
Maximum call stack size exceeded` in a Web Worker in Chrome, despite the README's promise that
hostile input cannot stop the module. `Poco::JSON::Parser` recurses once per level, and each level
takes little of the stack `checkStackSize` watches but much of the engine's own stack, which the
check cannot see. `createFromJSON` admits 8 levels of JSON per AST level, 8000 here.
`readASTJSON` now rejects a document nested more than 2000 levels deep, next to its size check.
The deepest document a query produces - on a server, at `MAX_PARSER_DEPTH` - is nested 1974
levels, and this module can only format documents up to about 800 levels deep anyway, so no real
document is turned away. The shared pre-scan in `ASTFromJSON.cpp` is left as it is: the server's
stack is large.
`test.mjs` gains a section that drives the deepest SQL inputs and these documents in a
`node:worker_threads` worker whose V8 stack is limited to 0.75 MB, about what a Chrome worker has,
on a freshly compiled module each time. `parser-cases` runs under `node:wasi` on Node's main
thread, which has more stack than a browser worker, so without it the limit would go untested.
It also gains real queries of the depth the stack size is about: a sum of 30 columns, 25 nested
`if` and 8 nested derived tables must round-trip through `ch_format_json`. The README says what
the guarantee assumes of the engine, and that an instance that has thrown a `RangeError` must be
replaced.
In headless Chrome 154, in fresh workers, 990 inputs - 18 shapes of SQL up to and past the depth
limits, through `ch_parse` and `ch_format`, and nested JSON through `ch_format_json` - now all
return a result or an error.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`ch_parse` held the producer to `checkDepth` and `checkSize`, which count AST
nodes, while `ch_format_json` counts more than nodes: `readFieldFromObjectImpl`
counts every value of a structured `Field` as well, and an `Array`, `Tuple` or
`Map` literal of any width is a single `ASTLiteral`. `ASTSetQuery`,
`ASTRenameQuery`, `ASTSystemQuery`, `ASTViewTargets`, `ASTEnumDataType` and
`ASTBackupQuery` count entries of their own the same way. A `SELECT` list of
49900 `*` and one 200-element array literal is therefore 49905 nodes and a
1005138-byte document that passed the producer, and `createFromJSON` then
refused it at 50106 counted elements - the advertised
`ch_parse` -> `ch_format_json` round trip did not hold for it.
The limits of the consumer now live in one function, `readASTJSON`, and
`serializeBody` runs it on its own output instead of restating its rules, which
a second implementation of them would be free to drift from. A tree that does
not survive the round trip answers a null `ast` with the reason, the way one
with no JSON representation at all does. The cost is one more pass over a
document of at most 1 MiB, and the module is 34 bytes smaller, because the two
sides no longer bound the raw text separately.
`test.mjs` gains that case, and a positive round trip of `Array`, `Tuple` and
nested literals well inside the budget, so the read-back is pinned in both
directions: 53/53 in the full build, 52/52 with `-DENABLE_DCL=OFF`, 37/37 in
each build without formatting.
Addresses the review comment on `utils/wasm-parser/wasm_parser.cpp`.
Four findings from the review of the JSON API:
* `ParserDiagnostics` is caller-owned and can be reused across queries, so
`tryParseQuery` now clears everything it fills in - the `Expected` variants,
the rightmost parsed position, the highlights, and the error token - while
keeping the knobs the caller set (`enable_highlighting`, `literal_token_map`).
Without this, a caller that enables highlighting once and parses repeatedly
gets the previous query's diagnostics mixed into the next result.
* The lexer shortcut in `tryParseQuery` returned before `parser.parse` ever ran,
so a query with an invalid token - an unclosed string literal, the shape an
editor sees while the user is still typing - came back with no highlights at
all, contrary to what `ch_parse` documents. The shortcut now runs the parse
for its highlights when the caller asked for them, and still reports the
lexical error, which is the better message for that position.
* `ch_parse` and `ch_format_json` were the two halves of one round trip with
different limits: the parse side bounded only the SQL text and the depth,
while the format side additionally rejected raw JSON over `MAX_QUERY_SIZE`
and trees over `MAX_AST_ELEMENTS`. The producer now holds itself to the
limits of the consumer, so an `ast` this module reports is always one it can
read back; a tree past them is reported as a null `ast` with the reason.
* `ch_format_json` now re-measures the assembled tree with `checkDepth` and
`checkSize`, the second pass `formatQueryFromJSON` makes on the server,
because a `readJSON` override can materialize nodes after the counting pass
of `createFromJSON`.
`test.mjs` covers the unclosed-string highlights and the round-trip invariant:
whatever `ch_parse` reports as an `ast`, `ch_format_json` reads back.
The module has no consumers yet, so there is no compatibility to keep:
`ch_check` answered a strict subset of `ch_parse` (the same return
code, with the error message inside the JSON document instead of
alone), so it is gone rather than kept alongside. `ch_format` stays -
it is not a compatibility entry point but the only way to pretty-print
a query whose AST has no JSON form, such as `GRANT`.
Sizes re-measured (each configuration a few hundred bytes smaller);
the ceilings still hold at about 8% over. All four configurations pass
`ctest`.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`ch_parse` parses a query and always answers with a JSON document. For
a query that parses: the AST (the same JSON the SQL function
`parseQueryToJSON` produces) and the syntax highlighting ranges. For
one that does not: the error message, its position (byte offset, line,
column), the expected-token list, and the highlights of the prefix that
did parse, so an editor keeps coloring while the user types.
`ch_format_json` is the reverse half, in builds with formatting: it
takes AST JSON - from `ch_parse` here or `parseQueryToJSON` on a server
- and formats it back into SQL.
`tryParseQuery` gains an optional `ParserDiagnostics` out-parameter (a
caller-provided `Expected`, which can have highlighting enabled, plus
the token the error message points at), so one parse yields the
canonical message and the machine-readable details together.
The error boundary now recovers anything deriving from
`Poco::Exception`, not just `DB::Exception`: the document handed to
`ch_format_json` is arbitrary, and `Poco::JSON::Parser` and the
`readJSON` overrides report a malformed one by throwing Poco exception
types (their `catch` blocks are dead under `-fignore-exceptions`).
The walk over the Itanium `__si_class_type_info` chain in `__cxa_throw`
is what establishes the derivation without libc++abi.
AST JSON goes with `-DENABLE_FORMATTING`: `writeJSON`/`readJSON` for
every node plus `Poco::JSON` and `Poco::Dynamic::Var` cost about 715 KB
raw / 172 KB brotli, while the errors-and-highlights half of `ch_parse`
costs about 2 KB and is in every build. Measured sizes and ceilings
updated accordingly; all four configurations pass `test.mjs`.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`utils/wasm-parser/build.sh` compiled a hand-written list of sources with a hand-written
set of flags, piped the compiler's output through `grep error:`, and then linked whatever
objects happened to be there. On `master` today it produces nothing at all, in two
independent ways:
* `src/Core/Streaming/Settings.cpp` appeared the day before the build was merged and was
never added to the list, so the link fails on `DB::WatermarkSettings::clone`. The script
swallowed the compile errors that preceded it, which is how a source list drifts
unnoticed.
* `-mllvm -wasm-enable-sjlj` on the link line does not reach the LTO backend, so `setjmp`
and `longjmp` stay undefined. Passing it as `-Wl,-mllvm,...` does reach it - and then
produces a module that links cleanly and throws an uncaught `WebAssembly.Exception` out
of `ch_check` the first time a query reports its error by throwing. LLVM's WebAssembly
`setjmp` lowering is only correct on a single translation unit, not on LTO-merged
bitcode.
The build is now a CMake project of its own, cross-compiled with the toolchain file
wasi-sdk ships. CMake fails the build on a compile error rather than reporting it and
carrying on, tracks dependencies, and gives the two tests a home: `parser-cases` runs
`test.mjs` under `node`, and `parser-size` asserts a byte ceiling per configuration, since
this is what a browser downloads.
The `setjmp` boundary moves into `wasm_sjlj.c`, the one translation unit outside the LTO
unit, which is what makes the lowering correct again. It is C so that
`-fvirtual-function-elimination`, which assumes the LTO unit is the whole program, still
sees every virtual call. The frames in between are LTO'd and optimized exactly as before -
a WebAssembly exception unwinds through them on its own - so the module does not grow.
The source list also drops `Poco/Path.cpp`, which nothing in the closure needs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The module aborted on any query the parser rejects by throwing rather than by
returning `false` - `SELECT sum(x) OVER (ROWS BETWEEN UNBOUNDED FOLLOWING AND
CURRENT ROW)`, for one - because `-fignore-exceptions` leaves nothing to catch.
`__cxa_throw` now jumps back to a `setjmp` boundary armed by `ch_check` and
`ch_format`, which report the message as a parse failure; the module stays
usable afterwards, which the test now checks. Recovery is gated on the thrown
type being exactly `DB::Exception`; anything else still stops the module.
This costs about 570 bytes and `-lsetjmp`, against the 262 KB and the
exception-handling proposal that `-fwasm-exceptions` would cost. `longjmp` runs
no destructors, so such a query leaks what the parser allocated below the
boundary - documented at `__cxa_throw`.
Ported from `parser-standalone-build` (ebca121740, 905c264506, 6c4bdaeaee),
which https://github.com/ClickHouse/ClickHouse/pull/112067 supersedes; flagged in
https://github.com/ClickHouse/ClickHouse/pull/112067#issuecomment-5125769942
Access management - `CREATE USER`, `CREATE ROLE`, quotas, row policies, settings
profiles, masking policies, `GRANT`, `REVOKE`, `CHECK GRANT`, `SET ROLE`,
`EXECUTE AS`, `SHOW GRANTS`, `SHOW ACCESS`, `SHOW CREATE USER`,
`SHOW PRIVILEGES` - is 201779 bytes of the 1168636-byte WebAssembly parser, and a
SQL editor in a browser has little use for it. `build.sh --no-dcl` leaves it out,
which also brings the module below what it costs with formatting removed:
everything 1168636 362827 gzipped
--no-dcl 966857 296331
--no-formatting 926511 298006
--no-formatting --no-dcl 754152 242329
`CLICKHOUSE_PARSER_NO_DCL` takes these out of the dispatch in `ParserQuery.cpp`
and `ParserQueryWithOutput.cpp`, and the linker drops `src/Parsers/Access` and
`src/Access/Common` with them - verified with `llvm-nm`: no `ParserGrantQuery`,
no `AccessFlags`. `DEFINER` on a view still parses, being part of `CREATE`.
The alternatives are grouped into five functions rather than guarded one by one,
because each has to keep its place in the chain: `SET ROLE` is tried before
`SET`, which would otherwise read it as the shorthand for a `Bool` setting named
`ROLE`, and `SHOW CREATE USER` before `SHOW TABLES`.
The module gains a `ch_features` export saying which of these switches it was
built with, so one piece of JavaScript can drive more than one build; `test.mjs`
uses it to decide whether the DCL cases are expected to parse.
The README gains what each other family of statements costs, measured the same
way. Nothing else comes close - `CREATE TABLE` is 56533, `ALTER` 35978, `SYSTEM`
32086 - and they overlap too much to be worth an option each.
Turning an AST back into SQL is 242 KB of the 1.17 MB WebAssembly parser - a
fifth of it - and none of it can be dropped piecemeal. `formatImpl` is virtual,
every AST class overrides it, and the linker keeps all 116 implementations as
long as one call goes through that slot, so removing the call from the entry
point saves 48 bytes. `build.sh --no-formatting` removes the last call instead
and produces a 926 KB module that answers whether a query parses.
The switch is `CLICKHOUSE_PARSER_NO_FORMATTING`, and it is confined to the four
functions in `IAST.cpp` that call `formatImpl`, plus
`formatWithPossiblyHidingSensitiveData`, which becomes a placeholder. Nothing in
the 116 AST classes needs an `#ifdef`; they simply become unreachable.
That still leaves parsing itself, which reaches the formatter in a few places
that have to keep a fragment of the query as a string: the type in
`CAST(x, 'T')` and `x::T`, the column type inside `defaultValueOfTypeName` for
`EPHEMERAL`, and the settings in the `viewExplain` a `(EXPLAIN ... SELECT ...)`
subquery turns into. These now go through `astText`, which formats as before,
and in a build with no formatter uses the text the user wrote - captured with
the new `textBetween`.
The two are not interchangeable, which is why `astText` exists rather than the
source text being used everywhere: the formatted spelling is canonical and is
what ends up in table metadata, while the source text carries the user's line
breaks into it. `CREATE TABLE t (x UInt8, e Enum8\n (\n 'hello' = 1 ...`
would store the newlines and indentation inside the string literal, and
`SHOW CREATE TABLE` would print them back.
One behaviour change: the `EPHEMERAL` type text was produced by
`formatForLogging`, which hides secrets and applies the server's query masking
rules to it - a data type has no secrets, and the comment there said as much.
It now uses the same call as the other three sites.
`src/Parsers` has no `catch`, and a syntax error is reported by `tryParseQuery` returning null,
not by throwing. So in a standalone build an exception can only mean a bug or a resource limit,
and there is nothing to unwind to - the exception machinery is pure overhead.
The build now passes `-fignore-exceptions`, which compiles `throw`, `try` and `catch` but emits no
landing pads and no unwind tables. This needs no source changes at all - in particular no macro
over the `throw` keyword, which would have been undefined behaviour and would have mangled the
`throw()` exception specifications in Poco. What is left is a call to `__cxa_throw`, and defining
that in the WebAssembly runtime keeps libc++abi out of the bundle entirely: the exception object
is constructed, its message printed, and the module stops.
`CLICKHOUSE_PARSER_MINIMAL_BUILD` now has a job: `Common/StackTrace.h` is included by
`Common/Exception.h` and so by most of the tree, and it needs `<csetjmp>`, `<csignal>` and
`<ucontext.h>`, none of which exist without exception handling on WebAssembly. The parts that
need them are guarded out rather than making the header unavailable. Poco's `SignalHandler` is
guarded the same way, and `Thread_POSIX` stops referring to it.
Measured, `-Os`, stripped:
2278794 -> 2016304 bytes, 618187 -> 569362 gzipped
and the module no longer needs an engine implementing the exception-handling proposal - it now
loads on stock Node 20 with no flags, where before it required `--experimental-wasm-exnref`.
Separately, and because they are an obsolete C++ feature in a library we own: every `throw()`
exception specification in `base/poco` is now `noexcept`. There were nineteen, plus two written
`throw ()`. None of them were dynamic specifications, so this is a straight rename.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An experiment in how small `src/Parsers` gets on its own. The Web UI currently ships only
the lexer (`Lexer.cpp` builds to a 6 KB `.wasm` with no dependencies); a real parser would
give exact syntax errors with positions and query pretty-printing in the browser.
`utils/wasm-parser/build.sh` compiles 500 translation units with wasi-sdk and links a
2.7 MB module (789 KB gzipped) exporting a five-function C interface. `test.mjs` drives it
through `SELECT`, `CREATE TABLE`, window functions, recursive CTEs, `ALTER`, `INSERT`,
`SYSTEM`, `CREATE USER`, `GRANT`, and two deliberate syntax errors.
`-fwasm-exceptions` is required rather than optional: the parser reports errors by
throwing, and the wasi-sdk headers refuse `setjmp.h` without it.
`wasm_runtime.cpp` substitutes no-ops for the pieces a browser has no use for, each of
which would otherwise dominate the bundle. Stack traces are the sharpest case:
`Common/Exception.cpp` cannot compile against stock libc++ at all, because it
`static_assert`s on `STD_EXCEPTION_HAS_STACK_TRACE` from ClickHouse's patched libc++.
`Core/Settings.cpp` is left out because `ParserSetQuery` calls `Settings::castValueUtil`
once - only to ask whether a bare `SET x` names a `Bool` setting - and that single call
pulls in every `SettingField*Traits` specialization, more code than the parser itself.
The timezone database that `contrib/cctz-cmake` generates into the binary is left out
too, as is query masking.
`shim/` holds the POSIX headers wasi-libc omits (`netdb.h`, `ucontext.h`, `sched.h`,
`pwd.h`, `fenv.h` rounding modes, `sun_path`, `__u6_addr`). Nothing calls into them; they
exist so headers that name those types in signatures still compile.
The README records where the size goes. The largest remaining cut is access-entity DDL:
`CREATE USER`/`ROLE`/`QUOTA`/`ROW POLICY` and `GRANT` account for 334 KB of parser and
`Access` code, roughly a third of the bundle once the re2 and abseil they share with
`COLUMNS('regexp')` matchers is counted.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
