* Restrict Kafka client params that execute commands or load libraries
Kafka source configs can be submitted through the REST API, so they should not be able to make librdkafka run shell commands or load native libraries. Kerberos ticket refresh is now opt-in through the operator-set `QW_KAFKA_KINIT_CMD` environment variable.
* Bump tantivy to e1acb3e
* Bump version to 0.9.2
* Add security note to REST API docs
* Treat an empty QW_KAFKA_KINIT_CMD as unset
An empty or whitespace-only value now behaves like an unset variable and keeps Kerberos ticket refresh disabled.
MCP negotiates its protocol revision in-band (initialize and the
MCP-Protocol-Version header), so nesting it under the versioned REST API
adds no value. /mcp matches the convention used by MCP clients and servers.
Add bounded error labels for reader creation, warmup, Tantivy search,
and panic failures. Expose these categories in documentation and Grafana.
Co-authored-by: Pascal Seitz <pascal.seitz@gmail.com>
* feat(es-compat): add index_filter support for field capabilities API
Implements index_filter parameter support for the ES-compatible
_field_caps endpoint, allowing users to filter field capabilities
based on document queries.
Changes:
- Add query_ast field to ListFieldsRequest and LeafListFieldsRequest protos
- Parse index_filter from ES Query DSL and convert to QueryAst
- Pass query_ast through to leaf nodes for future filtering support
- Add unit tests for index_filter parsing
- Add REST API integration tests
Note: This implementation accepts and parses the index_filter parameter
for API compatibility. Full split-level document filtering will be
added as a follow-up enhancement.
Closes#5693
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: ruo <ruoliu.dev@gmail.com>
* feat(es-compat): implement split-level filtering for field_caps index_filter
Address PR review comments for index_filter support in _field_caps API:
- Extract `parse_index_filter_to_query_ast()` function with clean prototype
- Implement split-level filtering via `split_matches_query()` using
lightweight `query.count()` execution (no document materialization)
- Add proper async handling with ByteRangeCache, warmup(), and
run_cpu_intensive() for Quickwit's async-only storage
- Add metastore-level pruning:
- Tag extraction via `extract_tags_from_query()`
- Time range extraction via `refine_start_end_timestamp_from_ast()`
- Build DocMapper only when query_ast is provided (no overhead for
common path without index_filter)
- Fix REST API tests: use `json:` key (not `json_body:`), use lowercase
term values to match tokenizer behavior
- Update tests to run against both quickwit and elasticsearch engines
Two-level filtering now implemented:
1. Metastore level: tags + time range from query AST
2. Split level: lightweight query execution for accurate filtering
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: ruo <ruoliu.dev@gmail.com>
* refactor(es-compat): use best-effort metadata filtering for index_filter
Remove heavy split-level query execution for field_caps index_filter.
The implementation now aligns with ES's "best-effort" approach that uses
metadata-level filtering only (time range, tags) instead of opening
splits and executing queries.
Changes:
- Remove split_matches_query function (no longer opens splits)
- Remove query_ast and doc_mapper from LeafListFieldsRequest proto
- Keep metadata-level filtering in root_list_fields:
- Time range extraction from query AST
- Tag-based split pruning
- Simplify leaf_list_fields to just return fields from all splits
This matches ES semantics: "filtering is done on a best-effort basis...
this API may return an index even if the provided filter matches no
document."
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: ruo <ruoliu.dev@gmail.com>
* fix(es-compat): reject empty index_filter to match ES behavior
- Remove empty object {} handling in parse_index_filter_to_query_ast
- ES rejects empty index_filter with 400, now QW does too
- Add tag_fields config and tag-based index_filter test
- Update unit and integration tests accordingly
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: ruo <ruoliu.dev@gmail.com>
* Added ref doc for the new functionality.
* cargo fmt fix
---------
Signed-off-by: ruo <ruoliu.dev@gmail.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: fulmicoton <paul.masurel@datadoghq.com>
* Add failing unit test
* Fix index update endpoint panic when index_uri is the bucket root
* Simplify update endpoint to use same default URI as create
* Better error messages in integ tests
* Initial fix suggestion from Trinity
* Add rest api test and clarify docs about leniency
* Add missing field test on wildcard query
* Fix add query building unit tests
* Forgotten staging file
* Allowing failed splits in root search.
This PR adds a query string parameter (`allow_partial_search_results`) in
the elasticsearch api controlling encounterring an error on a subset
of the splits, should fail a search query or if a response
(reflecting the partial set of successful splits) should be returned.
It changes the behavior of the root search gRPC call, that will not return
an error in presence of partial split errors, but instead returns the list of splits that failed.
This PR also changes the default behavior of the elasticsearch rest API: following elasticsearch behavior on failing shards, quickwit will allow split errors.
The Quickwit API default behavior, on the other hand, is unchanged.
Closes#5411
* Apply suggestions from code review
Co-authored-by: trinity-1686a <trinity@quickwit.io>
* Changing the moment when aggregation fast fields are checked.
The logic is now aligned with what is done for queries:
we check the schema at the root level.
If the aggregation does not make sense for the current
index docmapping, the query is rejected.
---------
Co-authored-by: trinity-1686a <trinity@quickwit.io>
* Add ES Resolve index API handler
This [method][1] is used by Opensearch Dashboards when user typing index
name in pattern creation UI.
OpenSearch Dashboard makes first request with `*:*` pattern which
should be converted to `*` as QuickWit do not support `:` in the index
name. This part is not implemented within this PR.
[1]: https://www.elastic.co/guide/en/elasticsearch/reference/7.17/indices-resolve-index-api.html
* Rename variables according to type
Seems left over from prev development as are we
[passing](quickwit/quickwit-serve/src/elasticsearch_api/mod.rs:81)
`metastore` service.
* Add `_cat` ES API documentation
Took params from `cat_indices.rs` and followed common structure of the
page.
* Added rest api integration test.
---------
Co-authored-by: Paul Masurel <paul@quickwit.io>
David Yaffe