mirror of
https://github.com/qdrant/qdrant.git
synced 2026-10-29 00:54:54 +08:00
dev
707
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Andrey VasnetsovandClaude Opus 5.5
|
64b5ae0ea3 |
Default to 2 shards per node in 3+ node clusters with few collections (#11255)
* Default to 2 shards per node while collections are fewer than nodes Auto-sharded collections created in a cluster without an explicit shard number get 2 shards per node while the cluster holds fewer collections than nodes, and 1 per node otherwise. Spare shards let new nodes take whole shards instead of resharding, which re-indexes the data. Clusters with many collections keep the previous shard count. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Don't document the default shard number logic in the API Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Keep one shard per node in clusters of fewer than 3 nodes Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Rewrap default_shard_number_per_node doc Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
|
150b787fbf |
feat(text-index): add code tokenizer (#11180)
* feat(text-index): add `code` tokenizer Splits source code into identifiers and their subwords, following Lucene's WordDelimiterGraphFilter with catenate_all: snake_case, camelCase, PascalCase, plus an acronym rule (HTTPServer -> http, server) with a plural exception (getURLs -> get, urls). Digits stay attached to letters (utf8, k8s), and words that start with a digit are number literals and are not split. Documents index subwords in text order, then each multi-part identifier joined (getUserName -> getusername) at the end of the value, so phrase matching over subwords is unaffected. Queries are split into subwords only. Also adds an ASCII fast path to TokensProcessor lowercasing (no allocation for already-lowercase tokens) and a tokenizer throughput bench. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LupJerJ3WBAhW5RK8kvdtt * fix(text-index): keep code tokenizer's joined identifiers out of phrases Phrase queries were tokenized as documents, so the code tokenizer appended joined identifiers to them: `getUserName(id)` became `get user name id getusername`, which no document has contiguously, and existing phrases stopped matching. In documents, joined identifiers sat next to each other and to the last subword, so phrases such as `hashmap getusername` matched text that does not contain them. - Phrase queries use a new `TokenizerTextKind::Phrase`: subwords only for the code tokenizer, unchanged for the others. - Joined identifiers are detached tokens (`Tokenizer::tokenize_doc_detached`). With phrase matching on, the index puts a sentinel before each of them, so they only match a phrase of themselves alone; the payload check does the same with a token id no query has. - Document length discounts the inserted sentinels by count, so it is the same with phrase matching on and off. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LupJerJ3WBAhW5RK8kvdtt * feat(text-index): join only camelCase identifiers in the code tokenizer A joined identifier helps when someone types an identifier as one lowercase word, as people do with camelCase (`hashmap`, `getusername`). snake_case is typed with its `_`, which the query splits into subwords anyway, so its joined form only grew the vocabulary. Join each camelCase run between `_` separators instead of the whole identifier (`test_getUserName` -> `getusername`). Without lowercasing, a query word with case changes is split, so it can never match a joined identifier: emit none. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LupJerJ3WBAhW5RK8kvdtt --------- Co-authored-by: Andrey Vasnetsov <andrey@qdrant.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Andrey VasnetsovandClaude Opus 5.5
|
e61cc8605c |
Enable transfer_raw_points and transfer_raw_payloads by default (#11253)
Shard transfers now send points as storage-native bytes, with payloads as their stored blobs, for every collection. Both are read on the sending side only. Every 1.19 release already accepts raw points and raw payloads, so a rolling upgrade from 1.19 to 1.20 can mix versions without breaking transfers. The `all` preset turns them on as well. Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
Andrey VasnetsovandClaude Opus 5.5
|
0441071b38 |
Add CLAUDE.md with repo guidance for coding agents (#11204)
* Add CLAUDE.md with repo guidance for coding agents
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* CLAUDE.md: match AI disclosure rule to CONTRIBUTING.md
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* Revert "CLAUDE.md: match AI disclosure rule to CONTRIBUTING.md"
This reverts commit
|
||
|
|
0cb222e367 |
[BM25] Score without positions from a term frequency file (#11165)
* [BM25] Score without positions from a term frequency file A scoring text index no longer requires phrase_matching. Without it, the index keeps an ids-only postings.dat, unchanged, and a new term_frequencies.dat: each posting list's frequencies in posting order, bit-packed per 128-entry block aligned with the postings chunks, at the width of the block's largest value, so every frequency is exact. A frequency is stored minus one, so a block of ones takes no bytes. With phrase_matching, tf still comes from the positions. - the mutable index counts tf from the token stream and keeps the ordered document only for phrase matching; its stored record keeps repeated tokens when lengths are recorded - the on-disk index reads the file header at open and only the query terms' headers and data per query, in two batched reads - a file that does not match postings.dat, or is malformed, is ignored: the index rebuilds, a read-only one reports the field absent - normalized() no longer forces phrase_matching On the bm25_corpus fixture the text index takes 0.28x (20k documents) and 0.18x (100k documents) of the positional one on disk. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Sync scoring docs with the counts-only term frequency file Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Reject a malformed term frequency remainder at parse A missing width byte, a width above 32 or too few remainder bits made scoring panic or read garbage frequencies. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Check term frequencies survive a gridstore reload, drop stale scoring docs - doc_len_survives_gridstore_reload also asserts tf after reopen - block length check moves from a const assert to a unit test - docs and error message no longer say scoring needs or implies positions Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Score from the term frequency file with or without positions Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Check term frequencies at query time in debug builds only Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Fill postings at builder add, keep scoring documents as written Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Open the term frequency file with the postings' placement Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Drop the no-op TextIndexParams::normalized Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Write and remove text index sidecars through the index fs Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: generall <andrey@vasnetsov.com> |
||
|
Andrey VasnetsovandClaude Opus 5.5
|
5c5ff38461 |
Nested payload index: index the null index per element too (#11207)
The null index of a `nested` field stayed an index of points, so it was the one index of the field not keyed by elements. It is now wrapped like the others: built and updated from each element, with its own element map, and opened over element offsets. `is_empty` on such a field matches a point when one of its elements lacks the field, and never a point without elements, like a nested filter over the payload. `is_null` already matched when any value is null. Opening an index takes the number of ids it is keyed by instead of the id tracker, so a per-element null index is sized by its elements. Its cardinality estimate is over all elements, not only those with values. Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
|
69e76786d8 |
Per-element (nested) payload indexes: API and index wrapper (#11126)
* Add per-element (nested) payload indexes New `nested` parameter of the create-index request: index every object of an array as a separate entry, e.g. `offers[].price` with `nested: offers`. The per-element index wraps an unchanged field index keyed by element offsets from an element map, shared by all per-element indexes of the same array. Every read translates elements back to points, so the index behaves exactly like the flattened index of the field. Nested filter processing is not changed yet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BbgxnY8sJkEr7N9HG6Azkn * Nested payload index: clippy fixes, transition and rebuild tests Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BbgxnY8sJkEr7N9HG6Azkn * Nested payload index: support full-text indexes A full-text condition matches a point when a single element matches it, and BM25 scores every element as its own document: corpus statistics are gathered over elements, and a point scores as its best element. Bm25Accept gains an element mode, so the outer filter is checked on the owning point while the index scores element ids. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Nested payload index: one element map per index An element is identified by its point and its position among the objects of the array, which depend only on the current payload. Element offsets are private to each index, so per-element indexes no longer share a map per array. Every per-element index owns its element map, stored next to its inner index and opened, built, flushed and wiped with it. Immutable indexes get a map of per-point start offsets, written once at build; mutable ones keep the Blobstore-backed map. The map's mutability follows the stored index type. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Destructure create-index operations and index types So a field added later can't be silently ignored: do_create_index and internal_create_index destructure their operation, and the per-element index build and open destructure FullPayloadIndexType. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Nested payload index: `nested` is a parameter of the index params `nested` was a top-level create-index parameter stored inside the params, so every API boundary split it out and merged it back. It is now an ordinary parameter of every index type's params, in REST and in each gRPC `*IndexParams` message: "field_schema": { "type": "integer", "nested": "offers" } The create-index request validates it against the field name and gates it on the peer version; normalization stores it as the array key. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Nested payload index: exhaustive matches instead of matches! So a new index variant or storage type has to be decided on explicitly. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Nested payload index: bit-packed starts file for immutable element maps The start offsets of an immutable element map are stored with BitPacker4x in blocks of 128: the deltas are the element counts of the points, packed with the bit width of the largest one in the block. With up to three elements per point that is about 2 bits per point instead of 4 bytes, so opening the map reads a fraction of the array it builds. The file starts with a 32-byte header: magic, format version, the number of starts, and reserved bytes for later fields. A file with another magic or version is invalid, which rebuilds the field's per-element indexes. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Nested payload index: stream order-by in runs of equal values Order-by on a per-element index collected and sorted the whole range. The inner index already streams pairs sorted by value, so only the points within one value need sorting: PointRuns buffers one run of equal values at a time, maps its elements to points, sorts and deduplicates them, and yields. Each end of the double-ended iterator takes whole runs from what neither end has taken, so a value never ends up in both. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Nested payload index: open immutable indexes without reading them Opening a per-element index counted its indexed points by checking every element of the inner index, which read a cold index from disk in full. The build pass now counts points with a value an index may store, and an immutable element map keeps the count in its header. Opening takes the deleted points with elements off it: approximate, since such a point is taken off even if it had no value. Mutable maps keep counting, their indexes are in RAM. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Nested payload index: facets reuse one buffer for the points of a value Facets collected the points of every value into a new vector before sorting and deduplicating them. ElementPoints keeps one buffer for all values of a facet call. Counting uses the same sorted buffer: in an immutable map the points already come sorted, so the sort is close to linear. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Nested payload index: document why ElementPoints keeps its buffer Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Nested payload index: per-element text semantics apply to every query Document on the text index's `nested` parameter that every query on the field treats each element as its own document, not only nested filters. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Nested payload index: assert points come in order when building An immutable element map needs payload storage to visit points in ascending order. The payload storage contract now states it, the build pass of a per-element index asserts it in debug builds, and the test-only in-memory payload storage iterates in point order. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Nested payload index: reuse freed offsets in mutable element maps A mutable element map gave every growing point a fresh contiguous range at the end and never reused offsets. A point that is cleared and refilled, or that grows and shrinks, leaked a range each time: the map, the per-id arrays of the mutable inner indexes, and the deleted mask grew without bound, and the u32 offsets could run out, failing writes. Every point now keeps the list of its element offsets in array order, and freed offsets are given out before new ones. The free list is rebuilt at open. A reused offset is safe: update_point writes every element of the point, and an unowned offset is never resolved. Element map tests move to their own file, with a randomized test that no offset is ever shared between points. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Nested payload index: keep build-time counts on reopen * Nested payload index: use Logstore in serverless mode * Nested payload index: encapsulate payload input paths and builders * Nested payload index: clarify element offset annotations * Nested payload index: compress immutable mappings with Zstd * Blobstore: iterate values in ascending point order Async read pipelines (io_uring) complete reads out of order, which broke the ascending-order contract of PayloadStorageRead::iter that immutable nested indexes rely on. Buffer each iteration batch and pass it to the callback sorted by point offset. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Nested payload index: size payload blocks in points Inner blocks count elements, while the flattened index of a map or bool field counts each value once per point. Estimate the block cardinality in points and skip blocks that fall below the threshold, so HNSW payload-block building matches the flattened index more closely. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Nested payload index: estimate point cardinality like flattened indexes Linear scaling by all elements underestimated: elements without the field diluted the ratio, and it assumed matches cluster into as few points as possible. Use estimate_multi_value_selection_cardinality over elements with values instead, bounding points by that count so the estimator's assertion holds for immutable indexes after deletions. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Nested payload index: count points with values of the indexed type The immutable builder counted indexed points with a type-agnostic check, so a string under an integer nested index counted in immutable segments but not in mutable ones or the flattened index. Make add_point of field index builders and ValueIndexer report whether the payload held a value of the index's type, and count from that. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Andrey Vasnetsov <andrey@qdrant.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Lenin Mookiahandqdrant-cloud-bot
|
ea15064861 |
docs: clarify zero HNSW m and payload links (#10744)
Co-authored-by: qdrant-cloud-bot <111755117+qdrant-cloud-bot@users.noreply.github.com> |
||
|
Andrey VasnetsovandClaude Opus 5.5
|
4505b13c9d |
[BM25] Gather text statistics over the idf corpus (#11081)
* [BM25] Gather text statistics over the idf corpus A text query with a corpus-scoped `idf` search param was refused: text statistics always covered every point of the shard. The corpus now scopes them, as it does for sparse vectors with the IDF modifier. Each segment resolves the corpus filter once, and counts `N`, `df` and the token total behind `avgdl` over the corpus points that hold a token. The corpus does not filter the ranked points. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * [BM25] Leave the idf corpus out of the text rate cost Sparse search does not price its corpus filter either; pricing it is a separate change for both. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * [BM25] Probe small corpora into long postings Scoring prunes common terms once its top results fill, so walking their whole posting lists for the corpus frequencies could cost more than the query. A term whose posting list has more chunks than the corpus has documents now has the documents probed into it instead: a probe decompresses at most one chunk, a walk every chunk. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
|
fd1b6da31e |
Nest a text query's BM25 parameters per scorer (#11012)
* Nest a text query's BM25 parameters per scorer
A text query carried BM25's k and b as top-level fields, so a second
scoring type would have had to add its own parameters next to them, valid
for some fields and silently meaningless for others. The parameters now sit
in a oneof keyed by scorer: TextQuery { query, oneof params { Bm25Params
bm25 } }, and REST takes { "query": ..., "bm25": { "k", "b" } }. Without
parameters, the field's scorer runs with its defaults.
The collection checks the parameter variant against the field's scoring
type before any shard runs, with an exhaustive match on both enums, so a new
scoring type or parameter variant must say which pairs match. Shards still
receive resolved k1/b in TextScoringInternal, so the internal protocol is
unchanged. TextQuery is not released yet, so the shape changes in place;
fields 2 and 3 are reserved so the old k and b numbers are never reused
(a dev-build client still sending them gets the defaults).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* Refuse unknown text query fields, match the no-parameters case
Address review: a text query with parameters outside `bm25` (the former
shape) or a misspelled one inside it was accepted and scored with the
defaults. TextQueryInput and Bm25Params now deny unknown fields, so such a
body is a format error (400).
The scoring-type check matched a query without parameters with a wildcard,
so a new scoring type would have taken BM25 defaults without a compile
error. It is now matched per scoring type like the other pairs.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* Accept unknown text query fields, like the rest of the Query API
Revert the deny_unknown_fields part of the previous commit: no Query API
input struct refuses unknown fields, and inside the untagged Query enum a
refusal surfaces as the generic "Expected some form of vector, id, or a type
of query" error. The former top-level k and b shape it guarded against was
never released.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* Key a text query's scorer parameters under `scoring`
REST had `bm25` directly on the text query, so a second scorer would have
added a sibling field, and both could be set at once. The parameters now sit
under `scoring`, keyed by the scorer's name, matching the index config's
`scoring` and leaving the top level free for parameters shared by all
scorers. The gRPC oneof is renamed to `scoring` too; its field number is
unchanged.
`TextQueryScoring` is an untagged enum of named structs, like `Query`, so
the schema has no anonymous objects.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: generall <andrey@vasnetsov.com>
|
||
|
|
24f67eb18e |
Add the Turbo16 vector datatype (#11024)
* Add the Turbo16 vector datatype TurboQuant at 16 bits per element: the shared Hadamard rotation, then a per-vector absmax scale onto the uniform grid [-32767, 32767], stored as little-endian i16. Like Turbo8 there is no codebook: the code is the value, and the per-vector scale cancels in the existing `l2 / ||c||` renormalization. A vector takes `2 * dim + 4` bytes (`+ 8` for Euclid/Manhattan) - the size of Float16 plus the extras - with ~7x lower reconstruction error than Float16. - quantization: `TQBits::Bits16` (Normal mode only, no TQ+). New `simd/query16bit` with one set of kernels for both scoring paths: `madd_epi16` on SSE4.1 / AVX2 / AVX-512BW, `vmull_s16` + `vpadalq_s32` on NEON. A pair of full-grid products fills an i32 lane, so the x86 kernels convert each pair sum to f32 (exact products, f32 accumulation); NEON widens into i64 and is exact. The asymmetric query is quantized onto the same grid with its own scale. - API/segment: `Datatype::Turbo16` (REST "turbo16", gRPC `Turbo16 = 6`), served by the existing turbo storages. Rejected for sparse vectors. - Tests: kernel parity/overflow, turbo storage tests over Bits16, model-test candidate, integration cases, test_turbo16_storage.py, a 16-bit cold-score criterion bench. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019inxbwm1wokue8vUje63p8 * NEON: four accumulator chains and 16/8-code steps in the 16-bit kernel Measured on Neoverse N2 (criterion `query16bit_score_cold`): equal to the two-chain loop with the vectors in L2, 1-12% faster from DRAM. The 16- and 8-code steps leave at most 7 codes to the scalar tail instead of 15. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019inxbwm1wokue8vUje63p8 --------- Co-authored-by: Andrey Vasnetsov <andrey@qdrant.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Andrey VasnetsovandClaude Opus 5.5
|
9d40cef8c4 |
Record collection creation time (#10992)
* Record collection creation time Stamp `created_at` on the create collection operation when it is submitted, so every peer persists the same value in the collection config. Existing values are kept, so collections re-created from a config (snapshot recovery, single-to-cluster migration) retain their original creation time. Return it in collection info over REST and gRPC when present; collections created before this change have no value. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Truncate collection creation time to milliseconds Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
Andrey VasnetsovandClaude Opus 5.5
|
6a9f1b1732 |
feat(audit): configurable audit log output (file / stdout / stderr) (#10466)
* feat(audit): optional stdout output for audit log Add `audit.stdout` (default false) to additionally write every audit event to stdout alongside the rolling log file, for setups where logs are collected from the container's output stream. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat(audit): configurable audit log output destination Replace the `audit.stdout` flag with `audit.output`, one of `file` (default), `stdout`, `stderr`, `file_and_stdout`, `file_and_stderr`. A single value keeps it settable via `QDRANT__AUDIT__OUTPUT`. Console output now goes through its own non-blocking writer instead of a synchronous write on the request path. With console-only output no audit dir is created and the audit read API returns no local entries. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(audit): tag audit log lines with "kind": "audit" Each written audit line now carries a fixed `kind` field, so audit entries can be filtered from regular logs when sharing stdout/stderr. The field is added on the write path only; `AuditEvent` and the audit API responses are unchanged, and files without it remain readable. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
24b7ca20a2 |
Add 8-bit TurboQuant quantization (#10939)
`TurboQuantBitSize::Bits8` makes the 8-bit TurboQuant encoding from the Turbo8 datatype available as search-side quantization over regular float storage. Bits8 has no TQ+ mode (its per-vector grid scale replaces the per-coordinate fit, and `encode` rejects TQ+), while turbo quantization always encoded with TQ+. `QuantizedVectors::tq_mode` now picks the mode from the bit width: Normal for Bits8, TQ+ for the others. It covers dense and multivector creation, the vector-size calculation, and the mutable storage that appendable segments get, so Bits8 quantizes appendable segments like the other widths. - API: REST "bits8", gRPC `Bits8 = 4`, edge FFI/Python, and `edge_tool --quantization turbo8`. - Tests: appendable segment create/score/reload for Bits4 and Bits8, update-only overlay byte parity for both widths, HNSW quantized search at Bits8 for all four distances, a model-test candidate, and a bits8 vector in test_turboquant.py. Claude-Session: https://claude.ai/code/session_019inxbwm1wokue8vUje63p8 Co-authored-by: Andrey Vasnetsov <andrey@qdrant.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
20711cc490 |
Add the Turbo8 vector datatype (#10874)
TurboQuant at 8 bits per element: the shared Hadamard rotation, then a per-vector absmax scale onto the uniform grid [-127, 127], stored as s + 128. There is no codebook, so the SIMD kernels need no lookup: the code is the value (`maddubs` / `VPDPBUSD` on x86, `SDOT` after one XOR on aarch64). The per-vector scale is not stored; TurboQuant's existing `l2 / ||c||` renormalization cancels it in scoring and dequantization. A vector takes `dim + 4` bytes (`dim + 8` for Euclid/Manhattan). - quantization: `TQBits::Bits8`; asymmetric scoring reuses `QuerySimd` as the `PLANES = 1` width; new `simd/query8bit` with exact symmetric kernels (SSE4.1, AVX2, AVX-512 VNNI, NEON, NEON+SDOT). TQ+ is not supported at 8 bits. On AVX2, batches of 8-bit vectors over 512 bytes go through the per-vector kernel (the batch loop was up to 2.9x slower on some Zen 3 hosts). - API/segment: `Datatype::Turbo8` (REST "turbo8", gRPC `Turbo8 = 5`), served by the existing turbo storages with the bit width passed from the datatype. Rejected for sparse vectors. - Tests: kernel parity/saturation, turbo storage tests over both widths, model-test candidate, integration cases, test_turbo8_storage.py. Claude-Session: https://claude.ai/code/session_01FNETeXVTeP72vQpadiUK54 Co-authored-by: Andrey Vasnetsov <andrey@qdrant.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Arnaud GourlayandClaude Opus 5.5
|
3833ed975f |
Query text indexes with a TextQuery through the Query API (#10786)
A text query ranks by BM25 over the text index of the payload field named by `using`. The proto TextQuery message is canonical (query, k, b), and REST takes either a string or the struct, normalized into one internal type, so new parameters land as fields rather than a second spelling. `using` resolves against payload fields only, and a qdrant/bm25 document keeps the sparse route. The collection checks that the field has a scoring text index and that k and b are in range before any shard runs. Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
Arnaud GourlayandClaude Opus 5.5
|
4eb7bf1335 |
[BM25] Add the scoring param to text indexes (#10770)
* Add the scoring param to text indexes
A text index with `scoring` set records document lengths and ranks by
BM25 with length normalization, replacing the const that kept scoring
off everywhere.
- TextIndexParams::scoring: Option<TextScoringParams { type: bm25 }>;
REST also reads `true` and `false`, normalized as it is read so the
stored schema holds one shape, and OpenAPI shows both forms; gRPC
carries the message only (TextIndexParams.scoring = 12)
- scoring implies phrase_matching, for the positions term frequencies
come from: normalized where index creation enters (before both the
consensus op and the direct index op are built), in the collection's
create path, and where segments build text indexes
- changing it rebuilds the index, as for any params difference
- edge does not expose it yet
- tests: serde forms, normalization, classify, the gRPC round trip, and
a collection scoring with recorded lengths against BM25 by definition
- OpenAPI regenerated
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* Test the text index scoring guards now that params carry scoring
With `scoring` on the params instead of a const, the guards that read it
can be driven directly:
- new_mmap and the immutable read-only open report an index without a
length sidecar absent under scoring, and open it otherwise
- test_congruence asserts doc_len_batch and total_tokens after a reopen
too, the reopen now asking for the lengths the builders record
- the positions-off, lengths-on tests are marked as TF-only tier guards,
a shape params no longer produce since scoring implies phrase_matching
- stale comments about the scoring const updated
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* Fit the text scoring param to dev's in-place enable_hnsw
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* Name the scoring field in the text index param validators
The exhaustive destructurings added by #10862 did not list the new
scoring field, so the build broke once the stack sat on dev.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
||
|
|
eaf73df780 |
Report which algorithm ran a filtered graph search (#10781)
* Report search algorithm in index's search telemetry * Add too_many_arguments annotation * Mark filtered_acorn optional * Time ACORN and HNSW filtered graph searches under separate counters Decide the graph algorithm at dispatch, reusing its cardinality estimate, so each filtered graph search is counted once: under filtered_acorn or filtered_large_cardinality. --------- Co-authored-by: generall <andrey@vasnetsov.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Tim Visée
|
1ec4c70144 |
Report transferred bytes for snapshot transfer (#10813)
* Report bytes transferred and rate in snapshot transfer comment While downloading a shard snapshot, the receiver now reports the number of bytes transferred and the current transfer rate in the transfer comment, e.g. `downloading (12.34s) | 1.23 GiB transferred, 45.60 MiB/s`. * Test snapshot transfer comment reports bytes transferred and rate * Document that the target peer also reports transfer comments |
||
|
Andrey Vasnetsovandqdrant-cloud-bot
|
a25e72713c |
Refactor metrics rendering into private modules (#10799)
* [AI] Refactor metrics rendering into private modules * Update metrics allow-list path in DEVELOPMENT.md Point contributors at requests.rs after the metrics module split. --------- Co-authored-by: qdrant-cloud-bot <111755117+qdrant-cloud-bot@users.noreply.github.com> |
||
|
Andrey VasnetsovandClaude Opus 5.5
|
56a7b8a84f |
Compact the payload tracker of non-appendable segments the optimizer builds (#10748)
Behind the new compact_logstore_tracker feature flag, implied by serverless_compatible. SegmentBuilder calls PayloadStorageEnum::make_immutable before its flush of a non-appendable segment's payload storage. Blobstore::make_immutable switches an append-only storage to the compacted tracker in memory, from the mappings in RAM plus whatever part was flushed, and removes the append-only tracker file; the next flush saves the compacted one. The mutable mode is left as is. Sparse vector storages keep the append-only tracker: plain nearest queries never read them. CompactedTracker::from_tracker no longer writes, and the tracker keeps its fs as an object-safe TrackerFs instead of a boxed closure. Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
|
6cd816acac |
Add match: { substring } filter condition (#10711)
* Add `match: { substring }` filter condition
Unindexed `text` and `text_any` matching became token-aware in #10341 and
#10593. Users who relied on the old raw substring behaviour get it back as
an explicit condition: `match: { "substring": "..." }` selects points with a
string value containing the given string, byte-wise and case-sensitive,
consistent with exact keyword and prefix matching.
Execution: a keyword index (with or without the `prefix` option) serves the
condition by scanning its value dictionary and uniting the postings of the
matching keys; cardinality reuses the prefix estimator, generalised into
`keys_union_cardinality`. The per-point checker goes through the forward
index. Without a keyword index the condition falls back to reading the
payload. Text, bool, integer and uuid indexes decline it.
Strict mode: the condition requires the `KeywordMatch` capability, so with
`unindexed_filtering_retrieve: false` it is rejected on unindexed and on
text-indexed fields and allowed on any keyword index.
API: `MatchSubstring` in the REST `Match` union with regenerated OpenAPI,
gRPC `Match.substring = 12`, edge python `MatchSubstring`, edge ffi
`Match::Substring`.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* Test substring fallback on a text-indexed field, document estimator params
A text index cannot serve `substring`, so on a field that has only a text
index the condition runs through the payload fallback; only strict mode may
reject it. Pin that in the OpenAPI suite and reword the strict-mode unit
test comment, which read as if the text index itself blocked the query.
Also spell out what `keys` and `postings` mean in `keys_union_cardinality`.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* Serve `match: { substring }` from the keyword key dictionary
The condition used to enumerate keys through `MapIndexRead::for_each_value`,
which on the on-disk variant drags the whole `value_to_points` file through
`for_each_entry`, plus one random read per matching key for its postings
count. Query planning paid that scan in full, before deciding whether to use
the clause at all.
Route it through the `prefix_index.bin` key dictionary instead: front-coded
keys with their postings counts inline, no postings. Estimation now reads
keys only and never touches `value_to_points`; filtering takes the matched
key list and resolves postings in one batched read, as prefix matching
already does.
This makes the `prefix` option a requirement: a keyword index without it has
no key dictionary, so it declines the condition and falls back to the payload
scan, the same as a text index. Strict mode follows — substring now infers
`KeywordPrefix`, so `unindexed_filtering_retrieve: false` names
`keyword (with prefix: true)` as the index to create.
`PrefixIndex::for_each_key` reads blocks in ~1 MiB chunks rather than the
whole key section at once: a substring cannot be pruned by the block index,
so the one-shot read would grow with the dictionary.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Sync MatchSubstring OpenAPI description with Rust docs
After routing substring matching through the prefix key dictionary,
the schema docstring required regenerating so OpenAPI stays consistent.
* Scan the map index keys for substring match without a dictionary
Without the keyword dictionary a substring condition was declined by the
field index and left to the per-point condition checker, which reads the
forward index for every candidate point. Enumerate the distinct keys of
`values_to_points` instead: the same one-pass-over-distinct-values shape as
the dictionary scan, only over a structure that interleaves keys with their
postings. Filtering and cardinality estimation are then always served, so
the condition can act as a primary clause on a plain keyword index.
Prefix matching keeps its per-point fallback: an ordered dictionary is what
makes a prefix a bounded range, and enumerating every key to answer one is
not a trade worth making implicitly.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Reject substring matching in strict mode
A substring condition is answered by looking at every distinct value of the
field: no index gives it a bounded access path, so there is no index a user
could create to make it affordable. Reject it under strict mode instead,
wherever a filter reaches verification — read and write filters, nested
sub-filters, and prefetch filters.
Filter limits are now checked before the unindexed-field check, so the
rejection is not reported as "create an index for this key", advice that
would lead to the same rejection afterwards.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Update the strict mode substring test to the new rejection
The test asserted that substring filtering under strict mode asks for a
keyword index with the `prefix` option. It is now rejected whatever index
the field carries, so every case in the test gets the same answer.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Estimate a substring condition without scanning
Counting the keys a substring matches costs the same scan as answering the
condition, and `filter` then repeats it to collect those keys. Report the
uninformed estimate instead — the one an unindexed condition has always
reported — and keep the primary clause, so the scan happens once, in
`filter`, and only when the planner picks the condition to drive iteration.
With no counts to collect, `substring_scan` collapses into `substring_keys`:
the in-RAM variants no longer look up a posting count per matched key.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Don't to parse everything as UTF-8
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: qdrant-cloud-bot <111755117+qdrant-cloud-bot@users.noreply.github.com>
Co-authored-by: timvisee <tim@visee.me>
|
||
|
xzfc
|
20229f99ba |
[combined-storage] Combined storage write (#10669)
* HNSWIndex::build(): add `inline_vectors` arg Let the caller decide whether to use `inline_vectors` format. * SegmentBuilder::build: finalize GraphInline vector storage Instead of old "graph-with-vectors plus regular vector storage", keep only the graph-with-vectors storage. * Gate the GraphInline segment build behind a feature flag |
||
|
Anton Antonov
|
88768e1b3d |
fix: expose ReshardingStage in telemetry (#10618)
* fix: expose ReshardingStage in internal telemetry Useful for cluster-manager operations. Non-breaking change for existing API. The current field values look stable enough, so it's not worrisome to keep them stable. Signed-off-by: Anton Antonov <anton.synd.antonov@gmail.com> * fix: address comment, also document uuid for symmetry Already returned, just document it in OpenAPI schema. Signed-off-by: Anton Antonov <anton.synd.antonov@gmail.com> --------- Signed-off-by: Anton Antonov <anton.synd.antonov@gmail.com> |
||
|
Tim Visée
|
cc7a209c76 |
Persist proxy segment changes across restart, don't stall WAL ack's (#10349)
* segment: move proxy pending change types into segment crate Move the types describing the changes a proxy segment buffers — point deletes (`ProxyDeletedPoint`), payload index changes (`ProxyIndexChange`, `ProxyIndexChanges`) and vector name changes (`IntendedVector`, `ProxyVectorNameChanges`) — from `shard::proxy_segment` into a new `segment::pending_changes` module. Pure move, no behavior change: the proxy segment re-exports them from their old location. Having them in the segment crate lets both the proxy segment and the segment load path share them, in preparation for persisting pending proxy changes to disk and replaying them on restart. * segment: add PendingChange describing a persisted proxy operation Add the `PendingChange` enum with one variant per operation type a proxy segment buffers — point delete, payload index change, vector name change — each carrying the operation version it was issued with. This is the shape in which pending proxy changes are persisted to disk. Derive serde on it and on the buffered change types it embeds, so entries can be serialized into a log file and read back. `PartialEq` on those types lets a persisted batch be matched against the in-memory pending buffer after a flush. * segment: add PendingChanges component persisting proxy changes to a log Add `PendingChanges`, the component that manages the operations a proxy segment buffers for one proxy layer, and persists them to disk so they no longer only live in memory. It keeps the same per-type buffers the proxy segment served its reads from (point deletes, payload index changes, vector name changes), plus a single registration-ordered buffer of everything not yet persisted. `flusher()` writes that buffer into an append-only log file inside the wrapped segment's directory: `pending_changes.log` for the inner most proxy layer, with the layer number as a suffix for each layer above it. Appends follow the mutable ID tracker: all new entries are serialized into one buffer and written with a single call on an append-mode file, then fsynced, so a crash can only leave a torn entry at the very end. Loading truncates such an entry — its operations were never durable and thus never acknowledged in the WAL — but fails hard on a malformed entry in the middle, which cannot be explained by a torn append. The component tracks the highest operation version the log covers. Every registered operation at or below it is either durable in the log or was a no-op that does not need recovery; a flusher advances it to the proxy's version even when there is nothing to write. The pending buffer is deliberately not cleared when the proxy propagates its changes to the wrapped segment, as that only makes them durable once the wrapped segment flushes. Replaying an entry twice is a version-gated no-op. A log file left behind by a previous proxy on the same segment is adopted by `open()`: new entries are appended after it and its highest version is taken over, while its entries are not loaded into the buffers as they are already applied to the segment. `load()` also reconstructs the buffers, for callers that do want the buffered state. * segment: replay persisted pending proxy changes onto a segment on load Add `recover_pending_changes`, to be called when a segment is loaded on restart, before regular WAL replay. If the segment directory holds pending changes log files, the proxies that wrote them did not propagate their buffered state into the segment before the process stopped. Instead of reconstructing the proxies, replay all logged operations directly onto the segment: inner most proxy layer first, each file in append order, through the regular version-gated segment operations (`apply_change`). Entries the segment already applied are silently skipped, so a stale file is harmless. The segment is force-flushed before the files are removed; a crash in between merely replays the files once more. * segment: test PendingChanges component Cover the pending changes component: registering and flushing each operation type and reconstructing the buffers from the log, log file naming per proxy layer and gap-tolerant listing, covering the proxy version without entries, operations registered while a flusher is captured, flushers of a dropped component, torn-tail truncation versus mid-file corruption, adoption of an existing log, and replaying logs onto a real segment: fresh, stale (already applied), multi-layer, and vector name changes. * segment: include pending changes logs in segment snapshots Register the pending changes log files of a segment in its snapshot: add them to `snapshot_files` next to the segment state and version files, existence-guarded, and to the segment manifest as unversioned files. Full, partial and streamed snapshots therefore all carry them. The recovery side needs no changes: a restored segment is loaded like any other, which replays and removes the logs. * shard: back proxy segment pending changes by PendingChanges component Replace the proxy segment's separate `deleted_points`, `changed_indexes` and `changed_vector_names` fields with a single `PendingChanges` component. Reads keep going through the same per-type buffers, now behind accessors; writes go through the component's `register_*` methods, which additionally queue every operation for persistence. Opening the component is fallible, as it adopts a pending changes log a previous proxy may have left in the wrapped segment's directory, so `UnsyncedProxySegment::new` now returns a result. Wrapping another proxy opens the next proxy layer up, writing to its own dedicated log file. No behavior change yet: the proxy still flushes and reports persistence exactly as before, nothing is written to the log. * shard: persist proxy pending changes on flush, stop holding back WAL ack Hook the pending changes component into the proxy segment's flush: the proxy flusher first persists the buffered operations into the pending changes log, then passes the flush along to the wrapped segment. The proxy's `persistent_version` now covers what the log durably holds on top of what the wrapped segment persisted itself. That is what lifts the WAL cap proxies imposed so far. `flush_all` compares each segment's version against its persistent version; a proxy used to report only the wrapped segment's persisted version while its own version climbed with every buffered operation, so the WAL could never be acknowledged past the point the proxy was created at, and a restart replayed all of it — potentially very expensive operations, such as an update by filter, all over again. With the buffered state durable on disk the generic rule acknowledges the full version, and a restart recovers it from the log instead. Dropping a proxy's data drops the component first, which waits for any in-flight pending changes flusher so it cannot append to the segment directory while that is being deleted. Update the proxy flush test to the new semantics, add a segment holder test asserting the acknowledged version advances past a proxied delete, and update the ack pin rationale in `finish_optimization`: the pin is still needed after the proxies leave the holder, it just snapshots a persistent version that now includes the log. * shard: propagate proxy changes when unwrapping on optimizer cancel When an optimization is cancelled or fails, `unwrap_proxy` puts the wrapped segments back into the segment holder. Propagate the changes buffered in each proxy into its wrapped segment first, as the snapshot unproxy path already does, instead of dropping them with the proxy. The pending changes log is deliberately left in place when unwrapping: deleting it before the wrapped segment has flushed the propagated changes would not be crash safe. It is cleaned up on restart and when the segment directory is dropped, and a new proxy on the same segment adopts and appends to it; replaying a stale file is safe because all operations are version gated. * shard: test persisted proxy pending changes Test the proxy segment against its persisted pending changes: buffered changes survive dropping the proxy without propagation and are replayed onto the segment when it is loaded again; unwrapping leaves the log in place and a new proxy on the same segment adopts and appends to it; layered proxies each persist into their own log file and a restart replays both; and a persisted log is part of the segment manifest and snapshot. * collection, edge: recover persisted proxy changes on segment load Replay the pending changes logs left behind by proxy segments onto each segment when a shard loads its segments, right after consistency repair and before the payload index rebuild, vector name reconciliation and WAL replay. Proxy state that made it to disk no longer holds back the WAL acknowledge, so this is where it must be recovered from. Proxies are not reconstructed: the segment holder starts with plain segments carrying the replayed operations, and the logs are removed once the segment flushed them. * collection: test crash recovery through persisted proxy changes End-to-end test of the persisted pending changes: wrap every segment of a local shard in a proxy, delete points so the deletes are only buffered, flush, and assert the acknowledgeable version covers them. Then acknowledge the WAL up to that version, drop the shard without ever propagating the proxies, and load it again: the deletes are gone from the WAL and must come back through the pending changes logs. The delete under test is deliberately not the last WAL entry, as the acknowledge never passes the last entry and that one is always replayed. * segment: make replaying persisted proxy changes on load an explicit mode Add `PersistedProxyChanges` to state whether persisted pending proxy changes are replayed onto a segment when it is loaded. `Replay`, the default, recovers them and removes the logs as before. `Ignore` leaves both the segment and the log files untouched and logs at debug level that replaying was skipped; it is for segment files that mirror those of another writer, where replaying would make the local copy diverge from what the writer's manifest describes. All callers pass `Replay` for now, no behavior change. * collection: do not replay persisted proxy changes on partial snapshot recovery Partial snapshots are recovered by read replicas in a read/write segregation setup. A read replica must not mutate its segments, so it cannot replay the persisted proxy segment changes on load and must ignore them instead: its segment files are a local copy of the writer's that must stay a faithful mirror of them, as later partial snapshots are diffed against what the writer's manifest describes. Replaying would mutate the segment files and remove the logs, making the copy diverge. Thread the replay mode through `LocalShard::load` as a dedicated `PersistedProxyChanges` argument, derived from the recovery type: `RecoveryType::Full` replays as before, `RecoveryType::Partial` ignores the persisted changes and leaves the logs in place. Regular shard loads replay. Extend the crash recovery test with an ignoring load first: the delete under test must not come back and the logs must survive, before a replaying load recovers it. * Persist wrapped segment before pending changes Prevents raising version of proxy segment too early * Fix comment * Fix crash window, only ready optimized segment after propagating changes The optimizer renamed a newly built segment into segments_path and wrote its version file before finish_optimization propagated the proxies' buffered changes into it. A crash in that window left the segment restart-loadable but stale, permanently losing or resurrecting points. Defer the version file save until finish_optimization has fully reconciled proxy changes into the segment, including the post-swap dedup pass, so it stays invisible to restart and snapshot recovery until then. SegmentBuilder::build() gains a `ready` flag; load_segment gains `ignore_missing_version` for the one caller reloading before that point. Incidentally also closes the crash-unsafe cancellation-orphan cleanup gap noted in #9217, since a cancelled build is discarded on restart the same way. * Force flush optimized segment, otherwise we may lose proxy changes * Don't force flush after replay, defer deleting log files until flush * Include persisted proxy changes log file in segment manifest * Add random ID to proxy log files, prevent instance conflicts * Rename proxy log file, always include level * Delete proxy log file on unproxy, defer until next flush cycle * Fix truncation * Reformat * Lock persisted segments behind runtime feature flag * Enable necessary feature flags in tests * Fix linters |
||
|
Andrey VasnetsovandClaude Fable 5.1
|
95eb3c3f79 |
Support cached id tracker memory placement (#10598)
Thread a `Populate` through the disk-resident id tracker's open paths (`DiskMappingReader`, `DiskIdTracker::open`, `ReadOnlyDiskIdTracker`, `ReadOnlyIdTrackerEnum`) so a `cached` placement primes the page cache with the mapping files on load instead of leaving them to page in on demand. The populate is derived from the segment config's placement at load time, clamped by low-memory mode, in both the writable segment open and the read-only one. The update-only lookup path keeps its transfer-nothing policy, and the build-time open stays cold: the built segment is reloaded anyway. `cached` is no longer rejected by validation. Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
Andrey VasnetsovandClaude Fable 5.1
|
81bb80a5d3 |
Expose id tracker memory placement in collection config (#10597)
Add `id_tracker: { memory: cold | pinned }` to CollectionParams,
CollectionParamsDiff and CreateCollection (REST + gRPC `IdTrackerParams`),
mirroring `payload: { memory }`. `cold` builds the disk-resident id tracker,
`pinned` the in-RAM immutable one. Unset keeps the current behavior: the
`serverless_compatible` feature flag decides.
The requested placement is persisted as an optional `id_tracker_memory` on
SegmentConfig (skipped when unset, so existing configs are unchanged); the
segment builder resolves it through `SegmentConfig::id_tracker_memory_placement`
instead of reading the feature flag directly.
The config mismatch optimizer rebuilds non-appendable segments whose effective
placement differs from the requested one. Appendable segments are skipped: they
always use the mutable tracker and get the current config when indexed.
`cached` is rejected by validation: the disk mapping reader has no
populate-on-open path.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
|
||
|
陈志谦
|
f24b7a5910 |
[AI] docs: update coverage instructions for the renamed coverage script (#10483)
tools/coverage.sh was split into tools/unit-test-coverage.sh and tools/integration-test-coverage.sh in #6414, but DEVELOPMENT.md still pointed at the old path and the new script's own usage header kept the old name. Updated both to the unit-test script. |
||
|
Tim Visée
|
52d8e39451 |
Remove RocksDB references (#10561)
* Remove RocksDB specifics from shell.nix * Re-enable sparse benches, replace RocksDB structures * Rewrite congruence test, in-memory ID tracker vs mutable ID tracker * Remove RocksDB flag from test * Remove RocksDB tool * Remove RocksDB comments * Bump OpenAPI spec |
||
|
xzfc
|
51cf88e7de |
[combined-storage] Integrate VectorStorageType::GraphInline reading (#10515)
* Placement accessors on VectorDataConfig * Wire up the GraphInline storage type * Let the HNSW index reuse the storage's links handle |
||
|
Andrey Vasnetsov
|
fec1f39d54 |
Probe large memory reports with temporary workers (#10458)
* Probe memory-report files concurrently * Simplify memory-report file probing and reuse workers * Scope file-probe workers to large memory reports * Fix OpenAPI consistency after optimizer description correction * Destructure collection memory merge result exhaustively |
||
|
xzfc
|
47e858c94e |
Remove unused (Sparse)VectorStorageType::Empty (#10431)
These were added for named-vector CRUD (
|
||
|
|
e0110f3fe8 |
feat: optional dial9 Tokio telemetry behind a dial9 feature (#10442)
* Add optional dial9 Tokio telemetry behind a `dial9` feature Integrate dial9 so storage runtimes can emit production-friendly Tokio traces. Recording is off unless the crate is built with `--features dial9` and DIAL9_ENABLED=true is set at runtime; with the feature off, runtime construction is byte-for-byte unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y7X6MkjY3P7wpP2MfHdTDY * Enable dial9 CPU and schedule profiling Turn on cpu-profiling and sched events behind the same `dial9` feature, add the DIAL9_CPU_* / DIAL9_SCHEDULE_* env knobs, and document the frame pointer rustflags the stack unwinder needs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y7X6MkjY3P7wpP2MfHdTDY * Harden dial9 env parsing and the writer-failure path - Reset Cargo.lock to the branch point and re-resolve, so the diff is additive instead of re-resolving unrelated packages. This drops the heck 0.5.0 -> 0.4.1 downgrade, which sat in the default build graph and would have changed proto codegen identifier casing. The remaining non-additive entry, toml_parser 1.0.9 -> 1.1.3, is forced by proc-macro-crate via dial9-trace-format-derive. - Parse DIAL9_* booleans the way dial9 does, accepting 1/y/yes/on and 0/n/no/off and warning on anything else. `str::parse::<bool>` took only exact lowercase true/false, so DIAL9_CPU_PROFILE_ENABLED=0 silently left 99 Hz sampling on and DIAL9_ENABLED=1 silently left recording off. - Require the numeric knobs to be positive. A zero disk budget made dial9 evict everything and stop recording within seconds while the log still reported telemetry enabled. - Treat a set-but-empty DIAL9_TRACE_DIR as unset. It skipped the /tmp fallback and wrote up to the full budget into the working directory, which is /qdrant next to storage/ in the official image. - Return a disabled guard as soon as the trace writer fails, before with_cpu_profiling and with_sched_events run. Those start their profilers eagerly, opening a perf event per thread and installing a process-global signal handler that build() would then discard. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y7X6MkjY3P7wpP2MfHdTDY * Correct the dial9 docs and give them their own section - `--cfg tokio_unstable` is required for any task data at all, not merely for fuller coverage: dial9's poll, spawn and terminate hooks are all `#[cfg(tokio_unstable)]`, and nothing in the repo sets the flag. Without it there is no task timeline and DIAL9_TASK_TRACKING_ENABLED does nothing. - Document `-C debuginfo=2`. `[profile.perf]` inherits `release` and sets no `debug` key, so the documented build symbolized off the ELF symtab with inlined callees collapsed and no file or line, unlike `[profile.bench]` which sets `debug = true` for this reason. - Move the dial9 material out from between the feature list and the prose that belongs to it. Those paragraphs describe `tracing` instrumentation and read as dial9's when the example is wedged in front of them, which points readers at `#[tracing::instrument]` for a tool that records Tokio runtime events and no tracing spans. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y7X6MkjY3P7wpP2MfHdTDY * Use cfg_select! --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: timvisee <tim@visee.me> |
||
|
Mohamed Salah
|
aa4a62f5b2 |
docs(schema): declare enforced 1..=65536 bound on VectorParams.size (#10324)
* docs(schema): declare enforced 1..=65536 bound on VectorParams.size The REST layer enforces an upper bound of 65536 on VectorParams.size via a custom validator (validate_nonzerou64_range_min_1_max_65536), but custom validators contribute no bounds to the generated JSON schema - so the published OpenAPI document only declared minimum: 1, while DenseVectorConfig.size already documented both bounds. Add an explicit #[schemars(range(min = 1, max = 65536))] attribute so clients validating requests against the schema see the same contract the server enforces, update docs/redoc/master/openapi.json accordingly, and pin the bound with a unit test asserting the generated schema. Fixes #9942 * test(openapi): accept documented size bound as a rejection path test_vector_dimension_limit asserted that an oversized VectorParams.size reaches the server and returns the exact runtime 422 message. Now that the enforced 1..=65536 bound is documented in the served OpenAPI schema (#9942), request_with_validation rejects such payloads client-side before sending. Accept either layer: a client-side jsonschema.ValidationError or the server-side validation error. * test(openapi): handle both rejection layers in dimension limit pytest.raises only covered the client-side jsonschema rejection; if the request reached the server instead, the test would fail on an unhandled response. Use try/except around request_with_validation and assert the server-side status and exact error message in the else branch. * test(openapi): assert exact HTTP 422 on server-side rejection A broad not-ok check would pass on any error status carrying the same error text; pin the documented contract to 422. * refactor(tests): address review feedback Remove the unit test asserting the generated VectorParams schema shape - it only restates the schemars attribute and adds maintenance cost. Reduce test_vector_dimension_limit to its actual contract: an oversized dimension is rejected by the documented OpenAPI schema before the request is sent. |
||
|
Kumar ShivenduandClaude Opus 5
|
83fe47c90a |
feat: add a dedicated min operator to score formulas (#10296)
Follow-up to #10287, which added `max`. Expressing a minimum still required spelling out `(a + b - |a - b|) / 2`, the sign flip of the max identity — drop the `neg` and you silently get a maximum instead. It also only works for two operands and mentions each one twice, so the scorer walks every sub-tree twice per candidate point. The pair is what makes clamping expressible: {"max": [0.0, {"min": [1.0, "$score"]}]} `min` mirrors `max` throughout, and both guard helpers introduced in #10287 already took an `operator: &str`, so they are reused unchanged: an empty operand list is rejected at parse time rather than folding to +infinity, and the Edge FFI rejects it at construction time. The result needs no `is_finite` check, since `min` cannot produce a non-finite value from finite inputs. The unindexed-field walker shares one arm for `Max | Min` as the bodies are identical, with a test pinning `min` separately so a later split cannot silently drop it. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Kumar ShivenduandClaude Opus 5
|
e2d42462fa |
feat: add a dedicated max operator to score formulas (#10287)
* feat: add a dedicated max operator to score formulas
Expressing a maximum in a score formula required spelling out the
arithmetic identity `(a + b + |a - b|) / 2`. That is easy to get wrong
(the `/ 2` is load-bearing), only works for two operands, and mentions
each operand twice, so the scorer evaluates every sub-tree twice per
candidate point.
`max` is variadic, mirroring `sum` and `mult`:
{"max": ["$score", {"mult": [0.5, "popularity"]}]}
Unlike `sum` and `mult`, `max` has no identity element for the empty
case, so an empty operand list is rejected at parse time rather than
folding to -infinity and scoring every point with a non-finite value.
The check lives in `ExpressionInternal::parse_and_convert`, which every
entry point passes through, and the Edge FFI additionally rejects it at
construction time to match how that crate validates elsewhere.
The result needs no `is_finite` check: unlike `log10`, `exp`, `div`,
`sqrt` and `pow`, `max` cannot produce a non-finite value from finite
inputs, so it follows the existing `sum` convention.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* test: cover max error propagation and datetime operands
An operand that fails must fail the whole expression rather than being
passed over in favour of a finite sibling. Covered with the failure both
before and after the finite operand: `mult` short-circuits on zero and
so can skip evaluating later operands, and this pins down that `max`
must not grow a similar shortcut that would swallow an error.
Also covers `max` over datetime operands, which reach the scorer through
a separate conversion to seconds, so that "score by whichever timestamp
is newer" is verified rather than assumed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Kumar ShivenduandClaude Opus 4.8
|
ce2f31c773 |
Detect container runtime beyond Docker in telemetry (#10132)
* telemetry: detect container runtime, not just Docker `is_docker` only checks `/.dockerenv`, a marker the Docker daemon creates. containerd/CRI-O (Kubernetes), Podman, etc. don't, so a containerized node — notably every Qdrant Cloud pod — reported `is_docker: false`, indistinguishable from bare metal. Add a `container_runtime` field (enum: none/docker/kubernetes/other) detected from well-known markers, most specific first: KUBERNETES_SERVICE_HOST → /.dockerenv → other container markers → none. Only docker and kubernetes are enumerated (the modes that matter for Qdrant); the rest fold into `other`. `is_docker` is kept but derived (== docker). Detected once via LazyLock. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Regenerate OpenAPI spec Add the ContainerRuntime schema and container_runtime field. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
xzfc
|
74dd4b71e3 |
Integrate batched HNSW (#10194)
* [14] HnswGraph: wrapper enum over in-RAM and batched graphs * [15] HnswGraph: route async backends to the batched graph * [15.a] De-tautologize `test_open_matrix` Anti-pattern: `expect_batched` mirrors `format_is_batched` logic. * [16] HNSW healing: reopen as direct * [17] Add async_hnsw_graph feature flag * Batch size |
||
|
David Dallakyan
|
79c71a644b |
docs(collection): align SnapshotRecover.priority rustdoc with SnapshotPriority (#10233)
Replica prefers existing replica data, then this peer is synchronized from other replicas — not from the snapshot. Also fix the "if will" typo. |
||
|
Andrey VasnetsovandClaude Fable 5
|
e32d3fbf89 |
Add acosh expression to formula query (#10231)
Unary inverse hyperbolic cosine, parallel to sqrt/ln/exp/log10, in REST, gRPC, and edge (FFI + Python) interfaces. Inputs below 1 produce the same NonFiniteNumber error as an invalid sqrt or ln. Closes #10186 Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
David Dallakyan
|
e45c248d38 |
add missing timeout parameter for get point API (#10235)
GET /collections/{name}/points/{id} already accepts ReadParams.timeout;
OpenAPI only documented consistency. Match the POST /points query parameters.
|
||
|
Ivan PleshkovandClaude Opus 5
|
86b9330628 |
transfer: send raw payloads, behind feature flags (#10066)
A raw point can carry its payload as the byte blob it is stored as, mirroring `PointStructRaw.raw_payload` on the internal gRPC API. The blob travels from the sending node into the receiving node's WAL untouched, so the sender never parses the payload it read and neither node builds a protobuf value tree for it. It is parsed exactly once, where the operation is unpacked for apply (`process_point_operation`), because that is the first place the parsed form is actually needed: `set_full_payload` goes through the payload index, which cannot be updated from bytes. The gRPC boundary therefore only checks the encoding tag and rejects a point that sets both payload fields, the way the enclosing request already rejects both `points` and `raw_points`. Moving the parse onto the apply path makes its error classification load-bearing, so a malformed blob is reported as `OperationError::MalformedPayloadBlob` — the payload sibling of `MalformedVectorBlob`, mapped to `CollectionError::BadInput` for the same reason: a bad blob that reached the WAL has to be skipped on replay instead of crash-looping recovery. Three consequences of the blob living that long are handled explicitly rather than by convention: - `decode_payload_raw` takes the blob only once it has parsed, so a failure leaves the point holding it instead of holding neither representation. - `upsert_points_raw` and `sync_points_raw` refuse a point that still carries a blob. They read the parsed payload, so such a point would otherwise be stored with no payload at all, and a `debug_assert!` would not catch it in release. - `is_equal_to` compares blob to stored blob as bytes. A differing encoding costs a redundant upsert on sync, never a skipped one. The `raw_payload_transfer` bench measures the trade, per 100-point batch (one transfer batch) at payloads of ~200 B / ~700 B / ~7 KB: - Sender, storage bytes to wire: 16x / 37x / 113x faster. This is where the whole win is — no parse of the blob that was read, no value tree built. - WAL encode: 5x / 11x / 25x faster, writing a byte string instead of a map. - Receiver, wire to applicable point: 1.09x / 1.10x / 1.06x. Near neutral, as it swaps walking a prost value tree for a JSON parse. - Wire bytes: ~6% smaller. WAL bytes: 10-32% *larger*, because the blob is JSON while a parsed payload is written as a compact CBOR map. The WAL growth is accepted rather than fixed: decoding earlier to win those bytes back costs a second full deserialization, and would leave the receiving side with a `payload_raw` that is never populated. Making the blob itself compact belongs in the payload storage encoding (`RawPayloadEncoding` is the extension point for it), not here. Two flags, both off by default and both sender-only (nodes accept raw points and raw payloads regardless), read where the transfer batch is prepared: - `transfer_raw_points` transfers every collection as raw points, not only those whose vector storage would drift in a decode-encode round-trip. - `transfer_raw_payloads` ships the blob a raw read hands out; without it the prepared batch decodes it back into the parsed payload, and the wire message is exactly what it is today. Neither is enabled by `all`: a node only accepts them once it runs a version that understands them, so they can only be switched on a release later. Nothing enforces that yet — the transfer has no peer-version gate. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Andrey VasnetsovandClaude Fable 5
|
6633205704 |
[UpdateOnly] create Blobstore-backed storages append-only under a feature flag (#10154)
* [UpdateOnly] create Blobstore-backed storages append-only under a feature flag A new `append_only_storages` feature flag, enabled by `serverless_compatible`, switches every Blobstore creation site — the payload storage, the appendable field indexes (numeric, map, geo, full-text) and the sparse vector storage — to the append-only Logstore mode. One shared helper maps each site's Gridstore layout to its Logstore counterpart, carrying the page size and compression over; blocks and regions have no append-only equivalent. Only creation consults the flag: an existing storage keeps its persisted mode, both modes are always readable, so flipping the flag never strands data. Two changes make the flag usable rather than booby-trapped: `Logstore::delete_value` now succeeds trivially where nothing is stored, as mutable mode does, and errors only for a stored value. The ordinary write paths delete defensively — an index clears a slot before filling it, an empty value is stored as a deletion — and only ever hit occupied slots when something is genuinely mutated in place. A segment derives `append_only_storages` from the persisted payload storage mode when it opens — not from the flags, which may have changed since it was created — and it forces append-only mutation semantics on itself: every mutation clones to a fresh slot, and the same-operation slot-reuse shortcut is disabled, since the second step of a multi-step write would rewrite a payload row those storages cannot rewrite. The end-to-end test runs as its own binary (feature flags are process-global) with `serverless_compatible` on: the segment comes out holding Logstore storages, and upserts, updates of existing points, multi-step same-operation writes, deletes, an index build over existing points, a flush and a reload all run against them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * [UpdateOnly] assume the flag pairing instead of deriving it, trim the docs Per review: `append_only_storages` without `append_only_mutations` is not a state to defend against — `init_feature_flags` forces the pairing, and the same-operation slot-reuse check reads the flag directly. That deletes the segment-side derivation: the `append_only_storages` segment field, the persisted-mode read at open, and the `is_append_only` accessor chain through `Blobstore`, `PayloadStorageImpl` and `PayloadStorageEnum`. Docstrings and comments trimmed to the guarantees; how the write paths use them is their own business. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * [UpdateOnly] restructure the creation config around mode-neutral options `CreateOptions` in the blobstore crate holds what a caller actually decides — page size, block size, compression — and `into_config(append_only)` turns them into the config of either mode, each taking the fields it can express. The segment-side `storage_config` supplies only the mode, from the feature flag. That removes the misnomer chain the previous cut left behind: nothing named gridstore returns a config that might not be one, and no call site builds a `GridstoreConfig` just to have its fields repacked. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * [UpdateOnly] keep Logstore strict; fix the callers that deleted nothing Per review, `Logstore::delete_value` goes back to an unconditional error: a delete reaching an append-only storage is a caller bug to fix, not a case to absorb. The callers that issued vacuous deletes are fixed instead: - The numeric and geo indexes only delete from the storage when their in-memory index actually held values at the slot — the two are written in lockstep, so an empty slot has nothing stored either. The map and text indexes already worked this way. - The sparse storage skips the delete for keys at or past its end, where nothing was ever stored. Each removed call was wasted work in mutable mode too. The e2e test now also drives a numeric index and the sparse storage against append-only mode, and asserts that deleting a stored sparse vector fails. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * [UpdateOnly] drop the same-op slot reuse; upserts write the whole point at once The append-only path never needs a multi-step point write: the one real multi-stepper was the shard's upsert — `upsert_point` followed by a payload step under one operation number — and it now goes through `upsert_moved_point`, which writes vectors and payload as one operation and one slot. With that, the same-operation slot-reuse carve-out in `handle_point_mutate` has nothing to carry: on an append-only segment every mutating step clones to a fresh slot, unconditionally, and the `append_only_storages` special case disappears with it. The version gate skips only on strictly newer versions, so a caller that still multi-steps stays correct — it pays a slot per step. `PointToUpsert` now exposes the point's parts — raw vectors, decoded vectors, payload — and both write paths are provided from them: `upsert_into` hands the parts to `upsert_moved_point`, `write_moved` adapts them to the copy-on-write move callback. The two hand-written `upsert_into` bodies and the follow-up payload helper are gone. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Regenerate OpenAPI for the `append_only_storages` feature flag Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * one extra debug assertion * fmt --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Andrey VasnetsovandClaude Opus 5
|
aa6c5d8403 |
Global quota API (#10035)
* feat: global quota API Memory and disk are node-wide resources, so configuring their thresholds per collection through strict mode makes little sense. Move them behind a single cluster-wide `QuotaManager`. The quota config is seeded from `storage.quotas` in the settings (and so from env vars), overridden by `quota.json` in the storage directory, and updated cluster-wide through a new `SetQuotaConfig` consensus operation which rewrites that file on every peer. Raft snapshots carry it too, so a peer that joins by snapshot picks it up. Quotas are enforced wherever the strict mode memory and disk checks used to run, but no longer gated behind `strict_mode.enabled`: a value set in an enabled strict mode config still wins per resource, the quota is the default. Rejections name both the condition that tripped and the config that governs it. `GET /quotas` reports the config plus current utilization to global read users; `PUT /quotas` replaces it for global manage users. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: cover the quota endpoints in the API consistency checks `test_all_rest_endpoints_are_covered` and the OpenAPI endpoint count both break on any new REST endpoint. Add `GET`/`PUT /quotas` to `ACTION_ACCESS` with their JWT access tests, and bump the expected API count. The quota endpoints stay out of `REST_ENDPOINT_WHITELIST`: that list is for data-plane endpoints reported per-endpoint in metrics. Also add a Raft snapshot CBOR compatibility test — snapshots are exchanged between peers of different versions during a rolling upgrade, so `quota_config` must be absent-tolerant in both directions. Review feedback: persist through `SaveOnDisk`, which already implements the write-before-swap protocol this was doing by hand; validate the config at both persistence boundaries, since a hand-edited quota file or a config arriving through consensus does not pass the REST handler's validation, and a `0%` limit would reject every update forever. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test: assert seeding a quota from invalid settings persists nothing Follow-up to review feedback claiming `SaveOnDisk::load_or_init` writes the init value before it is validated. It does not — only `SaveOnDisk::new` persists — but the property matters: were seeding to persist first, invalid settings would leave a `quota.json` that fails validation on every subsequent start, and the node could only be recovered by deleting it by hand. Pin it down with a test. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor: make QuotaManager the single reader of memory and disk The quota checks measured memory and disk themselves, while the optimizer and the WAL disk watcher each called `fs4::available_space` behind their own ad-hoc caches. Fold all of it into QuotaManager: it owns the readings, the freshness policy, and the limits they are compared against. Moves the module to `lib/shard`, since the optimizer sits below `storage` and has to reach it; `storage::quota` re-exports it, so consensus, the `/quotas` API and StorageConfig are unchanged. The manager is installed as a process singleton by TableOfContent, ahead of loading any collection. - Callers hand in QuotaLimits overrides instead of a StrictModeConfig, and an override can now only tighten. A collection-level admin could raise `max_disk_usage_percent` past a cluster-wide limit that needed global manage rights to set; ties resolve to the quota so the rejection names the knob that actually has to change. - Measurements are cached for 5s, but a reading at or above its limit is never reused: a rejected client retries, and freeing the resource has to take effect on the next request rather than a TTL later. - `fits_on_disk` sizes an optimization against physical free space only, never the configured limits. Optimizations are what free a full disk, so the quota must not be what stops one. - `percent_of` widens to u128 instead of saturating the multiply, which under-reported utilization (failing open) above ~184 PB. - StorageConfig::quotas is optional; absent means no quota is enforced. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: don't recover dead replicas onto a node at a resource limit Recovering a dead replica pulls a whole copy of its shard onto this node. If it is already at its memory or disk quota that transfer cannot finish, and starting it only pushes the node further past the limit. Skip it and reconsider on a later sync, once the resource frees up. Adds QuotaManager::check_capacity for work that lands bytes here without being an update. Unlike fits_on_disk the configured limits do apply: taking on a replica is not what frees a full node, so there is no deadlock to avoid by letting it through. The check is hoisted out of the per-shard loop because a node over its limit re-measures on every call, so checking per dead shard would cost a statvfs each. It is free when no quota is configured. Also trims the comments across the quota module, which had grown well past what the code needs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test: drop trivial and duplicated quota tests Six tests removed, ~140 lines, with no loss of coverage: - a_rejection_names_the_knob_that_has_to_change asserted that a format! contains its own literals; the message is covered end-to-end by the override test and by test_global_quota.py. - a_node_over_its_quota_has_no_capacity_to_take_on_a_replica was 30 lines for check_capacity, a one-line delegation to check_update the test above it already calls. - a_rejecting_measurement_is_never_served_from_the_cache duplicated the meter test, which proves the same rule with an injected reader instead of inferring it from the real filesystem. - free_space_is_reported_without_enforcing_anything covered a one-line accessor, and its point is what the fits_on_disk test is for. - The two resolve tests and the three meter tests each collapse into one. DiskFit::Unknown keeps its coverage as two lines inside the fits_on_disk test rather than its own fixture. The snapshot compat pair becomes one test: the second only asserted cluster_metadata.is_empty(), which says nothing about quotas — the real check was the deserialize, now an expect that states it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: re-measure free space as the disk fills, and drop a Windows-only assert Two CI failures, both from this branch. e2e test_low_disk: the DiskUsageWatcher I replaced escalated to checking on every call once free space fell below 512 MB. Folding it into the quota manager lost that — available_bytes passed no limit, so a reading was reused for the full 5s however little space was left. On a disk filling as fast as that test fills it, 5s blind is enough to actually run out and the WAL write dies instead of returning "No space left on device". available_bytes now takes a watch_below level and never reuses a reading under it, which is what the old ladder was expressing. The watcher passes max(min_free, 512 MB), so the escalation point is back; above it the 5s cache still costs fewer syscalls than the old 128-call ladder. fits_on_disk gets the same rule by passing required_bytes, so a merge that does not fit re-checks rather than sitting on a stale sample. Windows: fits_on_disk on a missing path was asserted to be Unknown, but GetDiskFreeSpaceEx resolves up to the containing drive and succeeds — as common::disk_usage's own test documents. Dropped; the branch is a two-line else and is not portably reachable. Also renames an_optimization_is_sized_against_the_disk_not_the_quota, which needed explaining to be understood. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: report the quota config in telemetry Reads it from the quota manager rather than the settings, so it is the config the node is actually enforcing: a peer that missed a consensus update reports what it is applying, not what the cluster agreed on. Gated on global access, the same access `GET /quotas` requires, and left out of `PeerTelemetry` — a quota is per-node state, so each peer reports its own. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: regenerate OpenAPI, and cover the quota in the telemetry key sets Two CI failures from the previous commit. Referencing QuotaConfig from TelemetryData moves its definition earlier in `components/schemas`, because TelemetryData is generated ahead of QuotaStatus. Regenerated rather than hand-patched, so the schema is a pure move. test_telemetry_detail asserts the exact set of top-level telemetry keys. The quota is reported at every level, including 0 — it is three scalars, it is the default the endpoint serves, and it is what explains an update being rejected — so both key sets gain it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: remove `max_disk_usage_percent` from strict mode Disk is a node-wide resource, so a per-collection percentage of it never meant anything a caller could act on: the limit describes how full the *node* is, and which collection the write happens to target has nothing to do with it. The global quota is where it belongs. It shipped in 1.18.2 without documentation, so this drops it outright rather than deprecating. Removal is soft in every direction: StrictModeConfig has no `deny_unknown_fields`, so a client still sending it gets it ignored rather than a 400, and the same struct deserializes the persisted collection config, so collections created on 1.18.2+ keep loading. Proto field 22 is reserved so the number is never reused. The e2e test becomes a quota test — the fixture and the timing are the interesting parts and they carry over unchanged; only how the threshold is configured differs. `max_resident_memory_percent` was documented and stays for now. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor: enforce the strict mode memory limit outside the quota `max_resident_memory_percent` was folded into the quota as an override, which meant the quota check had to know about strict mode, and retiring the setting would mean unpicking `EffectiveLimit` and `LimitSource` from the resolution logic. It is now a check of its own in `verification/mod.rs`, next to the strict mode checks it belongs with, borrowing only the measurement from the quota manager — which stays the node's single reader of process memory, so both checks still share one reading. Deleting the setting later is deleting one function and its one caller. `QuotaManager::check_update` takes no arguments and consults the quota alone. A collection can still only tighten the limit for itself, because its own check runs in addition rather than in place of the quota's, and each rejection now names the config that has to change without having to carry a `LimitSource` to say so. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor: enforce the quota on the update path, not in strict mode The quota check sat inside `check_strict_mode_toc_batch` only because that was the one place holding the collection's strict mode config. It doesn't need one any more, and the placement had a real cost: coverage depended on each handler remembering to ask for a strict mode check, and four of the internal update RPCs do — `sync_internal`, which moves the most bytes onto a node, does not. It now runs in `Collection::update_from_client` and `update_from_peer`, which every update passes through. `update_from_client` checks ahead of the shard split, so an operation is accepted or refused whole rather than landing on some shards and being refused by others. Classification moves with it, from ~10 `consumes_memory` impls on request DTOs to one exhaustive `CollectionUpdateOperations::consumes_quota`. The internal enum has variants — raw upserts, conditional upserts, the syncs — that have no client-facing request type, so per-DTO impls structurally could not classify them. Shard-transfer syncs stay excluded, as they are today: a transfer is sized up once before it starts, and refusing its batches partway abandons work that is nearly done only for it to restart from the beginning. Index and named-vector creation reach shards through consensus, past this check — a peer must not refuse what the cluster agreed to — so they keep their pre-consensus check, now against the quota manager directly. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: deprecate `max_resident_memory_percent` in strict mode Same reason the disk threshold went: memory is node-wide, so a per-collection percentage of it caps how full the *node* is, which has nothing to do with which collection is being written to. The node-wide quota caps it once for everything. Unlike the disk threshold this one shipped documented, in 1.18.0, so it keeps working — as a limit a collection can tighten for itself, never lift — and gets the usual markers: `#[deprecated]` on both Rust structs, `[deprecated = true]` on proto field 21, and `deprecated: true` in the OpenAPI schema, which schemars derives from the attribute. The note names 1.21 as the removal. Recording a version matters here: the audit in docs/plans/overdue-deprecations.md found that this repo has never written a removal deadline down, and members of the 1.15.0 deprecation batch are still in tree. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: reconcile the quota readers with #9891 #9891 landed effective (cgroup) figures in telemetry while this branch was making QuotaManager the single reader of memory and disk. Two collisions, neither of which git sees. `segment::utils::mem::total_memory_bytes` is now a shared accessor with a 5s TTL, so a cgroup resize is picked up. The quota module had its own `OnceLock` copy that froze the value at startup — exactly what #9891 set out to fix — so it delegates to the shared one instead. Telemetry's new `disk_size` called `common::disk_usage::disk_usage` directly. That reader lost its TTL cache on this branch when the caching moved into the quota manager's meter, so it would have taken an uncached `statvfs` on every telemetry request, and it put a second disk reader back in the tree. It goes through `QuotaManager::disk_capacity_bytes` now, sharing the reading the quota check already takes. Verified it still reports the storage filesystem, matching `df`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor: split the quota manager by what each half does `manager.rs` had grown to 450 lines holding three separate jobs: owning the config and its file, taking the readings, and comparing one against the other. - `manager/store.rs` — the `Store` enum, `QUOTA_CONFIG_FILE`, and config validation, which is now the store's own business rather than something every caller has to remember to do first. - `manager/measure.rs` — every reading, and `DiskFit`. The "nothing else calls `statvfs` or reads process RSS" claim is now checkable by looking at one file. - `manager/enforce.rs` — `check_update` / `check_capacity` and the threshold comparison. - `manager/mod.rs` — the struct, its construction, and the config accessors: what a reader needs to see first. Tests move with their subject. No behaviour change: `set_config` used to validate before delegating to the store, and now the store validates on write, which is the same order of operations from the outside. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: count copy-on-write deletes toward the quota Dropping a vector or a payload key does not free anything on its own: copy-on-write rewrites the point to produce the version without that field, so storage grows first and is only reclaimed once the optimizer gets to it. Gating those as if they were reclaiming space let a full node keep taking writes that make it fuller. Deleting whole points stays exempt. That is the one operation that has to work on a node at its limit, or there is no way back under it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: say that the quota's reported usage is per node `GET /quotas` returns one cluster-wide config and one set of utilization figures, which reads as though both describe the cluster. They do not: memory and disk are node-local, so `usage` is whatever the peer that served the request is seeing, and a peer under its limit says nothing about the others. Also corrects `resident_memory_percent`, which claimed to be a share of total system memory. It is a share of the memory available to the process, which under a cgroup is the limit rather than the host's RAM. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: treat a node over its quota as a failed replica, not a bad request A quota rejection described the request as invalid (400) and was classified non-transient, which is how the replica set recognises errors that every replica would produce alike. A quota is the opposite: the input is fine and the answer depends on which machine you ask. On the default `wait=false` path that combination silently dropped the write — `update.rs` only deactivates transient failures when nothing completed — leaving the replica Active and permanently missing data its co-replicas had. It is now `InsufficientStorage`, transient, HTTP 507 / gRPC `ResourceExhausted`. So a node that is out of room is handled like one that is offline: - last active replica, or every replica over quota: nothing could take the write, and the client is told the cluster is out of room. - more than one replica: the full node is deactivated through the same path a dead peer takes, and the update stands if enough replicas accepted it. `check_capacity` already keeps recovery off that node until it has room. The check also moves off `update_from_client`, which applied the coordinator's own limit to the whole operation even when it held no replica of the shards being written. Each replica set now gates its own local write and records the refusal as a failure of this peer, so a node only ever answers for itself. `ResourceExhausted` is shared with rate limiting, and the reverse conversion mapped it straight to `RateLimitExceeded` — a forwarded rejection came back as 429. Statuses now carry a marker so the two stay distinguishable across the wire. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: report quota pressure per node, and across the cluster A quota is node-local, so finding out which node has hit one meant asking each of them in turn — and nothing at all showed up in monitoring. `/metrics` gains a `quota_exceeded` gauge for the local node. It is emitted only while the quota is enabled: with it off the value would be a constant 0 that says nothing about the node, and an alert built on it would go quiet rather than fire if someone disabled the quota. Telemetry's `quota` field carries the same verdict alongside the config, since that is where the metric is derived from. `GET /quotas` now answers for the whole cluster. A new `GetQuotaUsage` RPC on the internal `QdrantInternal` service returns what one peer is using, and the handler fans it out to every known peer in parallel. Peers that do not answer are left out rather than failing the request — the nodes that are out of room are exactly the ones most likely to time out, and a partial answer still names them. Outside distributed mode the field is absent rather than a map of one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: report the quota metric per resource `quota_exceeded` was one flag for the whole node, which does not say what to go and fix — disk is freed by deleting or optimizing, memory by unloading. It now carries a `resource` label: quota_exceeded{resource="memory"} 0 quota_exceeded{resource="disk"} 1 A resource with no limit gets no series at all, for the same reason the metric is absent while the quota is disabled: a series that can never reach 1 reads as healthy and would quietly carry an alert that cannot fire. `QuotaManager::exceeded` returns the per-resource verdict, with `None` for a resource this node does not cap. Telemetry reports the same breakdown, since the metric is derived from it. The peer usage RPC keeps a single flag — it sits next to both percentages, so it only has to answer "is this peer refusing writes". Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: drop a no-op error conversion the linter caught `check_global_access` already returns a `StorageError`, so mapping it through `StorageError::from` converted the type to itself and tripped `clippy::useless_conversion`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: hold a tripped quota until usage clears a release margin A resource resting on its limit crosses it in both directions on the noise between two readings, and each crossing is expensive: the node refuses a write, its replica is deactivated, usage dips, recovery starts sending a whole shard copy back, and the arriving data pushes it over again. The loop sustains itself, and every lap costs a shard transfer. A limit now trips at its configured value but only clears once usage has fallen 5 percentage points below it, so the crossing has to be real. The margin is floored at 1%, since a limit smaller than the margin would otherwise be impossible to fall back under and would strand the node. The verdict is carried on the manager rather than recomputed, which makes it the thing reporting shows: expect `exceeded` to be set while the utilization next to it is already back under the limit. Rejections say so too, rather than claiming a limit that is no longer exceeded: Disk usage is at 87% of total capacity. It reached the configured limit of 90% and has to fall below 85% before this node takes writes again. Changing the config clears the verdicts. New limits are a deliberate act, and should not be held back by the margin of a limit that no longer exists. Both resources are now evaluated on every check instead of stopping at the first failure, so a verdict is never left behind reporting a reading that has since been superseded. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: make the quota release margin configurable 5 points is a guess about how noisy a deployment's usage is, which is not something one number can be right about: a node whose disk moves in gigabyte steps needs a wider margin than one that creeps, and an operator who wants the old flip-on-every-reading behaviour should be able to ask for it. `release_margin_percent` joins the rest of the quota config, so it seeds from `QDRANT__STORAGE__QUOTAS__RELEASE_MARGIN_PERCENT`, replicates through consensus, and changes with `PUT /quotas`. Defaults to 5 and is filled in when a request omits it, so it always answers with the margin actually in force rather than leaving the caller to assume one. `0` releases as soon as usage is back under the limit. `QuotaConfig` grows a hand-written `Default` for it, since deriving one would have quietly defaulted the margin to 0 and disabled the hysteresis for anyone constructing a config in code. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor: leave the release margin unset by default, and hold verdicts in atomics `release_margin_percent` is `null` unless someone sets it, rather than materialising 5 into every config. A quota written today then does not pin a number a later release may want to revise, and `{"enabled": false}` still round-trips as itself. `QuotaConfig::limits` resolves it, next to `enabled`, so enforcement never sees the unset case. The verdicts move from a `Mutex<QuotaExceeded>` to one `AtomicBool` per resource. They are judged independently and nothing reads them as a pair, so the lock only added contention to the path every update takes; a verdict that races a concurrent check is re-decided by the next one from a fresh reading. That also drops the tri-state. Only "was this over its limit" has to survive between checks — whether a resource is enforced at all follows from the config and the reading, so it is derived when reporting rather than stored. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: drop a comment arguing with a design that was never here Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Kumar ShivenduandClaude Opus 4.8
|
7743616b3b |
Report effective (cgroup) CPU, RAM and disk in telemetry (#9891)
* Report effective (cgroup) CPU, RAM and disk in telemetry
The `system` block reported host-level figures that ignore the limits the
kernel actually enforces on the process:
- `cores` <- sys_info::cpu_num() (host socket count)
- `ram_size` <- sys_info::mem_info() (host total RAM)
- `disk_size` <- sys_info::disk_info() (container root fs)
On any cgroup-limited deployment (containers, Kubernetes pods, systemd
slices) these overstate what Qdrant can use, are misleading for capacity /
oversubscription analysis, and don't match how Qdrant sizes itself.
Report the effective values instead, reusing existing helpers:
- `cores` -> common::cpu::get_num_cpus() (already drives sizing)
- `ram_size` -> segment::utils::mem::total_memory_bytes()
(cgroup limit via cgroups_rs, else sysinfo host total)
- `disk_size` -> common::disk_usage::disk_usage(storage_path)
(data-volume capacity, cached; sys_info host disk fallback)
`Mem::new()` is not free (it builds a sysinfo System and loads the cgroup
memory controller), so `total_memory_bytes()` caches with a 5s TTL — matching
the disk-usage cache — instead of recomputing per call. The TTL (rather than
caching once) means an in-place cgroup memory resize is reflected within a few
seconds, consistent with how `disk_size` and `cores` already behave. The
strict-mode helper in `collection` now delegates to this shared accessor
(dropping its own OnceLock), so it too becomes resize-aware.
ram_size/disk_size stay in KiB to match the previous unit.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Simplify comments and code
* Simplify openapi spec
* minor comment improve
* mem: cache total_memory_bytes like disk_usage (5s TTL)
`total_memory_bytes()` mirrors `common::disk_usage::disk_usage`: a small 5s
TTL cache over `Mem::new().total_memory_bytes()`. `Mem::new()` is not free
(builds a sysinfo System + loads the cgroup controller), and the short TTL
keeps the value in step with an in-place cgroup memory resize rather than
freezing at startup. Shared by telemetry and strict-mode, like the disk cache.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Regenerate OpenAPI spec
Add the ram_size / disk_size field descriptions produced by schemars from the
updated telemetry doc comments, keeping the generated spec consistent.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* mem: address review — parking_lot mutex, hold lock, saturating_duration_since
- Use parking_lot::Mutex (no poisoning; lock() returns the guard directly).
- Hold the lock across the whole method — single acquisition, simpler.
- saturating_duration_since instead of duration_since (no panic on a cached
timestamp spuriously ahead of now).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
Andrey VasnetsovandClaude Opus 5
|
c124db6905 |
Make deprecated on_disk_payload optional in the REST schema (#10020)
* Make deprecated `on_disk_payload` optional in the API schema `CollectionParams::on_disk_payload` was the last place the deprecated flag was still a bare bool, in both the REST schema and the gRPC `CollectionParams` message. Clients generated from those schemas model it as a required bool, so removing the field in a future version would break them. Make it optional in both schemas while keeping it populated on every path that builds `CollectionParams`, so responses and the persisted collection config still carry a value and existing clients keep working until it is removed. The gRPC change is wire-compatible: proto3 `optional` only adds a synthetic oneof for presence tracking, the field number and wire type are unchanged. It does change what an absent field means, though, so decode absent as `false` when reading a remote peer's collection info -- peers that predate the change encode a plain bool, which is omitted from the wire when false. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Keep gRPC `on_disk_payload` a plain bool Adding proto3 `optional` would let a new client tell "unset" from `false`, but it also changes what an absent field means. A pre-upgrade server encodes a plain bool, which is omitted from the wire when false, so a client generated from the new proto would decode `false` as "unset" -- and upgrading the client first is the order we recommend. gRPC does not need the change anyway: protobuf tolerates a missing field by design, so a gRPC client does not break at runtime when we stop sending this one. The breakage this addresses is on the REST side, where a generated model with a required non-nullable bool fails to deserialize a response that omits the field. JSON encodes `false` explicitly, so the REST schema change carries no equivalent ambiguity. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
0a16a62f99 |
feat: io_uring setting to control which components use the io_uring backend (#10008)
* feat: `io_uring` setting to control which components use the io_uring backend A few components have both an mmap and an io_uring variant reading the very same files: the immutable dense vector storages, the single-file TurboQuant storage, and the mmap payload storage. Until now the choice was a side effect of `async_scorer` — a vector-search knob — plus, for the payload storage, a feature flag that was parked off because io_uring is ~2x slower than mmap when the data fits the page cache (#9310, #9409). Add `storage.performance.io_uring`, optional, with two modes: - unset (default): unchanged behaviour. The vector storages keep following `async_scorer`; the payload storage stays on mmap. - `disabled`: no component uses io_uring. - `auto`: a component uses io_uring when its memory placement is `cold` (data is left on disk, so reads hit the disk and there is something to gain), its feature flag allows it, and the kernel supports io_uring. Components meant to sit in RAM keep using mmap. The decision lives in one place, `segment::common::io_uring::use_io_uring`, so the openers no longer each reach for the async-scorer global. Kernel support is now probed up front through `is_io_uring_supported()` instead of opening a file and falling back on error. `async_payload_storage` now defaults to on: it no longer decides anything by itself, it only lifts the ban, and the payload storage no longer follows `async_scorer` at all — so turning it on cannot silently move an existing `async_scorer: true` deployment onto the slower path. Which backend a component ended up on depends on the config, the placement and the kernel at once, so report it in `SegmentInfo`: `vector_data[name].io_backend` and `payload_storage_io_backend`, both `"mmap" | "io_uring"`, absent for components that have no such choice. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Trim comments, drop trivial tests Two tests were only restating their own implementation: `test_mode_round_trip` round-tripped the encode/decode pair next to it, and `test_io_uring_config` checked that serde deserializes a two-variant enum. The mode matrix test stays, it is the one that pins the semantics. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Flatten `IoBackend` in OpenAPI, derive `JsonSchema` for `IoUringMode` Per-variant doc comments on a plain string enum make schemars emit a `oneOf` of anonymous single-value objects instead of a flat `enum`. Move the variant descriptions into the enum doc, as `Memory` and friends already do. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Update lib/segment/src/vector_storage/turbo/turbo_vector_storage.rs Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com> * Update lib/segment/src/types.rs Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com> * Update lib/segment/src/types.rs Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com> * Update lib/segment/src/types.rs Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com> * Update lib/segment/src/types.rs Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com> * upd openapi schema * Update lib/common/common/src/flags.rs Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com> * Require kernel io_uring support in the async-scorer fallback `use_io_uring` returned `get_async_scorer()` verbatim when the `io_uring` setting is unset, so an enabled async scorer on a kernel without io_uring opened the io_uring storage, failed, and fell back to mmap with an error log per segment. Gate that branch on `is_io_uring_supported()` too, like `Auto` already is, so the component just stays on mmap. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * upd openapi schema --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com> |
||
|
Andrey Vasnetsov
|
9e53738c19 | do not generate oneOf (#10011) | ||
|
|
8a7325ad5d |
Remove deprecated search endpoints from OpenAPI, deprecate them in gRPC (#9982)
* Remove deprecated search/recommend/discover endpoints from OpenAPI Remove deprecated REST API endpoint definitions from the OpenAPI generator. These endpoints were deprecated in v1.13.3 (`f4ced2567`, #5907, 2025-01-30) in favor of the universal `/points/query` endpoint: - POST /points/search - POST /points/search/batch - POST /points/search/groups - POST /points/recommend - POST /points/recommend/batch - POST /points/recommend/groups - POST /points/discover - POST /points/discover/batch Also removes the corresponding request types from the schema generator and updates the expected API count in the consistency check. Co-authored-by: Cursor <cursoragent@cursor.com> * Migrate OpenAPI integration tests to /points/query The deprecated /points/search, /points/recommend and /points/discover endpoints (along with their /batch and /groups variants) were removed from the OpenAPI spec, which caused validation failures in the Python integration test harness. This commit migrates the affected tests to the universal /points/query endpoint: - Delete tests dedicated to the deprecated endpoints: test_recommend.py, test_discover.py, test_multicollection_reco.py, test_recommendation_multivector.py - Refactor remaining tests to call /points/query (and /query/batch, /query/groups), translating request bodies (vector -> query / using, positive/negative -> query.recommend, target/context -> query.discover) and unwrapping the new result.points response shape. - Drop equivalence assertions against the now-removed legacy endpoints. Co-authored-by: Cursor <cursoragent@cursor.com> * Relax non-empty assertions in migrated recommend/discover tests The previous migration added `len(...) > 0` assertions to tests that previously only checked equivalence between the deprecated and new API. These assertions are too strict because the parametrized `query_filter` cases legitimately produce empty result sets. Drop the `> 0` assertion and rely on `request_with_validation` to verify the response is well-formed and HTTP OK. Co-authored-by: Cursor <cursoragent@cursor.com> * Migrate remaining OpenAPI tests off deprecated search endpoints Tests added to dev after the original migration was written still call /points/search and /points/recommend/groups through `request_with_validation`, which resolves the endpoint against the OpenAPI spec and therefore breaks once the endpoint is not in the spec: - test_turbo4_storage.py, test_sparse_idf_corpus.py, test_validation.py: translate /points/search to /points/query (vector{name,vector} -> query + using, result -> result.points). - test_group.py: drop the /points/recommend/groups half of the lookup_from validation test in favour of the query equivalent. test_sparse_idf_corpus.py's test_query_api_supports_idf_corpus goes away: with the helper on /points/query every test in the file now exercises what it asserted. Also record why test_recommend_group cannot assert on its groups: it uses every point in the collection as a recommend example, so all of them are excluded and the result is legitimately empty. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Regenerate openapi.json without the deprecated search endpoints Drops the 8 deprecated paths and the request schemas that only they referenced: Search/Recommend/Discover request (+Batch, +Groups) types and their exclusive dependencies (NamedVector, NamedSparseVector, NamedVectorStruct, UsingVector, RecommendExample, ContextExamplePair). Regenerated output is a strict subset of the previous spec, and every remaining $ref still resolves. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Deprecate the search/recommend/discover RPCs in gRPC The REST counterparts have carried `deprecated: true` since v1.13.3 and are now gone from the OpenAPI spec, while the gRPC RPCs never got any deprecation annotation at all. Mark all 8 with `option deprecated = true` so generated clients warn, and point each doc comment at its `Query` replacement. tonic puts `#[deprecated]` on the generated client methods only; the server trait gets the doc comment alone, so our own `impl` is unaffected. The RPCs keep serving traffic — this is annotation only. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Restore the deleted recommend/discover suites on /points/query The earlier migration deleted these four files outright, but the query-side tests it left behind are all shallow smoke tests (`len(result) > 0`, `"points" in result[0]`). The deleted ones carried invariants with no query-API equivalent anywhere, so deleting them was a real loss of coverage rather than de-duplication: - test_recommend.py: default strategy equals average_vector; batch results identical to sequential singles across six request shapes; best_score with only negatives yields all-negative scores; best_score with a single positive orders identically to a nearest query; raw vectors as examples equal ids as examples. - test_discover.py: context-only scores are all <= 0; target-only orders identically to a nearest query but scores differently; with a fixed context the integer part of the score is stable while the decimal part moves, and vice versa with a fixed target; batch equals singles; lookup_from by id equals by vector. - test_multicollection_reco.py: cross-collection lookup_from, plus wrong-vector-size, unknown-collection and unknown-vector rejections. - test_recommendation_multivector.py: the same recommend invariants over a max_sim multivector collection, which the query suite never covered. Only test_recommend_missing_lookup_from_collection_with_raw_vector is dropped as genuinely redundant — test_query.py's test_query_missing_lookup_from_collection covers query, query/batch and prefetch. Two request-shape differences the translation had to absorb: - Giving no examples at all is 422 (a RecommendInput validation rule), where the legacy API reported 400 from the query itself. A malformed example, such as an empty vector, is still 400. - DiscoverInput requires the `context` key and accepts only an explicit null to mean "no context", so target-only discover must spell it out. The legacy API let it be omitted. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
|
Andrey VasnetsovandClaude Opus 4.8
|
eafabd267f |
docs: describe StartResharding fields in OpenAPI (#9946)
Add doc comments to `StartResharding` fields so the generated OpenAPI spec explains what a user has to pass, and regenerate the spec. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |