Kumar Shivendu 2e47b186fe feat(text-index): rank exact identifiers first with the code tokenizer (#11263) ...

* feat(text-index): rank exact identifiers first with the code tokenizer

Ranking queries (BM25 text query, BM25 inference) now get joined
identifiers too, not only documents: a query for `getUserName`,
`get_user_name` or `getusername` also matches `getusername`, so a point
with that exact identifier outranks one with the same words scattered.
Filter and phrase queries are unchanged, since they need every token.

Documents now also join `snake_case` identifiers (`get_user_name` ->
`getusername`), so a camelCase or lowercase query finds them too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(text-index): route ranking queries through TokenizerTextKind

Add `TokenizerTextKind::RankingQuery` and dispatch it in `Tokenizer::tokenize`,
instead of `tokenize_ranking_query` special-casing the code tokenizer.
Rename the code tokenizer methods by what they emit: `tokenize_with_joined`
(subwords + joined identifiers) and `tokenize_subwords`.

Strip `ARRAY_BOUNDARY_SENTINEL` inside `tokenize` for every query kind, so
direct `tokenize(Query | Phrase)` callers no longer see it either.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: generall <andrey@vasnetsov.com>
(cherry picked from commit 6708deb95b)

2026-10-21 18:58:26 +02:00

..

benches

common::{counter,uio_trace} => common::ambient (#11040)

2026-10-21 18:57:45 +02:00

src

feat(text-index): rank exact identifiers first with the code tokenizer (#11263)

2026-10-21 18:58:26 +02:00

Cargo.toml

Ambient counters (#10967)

2026-10-21 18:57:33 +02:00