Arnaud Gourlay 5ad2da1463 [AI] perf: BM25 top-k scores long queries term at a time (#11072) ...

* perf: BM25 top-k scores long queries term at a time

Squashed for the rebase onto the term frequency file (#11165):
- Score the dense pass one window of point offsets at a time
- Check for cancellation in the dense pass and drop its touched array
- Lower the BM25 dense cut-off to 0.5 postings per document
- Explain the BM25 dense window size by its own measurement
- Fix the build after rebasing onto the direct full text index open

* Check the dense pass's filter in batches

The dense pass checked the filter one document at a time. A filter that
reads storage then issued one read per document, serially. Documents above
the threshold are now gathered into blocks of BLOCK and checked with
check_batched, as document at a time does, and offered in document order
against the current threshold, so the ranking is unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
(cherry picked from commit abc2607571)

2026-10-21 18:58:24 +02:00

..

src

[AI] perf: BM25 top-k scores long queries term at a time (#11072)

2026-10-21 18:58:24 +02:00

Cargo.toml

Bump http 1.3.1 (#6849)

2025-07-17 13:55:49 +02:00