fix(search): preserve Unicode ranges and whole-word boundaries

Lowercase the complete text and query consistently so Greek final sigma
matches exact uppercase words. Map the final folded unit back to the end
of its original character, including partial dotted-I expansions, and
reject projected ranges that overlap an earlier original-text match.

Treat every Unicode mark category as part of a word in both JS and Rust.
Use unicode-properties, already in the lockfile, for the Rust categories.
Keep Unicode lowercasing semantics without accent removal or case folding.

Add shared matcher and Rust regressions, plus real painted text/code parity
checks for sigma, expansions, original offsets, and decomposed accents.
Update the README contract and the pending native search changeset.

Validation: release native build, 268 Rust tests, and Unicode paint tests pass.
The full React suite has one separate display-link wake timeout.

Fixes #95
Fixes #96

Session: ses_ea76fa70affeF580gmCkDnrEtT
Discord: https://discord.com/channels/1422625037164351591/1562029343889428490

.changeset/highlight-search-needle.md

@@ -2,4 +2,15 @@
 '@gpuix/native': patch
 ---
 
-Typing in a find-in-page query no longer rebuilds the search for every block of text. The query is lowercased and compiled once per keystroke and shared by all the text it is matched against, so each keystroke over a long document costs a few percent less CPU. Matches are the same as before: leftmost, non-overlapping, case-insensitive by default.
+Typing in a find-in-page query no longer rebuilds the search for every block of text. The query is lowercased and compiled once per keystroke and shared by all the text it is matched against, so each keystroke over a long document costs a few percent less CPU. Matches stay leftmost, non-overlapping, and case-insensitive by default.
+
+Fix Unicode matching in both `findRanges` and native highlights. Exact uppercase
+Greek words match correctly, partial lowercase expansions such as `İ` / `i`
+cover the complete original character, and ranges keep their original offsets.
+Whole-word searches no longer treat combining marks as word boundaries, so
+`cafe` does not match the prefix of `cafe\u0301` with `wholeWord: true`.
+Matching still uses Unicode lowercasing, not accent removal or full case folding.
+
+Fixes #95
+
+Fixes #96

README.md

@@ -1,3 +1,6 @@
+> [!IMPORTANT]
+> Remove this line to confirm you've reviewed this PR before submitting.
+
 # GPUIX
 
 **React and Solid** for [GPUI](https://github.com/zed-industries/zed/tree/main/crates/gpui), Zed's GPU UI framework.
@@ -2762,7 +2765,7 @@ for those.
 |---|---|
 | `query` | substring to match, case-insensitive by default |
 | `caseSensitive` | exact case only |
-| `wholeWord` | neither neighbour may be alphanumeric or `_` |
+| `wholeWord` | neither neighbour may be Unicode Alphabetic, a number, a mark, or `_` |
 | `ranges` | explicit `[start, end)` UTF-16 pairs |
 | `color` / `activeColor` | any CSS colour; defaults come from the theme |
 | `activeIndex` | which match gets `activeColor`, for a find cursor |
@@ -2776,8 +2779,22 @@ persistent mention tint. Later entries draw on top.
 
 Matches are **non-overlapping** and leftmost-first. Case-insensitive matching
 uses Unicode **lowercasing**, not full case folding, so `ff` does not match `ff`.
-A word boundary is any code point that is not Unicode Alphabetic, a digit,
-or `_`.
+Text and query both lowercase as whole strings, including Greek final sigma.
+Matches inside a lowercase expansion cover the complete original character.
+Returned ranges stay non-overlapping in the original text.
+
+```ts
+findRanges({ text: 'ΟΣ', query: 'ΟΣ' }) // [[0, 2]]
+findRanges({ text: 'İ', query: 'i' })   // [[0, 1]], covers the original İ
+```
+
+A **word boundary** is any code point that is not Unicode Alphabetic, a number,
+a combining mark, or `_`. This does not normalize accents or remove them:
+
+```ts
+findRanges({ text: 'cafe\u0301', query: 'cafe', wholeWord: true }) // []
+findRanges({ text: 'cafe\u0301', query: 'cafe\u0301', wholeWord: true }) // [[0, 5]]
+```
 
 A match never crosses a line, exactly like browser find. It **does** cross the
 several host nodes React creates for one interpolated line, which matters more

packages/native/Cargo.lock

@@ -2647,6 +2647,7 @@ dependencies = [
  "time",
  "ttf-parser",
  "two-face",
+ "unicode-properties",
  "unicode-segmentation",
  "url",
  "wasm-bindgen",

packages/native/Cargo.toml

@@ -31,6 +31,7 @@ gpui = { path = "../../zed/crates/gpui", default-features = false, features = ["
 thiserror = "2"
 pulldown-cmark = { version = "0.12", default-features = false }
 unicode-segmentation = "1"
+unicode-properties = { version = "0.1.4", default-features = false, features = ["general-category"] }
 web-time = "1"
 # Checks fonts handed to `registerFont`; GPUI's text system aborts on malformed ones.
 ttf-parser = { version = "0.25", default-features = false, features = ["std"] }

packages/native/js/__tests__/host-runtime.test.ts

@@ -191,6 +191,36 @@ describe("shared host runtime", () => {
       .toEqual([[0, 3], [4, 7]])
   })
 
+  it("maps contextual lowercase and expanded characters to complete original ranges", () => {
+    const cases: Array<{ text: string; query: string; expected: Array<[number, number]> }> = [
+      { text: "ΟΣ", query: "ΟΣ", expected: [[0, 2]] },
+      { text: "ΟΣ ΟΣ", query: "ος", expected: [[0, 2], [3, 5]] },
+      { text: "ΟΣΑ", query: "οσα", expected: [[0, 3]] },
+      { text: "İ", query: "i", expected: [[0, 1]] },
+      { text: "xİx", query: "xi", expected: [[0, 2]] },
+      { text: "İ", query: "\u0307", expected: [[0, 1]] },
+      { text: "İ", query: "i\u0307", expected: [[0, 1]] },
+      { text: "İİİ", query: "\u0307i", expected: [[0, 2]] },
+      { text: "\u{10400}İ fox", query: "i", expected: [[2, 3]] },
+      { text: "\u{10400}İ fox", query: "\u{10428}", expected: [[0, 2]] },
+      { text: "ff", query: "ff", expected: [] },
+      { text: "İstanbul", query: "istanbul", expected: [] },
+    ]
+    for (const { text, query, expected } of cases) {
+      expect(findRanges({ text, query }), JSON.stringify({ text, query })).toEqual(expected)
+    }
+    expect(findRanges({ text: "İ", query: "i", caseSensitive: true })).toEqual([])
+  })
+
+  it("keeps all Unicode mark categories inside a whole word", () => {
+    for (const mark of ["\u0301", "\u093e", "\u20dd"]) {
+      expect(findRanges({ text: `cafe${mark}`, query: "cafe", wholeWord: true })).toEqual([])
+      expect(findRanges({ text: `${mark}cafe`, query: "cafe", wholeWord: true })).toEqual([])
+      expect(findRanges({ text: `cafe${mark}`, query: `cafe${mark}`, wholeWord: true })).toEqual([[0, 5]])
+    }
+    expect(findRanges({ text: "cafe\u0301", query: "cafe", wholeWord: true, caseSensitive: true })).toEqual([])
+  })
+
   it("observes selected text and removes its callback", () => {
     const renderer = new RecordingRenderer() as RecordingRenderer & {
       getSelectedText(): string | null

packages/native/js/host-runtime.ts

@@ -132,7 +132,7 @@ export interface FindRangesOptions {
   wholeWord?: boolean
 }
 
-const WORD_CHAR = /[\p{Alphabetic}\p{N}_]/u
+const WORD_CHAR = /[\p{Alphabetic}\p{N}\p{M}_]/u
 
 function wordCharBefore(text: string, end: number): boolean {
   if (end <= 0) return false
@@ -147,16 +147,15 @@ function wordCharAt(text: string, start: number): boolean {
 }
 
 function fold(text: string): { folded: string; map: number[] } {
-  let folded = ""
+  const folded = text.toLowerCase()
   const map: number[] = []
+  // Contextual lowercase changes sigma's value, not its length, so scalar lengths map both.
   for (let index = 0; index < text.length;) {
     const char = String.fromCodePoint(text.codePointAt(index) ?? 0)
     const lower = char.toLowerCase()
     for (let unit = 0; unit < lower.length; unit += 1) map.push(index)
-    folded += lower
     index += char.length
   }
-  map.push(text.length)
   return { folded, map }
 }
 
@@ -172,8 +171,11 @@ export function findRanges(options: FindRangesOptions): Array<[number, number]>
     if (at < 0) break
     from = at + needle.length
     const start = map ? map[at]! : at
-    const end = map ? map[from]! : from
+    // The final folded unit can be inside an expansion; include its complete source character.
+    const last = map ? map[from - 1]! : from - 1
+    const end = map ? last + (text.codePointAt(last)! > 0xffff ? 2 : 1) : from
     if (start >= end) continue
+    if (start < (ranges.at(-1)?.[1] ?? 0)) continue
     if (wholeWord && (wordCharBefore(text, start) || wordCharAt(text, end))) continue
     ranges.push([start, end])
   }

packages/native/js/host.ts

@@ -370,7 +370,7 @@ export interface HighlightSpec {
    */
   query?: string
   caseSensitive?: boolean
-  /** Only match when neither neighbour is alphanumeric or `_`. */
+  /** Only match when neither neighbour is Unicode Alphabetic, a number, a mark, or `_`. */
   wholeWord?: boolean
   /**
    * Explicit `[start, end)` pairs in UTF-16 code units, the units `indexOf` and

packages/native/src/text/search.rs

@@ -16,6 +16,7 @@ use std::ops::Range;
 use std::sync::Arc;
 
 use gpui::Hsla;
+use unicode_properties::{GeneralCategoryGroup, UnicodeGeneralCategory};
 
 use crate::retained_tree::RetainedTree;
 
@@ -218,7 +219,7 @@ impl std::fmt::Debug for Needle {
 //
 // This is Unicode default lowercasing (`str::to_lowercase`), NOT full case
 // folding. `ff` does not match `ff`, and `İ` lowercases to `i` plus a combining
-// dot. `findRanges` in `packages/react/src/hooks/use-text-search.ts` uses
+// dot. `findRanges` in `packages/native/js/host-runtime.ts` uses
 // JS `toLowerCase`, which follows the same Unicode rules.
 
 /// Lowercase `text` and record, for every folded byte, the byte offset of the
@@ -228,18 +229,17 @@ impl std::fmt::Debug for Needle {
 /// (`İ` is 2 bytes and folds to 3), so a match offset in folded space does not
 /// index the original. The map converts it back exactly.
 fn fold(text: &str) -> (String, Vec<u32>) {
-    let mut folded = String::with_capacity(text.len());
+    let folded = text.to_lowercase();
     let mut map: Vec<u32> = Vec::with_capacity(text.len() + 1);
+    // Contextual lowercase only changes sigma, which keeps the same byte length.
     for (ix, ch) in text.char_indices() {
         for lower in ch.to_lowercase() {
             let len = lower.len_utf8();
-            folded.push(lower);
             for _ in 0..len {
                 map.push(ix as u32);
             }
         }
     }
-    map.push(text.len() as u32);
     (folded, map)
 }
 
@@ -279,7 +279,7 @@ fn utf16_range_to_bytes(text: &str, start: usize, end: usize) -> Option<Range<us
 }
 
 fn is_word_char(ch: char) -> bool {
-    ch.is_alphanumeric() || ch == '_'
+    ch.is_alphanumeric() || ch == '_' || ch.general_category_group() == GeneralCategoryGroup::Mark
 }
 
 fn is_whole_word(text: &str, range: &Range<usize>) -> bool {
@@ -453,11 +453,22 @@ fn matches_in(
         return out;
     }
     for hit in spec.needle.find(folded) {
-        let (Some(&start), Some(&end)) = (fold_map.get(hit.start), fold_map.get(hit.end)) else {
+        let (Some(&start), Some(&last)) = (fold_map.get(hit.start), fold_map.get(hit.end - 1))
+        else {
             continue;
         };
-        let range = start as usize..end as usize;
-        if range.start >= range.end {
+        // A hit may end inside an expansion; include the complete source character.
+        let Some(last_char) = text
+            .get(last as usize..)
+            .and_then(|tail| tail.chars().next())
+        else {
+            continue;
+        };
+        let range = start as usize..last as usize + last_char.len_utf8();
+        if out
+            .last()
+            .is_some_and(|previous| range.start < previous.end)
+        {
             continue;
         }
         if !spec.whole_word || is_whole_word(text, &range) {
@@ -785,9 +796,14 @@ mod tests {
     #[test]
     fn a_case_insensitive_needle_searches_the_lowercased_text() {
         let needle = Needle::new("FOO", false);
-        assert_eq!(needle.find("a foo and a FOO").collect::<Vec<_>>(), vec![2..5]);
         assert_eq!(
-            needle.find(&"a foo and a FOO".to_lowercase()).collect::<Vec<_>>(),
+            needle.find("a foo and a FOO").collect::<Vec<_>>(),
+            vec![2..5]
+        );
+        assert_eq!(
+            needle
+                .find(&"a foo and a FOO".to_lowercase())
+                .collect::<Vec<_>>(),
             vec![2..5, 12..15]
         );
     }
@@ -876,6 +892,40 @@ mod tests {
         assert_eq!(&text[hits[0].clone()], "İstanbul");
     }
 
+    #[test]
+    fn contextual_lowercase_and_partial_expansions_keep_original_ranges() {
+        for (text, query, expected) in [
+            ("ΟΣ", "ΟΣ", vec![0..4]),
+            ("ΟΣ ΟΣ", "ος", vec![0..4, 5..9]),
+            ("ΟΣΑ", "οσα", vec![0..6]),
+            ("İ", "i", vec![0..2]),
+            ("xİx", "xi", vec![0..3]),
+            ("İ", "\u{307}", vec![0..2]),
+            ("İ", "i\u{307}", vec![0..2]),
+            ("İİİ", "\u{307}i", vec![0..4]),
+            ("\u{10400}İ fox", "i", vec![4..6]),
+            ("\u{10400}İ fox", "\u{10428}", vec![0..4]),
+        ] {
+            assert_eq!(find(text, &spec(query)), expected, "{text:?} / {query:?}");
+        }
+    }
+
+    #[test]
+    fn all_mark_categories_continue_a_whole_word() {
+        let mut prefix = spec("cafe");
+        prefix.whole_word = true;
+        for mark in ['\u{301}', '\u{93e}', '\u{20dd}'] {
+            let text = format!("cafe{mark}");
+            assert!(find(&text, &prefix).is_empty(), "{text:?}");
+            assert!(find(&format!("{mark}cafe"), &prefix).is_empty());
+            let mut full = spec(&text);
+            full.whole_word = true;
+            assert_eq!(find(&text, &full), vec![0..text.len()]);
+        }
+        prefix.case_sensitive = true;
+        assert!(find("cafe\u{301}", &prefix).is_empty());
+    }
+
     #[test]
     fn matches_containing_an_emoji() {
         let text = "hi 👋 there";

packages/react/src/__tests__/highlight.test.tsx

@@ -467,6 +467,36 @@ describe("useTextSearch", () => {
 })
 
 describe("findRanges", () => {
+  it("matches Unicode ranges identically in JS, retained text, and native code", () => {
+    const { render, renderer } = createTestRoot()
+    const cases: Array<{ text: string; query: string; wholeWord?: boolean; expected: number[][] }> = [
+      { text: "ΟΣ", query: "ΟΣ", expected: [[0, 2]] },
+      { text: "ΟΣ ΟΣ", query: "ος", expected: [[0, 2], [3, 5]] },
+      { text: "ΟΣΑ", query: "οσα", expected: [[0, 3]] },
+      { text: "İ", query: "i", expected: [[0, 1]] },
+      { text: "xİx", query: "xi", expected: [[0, 2]] },
+      { text: "İ", query: "\u0307", expected: [[0, 1]] },
+      { text: "İİİ", query: "\u0307i", expected: [[0, 2]] },
+      { text: "\u{10400}İ fox", query: "i", expected: [[2, 3]] },
+      { text: "cafe\u0301", query: "cafe", wholeWord: true, expected: [] },
+      { text: "cafe\u093e", query: "cafe", wholeWord: true, expected: [] },
+      { text: "cafe\u20dd", query: "cafe", wholeWord: true, expected: [] },
+      { text: "cafe\u0301", query: "cafe\u0301", wholeWord: true, expected: [[0, 5]] },
+    ]
+    for (const { text, query, wholeWord, expected } of cases) {
+      expect(findRanges({ text, query, wholeWord })).toEqual(expected)
+      for (const kind of ["text", "code"] as const) {
+        render(
+          <div highlight={{ query, wholeWord }}>
+            {kind === "text" ? <text>{text}</text> : <code code={text} />}
+          </div>
+        )
+        const ranges = renderer.getPaintedHighlights().map(({ start, end }) => [start, end])
+        expect(ranges, JSON.stringify({ kind, text, query, wholeWord })).toEqual(expected)
+      }
+    }
+  })
+
   it("mirrors the native matcher for the common cases", () => {
     expect(findRanges({ text: "Fox fox FOX", query: "fox" })).toEqual([
       [0, 3],