Move source access and logical spans into the tokenizer source API, and keep input and encoding ownership in the reader. Export the incremental decoder helper for parser generator extensions.
* gh-153569: move tokenizer input state and relocation into the reader
* gh-153569: borrow diagnostic lines through the source API
* gh-153569: group indentation and logical-line state
* gh-153569: remove unused tokenizer cursor and source lookup APIs
* gh-153569: report tokenizer diagnostics without rewinding the scanner
* gh-153569: use tokenizer views and remove obsolete API remnants
Make pegen and _tokenize consume tokenizer operations instead of accessing the tokenizer layout. Return token kind, span, location, and raw-string context together, so parser actions use the context of the cached token.
Represent persistent scanner positions as offsets and remove pointer save/restore bookkeeping when input storage grows.
Formatted-string expressions and comments can outlive the buffer window where
scanning began. Pointer boundaries require buffer relocation to repair tokenizer
state. Record logical source ranges instead. Retain the active input window while a
formatted string is open, give each mode ownership of its comment spans, and
materialize text through shared span views.
Represent token boundaries as logical source offsets with explicit start
and end locations, and adapt pegen and _tokenize to consume them.
Use SourceText to own decoded input across all reader modes. Streaming
readers discard consumed windows while preserving logical offsets and
reusing the allocation. Prepared and interactive input retain their source.
Force source relocation on growth in debug builds and add coverage for
formatted-string growth, interactive input, storage reuse, and offset
limits. Scanner positions remain pointers, rebased when storage moves.
Pablo Galindo Salgado