8 Commits

Include renames

Author	SHA1	Message	Date
Pablo Galindo Salgado	6af655301a	gh-159465: Improve incomplete input detection (#159466) ... * gh-159465: Improve incomplete input detection * gh-159465: Track newline effects and preserve parser layout * gh-159465: Share incomplete-input error cleanup	2026-10-18 11:54:19 +01:00
Pablo Galindo Salgado	f892b11f22	gh-153569: Simplify tokenizer source views and reader ownership (#158849) ... Move source access and logical spans into the tokenizer source API, and keep input and encoding ownership in the reader. Export the incremental decoder helper for parser generator extensions.	2026-10-13 13:01:17 +00:00
Pablo Galindo Salgado	0b8aa89a05	gh-153569: Simplify tokenizer state (#158124) ... * Use diagnostic line for missing-brace errors * Keep implicit prepared newline metadata in the reader * Reuse parsed formatted string kind in lexer * Transfer owned tokenizer encoding strings * Remove unused tokenizer state and redundant branches * Avoid rescanning partial decoded tokenizer lines * Reuse prepared tokenizer line terminators for CR detection * Reuse interactive tokenizer chunks without carriage returns * Release tokenizer constructor resources on allocation failure * Narrow interactive newline normalization wrapper * Close duplicated tokenizer descriptors when fdopen fails * Preserve cached tokenizer lines when newline allocation fails * Collect cycles involving tokenizer readline callbacks * Preserve buffered text when decoding source files * Clear the transferred syntax error line before cleanup * Propagate tokenizer column conversion failures * Allocate parser repetition buffers on first match * Allocate type comment storage only when needed * Reuse decoded multiline text when counting token columns * Skip empty decoded tokenizer chunks * Check parser array growth before allocating * Name tokenizer bounds and derive byte lengths from their data * Set the tokenizer exception when file input size overflows * gh-153569: Send exact line endings in the REPL test * Share parser repetition buffer growth * Fold Unicode input coverage into the existing REPL test * Keep parser array growth policy private * Drop obsolete bitmap stress from source discard test * Preserve input failures while scanning string literals * fixup! Preserve input failures while scanning string literals * fixup! fixup! Preserve input failures while scanning string literals * gh-153569: Remove assertion-only lexer bookkeeping	2026-10-05 14:28:51 +01:00
Pablo Galindo Salgado	2a96282d9a	gh-153569: consolidate tokenizer input, layout, and diagnostics (#157055) ... * gh-153569: move tokenizer input state and relocation into the reader * gh-153569: borrow diagnostic lines through the source API * gh-153569: group indentation and logical-line state * gh-153569: remove unused tokenizer cursor and source lookup APIs * gh-153569: report tokenizer diagnostics without rewinding the scanner * gh-153569: use tokenizer views and remove obsolete API remnants	2026-09-24 16:26:47 +00:00
Pablo Galindo Salgado	d8a1072491	gh-153568: Reduce parser allocations (#157374)	2026-09-16 16:10:40 -04:00
Pablo Galindo Salgado	0d6671d744	gh-153568: Skip newline translation for source without carriage returns (#153584) ... The tokenizer copied every input byte by byte to normalize newlines, but source without a carriage return needs no translation and can be copied verbatim.	2026-09-14 16:24:02 +00:00
Pablo Galindo Salgado	7bfa97f4dc	gh-153569: hide tokenizer state behind token and source operations (#156654) ... Make pegen and _tokenize consume tokenizer operations instead of accessing the tokenizer layout. Return token kind, span, location, and raw-string context together, so parser actions use the context of the cached token. Represent persistent scanner positions as offsets and remove pointer save/restore bookkeeping when input storage grows.	2026-09-12 14:30:00 +01:00
Pablo Galindo Salgado	f5dbcbafae	gh-153569: Unify tokenizer input readers (#156472)	2026-08-28 17:26:05 +02:00