Jeong, YunWon f090ae3237 Rewrite _tokenize with 2-phase model ...

Replace per-line reparsing with single-pass tokenization:
- Read all lines via readline, parse once, yield tokens
- Fix token type values (COMMENT=65, NL=66, OP=55)
- Fix NEWLINE/NL end positions and implicit newline handling
- Fix DEDENT positions via look-ahead to next non-DEDENT token
- Handle FSTRING_MIDDLE brace unescaping ({{ → {, }} → })
- Emit implicit NL before ENDMARKER when source lacks trailing newline
- Raise IndentationError from lexer errors
- Remove 13 expectedFailure marks for now-passing tests

2026-03-10 11:28:16 +09:00

..

src

Rewrite _tokenize with 2-phase model

2026-03-10 11:28:16 +09:00

build.rs

reason inside #[allow] (#7049)

2026-02-08 13:11:50 +00:00

Cargo.toml

Base implementation of _tokenize module

2026-03-10 11:28:16 +09:00