AST nodes had an explicit `__dict__` slot, so every node built by
`ast.parse` allocated its own dict. With `Py_TPFLAGS_MANAGED_DICT`,
node attributes are stored as inline values with shared keys instead.
`ast.parse` is ~15.8% faster, walking trees is ~15.9% faster, and
the memory used by the AST trees is reduced by ~51.6%.
Literal parts of f-strings and t-strings are decoded only after the
whole string has been parsed, so a decode error took the location of
the last token. The resulting SyntaxError pointed at the closing quote,
and for multi-line strings it could be on the wrong line.
Raise the error at the literal part being decoded instead.
A bare name or number followed by a token that cannot extend an expression now produces its AST node directly. This avoids the expression precedence chain for common arguments, assignments, and subscripts.
Rules declare the hook with a fastpath flag. The Rule constructor exposes memo and fastpath attributes, and the C generator emits the hook call.
Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
Add an @include meta to pegen: the rules of an included grammar file are
appended to the including file's rules. The C generator emits the rules
of each included file into a separate C file next to the main output.
Use it to move the invalid_* rules from Grammar/python.gram into
Grammar/invalid.gram, which generates Parser/parser_invalid.c.
`KeywordCollectorVisitor` checked the quoted spelling of a keyword against
keys stored without quotes, so the check never matched and every
occurrence of a keyword in the grammar took a new token type. The 35
hard keywords were numbered between 500 and 721 instead of 500 to 534.
Since gh-105042, `tokenize` tolerates mismatched brackets. That also let
a mismatched closing bracket pop the brace that opened an f-string
replacement field, which the f-string state relies on. In `f'{]a'`, this
triggered an assertion failure in _PyLexer_scan_string.
Check the replacement field brace in both modes.
Move source access and logical spans into the tokenizer source API, and keep input and encoding ownership in the reader. Export the incremental decoder helper for parser generator extensions.
* gh-153569: move tokenizer input state and relocation into the reader
* gh-153569: borrow diagnostic lines through the source API
* gh-153569: group indentation and logical-line state
* gh-153569: remove unused tokenizer cursor and source lookup APIs
* gh-153569: report tokenizer diagnostics without rewinding the scanner
* gh-153569: use tokenizer views and remove obsolete API remnants
Replace soft deprecated PyBytes_FromStringAndSize(NULL, size) with a
simple PyMem_Malloc(). Avoid also "s = buf" to make the code easier to follow
(and easier to debug).
Add non-ASCII tests to test_string_literals.
* gh-153568: Don't materialize parser token text that is never read
Only tokens whose text is actually consumed get a bytes object;
operators and structural tokens no longer allocate one.
* Update Parser/pegen.c
Co-authored-by: Maurycy Pawłowski-Wieroński <maurycy@maurycy.com>
* gh-153568: Preserve keyword token text
---------
Co-authored-by: Maurycy Pawłowski-Wieroński <maurycy@maurycy.com>
The tokenizer copied every input byte by byte to normalize newlines,
but source without a carriage return needs no translation and can be
copied verbatim.
* gh-153568: Make the parser's per-rule stack check an inline comparison
The parser now caches the thread state and stack limit up front and
only falls back to the full check when the stack is nearly exhausted.
* fixup! gh-153568: Make the parser's per-rule stack check an inline comparison
* fixup! gh-153568: Make the parser's per-rule stack check an inline comparison
Make pegen and _tokenize consume tokenizer operations instead of accessing the tokenizer layout. Return token kind, span, location, and raw-string context together, so parser actions use the context of the cached token.
Represent persistent scanner positions as offsets and remove pointer save/restore bookkeeping when input storage grows.
Formatted-string expressions and comments can outlive the buffer window where
scanning began. Pointer boundaries require buffer relocation to repair tokenizer
state. Record logical source ranges instead. Retain the active input window while a
formatted string is open, give each mode ownership of its comment spans, and
materialize text through shared span views.
Represent token boundaries as logical source offsets with explicit start
and end locations, and adapt pegen and _tokenize to consume them.
Use SourceText to own decoded input across all reader modes. Streaming
readers discard consumed windows while preserving logical offsets and
reusing the allocation. Prepared and interactive input retain their source.
Force source relocation on growth in debug builds and add coverage for
formatted-string growth, interactive input, storage reuse, and offset
limits. Scanner positions remain pointers, rebased when storage moves.
The callers of _PyTokenizer_syntaxerror_known_range() pass columns in bytes,
but SyntaxError.offset and end_offset are columns in characters.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
If the console cancels a read because of Ctrl+C, but the SIGINT handler
does not raise an exception, restart the read instead of ending it as if
at end of file.
Co-authored-by: Valeriya Sinevich <valeriya.sinevich@phystech.edu>
Co-authored-by: Zackery Spytz <zspytz@gmail.com>
Move the invalid_factor rule from term to factor and the
invalid_arithmetic rule from shift_expr to sum, so that they are
tried at all positions where the invalid construct can occur.
Previously a generic "invalid syntax" error was reported for
constructs like "1 << 2 + not x" or "1 * + not x".
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
sobolevn