The tokenizer copied every input byte by byte to normalize newlines,
but source without a carriage return needs no translation and can be
copied verbatim.
* gh-153568: Make the parser's per-rule stack check an inline comparison
The parser now caches the thread state and stack limit up front and
only falls back to the full check when the stack is nearly exhausted.
* fixup! gh-153568: Make the parser's per-rule stack check an inline comparison
* fixup! gh-153568: Make the parser's per-rule stack check an inline comparison
Make pegen and _tokenize consume tokenizer operations instead of accessing the tokenizer layout. Return token kind, span, location, and raw-string context together, so parser actions use the context of the cached token.
Represent persistent scanner positions as offsets and remove pointer save/restore bookkeeping when input storage grows.
Formatted-string expressions and comments can outlive the buffer window where
scanning began. Pointer boundaries require buffer relocation to repair tokenizer
state. Record logical source ranges instead. Retain the active input window while a
formatted string is open, give each mode ownership of its comment spans, and
materialize text through shared span views.
Represent token boundaries as logical source offsets with explicit start
and end locations, and adapt pegen and _tokenize to consume them.
Use SourceText to own decoded input across all reader modes. Streaming
readers discard consumed windows while preserving logical offsets and
reusing the allocation. Prepared and interactive input retain their source.
Force source relocation on growth in debug builds and add coverage for
formatted-string growth, interactive input, storage reuse, and offset
limits. Scanner positions remain pointers, rebased when storage moves.
The callers of _PyTokenizer_syntaxerror_known_range() pass columns in bytes,
but SyntaxError.offset and end_offset are columns in characters.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
If the console cancels a read because of Ctrl+C, but the SIGINT handler
does not raise an exception, restart the read instead of ending it as if
at end of file.
Co-authored-by: Valeriya Sinevich <valeriya.sinevich@phystech.edu>
Co-authored-by: Zackery Spytz <zspytz@gmail.com>
Move the invalid_factor rule from term to factor and the
invalid_arithmetic rule from shift_expr to sum, so that they are
tried at all positions where the invalid construct can occur.
Previously a generic "invalid syntax" error was reported for
constructs like "1 << 2 + not x" or "1 * + not x".
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Add '+' alternatives to signed_number and signed_real_number grammar
rules, mirroring how unary minus is already handled for pattern matching.
Unary plus is a no-op on numbers so the value is returned directly without
wrapping in a UnaryOp node.
Rename from _Py_INTERNAL_ABI_SLOT to _Py_ABI_SLOT
and define the macro using _PyABIInfo_DEFAULT.
Use the ABI slot in stdlib extension modules to enable running
a check of ABI version compatibility.
_tkinter, _tracemalloc and readline don't use the slots, hence they need
explicit handling.
Co-authored-by: Victor Stinner <vstinner@python.org>
Guard against NULL pointer arithmetic in `_PyLexer_remember_fstring_buffers`
and `_PyLexer_restore_fstring_buffers`. When `start` or `multi_line_start`
are NULL (uninitialized in tok_mode_stack[0]), performing `NULL - tok->buf`
is undefined behavior. Add explicit NULL checks to store -1 as sentinel
and restore NULL accordingly.
Add test_lexer_buffer_realloc_with_null_start to test_repl.py that
exercises the code path where the lexer buffer is reallocated while
tok_mode_stack[0] has NULL start/multi_line_start pointers. This
triggers _PyLexer_remember_fstring_buffers and verifies the NULL
checks prevent undefined behavior.
Pablo Galindo Salgado