sobolevn
7b12a415c8
gh-152631: Fix ast.parse on lazy imports with older feature_version ( #152632 )
2026-10-25 20:35:59 +00:00
Pablo Galindo Salgado and Lysandros Nikolaou
b77cb891e9
gh-153568: Skip the expression precedence chain for single-token atoms ( #153575 )
...
A bare name or number followed by a token that cannot extend an expression now produces its AST node directly. This avoids the expression precedence chain for common arguments, assignments, and subscripts.
Rules declare the hook with a fastpath flag. The Rule constructor exposes memo and fastpath attributes, and the C generator emits the hook call.
Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com >
2026-10-23 14:23:10 -07:00
Lysandros Nikolaou
e548b472f3
gh-153195: Allow splitting the PEG grammar into several files ( #159917 )
...
Add an @include meta to pegen: the rules of an included grammar file are
appended to the including file's rules. The C generator emits the rules
of each included file into a separate C file next to the main output.
Use it to move the invalid_* rules from Grammar/python.gram into
Grammar/invalid.gram, which generates Parser/parser_invalid.c.
2026-10-23 12:58:39 -07:00
Lysandros Nikolaou
0b8bb7154c
gh-160284: Number each keyword once in the PEG generator ( #160290 )
...
`KeywordCollectorVisitor` checked the quoted spelling of a keyword against
keys stored without quotes, so the check never matched and every
occurrence of a keyword in the grammar took a new token type. The 35
hard keywords were numbered between 500 and 721 instead of 500 to 534.
2026-10-22 19:25:07 +00:00
Pablo Galindo Salgado
0b8aa89a05
gh-153569: Simplify tokenizer state ( #158124 )
...
* Use diagnostic line for missing-brace errors
* Keep implicit prepared newline metadata in the reader
* Reuse parsed formatted string kind in lexer
* Transfer owned tokenizer encoding strings
* Remove unused tokenizer state and redundant branches
* Avoid rescanning partial decoded tokenizer lines
* Reuse prepared tokenizer line terminators for CR detection
* Reuse interactive tokenizer chunks without carriage returns
* Release tokenizer constructor resources on allocation failure
* Narrow interactive newline normalization wrapper
* Close duplicated tokenizer descriptors when fdopen fails
* Preserve cached tokenizer lines when newline allocation fails
* Collect cycles involving tokenizer readline callbacks
* Preserve buffered text when decoding source files
* Clear the transferred syntax error line before cleanup
* Propagate tokenizer column conversion failures
* Allocate parser repetition buffers on first match
* Allocate type comment storage only when needed
* Reuse decoded multiline text when counting token columns
* Skip empty decoded tokenizer chunks
* Check parser array growth before allocating
* Name tokenizer bounds and derive byte lengths from their data
* Set the tokenizer exception when file input size overflows
* gh-153569: Send exact line endings in the REPL test
* Share parser repetition buffer growth
* Fold Unicode input coverage into the existing REPL test
* Keep parser array growth policy private
* Drop obsolete bitmap stress from source discard test
* Preserve input failures while scanning string literals
* fixup! Preserve input failures while scanning string literals
* fixup! fixup! Preserve input failures while scanning string literals
* gh-153569: Remove assertion-only lexer bookkeeping
2026-10-05 14:28:51 +01:00
Sergey B Kirpichev
198bc76836
gh-151464: exclude '<>' token from tokenize output ( #154854 )
...
* gh-151464: exclude '<>' token from tokenize output
Was:
```
$ echo '1 <> 2' | python -m tokenize
1,0-1,1: NUMBER '1'
1,2-1,4: OP '<>'
1,5-1,6: NUMBER '2'
1,6-1,7: NEWLINE '\n'
2,0-2,0: ENDMARKER ''
```
Now (regardless on ``__future__.barry_as_FLUFL`` import):
```
$ echo '1 <> 2' | ./python -m tokenize
1,0-1,1: NUMBER '1'
1,2-1,3: OP '<'
1,3-1,4: OP '>'
1,5-1,6: NUMBER '2'
1,6-1,7: NEWLINE '\n'
2,0-2,0: ENDMARKER ''
```
in accordance with the Grammar:
https://docs.python.org/3.14/reference/lexical_analysis.html#operators-and-delimiters
Also adds a custom error message for ``<>`` ("not equal" in Pascal
and Python 2).
* +1
* address review: lowercase and move invalid rule
* address review: news
* address review: move test_guido_as_bdfl_ineq_tokens()
* address review: revert _PyTokenizer_From* changes
* + revert unrelated change
* address review: remove whatsnew entry
2026-09-25 19:51:59 +01:00
7884f48af4
gh-152708: Fix AST generation asymmetry for unary positive in match statements ( #152712 )
...
Co-authored-by: Petr Viktorin <encukou@gmail.com >
Co-authored-by: johnslavik <bartosz@ilikepython.com >
2026-09-23 19:56:53 -07:00
Pablo Galindo Salgado
d8a1072491
gh-153568: Reduce parser allocations ( #157374 )
2026-09-16 16:10:40 -04:00
Pablo Galindo Salgado
89fe9857dd
gh-153568: Reuse shared prefixes in generated parser rules ( #157465 )
2026-09-14 20:26:18 -04:00
Pablo Galindo Salgado
c3f23734a8
gh-153568: Share expression parsing with assignment targets ( #157450 )
...
* gh-153568: Share expression parsing with assignment targets
* Clarify target parsing contracts and AST expectations
2026-09-14 16:53:25 +00:00
Pablo Galindo Salgado
f88d6687b6
gh-153568: Make the parser's per-rule stack check an inline comparison ( #153573 )
...
* gh-153568: Make the parser's per-rule stack check an inline comparison
The parser now caches the thread state and stack limit up front and
only falls back to the full check when the stack is nearly exhausted.
* fixup! gh-153568: Make the parser's per-rule stack check an inline comparison
* fixup! gh-153568: Make the parser's per-rule stack check an inline comparison
2026-09-14 16:08:59 +00:00
Pablo Galindo Salgado
ac86e12d74
gh-153568: Stop re-walking memo lists when growing left-recursive rules ( #153574 )
...
The left-recursion driver now keeps a direct reference to its memo
entry and updates it in place on every iteration.
2026-09-14 11:41:14 -04:00
Petr Viktorin and Bartosz Sławecki
a85830c7f8
gh-154775: Don't accept duplicate plus sign when matching complex number (GH-154776)
...
Co-authored-by: Bartosz Sławecki <bartosz@ilikepython.com >
2026-07-28 09:19:46 +02:00
Serhiy Storchaka and Claude Fable 5
1034e73f69
gh-153171: Produce specialized syntax errors for 'not' after an operator in more positions (GH-153175)
...
Move the invalid_factor rule from term to factor and the
invalid_arithmetic rule from shift_expr to sum, so that they are
tried at all positions where the invalid construct can occur.
Previously a generic "invalid syntax" error was reported for
constructs like "1 << 2 + not x" or "1 * + not x".
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
2026-07-06 13:00:05 +03:00
sobolevn
ec5f154e33
gh-150459: Fix SyntaxError message for from x lazy import y ( #150877 )
2026-07-04 14:17:58 +01:00
Aniket
0f501b9b74
gh-146495: Improve SyntaxError message for && and || operators ( #150906 )
2026-06-27 18:57:10 +02:00
Bartosz Sławecki
448d7b96c1
gh-145239: Accept unary plus literal pattern ( #148566 )
...
Add '+' alternatives to signed_number and signed_real_number grammar
rules, mirroring how unary minus is already handled for pattern matching.
Unary plus is a no-op on numbers so the value is returned directly without
wrapping in a UnaryOp node.
2026-04-23 22:07:28 +03:00
Stan Ulbrych
e1c224624a
gh-145783: Propagate errors raised in NEW_TYPE_COMMENT ( #145784 )
2026-03-13 11:05:20 +00:00
Pablo Galindo Salgado and Bartosz Sławecki
09e8c38231
gh-145241: specialize SyntaxError for single trailing-comma with item ( #145282 )
...
Co-authored-by: Bartosz Sławecki <bartosz@ilikepython.com >
2026-02-28 02:24:06 +00:00
46d5106cfa
gh-142349: Implement PEP 810 - Explicit lazy imports ( #142351 )
...
Co-authored-by: T. Wouters <twouters@meta.com >
Co-authored-by: Brittany Reynoso <breynoso@meta.com >
Co-authored-by: Dino Viehland <dinoviehland@meta.com >
2026-02-12 00:15:33 +00:00
adam j hartz and Jelle Zijlstra
ccbe41e27c
gh-143055: Implementation of PEP 798 ( #143056 )
...
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com >
2026-01-30 20:37:52 -08:00
Pablo Galindo Salgado
c433986005
gh-142236: Improve error location for missing comma in string concatenations ( #142330 )
2025-12-11 14:47:26 +00:00
Victor Stinner
7fe1a18b77
gh-130396: Remove _Py_ReachedRecursionLimitWithMargin() function ( #141951 )
...
Move the private function to the internal C API (pycore_ceval.h).
2025-11-27 12:32:00 +01:00
Brian Schubert
3dab11f888
gh-138944: Fix SyntaxError message for invalid syntax following valid import-as statement ( #138945 )
2025-10-26 22:35:21 +00:00
sobolevn and Bartosz Sławecki
92c0c45563
gh-138857: Improve error message for case outside of match ( #138858 )
...
* gh-138857: Improve error message for `case` outside of `match`
---------
Co-authored-by: Bartosz Sławecki <bartosz@ilikepython.com >
2025-10-24 11:20:54 +02:00
Bartosz Sławecki
b3b0d75069
gh-140253: Improve the syntax error from an ill-positioned double-star subpattern ( #140254 )
2025-10-22 21:29:14 +03:00
sobolevn
cf8f36fe0a
gh-138716: Fix assert a := b syntax error message ( #138718 )
2025-09-10 16:05:16 +03:00
sobolevn
6bc65c30ff
gh-136616: Improve assert syntax error messages ( #136653 )
2025-09-09 23:26:22 +03:00
Pablo Galindo Salgado
4e08a9f97a
gh-137078: Fix keyword typo recognition when executed over files ( #137079 )
2025-08-15 15:14:13 +00:00
c5e77af131
gh-132661: Disallow Template/str concatenation after PEP 750 spec update ( #135996 )
...
Co-authored-by: sobolevn <mail@sobolevn.me >
Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com >
2025-07-21 08:44:26 +02:00
sobolevn
7e33558455
gh-135422: Fix regression in SyntaxError messages after #134036 ( #135423 )
2025-06-30 21:52:26 +03:00
Bénédikt Tran
754e7c9b51
gh-133157: remove usage of _Py_NO_SANITIZE_UNDEFINED in Parser/pegen.c ( #134048 )
2025-06-10 01:08:30 +01:00
sobolevn
0d9ccc87a2
gh-134036: Improve error messages for invalid raise statements ( #134077 )
2025-06-06 01:51:06 +01:00
Victor Stinner
6e80f11eb5
gh-135028: Increase parser MAXSTACK for nested parenthesis ( #135031 )
2025-06-03 08:40:45 +02:00
sobolevn
84914ad0e5
gh-133999: Fix except parsing regression in 3.14 ( #134035 )
2025-05-17 17:57:02 +03:00
Stan Ulbrych
dbca27cfca
gh-133379: Fix misuse of the term "arguments" in error messages (GH-133382)
...
The right term is "parameters".
2025-05-10 15:00:43 +03:00
sobolevn
ca0a96dfaa
gh-133194: Fix regression with PEP 758 parsing on older feature_version ( #133289 )
...
gh-133192: Fix regression with PEP 758 parsing on older `feature_version`
2025-05-03 10:33:14 +03:00
sobolevn
a6ddd078d0
gh-123539: Improve SyntaxError msg for import as with not a name ( #123629 )
2025-05-02 08:34:13 +00:00
sobolevn
5cdd49b3f4
gh-133196: Guard PEP 750 grammar with CHECK_VERSION ( #133225 )
2025-05-01 09:17:07 +02:00
sobolevn
b1f893875b
gh-133194: Add CHECK_VERSION to new PEP758 grammar ( #133195 )
2025-04-30 13:39:26 +03:00
60202609a2
gh-132661: Implement PEP 750 ( #132662 )
...
Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com >
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com >
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com >
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com >
Co-authored-by: Wingy <git@wingysam.xyz >
Co-authored-by: Koudai Aono <koxudaxi@gmail.com >
Co-authored-by: Dave Peck <davepeck@gmail.com >
Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu >
Co-authored-by: Paul Everitt <pauleveritt@me.com >
Co-authored-by: sobolevn <mail@sobolevn.me >
2025-04-30 11:46:41 +02:00
Steele Farnsworth
99b71efe8e
gh-129858: Special syntax error for elif block after else ( #129902 )
2025-04-25 01:25:48 +00:00
Pablo Galindo Salgado and Łukasz Langa
bf3a0a1c0f
gh-132449: Improve syntax error messages for keywords with typos ( #132450 )
...
Signed-off-by: Pablo Galindo <pablogsal@gmail.com >
Co-authored-by: Łukasz Langa <lukasz@langa.pl >
2025-04-22 11:01:55 +02:00
Pablo Galindo Salgado
c2ac662f28
gh-131831: Implement PEP 758 – Allow except and except* expressions without parentheses ( #131833 )
2025-04-01 19:04:56 +00:00
Mark Shannon
014223649c
GH-130396: Use computed stack limits on linux (GH-130398)
...
* Implement C recursion protection with limit pointers for Linux, MacOS and Windows
* Remove calls to PyOS_CheckStack
* Add stack protection to parser
* Make tests more robust to low stacks
* Improve error messages for stack overflow
2025-02-25 09:24:48 +00:00
Petr Viktorin
ef29104f7d
GH-91079: Revert "GH-91079: Implement C stack limits using addresses, not counters. (GH-130007)" for now (GH130413)
...
Revert "GH-91079: Implement C stack limits using addresses, not counters. (GH-130007)" for now
Unfortunatlely, the change broke some buildbots.
This reverts commit 2498c22fa0
2025-02-24 11:16:08 +01:00
Mark Shannon
2498c22fa0
GH-91079: Implement C stack limits using addresses, not counters. (GH-130007)
...
* Implement C recursion protection with limit pointers
* Remove calls to PyOS_CheckStack
* Add stack protection to parser
* Make tests more robust to low stacks
* Improve error messages for stack overflow
2025-02-19 11:44:57 +00:00
bcc9a5dddb
gh-129515: Clarify syntax error messages for conditional expressions ( #129880 )
...
Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com >
Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com >
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com >
2025-02-18 21:43:19 +00:00
Pablo Galindo Salgado
3bd3e09588
gh-125331: Allow the parser to activate future imports on the fly ( #125482 )
2025-02-14 04:54:56 +00:00
Pablo Galindo Salgado
6fb5138776
gh-88535: Improve syntax error for wrongly closed strings ( #26633 )
2025-02-13 01:30:20 +00:00
sobolevn