cpython

0 Fork 0

mirror of https://github.com/python/cpython.git synced 2026-10-28 10:25:05 +08:00

Files

T

Sergey B Kirpichev 198bc76836 gh-151464: exclude '<>' token from tokenize output (#154854 )

* gh-151464: exclude '<>' token from tokenize output

Was:
```
$ echo '1 <> 2' | python -m tokenize
1,0-1,1:            NUMBER         '1'
1,2-1,4:            OP             '<>'
1,5-1,6:            NUMBER         '2'
1,6-1,7:            NEWLINE        '\n'
2,0-2,0:            ENDMARKER      ''
```

Now (regardless on ``__future__.barry_as_FLUFL`` import):
```
$ echo '1 <> 2' | ./python -m tokenize
1,0-1,1:            NUMBER         '1'
1,2-1,3:            OP             '<'
1,3-1,4:            OP             '>'
1,5-1,6:            NUMBER         '2'
1,6-1,7:            NEWLINE        '\n'
2,0-2,0:            ENDMARKER      ''
```
in accordance with the Grammar:
https://docs.python.org/3.14/reference/lexical_analysis.html#operators-and-delimiters

Also adds a custom error message for ``<>`` ("not equal" in Pascal
and Python 2).

* +1

* address review: lowercase and move invalid rule

* address review: news

* address review: move test_guido_as_bdfl_ineq_tokens()

* address review: revert _PyTokenizer_From* changes

* + revert unrelated change

* address review: remove whatsnew entry

2026-09-25 19:51:59 +01:00

lexer

gh-151464: exclude '<>' token from tokenize output (#154854 )

2026-09-25 19:51:59 +01:00

tokenizer

gh-153569: consolidate tokenizer input, layout, and diagnostics (#157055 )

2026-09-24 16:26:47 +00:00

action_helpers.c

gh-151464: exclude '<>' token from tokenize output (#154854 )

2026-09-25 19:51:59 +01:00

asdl_c.py

gh-157407: Fix a compilation error on Windows. (#158048 )