Pablo Galindo SalgadoandSergey B Kirpichev 66f8dbe0f5 [3.14] gh-151464: exclude '<>' token from tokenize output (GH-154854) (#158190) ...

* gh-151464: exclude '<>' token from tokenize output

Was:
```
$ echo '1 <> 2' | python -m tokenize
1,0-1,1:            NUMBER         '1'
1,2-1,4:            OP             '<>'
1,5-1,6:            NUMBER         '2'
1,6-1,7:            NEWLINE        '\n'
2,0-2,0:            ENDMARKER      ''
```

Now (regardless on ``__future__.barry_as_FLUFL`` import):
```
$ echo '1 <> 2' | ./python -m tokenize
1,0-1,1:            NUMBER         '1'
1,2-1,3:            OP             '<'
1,3-1,4:            OP             '>'
1,5-1,6:            NUMBER         '2'
1,6-1,7:            NEWLINE        '\n'
2,0-2,0:            ENDMARKER      ''
```
in accordance with the Grammar:
https://docs.python.org/3.14/reference/lexical_analysis.html#operators-and-delimiters

Also adds a custom error message for ``<>`` ("not equal" in Pascal
and Python 2).

* +1

* address review: lowercase and move invalid rule

* address review: news

* address review: move test_guido_as_bdfl_ineq_tokens()

* address review: revert _PyTokenizer_From* changes

* + revert unrelated change

* address review: remove whatsnew entry

(cherry picked from commit 198bc76836)

Co-authored-by: Sergey B Kirpichev <skirpichev@gmail.com>

2026-09-25 20:38:45 +00:00

..

python.gram

[3.14] gh-151464: exclude '<>' token from tokenize output (GH-154854) (#158190)

2026-09-25 20:38:45 +00:00

Tokens

[3.14] gh-133555: Allow regenerating the parser with Python < 3.14 (GH-133557) (#133630)

2025-05-08 01:54:42 +00:00