Skip to content

[3.14] gh-151464: exclude '<>' token from tokenize output (GH-154854) - #158190

Open
pablogsal wants to merge 1 commit into
python:3.14from
pablogsal:gh-154854-3.14
Open

pablogsal wants to merge 1 commit into
python:3.14from
pablogsal:gh-154854-3.14

Conversation

@pablogsal

@pablogsal pablogsal commented Sep 25, 2026 •

Copy link
Copy Markdown
Member

Manual backport of #154854 to 3.14. The comparison grammar differs from main, so this adds the diagnostic to the 3.14 rules and regenerates the parser. The tokenizer, syntax, Barry, and future-import suites pass locally.

@pablogsal

Copy link
Copy Markdown
Member Author

The required-check aggregate is red because the two Windows ARM jobs fail before compiling CPython: this runner has Visual Studio 18 but lacks the v143 toolset requested by the 3.14 branch. ABI, lint, review gates, and the other required platform jobs passed. The focused tokenizer, syntax, Barry, and future-import suites also pass locally. I am merging this backport with the infrastructure failure noted here.

…onGH-154854)

* pythongh-151464: exclude '<>' token from tokenize output

Was:
```
$ echo '1 <> 2' | python -m tokenize
1,0-1,1:            NUMBER         '1'
1,2-1,4:            OP             '<>'
1,5-1,6:            NUMBER         '2'
1,6-1,7:            NEWLINE        '\n'
2,0-2,0:            ENDMARKER      ''
```

Now (regardless on ``__future__.barry_as_FLUFL`` import):
```
$ echo '1 <> 2' | ./python -m tokenize
1,0-1,1:            NUMBER         '1'
1,2-1,3:            OP             '<'
1,3-1,4:            OP             '>'
1,5-1,6:            NUMBER         '2'
1,6-1,7:            NEWLINE        '\n'
2,0-2,0:            ENDMARKER      ''
```
in accordance with the Grammar:
https://docs.python.org/3.14/reference/lexical_analysis.html#operators-and-delimiters

Also adds a custom error message for ``<>`` ("not equal" in Pascal
and Python 2).

* +1

* address review: lowercase and move invalid rule

* address review: news

* address review: move test_guido_as_bdfl_ineq_tokens()

* address review: revert _PyTokenizer_From* changes

* + revert unrelated change

* address review: remove whatsnew entry

(cherry picked from commit 198bc76)
@pablogsal

Copy link
Copy Markdown
Member Author

The VS 2026 toolset and UWP fixes have now merged in #158188. I rebased this backport onto the updated 3.14 branch, and its checks are running again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants