Repository navigation
FIX: escape literal hyphens in TokenBijectionConverter to avoid delimiter collision - #3129
Open
Kushal Talati (kushaltalati) wants to merge 2 commits into
Open
Kushal Talati (kushaltalati) wants to merge 2 commits into
Kushal Talati (kushaltalati) wants to merge 2 commits into
Conversation
…iter collision A plaintext hyphen is the same character as the delimiter that joins code words, so e-mail encoded identically to email and decoded without the hyphen. Double a literal hyphen on encode and collapse it on decode, matching the apostrophe and digit escapes in DigitBijectionConverter (microsoft#2696, microsoft#2741). Closes microsoft#3128
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Closes #3128.
TokenBijectionConverterjoins the code words of one plaintext word with-and passes every unmapped character through unchanged, so a literal hyphen in the prompt is emitted as the same character as the delimiter and lands exactly where an inserted delimiter would.e-mailandemailencode to the same text, anddecode()reads both back asemail;self-harmcomes back asselfharm. The class docstring promises the encoding is unambiguously decodable, and the teaching instructions give the target no way to write a literal hyphen.This is the third instance of the collision class already fixed in this file for
DigitBijectionConverter: #2696 (literal apostrophes vs._CASE_MARKER) and #2741 (literal digits vs. digit tokens). The fix follows the same pattern:encode()writes a literal hyphen as a doubled delimiter (--), which can never be produced by joining code words._decode_patterngains a leading--alternative so a doubled delimiter is consumed as one literal hyphen before any code-word run is considered, anddecode()restores it.get_teaching_instructions()tells the target to write a literal hyphen as two hyphens._TOKEN_DELIMITERdocuments the collision, in the style of the existing_CASE_MARKER/_LITERAL_MARKERcomments.Hyphens that are not directly between two letters (
a - b,-lead,trail-) already survived because the pattern requires a code word on both sides; they still round-trip, now via the escape.Tests and Documentation
Added to
tests/unit/converter/test_bijection_converter.py:test_token_converter_round_trips_literal_hyphens, parametrized overe-mail,well-known,self-harm,Self-Harm,a - b,-lead,trail-,double--dash,---.test_token_converter_literal_hyphen_is_distinct_from_delimiter:e-mailandemailencode differently, and the hyphenated form is the first code word,--, then the rest of the joined form.test_token_converter_teaching_instructions_explain_hyphen_escape.With only the source change reverted, 6 of the new cases fail (the four hyphenated words plus the two new assertions); with it, the file is 151 passed and
tests/unit/converteris 2114 passed.ruff checkandruff format --checkare clean. The converter notebook indoc/code/converters/1_text_to_text_converters.pyuses a prompt without a hyphen, so its output is unchanged and no JupyText run was needed.