Skip to content

FIX: escape literal hyphens in TokenBijectionConverter to avoid delimiter collision - #3129

Open
Kushal Talati (kushaltalati) wants to merge 2 commits into
microsoft:mainfrom
kushaltalati:fix/token-bijection-hyphen
Open

Kushal Talati (kushaltalati) wants to merge 2 commits into
microsoft:mainfrom
kushaltalati:fix/token-bijection-hyphen

Conversation

@kushaltalati

Copy link
Copy Markdown

Description

Closes #3128.

TokenBijectionConverter joins the code words of one plaintext word with - and passes every unmapped character through unchanged, so a literal hyphen in the prompt is emitted as the same character as the delimiter and lands exactly where an inserted delimiter would. e-mail and email encode to the same text, and decode() reads both back as email; self-harm comes back as selfharm. The class docstring promises the encoding is unambiguously decodable, and the teaching instructions give the target no way to write a literal hyphen.

This is the third instance of the collision class already fixed in this file for DigitBijectionConverter: #2696 (literal apostrophes vs. _CASE_MARKER) and #2741 (literal digits vs. digit tokens). The fix follows the same pattern:

  • encode() writes a literal hyphen as a doubled delimiter (--), which can never be produced by joining code words.
  • _decode_pattern gains a leading -- alternative so a doubled delimiter is consumed as one literal hyphen before any code-word run is considered, and decode() restores it.
  • get_teaching_instructions() tells the target to write a literal hyphen as two hyphens.
  • A comment next to _TOKEN_DELIMITER documents the collision, in the style of the existing _CASE_MARKER / _LITERAL_MARKER comments.

Hyphens that are not directly between two letters (a - b, -lead, trail-) already survived because the pattern requires a code word on both sides; they still round-trip, now via the escape.

Tests and Documentation

Added to tests/unit/converter/test_bijection_converter.py:

  • test_token_converter_round_trips_literal_hyphens, parametrized over e-mail, well-known, self-harm, Self-Harm, a - b, -lead, trail-, double--dash, ---.
  • test_token_converter_literal_hyphen_is_distinct_from_delimiter: e-mail and email encode differently, and the hyphenated form is the first code word, --, then the rest of the joined form.
  • test_token_converter_teaching_instructions_explain_hyphen_escape.

With only the source change reverted, 6 of the new cases fail (the four hyphenated words plus the two new assertions); with it, the file is 151 passed and tests/unit/converter is 2114 passed. ruff check and ruff format --check are clean. The converter notebook in doc/code/converters/1_text_to_text_converters.py uses a prompt without a hyphen, so its output is unchanged and no JupyText run was needed.

…iter collision

A plaintext hyphen is the same character as the delimiter that joins code
words, so e-mail encoded identically to email and decoded without the hyphen.
Double a literal hyphen on encode and collapse it on decode, matching the
apostrophe and digit escapes in DigitBijectionConverter (microsoft#2696, microsoft#2741).

Closes microsoft#3128

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BUG TokenBijectionConverter cannot represent a literal hyphen, so "e-mail" encodes identically to "email" and decodes without the hyphen

2 participants