[Kushal Talati (kushaltalati)]

Description

Closes #3126.

PlagiarismScorer._tokenize dropped any punctuation that is not a fraction slash without inserting a token boundary, so a decimal point or thousands separator between two digits joined the digits on either side: 2.5 tokenized as 25, and 25.00 and 2,500 both as 2500. Because reference and response then tokenized identically, the verbatim fast path returned 1.0 on all three metrics, and a FloatScaleThresholdScorer on top returned True for a response that differs from the reference by a factor of ten or a hundred.

This extends the digit-flanked separator rule introduced in #3053 for fractions. A new _NUMERIC_SEPARATORS table lists the separators that are kept when they sit between two decimal digits and the form each takes in the token: both slash variants still fold to /, and . and , are kept as written. Punctuation anywhere else behaves exactly as before (a/b → ab, don't → dont, trailing 2. → 2), so no existing test changes.

Trade-off: a response that writes 1000 where the reference wrote 1,000 now scores slightly below 1.0 rather than matching exactly. That is the same trade #3053 made for fractions and seems right for a scorer whose job is detecting reproduction.

Tests and Documentation

Two existing parametrized tests in tests/unit/score/test_plagiarism_scorer.py gain cases:

• test_tokenize_fraction_and_mark_boundaries: 2.5, 25.00, 2,500, 1.2.3 and fullwidth ２.５ each stay one token; 2., .5, 2..5, a.b, a,b, 1,a keep the previous behaviour.
• test_score_text_fraction_not_concatenated_async: 2.5/25, 25.00/2500, 2,500/2500, 25.00/2,500 score 0.75 / 0.75 / 2⁄3 across LCS / Levenshtein / Jaccard and False through the threshold scorer, instead of 1.0 and True.

With only the source change reverted the 17 new cases fail; with it they pass. tests/unit/score: 3614 passed, 44 skipped. ruff check and ruff format --check are clean. The _tokenize docstring is updated; no notebook documents the tokenizer's separator handling, so no JupyText run was needed.

[Kushal Talati (kushaltalati)]

Kushal Talati (Kushal Talati (@kushaltalati)) please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.

@microsoft-github-policy-service agree [company="{your company}"]

Options:

• (default - no company specified) I have sole ownership of intellectual property rights to my Submissions and I am not making Submissions in the course of work for my employer.

@microsoft-github-policy-service agree

• (when company given) I am making Submissions in the course of work for my employer (or my employer has intellectual property rights in my Submissions by contract or applicable law). I have permission from my employer to make Submissions and enter into this Agreement on behalf of my employer. By signing below, the defined term “You” includes me and my employer.

@microsoft-github-policy-service agree company="Microsoft"

Contributor License Agreement

@microsoft-github-policy-service agree