Skip to content

join: split default fields at the blanks of the locale - #15856

Open
komal0000 wants to merge 4 commits into
uutils:mainfrom
komal0000:fix/join-locale-blanks
Open

komal0000 wants to merge 4 commits into
uutils:mainfrom
komal0000:fix/join-locale-blanks

Conversation

@komal0000

Copy link
Copy Markdown
Contributor

Without -t, join split fields only at space, tab and newline. In a UTF-8 locale GNU join also splits at multi-byte blanks such as U+3000 IDEOGRAPHIC SPACE, so a line like "root<U+3000>/bin/sh" took the whole line as its key and never paired with "root" in the other file.

Walk non-ASCII lines by character with Encoding::blank_len(), the class cut -w already uses. Single-byte locales and ASCII lines keep the memchr loop.

Fixes #15771

Without -t, join split fields only at space, tab and newline. In a UTF-8
locale GNU join also splits at multi-byte blanks such as U+3000
IDEOGRAPHIC SPACE, so a line like "root<U+3000>/bin/sh" took the whole
line as its key and never paired with "root" in the other file.

Walk non-ASCII lines by character with Encoding::blank_len(), the class
cut -w already uses. Single-byte locales and ASCII lines keep the memchr
loop.

Fixes uutils#15771

Co-authored-by: Sylvestre Ledru <733326+sylvestre@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings October 26, 2026 03:58

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing. Review the Copilot code review billing settings.

@codspeed

codspeed Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

Merging this PR will degrade performance by 3.3%

❌ 1 regressed benchmark
✅ 9 untouched benchmarks
⏩ 462 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ CPU Simulation join_partial_overlap 18.6 ms 19.2 ms -3.3%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing komal0000:fix/join-locale-blanks (8c37b07) with main (3544ef7)

Open in CodSpeed

Footnotes

  1. 462 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

…anks

Only lines that contain a 0xE1, 0xE2 or 0xE3 byte can hold a multi-byte
blank, so other non-ASCII lines (accented Latin text, for example) no
longer go through the per-character scan.
Copilot AI balanced review requested due to automatic review settings October 26, 2026 13:57

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing. Review the Copilot code review billing settings.

Copilot AI balanced review requested due to automatic review settings October 30, 2026 10:10

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing. Review the Copilot code review billing settings.

Copilot AI balanced review requested due to automatic review settings October 30, 2026 10:11

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing. Review the Copilot code review billing settings.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

join: multi-byte blanks like U+3000 do not split fields, unlike GNU 9.5+

2 participants