Skip to content

test(strings): cover Unicode whitespace and sentinel boundaries - #519

Merged
JohnnyWilson16 merged 1 commit into
FreshCode-Org:mainfrom
WilliamK112:codex/unicode-sentinel-tests-498
Oct 13, 2026
Merged

JohnnyWilson16 merged 1 commit into
FreshCode-Org:mainfrom
WilliamK112:codex/unicode-sentinel-tests-498

Conversation

@WilliamK112

Copy link
Copy Markdown
Contributor

Add six requested Unicode regression cases to tests/test_strings.py for object and pandas string columns. They cover default fd.clean() and explicit stripping, NBSP-padded sentinels/values, zero-width-space content, internal spacing, repair counts, and stripping disabled. Populated row identifiers keep sentinel-to-missing conversions observable without dropping rows.

NBSP is stripped at cell boundaries and can expose a sentinel; U+200B follows the existing Python whitespace rules and remains content. This PR changes tests only.

Fixes #498.

Validation:

  • pytest tests/test_strings.py --no-cov: all 29 cases pass on pandas 2.3.3; 26 pass and three existing Arrow cases skip on pandas 1.5.3.
  • Full fast lane: 7,608 passed, 49 skipped, five failures reproduced on clean main@ba3f01f in this Windows environment, 94.98% coverage.
  • Full Ruff check, Linux-platform mypy with CI's NumPy cap, and diff checks pass.

Draft pending #518: the floating Polars 2.0 matrix has an existing all-columns-dropped parity failure, addressed by that PR. Keep this test-only scope separate; refresh from main and mark ready after the fix lands and hosted checks pass.

Signed-off-by: WilliamK112 <164879897+WilliamK112@users.noreply.github.com>
@coderabbitai

coderabbitai Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 7beec061-fb67-4ce6-8c6e-32d50eae54ef
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@JohnnyWilson16 JohnnyWilson16 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thorough test coverage for #498. Correctly verifies U+00A0 stripping, U+200B preservation, and audit report counts across string and object dtypes without test flakiness. Ready to merge.

@JohnnyWilson16
JohnnyWilson16 marked this pull request as ready for review October 13, 2026 12:18
@JohnnyWilson16
JohnnyWilson16 merged commit f393aed into FreshCode-Org:main Oct 13, 2026
15 of 19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add unit tests for unicode whitespace and zero-width spaces in sentinel normalization

2 participants