Skip to content

Write redirected and piped output as UTF-8, and stop the suite inheriting colour settings - #1753

Open
tleonhardt wants to merge 4 commits into
mainfrom
utf8-redirection-main
Open

Write redirected and piped output as UTF-8, and stop the suite inheriting colour settings#1753
tleonhardt wants to merge 4 commits into
mainfrom
utf8-redirection-main

Conversation

@tleonhardt

@tleonhardt tleonhardt commented Sep 7, 2026

Copy link
Copy Markdown
Member

Two independent fixes, both already reviewed and merged to consolidated_toolbar in #1752.
Ported here because main is the next patch release and both problems are present on it
today. The third commit from #1752 — a CommandToolbar race — is not included, because
that module does not exist on main.

1. Redirection and piping fail on current Windows (user-visible data loss)

Command output is rendered by Rich and contains box-drawing characters, but redirection
targets and pipes are opened with the locale's encoding. On Windows that is the ANSI code
page, and no cp125x code page can represent them, so redirecting raises
UnicodeEncodeError and the user is left with an empty file plus advice to set
PYTHONIOENCODING.

Measured on main at ca0ddbe7, running help > out.txt under a non-UTF-8 default encoding:

file size: 1 bytes
contains help text: False

After this change the same probe writes 153 bytes containing the help text.

Who this affects

This is not limited to old systems. A fully updated Windows 11 with default settings is
affected. The character that breaks is U+2500 (), and encodability by code page is:

Windows ANSI code page Encodes U+2500?
cp1252 US / Western European No
cp1250 Central European No
cp1251 Cyrillic No
cp1253–1258 Greek, Turkish, Hebrew, Arabic, Baltic, Vietnamese No
cp932 / cp936 / cp949 / cp950 CJK Yes

Only the CJK double-byte code pages escape, because they happen to include box drawing.

By interpreter, sys.flags.utf8_mode is 0 on Python 3.11, 3.12, 3.13 and 3.14, and 1
on 3.15 — PEP 686 makes UTF-8 mode the default there. So every Python version this project
currently supports below 3.15 is affected.

The other escapes are Windows' "Beta: Use Unicode UTF-8 for worldwide language support"
option, which is off by default, and setting PYTHONUTF8=1 or -X utf8.

In short: a user on Windows 11, US locale, Python 3.13, typing help > out.txt in any cmd2
app loses their output. That is the common case, not an edge case.

main's own tests do not catch this — nothing there redirects Rich-heavy output and reads it
back, so the suite passes while the product is broken. This PR adds tests that do.

2. The suite inherits the caller's colour environment

Rich and cmd2 consult several environment variables when deciding whether to emit styling,
and the suite inherits them. Exporting any one makes large numbers of unrelated tests fail
depending on who runs the suite — measured on main:

Variable Tests failed
NO_COLOR=1 15
FORCE_COLOR=1 53
TTY_COMPATIBLE=1 53

The failures look like product regressions, which makes them expensive to diagnose; this
cost a reviewer and me a full round trip before the cause was found. An autouse fixture now
neutralizes them, and a guard test fails if any reaches a test again. Tests that exercise
these variables still set them explicitly, because a test's own monkeypatching runs after
the fixture.

Verification

Full suite under every environment that previously broke it:

baseline           1863 passed, 2 skipped
NO_COLOR=1         1863 passed, 2 skipped
FORCE_COLOR=1      1863 passed, 2 skipped
TTY_COMPATIBLE=1   1863 passed, 2 skipped
non-UTF8 locale    1863 passed, 2 skipped

make check, make test and make docs-test all pass. Reverting either encoding change
fails the new tests.

The pipe test runs its filter through sys.executable rather than cat, matching the
convention in the existing pipe tests — cmd.exe has no cat, and this fix exists for
Windows. Note that CI would not have caught that: GitHub's Windows runners have Git for
Windows' utilities on PATH.

Note for merging

These commits also exist on consolidated_toolbar via #1752, so the same change appears in
both histories. The content is identical, so the eventual merge resolves cleanly.

Rich and cmd2 consult several environment variables when deciding whether to
emit styling, and the suite inherited them. Exporting any one of them made large
numbers of unrelated tests fail depending on who ran the suite: NO_COLOR failed
15 tests, and FORCE_COLOR and TTY_COMPATIBLE 53 each. The failures look like
product regressions, which makes them expensive to diagnose.

Neutralize them for every test. Tests that exercise these variables set them
explicitly, which still works because a test's own monkeypatching runs after the
fixture. A guard test fails if any of them reaches a test again.

(cherry picked from commit 2f786cb)
Command output is rendered by Rich and routinely contains non-ASCII, but
redirection targets and pipes were opened with the locale's encoding. On any
system whose default is not UTF-8 -- a Windows console using a legacy code page,
for instance -- redirecting output raised UnicodeEncodeError, and the user was
left with an empty file and advice to set PYTHONIOENCODING.

Open both with UTF-8 explicitly. Two tests that read redirected output back were
relying on the locale encoding for decoding as well, so they now name it too.

(cherry picked from commit b490a36)
The test piped through `cat`, which cmd.exe does not provide. On a Windows
system without Unix utilities installed it would fail before reaching the
encoding behavior it exists to check -- and Windows is exactly what the UTF-8
redirection fix targets.

Use a sys.executable pass-through instead, matching the pipe tests already in
tests/test_command_toolbar.py. Reverting either the pipe or the redirect
encoding still fails these tests.

(cherry picked from commit 7b70fbc)
@codecov

codecov Bot commented Sep 7, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.63%. Comparing base (ca0ddbe) to head (c7436ed).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1753      +/-   ##
==========================================
- Coverage   99.64%   99.63%   -0.02%     
==========================================
  Files          23       23              
  Lines        5973     5973              
==========================================
- Hits         5952     5951       -1     
- Misses         21       22       +1     
Flag Coverage Δ
unittests 99.63% <100.00%> (-0.02%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

@tleonhardt tleonhardt self-assigned this Sep 7, 2026
@tleonhardt

Copy link
Copy Markdown
Member Author

@kmvanbrunt @bambu This is a bug fix to merge to main for fixing some edge cases on older Windows systems where utf-8 isn't the default encoding. This manually sets utf-8 as the encoding when piping or redirecting to ensure any Rich text styling is supported. Everything LGTM. But if either of you have time for a quick review, great.

Measured which Windows ANSI code pages can represent the box-drawing characters
Rich emits: every cp125x code page fails, covering US and Western European,
Central European, Cyrillic, Greek, Turkish, Hebrew, Arabic, Baltic and
Vietnamese systems. Only the CJK double-byte code pages survive.

Calling it a legacy code page was wrong: cp1252 is the default on a current,
fully updated Windows 11, and Python only defaults to UTF-8 mode in 3.15.
@tleonhardt tleonhardt added the bug label Sep 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant