Consolidated toolbar during command execution - #1745
Draft
tleonhardt wants to merge 27 commits into
Draft
Conversation
… and refreshing during command execution. Prompts, pagers, and shell commands temporarily suspend it while using the terminal. Try the work command examples/getting_started.py for a demonstration. The feature uses two prompt-toolkit displays at different times: the existing PromptSession while waiting for input, and a dedicated CommandToolbar while commands execute. - Command execution stays on the main thread. The command loop starts the toolbar’s UI in a background thread, using the same content callback, styling, and refresh interval. - Output appears above the toolbar. Stable stream wrappers route Python and piped subprocess output through prompt-toolkit’s output proxy. Their identities remain consistent across cmd2 redirection and toolbar suspension. - Input remains coordinated. The toolbar handles terminal position reports, saves typed-ahead keys for the next prompt, and forwards Ctrl-C to the main thread. - Other terminal interfaces get exclusive access. Nested prompts, pagers, and shell commands temporarily suspend the toolbar. Custom commands can do the same with suspend_bottom_toolbar(). - Cleanup restores normal terminal operation. When execution ends, buffered output is flushed, workers are stopped, and the original streams are restored. Most supporting code lives in cmd2/command_toolbar.py, with lifecycle integration in cmd2/cmd2.py. Because get_bottom_toolbar() runs in the UI thread during commands, shared mutable state should be protected with a lock.
1. Pipes to interactive targets (cmd2.py:3366, command_toolbar.py:43) — the pipe process now inherits the real terminal via the new pipe_target() helper, and the toolbar suspends for the life of the pipe (held in RedirectionSavedState.toolbar_suspension, released in _restore_output). Streams with no file descriptor still fall back to PIPE. Verified under a real pty: alias | python3 -c "...isatty()" printed PIPE_ISATTY True; before the fix that command produced no output at all. 2. Ctrl-C not reaching subprocesses (command_toolbar.py:161) — POSIX now uses os.killpg(os.getpgrp(), SIGINT), matching what the terminal driver does. Verified under a pty with a child that reports its own SIGINT: post-fix CHILD_GOT_SIGINT appears and the child exits immediately; pre-fix it never fired and the child ran to completion. Windows keeps interrupt_main() — broadcasting a console control event there would hit unrelated processes on the same console. Ctrl-\ stays inert, which matches the main prompt's existing behavior; documented rather than changed. 3. Failing stop() disabling the toolbar (cmd2.py:2098) — _command_toolbar = None moved into its own finally. 4. Writes racing _pause() (command_toolbar.py) — an RLock shared by the toolbar and its ToolbarStreams serializes writes against the proxy swap. The lock is deliberately released before _thread.join(), since the toolbar thread can itself be blocked writing through a stream. 5. Post-startup failures swallowed — new _app_exited() detaches the streams when the display stops on its own, so output goes to the terminal instead of a dead proxy, and reports the error. Worth noting: the review's stated trigger doesn't hold — a get_bottom_toolbar() that raises on refresh is caught by asyncio's default handler and the thread survives. The fix targets the case where the thread genuinely dies. 6. Final flush cancelled before writing — does not reproduce. Application.run_async already awaits wait_for_cpr_responses() and any in-flight run_in_terminal futures before its loop closes (application.py:770-779). I built the exact scenario — an output with responds_to_cpr=True and a CPR future confirmed pending at pause time — and the trailing text arrived with and without the proposed fix. I reverted my drain implementation, which was adding a run_in_terminal round-trip and up to a second of CPR waiting at every command boundary for nothing. I kept the test as a regression guard and a two-line guard against Application.exit() raising in a loop callback if the app has already stopped. 7. Docs/example mismatch — do_work now uses Annotated[int, Argument(nargs="?")], so work and work 5 both work. Verified in the real app.
Three tests called _read_raw_input() after the toolbar context had exited, so no prompt-toolkit app session was active. patch_stdout() then asked the default app session for its output, which lazily builds a real terminal Output: harmless Vt100_Output on POSIX, but Win32Output on Windows, which raises NoConsoleScreenBufferError without a console screen buffer of the kind GitHub Actions does not provide. Bind the fixture's app session to the test pipe and recording output so every prompt-toolkit call in these tests resolves to the test terminal rather than the ambient one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C7nkx8kKC2mpaMGfLwXe6J
…d an embedded pager Following prompt-toolkit’s [layout composition model](https://python-prompt-toolkit.readthedocs.io/en/3.0.52/pages/full_screen_apps.html#the-layout). - CommandToolbar (cmd2/command_toolbar.py:101) now reuses main_session.app and its actual toolbar container, eliminating the second application and duplicated display configuration. - The new pager (cmd2/pager.py:37) shares that application. The toolbar continues refreshing during scrolling, searching, and resizing. Colors, wrapped lines, and horizontal scrolling are supported. - Enabling the toolbar selects the embedded pager. Set self.use_builtin_pager = False to retain your external pager. An external pager such as less takes control of the terminal; a layout alone cannot reserve space around it. That path still temporarily suspends the toolbar. This simplifies rendering ownership, although the embedded pager adds code. The command UI thread and output proxy remain necessary for synchronous commands. There’s also one guarded dependency on PromptSession’s internal layout structure, verified against versions 3.0.52 and 3.0.53.
Closing the terminal's input ends the display's Application.run() with EOFError, which _resume() swallows because it is an ordinary shutdown rather than a failure the running command should hear about. Nothing exercised that handler, leaving it as the one uncovered line in the new module. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C7nkx8kKC2mpaMGfLwXe6J
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1745 +/- ##
==========================================
+ Coverage 99.64% 99.69% +0.04%
==========================================
Files 23 25 +2
Lines 5973 6464 +491
==========================================
+ Hits 5952 6444 +492
+ Misses 21 20 -1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. |
page() constructed a full Pager -- a TextArea, a SearchToolbar, an ANSI lexer and its key bindings -- and then threw it away whenever the output fit on screen, which is a common case for ppaged(). Only the line measurements were ever used on that path. Move the size check into a module-level output_fits() that measures the text directly, and build the Pager only once the output has to be scrolled. A shared _fragments() helper does the one ANSI parse for both, so the measurement cannot drift from what the view would render. The fast path's size check drops from ~0.15ms to ~0.02ms per call, so it isn't a massive speedup or anything, but still cleaner.
1. Chop-mode paging no longer loses your column — cmd2/pager.py:_scroll() The destination column now comes from the window's current horizontal_scroll when chop is set, instead of target * window_width (always 0 in chop mode) 2. A toolbar that can't start no longer takes the session down - cmd2/cmd2.py _command_toolbar_context() catches startup failure, reports once via perror, sets a new _command_toolbar_disabled flag so later commands don't retry and re-spam and runs the command without a toolbar 3. ppaged() no longer starts a toolbar outside the command loop — it now pages inside a toolbar only when one is already running, and otherwise falls through to the external pager. The documentation has been updated to contain truthful statements in this regard. 4. use_builtin_pager documented as opt-out only — initialization.md, os.md, prompt.md and the ppaged() docstring now state that it cannot turn the embedded pager on where no toolbar is running, since the two share one display.
…g visible hyperlink text, colors, and width calculations When ppaged() receives Rich output containing hyperlinks, Rich emits OSC 8 sequences that prompt-toolkit's ANSI parser does not understand. Their contents become visible pager text: a linked click here renders as 8;id=...;https://example.comclick here8;;. This also corrupts width calculations and searchable text. This change strips unsupported OSC sequences before passing captured output to cmd2's new internal pager.
…toolbar.py Also: - Moved common code used by test_command_toolbar.py and test_pager.py to conftest.py Since the pager is effectively its own component, it feels cleaner to have the tests in their own file.
Cause: rich's Style.render() skips the OSC 8 hyperlink wrapper entirely when legacy_windows=True. Console(force_terminal=True) on Windows auto-detects a legacy console, so the fixture string never contained \x1b]8; — the failure was in the test's setup, not in the pager. Fix (tests/test_pager.py:31): pass legacy_windows=False to the Console, so the fixture emits real hyperlink sequences on every platform.
test_command_toolbar_ui_call_reports_display_failure patched call_soon_threadsafe on the toolbar's live event loop and swallowed every call, not just the one it meant to drop. The patch was still installed while asyncio.run() tore the loop down, and asyncio resolves the default executor's join future through call_soon_threadsafe, so that report was dropped too. On Linux and macOS prompt_toolkit registers stdin with loop.add_reader, no default executor is ever created, and shutdown_default_executor() returns immediately. On Windows prompt_toolkit parks its WaitForMultipleObjects watcher in loop.run_in_executor(), so the executor exists and the loop waits on a future nothing resolves: 300 seconds on Python 3.12 and later, and forever before that, where shutdown_default_executor() takes no timeout. Windows CI went from 34s to 345s on 3.14, and hung for six hours on 3.11. Drop only the request made on the command thread and pass everything else through. The display still dies with the ValueError the test asserts on. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ej3Nyn6vGngxYqm9FA1Jw5
tleonhardt
marked this pull request as ready for review
September 6, 2026 15:13
…Windows It now has three commands, in the order your colleague should try them: 1. slow — sleeps 5 seconds. Does TOOLBAR IS HERE stay visible the whole time? This is the discriminating test: if the toolbar isn't up during a command, the real bug is that CommandToolbar never starts on Windows, and the pager is just collateral damage. 2. longout — pages 500 lines. Built-in pager ends in q: quit; the external one shows -- More --. 3. diag — dumps the guard state to real stderr (so it survives whatever the pager does to the screen). The single most useful thing to report is the slow result plus the diag block — that alone should help narrow down the underlying root cause.
tleonhardt
marked this pull request as draft
September 6, 2026 19:05
read_input() and read_secret() each build their own PromptSession and passed no bottom_toolbar, while _read_raw_input() suspends the main display. The toolbar therefore vanished entirely for as long as a nested prompt was up, which is a visible inconsistency for a bar documented as persistent. Pass the toolbar to both sessions when the application has one, gated on main_session.bottom_toolbar being set so an app without a toolbar does not sprout one. Also inherit refresh_interval, so a clock or status display keeps ticking while the nested prompt waits rather than freezing at its first render. select() is deliberately left alone: prompt-toolkit's choice() accepts a toolbar but has no refresh_interval, so its toolbar would go stale while the selection sits idle.
…nd a UI-call race (#1752) * Render test output independently of the caller's environment Rich and cmd2 consult several environment variables when deciding whether to emit styling, and the suite inherited them. Exporting any one of them made large numbers of unrelated tests fail depending on who ran the suite: NO_COLOR failed 15 tests, and FORCE_COLOR and TTY_COMPATIBLE 53 each. The failures look like product regressions, which makes them expensive to diagnose. Neutralize them for every test. Tests that exercise these variables set them explicitly, which still works because a test's own monkeypatching runs after the fixture. A guard test fails if any of them reaches a test again. * Write redirected and piped output as UTF-8 Command output is rendered by Rich and routinely contains non-ASCII, but redirection targets and pipes were opened with the locale's encoding. On any system whose default is not UTF-8 -- a Windows console using a legacy code page, for instance -- redirecting output raised UnicodeEncodeError, and the user was left with an empty file and advice to set PYTHONIOENCODING. Open both with UTF-8 explicitly. Two tests that read redirected output back were relying on the locale encoding for decoding as well, so they now name it too. * Fix a race that reported a timeout for a successful UI call _call_in_ui() polls the pending future with a 0.1s timeout and, on expiry, re-raises when the future is already done. That branch exists because concurrent.futures.TimeoutError is TimeoutError on Python 3.11+, so a callback raising a timeout of its own cannot be told apart by type from the poll expiring. Re-raising the caught exception conflates the two. When the callback completes in the window between the poll expiring and the future being inspected, the caller is told the call timed out even though it succeeded. Ask the future for its outcome instead: a callback that raised a timeout still propagates it, and one that produced a value now returns it. Found while investigating an intermittent failure of test_command_toolbar_ui_call_propagates_failures, which reproduced once in 60 runs before this change and not once in 120 after. The added regression test drives the interleaving deterministically rather than relying on timing. (cherry picked from commit 84bc19e) * Run the pipe encoding test through this interpreter, not cat The test piped through `cat`, which cmd.exe does not provide. On a Windows system without Unix utilities installed it would fail before reaching the encoding behavior it exists to check -- and Windows is exactly what the UTF-8 redirection fix targets. Use a sys.executable pass-through instead, matching the pipe tests already in tests/test_command_toolbar.py. Reverting either the pipe or the redirect encoding still fails these tests.
Measured which Windows ANSI code pages can represent the box-drawing characters Rich emits: every cp125x code page fails, covering US and Western European, Central European, Cyrillic, Greek, Turkish, Hebrew, Arabic, Baltic and Vietnamese systems. Only the CJK double-byte code pages survive. Calling it a legacy code page was wrong: cp1252 is the default on a current, fully updated Windows 11, and Python only defaults to UTF-8 mode in 3.15.
3 tasks
Member
Author
|
This PR is a better attempt at making the bottom bar persistent. It works in nearly all situations, but involved some complexities like needing to introduce code to support a custom pager. It works and has been validated to do what it is intended to do. However it continues to have an annoying issue with flicker of the bottom bar that has always existed. While we could review it and merge it in, I'm holding off on that for now as I explore options for truly eliminating the flicker to provide a better user experience. As I've learned that is not an easy problem to solve. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Today
enable_bottom_toolbar=Trueonly shows a toolbar whileprompt-toolkitis waiting for input — it vanishes the moment a command starts running. This branch makes the toolbar persistent: it stays visible and keeps refreshing while commands execute, with command output scrolling above it.Because a persistent toolbar owns the bottom of the terminal, anything else that wants the terminal has to coordinate with it. That drives the two other pieces of this branch: an embedded pager that shares the toolbar's application, and a suspension mechanism for the places that genuinely need the raw terminal (nested prompts, shell commands, Python/IPython shells, external pagers, and pipes to interactive processes).
What changed
Persistent toolbar (
cmd2/command_toolbar.py, new)CommandToolbarreusesmain_session.appand its existing toolbar container rather than standing up a secondprompt-toolkitapplication, so content callback, styling, and refresh interval are shared with the prompt.ToolbarStreamwrappers route Python-level and piped subprocess output throughprompt-toolkit's output proxy so writes land above the toolbar. Their identities stay consistent across cmd2 redirection and toolbar suspension.RLockshared by the toolbar and its streams serializes writes against the proxy swap during pause/resume.Ctrl-Cis forwarded to the process group (os.killpgon POSIX,_thread.interrupt_main()on Windows), andCtrl-Zsuspend-to-background is honored where supported._app_exited()detaches the streams so output goes back to the real terminal and the error is reported, instead of writing into a dead proxy.Embedded pager (
cmd2/pager.py, new)ppaged()uses a built-in pager that shares the toolbar's application, so the toolbar keeps rendering while you scroll. Supports colors, wrapped lines, chopped lines with horizontal scrolling (including on Windows, which the externalmorenever did), search (/,n/N),g/G, andq.use_builtin_pagerattribute, defaulting toenable_bottom_toolbar. Set it toFalseto keep using the configured externalpager/pager_chop; an external pager takes over the terminal, so that path still suspends the toolbar.Suspension (
cmd2/cmd2.py)Cmd.suspend_bottom_toolbar()context manager for application code that callsinput(), drives its own terminal UI, or starts a subprocess that inherits the terminal.@suspend_toolbardecorator on_read_raw_input(),select(),do_shell(),_run_python(),do_ipy(), and_run_cmdfinalization_hooks().pipe_target()helper, and the toolbar suspends for the life of the pipe (tracked onRedirectionSavedState.toolbar_suspension, released in_restore_output()). Streams with no file descriptor still fall back toPIPE. Previouslyalias | fzf-style commands produced no output at all.onecmd_plus_hooks()from the interactive loop and around startup commands. It is not started for non-interactive input or foronecmd_plus_hooks()calls made outside the loop.toolbar.stop()no longer disables the toolbar for the rest of the session —_command_toolbar = Nonemoved into its ownfinally.Notes and trade-offs
get_bottom_toolbar()now runs in a background UI thread during command execution. Applications that read state a command mutates should guard it with a lock. This is documented.lessbehavior opt out withuse_builtin_pager = False.PromptSession's internal layout structure, verified against prompt-toolkit 3.0.52 and 3.0.53; it raises a clearRuntimeErrorif the structure moves.Ctrl-\(SIGQUIT) is inert while the toolbar is up because raw mode clearsISIG. This matches existing behavior at the main prompt, so it is documented rather than changed.Tests
tests/test_command_toolbar.pywith 31 tests covering startup/shutdown, output routing, suspension paths, pipe handling,Ctrl-Cforwarding, the pager, and the EOF shutdown path.patch_stdout()cannot lazily construct a realWin32Outputand raiseNoConsoleScreenBufferErroron runners with no console screen buffer.Docs
docs/features/prompt.md: persistent behavior, refresh, the pager controls,suspend_bottom_toolbar(), and threading guidance.docs/features/os.mdanddocs/features/initialization.md:use_builtin_pagerand the external-pager opt-out.examples/getting_started.py: newwork [seconds]command demonstrating the toolbar clock updating while a command prints output.NOTES
This is one possible architecture and approach to solving this problem.
I did a bunch of manual testing on both macOS and Linux and everything looks good there, but I need someone else to help test on Windows.
TODO
test_command_toolbar.pyfile.test_command_toolbar.pygets stuck indefinitelygetting_started.pyapp on Windows from this branchCloses #1743