Make live view usable on touch devices by masnwilliams · Pull Request #448 · kernel/kernel-images

masnwilliams · 2026-10-09T20:55:12Z

Summary

Live view is hard to use from a phone or tablet: a one-finger swipe is sent as a mouse drag (it selects text instead of scrolling), a desktop-sized stream is too small to read or hit, and the soft keyboard only opens from a small icon. This PR adds a touch layer to the live view client. It is client-only: the remote browser still receives the same mouse, wheel and key messages it does today.

Changes

Gestures (src/utils/touch-gestures.ts)

Tap clicks at the tapped point.
One-finger swipe scrolls through the existing wheel message, so content follows the finger, with client-side momentum after a fling.
Long-press then release right-clicks; long-press then move drags (text selection, sliders, drag and drop).
Two-finger pinch and pan zoom the video locally (src/utils/zoom-pan.ts), up to 3x the fitted size, or 1.5x the stream's native pixel size for large streams (at most 8x). The zoomed video can fill the letterbox bars. Double-tap resets the zoom, and a "2.5× · Reset" chip shows while zoomed. Pointer coordinates are mapped through the zoom, so taps land on the right remote pixel.

Keyboard

A tap can raise the soft keyboard. iOS only opens it when an input is focused inside the user gesture, so the decision is made in touchend using the remote cursor shape: neko already sends cursor images over the data channel, and Chromium shows a text (I-beam) cursor over editable fields. The pointer moves to the touch point on touchstart, so the cursor update usually arrives before the finger lifts. If it arrives later, a "Tap to type" button appears at the tap point instead. Pages often show another cursor briefly before the I-beam after a click, so any text cursor within 800ms of the tap counts; only the end of that window means the tap was not on text. The client also remembers where a text cursor was seen, so a tap on that spot, such as a second tap on the same field, raises the keyboard inside the tap. Tapping a non-text area closes the keyboard. See src/utils/cursor-shape.ts.
Text from keyboards that report keyCode 229 (Android) arrives only as input and composition events, which the Guacamole keyboard ignores. These are now typed, including composition updates and backspace.
Upper-case letters from soft keyboards were typed in lower case, because no Shift press is reported and the X server maps the keysym to the unshifted key. Shift is now sent around them on touch devices.
The 30px icon is replaced by a keyboard control (src/components/touch-controls.vue) that stays off the stream whenever it can:
- Bottom band: when the stream leaves at least a 52px band below it (a 44px target plus margins and the safe-area inset), the control sits there, labelled "Keyboard" / "Hide keyboard".
- Side strip: when the spare room is beside the stream (for example a portrait stream on a shorter screen), the control sits in a 52px strip on the left or right.
- Shrinking: when neither band is big enough, the video is shrunk by up to 10% to make one. A stream that exactly fills its frame needs 6–8% at phone sizes (52px of 664–844px). Beyond 10% the stream gets noticeably smaller, so the control overlays the stream instead.
- Overlay handle: a 44px handle docked to the left or right edge, safe-area aware, starting on the left at mid-height. Released after a drag, it snaps to the nearest edge.
- In every mode: the control can be dragged along its band, strip or edge, and dragging never sends input to the page. It fades to 30% after 3s without a tap or drag and wakes on any touch on the view. While faded, the overlay handle lets taps through to the page underneath. The position is stored in localStorage, the control moves up above an overlaid soft keyboard (using visualViewport), and it shows on any touch device rather than only below 768px or with ?embed.
The overlay textarea gets autocapitalize, autocorrect and autocomplete off, and a 16px font so iOS does not zoom the page when it focuses.

Layout

.neko-main no longer has min-width: 360px, which pushed the controls off-screen in narrow embeds.
Heights use 100dvh, so the bottom controls are not hidden under mobile browser toolbars.
viewport-fit=cover makes the safe-area insets apply.

URL option

?tapKeyboard=auto (default): raise the keyboard when the tap lands on a text cursor.
chip: show the "Tap to type" button there instead.
always: raise it on every tap.
off: keyboard button only.

Desktop mouse and keyboard handling is unchanged. The new input paths only run for touch events, or on touch devices (Shift wrapping and input handling). The touch-only branch in focusOverlay is removed because touch no longer synthesizes mouse events.

Testing

bun test tests: 39 pass, including the new tests/touch-input.test.ts (gesture classification, zoom math and clamping, keysym and Shift mapping, composition diffing) and tests/touch-controls.test.ts (band, strip, shrink and overlay layout; drag and snap; stored position). Lint shows no new warnings compared with main, and npm run build succeeds.
Hot-patched the built client onto a running headful browser (390x844, with and without kiosk mode, plus a 1920x1080 session) and drove it from Chromium with iPhone 13 device emulation, using trusted touch input over CDP. Checked on the remote page:
- A 300px swipe scrolled about 380 remote px, matching the finger at that scale, and a fling continued about 690px.
- Tap clicked the mapped pixel, and long-press produced a contextmenu.
- Pinch to 2.5x and 4x worked, taps while zoomed hit the exact remote pixel, and double-tap reset.
- Tapping an input raised the keyboard inside the tap on 40–160ms taps. Tapping a non-text area closed it.
- Text typed through input events, IME composition with a keyCode 229 backspace, and key events with capitals all arrived correctly.
- With the browser UI visible, tapping the address bar raised the keyboard and typing a URL plus Enter navigated.
- Keyboard control layout:
  - A 390x664 viewport over a portrait stream put it in a left strip with no overlap.
  - Dragging it across moved it to a right strip, sent no input to the page, and was remembered after a reload.
  - A frame exactly the stream's size shrank the video by 5.7% for a bottom band, and a taller frame got a bottom band with no shrink.
  - A 195x422 frame used the overlay handle. Faded, it let a tap through to the page and woke up; dragged, it snapped to the right edge without sending input.
- The keyboard path was also run in Playwright WebKit with an iPhone profile, with the video tracks disabled because Linux WebKit crashes decoding them. The data channel, cursor images, touch input and focus all run for real. On a real search page whose input briefly shows an arrow before the I-beam, and with 200ms of added cursor latency, a quick first tap on the field showed the "Tap to type" button, and a second tap on the same field focused the input synchronously inside the tap. That textarea was focusable (visible, not disabled or readonly, full size), and the typed text reached the remote field. Chromium behaved the same.
- With the current client, the same swipe only selected text, the keyboard never opened from a tap, input-event text was dropped, and capitals arrived in lower case.
- Desktop mouse click, Shift-typing and the keyboard path were unchanged.

Not verified yet:

Not on a physical iPhone or Android device. In particular, whether iOS raises the keyboard from focus inside touchend, real Gboard behaviour, and timing over cellular networks.
The legacy (non-module) bundle was not exercised.

Known gaps

Chromium shows the text cursor over any selectable text, not only inputs, so in auto mode tapping plain paragraph text can open the keyboard. ?tapKeyboard=chip is the less intrusive option.
The client cannot see when the remote field loses focus, so the keyboard stays open after a form submit or navigation until the user taps a non-text area or "Hide keyboard".
Over a real network the cursor update usually arrives after a quick tap ends, so the first tap on a field shows the "Tap to type" button rather than the keyboard. A second tap on the field raises it.
The remote cursor is still drawn in the video.

Future work

Native touch input (forwarding real multi-touch to the remote browser) would need new data channel messages in neko. The remote browser currently reports maxTouchPoints 0 and a fine pointer, so pages would see touch events from a browser that says it has no touch support. Native touch should come with a consistent mobile device configuration, not on its own.
A guest-side signal for focused editable fields would replace the cursor heuristic and fix the keyboard staying open after submit.

Note

Medium Risk
Large client-only input refactor on the live view path; remote control still uses the same messages, but touch keyboard/cursor heuristics and new gesture code could mis-route clicks, scroll, or typing on edge cases.

Overview
Adds a touch-first live view layer in the chromium-headful client so phones/tablets can scroll, zoom, and type into the remote session without changing server wire protocol.

Gestures and zoom: Replaces touch-to-synthetic-mouse with TouchGestures (tap click, swipe → wheel with fling, long-press right-click / drag, pinch zoom via ZoomPan). Taps map through local zoom; double-tap and a zoom chip reset zoom.

Keyboard: Parses remote cursor images from the WebRTC data channel and classifies I-beam cursors to decide when to open the soft keyboard on touchend (with ?tapKeyboard= modes and a “Tap to type” chip). Handles Android input/IME text, auto-Shift for capitals, and a draggable keyboard control (touch-controls) with band/side/overlay layout and localStorage position. Overlay textarea and layout tweaks (100dvh, viewport-fit=cover, removed min-width: 360px) target mobile safe areas and browser chrome.

Tests: New unit tests for gesture/zoom/text helpers and control layout math. Desktop mouse/keyboard paths stay as before for non-touch flows.

^{Reviewed by Cursor Bugbot for commit 4e938a0. Bugbot is set up for automated code reviews on this repo. Configure here.}

Add a gesture layer for touch input: tap clicks, one-finger swipes scroll with momentum, long-press right-clicks or drags, and two-finger pinch zooms and pans the video locally. Raise the soft keyboard from the tap when the remote cursor is a text cursor, type Android input and composition events, and send Shift for upper case letters from soft keyboards. Replace the small keyboard icon with a labeled button inside the safe area and fix the layout below 360px and under mobile browser toolbars.

cursor