Skip to content

Add a ChatStream-Octane experimental workload - #602

Open
jrmuizel wants to merge 10 commits into
WebKit:mainfrom
jrmuizel:chat-stream-octane
Open

jrmuizel wants to merge 10 commits into
WebKit:mainfrom
jrmuizel:chat-stream-octane

Conversation

@jrmuizel

Copy link
Copy Markdown
Contributor

An LLM chat client in Octane (chat-stream/octane/App.tsrx): a sidebar of conversations, a message list and a composer, with replies streamed a few tokens at a time and rendered as memoized Markdown. It commits like the Octane fixture of the chat-stream benchmark in the Octane repository: event handlers only set state, which the steps commit through window.__benchFlush, and each streamed chunk commits with flushSync.

The corpus, Markdown parser, network stub and stylesheet are in shared/, and the build is in dist/octane.

An LLM chat client in Octane (chat-stream/octane/App.tsrx): a sidebar of
conversations, a message list and a composer, with replies streamed a few
tokens at a time and rendered as memoized Markdown. It commits like the
Octane fixture of the chat-stream benchmark in the Octane repository: event
handlers only set state, which the steps commit through window.__benchFlush,
and each streamed chunk commits with flushSync.

The corpus, Markdown parser, network stub and stylesheet are in shared/, and
the build is in dist/octane.
@netlify

netlify Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

✅ Deploy Preview for webkit-speedometer-preview ready!

Built without sensitive environment variables

Name Link
🔨 Latest commit 7fb51c2
🔍 Latest deploy log https://app.netlify.com/projects/webkit-speedometer-preview/deploys/6afb59e417d064caff1a8b56
😎 Deploy Preview https://deploy-preview-602--webkit-speedometer-preview.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.
🤖 Make changes Run an agent on this branch

To edit notification comments on pull requests, go to your Netlify project configuration.

StreamReply1, StreamReply2 and StreamReply3 become a single StreamReplies
step that sends the three prompts in turn in the first conversation,
streaming each reply to completion before sending the next.

Also fix the README's reference to chatStreamHelpers, which is named
chatStreamTests.
@camillobruni

camillobruni commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

We have 2 very fast steps on my m4:

ChatStream-Octane
ChatStream-Octane/StreamReplies	5.66	±	0.94 (16.7%)	ms
ChatStream-Octane/SwitchConversations	15.50	±	0.80 (5.1%)	ms
ChatStream-Octane/StreamIntoLongConversation	15.96	±	0.66 (4.1%)	ms
ChatStream-Octane/TypePrompt	4.53	±	0.39 (8.6%)

=> StreamReplies and TypePrompt should probably be bumped by 2x

Build the Markdown views the way a design-system component catalog is
built, as AssistantChat-Octane's catalog is: the app root provides theme,
locale and layout contexts, each message a message context, and each reply
streaming, typography and density contexts. Every block renders through a
memoized chain (MarkdownBlock, the block's own component, Frame, Part), and
each of them, the list items, inline spans and code lines reads the
contexts it depends on. That exercises Octane's context dependency tracking
across long chains of memoized ancestors, where AssistantChat-Octane's
streaming steps spend much of their time.

The streaming cursor's selector moves one level down, into the block frame.
Replace the catalog's decorative contexts (theme, locale, layout, message,
streaming, typography, density) and the Frame and Part layers with the kinds
of context the reply renderer of a large chat service's web app reads, as
found in its production bundle:

- Layout flow. Each reply and each list provide a new flow value whenever
  they render, with each child's previous sibling. Each block's wrapper and
  each list item read it and compute their spacing with that renderer's
  spacing rules, so every chunk re-renders every block's wrapper while memo
  skips the content of the blocks that did not grow.
- The colour scheme, read by code blocks and inline code.
- Streamed words. Text in a reply that streams in renders one memoized span
  per word. Each word reads four animation contexts and starts its fade-in
  from a layout effect with the delay a scheduler gives it, staggering words
  as that renderer does. The scheduler reads the stream's virtual clock,
  80 ms per chunk, kept by shared/network.js, instead of performance.now().

The steps end the word animations once a reply is complete, through
window.finishWordAnimations(), so none of that work lands in a later step.
Checking the previous commit against the reference renderer's bundle and
its live DOM turned up several differences, fixed here:

- The word animation contexts are whether the reply is eligible (it is
  streaming) and whether animation is enabled (the mode is not disabled),
  not "animate" and "visible".
- Text splits into words only inside a text component, which provides a
  context saying so; paragraphs, headings and list items provide it.
- There is no wrapper element per block. Each block component reads the
  flow and puts its spacing on its own element, and memo(Inlines) skips the
  content of the blocks that did not grow.
- A list provides its items' layout (the marker kind), not a flow. Its
  items read the reply's flow, in which they have no previous sibling, so
  they are not spaced, as in the reference.
- Paragraphs, headings and bold and italic runs resolve their text colour
  against the colour scheme.

The previous-sibling map is now keyed by node rather than by id, since
block and list-item ids are both positions and could collide.
Replies can now contain a ```ui fence: an indented tree of rows, columns,
cards, titles, text, captions, badges, buttons and dividers. Every streamed
reply has one, typically a row of option cards, and so do about a third of
the others. The client parses the fence as it streams, so cards appear piece
by piece, and unchanged subtrees keep their identity across parses.

This gives the layout flow the work it does in the reference renderer.
Every container provides its children a new flow on every render, and every
node reads it and computes its spacing with that renderer's rules for the
reply, a column and a row; a row with a fixed gap turns them off. A reply's
buttons are disabled through the flow while it streams. A card's details
stay out of its flow's sibling map until its Details button opens them, so
opening them changes which sibling the nodes after them are spaced from.

A new step, ExpandCards, opens the details of every card in the 200-message
conversation one at a time.
The draft moves from App into a memoized Composer, as on the reference chat
app, whose composer does not re-render the transcript. A keystroke now
re-renders only the composer, not the app root and every message. App
passes the composer a send callback that changes only with the active
conversation and the stream, so the composer also skips the renders a
streamed chunk causes.

Typing no longer makes every message's memo bail, so it no longer pays the
context walks those bails took after a card had updated its own state.
Measured on the reference chat app's live site, its composer:

- sizes its textarea to its content in CSS (field-sizing: content, one
  24 px row up to eight), so no script reads layout while typing;
- shows its send button once the draft has text;
- expands, the textarea over a row of actions, once a ResizeObserver on the
  textarea sees the text wrap, and animates its frame to the content's
  height, which another ResizeObserver measures;
- publishes its height, from a ResizeObserver on the footer, as a CSS
  variable the page lays out around it, since it overlays the transcript;
- saves the draft to localStorage 250 ms after the last change and at least
  once a second while it changes, and restores it on load.

The composer now does the same. A step that types ends by settling the
pending draft save and the height transition through
window.flushComposer(), so none of that work lands in a later step, and the
suite's prepare empties the composer and the saved draft through
window.resetComposer(), so every iteration starts from the same state.
TypePrompt lays out each keystroke, as each gets its own frame when a person
types.
… key events

The composer's state is now one signal scope (composer-state.tsrx), as the
reference chat app's composer controller is: the draft, focus, the streaming
state and what the ResizeObservers measure, and, derived from them, the
layout, the presentation controls, the submit inputs, whether the draft can
be submitted and what the actions show, under the reference's names. The
textarea's value is bound to the draft signal, and each of the composer's
parts reads the signals it shows as it renders, so a keystroke re-renders the
parts whose state it changes. The composer moves to its own module,
Composer.tsrx.

Since a component reads signals as it renders, Octane tracks the signal reads
of every component's render, in the transcript too, as on the reference. That
makes the other steps 20-50% slower: one signal-reading component added to
the previous version costs them as much.

TypePrompt now focuses the composer and types each character with the
events a keyboard produces: keydown, keypress, beforeinput, the edit, input
and keyup.
@jrmuizel

Copy link
Copy Markdown
Contributor Author

I've added a lot more complexity to this benchmark. The times have gone up quite a bit. Maybe too much?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants