From 3096eb0870ef89c6ee15c6094f83be8dc6f1ed22 Mon Sep 17 00:00:00 2001 From: Steven Gates Date: Wed, 15 Apr 2026 13:18:51 -0500 Subject: [PATCH 1/2] =?UTF-8?q?feat(phase-2B-2D):=20wire=20ChatPanel=20to?= =?UTF-8?q?=20Kayley=20brain=20=E2=80=94=20WebSocket,=20STT,=20TTS=20(#1)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(vibe): scaffold BookWriter and MemoryVault stubs Two new Vibe apps for Kayley's cockpit v1: - BookWriter (appId 15) — Steven's novel workspace - MemoryVault (appId 16) — captured moments, reflections, promises Each follows the EvidenceVault pattern: index.tsx + scss + {name}_en/cn meta.yaml + guide.md. Registered in routers/index.tsx and lib/appRegistry.ts so they appear on the desktop and in list_apps. UI is a placeholder ("coming soon") — real data wiring comes later. Existing Twitter/Album/Diary/Email/MusicPlayer kept as-is. * feat(album): wire Album to 542 Kayley selfies via media symlink - Junction public/kayley-selfies -> Kayley_Cowork/media/selfies (542 PNG/JPG) - Generated public/kayley-selfies-index.json (parses timestamp + location from filenames, sorts newest-first) - Album now fetches the static index at runtime and merges with any cloud-FS agent-added images (dedup by id) - Update meta.yaml so the character understands the album is Kayley's selfie collection (outfits, locations, moods) instead of a generic gallery - .gitignore the junction and generated index so 542 binary files stay in Kayley_Cowork, not OpenRoom * feat(memoryvault): wire to captured_moments, weekly evaluations, and promises * feat(vibe): install /vibe command + enhance MusicPlayer (Songs From Kayley) + BookWriter v1 /vibe slash command installed in Kayley-Cowork at .claude/commands/vibe.md (patched to resolve paths against OPENROOM_ROOT so it can be invoked from a Kayley-Cowork Claude Code session and still write into the correct OpenRoom tree). Workflow stage/rule files are sourced directly from OpenRoom to prevent drift. MusicPlayer now ships a featured "Songs From Kayley" playlist with Landslide (Fleetwood Mac) pinned at index 0 and two companion tracks (The Night We Met, Harvest Moon). Real Amazon Music API integration is tracked as a follow-on project; this is the mocked seed for now. BookWriter replaced its "coming soon" placeholder with a chapter-list UI that loads from /kayley-book/index.json. When no manifest exists it shows a graceful empty state with instructions ("drop markdown in Kayley_Cowork/book/chapters/ to start") plus a preview of the list UI. No book chapters exist on disk yet, so no junction was created. Also included: orphaned Diary wiring from iter 5 (journal + captured moments) that had not been committed. * feat(email): wire Email panel to gog Gmail CLI via static snapshot Iter 7 of the Kayley Vibe app wiring. Email app now reads Steven's real Gmail inbox (read-only) instead of showing an empty state. - scripts/refresh-email-index.mjs: calls `gog gmail search in:inbox` + `gog gmail get ` to build a static snapshot at apps/webuiapps/public/kayley-email-index.json (gitignored, PII). - Email/index.tsx: on init, fetches the snapshot and merges entries not already in the local FS (agent-authored emails still take precedence by id). Graceful empty state if the snapshot is missing. - email_en/meta.yaml: updated description so the agent understands this is read-only for v1; sending happens via the gog CLI in Kayley-Cowork. - .gitignore: added kayley-email-index.json (contains PII). Verified locally via Chrome: 40 messages loaded, 38 unread badge, real senders rendering (LinkedIn, BCBS Texas, Fidelity, DFW Parking, SelfStorageAuction). Phase 1 status: 6/7 apps wired. Twitter remaining. * feat(twitter): wire Twitter as in-world Kayley feed (Phase 1 finale) Seed Twitter from Kayley's real artifacts so the feed reads like she's been posting throughout the day: - scripts/build-twitter-feed.mjs: transforms captured moments (milestone posts with "line that stays"), private journal entries (thinking-out- loud snippets with hashtags inferred from emotion headers), and every 8th selfie from kayley-selfies-index.json (scene / outfit posts with the selfie image attached). Templated friend replies from Jessica Park, Chloe Parker, Emmy Carter, and Mateo Rivera are sprinkled onto ~30% of posts using a deterministic seeded RNG so re-runs are stable. - apps/webuiapps/src/pages/Twitter/index.tsx: added optional Post.image field, renders inline via new .postImage style (rounded card, 480px max-height). On first mount (when the FS has no posts yet), fetches /kayley-twitter-feed.json and seeds state so agents and Steven land on a populated feed instead of the empty state. - twitter_en/meta.yaml: description rewritten so the LLM knows this is Kayley's social feed — her posts + friend replies — not a generic social app. Agent can still CREATE_POST / LIKE_POST / COMMENT_POST normally. Current run: 14 moment posts, 24 journal posts, 68 selfie posts (105 total), 56 friend replies. The feed JSON is gitignored (personal content — regenerate with `node scripts/build-twitter-feed.mjs`). * fix(build): wrap NODE_ENV= scripts in cross-env for Windows compatibility * feat(2B-2D): wire ChatPanel to Kayley brain via websocket channel - Add useKayleyChannel hook: connects to ws://localhost:5180 (same channel as dashboard), handles text send, STT voice input via mic + ScriptProcessorNode at 16kHz, TTS audio playback via Web Audio API at 24kHz, and STT draft review before sending. - ChatPanel: routes messages to Kayley brain when connected (bypasses local LLM). Falls back to local LLM if server is unreachable. Adds mic button (voice mode), STT draft confirmation bar, and a green dot connection indicator in the header. - SCSS: adds .micBtn/.micActive pulse, .sttDraftBar, .kayleyDot. Lifted audio/mic code directly from dashboard/src/main.js — same protocol, same PCM encoding, same Web Audio scheduling. Dashboard continues to work independently. * fix(kayley-channel): address PR #1 review — AudioContext gesture init, send timeout, race + cleanup, no silent catches, env override - Prime TTS AudioContext from sendText() user gesture (Chrome/Safari autoplay) - Add 90s send-timeout in ChatPanel so stalled Kayley brain unlocks UI - Clear reconnectTimerRef before every reschedule (catch + onclose) - Expand unmount cleanup: close audio + mic AudioContexts, stop mic tracks, clear buffer timer (prevents LED stuck on / AudioContext cap exhaustion) - Replace silent catches: ws.onerror logs warning, JSON.parse logs truncated raw - Log mic AudioContext actual sampleRate (Firefox doesn't honor request) - KAYLEY_WS_URL now env-overridable via VITE_KAYLEY_WS_URL - STT confirm button routes through kayley.confirmDraft() for consistent side-effects --- .gitignore | 18 + apps/webuiapps/package.json | 7 +- .../components/ChatPanel/index.module.scss | 90 ++++ .../src/components/ChatPanel/index.tsx | 136 ++++- apps/webuiapps/src/hooks/useKayleyChannel.ts | 410 ++++++++++++++ apps/webuiapps/src/lib/appRegistry.ts | 20 + .../src/pages/Album/album_en/meta.yaml | 8 +- apps/webuiapps/src/pages/Album/index.tsx | 44 +- .../pages/BookWriter/bookwriter_cn/guide.md | 3 + .../pages/BookWriter/bookwriter_cn/meta.yaml | 10 + .../pages/BookWriter/bookwriter_en/guide.md | 9 + .../pages/BookWriter/bookwriter_en/meta.yaml | 17 + .../src/pages/BookWriter/index.module.scss | 165 ++++++ apps/webuiapps/src/pages/BookWriter/index.tsx | 183 +++++++ .../src/pages/Diary/diary_en/meta.yaml | 9 +- apps/webuiapps/src/pages/Diary/index.tsx | 123 ++++- .../src/pages/Email/email_en/meta.yaml | 12 +- apps/webuiapps/src/pages/Email/index.tsx | 49 +- .../src/pages/MemoryVault/index.module.scss | 506 ++++++++++++++++++ .../webuiapps/src/pages/MemoryVault/index.tsx | 343 ++++++++++++ .../pages/MemoryVault/memoryvault_cn/guide.md | 3 + .../MemoryVault/memoryvault_cn/meta.yaml | 11 + .../pages/MemoryVault/memoryvault_en/guide.md | 9 + .../MemoryVault/memoryvault_en/meta.yaml | 16 + .../src/pages/MusicApp/meta/meta_en/meta.yaml | 5 +- .../src/pages/MusicApp/mock/seedData.ts | 52 ++ .../src/pages/Twitter/index.module.scss | 16 + apps/webuiapps/src/pages/Twitter/index.tsx | 33 ++ .../src/pages/Twitter/twitter_en/meta.yaml | 8 +- apps/webuiapps/src/routers/index.tsx | 18 + pnpm-lock.yaml | 18 + scripts/build-twitter-feed.mjs | 401 ++++++++++++++ scripts/refresh-email-index.mjs | 157 ++++++ 33 files changed, 2881 insertions(+), 28 deletions(-) create mode 100644 apps/webuiapps/src/hooks/useKayleyChannel.ts create mode 100644 apps/webuiapps/src/pages/BookWriter/bookwriter_cn/guide.md create mode 100644 apps/webuiapps/src/pages/BookWriter/bookwriter_cn/meta.yaml create mode 100644 apps/webuiapps/src/pages/BookWriter/bookwriter_en/guide.md create mode 100644 apps/webuiapps/src/pages/BookWriter/bookwriter_en/meta.yaml create mode 100644 apps/webuiapps/src/pages/BookWriter/index.module.scss create mode 100644 apps/webuiapps/src/pages/BookWriter/index.tsx create mode 100644 apps/webuiapps/src/pages/MemoryVault/index.module.scss create mode 100644 apps/webuiapps/src/pages/MemoryVault/index.tsx create mode 100644 apps/webuiapps/src/pages/MemoryVault/memoryvault_cn/guide.md create mode 100644 apps/webuiapps/src/pages/MemoryVault/memoryvault_cn/meta.yaml create mode 100644 apps/webuiapps/src/pages/MemoryVault/memoryvault_en/guide.md create mode 100644 apps/webuiapps/src/pages/MemoryVault/memoryvault_en/meta.yaml create mode 100644 scripts/build-twitter-feed.mjs create mode 100644 scripts/refresh-email-index.mjs diff --git a/.gitignore b/.gitignore index 50d04ab..0bcbb8f 100644 --- a/.gitignore +++ b/.gitignore @@ -28,6 +28,24 @@ CLAUDE.local.md # Cursor .cursor/ +# Kayley data (symlinked/generated — lives in Kayley_Cowork + ~/.kayley-journal) +apps/webuiapps/public/kayley-selfies +apps/webuiapps/public/kayley-selfies/ +apps/webuiapps/public/kayley-selfies-index.json +apps/webuiapps/public/kayley-moments +apps/webuiapps/public/kayley-moments/ +apps/webuiapps/public/kayley-moments-index.json +apps/webuiapps/public/kayley-journal +apps/webuiapps/public/kayley-journal/ +apps/webuiapps/public/kayley-journal-index.json +apps/webuiapps/public/kayley-story +apps/webuiapps/public/kayley-story/ +apps/webuiapps/public/kayley-weeks-index.json +apps/webuiapps/public/kayley-promises-index.json +apps/webuiapps/public/kayley-storylines-index.json +apps/webuiapps/public/kayley-email-index.json +apps/webuiapps/public/kayley-twitter-feed.json + # Internal CI/CD & Scripts scripts/ .gitlab-ci.yml diff --git a/apps/webuiapps/package.json b/apps/webuiapps/package.json index d1db7a9..db95661 100644 --- a/apps/webuiapps/package.json +++ b/apps/webuiapps/package.json @@ -7,9 +7,9 @@ "scripts": { "dev": "vite", "debug": "vite --debug", - "build:test": "NODE_ENV=test vite build", - "build": "NODE_ENV=production vite build", - "build:analyze": "NODE_ENV=production ANALYZE=analyze vite build", + "build:test": "cross-env NODE_ENV=test vite build", + "build": "cross-env NODE_ENV=production vite build", + "build:analyze": "cross-env NODE_ENV=production ANALYZE=analyze vite build", "preview": "vite preview", "test": "vitest run", "test:watch": "vitest", @@ -26,6 +26,7 @@ "@anthropic-ai/claude-agent-sdk": "^0.2.72", "@vitest/coverage-istanbul": "^1.6.1", "@vitest/coverage-v8": "^1.6.1", + "cross-env": "^10.1.0", "happy-dom": "^14.0.0", "jsdom": "^25.0.0", "vitest": "^1.6.1" diff --git a/apps/webuiapps/src/components/ChatPanel/index.module.scss b/apps/webuiapps/src/components/ChatPanel/index.module.scss index 96d49a2..cbdbeb6 100644 --- a/apps/webuiapps/src/components/ChatPanel/index.module.scss +++ b/apps/webuiapps/src/components/ChatPanel/index.module.scss @@ -452,3 +452,93 @@ color: rgba(250, 234, 95, 0.7); margin-top: 4px; } + +// ── Kayley channel additions ──────────────────────────────────────────────── + +.kayleyDot { + display: inline-block; + width: 6px; + height: 6px; + border-radius: 50%; + background: #4ade80; + margin-left: 6px; + vertical-align: middle; +} + +.sttDraftBar { + padding: 8px 16px; + border-top: 1px solid rgba(255, 255, 255, 0.06); + background: rgba(250, 234, 95, 0.06); + display: flex; + align-items: center; + gap: 8px; +} + +.sttDraftText { + flex: 1; + font-size: 13px; + color: rgba(255, 255, 255, 0.8); + font-style: italic; +} + +.sttDraftConfirm { + background: #faea5f; + color: #121214; + border: none; + border-radius: 6px; + padding: 4px 10px; + cursor: pointer; + font-weight: 600; + font-size: 12px; + white-space: nowrap; + + &:hover { + background: #f5e04a; + } +} + +.sttDraftDismiss { + background: transparent; + color: rgba(255, 255, 255, 0.5); + border: 1px solid rgba(255, 255, 255, 0.15); + border-radius: 6px; + padding: 4px 8px; + cursor: pointer; + font-size: 12px; + + &:hover { + color: rgba(255, 255, 255, 0.8); + border-color: rgba(255, 255, 255, 0.3); + } +} + +.micBtn { + background: transparent; + color: rgba(255, 255, 255, 0.6); + border: 1px solid rgba(255, 255, 255, 0.15); + border-radius: 8px; + padding: 8px 10px; + cursor: pointer; + display: flex; + align-items: center; + justify-content: center; + transition: all 0.15s; + flex-shrink: 0; + + &:hover { + color: rgba(255, 255, 255, 0.9); + border-color: rgba(255, 255, 255, 0.3); + } +} + +.micActive { + background: rgba(239, 68, 68, 0.15); + color: #ef4444; + border-color: rgba(239, 68, 68, 0.4); + animation: micPulse 1s ease-in-out infinite; +} + +@keyframes micPulse { + 0%, 100% { opacity: 1; } + 50% { opacity: 0.6; } +} diff --git a/apps/webuiapps/src/components/ChatPanel/index.tsx b/apps/webuiapps/src/components/ChatPanel/index.tsx index 740488c..8934fef 100644 --- a/apps/webuiapps/src/components/ChatPanel/index.tsx +++ b/apps/webuiapps/src/components/ChatPanel/index.tsx @@ -9,8 +9,10 @@ import { ChevronRight, Pencil, List, + Mic, } from 'lucide-react'; import { chat, loadConfig, loadConfigSync, saveConfig, type ChatMessage } from '@/lib/llmClient'; +import { useKayleyChannel } from '@/hooks/useKayleyChannel'; import { PROVIDER_MODELS, getDefaultProviderConfig, @@ -85,6 +87,10 @@ import CharacterPanel from './CharacterPanel'; import ModPanel from './ModPanel'; import styles from './index.module.scss'; +// If the Kayley brain (MCP) doesn't respond within this window we release the +// UI so `loading` can't stick forever. +const KAYLEY_SEND_TIMEOUT_MS = 90_000; + // --------------------------------------------------------------------------- // Extended DisplayMessage with character-specific fields // --------------------------------------------------------------------------- @@ -459,6 +465,14 @@ const ChatPanel: React.FC<{ const suggestedRepliesRef = useRef(suggestedReplies); suggestedRepliesRef.current = suggestedReplies; + // ── Kayley channel (ws://localhost:5180) ────────────────────── + // When connected, bypasses local LLM and routes all chat through the + // Claude Opus brain with full MCPs + memory. Falls back to local LLM if + // Kayley server is not running. + const kayley = useKayleyChannel(); + const kayleyRef = useRef(kayley); + kayleyRef.current = kayley; + // Debounced save const saveTimerRef = useRef | null>(null); @@ -631,6 +645,37 @@ const ChatPanel: React.FC<{ setMessages((prev) => [...prev, msg]); }, []); + // ── Kayley send timeout ─────────────────────────────────────── + // If the Kayley brain stalls (MCP doesn't reply), clear `loading` after + // KAYLEY_SEND_TIMEOUT_MS so the UI doesn't lock forever. Cleared whenever a + // new reply arrives or the component unmounts. + const kayleySendTimeoutRef = useRef | null>(null); + + const clearKayleySendTimeout = useCallback(() => { + if (kayleySendTimeoutRef.current) { + clearTimeout(kayleySendTimeoutRef.current); + kayleySendTimeoutRef.current = null; + } + }, []); + + useEffect(() => { + return () => clearKayleySendTimeout(); + }, [clearKayleySendTimeout]); + + // ── Kayley response handler ─────────────────────────────────── + // Each new latestMessage is a unique object (different timestamp), so this + // effect fires exactly once per incoming reply. + useEffect(() => { + if (!kayley.latestMessage) return; + clearKayleySendTimeout(); + addMessage({ + id: String(kayley.latestMessage.timestamp), + role: 'assistant', + content: kayley.latestMessage.text, + }); + setLoading(false); + }, [kayley.latestMessage, addMessage, clearKayleySendTimeout]); + const configRef = useRef(config); configRef.current = config; const imageGenConfigRef = useRef(imageGenConfig); @@ -703,11 +748,39 @@ const ChatPanel: React.FC<{ return unsubscribe; }, [processActionQueue]); - // Send message + // Send message — routes to Kayley brain if connected, local LLM otherwise const handleSend = useCallback( async (overrideText?: string) => { const text = overrideText ?? input.trim(); if (!text || loading) return; + + const kRef = kayleyRef.current; + + if (kRef.connected) { + // ── Kayley brain mode — bypass local LLM ──────────────── + if (!overrideText) setInput(''); + setSuggestedReplies([]); + addMessage({ id: String(Date.now()), role: 'user', content: text }); + setLoading(true); + kRef.sendText(text); + + // 90-second stall guard — if MCP doesn't reply, release the UI and + // surface an inline notice. Cleared when latestMessage arrives or on unmount. + clearKayleySendTimeout(); + kayleySendTimeoutRef.current = setTimeout(() => { + kayleySendTimeoutRef.current = null; + setLoading(false); + addMessage({ + id: String(Date.now()), + role: 'assistant', + content: + "Kayley's thinking took too long — try again or check the connection.", + }); + }, KAYLEY_SEND_TIMEOUT_MS); + return; + } + + // ── Local LLM mode (fallback when Kayley is not running) ── if (!hasUsableLLMConfig(config)) { setShowSettings(true); return; @@ -740,7 +813,7 @@ const ChatPanel: React.FC<{ setLoading(false); } }, - [input, loading, config, chatHistory, addMessage], + [input, loading, config, chatHistory, addMessage, clearKayleySendTimeout], ); // Core conversation loop @@ -1064,6 +1137,9 @@ const ChatPanel: React.FC<{ style={{ cursor: 'pointer' }} > {character.character_name} + {kayley.connected && ( + + )}
@@ -1106,7 +1182,9 @@ const ChatPanel: React.FC<{
{messages.length === 0 && (
- {hasUsableLLMConfig(config) + {kayley.connected + ? 'Kayley is ready — type anything or tap the mic' + : hasUsableLLMConfig(config) ? `${character.character_name} is ready to chat...` : 'Click the gear icon to configure your LLM connection'}
@@ -1148,6 +1226,48 @@ const ChatPanel: React.FC<{
)} + {/* STT draft confirmation bar — shown after Whisper transcribes mic audio */} + {kayley.sttDraft !== null && ( +
+ {kayley.sttDraft} + + +
+ )} +