Add fully local voice agent example (examples/voice_agents/local_only)

examples/voice_agents/local_only/README.md

@@ -0,0 +1,80 @@
+# Fully local voice agent
+
+A voice agent where every stage runs on your own hardware. No API keys, no cloud calls, no network dependency once models are downloaded.
+
+| Stage | Component | How it connects |
+|---|---|---|
+| STT | [speaches](https://github.com/speaches-ai/speaches) (faster-whisper) | openai plugin + `base_url` |
+| LLM | [Ollama](https://ollama.com) | `openai.LLM.with_ollama()` |
+| TTS | [kokoro-fastapi](https://github.com/remsky/kokoro-fastapi) | openai plugin + `base_url` |
+| VAD | silero | local ONNX (built in) |
+| Turn detection | livekit turn-detector | local model (built in) |
+
+The trick is that everything speaks the OpenAI API: the openai plugin's `base_url` parameter turns it into a universal adapter for any OpenAI-compatible local server. No new plugins needed.
+
+## Setup
+
+### 1. Ollama (LLM)
+
+```bash
+curl -fsSL https://ollama.com/install.sh | sh
+ollama pull llama3.2:3b
+```
+
+Any chat model works. Smaller models answer faster; see the latency table below before choosing.
+
+### 2. speaches (STT)
+
+```bash
+docker run -d --gpus all -p 8000:8000 ghcr.io/speaches-ai/speaches:latest-cuda
+```
+
+CPU-only image also available (`latest-cpu`) — expect higher STT latency.
+
+### 3. kokoro-fastapi (TTS)
+
+```bash
+docker run -d --gpus all -p 8880:8880 ghcr.io/remsky/kokoro-fastapi-gpu:latest
+```
+
+### 4. Run the agent
+
+```bash
+python agent.py console   # terminal mode, local mic/speakers, no LiveKit server needed
+python agent.py dev       # connect to a LiveKit server for real WebRTC sessions
+```
+
+Model and endpoint choices are environment-overridable — see the constants at the top of `agent.py`.
+
+## Measured latency
+
+Honest numbers, measured on real hardware with `bench.py` (5 runs per stage, warm path, medians). Reproduce them yourself: `python bench.py`.
+
+**Test rig:** RTX 5080 16GB (Ollama LLM) + RTX 4060 Ti 16GB (kokoro container), STT on the 5080 via speaches CUDA container. Consumer parts, nothing exotic.
+
+| Stage | Metric | Median | Spread (n=5) |
+|---|---|---|---|
+| TTS (kokoro) | first audio byte | **36ms** | 35–38ms |
+| STT (faster-whisper-small) | full transcription, ~3s utterance | **359ms** | 356–366ms |
+| LLM (llama3.2:3b) | time to first token | **127ms** | 115–130ms |
+| LLM (llama3.2:3b) | generation throughput | **170 tok/s** | — |
+
+Sum of stages after end of speech: **~520ms** before the agent starts speaking (plus turn-detection time). That's cloud-competitive, from hardware you own.
+
+**Cold starts are real and separate:** first STT request loads the model (~20s), first LLM request a few seconds. They happen once per server lifetime — pre-pull the STT model with `curl -X POST http://localhost:8000/v1/models/Systran/faster-whisper-small` and send one warmup request at deploy time.
+
+## Failure modes encountered
+
+Recorded because the next person will hit them too:
+
+1. **Thinking models are unusable for voice without care.** qwen3:4b streamed its first *reasoning* token at 142ms but its first *content* token at 18 seconds — the agent has the answer's ingredients almost instantly and says nothing while it deliberates. None of the obvious switches fixed it cleanly on Ollama's OpenAI-compatible endpoint: `/no_think` is ignored by newer qwen3 templates, `think: false` still stalled multiple seconds, and `reasoning_effort: "none"` moved the chain-of-thought *into* the content stream, which a voice agent would happily read aloud. Pick a non-thinking model for the voice path; that's why this example defaults to `llama3.2:3b`.
+2. **New GPU architectures vs. prebuilt images.** The kokoro-fastapi GPU image's PyTorch build had no kernels for the RTX 5080 (Blackwell, sm_120) — `CUDA error: no kernel image is available`. Fix: pin the container to a supported GPU (`--gpus '"device=1"'` for the Ada card here) or use the CPU image.
+3. **First-request latency is model load, not inference.** See cold starts above — don't benchmark (or alert on) the first request after a restart.
+
+## Why run local?
+
+- **Privacy** — audio never leaves the machine. Healthcare, legal, and anything under NDA becomes buildable.
+- **Cost** — a busy agent's per-minute cloud bill becomes electricity.
+- **The edge** — sites with no connectivity, or where connectivity is the failure mode.
+
+The tradeoff is honesty about latency and quality: a local 4B model is not GPT-4.1, and this README's numbers table exists so you can decide with data instead of vibes.

examples/voice_agents/local_only/agent.py

@@ -0,0 +1,110 @@
+"""Fully local voice agent — zero cloud calls.
+
+Every stage of the voice loop runs on your own hardware:
+
+    STT  — speaches (faster-whisper) via its OpenAI-compatible endpoint
+    LLM  — Ollama via ``openai.LLM.with_ollama()``
+    TTS  — kokoro-fastapi via its OpenAI-compatible endpoint
+    VAD  — silero (local ONNX model)
+    Turn detection — LiveKit turn-detector (local model)
+
+Prerequisites (see README.md for setup):
+
+    Ollama serving a chat model      http://localhost:11434
+    speaches serving faster-whisper  http://localhost:8000
+    kokoro-fastapi serving kokoro    http://localhost:8880
+
+No API keys required. The OPENAI_API_KEY values passed below are
+placeholders — the openai plugin requires a non-empty key, but the local
+servers never check it.
+"""
+
+import logging
+import os
+
+from livekit.agents import (
+    Agent,
+    AgentServer,
+    AgentSession,
+    JobContext,
+    MetricsCollectedEvent,
+    cli,
+    metrics,
+)
+from livekit.plugins import openai, silero
+
+# NOTE: livekit.plugins.turn_detector is marked deprecated in favor of
+# livekit.agents.inference.TurnDetector — but the replacement routes through
+# LiveKit's inference gateway (it takes api_key/api_secret), while this plugin
+# runs the turn-detection model locally. For a fully offline pipeline the
+# plugin is currently the only option.
+from livekit.plugins.turn_detector.multilingual import MultilingualModel
+
+logger = logging.getLogger("local-only-agent")
+
+# Local endpoints — override via environment if your servers live elsewhere.
+STT_BASE_URL = os.environ.get("LOCAL_STT_URL", "http://localhost:8000/v1")
+LLM_BASE_URL = os.environ.get("LOCAL_LLM_URL", "http://localhost:11434/v1")
+TTS_BASE_URL = os.environ.get("LOCAL_TTS_URL", "http://localhost:8880/v1")
+
+STT_MODEL = os.environ.get("LOCAL_STT_MODEL", "Systran/faster-whisper-small")
+# Use a non-thinking model for the voice path. Thinking models (e.g. qwen3)
+# stream reasoning tokens first and content tokens seconds later — the agent
+# stays silent while it deliberates. See README "Failure modes encountered".
+LLM_MODEL = os.environ.get("LOCAL_LLM_MODEL", "llama3.2:3b")
+TTS_MODEL = os.environ.get("LOCAL_TTS_MODEL", "kokoro")
+TTS_VOICE = os.environ.get("LOCAL_TTS_VOICE", "af_heart")
+
+
+class LocalAgent(Agent):
+    def __init__(self) -> None:
+        super().__init__(
+            instructions=(
+                "You are a voice assistant running entirely on local hardware. "
+                "Keep responses concise and conversational — they will be spoken aloud. "
+                "Do not use emojis, asterisks, markdown, or other special characters."
+            ),
+        )
+
+    async def on_enter(self) -> None:
+        self.session.generate_reply(
+            instructions="Greet the user briefly and mention you are running fully offline."
+        )
+
+
+server = AgentServer()
+
+
+@server.rtc_session()
+async def entrypoint(ctx: JobContext) -> None:
+    session: AgentSession = AgentSession(
+        stt=openai.STT(
+            model=STT_MODEL,
+            base_url=STT_BASE_URL,
+            api_key="not-needed",
+        ),
+        llm=openai.LLM.with_ollama(
+            model=LLM_MODEL,
+            base_url=LLM_BASE_URL,
+        ),
+        tts=openai.TTS(
+            model=TTS_MODEL,
+            voice=TTS_VOICE,
+            base_url=TTS_BASE_URL,
+            api_key="not-needed",
+        ),
+        vad=silero.VAD.load(),
+        turn_detection=MultilingualModel(),
+    )
+
+    # Log per-stage metrics — with a fully local stack these numbers are
+    # your hardware's honest answer, not a provider's SLA.
+    @session.on("metrics_collected")
+    def _on_metrics_collected(ev: MetricsCollectedEvent) -> None:
+        metrics.log_metrics(ev.metrics)
+
+    await session.start(agent=LocalAgent(), room=ctx.room)
+
+
+if __name__ == "__main__":
+    cli.run_app(server)

examples/voice_agents/local_only/bench.py

@@ -0,0 +1,165 @@
+"""Per-stage latency benchmark for the local voice stack.
+
+Measures each stage directly against its endpoint — the same numbers the
+agent's metrics_collected event reports, but reproducible standalone:
+
+    TTS  — time to first audio byte, and total synthesis time
+    STT  — full transcription latency for the TTS-generated utterance
+    LLM  — time to first streamed token (TTFT), and tokens/sec
+
+Run after all three servers are up:  python bench.py
+"""
+
+import io
+import json
+import statistics
+import time
+import urllib.request
+
+STT_URL = "http://localhost:8000/v1/audio/transcriptions"
+LLM_URL = "http://localhost:11434/v1/chat/completions"
+TTS_URL = "http://localhost:8880/v1/audio/speech"
+
+STT_MODEL = "Systran/faster-whisper-small"
+LLM_MODEL = "llama3.2:3b"
+TTS_MODEL = "kokoro"
+TTS_VOICE = "af_heart"
+
+RUNS = 5
+UTTERANCE = "What's the weather looking like in Hartford today?"
+
+
+def bench_tts() -> tuple[list[float], list[float], bytes]:
+    first_byte, total, audio = [], [], b""
+    for _ in range(RUNS):
+        req = urllib.request.Request(
+            TTS_URL,
+            data=json.dumps(
+                {
+                    "model": TTS_MODEL,
+                    "voice": TTS_VOICE,
+                    "input": UTTERANCE,
+                    "response_format": "wav",
+                    "stream": True,
+                }
+            ).encode(),
+            headers={"Content-Type": "application/json"},
+        )
+        t0 = time.perf_counter()
+        with urllib.request.urlopen(req) as resp:
+            chunk = resp.read(4096)
+            first_byte.append(time.perf_counter() - t0)
+            buf = io.BytesIO()
+            buf.write(chunk)
+            while chunk := resp.read(65536):
+                buf.write(chunk)
+            total.append(time.perf_counter() - t0)
+            audio = buf.getvalue()
+    return first_byte, total, audio
+
+
+def bench_stt(audio_wav: bytes) -> list[float]:
+    boundary = "benchboundary"
+    body = (
+        (
+            f'--{boundary}\r\nContent-Disposition: form-data; name="model"\r\n\r\n{STT_MODEL}\r\n'
+            f'--{boundary}\r\nContent-Disposition: form-data; name="file"; filename="u.wav"\r\n'
+            f"Content-Type: audio/wav\r\n\r\n"
+        ).encode()
+        + audio_wav
+        + f"\r\n--{boundary}--\r\n".encode()
+    )
+    times = []
+    for _ in range(RUNS):
+        req = urllib.request.Request(
+            STT_URL,
+            data=body,
+            headers={"Content-Type": f"multipart/form-data; boundary={boundary}"},
+        )
+        t0 = time.perf_counter()
+        with urllib.request.urlopen(req) as resp:
+            out = json.load(resp)
+        times.append(time.perf_counter() - t0)
+    print(f"  STT transcript: {out.get('text', '?')!r}")
+    return times
+
+
+def bench_llm() -> tuple[list[float], list[float]]:
+    ttft, tps = [], []
+    for _ in range(RUNS):
+        req = urllib.request.Request(
+            LLM_URL,
+            data=json.dumps(
+                {
+                    "model": LLM_MODEL,
+                    "stream": True,
+                    "messages": [
+                        {"role": "system", "content": "You are a concise voice assistant."},
+                        {"role": "user", "content": UTTERANCE},
+                    ],
+                }
+            ).encode(),
+            headers={"Content-Type": "application/json"},
+        )
+        t0 = time.perf_counter()
+        first, ntok = None, 0
+        with urllib.request.urlopen(req) as resp:
+            for line in resp:
+                if not line.startswith(b"data: ") or line.strip() == b"data: [DONE]":
+                    continue
+                delta = json.loads(line[6:])["choices"][0]["delta"].get("content")
+                if delta:
+                    ntok += 1
+                    if first is None:
+                        first = time.perf_counter() - t0
+        elapsed = time.perf_counter() - t0
+        ttft.append(first if first is not None else elapsed)
+        tps.append(ntok / elapsed if elapsed else 0.0)
+    return ttft, tps
+
+
+def report(name: str, values: list[float], unit: str = "ms") -> None:
+    scale = 1000 if unit == "ms" else 1
+    vals = [v * scale for v in values]
+    print(
+        f"  {name}: median {statistics.median(vals):.0f}{unit}  "
+        f"(min {min(vals):.0f}, max {max(vals):.0f}, n={len(vals)})"
+    )
+
+
+def warmup() -> None:
+    """One throwaway call per endpoint so measurements reflect the warm path.
+
+    Cold starts are real (model load into VRAM: ~20s for faster-whisper-small,
+    a few seconds for the LLM) but they happen once per server lifetime —
+    report them separately, don't let them pollute the steady-state numbers.
+    """
+    global RUNS
+    runs, RUNS = RUNS, 1
+    try:
+        _, _, audio = bench_tts()
+        bench_stt(audio)
+        bench_llm()
+    finally:
+        RUNS = runs
+
+
+if __name__ == "__main__":
+    print("Warming up all three endpoints (cold-start excluded from numbers)...")
+    warmup()
+    print(f"\nBenchmarking local stack ({RUNS} runs per stage)\n")
+
+    print("[TTS] kokoro")
+    fb, total, audio = bench_tts()
+    report("first audio byte", fb)
+    report("full synthesis", total)
+    print(f"  audio size: {len(audio) // 1024} KiB\n")
+
+    print(f"[STT] {STT_MODEL}")
+    report("full transcription", bench_stt(audio))
+    print()
+
+    print(f"[LLM] {LLM_MODEL}")
+    ttft, tps = bench_llm()
+    report("time to first token", ttft)
+    print(f"  throughput: median {statistics.median(tps):.0f} tok/s")