feat: governed realtime media membrane

[sandl99]

Problem Statement

Related: NVIDIA/NemoClaw#6207.
VoiceClaw referencces

• Prioritized VoiceClaw platform-issues spreadsheet
• Detailed VoiceClaw platform-issues document

P0: OpenShell needs a governed realtime media path for audio streams.
OpenShell 0.0.72 can relay WebSocket binary bytes, but only the WebSocket upgrade and text messages have policy semantics today. WebRTC/TURN-style media is also outside the current TCP CONNECT proxy model.

VoiceClaw needs realtime audio without turning the sandbox boundary into an opaque raw socket escape. Media must remain deny-by-default, auditable, rate-limited, and bound to declared destinations and sessions.

Who will use this

• VoiceClaw realtime voice assistant.
• OpenClaw voice-call and Talk sessions.
• NemoClaw sandboxes running voice-enabled OpenClaw.
• NVIDIA speech integrations such as Parakeet ASR, Magpie TTS, and PersonaPlex once its serving wrapper is confirmed.
• Operators and security teams who need media egress governed by OpenShell policy instead of hidden behind host bridges.

Concrete protocol evidence

• Parakeet is served through NVIDIA ASR NIM / Riva Speech Recognition:
  • HTTP REST for offline transcription.
  • gRPC over HTTP/2 for RivaSpeechRecognition/Recognize and bidirectional StreamingRecognize.
  • Realtime WebSocket for /v1/realtime?intent=transcription, using JSON events with base64 audio.
• Magpie is served through NVIDIA TTS NIM / Riva Speech Synthesis:
  • HTTP REST for /v1/audio/synthesize and /v1/audio/synthesize_online.
  • gRPC over HTTP/2 for RivaSpeechSynthesis/Synthesize and server-streaming SynthesizeOnline.
  • Realtime WebSocket for /v1/realtime?intent=synthesize, using JSON events.
• PersonaPlex is publicly described as a full-duplex speech-to-speech model/checkpoint, not a documented Speech NIM API surface:
  • Treat its network protocol as unconfirmed until the serving wrapper is known.
  • If wrapped as Riva/NIM, expect gRPC or realtime WebSocket.
  • If exposed directly for browser voice, expect WebSocket or WebRTC.

Proposed Design

Add a governed media protocol surface to OpenShell.

This could be a new protocol: media, or staged support through:

1. WEBSOCKET_BINARY for realtime audio over WebSocket.
2. Realtime WebSocket JSON media policy.
3. protocol: grpc for streaming gRPC.
4. Optional TURN/WebRTC support later, only when there is a concrete sandbox use case.

The media policy should include:

• transport allowlist
• codec allowlist
• destination and session binding
• direction controls
• max frame/message size
• bitrate limits
• session duration limits
• concurrency limits
• OCSF media session logging

Alternatives Considered

• Keep using raw L4 passthrough. This moves bytes, but it cannot express media policy, session caps, gRPC method policy, or useful audit semantics.
• Keep VoiceClaw audio on the host forever. This avoids sandbox risk but forces every voice/media integration to rebuild a host-side shadow plane.
• Add raw UDP/RTP/WebRTC first. The current VoiceClaw POC does not require that inside the sandbox. Start with WebSocket binary and gRPC streaming, then add TURN/WebRTC only with a concrete requirement.

Agent Investigation

• Reviewed NemoClaw #6207, which frames this as the P0 OpenShell ask: governed realtime media without raw socket escape.
• Inspected local ../openclaw Voice Call code. Carrier and provider paths use WebSocket media streams; some providers use JSON events with base64 audio and some use binary audio frames.
• Checked OpenShell docs/code. protocol: websocket governs upgrade and client text frames; binary WebSocket frames are relayed but not policy-governed. protocol: grpc is not currently listed as an L7 protocol.
• Checked NVIDIA Speech NIM docs for Parakeet and Magpie protocols.

Sources:

• NVIDIA Speech NIM overview: https://docs.nvidia.com/nim/speech/latest/about/index.html
• ASR support matrix / Parakeet: https://docs.nvidia.com/nim/speech/latest/reference/support-matrix/asr.html
• ASR gRPC API: https://docs.nvidia.com/nim/speech/latest/reference/api-references/asr/protos.html
• ASR realtime WebSocket API: https://docs.nvidia.com/nim/speech/latest/reference/api-references/asr/realtime-asr.html
• TTS support matrix / Magpie: https://docs.nvidia.com/nim/speech/latest/reference/support-matrix/tts.html
• TTS HTTP API: https://docs.nvidia.com/nim/speech/latest/reference/api-references/tts/http-tts.html
• TTS gRPC API: https://docs.nvidia.com/nim/speech/latest/reference/api-references/tts/protos.html
• TTS realtime WebSocket API: https://docs.nvidia.com/nim/speech/latest/reference/api-references/tts/realtime-tts.html
• PersonaPlex paper: https://arxiv.org/abs/2602.06053

Acceptance Criteria

• Realtime audio can be allowed without granting arbitrary raw socket egress.
• WebSocket binary media is denied unless explicitly allowed.
• Realtime WebSocket JSON media can be governed by event type and size limits.
• gRPC streaming can be restricted by host, service, method, metadata, size, rate, duration, and concurrency.
• Media sessions have frame/message size, bitrate, duration, and concurrency caps.
• OCSF logs record allow, deny, and limit decisions without logging audio payloads, base64 media, or secrets.

Non-Goals

• Unrestricted raw socket escape.
• Generic UDP/RTP/WebRTC support in the first implementation.
• Audio transcription or semantic inspection inside OpenShell.
• OpenClaw provider implementation.
• NemoClaw plugin packaging work.

Checklist

• I've reviewed existing issues and the architecture docs
• This is a design proposal, not a "please build this" request

[johntmyers]

@sandl99 are you able to get media working by adding a target endpoint and using tls:skip to allow traffic to pass through un-inspected? I'd like to just know if the proxy will pass this traffic currently.

As far as adding new protocol: media|grpc types, we are unlikely to support that. This is a great candidate for using middleware: #1738 - so you could develop a small service that analyzes payloads and annotate allow/deny semantics that will trigger the OpenShell Proxy to take action.

What is the concrete MVP you would want to build here? Can we start with one of Parakeet, Magpie, etc and a small governance surface area? If the MVP requirements can be narrowed down we can work with you on using the upcoming middleware feature.

[sandl99]

Adding replies from Voice team on your ask @johntmyers
tested on OpenShell 0.0.72 (the version latest NemoClaw pins), with streaming gRPC (local ASR NIM + a cloud gRPC server) and WebSockets.

Q1 — yes, both items work through the proxy today, without enforcement:
• gRPC over HTTP/2: streaming gRPC works end-to-end — verified both local (StreamingRecognize to a local ASR NIM) and cloud (server-streaming to a gRPC server). Plaintext gRPC rides plain L4 passthrough; TLS gRPC works via a raw-passthrough endpoint (access: full + tls: skip) — an existing OpenShell policy option that NemoClaw's built-in WhatsApp channel already relies on. (On the default TLS path the proxy MITM-terminates and drops h2 ALPN → HTTP/1.1, which breaks gRPC, so TLS gRPC just needs the raw-passthrough policy.)
• Streaming WebSocket (text + binary): works over L4 — sustained bidirectional streaming, both frame types. Deny-by-default correctly blocks until the endpoint is allowed.
• Everything we tested is TCP-based (gRPC/HTTP2 + WebSocket). WebRTC (UDP) can't terminate inside the sandbox — it can give a better real-time experience on lossy networks, but it isn't required here: a WebSocket audio path is enough to unblock the in-sandbox POC.→ So yes @San Dang — it works; the in-sandbox voice POC is unblocked on the transport side.

2 — practical policies to enforce in a media WS stream once the upgrade lands:
today protocol: websocket only covers the handshake + WEBSOCKET_TEXT frames — there's no rule type for binary frames, and no L7 for gRPC. Practical examples:
• gate binary frames (and direction) while still validating the interleaved JSON control frames (e.g. OpenAI Realtime / Twilio Media Streams config messages)
• for gRPC: an HTTP/2-aware method/:path allowlist
(plus, as defense-in-depth on an opaque binary channel: volume / rate / duration caps)
This maps directly to the middleware "parse the protocol → signal allow/deny", you mentioned earlier.

@Dharmendra Choudhary (SW-TEGRA) for the POC on Nemoclaw community we have two paths from here:
(1) fully in-sandbox — swap the browser leg WebRTC→WebSocket, STT/TTS via gRPC to NIMs inside;
(2) hybrid — keep UI+Talker+WebRTC on the host (preserve WebRTC UX), and let the in-sandbox OpenClaw voice-call use gRPC to the NIMs.

@johntmyers @pimlock I noticed that #2027 was merged as a middleware feature.
I reviewed the current implementation and want to confirm the intended path for gRPC. The current middleware contract exposes EvaluateHttpRequest for buffered HTTP/1.x requests. HTTP/2 is not inspected, TLS termination advertises only http/1.1, and tls: skip relays encrypted traffic without middleware evaluation or credential injection.

Do we have any plan for HTTP/2 middleware in near timeline to unblock us ?