WIP: Custom Source#560
Open
MiloszFilimowski wants to merge 78 commits into
Open
Conversation
Throwaway Phase 0 spike proving react-native-webgpu can render (clear) into a GPUTexture backed by an imported IOSurface with usage RENDER_ATTACHMENT, and that the pixels land in the surface. - add react-native-webgpu@0.5.15 to fishjam-chat deps - webgpu-spike/iosurfaceRenderSpike.ts: import IOSurface via createTestVideoFrame -> importSharedTextureMemory -> createTexture(RENDER_ATTACHMENT|COPY_SRC) -> beginAccess -> render-pass clear -> copyTextureToBuffer -> endAccess -> mapAsync readback + BGRA compare; plus a re-import persistence check - webgpu-spike/IosurfaceSpikeScreen.tsx + app/(tabs)/webgpu-spike.tsx: dev-only "WebGPU Spike" tab with a Run button (no on-screen Canvas) All spike code isolated in webgpu-spike/ behind PHASE0 headers for trivial deletion. Logs PHASE0 PASS / PHASE0 FAIL to console. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…on mount Bump react-native-reanimated to ~4.2.3 and react-native-worklets to ~0.7.4 to satisfy react-native-webgpu 0.5.15 peer requirements. Add an auto-run-on-mount to the IOSurface spike screen so the gate can be triggered via deep link. GATE RESULT: PASS on iOS simulator — Dawn accepts RENDER_ATTACHMENT on an imported IOSurface; rendered color lands in the surface and persists across a fresh re-import. Adapter exposes shared-texture-memory-iosurface and shared-fence-mtl-shared-event. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Isolated webgpu-track/ module (WebGPUVideoTrack + useWebGPUCustomSource + WebGPUTrackToggle) that renders an animated scene zero-copy into IOSurface-backed buffers and publishes them via useCustomSource. In-call 'WebGPU' toggle wired into the room screen. Default scene: animated color-cycling clear. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The package re-exports createCustomVideoTrack/pushCustomVideoFrame as named exports from src/index.ts; importing createCustomVideoTrack as default made it undefined at runtime (TypeError: createCustomVideoTrack.default is not a function). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
VideosGrid only rendered cameraTrack/screenShareVideoTrack, so the published WebGPU customVideo track produced no tile on either peer. Render peer.customVideoTracks too and include them in the hasVideoTrack check. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ustomVideo A published video track is no different from a camera at the media layer (proven by webrtc-internals: VP8 480x480@30fps, decoded on the remote with 0 loss). It only failed to render because useCustomSource tagged it metadata.type=customVideo, which receivers (and the dashboard) bucket separately from the camera. Switch to useCamera().setCameraTrackMiddleware: return the WebGPU-fed native track as the camera's replacement track (replaceTrack, same msid) so it publishes as a normal camera stream and renders on any receiver with zero changes. Revert the receiver-side VideosGrid customVideoTracks rendering (no longer needed). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…rack
De-risks the unproven plumbing for "camera -> external texture -> TypeGPU
effect -> IOSurface custom track -> WebRTC", on top of the existing zero-copy
WebGPU->WebRTC track. No on-screen Canvas: output is the WebRTC stream only,
verified like the color-cycle POC.
New isolated module examples/.../fishjam-chat/webgpu-camera/:
- cameraEffectShader.ts: WGSL grayscale effect + Android YUV/orientation prelude
- WebGPUCameraEffectPipeline.ts: builds the WGSL render pipeline off the TypeGPU
root's device; worklet-safe per-frame draw into the bgra8unorm IOSurface view
- useWebGPUCameraEffect.ts: VisionCamera (sole front-camera owner) frame ->
importExternalTexture -> grayscale -> inline begin/draw/submit/endAccess on the
IOSurface pool in the frame-processor worklet -> scheduleOnRN fence retain +
pushCustomVideoFrame -> publish via useCustomSource('webgpu')
- WebGPUCameraScreen.tsx: permission + status UI (no preview)
- app/(tabs)/webgpu-camera.tsx + _layout.tsx tab entry (PHASE 3a, deletable)
WebGPUVideoTrack.ts: add worklet-serializable pool accessors + JS-side
pushPreparedFrameAndRetain so the GPU half runs in the worklet and only the
native push hops to JS. Color-cycle POC path left intact.
Config: add react-native-vision-camera 5.0.10 + worklets plugin, typegpu, bump
reanimated 4.3.1 / worklets 0.8.3, unplugin-typegpu/babel (+worklets plugin last),
VisionCamera Expo config plugin + camera permission, @webgpu/types.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add react-native-vision-camera + worklets 5.0.10, Nitro modules, typegpu 0.11.6; bump reanimated 4.3.1 / worklets 0.8.3; @webgpu/types + unplugin-typegpu (babel). Remove the vision-camera Expo config plugin entry (no app.plugin.js in 5.0.10; camera permission stays via infoPlist). Keep pixelFormat 'native' (NV12 + importExternalTexture) — the correct zero-copy device path; note the iOS simulator cannot import external/YUV textures so this verifies on a device. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…the room screen
SDK: generalize useCustomSource so a custom source can publish its video as a
regular `camera`-typed track (a virtual camera) instead of the hardcoded
`customVideo`. Add an optional second arg
`useCustomSource(sourceId, { videoType?: 'camera' | 'customVideo'; audioType?: 'customAudio' })`,
default unchanged. The chosen types are stored per-source and threaded through
the custom-source manager into fishjamClient.addTrack, so metadata.type ends up
as 'camera' and usePeers buckets it as cameraTrack on every receiver.
Example: replace the standalone WebGPU camera tab with an in-call
WebGPUCameraToggle in app/room/[roomName].tsx. The effect hook stops the SDK
camera before VisionCamera opens its capture session (iOS camera-ownership
handoff) and publishes via useCustomSource('webgpu', { videoType: 'camera' });
the toggle restores the SDK camera on off. Color-cycle webgpu-track POC left
intact.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…d error) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ture import iOS camera frames are biplanar NV12 (Y+CbCr). Importing them as a GPUExternalTexture needs the dawn-multi-planar-formats device feature; without it importExternalTexture fails BeginAccess on every frame on a real device. Add it to the useDevice requiredFeatures (matches the wgpu VisionCamera reference) + log device features. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Dawn requires beginAccess/endAccess to run on the SAME runtime that called
importSharedTextureMemory. The output pool was imported on the JS runtime in
WebGPUVideoTrack.create(), so calling beginAccess from the VisionCamera worklet
failed on device (GPUSharedTextureMemory::beginAccess() failed).
Fix: the camera path now passes RAW serializable IOSurface descriptors
({ index, surfaceHandle, width, height }) into the worklet and imports each
surface on the worklet runtime lazily (cached per index in a worklet-side box),
so import + begin/endAccess all run on the worklet runtime. The JS-thread
color-cycle POC (WebGPUVideoTrack.renderFrame) is untouched.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Removes the Fishjam SDK camera entirely from the fishjam-chat example so VisionCamera (the WebGPU camera) is the sole camera owner. This avoids the SDK stopCamera/handoff race with the camera-typed custom track that drove a WebRTC reconnect loop and stale-track-id removeTrack errors. - useWebGPUCameraEffect: drop useFishjamCamera + stopFishjamCamera() handoff; VisionCamera is the only capture. Header comment updated. - WebGPUCameraToggle: drop useCamera/startCamera/isCameraOn + restore logic; toggle just flips enabled (guarded on deviceReady + camera permission). - room/[roomName]: drop useCamera + the in-call camera button; no stopCamera on unmount (mic stop kept). - room/preview: don't start the SDK camera (initializeDevices enableVideo:false, no startCamera, no RTCView preview). Join flow + mic init intact. Example-only (Metro reload). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…pipeline Replaces the Phase 3a grayscale effect with the real MediaPipe Selfie Segmentation CNN (pure TypeGPU compute shaders, no ML runtime) + temporal/ prior/bilateral post-process + a composite that keeps the person and replaces the background with a synthetic gradient. Setup-vs-worklet split: the tgpu root, parsed .ssgbin plan, all CNN slot buffers + weights, every compute/post-process/composite pipeline are built ONCE on the JS thread and flattened to raw NativeObjects via a recording proxy (nativeRecording.ts) — tgpu objects can't cross the worklet boundary. The VisionCamera worklet only imports the per-frame camera external texture, builds its bind groups, writes the per-frame crop uniforms, and replays the recorded native dispatches into the IOSurface. IOSurface pool / GPU fence / pushCustom- VideoFrame / publish-as-camera are unchanged. New deps: typed-binary ^4.3.3, expo-asset ~12.0.12. Model bundled as assets/models/selfie_segmenter.ssgbin; metro.config.js adds .ssgbin to resolver.assetExts. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
….11.8
recordComputeDispatch's proxy pass lacked dispatchWorkgroups, so tgpu
0.11.8's isGPUComputePassEncoder guard ("dispatchWorkgroups" in value &&
!("beginRenderPass" in value)) failed. .with(proxy) then mis-classified
the proxy as a bind-group-layout-map entry instead of priors.externalPass,
so dispatchWorkgroups() created a REAL encoder/pass, applied state there,
and submitted -- setPipeline was never seen by the proxy, throwing
"tgpu did not call setPipeline" at setup.
Add dispatchWorkgroups/dispatchWorkgroupsIndirect to the proxy so the
guard passes and .with(proxy) takes the external-pass branch. tgpu's real
_applyComputeState -> applyBindGroups then drives the proxy, emitting
setPipeline(rawPipeline) + setBindGroup(index, rawGroup) for EVERY used
bind-group layout -- including tgpu's auto-managed catchall group
(uniforms/consts/slots) at its own index -- guaranteeing the worklet
replay binds the exact same groups at the exact same indices as real
execution would.
Harden the expected-error handling: only swallow MissingBindGroupsError
for frame-dependent dispatches (external-texture group bound per-frame in
the worklet); rethrow any other error (or a missing-group error on a
fully-static dispatch) so internals drift surfaces at setup on the JS
thread. Same precise catch applied to recordRenderDispatch.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…are (Android AHardwareBuffer)
Adapt the iOS IOSurface render gate to also cover Android AHardwareBuffer,
keeping iOS behavior identical:
- Format: rgba8unorm on Android, bgra8unorm on iOS (matches each platform's
native surface layout / getPreferredCanvasFormat).
- Readback assertion: RGBA byte order on Android, BGRA on iOS.
- Features: log presence of rnwebgpu/native-texture,
shared-texture-memory-{ahardware-buffer|iosurface}, dawn-multi-planar-formats,
and opaque-ycbcr-android-for-external-texture. Request the Android camera gate
(opaque-ycbcr) on Android only; iOS keeps native-texture + multi-planar.
- AHB source: RNWebGPU.createTestVideoFrame is implemented on Android
(AHardwareBuffer_allocate, R8G8B8A8_UNORM) — used exactly like iOS. If ever
unavailable, the spike now reports it clearly instead of inventing an allocator.
- Console prefix PHASE A0 on Android, PHASE0 on iOS; on-screen PASS/FAIL box kept.
Existing "WebGPU Spike" tab/route runs unchanged on both platforms.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…pter features requestDevice() fails outright if the adapter lacks a requested feature, so don't request the camera-input gates (multi-planar / opaque-ycbcr) for the render spike — request only rnwebgpu/native-texture (auto-injects shared-ahardware-buffer+sync-fd), and LOG adapter.features so we still learn camera-feature availability for later phases. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…HardwareBuffer fix) The on-disk android/build.gradle is prebuild-generated (gitignored); add expo-build-properties with android.minSdkVersion 26 so a future prebuild keeps minSdk 26 (required by react-native-webgpu's AHardwareBuffer C++). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Pinpoints the render-gate failure: import+sample+readback of an AHardwareBuffer works; only beginAccess on a RENDER_ATTACHMENT texture throws — because createTestVideoFrame's AHB lacks GPU_FRAMEBUFFER usage. Confirms the device is fully capable; render-into-AHB needs a framebuffer-usable AHB (A1's native pool). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…on Android The A0 render gate failed only because createTestVideoFrame's AHB is sampled-only. Source the render surface from the native custom-track pool (GPU_FRAMEBUFFER usage) on Android to prove render-into-AHB. iOS still uses createTestVideoFrame. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…t SDK-camera middleware
The VisionCamera-only refactor removed the SDK camera, breaking the color-cycle
toggle (its setCameraTrackMiddleware needs a live SDK camera track). Publish via
useCustomSource('webgpu',{videoType:'camera'}).setStream instead — works on both
platforms with no SDK camera, and is the simplest producer to exercise the
Android native custom-track delivery (A1b).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The VisionCamera worklet path (importExternalTexture YUV decode + sync-fd fence) is iOS-only until A2. On Android it floods 'Exception in HostFunction' per frame and collides with the color-cycle on the same custom-source id. Bail early on Android so the color-cycle (A1b native-delivery test) can be exercised cleanly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…HostFunction throw Render-into-AHB passes (A0 spike), so the color-cycle's opaque 'Exception in HostFunction' is in endAccess / fence.export() (sync-fd) / pushCustomVideoFrame. Add per-step logging (first 3 frames) + try/catch labels to identify it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…droid=rgba8unorm) The color-cycle threw an opaque 'Exception in HostFunction' every frame on Android: WebGPUVideoTrack hardcoded format:'bgra8unorm' for the imported RENDER_ATTACHMENT texture, but Android allocates RGBA8 AHardwareBuffers (format=1, R8G8B8A8_UNORM). Dawn rejects beginAccess on a format-mismatched aliased texture, so the throw landed in beginAccess (before the instrumented fence/push try-block, which is why no step: logs appeared). Pick the format by platform, matching the passing Phase 0 spike: iOS bgra8unorm, Android rgba8unorm. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Drops the Android no-op gate and makes the worklet camera path platform-aware, reusing the now-proven A1b native AHB->EGLImage->OES->WebRTC output path (no native rebuild needed): - REQUIRED_FEATURES: Android camera frames are YUV_420 AHardwareBuffers imported via an external Ycbcr Vulkan sampler, so request opaque-ycbcr-android-for-external-texture (confirmed on the adapter) instead of iOS's dawn-multi-planar-formats (NV12 IOSurfaces). - Output surface texture format: rgba8unorm on Android (RGBA8 AHB) vs bgra8unorm on iOS, same root cause as the A1b color-cycle fix; closed over the worklet as a plain string. textureSampleBaseClampToEdge on the texture_external returns RGB on both platforms (Dawn injects YUV->RGB under the ycbcr feature) and importExternalTexture applies rotation+mirror, so the segmentation shaders are unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two Android color fixes after the segmentation path came up (it ran but rendered black, then wrong colors): 1. Composite render pipeline target was hardcoded bgra8unorm, but the Android output surface is rgba8unorm (RGBA8 AHardwareBuffer). The format mismatch rendered black under Dawn-Vulkan. Made the target platform-aware to match OUTPUT_SURFACE_FORMAT. 2. The camera external texture on Android is a BGRA8 buffer that Dawn samples as BGR in the .rgb channels, so the person region had red/blue swapped (the in-shader background gradient was correct, and the A0 spike proved the output path is byte-faithful, which localized the swap to the camera input). Swizzle the camera sample to .bgr on Android at both sampling sites (CNN preprocess + composite) via a build-time Platform.OS ternary. iOS (NV12) is unchanged. DEBUG_WORKLET_CLEAR left in (set false) as an isolated bisect switch until A2 is fully verified, then removed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…rough, drop ycbcr)
The segmentation rendered, but the face came out green with magenta highlights —
the signature of YCbCr<->RGB misinterpretation, not a plain R/B swap. The .bgr
swizzle didn't fix it (it shuffled an already-scrambled result).
Hypothesis: opaque-ycbcr-android-for-external-texture makes Dawn apply a YUV->RGB
matrix when sampling, but VisionCamera 'native' delivers a BGRA8 AHardwareBuffer
(already-RGB, pixelFormat=bgra8), so the matrix scrambles it.
This checkpoint, to read ground truth:
- drop the ycbcr feature on Android (plain RGBA external import, no YUV matrix)
- DEBUG_COMPOSITE_RAW_CAMERA: composite outputs the raw camera sample (no mask /
background mix) so the true channel order is visible
- CAMERA_SWIZZLE switch ('rgb' now) + applyCameraSwizzle helper to flip channel
order from one place once ground truth is known
All temporary, removed once color is correct.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ther way) VisionCamera returns a bgra8 AHardwareBuffer for both 'native' and 'rgb' on this device, so the format request doesn't change Dawn's behavior. Reverted to 'native' and recorded the finding: importExternalTexture on Android always routes the camera AHB through the Ycbcr sampler (the ycbcr feature is mandatory or import throws) and applies a YUV->RGB matrix to RGB data, causing the magenta/green color shift. Real fix is to import the camera AHB as a regular bgra8 sampled texture instead of an external texture. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…rklet Feed the two hand models from the live camera without any CPU resize plugin (vision-camera-resize-plugin is dead with VisionCamera 5): two tiny TypeGPU render passes (charades/gpu/handTensorPasses.ts) sample the already-imported camera external texture into offscreen rgba8 targets — an aspect-preserving 192x192 letterbox for the palm detector, and a rotated 224x224 crop (affine = cropToFrame from the P8 tracking glue) for the landmark model — then copyTextureToBuffer + mapAsync read them back on the worklet runtime. The readback continuation converts RGBA->float32, runs the boxed fast-tflite models (NitroModules.box across the worklet boundary), and advances the detect-then-track state: no tracked ROI -> letterbox+detector; tracked ROI -> crop+landmarks -> next ROI via the palm-only landmarksToRoi. One readback in flight at a time; the camera render/push never waits (results lag one frame). The tracking math gained 'worklet' directives (inert strings off-device; the 11 fixture tests still pass). TEMPORARY debug output: throttled console log of index-tip position, presence, pinch ratio, and inference time. P11 replaces the log with cursor writes through the HandSource seam. Requires a physical device (the simulator cannot import camera external textures). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rjc8UZojJbMo5oa9KwDriT
On-device the readback continuation threw "ReferenceError: Property 'MINIMUM_DETECTION_SCORE' doesn't exist" (and took the JS runtime down): react-native-worklets does not carry module scope into workletized functions — the failure surfaced in a DEFAULT PARAMETER expression, which the babel plugin's closure capture does not scan. Make every workletized tracking function fully self-contained: default parameters and body constants are inlined as local literals, with the exported module constants kept as the canonical documentation for JS-side callers and tests (comments mark each inlined copy). No semantic change — the 11 fixture tests pass unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rjc8UZojJbMo5oa9KwDriT
Observed on-device: "ReferenceError: Property 'RNWebGPU' doesn't exist" at hook render. react-native-webgpu binds the RNWebGPU global to the runtime that evaluates its module init; that binding can be absent on the current runtime (crash-recovery runtime reloads, install-ordering flakes). Instead of throwing on the bare global, detect the gap with typeof and re-run WebGPUModule.install() — it binds to the CALLING runtime and is safe to repeat — logging a warning when the heal path fires. Also includes the P10 stage diagnostics (encode / mapAsync-resolve counters) used to localize silent failures during device bring-up. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rjc8UZojJbMo5oa9KwDriT
An unexplained native EXC_BAD_ACCESS at camera launch blocks the GPU tensor-extraction path on-device, so validate the models first over a path with zero WebGPU involvement: every 3rd frame the worklet converts the camera Frame to a CPU nitro Image (VisionCamera FrameConverter), rotates it upright, letterbox-resizes to 192x192 for the palm detector / axis-aligned-crops + resizes to 224x224 for the landmark model (ROI rotation deliberately ignored in this smoke path), builds float tensors from raw pixels (handCpuFrameWorklet.ts, BGRA/RGBA/... channel orders), and runs the boxed fast-tflite models synchronously — detect-then-track as before, throttled logs of tip position / presence / pinch / total ms. The GPU passes (gpu/handTensorPasses.ts + the GPU half of handFrameWorklet) stay in the tree but are no longer wired; device run #1 proved the full GPU readback chain executes (mapAsync resolves on the worklet runtime), so it can return once the native crash is root-caused. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rjc8UZojJbMo5oa9KwDriT
On-device the landmark stage silently dropped every frame (presence < 0.5 ->
endless re-detect loop): react-native-nitro-image's iOS crop() computes the
target size and then ignores it, cropping from the start point to the frame's
bottom-right corner — the landmark model saw the hand plus half the scene
squeezed into 224x224.
Replace crop()+resize() with sampleRotatedCropTensor: a bilinear warp sampled
directly from the upright frame's raw pixel buffer using the lab's crop
affine (the CPU equivalent of cv2.warpAffine, black outside the frame). Bonus:
this is the PROPER rotated crop — the axis-aligned approximation goes away
and the CPU path now matches the validated pipeline's geometry.
Also fixes an on-device-only regression the parity line caught ("
invertAffineTransform is not a function"): the worklets babel transform
de-hoists function declarations and captures a worklet's dependencies at
definition time, so callees must be defined ABOVE callers — reordered
cropTransform.ts (vitest never sees this because the worklet plugin does not
run there). Adds a presence log when tracking drops.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rjc8UZojJbMo5oa9KwDriT
nitro-image's toRawPixelData reads the underlying cgImage directly, which stays in SENSOR orientation (landscape) while the Image's width/height report the flag-corrected portrait size — so the rotated-crop sampler warped portrait coordinates over a landscape buffer and the landmark model saw garbage (presence ~0.01, endless re-detect loop). The detector path was immune because its small resize() redraws the image, baking the flag. Bake explicitly with a same-size resize() before toRawPixelData on the landmark path, and log the raw buffer's dimensions + pixel format once for verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rjc8UZojJbMo5oa9KwDriT
…handoff Live finger detection is CONFIRMED on-device (presence 0.99, pinch ratio cleanly separating ~0.1 pinched vs ~0.8+ open at ~47ms/cycle). Two additions: 1. Buffer-scale fix: iOS bakes the orientation-corrected image at screen scale (3x: 2160x3840 for a 720x1280 frame). Tracking self-corrected into buffer space after the first success, but the detector->landmark handoff mixed canonical ROIs with the 3x buffer and only bootstrapped by luck — the observed slow re-detection. The sampling affine is now scaled into buffer pixels while landmarks stay canonical. 2. TEMPORARY debug viz: the hand cursor (thumb/index midpoint + pinch with lab-tuned 0.15/0.30 hysteresis) overrides the touch cursor in the brush pass while fresh — hover leaves a thin blue dot trail (live view of tracking), pinching paints thick red strokes. Existing Clear button wipes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rjc8UZojJbMo5oa9KwDriT
…rame The CPU hand path collapsed into a pile of coordinate-space patches because nitro-image logical sizes lie: every redraw outputs at screen scale (3x on this device), and each stage trusted a different notion of the frame's size — including the detector letterbox fill, which had been silently writing out of bounds the whole time (the detector was effectively running on garbage and still occasionally bootstrapping tracking by luck). Rip out all of it. Per processed frame: ONE small resize, ONE toRawPixelData, and the MEASURED buffer dimensions become the only coordinate space. Both model inputs (192 letterbox, 224 rotated crop) are produced by the same bilinear sampler from that single buffer; ROI, landmarks and the cursor all live in the same space (the brush maps them to display uv by one measured scale factor). Detector and landmarks now run in the SAME processed frame, so hand acquisition is a single cycle instead of a lucky multi-frame handoff. fillLetterboxTensor is gone from the path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rjc8UZojJbMo5oa9KwDriT
…nted Drawing axes were swapped 90° (draw left -> stroke goes down, draw up -> stroke goes left): VisionCamera's FrameConverter returns a UIImage that is already display-oriented via its orientation flag, and the hand path rotated it AGAIN by frame.orientation — sideways content in a portrait canvas (the models coped because MediaPipe's ROI rotation absorbs a global rotation, so detection kept working while the cursor space was rotated vs the display). Use the converter image as-is; the small resize() bakes the flag into real pixels. Logs the converter dims + what frame.orientation claims, in case some platform/device returns unoriented buffers after all. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rjc8UZojJbMo5oa9KwDriT
…code gone Promote the validated debug rig into feature-shaped code: - Extract the per-frame pipeline into hand/tflite/handTrackingWorklet.ts (createHandTrackingWorkletState + processHandTrackingFrame); the camera hook now just calls it and consumes the cursor. computeRoiFromDetectorTensors moves into hand/tracking/ with the rest of the pure math. - Live cursor ring: the composite fragment draws a soft white ring at the cursor (uniform-driven, never persisted into the strokes) — hover shows where you are aiming, pinching draws. The hover dot-trail hack is gone; hover no longer paints anything into the canvas. - Delete the dead ends: the unwired GPU tensor passes (handTensorPasses.ts, handFrameWorklet.ts), the P9 parity check + its screen line + the base64 decoder, and the 1.6MB fixtures are no longer pulled into the app bundle (vitest keeps using them). Per-frame debug logs removed (one-time buffer info + failure warnings stay). Touch input remains the fallback cursor while no hand is tracked (and the only input on simulators). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rjc8UZojJbMo5oa9KwDriT
…re now The webgpu-camera/ folder (TypeGPU selfie-segmentation CNN, WebGPUCameraToggle on the Room tab, WebGPUVideoTrack) and its selfie_segmenter.ssgbin weights were research scaffolding from the WebGPU POC; the charades pipeline forked its camera/composite machinery long ago and shares nothing with it at runtime. Revert the Room tab to its original shape, drop the ssgbin Metro asset extension, and delete the folder + model. With the segmentation code gone the app typechecks completely clean for the first time on this branch (its typegpu operator-type drift was the source of every remaining tsc error). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rjc8UZojJbMo5oa9KwDriT
…dget The ~45ms synchronous tracking cycle every 3rd frame blew VisionCamera's ~33ms budget and logged constant "frame-was-late" drops. Split it: tick % 3 == 0 PHASE A: Frame -> Image -> resize -> raw pixel copy tick % 3 == 1 PHASE B: tensors + models + cursor (on A's pixels) Each phase fits inside one frame interval, so the camera pipeline stops dropping frames; the cursor gains one frame (~33ms) of latency, which is imperceptible next to the existing inference cadence. (The purpose-built alternative — VisionCamera's AsyncRunner/runAsync — is not usable yet: its own source notes scheduleOnRuntime currently throws when called from a frame processor.) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rjc8UZojJbMo5oa9KwDriT
buildCharadesBundle threw at setup on-device: tgpu rejects struct field names that start with WGSL reserved keywords, and the cursor-ring uniform used `active`. Renamed to `strength` (shader side only; the TS-facing CursorIndicatorInput keeps `active` — the packed byte layout is unchanged). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rjc8UZojJbMo5oa9KwDriT
Drop the square POC framing: - Output track is now 540x960 (9:16 portrait, matching the front camera); the composite center-crops the camera to the OUTPUT aspect (computeAspectFillCrop — a no-op on a 9:16 camera). Strokes texture and IOSurface pool follow the new dimensions. - Brush and cursor ring measure distances in v-units with x scaled by the output aspect, so strokes and the ring stay perfectly ROUND on the non-square output (radii are fractions of the output height). - Game UI: starting the camera opens a full-screen Modal (covers the tab bar) — full-bleed video with the strokes + cursor ring, a translucent ✕ (stop) top-right and a Clear pill bottom-center. The Hover toggle is gone (pinch is the primary input; touch-drag stays as the silent fallback). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rjc8UZojJbMo5oa9KwDriT
The charades tab now has a role picker: join as the drawing Host or as a
guessing Viewer. Everyone gets the same layout — full-screen host video
with small viewer camera tiles floating above the bottom controls.
- charades/game/roster.ts: role peer-metadata + findHostPeer /
selectViewerPeers (non-host fallback keeps unknown clients visible)
- charades/ui/ViewerTilesStrip.tsx: horizontal strip of viewer camera
tiles with VAD speaking borders; excludes the host (their 'camera' is
the full-screen composite), self tile first on viewer devices
- charades/ui/ViewerGameSection.tsx: viewer full-screen Modal — renders
the host's remote camera track through CharadesCanvas and publishes
the viewer's plain SDK camera + microphone (the VisionCamera-owns-the-
camera constraint only applies to the host device)
- CharadesCanvas: placeholderText prop ('Waiting for the host…')
- charades.tsx: role state, distinct sandbox peer names (Host /
Viewer-xxxx — the room manager keys peers by name), role in
peerMetadata, tiles strip in the host game modal
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Playable end-to-end (honor system): the host types a phrase and starts a
round, viewers see the prompt, and a manual 'I guessed it' button stands
in for speech-to-text until P17 (it stays as the simulator fallback).
- charades/game/protocol.ts: two reliable-channel JSON messages —
round_start {roundId, phrase, hostName} and phrase_guessed {roundId,
guesserName, phrase}. ASCII-escaped JSON over Uint8Array (no
TextEncoder dependency); defensive decode returns null on garbage.
The channel has no loopback and no sender identity, so names ride in
the payloads and senders apply their own messages locally.
- charades/game/useCharadesGame.ts: awaiting_round → round_active →
round_won reducer shared by both roles; roundId dedupe (idempotent
rebroadcasts), stale-guess rejection (first processed guess wins),
5 s winner banner then local reset + onRoundEnded (host clears
strokes), host rebroadcast of the current round 2 s after the room
grows (late joiners), viewer reset when the host leaves,
initializeDataChannel once connected.
- PhraseEntry (host overlay, upper third so the keyboard misses it),
GuessBanner (5 s winner banner, both roles), status/phrase pills,
dataChannelError surfaced on both screens.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Speech-to-text infrastructure for viewer guess detection, without any runtime usage yet — this isolates the native-dependency phase (and its dev-client rebuild) from the detection logic. - react-native-executorch ^0.9.2 + expo resource fetcher; initExecutorch registered in index.js before the router entry (require, not import — ES imports would hoist past the registration), recipe re-applied from the P5/P6 history - iOS deployment target 15.1 → 17.0 in the fishjam expo plugin (ExecuTorch requirement; affects the whole example app, POC-accepted) - charades/game/phraseMatch.ts: bag-of-words matcher — lowercase, NFD diacritic strip, punctuation drop, whole words only; a guess counts when every phrase word appears in the rolling transcript. 10 vitest cases; test script now covers charades/game too. Note: executorch 0.9.2 ships Whisper only (no Moonshine despite current docs) — P17 uses WHISPER_TINY_EN. Verified: pods install and the dev client builds with ExecuTorch + fast-tflite side by side (their EXCLUDED_ARCHS merge warning — i386 vs x86_64 — is benign on arm64). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Viewers now guess hands-free: the local microphone track (the same one published to the room) is tapped natively via startAudioExtraction (16 kHz mono f32, 100 ms batches — local tracks resolve through the pcId −1 path) and fed into ExecuTorch's streaming Whisper; each transcript update runs the bag-of-words matcher and the first hit broadcasts phrase_guessed exactly once per round. - charades/game/useGuessDetection.ts: whisper-tiny.en + FSMN VAD mounted at viewer-screen entry so the ~150 MB first-run download starts before round one; detection pipeline only runs while a round is active. Gotcha encoded in a comment: the useSpeechToText methods change identity whenever its internal isGenerating flips, so the effect reads the hook through a ref and keys only on enabled/ready/phrase/mic — depending on the methods directly tears the stream down in a loop. - ViewerGameSection: download progress in the status pill, live transcript debug line, manual button relabeled as the fallback (simulator / model-still-downloading). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Found during the live two-device run: the library's WHISPER_TINY_EN constant resolves to the CoreML .pte on iOS, and that file is missing from HuggingFace for the v0.9.0 tag (HEAD → 404), so the model load died on launch. Pin an explicit config on the XNNPACK variant (verified present; it's what Android uses anyway). Also: don't show the 'Waiting for the host…' pill while the winner banner is up — it belongs to awaiting_round only. Verified end-to-end host(phone) ↔ viewer(simulator): round_start over the data channel, host drawing streamed full-screen to the viewer, model download progressing (XNNPACK), manual guess → winner banner on both devices → strokes cleared → next round. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
'Heard: …<last 3 words>' pill under the video while a round runs, so the viewer can tell their microphone → speech-to-text path is working (replaces the raw transcript debug line). Plus two hardening details from the proven FCE-3458 transcription example: nonCommitted.text can be undefined (would have rendered literally), and empty extraction batches are skipped before streamInsert. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ublishing createDataChannelFn created a BARE ConnectionManager when no connection existed yet — no ICE-candidate listeners, no track-manager wiring, no commands-queue init. Since onOfferData skips createNewConnection() when a connection already exists, every track added afterwards was queued but never published and local ICE candidates were never sent: the peer joined fine, the data channel even worked, but no media ever flowed. This is exactly the charades regression: the game calls initializeDataChannel() the moment the room connects, so a host joining an EMPTY room (no offerData pending) always hit the bare-connection path — video broke 100% of the time, while P13 (no data channels) and rooms with existing peers (offerData arrives first) worked. Fix: route data-channel connection creation through createNewConnection, the same fully-wired path onOfferData uses. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The SDK only publishes device tracks in two places: the 'joined' event handler (whatever device is already running at join time) and toggleDevice (which startStreams when connected). The viewer joins the room FIRST and then called the raw startCamera()/startMicrophone(), which only start the local device — so the viewer's camera and mic never reached the room (self tile 'No video', host heard nothing). The Room flow never hits this because preview.tsx starts the mic before joinRoom. Enable via toggleCamera()/toggleMicrophone() instead — the supported publish-while-connected path — and surface returned DeviceErrors instead of silently ignoring the result tuples. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The guessing pipeline was fed nothing: our submodule checkout of feat/custom-video-track predates fishjam-react-native-webrtc PR #60 (FCE-3503), which is what ADDED local-track audio extraction — iOS via an AVAudioEngine input tap, Android via the ADM samples-ready callback. Before it, startAudioExtraction on a local track attached a WebRTC track sink that upstream libwebrtc silently discards (LocalAudioSource::AddSink is a no-op), so the viewer's mic never produced a single batch for Whisper. Merged origin/master (PR #60 + VoIP #59 + docs) into the submodule's feat/custom-video-track branch; conflicts were the custom-video exports in src/index.ts and the relocation of Android audio extraction into AudioExtractionController (which now owns the miniaudio constant). Native change — dev clients must be rebuilt (yarn ios), a JS reload is not enough. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…hook (P1) New workspace package adapting VisionCamera v5 to Fishjam custom video tracks, following the source-hook family pattern (useCustomSource sibling). useVisionCameraSource(sourceId, options) creates a forwarding custom video track, publishes it under sourceId via the custom-source machinery, and forwards every camera frame zero-copy (getNativeBuffer -> forwardFrame), with an optional inference worklet running on the same frames. The hook owns the async track lifecycle and the publish/unpublish ordering; consumer code is fully declarative. Frame timestamps come from Frame.timestamp normalized to a monotonic nanosecond timeline: VisionCamera 5.x reports seconds on iOS but raw nanoseconds on Android, so the normalizer discriminates by magnitude and offsets from the first frame. Rotation maps frame.orientation to the forwardFrame rotation metadata. Registered in root workspaces, bump-version.sh, and a release.yaml publish job. The /webgpu render tier lands in the next phase. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds the /webgpu entry point with useVisionCameraWebGpuSource(sourceId, options): a source hook that creates the surface pool + pooled custom video track internally, publishes under sourceId, and hides the whole per-frame ceremony extracted from the charades POC — pool-slot round-robin, per-slot shared-texture imports cached on the frame runtime, begin/endAccess (endAccess now in a finally so a throwing render callback cannot poison a slot), GPU fence export, pushFrame with normalized capture timestamps, camera de-rotation at import, and buffer release ordering. The user's worklet gets onFrame(frame, render) where render() hands a command encoder, the camera texture, and the output texture; a single submit per frame. DX helpers, all raw WGSL (no typegpu dependency): - createCameraShaderBindings: sampleCamera(uv) with the Android BT.709 limited-range YUV decode injected per platform; the hook builds the per-frame camera bind group when the bindings are registered. - createCameraPassthroughPipeline / encodeCameraPassthrough: zero-WGSL camera-to-output pass with aspect-fill crop, usable under overlay passes. - createCameraTextureResolver: opt-in resolve to a plain rgba8unorm texture for pipelines that cannot sample texture_external. - Device management: shared app-wide GPUDevice requested with the platform's required camera-import features, or a validated override. react-native-webgpu is an optional peer loaded only by this entry; the main entry's dist stays free of it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…s (P3) Both source hooks gain an `enabled` option (the declarative sibling of VisionCamera's isActive): while false, no track or surface pool exists and nothing is published. Internal managed-track hooks reset their state in effect cleanup instead of the effect body (compliant with react-hooks/set-state-in-effect). README gains the WebGPU quickstart. Validated on a physical iPhone 17 Pro: the forward tier publishes at a steady 30fps with correct 720p YUV buffers and monotonic normalized timestamps (iOS Frame.timestamp empirically confirmed to be seconds). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Conflicts: yarn.lock only — resolved by taking main's lockfile and regenerating with yarn install. Bumps the new @fishjam-cloud/react-native-vision-camera-source package to 0.28.3, matching the lockstep release that landed on main. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Render callbacks previously had to call outputTexture.createView() every frame, and GPUTextureView has no destroy()/release() in react-native-webgpu, so the native wrappers pile up on the frame runtime until GC — a residual native-heap leak in a render loop. Add a cached outputView to WebGpuFrameRenderContext: the hook builds one view per pool-slot output texture and reuses it every frame, so consumers use context.outputView instead of creating a view per frame. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.