fix(asr): recover whole-window blank decodes with a length-perturbation ladder (#909) - #910
Merged
Conversation
…on ladder (#909) Parakeet TDT v3 has input cuts on which it emits nothing: the joint predicts blank at every frame from a fresh state for 11-13 s of clear speech. It reproduces in batch on the exact span, flips with tenths of a second of length, and the fp16 MLX port of the same checkpoint blanks on the same cuts, so this is the model, not CoreML quantization. The encoder output on such a cut is finite but lower in magnitude, and the outcome sits on a knife edge with respect to the length inputs (the mel normalization moves ~1% between a 12 s and a 13 s input). executeMLInferenceWithTimings, which every decode path goes through, now recovers an empty decode of a window that carries speech (>= 2 s, RMS >= -50 dBFS) by re-running from a copy of the entry decoder state with a ladder of length declarations - padding declared valid to the encoder, to the preprocessor, the audio declared 0.2 s shorter with that tail silenced, and the two combinations - keeping the first non-empty result. A good window is never touched. The ladder recovers 10 of the 11 reproduced spans and both real streaming losses. 40 longest LibriSpeech test-clean files: streaming WER 4.68% -> 4.05% at chunk 11, 7.43% -> 4.05% at chunk 7; batch 4.72% -> 3.16%.
Parakeet EOU Benchmark Results ✅Status: Benchmark passed Performance Metrics
Streaming Metrics
Test runtime: 1m8s • 09/11/2026, 02:03 PM EST RTFx = Real-Time Factor (higher is better) • Processing includes: Model inference, audio preprocessing, state management, and file I/O |
Supertonic3 Smoke Test ✅
Runtime: 0m47s Note: CI VMs lack a physical Neural Engine; the ANE-bucketed VectorEstimator falls back to CPU here. This validates download + variant resolution + synthesis, not ANE residency/perf. |
Offline VBx Pipeline ResultsSpeaker Diarization Performance (VBx Batch Mode)Optimal clustering with Hungarian algorithm for maximum accuracy
Offline VBx Pipeline Timing BreakdownTime spent in each stage of batch diarization
Speaker Diarization Research ComparisonOffline VBx achieves competitive accuracy with batch processing
Pipeline Details:
🎯 Offline VBx Test • AMI Corpus ES2004a • 1049.0s meeting audio • 150.5s processing • Test runtime: 2m 38s • 09/11/2026, 02:13 PM EST |
PocketTTS Smoke Test ✅
Runtime: 0m6s Note: PocketTTS uses CoreML MLState (macOS 15) KV cache + Mimi streaming state. CI VM lacks physical GPU — audio quality and performance may differ from Apple Silicon. |
VAD Benchmark ResultsPerformance Comparison
Dataset Details
✅: Average F1-Score above 70% |
Speaker Diarization Benchmark ResultsSpeaker Diarization PerformanceEvaluating "who spoke when" detection accuracy
Diarization Pipeline Timing BreakdownTime spent in each stage of speaker diarization
Speaker Diarization Research ComparisonResearch baselines typically achieve 18-30% DER on standard datasets
Note: RTFx shown above is from GitHub Actions runner. On Apple Silicon with ANE:
🎯 Speaker Diarization Test • AMI Corpus ES2004a • 1049.0s meeting audio • 58.6s diarization time • Test runtime: 2m 58s • 09/11/2026, 02:17 PM EST |
ASR Benchmark Results ✅Status: All benchmarks passed Parakeet v3 (multilingual)
Parakeet v2 (English-optimized)
Streaming (v3)
Streaming (v2)
Streaming tests use 5 files with 0.5s chunks to simulate real-time audio streaming 25 files per dataset • Test runtime: 9m41s • 09/11/2026, 02:13 PM EST RTFx = Real-Time Factor (higher is better) • Calculated as: Total audio duration ÷ Total processing time Expected RTFx Performance on Physical M1 Hardware:• M1 Mac: ~28x (clean), ~25x (other) Testing methodology follows HuggingFace Open ASR Leaderboard |
Sortformer High-Latency Benchmark ResultsES2004a Performance (30.4s latency config)
Sortformer High-Latency • ES2004a • Runtime: 3m 38s • 2026-09-11T18:06:47.189Z |
…only credible recoveries (#909) Review round 2. A streaming re-decode whose new audio holds no speech has an empty visible sequence but tokens suppressed before the cutoff: the window decoded fine and those tokens are seam evidence, so the ladder must not run on it. isWholeWindowBlank requires both lists empty; regression added. The energy gate is a non-silence test, not a speech test, so music or noise can reach the ladder. A recovered hypothesis now replaces the empty decode only when it is credible: at least two tokens at a mean confidence of 0.7 (genuine recoveries of the reproduced cuts score 0.89-0.93 over 33-62 tokens). On 80 MUSAN noise and 80 music files the ladder accepted nothing. The recovery is enabled for parakeet-tdt-0.6b-v3 only, the model it was demonstrated on.
#909) Music and noise decode to nothing on every window and pass the energy gate (160 of 160 MUSAN files), so without a budget a non-speech stream paid five extra passes per window while accepting nothing. The ladder now suspends after two consecutive failed recoveries and resumes when a window decodes normally.
…ed per transcription (#909) Review round 3. The previous budget suspended the ladder until a window decoded normally, which a run of pathological speech windows never does, and the counter outlived the file on a reused manager, so non-speech audio could disable the fix indefinitely for unrelated audio. EmptyDecodeRecoveryBudget replaces the counter: after two consecutive failed recoveries the ladder degrades to its first policy (encoderFull, one extra pass per empty window, the policy that recovered most of the reproduced cuts) so the recoverable speech window that follows is still probed, the full ladder returns every fifth empty window, and any recovery or normal decode restores it. The budget is reset at every batch transcribe entry and at the streaming manager's startStreaming, reset and error recovery. Regression covers two non-speech failures followed by a probed window, the periodic full ladder, and the per-session reset.
… the budget (#909) Review round 4. A reduced ladder loses for good a one-off cut that only a later policy flips (the next window's overlap covers only part of it, and single-shot decoding has no later chance), and any per-session budget needs every entry point to reset it (transcribeDiskBacked did not). With nothing cheaper than the model to tell speech from music or noise, the budget is removed: every window that decodes to nothing and carries energy gets all five policies. Measured cost on 30 s of MUSAN music on an M5 Pro, net of model load: batch 0.0 s -> 0.5 s, streaming 0.4 s -> 1.0 s, still tens of times faster than real time.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #909.
What the model does
Parakeet TDT v3 has input cuts on which it emits nothing: the joint predicts blank at every frame from a fresh state, for 11 to 13 s of clear read speech. It reproduces in batch on the exact span (
121-123859-00020–13.0 s empty, 0–12.9 s and 0–14.0 s correct), flips with tenths of a second of length, and the fp16 MLX port of the same checkpoint blanks on the same cuts (4507-16021-0032at 5–14.5, 5–14.8, 5–15.0, 5–15.2 s all empty inparakeet-mlx, 5–14 and 5–15.5 correct). So this is the model, not CoreML quantization, and no encoder precision removes it.Inside the pipeline the encoder output on a blank cut is finite, without NaNs, just lower in magnitude (median frame norm 0.38 vs 0.60 on the neighbouring good cut), and the joint never beats blank from the SOS state. The outcome sits on a knife edge with respect to the length inputs: the mel normalization moves by about 1 % between the 12 s and 13 s inputs, and declaring the zero padding valid to the encoder alone, or to the preprocessor alone, or trimming 0.2 s, each flips a different subset of the cuts back.
Change
AsrManager.executeMLInferenceWithTimings, which every decode path goes through (single-shot, batch chunks, streaming windows), recovers an empty decode of a window that carries speech (≥ 2 s of audio, RMS ≥ −50 dBFS over the actual length) by re-running from a deep copy of the decoder state the first attempt started from, with a ladder of length declarations, and keeps the first non-empty result:encoderFull:mel_lengthset to the padded frame count;preprocessorFull:audio_lengthset to the padded length;trimmedTail: the audio declared 0.2 s shorter on a frame boundary, with that tail silenced;trimmedTail + encoderFull;trimmedTail + preprocessorFull.The decoder still stops at the real frames under every policy. A good window is never touched: the ladder runs only after an empty decode, at one extra preprocessor + encoder + decoder pass per step, and a silent window (no speech energy) is left empty. Trimming costs the final 0.2 s, which the next window's overlap covers everywhere but at the very end of a stream, so it comes last.
The ladder recovers 10 of the 11 reproduced spans (the 13.1 s cut of case A is the holdout) and both real streaming losses. Docs: new "Empty-Window Recovery" section in
LongTranscription.md.Tests
EmptyDecodeRecoveryTests: the energy/length gate (speech-level tone retries; silence, near-silence, under 2 s, and zero padding do not), themel_lengthoverride (only the length changes, fused frontends pass through), and the ladder order and names. The model-level behavior is pinned by the batch benchmark and the streaming fixtures.Verification
Release build, M5 Pro. The three fixture clips at chunk sizes 6 to 11 and in batch, and the #855 repro clip, are unchanged (the ladder never engages on them). 40 longest LibriSpeech test-clean files (18 min, 2,691 reference words, same normalization as #908), no file worse in any run:
Batch improves too: the last, end-aligned chunk of a file can land on such a cut just as a streaming window can.
CI benchmark note
The CI ASR benchmark on this branch reports v3 test-other at 1.59% where the #908 heads reported 1.19%. The per-file artifacts of the two runs differ in exactly one record,
1688-142285-0002.flac, whose hypothesis string is byte-identical in both (You don't mean that you thought me so silly.) yet scores 0% WER in the #908 run and 10% WER / 2.9% CER in the #910 run. The other 99 records across both models and both subsets are identical. Locally the same file scores 0% in eight separate processes on this build, and the recovery ladder never engages on any of the 25 CI files for v2 or v3 on either subset. The CI scorer is producing different numbers for identical text between runs; filed separately as a benchmark nondeterminism.Round 2
isWholeWindowBlankrequires both lists empty; regression added for an all-suppressed window.All reproduced spans still recover, both real streaming losses still recover, the fixture clips are unchanged, and the 40-file WER is unchanged by the gates: 4.05% / 4.05% / 3.16%.
Round 3
EmptyDecodeRecoveryBudget: after two consecutive failed recoveries the ladder degrades to its first policy (encoderFull, one extra pass per empty window, the policy that recovered most of the reproduced cuts), so the recoverable speech window that follows non-speech audio is still probed; the full ladder returns every fifth empty window; any recovery or normal decode restores it. The budget is reset at every batchtranscribeentry and at the streaming manager'sstartStreaming,resetand error recovery. Regression covers two non-speech failures followed by a probed window, the periodic full ladder, and the per-session reset.Reproduced spans and both real streaming losses still recover; the 40-file WER on this head is unchanged: 4.05% / 4.05% / 3.16%.
Round 4
transcribeDiskBackeddid not. With nothing cheaper than the model itself to tell speech from music or noise, the budget is gone: every window that decodes to nothing and carries energy gets all five policies, and there is no state to reset.EmptyDecodeRecoveryBudget, its tests, the reset hook and the streaming reset calls are removed.LongTranscription.md.All ten previously recovered spans and both real streaming losses still recover (the 13.1 s cut of case A remains the holdout); the 40-file WER on this head is unchanged: 4.05% / 4.05% / 3.16%.