ibm-granite/granite-speech-streaming-webgpu

Granite Live Dictation

Real-time speech recognition running entirely in the browser on WebGPU, with `ibm-granite/granite-speech-5.0-470m-turboctc` — a 473M-parameter FastConformer CTC model. Audio never leaves the device.

Words appear as you speak: each one is grey while it is still provisional and turns solid once it has been confirmed.

How it works

continuous 16 kHz capture ─► rolling buffer + energy SpeechGate (speech/silence)
   segment = contiguous speech (gate speechStarted … speechEnded)
   every ~0.4 s while speaking:
      window = audio[segment start … now]              (grows; each window overlaps the last)
      hypo   = words( CTC transcription of window )     (one forward pass, greedy decode)
      ── LocalAgreement-2 ──
      a word is COMMITTED once two consecutive windows agree on it (longest common prefix)
      everything past the agreement is TENTATIVE (shown grey)
   on a pause: one clean full-window pass commits the segment and re-anchors

The window grows rather than sliding because the model returns text with no word timestamps — there is no safe place to trim a fixed window without risking a mid-word cut. Anchoring each window at the start of the current speech segment makes reconciliation a plain prefix comparison and keeps the window bounded (a phrase between pauses). This is a port of the streaming policy from IBM's macOS GraniteLiveDictation app.

CTC is what makes it viable in a browser: one encoder pass plus a greedy collapse per window, with no autoregressive decoding and no KV cache, so re-transcribing a growing window several times a second is affordable.

Models

The CTC model emits raw lowercase text with no sentence punctuation, so punctuation and capitalization are restored by a second, separate model (togglable in the UI). Both are cached after the first load.

Measured behaviour

Numbers from this demo's own pipeline (not official model benchmarks), on 8 minutes of LibriSpeech dev-clean — 73 utterances, 1150 words, via hf-internal-testing/librispeech_asr_dummy:

"Normalized" expands contractions and possessives on both sides: the model writes i am for i'm and linnell is for linnell's, which scores as an error against LibriSpeech orthography while the audio was read correctly. Roughly half the raw error rate is that mismatch rather than misrecognition.

Latency on an M1 Max: ~300 ms per window (10.24 s of padded frames), so tentative text lands ~0.4–0.7 s behind speech and a word commits about one step later. Falling back to WASM costs ~1320 ms per window, which is no longer live — the app says so in the footer when that happens.

Requirements

• —WebGPU on a desktop browser (Chrome/Edge 113+). Mobile is not supported.
• —Microphone permission for dictation. Uploading a file needs neither.
• —~550 MB downloaded on first visit (312 MB model, 209 MB punctuator, 26 MB runtime), then served from cache.

Running locally

python3 serve.py --http     # http://localhost:8000

localhost is already a secure context, so WebGPU and the microphone both work with no certificate warning. Use plain python3 serve.py (HTTPS on 8443) to reach the demo from another device.

Editing the app

The engine (log-mel front end, tokenizer, ONNX session, SpeechGate + Reconciler streaming logic) lives in granite-engine.js, a single classic script that exposes window.GraniteEngine — a DOM-free API (init, startDictation, transcribeFile, loadAudioFile, setPunctuation). It is hand-edited directly; there is no separate source/bundle step for it in this repo. The UI shell (index.html, style.css, app.js) drives that API and owns all DOM wiring, recorder state, and waveform rendering.

Why one script rather than ES modules — this is not a packaging preference. On a private Space, huggingface.co embeds the demo in a cross-site iframe, and browsers withhold the auth cookie from CORS-mode requests there. ES module fetches are CORS-mode, so a module script would be fetched but could never execute, and the page would sit there with no error. Classic scripts and plain fetch() are authenticated in that context, every time. The same constraint explains two other pieces of the app:

• —ORT locates its backend by import()ing ort-wasm-simd-threaded.jsep.mjs, so granite-engine.js fetches that glue itself and hands it over as a blob: URL, with the wasm bytes passed as wasmBinary (see configureOrtWasm). Without this, session creation fails with "no available backend found … initWasm() failed".
• —audioWorklet.addModule() also fetches as a module, so the worklet is loaded from a blob: URL too (see captureWorkletUrl).

None of this is needed for a public Space or for local serving; it is all harmless there, and it falls back to the direct URLs if a preload fetch fails.

Rebuilding the ONNX files

ctc_conformer is a custom architecture with no optimum/transformers.js exporter, so the graph is traced by hand:

python3 export_ctc_onnx.py                  # fp32 ONNX + front-end tables, with parity checks
python3 quantize_ctc.py --only q4f16         # 4-bit weights on an fp16 graph
python3 eval_ctc.py onnx-ctc/ctc_conformer_q4f16.onnx   # WER against LibriSpeech
deno run --allow-read test-frontend.mjs      # JS front-end vs the Python one
deno run --allow-read test-decode.mjs        # JS CTC collapse + ByteLevel decode vs `tokenizers`

export_ctc_onnx.py reads the packaged model from --model-dir (default /tmp/ctcwork/turboctc, i.e. a snapshot_download of the Hub repo — note that /tmp does not survive a reboot). It writes the 1.9 GB fp32 graph that quantize_ctc.py and eval_ctc.py both work from, so run it first if that file is absent.

Two constraints shape the export, both documented in export_ctc_onnx.py: feature frames are padded to a multiple of 512 (the encoder chunks attention into 128-frame blocks at every layer and subsamples time by 4, so no other frame count avoids a partial block), and the argmax happens inside the graph so a window returns ~1 KB of ids instead of 8 MB of logits.

Quantization is constrained by what ORT-web's WebGPU EP implements: MatMulNBits at 4 bits only (8-bit silently falls back to WASM), and fp16 activations only (the fp32 depthwise Conv kernel rejects the graph). See the header of quantize_ctc.py for the full comparison.

Project layout

index.html  style.css  shared/  assets/    UI shell: page, styles, recorder/waveform chrome
app.js                                     UI wiring — drives window.GraniteEngine
granite-engine.js                          DOM-free engine: front end, tokenizer,
                                            ONNX session, SpeechGate/LocalAgreement-2
                                            streaming, exposes window.GraniteEngine
ctc-capture-worklet.js                     continuous 16 kHz mic capture
punctuator.js                              punctuation/capitalization pass
ort/                                       vendored onnxruntime-web 1.26.0 (MIT)
onnx-ctc/                                  quantized model + front-end tables
export_ctc_onnx.py  quantize_ctc.py  eval_ctc.py    model pipeline
test-frontend.mjs  test-decode.mjs   parity tests against the Python path
serve.py                             local server (--http for localhost)
upload_space.py                      deploy to the Space (allowlist, --prune)

Deploying

python3 upload_space.py --dry-run    # exactly what would be sent
python3 upload_space.py              # upload only what changed
python3 upload_space.py --prune      # also delete Space files outside the allowlist

ALLOW in upload_space.py is the single source of truth for what the Space holds. It is an explicit allowlist rather than ignore patterns because the working tree also carries superseded model exports — with ignore rules, one missing pattern silently ships gigabytes.

Two Space-specific limits worth knowing:

• —Spaces cap repo storage at 1 GB, and the commit endpoint weighs the whole payload against the remaining headroom instead of deduplicating unchanged blobs. Re-sending the 312 MB weights on every deploy therefore fails with "Repository storage limit reached", which is why the uploader compares hashes first and sends only what changed. At ~550 MB used, roughly one more model update fits before you need super_squash_history or the weights in a separate model repo.
• —A private Space is only reachable through huggingface.co. The *.static.hf.space URL answers "Invalid username or password" on its own, because the auth cookie is set by the huggingface.co embedding flow.

The ONNX runtime is vendored in ort/ rather than loaded from a CDN: Edge's Tracking Prevention interferes with the jsdelivr request, and as a blocking script that stalls the whole page. See ort/README.md.

Limitations

• —English transcription only — no translation, unlike the earlier granite-speech-4.1-2b build of this demo.
• —Sentence punctuation and capitalization come from the separate punctuator; the CTC model itself emits lowercase, unpunctuated text (it does produce apostrophes, digits and decimals).
• —Numbers are written as numerals (1st, 3.14), which is a formatting difference from spelled-out references rather than a recognition error.
• —A run-on with no pause is force-committed at 10 s, which can split a phrase.