Scicom-intl/semantic-vad-eot-whisper-small

Semantic VAD — Whisper-small end-of-turn detector (audio only)

The whisper-small sibling of `Scicom-intl/semantic-vad-eot-whisper-tiny` and `…-whisper-base`: same recipe, same input contract, same data, an 88 M-parameter encoder. Given the last 8 seconds of a caller's 16 kHz audio it returns p(end of turn) — finished speaking vs paused mid-sentence — with no transcript.

88 M parameters · int8 ONNX 95 MB · the most accurate of the three (validation AUC 0.894 vs 0.877 base / 0.859 tiny; offline 0.86 / 0.88 / 0.99 at 0 / 0.2 / 0.6 s into a pause). CPU cost on one thread is ≈ 175–200 ms per prediction, 78 ms at four threads and 50 ms at eight (measured on a busy 164-core box), so it is the choice for nodes with cores to spare or for a GPU-served, batched endpoint (STT-API's RemoteEoT backend posts the PCM to a URL and is the hook for that); the tiny model remains the pick for a single CPU thread per agent.

Serving cost (int8 ONNX, measured 2026-09-08): 190–220 ms per prediction on one CPU thread standalone and 200 / 245 ms p50 / p90 inside a LiveKit agent process (≈ 3 % of a core per call; a 164-core node sustains ≈ 290 predictions/s with 64 dedicated single-thread processes at p90 244 ms). It keeps an agent worker thread busy for ~0.2 s per prediction, so run it with four threads, in a sidecar, or behind the GPU API rather than inside a dense agent process; latency does not depend on the audio length sent (fixed 8 s window).

Results

Compared with other open detectors (eot-bench, private telephony)

Same eot-bench harness for every model (100 ms causal grid over every pause ≥ 0.1 s, threshold × action_delay × timeout policy sweep, scalar metrics scored 0.2 s into each pause), random private telephony test turns, both language tags pooled. Third-party models run through eot-bench's own adapters with the language gate widened to Malay; the text detector on transcripts from our Whisper STT (segment timestamps interpolated to words, ~30 % of these short turns have no transcript); ultraVAD without the assistant context it was designed for (this set has none); LiveKit's cloud Turn Detector v1 streamed the 300-turn set once with the data owner's approval (292 of 300 turns scored, 8 failed the gateway handshake; LiveKit Cloud caps this project at ~5 streaming turns per minute, so the run was paced).

The 300 benchmark turns (300 eot / 191 hold spans) — every detector, LiveKit cloud v1 included:

LiveKit's cloud v1 lands between ultraVAD and the silence timer on this Malay-heavy telephony audio (no Malay, no telephony in its training); its local v1-mini does better. Scicom Semantic VAD is the enterprise member of this family, served from a GPU with dynamic batching, and is not public.

1 000 random test turns (1 010 eot / 569 hold spans) — the larger sample; the cloud detector was not run here:

In the pipeline and at fixed cut points

In a real LiveKit Agents 1.8 pipeline (Silero VAD → turn detector → endpointing, no STT, 300 recorded telephony turns, LiveKit defaults: VAD silence 0.55 s, min_delay 0.5 s, max_delay 3.0 s):

The cleanest separation of the family: mean p(eot) 0.83 on finished turns against 0.30 on mid-turn pauses at the moment LiveKit asks (tiny 0.81 / 0.42), so 0.5 is the natural threshold and 0.3 trades one more cut-off in 300 turns for a 97 % fast path. Per call in the pipeline's single CPU thread it took ≈200 ms on a heavily loaded box, inside LiveKit's 1 s prediction budget every time.

Offline, at fixed cut points relative to the start of each pause (AUC, same 300 turns, every pause):

Score smoothness along a pause is in line with the family (local std 0.034 over 200 ms, threshold flips 1.0 % per 20 ms step; smart-turn-v3 0.124 / 9.8 %).

Under LiveKit's [eot-bench](https://github.com/livekit/eot-bench) harness (100 ms causal grid over every pause ≥ 0.1 s, threshold × action_delay × timeout policy sweep; same adapter for all audio models, scored 0.2 s into each pause for the scalar metrics):

Where the tiny model only ties the VAD timer on latency at a 5 % cutoff budget, this one is ahead of it on every operating point of every subset, and ahead of the tiny model everywhere except latency at 10 % on the Malay 1 000-turn set. The harness asks within the first 100–300 ms of every pause, before an audio model has silence evidence; the extra encoder capacity buys the most exactly there (AUC 0.85 vs 0.80 at the pause start). In the LiveKit pipeline, which asks after the VAD's 0.4–0.55 s of silence, the two are one cut-off turn apart.

Files

Input: input_features [batch, 80, 800] float32 — Whisper log-mel of the last 8 s of audio, left-padded with zeros when shorter, do_normalize=False. Output: probability [batch, 1], already through the sigmoid.

Usage

Identical to the tiny model — substitute the repo id. In short (ONNX, no torch):

import numpy as np, onnxruntime as ort
from huggingface_hub import hf_hub_download
from transformers import WhisperFeatureExtractor

REPO, SR, WINDOW = "Scicom-intl/semantic-vad-eot-whisper-small", 16000, 8 * 16000
opts = ort.SessionOptions(); opts.intra_op_num_threads = 1
sess = ort.InferenceSession(hf_hub_download(REPO, "onnx/model.int8.onnx"), opts, providers=["CPUExecutionProvider"])
fe = WhisperFeatureExtractor(feature_size=80, sampling_rate=SR, chunk_length=8)

def p_end_of_turn(pcm: np.ndarray) -> float:
    """pcm: float32 in [-1, 1] at 16 kHz, the caller's audio up to *now* (any length)."""
    pcm = np.asarray(pcm, dtype=np.float32)
    if pcm.size and np.abs(pcm).max() > 1.5:   # int16-scale samples -> unit float
        pcm = pcm / 32768.0
    pcm = pcm[-WINDOW:] if len(pcm) >= WINDOW else np.pad(pcm, (WINDOW - len(pcm), 0))
    feats = fe([pcm], sampling_rate=SR, return_tensors="np", padding="max_length", max_length=WINDOW,
               truncation=True, do_normalize=False)["input_features"].astype(np.float32)
    return float(sess.run(None, {"input_features": feats})[0].reshape(-1)[0])

For LiveKit Agents use it as the backend of STT-API's SemanticVAD through a three-line predict(pcm) -> p(eot) backend around the ONNX snippet, exactly as on the tiny model's card.

Use p ≥ 0.5 (measured operating point above; 0.3 for a higher fast-path share). The PyTorch loading snippet (a WhisperEncoder subclass that accepts the 8 s window + the 3-layer head) is on the tiny model's card and works unchanged with this repo id (d_model 768).

License

Apache-2.0 (the Whisper encoder it fine-tunes is Apache-2.0).