livekit/turn-detector

1---2language:3  - en4  - es5  - fr6  - de7  - it8  - pt9  - nl10  - zh11  - ja12  - ko13  - id14  - tr15  - ru16  - hi17license: other18license_name: livekit-model-license19license_link: LICENSE20library_name: transformers21pipeline_tag: text-classification22base_model: Qwen/Qwen2.5-0.5B-Instruct23tags:24  - voice-ai25  - turn-detection26  - end-of-utterance27  - end-of-turn28  - conversational-ai29  - livekit30  - onnx31  - quantized32  - knowledge-distillation33---34 35# LiveKit Turn Detector36 37An open-weights language model for contextually-aware end-of-utterance (EOU) detection in voice AI applications. The model predicts whether a user has finished speaking based on the semantic content of their transcribed speech, providing a critical complement to voice activity detection (VAD) systems.38 39> **📖 For installation, usage examples, and integration guides, see the [LiveKit documentation](https://docs.livekit.io/agents/logic/turns/turn-detector/).**40 41## Table of Contents42 43- [Overview](#overview)44- [Model Variants](#model-variants)45- [How It Works](#how-it-works)46- [Architecture and Training](#architecture-and-training)47- [Supported Languages](#supported-languages)48- [Benchmarks](#benchmarks)49- [Usage](#usage)50- [Deployment Requirements](#deployment-requirements)51- [Limitations](#limitations)52- [License](LICENSE)53- [Resources](#resources)54 55## Overview56 57Traditional voice agents rely on voice activity detection (VAD) to determine when a user has finished speaking. VAD works by detecting the presence or absence of speech in an audio signal and applying a silence timer. While effective for detecting pauses, VAD lacks language understanding and frequently causes false positives. For example, a user who says *"I need to think about that for a moment..."* and then pauses will be interrupted by a VAD-only system, even though they clearly intend to continue.58 59This model adds semantic understanding to the turn detection process. It analyzes the transcribed text of a conversation in real time and predicts the probability that the user has completed their turn. When integrated into a voice pipeline alongside VAD, it substantially reduces unwanted interruptions while maintaining responsiveness.60 61The model is particularly effective in scenarios involving structured data input — such as dictating addresses, phone numbers, email addresses, and credit card numbers — where natural pauses between segments do not indicate completion.62 63## Model Variants64 65**Multilingual** (recommended) and  **English-only** (deprecated) are distributed as INT8 quantized ONNX models (`model_q8.onnx`) optimized for CPU inference.66 67> **⚠️ The English-only model (`EnglishModel`) is deprecated.** Use the **multilingual model (`MultilingualModel`)** for all new projects, including English-only applications. The multilingual model provides better accuracy across all languages — including English — thanks to knowledge distillation from a larger teacher model and an expanded training dataset. The English-only variant will not receive further updates.68 69## How It Works70 71The model operates on transcribed text from a speech-to-text (STT) system, not raw audio.72 731. **Input**: The recent conversation history (up to **6 turns**, truncated to **128 tokens**) is formatted using the [Qwen chat template](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct) with `<|im_start|>` / `<|im_end|>` delimiters. The final user message is left *without* the closing `<|im_end|>` token.74 752. **Prediction**: The model predicts the probability of the `<|im_end|>` token appearing next. A **high probability** indicates the user has likely finished their utterance. A **low probability** indicates they are likely to continue.76 773. **Thresholding**: Per-language thresholds (stored in `languages.json`) convert the raw probability into a binary decision. These thresholds are tuned to balance responsiveness and accuracy for each supported language.78 794. **Integration with VAD**: In the LiveKit Agents framework, the model works alongside the [Silero VAD](https://docs.livekit.io/agents/logic/turns/vad/) plugin. VAD handles speech presence detection and interruption triggering, while this model provides the semantic signal for when to commit a turn.80 81### Text Preprocessing82 83The **multilingual** variant applies the following normalization before inference:84 85- NFKC unicode normalization86- Lowercasing87- Punctuation removal (preserving apostrophes and hyphens)88- Whitespace collapsing89 90The **English-only** variant passes raw transcribed text without normalization.91 92## Architecture and Training93 94### Base Model95 96Both variants are fine-tuned from [Qwen/Qwen2.5-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct), selected for its strong performance on this task while enabling low-latency CPU inference.97 98### Knowledge Distillation99 100A **Qwen2.5-7B-Instruct** model was first fine-tuned as a teacher on end-of-turn prediction. Its knowledge was then distilled into the 0.5B student model. The distilled model approaches teacher-level accuracy while maintaining the efficiency of the smaller architecture, converging after approximately 1,500 training steps.101 102### Training Data103 104The training dataset is a mix of:105 106- **Real call center transcripts** covering diverse conversational patterns107- **Synthetic dialogues** emphasizing structured data input — addresses, email addresses, phone numbers, and credit card numbers108- **Multi-format STT outputs** to handle provider variation (e.g., "forty two" vs. "42"), ensuring consistent predictions across different STT engines without runtime overhead109 110Although structured data enhancements were added only to the English training set, performance improvements generalized across languages due to the multilingual knowledge encoded in the Qwen2.5 base model.111 112### Quantization113 114The trained model is exported to ONNX format and quantized to INT8 (`model_q8.onnx`), enabling efficient CPU-only inference with ONNX Runtime.115 116## Supported Languages117 118The multilingual model supports 14 languages. The model relies on the STT provider to report the detected language, which is then used to select the appropriate per-language threshold.119 120English, Spanish, French, German, Italian, Portuguese, Dutch, Chinese, Japanese, Korean, Indonesian, Turkish, Russian, Hindi121 122## Benchmarks123 124### Detection Accuracy (Multilingual Variant)125 126- **True positive** — the model correctly identifies the user has finished speaking.127- **True negative** — the model correctly identifies the user will continue speaking.128 129| Language | True Positive Rate | True Negative Rate |130|---|---|---|131| Hindi | 99.4% | 96.3% |132| Korean | 99.3% | 94.5% |133| French | 99.3% | 88.9% |134| Indonesian | 99.3% | 89.4% |135| Japanese | 99.3% | 88.8% |136| Dutch | 99.3% | 88.1% |137| Russian | 99.3% | 88.0% |138| German | 99.3% | 87.8% |139| Portuguese | 99.4% | 87.4% |140| Turkish | 99.3% | 87.3% |141| English | 99.3% | 87.0% |142| Chinese | 99.3% | 86.6% |143| Spanish | 99.3% | 86.0% |144| Italian | 99.3% | 85.1% |145 146### Improvement Over Prior Version147 148The multilingual v0.4.1 release achieved a **39.23% relative improvement** in handling structured inputs (emails, addresses, phone numbers, credit card numbers) compared to the prior version, reducing premature interruptions during data collection scenarios.149 150## Usage151 152The model is designed for use as a turn detection plugin within the [LiveKit Agents](https://github.com/livekit/agents) framework.153 154For complete installation instructions, code examples (Python and Node.js), and configuration options, see the **[LiveKit turn detector plugin documentation](https://docs.livekit.io/agents/logic/turns/turn-detector/)**.155 156For broader context on how turn detection fits into the voice pipeline — including VAD configuration, interruption handling, and manual turn control — see the **[Turns overview](https://docs.livekit.io/agents/logic/turns/)**.157 158## Deployment Requirements159 160- **Runtime**: CPU-only (no GPU required). Uses [ONNX Runtime](https://onnxruntime.ai/) with the `CPUExecutionProvider`.161- **RAM**: <500 MB for the multilingual model.162- **Instance type**: Use compute-optimized instances (e.g., AWS c6i, c7i). Avoid burstable instances (e.g., AWS t3, t4g) to prevent inference timeouts from CPU credit exhaustion.163- **LiveKit Cloud**: The model is deployed globally on LiveKit Cloud. Agents running there automatically use the optimized remote inference service with no local resource requirements.164 165## Limitations166 167- **Text-only input**: The model operates on STT-transcribed text and cannot incorporate prosodic cues such as pauses, intonation, or emphasis. Future versions may integrate multimodal audio features.168- **STT dependency**: Prediction quality depends on the accuracy and output format of the upstream STT provider. Mismatches between training and deployment STT formats may degrade performance.169- **Context window**: Limited to 128 tokens across a maximum of 6 conversation turns.170- **Language coverage**: Currently supports 14 languages. Performance on unsupported languages is undefined.171- **Realtime model compatibility**: Cannot be used with audio-native realtime models (e.g., OpenAI Realtime API) without adding a separate STT service, which incurs additional cost and latency.172 173## License174 175This model is released under the [LiveKit Model License](./LICENSE).176 177## Resources178 179- **[Documentation](https://docs.livekit.io/agents/logic/turns/turn-detector/)**: Full plugin documentation, installation, and integration guide.180- **[Turns Overview](https://docs.livekit.io/agents/logic/turns/)**: How turn detection fits into the LiveKit Agents voice pipeline.181- **[Blog: Improved End-of-Turn Model](https://blog.livekit.io/improved-end-of-turn-model-cuts-voice-ai-interruptions-39/)**: Technical deep dive on the multilingual distillation approach and benchmarks.182- **[Blog: Using a Transformer for Turn Detection](https://blog.livekit.io/using-a-transformer-to-improve-end-of-turn-detection/)**: Original blog post introducing the concept and architecture.183- **[Video: LiveKit Turn Detector](https://youtu.be/OZG0oZKctgw)**: Overview video demonstrating the plugin.184- **[GitHub: Plugin Source](https://github.com/livekit/agents/tree/main/livekit-plugins/livekit-plugins-turn-detector)**: Source code for the `livekit-plugins-turn-detector` package.185- **[PyPI](https://pypi.org/project/livekit-plugins-turn-detector/)** | **[npm](https://www.npmjs.com/package/@livekit/agents-plugin-livekit)**: Package registries.186