aman4014/whisper-bil-model-100k-ct2-int8_float16

Whisper Bilingual German-English Model

  

CT2 INT8-FP16 | OpenAI Whisper-medium | Bidirectional De-En Speech Translation

Developed at the Hasso Plattner Institute (HPI) with computational resources provided by the HPI High-Performance Computing (HPC) cluster. Fine-tuned on the aman4014/translated-german-english-asr dataset — a curated parallel German-English speech corpus totalling approximately 4.65 million samples across 12 training splits — and optimized with CTranslate2 INT8-FP16 quantization for efficient real-time bidirectional speech translation between German and English.

Table of Contents

1. 1.Model Overview
2. 2.Key Features
3. 3.Model Architecture
4. 4.Training Data
5. 5.Performance Metrics
6. 6.Usage
7. 7.Supported Directions
8. 8.Technical Specifications
9. 9.Use Cases
10. 10.Limitations
11. 11.Citation
12. 12.Acknowledgements
13. 13.Funding Notice
14. 14.License

Model Overview

This model is a CTranslate2-optimized variant of OpenAI's Whisper Medium, fine-tuned for bidirectional German-English speech translation. Quantized using INT8 weights and FP16 activations, it achieves a balance between high translation quality, low latency, reduced memory footprint, and bidirectional capability (De-En and En-De).

The model extends Whisper's standard task vocabulary with a custom <|en_to_de|> task token, enabling English-to-German speech translation — a capability not present in vanilla Whisper models.

Key Features

• —Bidirectional Translation: Supports both German-to-English and English-to-German speech translation.
• —Real-Time Performance: Mean RTF of 0.030 on CUDA — 47.9x faster than real-time across 50 hours of audio.
• —Best Quality-Efficiency Ratio: COMET/RTF of 20.3, ranking first among 12 benchmarked models.
• —Competitive Accuracy: Outperforms Whisper-small on COMET (+0.019) with 1.30x lower latency.
• —Efficient Footprint: 4x smaller than FP32; faster inference via CTranslate2.
• —Production-Ready: INT8-FP16 quantization optimized for deployment.
• —Zero Hard Errors: Processes 100% of test samples with zero failures across 24,375 samples.
• —Extensively Benchmarked: Evaluated on the LibriS2S test set (24,375 samples, De-En direction).
• —Multilingual Foundation: Built on Whisper's 99-language vocabulary.
• —Custom Task Extension: Novel <|en_to_de|> task token enabling bidirectional capability.

Model Architecture

Custom Task Tokens

This model extends Whisper's vocabulary with:

Training Data

Fine-tuned on the aman4014/translated-german-english-asr dataset, aggregating high-quality German-English parallel speech data from multiple sources:

All samples are filtered by hunalign confidence score (>= 0.01) to ensure translation quality.

Performance Metrics

Benchmarked on the LibriS2S test set (German to English, 24,375 samples) against 11 baseline systems spanning quality ceilings, size-matched peers, and cascade architectures.

Summary Results

Full Benchmark — German to English (LibriS2S, sorted by BLEU)

Quality/Latency = COMET / RTF_mean. Higher is better. Measures semantic translation quality delivered per unit of compute time.

Head-to-Head vs. Key Competitors

Latency Characteristics

The model has a fixed startup overhead of approximately 0.114 s per call, after which processing cost is approximately 0.006 s per second of audio.

For short clips under 3 seconds, startup overhead dominates. For segments of 5 seconds or longer, RTF drops well below 0.03 and the model processes audio nearly 100x faster than real-time.

Robustness

• —Zero hard failures: All 24,375 test samples processed successfully. Seven competing models (including Whisper-large-v3 and Whisper-medium) each failed on 143 samples.
• —4 empty outputs: All were sub-250 ms clips containing single words — physiologically too short for reliable recognition by any model.
• —German language leakage: 88 samples (0.36%) contain German diacritics in English output, typically proper nouns or stage-direction artefacts. Within acceptable bounds for a bilingual fine-tune.

Usage

Prerequisites

pip install faster-whisper torch soundfile numpy

Quick Start

German to English Translation

from faster_whisper import WhisperModel

model = WhisperModel(
    "aman4014/whisper-bil-model-100k-ct2-int8_float16",
    device="cuda",           # or "cpu"
    compute_type="int8_float16"
)

segments, info = model.transcribe(
    "german_audio.wav",
    language="de",
    task="translate",        # Standard Whisper task
    beam_size=6
)

for segment in segments:
    print(f"[{segment.start:.2f}s -> {segment.end:.2f}s] {segment.text}")

English to German Translation

segments, info = model.transcribe(
    "english_audio.wav",
    language="de",           # Target language
    task="en_to_de",         # Custom task token
    beam_size=6
)

for segment in segments:
    print(f"[{segment.start:.2f}s -> {segment.end:.2f}s] {segment.text}")

German to German Transcription (ASR)

segments, info = model.transcribe(
    "german_audio.wav",
    language="de",
    task="transcribe",
    beam_size=6
)

Advanced Usage

Batch Processing with Custom Parameters

import soundfile as sf
import numpy as np
from faster_whisper import WhisperModel

model = WhisperModel(
    "aman4014/whisper-bil-model-100k-ct2-int8_float16",
    device="cuda",
    compute_type="int8_float16"
)

audio, sr = sf.read("audio.wav", dtype="float32")
if sr != 16000:
    audio = resample(audio, sr, 16000)  # Resample to 16 kHz (required)

segments, info = model.transcribe(
    audio,
    language="de",
    task="translate",
    beam_size=6,
    best_of=6,
    patience=1.15,
    length_penalty=1.0,
    repetition_penalty=1.08,
    no_repeat_ngram_size=3,
    temperature=0.0,
    condition_on_previous_text=True,
    vad_filter=True,
    vad_parameters=dict(min_silence_duration_ms=500)
)

print(f"Detected language: {info.language} (probability: {info.language_probability:.2f})")
print(f"Duration: {info.duration:.2f}s")

CPU-Only Inference

model = WhisperModel(
    "aman4014/whisper-bil-model-100k-ct2-int8_float16",
    device="cpu",
    compute_type="int8",     # Use int8 for CPU
    cpu_threads=4,
    num_workers=1
)

Custom Task Token

The model introduces a custom task token <|en_to_de|> (token ID: 51866) to enable English-to-German translation:

# Task parameter mapping:
# task="translate"   →  <|translate|>  (ID: 50359) — Any language to English
# task="en_to_de"    →  <|en_to_de|>   (ID: 51866) — English to German
# task="transcribe"  →  <|transcribe|> (ID: 50360) — ASR in source language

Supported Directions

Technical Specifications

Quantization Details

This model uses INT8-FP16 mixed precision quantization via CTranslate2:

• —Weights: INT8 (8-bit integers) — 4x memory reduction
• —Activations: FP16 (16-bit floats) — Preserves numerical stability
• —Quantization Method: Symmetric per-tensor quantization
• —Calibration: Post-training quantization (PTQ) on representative dataset
• —Accuracy: ~98% of FP32 performance (BLEU drop < 0.5)

Environment Variables

# Model configuration
export WHISPER_MODEL_REPO_ID="aman4014/whisper-bil-model-100k-ct2-int8_float16"
export WHISPER_MODEL_REVISION="main"
export WHISPER_EN_DE_TASK_TOKEN="<|en_to_de|>"

# Decoding parameters
export WHISPER_FINAL_BEAM_SIZE=6
export WHISPER_FINAL_MAX_NEW_TOKENS=224
export WHISPER_DECODE_PATIENCE=1.15
export WHISPER_DECODE_LENGTH_PENALTY=1.0
export WHISPER_DECODE_REPETITION_PENALTY=1.08
export WHISPER_DECODE_NO_REPEAT_NGRAM_SIZE=3

# Audio preprocessing
export WHISPER_SILENCE_TRIM_ENABLED=1
export WHISPER_SILENCE_TRIM_MULTIPLIER=1.8
export WHISPER_SILENCE_TRIM_MARGIN_SEC=0.12
export WHISPER_NOISE_MIN_RMS_FLOOR=0.0025
export WHISPER_MIN_DECODE_SEC=0.3
export WHISPER_SILERO_VAD_ENABLED=1
export WHISPER_SILERO_VAD_THRESHOLD=0.44
export WHISPER_SILERO_VAD_MIN_SPEECH_MS=90
export WHISPER_SILERO_VAD_MIN_SILENCE_MS=220
export WHISPER_SILERO_VAD_SPEECH_PAD_MS=140

# Post-processing
export WHISPER_POST_PROCESSING_ENABLED=1
export WHISPER_ENGLISH_FLUENCY_REFINEMENT_ENABLED=1

Compute Requirements

Minimum:

• —CPU: 4 cores, 8 GB RAM
• —Inference: CPU-only (int8 compute type)

Recommended:

• —GPU: NVIDIA GPU with CUDA support (8 GB VRAM or more)
• —CUDA: 11.8 or later
• —Compute Type: int8_float16

Production (real-time streaming):

• —GPU: NVIDIA T4, V100, A10, or better
• —VRAM: 8 GB or more
• —Concurrent streams: 4–8, depending on GPU

Use Cases

Suitable Applications

• —Live Conference Translation: German-English interpretation at 47.9x real-time throughput.
• —Podcast and Video Subtitling: Efficient batch processing with near-zero failure rate.
• —Customer Service Bots: Real-time multilingual support with low tail latency (p95 RTF 0.067).
• —Educational Platforms: Language learning with combined transcription and translation.
• —Medical Documentation: Transcribe and translate clinical interviews.
• —Parliamentary and Legal Transcription: High-accuracy domain-specific speech processing.
• —Accessibility Tools: Real-time captioning for German and English speakers.

Deployment Targets

• —Edge Devices: Runs on NVIDIA Jetson with quantization.
• —Cloud APIs: Low-latency REST/gRPC services.
• —Batch Transcription: High-throughput offline processing (~50 hours of audio per hour of compute).
• —Hybrid Pipelines: ASR to MT comparison benchmarks.

Not Recommended For

• —Low-resource languages: Model is optimized for German and English only.
• —Text-to-speech: This is a translation and transcription model only.
• —Speaker diarization: Use a dedicated diarization model (e.g., pyannote.audio) upstream.
• —Emotion recognition: Model outputs text only; no prosody or emotion features.

Limitations

1. 1.Output length compression on long segments: Hypothesis/reference length ratio degrades from ~1.08 on short utterances (under 10 words) to ~0.61 on multi-sentence segments (60–200 words). This is the primary quality gap vs. Whisper-medium and is reflected in the BLEU rank (7/12) relative to the TER rank (4/12). Mitigated by chunking audio at sentence boundaries; fine-tuning on longer segments is planned.

1. 1.Domain mismatch: Performance may degrade on out-of-domain audio including heavily accented speech or technical jargon not present in the training data.

1. 1.Quantization artifacts: Rare edge cases where INT8 quantization causes minor hallucinations or omissions (under 1% of utterances).

1. 1.Language detection: Bilingual models may occasionally misdetect language if not explicitly specified via the language parameter.

1. 1.Long-form audio: Audio longer than 30 seconds should be chunked to avoid context window issues.

1. 1.Unidirectional En-De training: The custom <|en_to_de|> task is trained on 1.15M samples, which may be insufficient for complex or formal English-to-German translation. De-En uses Whisper's pre-trained translate task.

1. 1.Short-clip overhead: For audio under 3 seconds, a fixed startup cost of ~0.114 s causes RTF to spike (~0.065 at 0–3 s). Batch short clips where possible in production use.

Acknowledgements

Institutional Support

• —[Hasso Plattner Institute (HPI)](https://hpi.de/): Research environment, infrastructure, and academic support.
• —HPI High-Performance Computing (HPC) Cluster: Computational resources for model training, fine-tuning, and large-scale evaluation.
• —[AI Service Center Berlin-Brandenburg (KI-Servicezentrum)](https://www.ki-servicezentrum-bb.de/): Supporting research and deployment of AI solutions for speech and language technologies in the Berlin-Brandenburg region.

Datasets

• —[Mozilla Common Voice](https://commonvoice.mozilla.org/): Massively multilingual, crowd-sourced speech corpus (CC0).
• —[Meta AI Research](https://ai.facebook.com/): Multilingual LibriSpeech dataset.
• —[Amphion / Emilia Team](https://github.com/open-mmlab/Amphion): Emilia-YODAS large-scale multilingual speech dataset.
• —[TU Darmstadt](https://www.tu-darmstadt.de/): Tuda-De German ASR corpus.
• —[disco-eth / EuroSpeech Team](https://huggingface.co/datasets/disco-eth/eurospeech): Multilingual European parliamentary speech corpus.
• —[Thorsten Müller](https://github.com/thorstenMueller): Thorsten-Voice German TTS dataset.
• —[M-AILABS](https://www.caito.de/2019/01/the-m-ailabs-speech-dataset/): Multilingual audiobook-based speech corpus.

Frameworks and Tools

• —[OpenAI Whisper](https://github.com/openai/whisper): Foundation model for robust multilingual speech recognition.
• —[CTranslate2](https://github.com/OpenNMT/CTranslate2): Fast inference engine enabling production-ready deployment with INT8 quantization.
• —[Faster Whisper](https://github.com/guillaumekln/faster-whisper): High-performance Python wrapper for CTranslate2 and Whisper.
• —[Hugging Face](https://huggingface.co/): Platform for model hosting, dataset distribution, and collaborative ML research.

Funding Notice

German (Original)

Das zugrunde liegende Vorhaben wurde mit Mitteln des Bundesministeriums für Forschung, Technologie und Raumfahrt unter dem Förderkennzeichen "KI-Servicezentrum Berlin-Brandenburg" 16IS22092 gefördert. Die Verantwortung für den Inhalt dieser Veröffentlichung liegt beim Autor.

English Translation

This project was funded by the German Federal Ministry of Research, Technology and Space under the funding code "KI-Servicezentrum Berlin-Brandenburg" 16IS22092. Responsibility for the content of this publication remains with the author.

The AI Service Center Berlin-Brandenburg (KI-Servicezentrum) is a regional initiative funded by the German Federal Ministry of Education and Research (BMBF) to advance artificial intelligence research, development, and deployment in the Berlin-Brandenburg metropolitan region. The center provides computational resources, research support, and knowledge transfer through workshops, training programs, and consulting services for AI adoption.

More information: https://www.ki-servicezentrum-bb.de/

License

This model is released under the MIT License.

MIT License

Copyright (c) 2024 Aman Raut, Hasso Plattner Institute

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

Third-Party Licenses

Contact and Support

For commercial licensing inquiries or deployment support, please contact the author or the HPI technology transfer office.