voiceping-ai/whisper-ja-en-speech-translation

Whisper EN-JA Speech Translation

Bidirectional EN<->JA speech translation at 212 tok/s — 4x faster than Whisper large-v3 and SeamlessM4T v2, with only 756M parameters.

Benchmark

[image]

Quality scored on FLEURS test samples (1-5 scale: accuracy + fluency). Speed benchmarked on NVIDIA GPU with bfloat16.

Quick Start

pip install torch transformers librosa

import torch
import librosa
from transformers import WhisperProcessor, WhisperForConditionalGeneration

MODEL_ID = "voiceping-ai/whisper-ja-en-speech-translation"

processor = WhisperProcessor.from_pretrained(MODEL_ID)
model = WhisperForConditionalGeneration.from_pretrained(MODEL_ID)

# Load audio (16kHz mono)
audio, sr = librosa.load("english_audio.wav", sr=16000)

input_features = processor(
    audio, sampling_rate=16000, return_tensors="pt"
).input_features

# EN audio -> JA text: set language to source language
forced_decoder_ids = processor.get_decoder_prompt_ids(
    language="en", task="translate"
)
model.config.forced_decoder_ids = forced_decoder_ids

with torch.no_grad():
    predicted_ids = model.generate(input_features)

print(processor.batch_decode(predicted_ids, skip_special_tokens=True)[0])
# Example: "しかし、通信の速度が遅いため、西洋では二十五年から三十年ほど遅れをとることがあります。"

# JA audio -> EN text: set language to source language
forced_decoder_ids = processor.get_decoder_prompt_ids(
    language="ja", task="translate"
)
model.config.forced_decoder_ids = forced_decoder_ids

with torch.no_grad():
    predicted_ids = model.generate(input_features)

print(processor.batch_decode(predicted_ids, skip_special_tokens=True)[0])

Note on `forced_decoder_ids`: In newer versions of transformers (>=4.40), pass forced_decoder_ids via model.config.forced_decoder_ids rather than as a keyword argument to model.generate().

Standalone Inference Script

See `inference.py` for a complete standalone script that handles audio file input, device selection, and both translation directions.

# EN audio -> JA text
python inference.py audio_en.wav --direction en2ja

# JA audio -> EN text
python inference.py audio_ja.wav --direction ja2en

# Use GPU
python inference.py audio.wav --direction en2ja --device cuda:0

Model Details

Architecture

This model is a distilled variant of OpenAI Whisper large-v2:

The distilled architecture keeps the full 32-layer encoder for strong audio understanding while reducing the decoder from 32 to 4 layers for faster inference. This makes the model significantly faster than the full Whisper large-v2 while preserving translation quality.

How Translation Direction Works

Whisper's original translate task always outputs English. This model extends that capability by fine-tuning on bidirectional translation pairs, so the translate task can produce either Japanese or English depending on the source language token.

The translation direction is controlled via forced_decoder_ids:

• —language="en" + task="translate" = EN audio -> JA text
• —language="ja" + task="translate" = JA audio -> EN text

The language parameter specifies the source audio language, and the model outputs the translation in the opposite language.

Training

The model was fine-tuned for bidirectional speech translation (EN<->JA) using paired audio-text translation data in both directions.

• —Task: Speech translation (task="translate")
• —Encoder: Frozen during training (pre-trained representations preserved)
• —Decoder: Fine-tuned for translation output
• —Optimizer: AdamW
• —Learning rate: 2e-4 with cosine-with-restarts scheduler
• —Epochs: 20
• —Batch size: 72
• —Label smoothing: 0.1
• —Gradient checkpointing: Enabled
• —Audio filtering: Minimum 2 seconds duration
• —Text normalization: Applied during training (Japanese Kanji normalization, punctuation handling)

Evaluation

Evaluated on the FLEURS test set for both translation directions.

Metrics are computed with language-appropriate text normalization:

• —English: BasicTextNormalizer (lowercase, remove punctuation/articles)
• —Japanese: Ginza tokenization with Kanji display-form normalization and Japanese punctuation removal

Translation Examples

Side-by-side comparison on FLEURS test set samples:

EN -> JA

JA -> EN

Limitations

• —Audio length: Best performance on audio segments under 30 seconds
• —Language pair: Only supports EN<->JA translation (not general-purpose multilingual)
• —Translation quality: As a distilled model, quality may be lower than the full Whisper large-v2 on some inputs
• —Domain: Trained primarily on general-domain speech; specialized domains (medical, legal, etc.) may have lower accuracy
• —No timestamps: This model does not output timestamp tokens

License

Apache 2.0