didiudom94/whisper-small-ko-to-en-v4-cross-attention

Whisper Small for Korean-to-English Translation: Burn to Win (불꽃야구) Specialization Edition V2

This model is a fine-tuned, Parameter-Efficient (LoRA) quantized 4-bit version of openai/whisper-small. It is specifically optimized to perform end-to-end audio translation from Korean speech directly to time-synchronized English text subtitles.

Its primary design goal is capturing fast-paced sports commentary, baseball slang, emotional on-screen graphics, and background stadium dialogue from the Korean variety show "Burn to Win" (불꽃야구).

Model Description

• —Developed by: [Your Name / Username]
• —Model Type: Sequence-to-Sequence Generative Audio Transformer (Whisper)
• —Language(s): Audio Input: Korean (ko) -> Text Output: English (en)
• —Task: End-to-End Translation (task="translate")
• —Finetuned from model: openai/whisper-small
• —Quantization: 4-bit NormalFloat (NF4) via bitsandbytes

Training & Dataset Architecture

The model was trained on a robust 66-hour master dataset fusion matrix, balancing domain-specific variety show vocabulary with general conversational stability using a precise 70:30 ratio:

1. 1.Show-Centric Data (70%): 46 hours of clean broadcast audio slices paired with custom translated English subtitles mapping complex baseball jargon and variety show filler phrases.
2. 2.General Data (30%): 20 hours of conversational multi-domain Korean audio to stabilize English sentence structure and prevent domain-specific catastrophic forgetting, pulled via:
3. 3.didiudom94/my-zeroth-audio-dataset-with-text (4,000 balanced rows)
4. 4.Bingsu/KSS_Dataset (10,000 balanced rows)

LoRA Hyperparameters

• —Rank (\(r\)): 32
• —Alpha (\(\alpha\)): 64
• —Target Modules: Core Attention Projection Vectors (["q_proj", "v_proj", "out_proj", "k_proj"])
• —Trainable Parameters: 7,077,888 (~2.84% of total weights)

Training Hyperparameters

• —Hardware Platform: NVIDIA A100 / L4 GPU Ecosystem
• —Effective Batch Size: 32 (per_device_train_batch_size=32, gradient_accumulation_steps=1)
• —Precision: Native bfloat16 (bf16=True, tf32=True)
• —Learning Rate: 3e-5 (Linear decay with 500 warmup steps)
• —Epochs: 2 full passes over the fused master dataset
• —Optimization Metric Matrix: Cross-Entropy Validation Loss (metric_for_best_model="loss")

Empirical Performance Benchmarks

Evaluation was run using a 1,000-row unseen evaluation slice drawn directly from the master test matrix. The model demonstrates a near-doubling of string-level contextual translation accuracy over the original out-of-the-box Whisper Small base weights.

Visual Ground-Truth Subtitle Inspection Sample

• —Ground Truth (Reference): how is he that fast ?
• —Fine-Tuned Model Output: why is he so fast ? (Highly contextual, fluent substitution)
• —Base OpenAI Whisper Output: we should find nobody (Complete hallucination)

How to Use & Deploy

To load this fine-tuned checkpoint directly from the Hugging Face Hub using the peft and transformers ecosystem, use the following code snippet:

import torch
from transformers import WhisperProcessor, WhisperForConditionalGeneration
from peft import PeftModel, PeftConfig

peft_model_id = "YOUR_HF_USERNAME/YOUR_REPO_NAME"

# 1. Load base configuration and processor
config = PeftConfig.from_pretrained(peft_model_id)
processor = WhisperProcessor.from_pretrained(config.base_model_name_or_path, language="english", task="translate")

# 2. Initialize underlying base architecture 
base_model = WhisperForConditionalGeneration.from_pretrained(
    config.base_model_name_or_path, 
    torch_dtype=torch.bfloat16,
    attn_implementation="sdpa",
    device_map="auto"
)

# 3. Inject fine-tuned adapter layers
model = PeftModel.from_pretrained(base_model, peft_model_id)
model.eval()
print("🎉 Custom translation engine deployed successfully!")

Limitations & Best Practices

• —Audio Clutter: While the model handles high background noise and stadium cheering exceptionally well, rapid cross-talking during loud variety segments may cause missing caption links.
• —Long-Form Media: For video files over 30 minutes, it is highly recommended to run inference using stable-ts (Stable Whisper) with arguments chunk_size=30 and condition_on_previous_text=False to maintain constant timestamp stability and eliminate trailing token duplication bugs.