PogusTheWhisper/Pathumma-whisper-th-large-v3-natural-noise-finetuned

Pathumma Whisper Large V3 (TH) — Natural Noise-Robust Finetuned (v4, LoRA)

Model Description

This model is a Thai Automatic Speech Recognition (ASR) system based on `nectec/Pathumma-whisper-th-large-v3`, enhanced with LoRA (Low-Rank Adaptation) fine-tuning to improve robustness in noisy environments.

It uses WhisperForConditionalGeneration with SpecAugment and gradient checkpointing to improve performance on real-world noisy and spontaneous Thai speech. Training was done on a custom dataset simulating voice messages, ambient sound, and conversational noise.

Dataset

• —Name: `tingwry/asr-augmented`
• —Description: Thai ASR dataset augmented with realistic background noise (e.g., voice messages, ambient environments) to simulate common recording conditions.

Quickstart

import torch
from transformers import pipeline

device = "cuda" if torch.cuda.is_available() else "cpu"
torch_dtype = torch.bfloat16 if torch.cuda.is_available() else torch.float32

lang = "th"
task = "transcribe"

pipe = pipeline(
    task="automatic-speech-recognition",
    model="PogusTheWhisper/Pathumma-whisper-th-large-v3-natural-noise-finetuned",
    device=device,
    torch_dtype=torch_dtype,
    chunk_length_s=30,
    return_timestamps=False
)

pipe.model.config.forced_decoder_ids = pipe.tokenizer.get_decoder_prompt_ids(language=lang, task=task)

audio_path = "<Your wav file>"
result = pipe(audio_path)

print("Full Transcription:\n", result["text"])

Model Architecture

• —Base Model: nectec/Pathumma-whisper-th-large-v3
• —Adapter Type: LoRA
• —Target Modules: q_proj, k_proj, v_proj
• —LoRA Config:
• —r=8
• —lora_alpha=32
• —lora_dropout=0.1

SpecAugment

• —mask_time_prob = 0.2
• —mask_feature_prob = 0.2

Training Arguments

• —Epochs: 8
• —Learning Rate: 2e-5
• —Scheduler: Cosine
• —Warmup Ratio: 0.05
• —Batch Size: 4 (per device)
• —Precision: bf16
• —Optimizer: AdamW (fused)
• —Gradient Checkpointing: Enabled
• —Metric: CER
• —Generation Max Length: 256
• —Generation Beams: 5

Training Results

Evaluation Performance (in Percentage)

CER

WER

Limitations and Future Work

• —Trained on Thai-only speech, not multilingual
• —Evaluated using CER, WER ; Thai word segmentation metrics will be explored in future versions
• —May not generalize well to regional dialects or highly degraded audio
• —Future improvements may include domain adaptation (e.g., medical, legal) and dialect-specific tuning

Acknowledgements

• —NECTEC for the original base model
• —OpenAI for Whisper architecture
• —SuperAI Engineer Program for mentors support
• —ThaiSC (NSTDA Supercomputer Center) for GPU compute on LANTA cluster
• —Special thanks to P'Tik, P'Joe, P'Sam, P'nut and P'Earth
• —And The Scamper SS5 House

Built with `peft==0.15.2` and `transformers==4.x`