ysdede/Phi-4-mm-inst-asr-turkish-unf

<!-- This model card has been generated automatically according to the information the Trainer had access to. You should probably proofread and complete it, then remove this comment. -->

Phi-4-mm-inst-asr-turkish-unf

This model is a fine-tuned version of microsoft/Phi-4-multimodal-instruct.

Model Background: This benchmark evaluates a fine-tuned version of Microsoft's Phi-4-mm-instruct, a multimodal model not originally designed for Turkish ASR. Key points:

1. 1.Initial Limitations:
2. 2.No Turkish ASR support in base model
3. 3.Initial WER 100+%

1. 1.Fine-Tuning Process:
2. 2.Unfroze encoder layers for Turkish adaptation
3. 3.Trained for 1 epoch on Turkish audio-text pairs

1. 1.Current Status:
2. 2.Achieved significant WER reduction (100+% → 9.7% on CommonVoice)*
3. 3.Still under active development for better generalization
4. 4.Results shared as incremental progress documentation

Why This Matters:

• —Demonstrates adaptability of multimodal architectures
• —Provides baseline for Turkish ASR in resource-constrained scenarios
• —Encourages exploration of under-supported languages

• —Note on CommonVoice Results:
• —CommonVoice's relatively low WER (9.7%) may benefit from:
• —Potential speaker leakage between splits (same speakers in train/test)
• —Clean audio conditions despite non-professional recordings
• —Short utterance structure (average 4-5 seconds)
• —See below for full context on CommonVoice characteristics in the "Dataset Notes" section.

Benchmark Results

Testing Environment: Google Colab with L4 GPU (24 GB VRAM)

Dataset Notes:

• —Finetuning Datasets:
• —commonvoice_17_tr_fixed: Crowd-sourced clean speech (not professional studio recordings) with shuffled splits - potential speaker leakage (same speakers in train/test with different utterances)
• —khanacademy-turkish: Educational lectures with STEM vocabulary
• —yeni-split-0: Noisy real-world recordings

• —Benchmark-only Datasets:
• —turkishvoicedataset: Synthetic TTS news (clean but artificial prosody)
• —yeni-split-lq-noisy: Challenging noisy samples with alignment errors

Text Normalization Challenges: ⚠️ Current WER/CER scores may be inflated due to:

1. 1.Lack of standardized Turkish ASR text normalization pipeline
2. 2.Case/punctuation inconsistencies in references
3. 3.Agglutinative language morphology affecting word boundaries

Evaluation Note: For Turkish ASR benchmarking, I developed a text normalizer to address language-specific scoring challenges. While imperfect, it helps:

• —Convert numbers/dates to words
• —Standardize compound word formatting
• —Reduce punctuation-related mismatches

This preprocessing makes WER/CER calculations slightly fairer compared to raw scoring, though manual verification remains recommended. The tool is actively being refined based on validation set findings.

Performance Factors:

• —CommonVoice's relatively low WER (9.7%) likely benefits from:
• —High audio quality despite non-professional speakers
• —Potential speaker familiarity patterns (same speakers in both splits)
• —Short utterance structure (average 4-5 seconds)

Training procedure

finetuning Colab notebook

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training hyperparameters

The following hyperparameters were used during training:

• —learning_rate: 0.0001
• —trainbatchsize: 8
• —evalbatchsize: 8
• —seed: 42
• —optimizer: Use OptimizerNames.ADAMWTORCH with betas=(0.9,0.99) and epsilon=1e-07 and optimizerargs=No additional optimizer arguments
• —lrschedulertype: cosine
• —lrschedulerwarmup_ratio: 0.1
• —num_epochs: 1

Framework versions

• —Transformers 4.48.3
• —Pytorch 2.5.1+cu124
• —Datasets 3.3.2
• —Tokenizers 0.21.0