njand/wav2vec2-xls-r-latin

Wav2Vec2 XLS-R 300M - Latin ASR (Restored Classical Pronunciation)

Fine-tuned version of `facebook/wav2vec2-xls-r-300m` for Latin Automatic Speech Recognition (ASR), trained on reading passages from njand/llpsi-speech-dataset.

• —Live Demo: Gradio Interface
• —Source Code: GitHub Repository
• —Base Model: `facebook/wav2vec2-xls-r-300m`
• —Dataset: `njand/llpsi-speech-dataset` (currently private)
• —Pronunciation Standard: Restored Classical Pronunciation
• —Orthography: Macrons (ā, ē, ī, ō, ū, ȳ); consonantal j/v

🛠️ Model Variants & Optimization

To facilitate production deployment on CPU-based infrastructure, this repository provides the model in three formats:

Why Quantized? The INT8 version offers ~3x faster inference speed on standard CPUs compared to the original FP32 PyTorch weights while retaining near-identical transcription accuracy.

🚀 Quickstart

Option 1: Standard Inference (PyTorch)

from transformers import pipeline

transcriber = pipeline("automatic-speech-recognition", model="njand/wav2vec2-xls-r-latin")
result = transcriber("sample.wav")
print(result["text"])

Option 2: Optimized Inference (Optimum ONNX Runtime)

For CPU inference, I recommend using the INT8 quantized model. Install dependencies: pip install "optimum[onnxruntime]"

from optimum.onnxruntime import ORTModelForCTC
from transformers import AutoProcessor

# Load the quantized model
model = ORTModelForCTC.from_pretrained(
    "njand/wav2vec2-xls-r-latin", 
    file_name="model_quantized.onnx"
)
processor = AutoProcessor.from_pretrained("njand/wav2vec2-xls-r-latin")

# Inference pipeline
inputs = processor("sample.wav", return_tensors="pt")
outputs = model(**inputs)
# ... decode outputs

🔤 Orthography & Text Normalization

This model transcribes audio using the following orthographic conventions:

• —Vowel Quantity: Macronizes long vowels (ā, ē, ī, ō, ū, ȳ).
• —Consonantal vs. Vocalic Glides: Distinguishes consonantal j and v from vocalic i and u (e.g., ējiciō rather than eicio, vīvus rather than uiuus).
• —Casing & Punctuation: Transcribes lowercase text with no punctuation.

📊 Cross-Validation Performance

Evaluated across 5-fold cross-validation on njand/llpsi-speech-dataset:

Note: Normalized WER/CER measure core word recognition (macrons stripped, j/v → i/u), while Strict WER/CER enforce exact macron placement and j/v orthography.

🧠 Language Model Decoding (KenLM + pyctcdecode)

To improve generalization beyond the limited training audio, I integrated a unigram language model (KenLM) using `pyctcdecode`. This decoder is not provided in this Hub, but you can view it in action at the Live Gradio Demo.

Lexicon & Model Construction

• —Lexicon Source: Derived from the comprehensive, fully inflected, macronized wordlist in the Alatius Latin Macronizer.
• —Enclitic Expansion: Every entry was programmatically augmented with common Latin enclitics (-que, -ve, -ne).
• —Generalization: Latin's rich inflectional system means a limited audio training set struggles with unseen word forms. The unigram LM supplies a morphological prior, enabling the decoder to correctly recognize and macronize valid inflected forms and enclitic combinations even if they never appeared in the training audio.

Decoder hyperparameters were tuned per fold using Optuna.

Constrained Decoding CV Performance

Impact: Drops mean WER from 13.20% → 8.70% (a 34.12% relative error reduction across all 5 folds).

Note: Character Error Rate (CER) saw a parallel improvement from 2.75% → 2.08% (-24.30% relative reduction).

Recommended Hyperparameters

Global Optuna optimization across the full dataset yielded the following optimal decoding parameters:

{
  "alpha": 0.20108,
  "beta": -0.89233,
  "unk_score_offset": -93.31052,
  "beam_prune_logp": -104.90582,
  "beam_width": 64
}

⚡ Environmental Impact & Compute

• —Hardware: NVIDIA L4 GPU (24GB VRAM) via Modal
• —Total Training Time: 10.47 hours
• —Estimated Carbon Emissions: 0.2872 kg CO2eq

⚠️ Limitations & Out-of-Scope Use

• —Pronunciation Variances: Performance will drop on audio using Ecclesiastical / Italianate pronunciation (e.g., pronouncing c before e/i as /tʃ/ rather than /k/).
• —Audio Conditions: Optimized for clear, single-speaker reading. Background noise or overlapping speakers will reduce accuracy significantly.