cantonese-asr-lab/whisper-large-v2-cantonese-p2-lora

Whisper Large-v2 Cantonese P2 LoRA

Rank-8 LoRA adapters for openai/whisper-large-v2, trained in an independent three-epoch, two-seed Cantonese P2 follow-up. This repository is a PEFT adapter release, not a standalone full-model checkpoint. It was not produced by the source-alternating Whisper-small curriculum described elsewhere in the project.

Artifact identity

Seed 43 is placed at the repository root because it has the higher local-evaluation tolerance-two accuracy. Seed 42 has the slightly lower local-evaluation CER and remains separately downloadable.

Evaluation protocol

Both seeds use the common step-4,371 endpoint after three epochs over 23,304 unique utterances, with global batch 16 and one 80 GB H100 per arm. Evaluation uses the registered D0 decoder under transformers==4.57.6:

• —language="zh", task="transcribe";
• —num_beams=2, no_repeat_ngram_size=4;
• —repetition_penalty=1.05, max_length=225, early_stopping=True.

tol2 is the percentage of utterances with normalized character edit distance at most two. CER is character error rate. All table values are percentages.

Results

Task-aligned surfaces

The task-provided local evaluation set is denoted Public in the experiment artifacts. Because its tol2 chooses the repository-root seed, it is a disclosed selection surface for that adapter rather than an untouched test set.

For context, the corresponding Large-v2 Full-SFT recipes reach 89.46% tol2 for both seeds on Validation, but 94.68%/94.53% on local evaluation. Their CER is 6.85%/6.77% and 5.74%/5.58%, respectively. Full SFT leads on Validation while LoRA leads on local-evaluation CER in both seeds. This is an evaluation-surface-dependent result, not a general ordering of Full SFT and LoRA.

Relative to the previously released source-alternating Whisper-small system on the same local surface, seed 43 changes CER from 8.13% to 5.10%, a 3.03-point absolute and 37.31% relative decrease. This cross-model, cross-recipe comparison is not attributed to the source-alternating curriculum.

OOD script-normalization audit

The frozen scorer converts predictions from Traditional to Simplified Chinese but leaves OOD references in their predominantly Traditional form. The symmetric audit applies the same conversion to both sides without changing the decoded text.

Raw values remain the reproducible output of the frozen task scorer. Symmetric values are the appropriate companion when interpreting recognition after controlling the known reference/prediction script asymmetry. OOD does not select either adapter.

Model and training details

• —Base: openai/whisper-large-v2 (1,543,304,960 parameters).
• —LoRA: rank 8, alpha 16, dropout 0.05, attention q/v projections.
• —Trainable fraction: 0.254% of combined parameters.
• —Recorded peak allocated memory: 6.58 GiB.
• —Training data: 6,292 task-provided, 8,451 Common Voice zh-HK, and 8,561 MDCC utterances.
• —Schedule: three epochs, 4,371 optimizer steps, global batch 16, seeds 42/43.

The adapter is parameter- and memory-efficient, but was not faster than Large-v2 Full SFT in this run: recorded LoRA wall time was about 4:07 per seed, compared with approximately 3:53 and 3:41 for Full SFT. These records include checkpoint and evaluation overhead.

Usage

import torch
from peft import PeftModel
from transformers import WhisperForConditionalGeneration, WhisperProcessor

base_id = "openai/whisper-large-v2"
adapter_id = "cantonese-asr-lab/whisper-large-v2-cantonese-p2-lora"

processor = WhisperProcessor.from_pretrained(
    base_id,
    language="zh",
    task="transcribe",
)
base = WhisperForConditionalGeneration.from_pretrained(
    base_id,
    torch_dtype=torch.float16,
)
model = PeftModel.from_pretrained(base, adapter_id).cuda().eval()

generated_ids = model.generate(
    input_features,
    language="zh",
    task="transcribe",
    num_beams=2,
    no_repeat_ngram_size=4,
    repetition_penalty=1.05,
    max_length=225,
    early_stopping=True,
)
text = processor.batch_decode(generated_ids, skip_special_tokens=True)

To load seed 42, use the repository's seed42 subfolder. Reproducing the tables also requires the registered normalizer, manifest order, reference field, and pinned Transformers version.

Evidence and limitations

• —Immutable converged-P2 report
• —Repository at the documented revision

The local-evaluation surface participates in root-seed selection. Validation and local evaluation rank the six baseline families differently (two-seed mean tol2 Spearman correlation 0.60), but the available split metadata does not establish the cause. Symmetric OOD scoring is a post-hoc audit on one panel. The downloaded evidence bundle contains per-utterance products for 134 receipts; 28 later aggregate receipts do not have a per-utterance counterpart in that bundle. These adapters therefore document the registered recipes and surfaces without claiming a universal model-size or adaptation-family ranking.

Licenses and data access

The LoRA adapter files are released under the MIT License declared in this repository. The openai/whisper-large-v2 base model is separately distributed under the Apache License 2.0. Training data is not redistributed here and each source remains subject to its own terms. Information identifying the exact training-time task-data snapshot is available from the corresponding author by email; the address will be added with the final author record.