thomaseibner/whisper-large-v3-turbo-us-atc-v3

whisper-large-v3-turbo-us-atc-v3

A LoRA adapter for openai/whisper-large-v3-turbo, fine-tuned on 1,216 human-transcribed clips of live US air traffic control audio — Minneapolis (KMSP) terminal frequencies plus five satellite fields — captured from VHF AM airband receivers.

What is actually new here

This is the first model in this project trained against a verified holdout. Every earlier release, including …-us-atc-v2, was trained on corpora that overlapped their own evaluation sets by 33.7–41.4%.

⚠ Do not read v2's published numbers as a baseline for these. They are not comparable, in three independent ways: different eval sets, a train/eval overlap of 96 of 285 clips, and a scorer whose number-normalisation chain changed on 2026-09-20 (worth ~2.4 WER on its own).

Results

Eval: 285 clips, held out with verified 0% train overlap, stratified by annotator-judged noise.

⚠ 20.60 is the mean of two seeds (42, 43). The published weights are seed 43, which alone scores 20.01. Quote 20.60: taking the better of two draws is an upward-biased estimate of what the checkpoint will do.

⚠ The two seeds differ by 1.18 pooled WER — about double the spread on the previous corpus (0.49). Adding satellite-field clips made the result noisier across seeds, so a single-seed number from this data is less trustworthy than one from the earlier corpus. Differences below ~1.0 WER here are not readable, and on severe the two seeds differ by 3.06 identifier-recall points on their own.

On a second, fully independent holdout — 35 clips from satellite/GA fields that no model in this project has ever trained on, decoded through the production CTranslate2 path:

⚠ v2 still wins identifier recall on that set. If callsign accuracy matters more to you than word accuracy, v2 may remain the better choice. At n=35 neither gap is statistically resolvable.

⚠ What this model outputs, and what a pipeline adds afterwards

Three live transmissions through the deployed model, showing every layer. Everything in capitals below was produced after the model, not by it.

`zmp-darwin`, 125.500 MHz

`kmsp-dep-se`, 124.700 MHz

`kmsp-dep-se`, 124.700 MHz — a frequency handoff, and a self-correction

Here the model wrote 13485 — five bare digits, exactly as a controller says a frequency. Turning that into 134.850, recognising which en-route sector owns it, and marking the transmission as a handoff are all downstream work. The model also kept the speaker's self-correction verbatim — "FedEx 17, er, FedEx 1618" — which is correct: the target is what was said, disfluencies included, not a tidied version.

The model heard "zumbro" and "torgy" — the spoken words. ZMBRO and TORGY are FAA waypoint idents, vowel-compressed spellings that are never pronounced as written; 12R is a runway code where "12 right" was said; TORGY4 is a procedure revision number that appears nowhere in the audio at all. A separate resolver produced each, using ADS-B and flight-plan data the model never sees.

This is deliberate, and it is why the numbers above are measured before resolution. A model trained to emit ZMBRO would emit it in San Diego, where the waypoint does not exist. Local nouns belong in a resolver or in decode-time context, not in the weights — so the references this model is scored against record "killer", "coult", "dwight", never KKILR, COULT, DWIYT.

That this is the right target, and not merely our convention, shows up in the live stream: on two separate transmissions this model and the unrelated Qwen model, which share no weights, produced the same spoken spelling as each other —

Neither model emitted the ident, and neither should: "zumbro" and "muscle" are the sounds in the air, while ZMBRO and MUSCL are vowel-compressed spellings that are never pronounced as written. Two independent models agreeing on the spoken form is the corpus telling you where the boundary belongs — mapping one to the other is a lookup, and a lookup belongs outside the weights.

⚠ These three are illustrations of the layering, not a sample of accuracy. All three happen to be correct. The model's word error rate is ~20%, so roughly one word in five across the eval set is not.

The same transmissions in the application

[image]

⚠ This is the application, not the model's output. The sentences are this model's; everything else on those rows was added downstream. ZMBRO, FDX1618, ↑17000, hdg 120°, 134.850 ZMP Swinns Valley LOW (handoff) and ✓ heading matches track all come from the resolver and from ADS-B — compare the tables above to see what the model actually emitted. The badges reading fine-tuned-v3 mark which model transcribed each clip, and the millisecond figures beside them are its decode latency.

Two details worth noticing. The bottom row is greyed and marked `suffix` at 42% — the pilot said only "1618 heavy" with no airline word, so the pipeline could match a flight number but not confirm an operator, and it says so rather than guessing. And the top row shows the self-correction from the table above surviving all the way to the display: the model heard the stumble, and nothing downstream tidied it.

The same audio, two models

This project runs a second, unrelated model — qwen3-asr-0.6b-us-atc-v1, a full fine-tune of Qwen3-ASR-0.6B — as a shadow lane on the identical live stream. Over one 4-minute window, 27 airline transmissions, both models on the same audio: 14 agreed word for word.

⚠ There is no human reference for these clips, so "differs" does not mean "one is wrong." Two of the rows can be settled from outside the audio, and they do not both go the same way:

• —02:39:58 — Whisper v3 is right. The pilot really did stumble: "FedEx 17, er, FedEx 1618." v3 kept the self-correction; Qwen silently tidied it away. If you are building references or training data, that difference matters more than a WER point — the target is what was said, disfluencies included, and a model that quietly repairs speech produces labels that no longer describe the audio.
• —02:38:47 — Qwen is right. Delta 2929 appears in the next two transmissions and both models agree on it there, so v3's 2979 is the error. Consistency across nearby transmissions catches things a single clip cannot.

02:40:05 is the failure class worth fearing: not a digit but a different airline — Southwest 1670 against Delta 1170. A downstream resolver would attribute that transmission to the wrong aircraft, and nothing in the sentence looks wrong.

Training

Two recipe choices were measured rather than assumed, each against a step-matched control:

• —lr 1e-3 → 3e-4 is worth −1.53 WER, confirmed across two seeds. The widely-copied HF PEFT Whisper recipe uses 1e-3; on this corpus that is roughly 100× the full-fine-tune rate and materially too high.
• —Adapting all linear layers rather than `q,v` is worth +0.88 identifier recall at no readable WER cost — but only at the corrected learning rate. At lr 1e-3 the same change costs +5.74 WER through overfitting.

⚠ Limitations

• —One airport's terminal airspace. KMSP plus nearby satellite fields. Phraseology, accents, facility names and traffic mix are local; cross-corpus ATC transfer is known to be poor (published examples run 9.82% in-domain → 57.35% out).
• —Channel generalisation is largely untested. The main eval has only 1 of 285 clips on a frequency absent from training. A 35-clip channel-disjoint check exists and is included above; it is small.
• —Adding satellite-field clips traded accuracy between strata. At fixed compute they readably cost clean-audio accuracy (+1.52 WER) while buying noisy-audio accuracy (−0.67). Appending domain data to a small corpus is not neutral.
• —Not comparable to published ATC WERs. The same model scores 13.46 and 37.62 on "ATCO2" in two papers, differing only by split and normalisation.
• —Scored on what was said, never on resolved identifiers. References record "killer", "coult", "dwight" — not KKILR, COULT, DWIYT. Identifier resolution is a separate post-processing step and is not part of these numbers.

Scoring

Numbers come from one pinned chain: normalise → fold_numbers → thousands-collapse, reported per noise stratum, with a measured seed spread so that small differences are declared unreadable rather than claimed. Decoding used max_new_tokens=96, repetition_penalty=1.1, greedy, language="en".

Every figure above was re-derived from the two seeds' stored prediction files on 2026-09-21 immediately before publishing, rather than copied forward from a results table. The two runs were first checked to share identical references on all 285 clips, so the per-seed comparison is not confounded by a reference difference.

⚠ That re-derivation disagrees with this project's own results table by 0.04 pooled WER (20.60 vs a recorded 20.64) and 0.13 on `severe` (27.37 vs 27.50); `clean`, `noisy` and identifier recall reproduce exactly. The cause is not established. Both figures are far below the ~1.0 WER readable threshold on this corpus, so nothing in the conclusions moves either way — but the number published here is the one that can be reproduced today from the stored predictions, not the one that could not be.

Intended use

Transcribing live VHF air-traffic-control audio for situational awareness, logging and research.

⚠ Not for operational use. This is a research model on safety-critical speech with a ~20% word error rate. Do not use it for anything a controller or pilot would rely on.