thomaseibner/qwen3-asr-0.6b-us-atc-v1

qwen3-asr-0.6b-us-atc-v1

Qwen/Qwen3-ASR-0.6B, fully fine-tuned (no LoRA) on live US air traffic control audio from Minneapolis (KMSP) terminal frequencies plus nearby satellite fields, captured from VHF AM airband receivers.

⚠ Read this before the numbers

This model is not better than the Whisper fine-tune from the same project. On the identical 285-clip holdout, whisper-large-v3-turbo-us-atc-v3 scores 20.60 WER against this model's 23.84.

It is published because it fails differently. On the same clips the two models each win roughly a third and tie the rest, and an oracle picking the better transcript per clip beats the better single model by 5.04 WER — more than twice the size of any data or recipe effect measured in this project. Two models that make different mistakes are more useful than one better model when a human, or a second pass, gets to choose. That is the intended use here.

It is also trained mostly on machine labels. Of 14,256 training rows, 8,456 are pseudo-labels — transcripts produced by another model, not by a human. Only 5,800 are human-transcribed. See Training data.

Results

Eval: 285 clips, human-transcribed, held out with verified 0% train overlap, stratified by annotator-judged noise. Scored by one pinned chain (normalise → fold_numbers → thousands-collapse).

Two seeds (42, 43). Quote the mean; the published weights are seed 42.

Per noise stratum, with context, mean of both seeds:

⚠ The two seeds differ by 3.15 WER on `severe` alone (35.94 vs 39.09) against 0.61 pooled. Hard audio is where this model is least stable, and a single-seed number on severe should not be trusted.

What the ADS-B context does, and does not do

The live configuration passes a system prompt containing the callsigns of aircraft currently in the airspace, from ADS-B, plus a short glossary of standard ATC phraseology. For this model that is worth −0.40 WER (24.24 → 23.84) and +3.04 identifier recall — nearly all of the benefit is in getting callsigns right, which is what it was added for.

⚠ THE MODEL CAN TRANSCRIBE ITS OWN PROMPT INSTEAD OF THE AUDIO, AND THIS RECIPE IS NOT IMMUNE. A small model handed a word list sometimes returns the list. Every leaked word is an insertion, so it inflates WER while telling you nothing about whether the context helped.

The published weights show none of it. But an identically-trained second seed showed two, so this is a low-rate, seed-dependent risk of the recipe, not a property it is free of — and the honest reading of one clean seed is "not observed here", not "cannot happen". What leaked was the phraseology glossary, not the callsign list: one clip came back as "Minneapolis, 2 foxtrot, crossing direct roger, heavy, super, go around, climb and maintain, descend and maintain, cleared to land, cleared for takeoff, line up" against a five-word reference.

⚠ Because seed 43 carries those two clips, the two-seed means above are very slightly pessimistic for this failure rather than optimistic. If you pass a context list to a model this size, check its output for your own prompt.

How to supply the ADS-B context

The 23.84 / 90.27 row is not reproducible without this, so here is the exact construction.

The system message is a bare comma list — a glossary, then the aircraft actually airborne on that frequency at that clip's own timestamp. Nothing else. It is sent as an ordinary system message alongside the audio:

STATIC = (
    "Air traffic control radio. "
    "ILS, VFR, IFR, squawk, ident, heavy, super, go around, "
    "climb and maintain, descend and maintain, cleared to land, "
    "cleared for takeoff, line up and wait, contact departure, "
    "turn left heading, reduce speed, expedite, traffic in sight."
)

# callsigns as an EAR would hear them, best candidate first, at most 12
names = ["Southwest 3672", "Execjet 694", "Frontier 1615", "Delta 2982", "Delta 2437"]
system = STATIC + " " + ", ".join(names) + "." if names else STATIC

messages = [
    {"role": "system", "content": system},
    {"role": "user", "content": [
        {"type": "audio_url", "audio_url": {"url": f"data:audio/wav;base64,{b64}"}}]},
]
# vLLM OpenAI-compatible /v1/chat/completions
body = {"model": MODEL, "messages": messages, "temperature": 0.01, "max_tokens": 256}

A real one, emitted verbatim by the builder for eval clip 1566440:

Air traffic control radio. ILS, VFR, IFR, squawk, ident, heavy, super, go around, climb and
maintain, descend and maintain, cleared to land, cleared for takeoff, line up and wait, contact
departure, turn left heading, reduce speed, expedite, traffic in sight. Southwest 3672, Execjet
694, Frontier 1615, Delta 2982, Delta 2437.

Across the 1,383 cached eval contexts the roster holds a median of 10 aircraft; 506 hit the cap of 12 and only 4 are empty. So in practice the callsign list is about as long as the glossary in front of it.

The rules that make it work, each learned by breaking it

• —Spoken forms, never resolved identifiers. "Delta 766", never "DAL766". A model shown DAL766 learns to emit resolved tokens, and the reference says "delta 766" because that is what was said. Map the ICAO prefix to its telephony word, capitalise, and keep the digits as digits.
• —⚠ Declarative, not instructions. This model does not take orders. An instruction-shaped prompt — "Transcribe the transmission. Output only the transcription. Write numbers as digits…" — returned language None and an empty transcription on a clip that shorter prompts got right. Qwen3-ASR is ASR-only by design, to "mitigate instruction injection and instruction-following failures". The system slot is a place to put words the model might hear, not commands.
• —⚠ No preamble sentence. "Aircraft likely on this frequency right now:" is a sentence about the list rather than list content, and sentences are the shape that blanked the output. Let the callsigns join the glossary as more of the same.
• —⚠ Output format cannot be prompted at all. "Write numbers as digits" changed the output by not one character — byte-identical to no prompt. The model writes "one two five point five" where the reference says "125.5". Fold both sides at scoring time; never ask. Curiously, context does what instruction could not: with the roster present the same clip came back "Delta 2977" — correct case and digits — where the bare model wrote "delta twenty nine seven seven". Biasing appears to carry format with it.
• —Cap the list at ~12, ranked. A longer list is not a better hint, it is a wider net. Rank by in-sector, then route match, then told-to-call-this-frequency, then distance, and take the head.
• —An empty roster degrades to the glossary alone, never to a dangling label.
• —⚠ Never put anything derived from the clip's own transcript in the prompt. Our context endpoint also returns the tags this clip was assigned — which come from transcribing it. Feeding those back leaks the answer and produces a beautiful meaningless number. Only ADS-B/SFDPS-derived aircraft are used here, because those are independent of the audio.

⚠ The hint is often wrong, and that is the ceiling

The spoken truth is in the roster 80.5% of the time (62 of 77 clips naming an aircraft). A hint that does not contain the answer cannot supply it — and worse, it offers a plausible neighbour instead. On one eval clip the truth is "Delta 766"; the roster carries Delta 762 at 6 nm and no DAL766 at all. That near-miss is exactly what a biased decoder snaps to. Measure your own roster's hit rate before trusting a gain to biasing.

If you would rather not risk the glossary leaking

The two clips that reproduced the prompt reproduced the glossary, not the callsigns. A callsigns-only system message — ", ".join(names) + ".", no glossary — removes that surface entirely. It was built as a minimum-surface arm for exactly this reason. It is not the configuration these numbers were measured in, so treat it as a different setup rather than a free improvement.

The same audio, two models

The claim at the top of this card — that this model earns its place by failing differently — is testable on live audio. This model runs as a shadow lane beside whisper-large-v3-turbo-us-atc-v3 on the identical stream. Over one 4-minute window, 27 airline transmissions, both models on the same audio: 14 agreed word for word.

⚠ There is no human reference for these clips, so "differs" does not mean "one is wrong." Two of the rows can be settled from outside the audio, and they do not both go the same way:

• —02:39:58 — Whisper v3 is right. The pilot really did stumble: "FedEx 17, er, FedEx 1618." v3 kept the self-correction; Qwen silently tidied it away. If you are building references or training data, that difference matters more than a WER point — the target is what was said, disfluencies included, and a model that quietly repairs speech produces labels that no longer describe the audio.
• —02:38:47 — Qwen is right. Delta 2929 appears in the next two transmissions and both models agree on it there, so v3's 2979 is the error. Consistency across nearby transmissions catches things a single clip cannot.

02:40:05 is the failure class worth fearing: not a digit but a different airline — Southwest 1670 against Delta 1170. A downstream resolver would attribute that transmission to the wrong aircraft, and nothing in the sentence looks wrong.

Training data

The machine labels measurably help word accuracy. At genuinely matched gradient steps they are worth −2.48 WER with context and −5.08 without, against an otherwise identical human-only arm.

⚠ They do not measurably help identifier recall. +1.58 points, against a measured between-seed identifier spread that puts the readable threshold at ≈3.46. Treat the identifier half as unproven.

⚠ An earlier version of this comparison was confounded and is corrected here: the human-only arm ran 1,088 steps against the machine-label arm's 2,673, so data and compute moved together and the result was unattributable. The arm published here is step-matched at 1,090.

Training

training_args.json in this repo records every effective argument, the upstream trainer commit, and the library versions.

⚠ A learning-rate finding from the Whisper model in this project did NOT transfer, and the reason is structural. On Whisper, lowering the LoRA rate from 1e-3 to 3e-4 was the single largest improvement available (−1.53 WER). Halving this model's rate to 1e-5 cost +4.87 WER without context, +11.58 with it, and +14.13 on `severe`. The Whisper gain came from LoRA, where the working heuristic is a rate ~10× the full-fine-tune rate; this is a full fine-tune, where that heuristic says nothing. Copying a hyperparameter across because both are spelled "learning rate" is a category error.

⚠ Limitations

• —One airport's terminal airspace. KMSP plus nearby satellite fields. Phraseology, accents, facility names and traffic mix are local; cross-corpus ATC transfer is known to be poor.
• —Mostly pseudo-labelled. 59% of the training rows were never checked by a human, so any systematic error in the labelling model is partly baked in here.
• —`severe` audio is weak and unstable — 37.52 WER, with a 3.15 spread between seeds.
• —Not comparable to published ATC WERs. The same model can score 13.46 and 37.62 on the same public benchmark in two papers, differing only by split and normalisation.
• —Scored on what was said, never on resolved identifiers. References record "killer", "muscle", "dwight" — not KKILR, MUSCL, DWIYT. Identifier resolution is a separate post-processing step and is not part of these numbers.
• —Single-seed spread is real. Two seeds is enough to see that differences under ~1 WER pooled, and under ~3.5 identifier points, are not readable here. It is not enough to estimate that spread well.

Intended use

A second opinion alongside a stronger primary model, for transcribing live VHF air-traffic-control audio for situational awareness, logging and research. Its value is in disagreeing usefully.

⚠ Not for operational use. This is a research model on safety-critical speech with a ~24% word error rate. Do not use it for anything a controller or pilot would rely on.