Luigi/Qwen3-ASR-1.7B-TW-Agent

Qwen3-ASR-1.7B-TW-Agent

LoRA adapter turning Qwen/Qwen3-ASR-1.7B into a Taiwan-localized speech-to-action agent: it transcribes zh-TW/English speech and acts on it in the same decoder pass — one model instead of an ASR + LLM pipeline.

Unusually for a task fine-tune, it is better than its own base model at ASR in both languages while also being a competent tool-caller.

<transcript>請幫我轉接行銷部的陳怡君</transcript>
<tool_call>
{"name": "search_contacts", "arguments": {"query": "陳怡君", "department": "行銷"}}
</tool_call>

Benchmarks

ASR — 300 Common Voice 17 zh-TW test clips (CER, OpenCC s2tw-normalized) and 200 LibriSpeech test-clean clips (WER, case-insensitive); identical pipeline for every row:

Tool-calling — BFCL v4, 11 non-agentic categories (no java/js), 150 rows/category, scored with BFCL's official ast_checker. TW-BFCL v4 is a zh-TW localization of the same categories:

Training: joint SFT → on-policy distillation → format refresher

Three stages, each repairing what the previous one broke — the intermediate failures are reported because they are the interesting part:

1. 1.Joint multi-task SFT (82.7k rows: zh-TW + English ASR, telephone dialogs, ~54k function-calling examples at 50/50 zh/en with 13% "decline to call" negatives; 33.9% audio share; frozen audio encoder, LoRA r16 on the decoder). Gives strong tool-calling and the best zh CER (2.65) — but English ASR collapses to 30.23 WER, catastrophic forgetting from a Chinese-heavy mix.
2. 2.On-policy distillation (arXiv 2605.28139): the student transcribes audio itself, the frozen base Qwen3-ASR-1.7B scores the student's own transcript on the same audio, and the loss is a temperature-scaled KL over the union of teacher/student top-k tokens. 1500 steps took English 30.23 → 1.65 WER — supervised training on reference transcripts had failed here regardless of data volume, because it never visits the error states the model itself produces. Cost: the tool-call format eroded (ASR rollouts always end after one short block, so the model learned to emit <think> and stop).
3. 3.Text-FC refresher: 6000 function-calling rows, 188 steps at lr 2e-5. Restored tool-calling (BFCL 84.95) with English WER unchanged at 1.65.

Caveats

• —Trained for zh-TW and English; other languages are untested.
• —Never trained on 8 kHz telephone-channel audio — benchmarks are 16 kHz. Real phone deployments should be re-validated.
• —TW-BFCL scores are not comparable to English-BFCL leaderboard numbers: its gold answers are augmented with zh alternatives, and ~29% of AST rows are flagged _solvable=false (cross-lingual entity mismatch).
• —Multi-turn full-trajectory tool use is weaker than single-turn (~58% in earlier in-house testing).

Usage

from qwen_asr.core.transformers_backend.modeling_qwen3_asr import Qwen3ASRForConditionalGeneration
from qwen_asr.core.transformers_backend.processing_qwen3_asr import Qwen3ASRProcessor
from peft import PeftModel
import torch

base = "Qwen/Qwen3-ASR-1.7B"          # NOTE: the nested-config repo, not the -hf variant
proc = Qwen3ASRProcessor.from_pretrained(base)
thinker = Qwen3ASRForConditionalGeneration.from_pretrained(base, dtype=torch.bfloat16).thinker
model = PeftModel.from_pretrained(thinker, "<this repo>").cuda().eval()
# render system prompt + tools + an <|audio_pad|> user turn with the processor, then generate;
# output is <transcript>...</transcript> followed by a Hermes <tool_call> or a spoken reply.

For text-only use (e.g. BFCL harnesses), merge the adapter into a plain Qwen3ForCausalLM.

Smaller sibling: Luigi/Qwen3-ASR-0.6B-TW-Agent.