Modelpublic

kehanlu/interspeech-tutorial

sourceHugging Faceapache-2.0updated 14h agoView on Hugging Face

0likes

Model Card

Interspeech tutorial — DeSTA-style SpeechLLM checkpoints

Whisper-large-v3 encoder (frozen) → concat+MLP adapter → Qwen3-4B-Instruct-2507 + LoRA r32. Only the adapter and the LoRA weights are trained, so each checkpoint is ~147 MB; the base models are downloaded from their own repos at load time.

Code, configs and the full recipe: <https://github.com/kehanlu/interspeech-tutorial>

folder	training data	test-clean ASR	test-clean gender
`asr_gender`	281k ASR + a fresh 30% of the gender rows each epoch	1.81 WER	98.85
`selfgen`	281k self-generated conversational replies, no task labels	3.93 WER	98.24

asr_gender is ordinary task SFT, and it matches whisper-large-v3 on ASR (1.89) while also answering the gender question. It is the baseline.

selfgen is the interesting one: its targets were written by Qwen3-4B given only the transcript and the speaker's gender, so it has never seen a transcription or a gender label as a training target. The self-generation prompt was

<audio>{transcription} (Gender: {gender})</audio>

The audio is a passage read aloud from a book. Respond directly as a natural conversation partner. Do not mention the audio, the transcription, or the speaker attributes.

It can still do both tasks, but only if the prompt leaves room for a short answer. Asked the way asr_gender was trained ("Transcribe the speech into text") it replies with an essay about the passage and scores 52.35 WER; asked for a format it reaches 3.93:

prompt	ASR
`Transcribe the speech into text`	52.35 → 16.54 after clean-up
`Transcribe the speech word for word. Output only the transcription, with no explanation, in this format:\nAnswer: "<transcription>"`	8.94 → 3.93

Gender goes 81.87 → 98.24 the same way, with "The audio is a passage read aloud from a book. Is the speaker male or female? Answer with one word." The clean-up is the rule-based postprocess() in the tutorial repo's example/evaluate/evaluate_asr.py; it is a no-op on asr_gender, which already answers with a bare transcript.

Usage

The model code is in the GitHub repo:

bash

git clone https://github.com/kehanlu/interspeech-tutorial

python

from huggingface_hub import hf_hub_download
import sys, torch

sys.path.insert(0, "interspeech-tutorial")
from inference import SpeechLLMForInference

ckpt = hf_hub_download("kehanlu/interspeech-tutorial", "selfgen/model.ckpt")
pipe = SpeechLLMForInference.from_checkpoint(ckpt, dtype=torch.float16)   # float16 for a Colab T4
print(pipe.generate([{"role": "user",
                      "content": "<audio><|AUDIO|></audio>\n\nTranscribe the speech into text",
                      "audios": [{"audio": "sample.flac"}]}]))

A few LibriSpeech dev-clean clips are in samples/ with their transcripts in samples/samples.json.