Modelpublic

Kaousheik/tempo-sft-stage2-realonly

sourceHugging Faceotherupdated 1mo agoView on Hugging Face

0likes49downloads

Model Card

TEMPO — Table 2 — SFT (Ours) - Stage 2

[Paper](https://arxiv.org/abs/XXXX.XXXXX) · [All artifacts](https://huggingface.co/collections/Kaousheik/tempo-6a8a96ced2752994aa9034e2) · [Dataset](https://huggingface.co/datasets/Kaousheik/tempo)

Description

Real data only, no synthetic Stage 1 (1 epoch @ lr 5e-5). Isolates the contribution of the curriculum.

TEMPO is the first unified large audio-language model for timestamping across speech, sound and music. A single decoder handles all five tasks, emitting text interleaved with timestamp tokens and distinguished only by a task tag in the prompt. Three components make it work: atomic timestamp tokens (~601 dedicated tokens at 0.1 s resolution, so a timestamp is one categorical decision rather than several BPE fragments), a time-aware multi-modal projector injecting sinusoidal wall-clock encodings into frame embeddings, and a distance-aware Gaussian loss that gives partial credit to near misses instead of penalising every wrong timestamp equally.

Best for

—Assigning who spoke what, and when in meetings and conversational audio
—Localising a natural-language query to a time interval in an audio stream
—Dense captioning — segmenting a recording and describing each event with boundaries
—Timestamped music analysis — chord spans, tempo changes, instrument entries/exits

This checkpoint

Table 2 — SFT (Ours) - Stage 2 — reported results:

ASR MAE 0.85 / mIoU 64.8 / WER 47.0 · DER 25.1 · dense-cap eF1 55.2 · grounding F1 44.6

This variant uses the time-aware projector, so time_proj.pt is included and must be loaded.

TEMPO variants

checkpoint	paper row	notes
`tempo-rl-multitask`	Table 2 — SFT + Multi-task RL	Best overall. GRPO over all five tasks jointly.
`tempo-sft-stage1-2`	Table 2 — SFT (Ours) Stage 1+2	Full SFT recipe, no RL.
`tempo-sft-stage1`	Table 2 — SFT (Ours) Stage 1	Synthetic calibration only.
`tempo-sft-stage2-realonly` ← this model	Table 2 — SFT (Ours) Stage 2	Real data only, no curriculum.
`tempo-rl-singletask-music`	Table 2 — SFT + Single-task RL	GRPO on music captioning alone.
`tempo-ablation-both`	Table 3 — + Both (Ours)	Gaussian loss + time-aware projector.
`tempo-ablation-loss`	Table 3 — + Loss	Gaussian timestamp loss only.
`tempo-ablation-projector`	Table 3 — + Projector	Time-aware projector only.
`tempo-ablation-naive`	Table 3 — Naive	Neither component.

Plus the training and evaluation data: [`Kaousheik/tempo`](https://huggingface.co/datasets/Kaousheik/tempo) — five task configs with splits synthetic_stage1 / sft_stage2 / rl / val / evaluation.

Usage

python

import torch
from transformers import AutoProcessor, AutoModelForCausalLM

repo = "Kaousheik/tempo-sft-stage2-realonly"
processor = AutoProcessor.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype=torch.bfloat16, device_map="auto")

# REQUIRED: the time-aware projector lives outside the transformer checkpoint.
# Skipping this silently falls back to the stock AF3 projector and degrades timestamps.
from huggingface_hub import hf_hub_download
sd = torch.load(hf_hub_download(repo, "time_proj.pt"), map_location="cpu")
model.load_time_projector(sd)   # adapt to your loading code

Prompt with the task tag for the behaviour you want:

task	prompt tag	answer shape
multi-speaker ASR	`[speech:asr]`	`<\	t0\	> transcript <\	t1\	>` per utterance
speaker diarization	`[speech:diar]`	`<\	t0\	> Speaker N <\	t1\	>` per turn
audio temporal grounding	`[audio:ground]`	`<\	t0\	> to <\	t1\	>` per interval
dense audio captioning	`[audio:caption]`	`<\	t0\	> caption <\	t1\	>` per event
timestamped music captioning	`[audio:music]`	`[instrument]` / `[tempo]` / `[chord]` / `[stats]` spans

Timestamps are emitted as atomic tokens <|0.0|> … <|60.0|> at 0.1 s resolution.

Evaluation

Held-out benchmark of 10,521 question–answer pairs over six corpora. WER and DER are percentages (lower is better); mIoU is the symmetric harmonic mean of recall- and precision-side mean-best-IoU, so over-prediction cannot inflate it.

model	WER ↓	ASR mIoU ↑	DER ↓	diar mIoU ↑	dense eF1 ↑	ground F1 ↑
Audio Flamingo 3	209.7	19.4	103.2	2.9	3.8	3.3
Audio Flamingo Next	115.4	11.5	106.9	12.4	11.6	3.6
Qwen3-Omni	69.7	31.6	44.2	44.4	57.9	47.4
TimeAudio	–	–	–	–	43.6	40.1
TEMPO SFT Stage 1+2	44.7	63.8	25.4	70.5	58.5	46.2
TEMPO SFT + Multi-task RL	43.5	65.8	25.4	71.1	59.3	46.5

Training summary

Built on Audio Flamingo 3 — frozen Whisper-large audio encoder (~0.6B) + Qwen2-7B, joined by a two-layer MLP projector. Three stages:

stage	data	recipe
Stage 1 SFT	51,512 synthetic	2 epochs, lr 1e-4, LoRA r=128 α=256
Stage 2 SFT	32,726 real	2 epochs, lr 5e-5, from the merged Stage 1 checkpoint
GRPO	~35K	1,000 steps, lr 1e-5, LoRA r=256 α=512, 8 completions/prompt, KL β=0.01

Rewards for GRPO are the evaluation metrics themselves, bounded in [0, 1] and gated by a format check, so maximising reward and improving evaluation performance are aligned. Trained on NVIDIA A6000s; the audio encoder stays frozen throughout.

Limitations

—English only; timestamps are capped at 60 s by the token vocabulary, so longer audio must be chunked.
—Multi-speaker ASR remains hard in absolute terms (WER 43.5%) — useful for alignment, not a drop-in replacement for a dedicated ASR system.
—Chord F1 is low in absolute terms; music captioning is best treated as approximate.
—Speaker labels are consistent within a clip but arbitrary across clips.
—CHiME-6 is excluded from the released dataset (its licence forbids redistribution), so the public data is smaller than what the paper trained on.

Licence

Derived from Audio Flamingo 3, released by NVIDIA under a research-only licence, which governs this checkpoint. Non-commercial academic research only. Training corpora carry their own terms: CC BY 4.0 for AMI, ICSI, AudioSet Strong, TACOS, Slakh2100 and LibriSpeech; CC BY-NC 3.0 for ESC-50.

Citation

bibtex

@inproceedings{tempo,
  title     = {TEMPO: Temporally-grounded Multi-task Post-training for Large Audio-Language Models},
  author    = {TODO},
  booktitle = {TODO},
  year      = {2026}
}