Datasetpublic

Kaousheik/tempo

TEMPO — Temporally-grounded Multi-task Post-training for LALMs Training and evaluation data for TEMPO, a unified large audio-language model that assigns timestamps to events, speakers and sounds across speech, sound and music. Every example is a (audio, question, answer) triple whose answer is text interleaved with atomic timestamp tokens at 0.1 s resolution (<|0.0|>, <|0.1|>, … <|60.0|>), prefixed by a task tag. Structure There is one config per task and the… See the full description on the dataset page: https://huggingface.co/datasets/Kaousheik/tempo.

sourceHugging Facecc-by-4.0updated 1mo agoView on Hugging Face

0likes167downloads

Dataset Card

TEMPO — Temporally-grounded Multi-task Post-training for LALMs

Training and evaluation data for TEMPO, a unified large audio-language model that assigns timestamps to events, speakers and sounds across speech, sound and music. Every example is a (audio, question, answer) triple whose answer is text interleaved with atomic timestamp tokens at 0.1 s resolution (<|0.0|>, <|0.1|>, … <|60.0|>), prefixed by a task tag.

Structure

There is one config per task and the splits follow the paper's three-stage curriculum plus the held-out benchmark:

split	role
`synthetic_stage1`	Stage 1 SFT — synthetic corpus for temporal calibration
`sft_stage2`	Stage 2 SFT — real-world corpus
`rl`	GRPO with verifiable rewards
`val`	validation during SFT
`evaluation`	held-out benchmark reported in the paper

task	`synthetic_stage1`	`sft_stage2`	`rl`	`val`	`evaluation`	total
`asr`	8,783	5,604	5,647	355	1,189	21,578
`diarization`	8,783	5,645	5,627	342	1,181	21,578
`audio_grounding`	11,962	6,750	6,750	500	5,151	31,113
`dense_audio_captioning`	11,984	4,929	4,929	500	2,000	24,342
`music_captioning`	10,000	5,000	8,000	–	1,000	24,000
total	51,512	27,928	30,953	1,697	10,521	122,611

python

from datasets import load_dataset

# one task, one stage
ds = load_dataset("Kaousheik/tempo", "diarization", split="sft_stage2")

# the evaluation benchmark for a task
ev = load_dataset("Kaousheik/tempo", "music_captioning", split="evaluation")
print(ev[0]["question"], ev[0]["answer"][:200])

Fields

column	description
`audio`	decoded audio (`datasets.Audio`), 16 kHz
`audio_key`	stable `"<corpus>/<filename>"` identifier
`source`	originating corpus (see below)
`task`	one of the five task names
`split`	curriculum stage this row belongs to
`question`	the instruction shown to the model
`answer`	target string with timestamp tokens and task tag
`id`	per-row identifier
`audio_id`	source clip id (`audio_grounding` only, else `null`)
`source_file`	provenance of the row (evaluation ASR/diarization, else `null`)

Tasks and answer formats

task	tag	answer shape
`asr`	`[speech:asr]`	`<\	t0\	> transcript <\	t1\	>` per utterance
`diarization`	`[speech:diar]`	`<\	t0\	> Speaker N <\	t1\	>` per turn
`audio_grounding`	`[audio:ground]`	`<\	t0\	> to <\	t1\	>` per matching interval
`dense_audio_captioning`	`[audio:caption]`	`<\	t0\	> caption <\	t1\	>` per event
`music_captioning`	`[instrument]` / `[tempo]` / `[chord]` / `[stats]`	timestamped instrument entries/exits, tempo changes, chord spans, note statistics

Audio sources

corpus	tasks	unique clips
AMI	asr, diarization	8,825
ICSI	asr, diarization	3,970
AudioSet Strong	audio_grounding	11,252
TACOS	denseaudiocaptioning, audio_grounding (eval)	12,358
Slakh2100	music_captioning	15,781
synthetic (LibriSpeech + ESC-50)	asr, diarization, audiogrounding, denseaudio_captioning	32,729

84,915 unique audio clips back the rows above (clips are reused across splits and tasks).

CHiME-6 is not included. The paper's asr and diarization training data additionally drew on CHiME-6 (4,939 clips; 9,878 rows across sft_stage2, rl and val). CHiME-6 is distributed under its own registration-gated research licence that does not permit redistribution, so those rows are omitted here and the counts above are correspondingly lower than the paper's. Everything else is complete.

Encoding

All audio is 16 kHz. WAV sources are stored as FLAC, verified bit-exact against the originals; TACOS is kept in its original MP3. No resampling or channel mixing was applied.

Licensing

AMI, ICSI, AudioSet Strong, TACOS, Slakh2100 and LibriSpeech are CC BY 4.0; ESC-50 is CC BY-NC 3.0 (non-commercial research); CHiME-6 follows its own research licence (and is therefore excluded, see above). This collection is redistributed for non-commercial academic research only, consistent with each provider's intended use.