MounishAllam/qwen2.5-omni-audio-temporal-qlora

Qwen2.5-Omni-7B QLoRA — Audio Temporal Reasoning

A LoRA adapter for [Qwen/Qwen2.5-Omni-7B](https://huggingface.co/Qwen/Qwen2.5-Omni-7B), fine-tuned with QLoRA on AudioCaps to reduce audio hallucination and improve temporal-ordering accuracy when describing the sequence of sound events in an audio clip.

Full code, training recipe, and an honest results write-up (including failure cases and what didn't improve) is in the [GitHub repo](https://github.com/Mounish-Allam/AUDIO-TEMPORAL-REASONING-FINE-TUNNING). A static results showcase is also available as a Hugging Face Space.

What this adapter does

Given an audio clip and a prompt like "Describe what happens first, then next, and finally in this audio", the fine-tuned model answers in one plain sentence describing the sound events in the order they occur — trained to avoid inventing sounds that aren't in the clip and to prefer a concise, accurate answer over a rambling one.

This is the production / qlora_v3 run from the project's model registry: trained on 3,000 real AudioCaps clips for 2 epochs. See MODELS.md for the two smaller (300 / 1,000-sample) runs, whose weights were lost to an overwrite bug — this is the only run whose weights survive.

Results (real, not simulated)

Evaluated on 300 held-out AudioCaps test clips (train/test leakage-guarded). Sound event recall dipped slightly — the model got more conservative, not just more accurate; see the GitHub README's Failure analysis for real examples of both where this helps and where it still hallucinates.

Known limitation: this adapter was saved after the last training epoch, not the epoch with the best validation loss (val loss actually bottomed out at epoch 1 and rose slightly by epoch 2 — the trainer computes a best checkpoint but a bug in the training script never promotes it). The numbers above are still real, just from a slightly overfit checkpoint. See the GitHub README's "Model quality" section for the fix.

Usage

This adapter needs Qwen2.5-Omni's custom audio+text pipeline (chat template, audio preprocessing) — the cleanest way to run it is the GitHub repo's inference script, which handles that for you:

git clone https://github.com/Mounish-Allam/AUDIO-TEMPORAL-REASONING-FINE-TUNNING
cd AUDIO-TEMPORAL-REASONING-FINE-TUNNING
pip install -r requirements.txt

python scripts/run_inference.py \
    --audio_path your_clip.wav \
    --prompt "Describe what happens first, then next, and finally in this audio." \
    --lora_path MounishAllam/qwen2.5-omni-audio-temporal-qlora \
    --use_4bit

If you want to load it directly with transformers + peft (e.g. to inspect the adapter or integrate it into your own pipeline):

from transformers import Qwen2_5OmniProcessor, Qwen2_5OmniThinkerForConditionalGeneration
from peft import PeftModel

base_id = "Qwen/Qwen2.5-Omni-7B"
processor = Qwen2_5OmniProcessor.from_pretrained(base_id)
base_model = Qwen2_5OmniThinkerForConditionalGeneration.from_pretrained(
    base_id, device_map="auto"
)
model = PeftModel.from_pretrained(base_model, "MounishAllam/qwen2.5-omni-audio-temporal-qlora")

You'll still need Qwen2.5-Omni's chat-template and audio-preprocessing helpers to build a real prompt with audio input — see src/conversation.py and src/inference.py in the GitHub repo for that.

Training details

• —Base model: Qwen/Qwen2.5-Omni-7B, loaded in 4-bit NF4 (QLoRA)
• —LoRA config: r=32, alpha=64, dropout=0.05, target modules: q/k/v/o/gate/up/down_proj
• —Data: 3,000 real AudioCaps clips, 2 epochs, lr=1e-4, batch size 2 (grad accum 8)
• —Hardware: trained on a local RTX 5080; the same recipe fits a free Kaggle T4 GPU at smaller scale — see KAGGLE_GUIDE.md

License

MIT, matching the base model's Apache 2.0 permissiveness for the adapter weights themselves (no base model weights are redistributed here — this repo contains only the LoRA adapter).