kulsoom-abdullah/surgvivqa-qwen7b-audio

SurgViVQA-Audio — spoken-question video QA (LoRA adapter)

A QLoRA adapter that answers questions about colonoscopy video from spoken audio alone. No question text is given to the model at any point — the text prompt is a single fixed instruction with no question content.

• —Base: `kulsoom-abdullah/Qwen2-Audio-7B-Transcription` — Qwen2-VL-7B-Instruct with a Whisper-large-v3-turbo encoder projected into the embedding space
• —Code, full results, and prediction files: github.com/kulsoom-abdullah/SurgViVQA-Audio
• —Detailed results: `docs/RESULTS.md`

Results

Evaluated on 1,000 samples from a held-out patient (REAL-Colon video 002-004), frame-disjoint from training, stratified 50 rows across each of 20 question types.

Read the floor before the accuracy. Seven of the 20 question types have a single gold answer across all 50 of their test rows, so 35% of the test set cannot distinguish a model from a lookup table.

A control with no perception at all. Something that is not a model — it hears nothing, sees nothing, and knows only which of the 20 question types was asked, emitting one canned answer per type — scores:

This model scores 57.1% on the full test set and 49.5% on the 650 rows from the 13 question types whose answers vary. So on aggregate it does not beat a lookup table built from training answers; on the discriminative subset it clears that floor by 7.5 points. Aggregate accuracy on this benchmark measures answer priors more than it measures vision — prefer the 650-row figure when comparing systems.

Full floor analysis: https://github.com/kulsoom-abdullah/SurgViVQA-Audio/blob/main/docs/RESULTS.md

What this model does well

Both are balanced 25/25 splits, so these are genuine visual discrimination.

What it does not do

• —Temporal questions are not answered by any model. Across four stock VLMs spanning 7B–33B and two generations, every model lands within 5 points of chance on motion questions. The dataset samples 8 frames at a fixed offset regardless of where the labeled motion occurs, so the signal is frequently absent from the input.
• —Screen-space localization fails here but is solvable. Qwen3-VL-32B reaches +22 over floor on lesion_screen_position; this model sits at the floor, emitting one constant string. Same resolution, same frames — a capability gap, not a data limit.
• —Two losses to stock models: mucosa_visibility (48% vs 64–80%) and occlusion_check (50% vs 86–88%).
• —Output diversity is low — roughly one distinct answer string per question type, against 33 for Qwen3-VL-32B. Much of the aggregate score comes from having learned the correct constant for single-class question types.

Audio comprehension is measured separately from accuracy

Given audio and no question text, the un-fine-tuned base reproduced the question it had heard on 488 of 1,000 rows, and 472 of those (96.7%) matched the question actually spoken, out of 20 candidates — against a 5% chance rate and a 4.9% permutation null.

Audio comprehension is therefore present before task fine-tuning. What this adapter adds is answer format and partial visual grounding.

Usage

The question must be supplied as audio. The text prompt carries no question content.

import torch, librosa
from PIL import Image
from transformers import (
    Qwen2VLForConditionalGeneration, AutoProcessor, AutoTokenizer,
    WhisperFeatureExtractor, BitsAndBytesConfig,
)
from peft import PeftModel

BASE = "kulsoom-abdullah/Qwen2-Audio-7B-Transcription"
ADAPTER = "kulsoom-abdullah/surgvivqa-qwen7b-audio"
PROMPT = "Answer the question concisely based on the visual and audio evidence."

bnb = BitsAndBytesConfig(
    load_in_4bit=True, bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16, bnb_4bit_use_double_quant=True,
)
base = Qwen2VLForConditionalGeneration.from_pretrained(
    BASE, quantization_config=bnb, attn_implementation="sdpa", trust_remote_code=True,
)
model = PeftModel.from_pretrained(base, ADAPTER).eval()

tokenizer = AutoTokenizer.from_pretrained(BASE, trust_remote_code=True, use_fast=False)
processor = AutoProcessor.from_pretrained("Qwen/Qwen2-VL-7B-Instruct", use_fast=False)
processor.tokenizer = tokenizer
fe = WhisperFeatureExtractor.from_pretrained("openai/whisper-large-v3-turbo")

frames = [Image.open(p).convert("RGB") for p in frame_paths]   # 8 frames, 384px
audio, _ = librosa.load("spoken_question.mp3", sr=16000, mono=True)
input_features = fe(audio, sampling_rate=16000, return_tensors="pt").input_features
# 1500 audio tokens are injected ahead of the visual content; see
# src/evaluate_checkpoint.py in the GitHub repo for the full token layout.

This requires the forked transformers in the GitHub repo. Stock transformers silently drops the audio pathway — Qwen2VLForConditionalGeneration.forward has no input_features parameter, so the argument is discarded without error and the model answers from the frames alone. Verify before trusting any output:

import inspect
from transformers import Qwen2VLForConditionalGeneration as M
assert "input_features" in inspect.signature(M.forward).parameters

Training

Data. 2,302 training / 398 eval rows from REAL-Colon videos 002-001/002/003; 1,000 test rows from 002-004, frame-disjoint. Questions rendered to speech with edge-tts across 41 English voices split 23/9/9 with no voice appearing in two splits, stratified by accent region and gender, seeded.

Why audio directly instead of transcribing first

Not latency. Measured on matched hardware, weights, frames, and decoding budget, direct audio input is 0.88× — 13% slower than an ASR-then-text pipeline (1,896 ms vs 1,677 ms, n=100). The entire difference is prefill: the speech encoder emits a fixed 1,500 tokens for a 30-second window while these questions run under 4 seconds, so most of that context is encoded silence. Transcription costs only 73 ms.

The case for direct audio is that it removes transcription errors from the path to the answer, and keeps prosody and background sound available to the model rather than discarding them at a text bottleneck. Neither is exercised by this task, which uses clean synthetic speech. Longer utterances would shift the latency balance, since transcription cost scales with speech length and the audio prefix does not.

Limitations

• —Synthetic audio. Clean TTS, no noise or reverberation. Real operating-room acoustics are untested, and are the condition under which direct audio would be expected to hold up better than an ASR pipeline.
• —20 distinct questions total, one per question type. Question recognition from audio is a 20-way classification, not open-vocabulary speech understanding.
• —Single test patient. All 1,000 test rows come from one video.
• —Accent coverage does not match the intended deployment population. en-IN voices are included as the nearest available South Asian English and are explicitly not labeled Pakistani-accented.
• —Research artifact only. Not validated for clinical use in any form.

Credits

Built on Qwen2-VL and Whisper large-v3-turbo. Data from the REAL-Colon dataset (CC BY). Question templates follow the SurgViVQA benchmark; I am not affiliated with its authors.