dieKarotte/SO-Bench

SO-Bench: Spatial-Omni Benchmark for FOA Spatial Audio Question Answering

SO-Bench is an audio-only spatial question answering benchmark built from FOA spatial audio. Each example contains a first-order ambisonics (FOA) waveform and one natural-language question-answer pair. The questions cover sound event detection, localization, spatial relations, motion, and multi-step spatial reasoning.

SO-Bench is part of the Spatial-Omni: Spatial Audio Understanding Integration in Multimodal LLMs via FOA Encoding.

Dataset Contents

• —Audio format: FOA waveform files (.wav)
• —Split: test
• —QA metadata: one JSON object per question-answer pair

File Structure

SO-Bench/
  README.md
  score_predictions_llm_judge.py    # LLM-judge scorer for predictions

  audio/
    test/
      foa_413d6010b124f7adcc0b.wav
      foa_be5d2301ea1330fce366.wav
      ...

  qa/
    test.jsonl

Dataset Statistics

Bench sub-tasks is shown in figures below. <!-- figure --> <img src="figures/bench_tasks.jpeg" alt="SO-Bench task distribution" width="400"/>

Task distribution:

QA Metadata Format

Each line in qa/test.jsonl is one QA example.

Example:

{
  "qa_id": "qa_4e31e78e5a6f7579fe52",
  "split": "test",
  "audio_id": "foa_413d6010b124f7adcc0b",
  "audio_path": "audio/test/foa_413d6010b124f7adcc0b.wav",
  "task_name": "detect_time",
  "question_class": "detect",
  "answer_format": "structured_text",
  "question": "Listen to the audio clip and answer based only on what you hear. During which time interval can the sound of the gong be heard? Based on the audio clip, state which sound sources are audible and when each one occurs. Answer in one natural sentence and report times in seconds.",
  "answer": "The gong is audible from 18.5s to 20.0s.",
  "canonical_answer": "18.5s to 20.0s",
  "answer_meta": {"time_span": [18.5, 20.0]}
}

Important fields:

• —qa_id: anonymized QA identifier. Use this as the stable join key with your `predictions.jsonl` (see "Benchmark Methods" below).
• —split: dataset split. This release contains only test.
• —audio_id: anonymized FOA audio identifier.
• —audio_path: relative path to the FOA waveform.
• —task_name: task type for the question.
• —question_class: broad category such as detect, azimuth, elevation, distance, motion, or reasoning.
• —answer_format: expected answer style.
• —question: natural-language question.
• —answer: full natural-language answer.
• —canonical_answer: normalized answer target for evaluation or parsing.
• —answer_meta: structured ground-truth used by the LLM-judge scorer. Populated for the numeric and time-grounded tasks; empty {} otherwise. Possible keys:
• —azimuth_deg (float, for estimate_azimuth)
• —elevation_deg (float, for estimate_elevation)
• —distance_m (float, for estimate_distance)
• —time_span ([start_s, end_s], for detect_time, spatial_temporal, and single-event detect_source)
• —onset_time (float, for onset_from_location)
• —active_count (int, for count_sources)

Download

Install the Hugging Face CLI:

pip install -U "huggingface_hub[cli]"

Download the full benchmark:

hf download dieKarotte/SO-Bench \
  --repo-type dataset \
  --local-dir SO-Bench

Download only the QA metadata:

hf download dieKarotte/SO-Bench \
  --repo-type dataset \
  --local-dir SO-Bench \
  --include "qa/test.jsonl" \
  --include "README.md"

Benchmark Methods

After running your model on every entry in qa/test.jsonl and writing the outputs to a predictions.jsonl file, score them with the bundled score_predictions_llm_judge.py. The scorer normalizes each model output via an OpenAI-compatible chat-completions API, then applies task-specific local metrics (numeric thresholds, IoU, exact-match, WER) to the normalized fields.

predictions.jsonl schema

One JSON object per line, in the same order as qa/test.jsonl (or include qa_id so the scorer can match by id, which is the recommended path):

{
  "qa_id": "qa_4e31e78e5a6f7579fe52",
  "audio_path": "audio/test/foa_413d6010b124f7adcc0b.wav",
  "task_name": "detect_time",
  "prediction": "The gong is heard from 18.4s to 20.0s."
}

The scorer reads the model output from prediction (preferred), prediction_cleaned, or prediction_raw, in that order.

Running the scorer

export OPENAI_API_KEY=sk-...                          # or pass --api-key
# export OPENAI_BASE_URL=https://api.openai.com/v1    # optional, override base URL

python3 score_predictions_llm_judge.py \
  --predictions-jsonl your_predictions_file.jsonl \
  --qa-root /path/to/SO-Bench/qa \
  --split test \
  --model gpt-4o \
  --concurrency 256 \
  --force-refresh \
  --require-api \
  --output-json /tmp/llm_result.json \
  --judged-jsonl /tmp/llm_judged_records.jsonl \
  --cache-jsonl /tmp/llm_judge_cache.jsonl

Scoring rules per task

• —Numeric thresholds (binary correctness):
• —estimate_azimuth: wrap-around angular error ≤ --angle-threshold-deg (default 20°).
• —estimate_elevation: absolute error ≤ --elevation-threshold-deg (default 10°).
• —estimate_distance: absolute error ≤ --distance-threshold-m (default 1.0 m). Note: the prompt's near <1.5 m / mid-range 1.5–4.0 m / far >4.0 m buckets are textual hints to the model; the scorer judges by absolute MAE, not bucket membership.
• —onset_from_location: absolute error ≤ --time-threshold-s (default 0.2 s).
• —count_sources: exact integer match.
• —IoU: detect_time returns time-span IoU as the score directly.
• —Semantic source match (LLM-judged): detect_source, identify_source_by_doa, identify_source_by_location, multi_hop. Correct iff the LLM judge returns exact_or_synonym (or compatible_but_broader when --accept-broader-source is set).
• —detect_source defaults to --detect-source-time-policy event_only (label only). Pass event_times_iou to multiply the score by the time-span IoU when both spans exist.
• —Relation match (LLM-judged): compare_distance, compare_elevation, relative_left_right. Correct iff the LLM judge returns relation_match=true.
• —Spatial-temporal: spatial_temporal defaults to --spatial-temporal-time-policy semantic_times_iou: (source_match ∧ direction_match) × IoU(time_span). Pass semantic_only to drop the IoU factor.
• —Motion: classify_motion correct iff motion_match=true.
• —Speech transcript (local, no LLM call): speech_content is scored locally via word error rate against canonical_answer, with binary correctness threshold WER ≤ 0.5. The scorer also reports mean / median WER and the fraction at WER ≤ 0.3 / 0.5 / 1.0.

Citation and License

Please cite this dataset as appropriate for your use. If you redistribute or use the dataset in downstream work, make sure your usage is compatible with the licenses of the underlying audio and spatial data sources.

@misc{zhu2026spatialomnispatialaudiounderstanding,
      title={Spatial-Omni: Spatial Audio Understanding Integration in Multimodal LLMs via FOA Encoding},
      author={Zhiyuan Zhu and Yixuan Chen and Yiwen Shao and Wenxiang Guo and Changhao Pan and Yu Zhang and Yuxiang Wang and Wei Liu and Houhua Zhang and Chengkuan Zeng and Wenbo Cheng and Yunxi Liu and Rui Yang and Steve Yves and Liefeng Bo and Zhou Zhao},
      year={2026},
      eprint={2606.10738},
      archivePrefix={arXiv},
      primaryClass={eess.AS},
      url={https://arxiv.org/abs/2606.10738},
}