Elfsong/musicai-background-music-audio-llm-benchmark

Does Background Music Matter to Speech in Pre-trained Language Models

The completed September 2026 study covers 8 model families, 55 instrumental recordings, and 10 evaluation settings. It studies how adding background music to the same spoken question changes model responses.

• —Latest release and artifact guide
• —Technical report PDF
• —Complete LaTeX project
• —LaTeX GitHub repository
• —Matrices, figures, and supporting data
• —Regenerated speech and mixtures: 550 archives / 250,800 condition FLACs
• —Raw model outputs and scored conditions
• —Evaluation code, configurations, and analyses

The latest release preserves the original files below. It separately identifies regenerated audio, corrected expanded-cohort IFEval scores, backfilled historical models, and diagnostic runs. Refer to the release guide and report appendices for the final protocol and its limitations. The original speech waveforms from the first released cohort remain unavailable; the new archives contain the explicitly regenerated versions.

The following is the original repository documentation, retained as historical context. Its worker status and pilot scope predate the completed release above.

MusicAI ambient-music pilot

This repository tests whether a fixed spoken task changes when controlled synthetic background audio is mixed into the same waveform.

Fixed setup

• —TTS: Qwen3-TTS-12Hz-0.6B-CustomVoice, speaker Aiden
• —Task model: Qwen2.5-Omni-7B, text output only
• —16 exact-answer prompts across arithmetic, logic, science, and instruction following
• —6 conditions: clean, white noise at 10 dB SNR, calm/60 BPM and upbeat/150 BPM synthetic music at 10 dB and 0 dB SNR
• —deterministic task-model decoding
• —8 data-parallel inference shards, one 12-sample batch per GPU

A second sampling pilot repeats every audio input 8 times with temperature 0.7 and uses two 48-sample batches per GPU. It measures output variability and provides a more meaningful high-throughput GPU run than a synthetic stress test.

The original six-condition pilot uses locally generated oscillator music. A second real-music pilot adds 16 licensed excerpts spanning classical, pop, rap/hip-hop, rock, jazz, electronic, and folk/country, balanced into eight vocal and eight instrumental tracks. See GENRE_PILOT_REPORT.md for results and MUSIC_ATTRIBUTION.md before redistributing any audio.

Public benchmark suite

The expanded model-only benchmark uses a deterministic, speech-adapted 2,000-prompt suite:

• —500 GSM8K test questions
• —500 BBH test questions, balanced across 10 audio-suitable tasks
• —500 MMLU test questions, balanced across four subject families
• —500 IFEval prompts selected for spoken delivery

The final instrumental screen evaluates every fixed TTS waveform with 55 tracks across 11 genres at +10 dB speech-to-background SNR. Every genre-local group contains clean speech, white noise, and five music tracks in the same model batch. This paired batching is part of the evaluation protocol because greedy multimodal decoding can change under different padding and floating-point execution paths. Generated speech and condition audio are deleted after their corresponding inference stage; prompts, hashes, metadata, predictions, scores, and source music stimuli are retained.

This suite is an audio adaptation and stratified subset, not a claim of canonical text-leaderboard comparability. IFEval uses the official strict and loose instruction-following verifier; the other tasks use exact-answer scoring.

Run on Merlin worker 1091125

The current worker is already being prepared. To reproduce the environment on a fresh worker, run bash setup_worker.sh first. Model files and the virtual environment are stored under worker-local /tmp to keep the shared workspace small.

mlx worker login 1091125 -- bash -lc \
  'cd /mlx_devbox/users/mz.du/playground/MusicAI && bash run_pilot.sh'

mlx worker login 1091125 -- bash -lc \
  'cd /mlx_devbox/users/mz.du/playground/MusicAI && bash run_sampling_pilot.sh'

mlx worker login 1091125 -- bash -lc \
  'cd /mlx_devbox/users/mz.du/playground/MusicAI && bash run_genre_pilot.sh'

mlx worker login 1091125 -- bash -lc \
  'cd /mlx_devbox/users/mz.du/playground/MusicAI && bash run_genre_sampling_pilot.sh'

mlx worker login 1091125 -- bash -lc \
  'cd /mlx_devbox/users/mz.du/playground/MusicAI && bash run_benchmark_tts.sh'

mlx worker login 1091125 -- bash -lc \
  'cd /mlx_devbox/users/mz.du/playground/MusicAI && bash run_public_closed.sh'

mlx worker login 1091125 -- bash -lc \
  'cd /mlx_devbox/users/mz.du/playground/MusicAI && bash run_public_ifeval.sh'

mlx worker login 1091125 -- bash -lc \
  'cd /mlx_devbox/users/mz.du/playground/MusicAI && \
   /tmp/musicai-pilot-venv/bin/python scripts/score_public_benchmarks.py && \
   /tmp/musicai-pilot-venv/bin/python scripts/analyze_paired_effects.py'

Persistent outputs:

• —artifacts/tts/: one clean TTS waveform per prompt
• —artifacts/conditions/: condition-mixed waveforms
• —artifacts/eval_manifest.jsonl: audio metadata, hashes, and SNR
• —results/predictions.jsonl: sample-level model outputs and scores
• —results/summary.csv: condition-level accuracy and clean deltas
• —results/paired_flips.json: paired answer flips relative to clean
• —logs/gpu_samples.csv: one-second GPU telemetry during inference
• —results/gpu_summary.csv: measured mean, active mean, and peak utilization
• —data/benchmarks/benchmark_2000.jsonl: fixed expanded benchmark selection
• —data/benchmarks/benchmark_new_1600.jsonl: added prompts run in the expansion
• —results/public_full_500_benchmarks/: 500-prompt official scores and paired effects
• —EXPANDED_BENCHMARK_500_REPORT.md: expanded experiment report
• —results/exploratory_ungrouped/: archived pre-validation runs with mixed batches

Models and the isolated environment are intentionally stored on the worker's local /tmp volume, while all experiment code and outputs are stored here.