poonehmousavi/listen-to-reason-checkpoints

LISTEN-to-Reason — checkpoints

Graph + retrieval index + prototypes for LISTEN-to-Reason: a frozen text LLM answers audio questions from a serialized multimodal knowledge graph, never hearing the clip and never being fine-tuned.

No audio is redistributed here — only CLAP embeddings, graph structure, and the reference captions. See the attribution table for the licence that follows those captions.

git clone https://github.com/poonehmousavi/listen-to-reason && cd listen-to-reason
git checkout clean-pipeline
hf download poonehmousavi/listen-to-reason-checkpoints --repo-type dataset --local-dir checkpoint/
python -m src.release --verify checkpoint/     # sha256 + h5 row counts, run this first
python -m src.evaluate --dataset mmar          # expect: ours 0.412, blind 0.348

You additionally need the benchmark audio (MMAR / MMAU / SAKURA) — see the repo README.

LISTEN-to-Reason — checkpoint release (0102eb4)

13 files, 545 MB. Place them in checkpoint/ at the repo root; every path is a key in configs/config.yaml, so nothing here is hardcoded.

$ENV/bin/hf download <this repo> --repo-type dataset --local-dir checkpoint/
$ENV/bin/python -m src.release --verify checkpoint/   # checksums + h5 row counts

Verify before trusting a number. emb is a 164,003 x 512 float32 block, so a truncated or half-synced copy surfaces as a shape error half an hour into an eval rather than at transfer.

Runtime — needed to reproduce any number

Caches — optional, but they save hours of third-party fetching

Attribution — the corpus carries third-party caption text

audio_rag_corpus_clean.h5 stores a CLAP embedding and the caption for each reference clip. No audio is redistributed, but the captions are other people's work, and the licence follows them. Check this table before making anything here public or commercial.

Three things to know before you trust a number

1. 1.Corpus paths are portable placeholders, {data_root}/<source>/<tail> — the build machine's absolute paths were stripped before publishing. Nothing at inference opens them; only text and emb are read, with path used for provenance display and a basename-matched self-exclusion. That self-exclusion is a weak guard either way — what actually keeps benchmark clips out is the tau=0.95 overlap removal already applied here; see README section 5.1 for why clip-level disjointness was not enough.
2. 2.`audio_rag_corpus.h5` is deliberately NOT in this release. It is the pre-exclusion corpus and reproduces the contaminated MMAU 0.626 rather than the corrected 0.615. Neither are the documented negatives (universal_*.pt, route_learned.pt, router_feats_*.h5) — their own commits record them as not working.
3. 3.Every CLAP vector in here was embedded from one arbitrary 10 s window. laionclap picked it at random off the global numpy RNG; queries are deterministic as of the `clapcheck commit but these reference vectors predate it. They are stable references, so results reproduce — just do not mix them with chunk_mean` query embeddings without re-embedding the corpus. README section 5.7.

Quick check that it works

python -m src.clap_check                                   # encoder is deterministic
python -m src.evaluate --dataset mmar --limit 100 --tag smoke
python -m src.evaluate --dataset mmar

Compare against output/eval_mmar.md, which is tracked in the repo at the commit above — not against a number pasted here, which would go stale the first time the pipeline moved. MMAR is the right smoke target: it is the only benchmark with no containment in the corpus (0.1% of clips above the tau=0.99 duplicate threshold) and the only one on which the AF3 baseline has full caption coverage, so both comparisons in that report are honest ones.