WitneyWW/samaudio-quality-textprompt-vs-new

SAM-Audio quality: 8_sec_samaudio_textprompt vs 8_sec_samaudio_new_*

Comparison of two SAM-Audio-based sound-extraction pipelines used to build the JavisDiT train/val/test splits (/group2/ct/weihanx/JavisDiT/javisdit/script/{train,test,val}.jsonl, 32,102 / 1,784 / 1,783 clips):

• —old — 8_sec_samaudio_textprompt: single-pass text-prompted SAM-Audio separation.
• —new — 8_sec_samaudio_new_{train,test(unsuffixed dir "8_sec_samaudio_new"),val}: best-of-seed SAM-Audio (3 masks x 3 seeds) with an ImageBind text-audio alignment score picking the winning candidate per clip (selection.json).

Every id in all three splits has a target/residual WAV pair in both pipelines (0 missing either side, verified by directory listing against each split's jsonl).

Method

Signal-level metrics (full corpus, all 35,669 ids, both pipelines): loudness (RMS/peak dBFS), clipping ratio, silence ratio, and a target-vs-residual power ratio in dB (proxy for separation strength — how much energy landed in the extracted target vs what was left in the residual). These are computed identically for both pipelines directly from the WAV files, no model inference required.

ImageBind text-audio alignment (30 hand-picked samples, 10/split): the new pipeline already stores an ImageBind ib_ta score per clip from its own best-of-seed selection step (selection.json). The old pipeline never computed this, so for a same-protocol comparison we re-ran the identical scoring function (eval.javisbench.src.metrics.calc_imagebind_score, ImageBind-huge, javisdit conda env) on both pipelines' target.wav for a curated subset — picked to span the new pipeline's score distribution (high / median / low) and the biggest separation-ratio swings between pipelines — rather than the full corpus, since ImageBind inference at 35k x 2 pipelines wasn't warranted for this comparison.

Results

Full corpus, signal-level (mean per split x pipeline): see the metrics table and audio browser in the Space UI (index.html, data in data.json). Headline: the new pipeline's target audio is consistently louder (~9 dB higher RMS), has ~13-16 percentage points less silence, and has a 5-6 dB better target-vs-residual power ratio than the old pipeline, in all three splits. Zero clipping in either pipeline.

30-sample subset, recomputed ImageBind text-audio alignment (same protocol both pipelines): new pipeline scores higher on average in all three splits (train 0.083 -> 0.151, test 0.105 -> 0.120, val 0.120 -> 0.158) and wins on 20/30 individual samples overall (7/10 train, 7/10 test, 6/10 val) — a real but not universal improvement; roughly a third of curated samples still score better under the old pipeline.

Files

• —index.html — static browser UI (dropdowns + audio players + tables).
• —data.json — the same data as metrics_summary.csv / samples_manifest.csv, embedded for the page to fetch.
• —metrics_summary.csv — full-corpus aggregate signal metrics per split x pipeline.
• —samples_manifest.csv — the 30 curated samples: split, video_id, text prompt, selection reason, signal SRR both pipelines, and the recomputed ImageBind text-audio score both pipelines.
• —audio/<split>/<video_id>/{old,new}_{target,residual}.wav — the actual clips for the 30 curated samples.