intrect/artifactbench

1---2license: cc-by-nc-4.03task_categories:4  - audio-classification5tags:6  - ai-music-detection7  - benchmark8  - forensic9  - audio10language:11  - en12size_categories:13  - 1K<n<10K14configs:15  - config_name: default16    data_files:17      - split: train18        path: ai_tracks/*19---20 21# ArtifactBench — AI-Generated Music Detection Benchmark22 23## ArtifactBench v2 — lineage-aware frozen protocol24 25ArtifactBench v2 adds a metadata-first, lineage-aware evaluation protocol while26preserving the v1 and v1.1 releases below. Its frozen primary cohort contains27828 entries (605 AI-generated and 223 real) across 15 source strata, split by28content lineage into calibration, validation, and sealed-test partitions before29final model comparison.30 31The v2 package is under [`v2/`](v2/README.md). It contains the path-free frozen32manifest, per-model probabilities, structured inference and chunk failures,33source-level and paired metrics, uncertainty estimates, provenance records, and34checksums. It does **not** add or replace audio. Existing v1 audio shards and35manifests remain unchanged.36 37On the 562-track common-success sealed-test intersection, ArtifactNet v9.438obtains AUROC 0.982 and balanced accuracy 0.918; the public Deezer detector39obtains 0.761 and 0.776. These values are tied to the v2 cohort, calibration-only40threshold policy, and declared coverage rules and should not be compared as if41they were measured on the v1 cohort below.42 43Paper (preprint): [ArtifactBench: Lineage-Aware Evaluation of AI-Generated Music Detectors under Distribution Shift](https://arxiv.org/abs/2609.23550).44 45### Try ArtifactNet on your own audio46 47[**Try the free live demo →**](https://demo.intrect.io/?utm_source=huggingface&utm_medium=referral&utm_campaign=artifactbench_arxiv_202609&utm_content=dataset_card_demo) · [Scope and limitations](https://intrect.io/artifactnet/?utm_source=huggingface&utm_medium=referral&utm_campaign=artifactbench_arxiv_202609&utm_content=dataset_card_product)48 49Upload an audio file or choose a built-in example. No account is required within the demo’s free limits. This is the hosted ArtifactNet detector, not a browser-based reproduction of the four-model ArtifactBench evaluation; the live service may differ from the version-pinned research model.50 51 52---53 54## ArtifactBench v1 — original public benchmark55 56A multi-generator evaluation benchmark for AI-generated music forensic detection, covering 22 AI generators and 6 real music sources.57 58## Dataset Description59 60- **Total tracks**: 8,766 (4,383 AI + 4,383 Real, 1:1 balanced)61- **AI generators**: 22 (MusicGen, Stable Audio, Suno v3/v3.5/v4, Udio, Riffusion, DiffRhythm, Yue, Chirp v2/v3/v3.5, etc.)62- **Real sources**: 6 (SONICS, MoM, FMA, YouTube)63- **Format**: AI tracks as Parquet (audio bytes embedded), Real tracks as CSV (YouTube IDs for user download)64 65## Motivation66 67Existing benchmarks (SONICS: 5 generators, MoM: 6 generators) only measure in-distribution performance. Models reporting high F1 on these benchmarks fail catastrophically on out-of-distribution generators:68 69- CLAM (194M params, F1=0.925 on MoM) → F1=0.824 on ArtifactBench70- SpecTTTra (19M params, F1=0.97 on SONICS) → F1=0.766 on ArtifactBench71 72ArtifactBench evaluates what matters for deployment: generalization across diverse generators.73 74## Sanity Check Protocol75 76Per-source pass/fail thresholds:77- Real source FPR ≤ 5%78- AI source TPR ≥ 90% (Stable Audio: ≥ 60%)79- Codec invariance: mean Δ ≤ 0.15, max Δ ≤ 0.3580 81## Baseline Results82 83| Model | Params | F1 | FAIL | Suno v4 TPR | Real FPR |84|---|---|---|---|---|---|85| **ArtifactNet v9.4** | **4.2M** | **0.983** | **4/28** | **98%** | **1.5%** |86| CLAM (MoM) | 194M | 0.824 | 16/28 | 78% | 70.5% |87| SpecTTTra | 19M | 0.766 | 23/28 | 55% | 21.4% |88 89## Public Model Comparison — 8-way (v1.1 test partition, 2026-07-04)90 91Eight publicly-available detectors scored on identical files with the same runner92(adapters in `artifactbench/models/`), τ = 0.5. n = 2,104 (1,388 AI / 716 real —93the locally-restored real set; see the provenance caveat in `v1.1/RESULTS_8WAY.md`).94 95| Rank | Model | Params | F1 | Precision | Recall (TPR) | FPR | Sanity FAIL |96|---|---|---:|:---:|:---:|:---:|:---:|:---:|97| 1 | **ArtifactNet v9.4 (public ONNX)** | **4.2M** | **0.952** | 0.932 | 97.3% | 13.8% | 8/28 |98| 2 | AI-Music-Detection AST-60s | 90.8M | 0.840 | 0.848 | 83.1% | 28.9% | 16/28 |99| 3 | CLAM (MoM) | 194.3M | 0.787 | 0.711 | 88.3% | 69.7% | 14/28 |100| 4 | SpecTTTra α-120s | 18.7M | 0.777 | 0.880 | 69.5% | 18.4% | 22/28 |101| 5 | Deezer ISMIR fakeprint LR | 3.6K | 0.754 | 0.906 | 64.6% | 13.0% | 18/28 |102| 6 | FST (Mippia, arXiv:2601.13647) | 174.4M | 0.735 | 0.984 | 58.7% | 1.8% | 17/28 |103| 7 | DeepFense EAT+Nes2Net | — | 0.650 | 0.589 | 72.4% | 97.8% | 11/28 |104| 8 | SpecTTTra β-5s | 18.7M | 0.563 | 0.884 | 41.3% | 10.5% | 24/28 |105 106The ArtifactNet production pipeline (v9.7 / cnn_v95, PyTorch — not the public ONNX107export above) measured on the same partition: F1 0.984 / TPR 98.9% / FPR 4.2%108(1/28 FAIL). Per-source TPR/FPR tables, ROC summaries, and notes: `v1.1/RESULTS_8WAY.md`.109 110## Usage111 112```python113from artifactbench.bench import main114# or115# python -m artifactbench.bench --model artifactnet --manifest artifactbench_v1_manifest.json116```117 118 119## Per-Source Breakdown (v1.0.1)120 121| Source | Class | Tracks | bench_origin: test | Generator |122|---|---|---|---|---|123| aime_musicgen_large | AI | 200 | 30 | MusicGen Large |124| aime_musicgen_medium | AI | 200 | 30 | MusicGen Medium |125| aime_musicgen_small | AI | 200 | 30 | MusicGen Small |126| aime_riffusion | AI | 200 | 30 | Riffusion |127| aime_stable_audio_v1 | AI | 200 | 50 | Stable Audio v1 |128| aime_stable_audio_v2 | AI | 200 | 50 | Stable Audio v2 |129| aime_suno_v3 | AI | 200 | 30 | Suno v3 |130| aime_suno_v35 | AI | 200 | 30 | Suno v3.5 |131| aime_udio | AI | 200 | 30 | Udio (AIME) |132| mom_diffrythm | AI | 200 | 100 | DiffRhythm |133| mom_riffusion | AI | 200 | 100 | Riffusion (MoM) |134| mom_udio | AI | 200 | 100 | Udio (MoM) |135| mom_yue | AI | 200 | 100 | Yue |136| sonics_chirp-v2-xxl-alpha | AI | 200 | 80 | Chirp v2 |137| sonics_chirp-v3 | AI | 200 | 80 | Chirp v3 |138| sonics_chirp-v3.5 | AI | 200 | 80 | Chirp v3.5 |139| sonics_udio-120s | AI | 200 | 80 | Udio 120s |140| sonics_udio-30s | AI | 200 | 80 | Udio 30s |141| suno_cdn_latest | AI | 200 | 100 | Suno CDN (post-freeze) |142| suno_extra | AI | 200 | 80 | Suno extras |143| **udio_cdn_latest** | AI | **200** | 35 | Udio CDN (post-freeze) — v1.0.1 balanced |144| udio_extra | AI | 200 | 80 | Udio extras |145| sonics_real | Real | 500 | 300 | SONICS real partition |146| mom_real | Real | 400 | 200 | MoM real (mp3 + wav) |147| fma_hardneg | Real | 300 | 150 | FMA mp3 hard-negatives |148| mom_extra_real | Real | 200 | 110 | MoM extra real |149| mom_real_wav | Real | 200 | 42 | MoM real WAV variants |150| youtube_hardneg | Real | 200 | 73 | YouTube curated hard-negatives |151| **TOTAL** | — | **6,200** | **2,280** | 28 sources, 22 AI generators |152 153Real sources are intentionally over-represented (1,800 total) to enable rigorous FPR estimation across diverse codec and production conditions.154 155 156## Files157 158- `artifactbench_v1_manifest.json` — Track manifest with bench_origin tags159- `metadata.json` — Dataset statistics and generator list160 161## Citation162 163```bibtex164@article{oh2026artifactnet,165  title        = {ArtifactNet: Detecting AI-Generated Music via Forensic Residual Physics},166  author       = {Oh, Heewon},167  journal      = {arXiv preprint arXiv:2604.16254},168  year         = {2026},169  eprint       = {2604.16254},170  archivePrefix= {arXiv},171  primaryClass = {cs.SD},172  doi          = {10.48550/arXiv.2604.16254},173  url          = {https://arxiv.org/abs/2604.16254}174}175```176 177**arXiv**: [2604.16254](https://arxiv.org/abs/2604.16254) · **DOI**: [10.48550/arXiv.2604.16254](https://doi.org/10.48550/arXiv.2604.16254)178 179## License180 181CC BY-NC 4.0182 183 184## v1.1 (2026-07-03) — integrity-audit purged test partition185 186An audit of the test partition against the ArtifactNet v9.4 training manifest found 34187overlapping real tracks (all YouTube-derived); 5 further real tracks became unrecoverable.188`v1.1/` ships the purged partition (n = 2,224), a per-track status CSV, and official v1.1189result bounds. **v1 files are unchanged** — results computed on v1 remain reproducible.190See `v1.1/RESULTS_v1.1.md`.191 192 193## 8-way public model comparison (2026-07-04)194 195Extends the v1.1 evaluation with five more publicly-available detectors. The summary196table is in **Public Model Comparison** above; per-source breakdowns and notes are in197`v1.1/RESULTS_8WAY.md`.198