intrect/artifactbench

ArtifactBench — AI-Generated Music Detection Benchmark

ArtifactBench v2 — lineage-aware frozen protocol

ArtifactBench v2 adds a metadata-first, lineage-aware evaluation protocol while preserving the v1 and v1.1 releases below. Its frozen primary cohort contains 828 entries (605 AI-generated and 223 real) across 15 source strata, split by content lineage into calibration, validation, and sealed-test partitions before final model comparison.

The v2 package is under `v2/`. It contains the path-free frozen manifest, per-model probabilities, structured inference and chunk failures, source-level and paired metrics, uncertainty estimates, provenance records, and checksums. It does not add or replace audio. Existing v1 audio shards and manifests remain unchanged.

On the 562-track common-success sealed-test intersection, ArtifactNet v9.4 obtains AUROC 0.982 and balanced accuracy 0.918; the public Deezer detector obtains 0.761 and 0.776. These values are tied to the v2 cohort, calibration-only threshold policy, and declared coverage rules and should not be compared as if they were measured on the v1 cohort below.

Paper (preprint): ArtifactBench: Lineage-Aware Evaluation of AI-Generated Music Detectors under Distribution Shift.

Try ArtifactNet on your own audio

**Try the free live demo →** · Scope and limitations

Upload an audio file or choose a built-in example. No account is required within the demo’s free limits. This is the hosted ArtifactNet detector, not a browser-based reproduction of the four-model ArtifactBench evaluation; the live service may differ from the version-pinned research model.

ArtifactBench v1 — original public benchmark

A multi-generator evaluation benchmark for AI-generated music forensic detection, covering 22 AI generators and 6 real music sources.

Dataset Description

• —Total tracks: 8,766 (4,383 AI + 4,383 Real, 1:1 balanced)
• —AI generators: 22 (MusicGen, Stable Audio, Suno v3/v3.5/v4, Udio, Riffusion, DiffRhythm, Yue, Chirp v2/v3/v3.5, etc.)
• —Real sources: 6 (SONICS, MoM, FMA, YouTube)
• —Format: AI tracks as Parquet (audio bytes embedded), Real tracks as CSV (YouTube IDs for user download)

Motivation

Existing benchmarks (SONICS: 5 generators, MoM: 6 generators) only measure in-distribution performance. Models reporting high F1 on these benchmarks fail catastrophically on out-of-distribution generators:

• —CLAM (194M params, F1=0.925 on MoM) → F1=0.824 on ArtifactBench
• —SpecTTTra (19M params, F1=0.97 on SONICS) → F1=0.766 on ArtifactBench

ArtifactBench evaluates what matters for deployment: generalization across diverse generators.

Sanity Check Protocol

Per-source pass/fail thresholds:

• —Real source FPR ≤ 5%
• —AI source TPR ≥ 90% (Stable Audio: ≥ 60%)
• —Codec invariance: mean Δ ≤ 0.15, max Δ ≤ 0.35

Baseline Results

Public Model Comparison — 8-way (v1.1 test partition, 2026-07-04)

Eight publicly-available detectors scored on identical files with the same runner (adapters in artifactbench/models/), τ = 0.5. n = 2,104 (1,388 AI / 716 real — the locally-restored real set; see the provenance caveat in v1.1/RESULTS_8WAY.md).

The ArtifactNet production pipeline (v9.7 / cnnv95, PyTorch — not the public ONNX export above) measured on the same partition: F1 0.984 / TPR 98.9% / FPR 4.2% (1/28 FAIL). Per-source TPR/FPR tables, ROC summaries, and notes: `v1.1/RESULTS8WAY.md`.

Usage

from artifactbench.bench import main
# or
# python -m artifactbench.bench --model artifactnet --manifest artifactbench_v1_manifest.json

Per-Source Breakdown (v1.0.1)

Real sources are intentionally over-represented (1,800 total) to enable rigorous FPR estimation across diverse codec and production conditions.

Files

• —artifactbench_v1_manifest.json — Track manifest with bench_origin tags
• —metadata.json — Dataset statistics and generator list

Citation

@article{oh2026artifactnet,
  title        = {ArtifactNet: Detecting AI-Generated Music via Forensic Residual Physics},
  author       = {Oh, Heewon},
  journal      = {arXiv preprint arXiv:2604.16254},
  year         = {2026},
  eprint       = {2604.16254},
  archivePrefix= {arXiv},
  primaryClass = {cs.SD},
  doi          = {10.48550/arXiv.2604.16254},
  url          = {https://arxiv.org/abs/2604.16254}
}

arXiv: 2604.16254 · DOI: 10.48550/arXiv.2604.16254

License

CC BY-NC 4.0

v1.1 (2026-07-03) — integrity-audit purged test partition

An audit of the test partition against the ArtifactNet v9.4 training manifest found 34 overlapping real tracks (all YouTube-derived); 5 further real tracks became unrecoverable. v1.1/ ships the purged partition (n = 2,224), a per-track status CSV, and official v1.1 result bounds. v1 files are unchanged — results computed on v1 remain reproducible. See v1.1/RESULTS_v1.1.md.

8-way public model comparison (2026-07-04)

Extends the v1.1 evaluation with five more publicly-available detectors. The summary table is in Public Model Comparison above; per-source breakdowns and notes are in v1.1/RESULTS_8WAY.md.