Datasetpublic

joelleoqiyi/trace-rx-eval-predictions

TRACE-RX Evaluation Predictions Per-image detector scores from an independent evaluation of the two TechJam 2026 TRACE-RX detectors, run 30 Aug – 1 Sep 2026. No images here. Every file contains scores, labels, asset ids and transform names only — this is derived evaluation metadata, not a redistribution of any source imagery. The underlying corpora (Joshyxwa/data_draft, Joshyxwa/techjam2026, techjam-aigc/wildfake-eval-subset) keep their own terms, and data_draft's WildFake rows… See the full description on the dataset page: https://huggingface.co/datasets/joelleoqiyi/trace-rx-eval-predictions.

sourceHugging Faceotherupdated 23d agoView on Hugging Face

0likes66downloads

Dataset Card

TRACE-RX Evaluation Predictions

Per-image detector scores from an independent evaluation of the two TechJam 2026 TRACE-RX detectors, run 30 Aug – 1 Sep 2026.

No images here. Every file contains scores, labels, asset ids and transform names only — this is derived evaluation metadata, not a redistribution of any source imagery. The underlying corpora (Joshyxwa/data_draft, Joshyxwa/techjam2026, techjam-aigc/wildfake-eval-subset) keep their own terms, and data_draft's WildFake rows remain licence_audit_required.

Code and full write-up: https://github.com/BenyAlbatross/techjam-aigc/tree/joel-evals

Models evaluated

model	architecture	checkpoint
`techjam-aigc/trace-rx-m-v2`	DINOv2-base + authentic-prototype memory + residual head	`s4_detector.pt`, epoch 5, frozen encoder
`albagon/trace-rx-parallel-techjam2026`	two-branch (global + memory) with a learned fusion gate	`best_detector.pt`, epoch 8, frozen encoder

Inference only. Nothing was retrained.

Headline results

what	cost
six stacked random transforms	≤ 0.005 AUROC
one unseen generator	0.66 AUROC
trace-rx-parallel's fusion gate, off-distribution	0.12 AUROC

Transform robustness is solved on this data; generator generalisation is not, by roughly 130x. The parallel model's memory branch alone (0.596) beats its own shipped fused output (0.473) and beats trace-rx-m (0.572) on out-of-distribution data — the best detector measured here is a component the shipped model discards.

Full numbers in FINDINGS.md. Start with DEFINITIONS.md; several dataset names are near-homonyms.

Files

file	rows	what
`predictions/calib_trace_rx_m.parquet`	83,775	trace-rx-m, techjam2026 calibration x 15 official conditions
`predictions/calib_trace_parallel.parquet`	83,775	same, parallel — with `fused` / `glob` / `mem` branch logits
`predictions/chains_{m,p}.parquet`	33,510	calibration x chains of 1–6 transforms
`predictions/chains_wf_{m,p}.parquet`	30,000	data_draft WildFake x chains of 1–6
`predictions/trace_evalsubset_*.parquet`	varies	trace-rx-m on each of the 10 eval-subset configs
`predictions/parallel_*.parquet`	varies	parallel on the same 10, all three branch outputs
`predictions/trace_rx_m_scores.parquet`	10,000	trace-rx-m on all of data_draft
`*.json`	—	computed metric summaries

Scores are raw logits (higher = more likely AI), not calibrated probabilities.

Reproducing any table

python

import pandas as pd
from sklearn.metrics import roc_auc_score

d = pd.read_parquet("predictions/chains_wf_m.parquet")

# degradation curve: AUROC by number of stacked transforms
for k in range(1, 7):
    s = d[d.k == k]
    print(k, round(roc_auc_score(s.label, s.score), 4))

# the finding: adm and gan_based get BETTER with more transforms
for g in ["adm", "gan_based", "imagen"]:
    sub = d[(d.generator == g) | (d.label == 0)]
    a1 = roc_auc_score(sub[sub.k == 1].label, sub[sub.k == 1].score)
    a6 = roc_auc_score(sub[sub.k == 6].label, sub[sub.k == 6].score)
    print(f"{g:10s} k=1 {a1:.4f} -> k=6 {a6:.4f}  ({a6 - a1:+.4f})")

Caveats

—Prevalence varies by dataset — 0.268 on calibration, 0.500 on data_draft WildFake, 0.639 on several eval-subset configs. AUPRC is not comparable across them; always read it against its base rate.
—The calibration split is 88% trained-generator AI images, so its 0.874 headline mostly measures recognition of three memorised programs. Report the per-generator split beside it.
—One unanalysed thread: the chain column records the full ordered transform chain, and all 120 orderings of 3-family chains occur, but nobody has tested whether order matters (e.g. jpeg -> blur vs blur -> jpeg). The data is here for it.