Datasetpublic

henrywch2huggingface/Unsplash-Debiased_by_AI-18K

Unsplash-Debiased-by-AI-18K 17,812 high-quality image captions selected by a label-free, dual-modality, 6-judge debiasing pipeline from 32,135 Unsplash-40K captions. Each caption passes three gates: (a) high/mid tier in a debiased judge consensus (family-balanced × reliability-weighted, rank-calibrated, self-preference-dropped), (b) image quality (NIQE/MUSIQ/LIQE), and (c) image–caption alignment (CLIPScore/SigLIP). Images are NOT included. This dataset ships captions +… See the full description on the dataset page: https://huggingface.co/datasets/henrywch2huggingface/Unsplash-Debiased_by_AI-18K.

sourceHugging Facemitupdated 3mo agoView on Hugging Face

0likes21downloads

Dataset Card

Unsplash-Debiased-by-AI-18K

17,812 high-quality image captions selected by a label-free, dual-modality, 6-judge debiasing pipeline from 32,135 Unsplash-40K captions. Each caption passes three gates: (a) high/mid tier in a debiased judge consensus (family-balanced × reliability-weighted, rank-calibrated, self-preference-dropped), (b) image quality (NIQE/MUSIQ/LIQE), and (c) image–caption alignment (CLIPScore/SigLIP).

Images are NOT included. This dataset ships captions + Unsplash photo IDs/URLs + metadata only, to respect the Unsplash License. Reconstruct each image from unsplash_url (https://unsplash.com/photos/{unsplash_id}).

Produced by SC-LLMJudge — see the code + paper and the results report (doc/ANALYSIS.md).

How it was built

—Captioner: MiMo-V2.5 (one detailed paragraph per image, temp 0).
—6-judge panel: 3 text-LLM (DeepSeek-V4-Pro, Kimi-K2.6, MiMo-V2.5-Pro) score fluency/coherence/noobviouserrors; 3 VLM (GLM-4.6V, MiMo-V2.5-VL, Qwen3.5-397B) score relevance/conceptgrasp/detailfidelity. Listwise N=3, integer 1–5 + rationale.
—Debiasing: detect → mitigate → verify → aggregate loop that corrects score saturation, model-specific self-preference, verbosity, and judge redundancy, and verifies image-grounding with blinding/shuffle probes. Design: doc/DEBIAS_SYSTEM.md.

Schema

field	type	description
`unsplash_id`, `unsplash_url`	str	Unsplash photo id / page URL (fetch the image here)
`caption`	str	the MiMo-V2.5 caption being scored
`tier`	str	debiased consensus tier: `high` / `mid` (only these are released)
`overall`	float	overall debiased quality (rank-calibrated, 0–1)
`dispersion`	float	cross-judge disagreement (higher = less certain)
`n_judges`	int	judges that scored this item
`score_{relevance,concept_grasp,detail_fidelity,fluency,coherence,no_obvious_errors}`	float	per-dimension debiased consensus (0–1)
`iq_{niqe,musiq,maniqa,topiq_nr,clipiqa}`, `clipscore`, `liqe`, `siglip_sim`	float	image-quality + image-text-alignment covariates

Usage

python

from datasets import load_dataset
ds = load_dataset("<user>/Unsplash-Debiased-by-AI-18K", split="train")
print(ds[0]["caption"], ds[0]["unsplash_url"], ds[0]["overall"])
# to use images: download from unsplash_url per the Unsplash License

Intended use & limitations

—Use: a cleaner caption set for training/distillation/evaluation, or as a testbed for judge-bias research (per-caption debiased scores + covariates are all included).
—Label-free: selection reflects a debiased model consensus, not human gold labels; biases that co-vary with true quality (self-preference, calibration) are mitigated but not perfectly removed. A small human-anchor calibration is future work.
—Provenance: single captioner (MiMo-V2.5); English captions; Unsplash's curated, mostly high-quality photography (a ~9% low-quality tail was gated out).

Citation

bibtex

@misc{scllmjudge2026,
  title  = {Judging the Judges at Scale: A Label-Free, Dual-Modality System for
            Detecting and Eliminating Bias in LLM/VLM-as-a-Judge},
  author = {Anonymous}, year = {2026}
}