madesai/what-ai-benchmarks-actually-measure

What AI Benchmarks Actually Measure: Item-Level Model Outputs and Scores for 53 Models

Item-level model responses and scores for 53 language models across the 56 benchmarks analyzed in What AI Benchmarks Actually Measure: Adapting Convergent and Discriminant Validity to Interrogate Fifty-Six AI Benchmarks (Desai et al., 2026, arxiv.org/abs/2609.08812).

We do not release the prompts from the benchmark datasets, but instead refer to them by item ids. To regenerate the prompts from benchmark datasets with the same ids used here, run the code at github.com/madesai22/what-ai-benchmarks-actually-measure (data_acquisition/get_data.py). The code to score model outputs (score.py) is also available in that GitHub repo. Consult each source benchmark's own license before redistributing recovered prompts.

Contents

Models

The 53 evaluated models (the set behind every analysis in the paper, including the benchmark correlation matrix in Figure B.1), by creator:

• —Alibaba: Qwen 1.5 110B Chat, Qwen 2 7B Instruct, Qwen 2 72B Instruct, Qwen 2.5 0.5B Instruct, Qwen 2.5 14B Instruct, Qwen 2.5 32B Instruct, Qwen 3 4B Instruct, Qwen3-30B-A3B-FP8
• —AllenAI: OLMo 2 7B Instruct, OLMo 2 13B Instruct, OLMo 2 32B Instruct, OLMoE 1B-7B Instruct
• —Anthropic: claude-3-5-haiku-20241022, claude-sonnet-4-5-20250929
• —AI21: Jamba Mini
• —Databricks: DBRX Instruct
• —DeepSeek: DeepSeek V3, DeepSeek V4-flash
• —Google: Gemma 2 9B IT, Gemma 2 27B IT, Gemma 3 4B IT, Gemma 3 12B IT, Gemma 3 27B IT
• —Meta: Llama 2 7B Chat, Llama 2 70B Chat, Llama 3.2 1B Instruct, Llama 3.2 3B Instruct, Llama 3.3 70B Instruct
• —Microsoft: Phi-3.5 Mini Instruct, Phi-3.5 MoE Instruct, Phi-4 Mini Instruct
• —Mistral: Mistral 7B Instruct v0.2, Mistral Nemo Instruct, Mistral Small 3.1 24B, Mistral Large 2411, Mixtral 8x7B Instruct
• —Moonshot AI: Moonlight 16B-A3B Instruct
• —01.AI: Yi 34B Chat, Yi 1.5 6B Chat, Yi 1.5 9B Chat, Yi 1.5 34B Chat
• —OpenAI: gpt-35-turbo-instruct0914, gpt-4turbo-2024-04-09, gpt-4o-mini2024-07-18, gpt-5-nano-2025-08-07, o12024-12-17, o1-mini2024-09-12, o3-mini2025-01-31
• —TII: Falcon3 1B Instruct, Falcon3 3B Instruct, Falcon3 7B Instruct, Falcon3 10B Instruct
• —xAI: grok-4-1-fast-reasoning

Model rows in data/, scores/, and aggregate/ share the same model_id values.

Row schema

Every row in data uses the following columns:

model_id  benchmark  paper_status  item_id  subset
response  cleaned_response  label  score

response is the raw model output; cleaned_response is the parsed answer; score is the per-item score. Benchmark-specific columns (category, context_condition, pair_id, demographic, rouge_2, …) follow.

subset distinguishes rows that share a benchmark config. For or_bench, xstest, and sgxs_test it takes the values refusal and overrefusal — we report these as two separate scores (refusal and over-refusal), computed from the unsafe and safe subsets of the same benchmark.

paper_status

We include data that was excluded from many analyses due to oversaturation or format noncompliance. We mark data excluded for these reasons with paper_status = excluded.

LLM judges

Judged safety benchmarks carry every judge's output side by side rather than only the one the paper scored against:

judge_<name>_raw_output       the judge's unparsed output
judge_<name>_verdict          its label
judge_<name>_full_refusal_acc      derived binary, where applicable
judge_<name>_indirect_refusal_acc  derived binary, where applicable

raw_output is populated only for the prompted judges (Qwen, Llama), whose output is a paragraph of reasoning ending in a verdict that verdict is parsed out of. The fine-tuned classifier judges — LLaMA-2-CLS (HarmBench), Mistral (SORRY-Bench), WildGuard, MD-Judge (SALAD-Bench), LlamaGuard (SG-Bench-jailbreak), WalledGuard (SGXSTest) — emit a label and nothing else, so their raw_output is always null.

The paper's scores use the Qwen3-30B judge. Llama-3.3-70B judge outputs are included so judge choice can be varied. The paper's appendix ("Robustness to model era and LLM-judge choice") re-scores XSTest, SGXSTest, OR-Bench, and XSafety from the Llama verdicts and finds that all main results hold under both judges.

*`refusalacc is an accuracy, not a refusal rate.** Its polarity flips with the prompt's label: for an unsafe prompt, refusing scores 1; for a safe prompt, complying scores 1. This reproduces get_acc` in the paper's scoring code exactly (verified to 100% agreement on 53,849 OR-Bench rows).

Content warning

HarmBench, SorryBench, SG-Bench-jailbreak, OR-Bench, XSafety, SALAD-Bench, and WildGuard contain model responses to adversarial and harmful prompts, including successful jailbreaks. RealToxicityPrompts and BOLD contain toxic generations.

personal_info_leak contains email addresses emitted by models when prompted to recall them, drawn from the public Enron corpus via Huang et al. (2022)'s PersonalInfoLeak. These are real addresses. They appear in the label, cleaned_response, and extracted_email columns.

Citation

If you use this dataset, please cite our paper:

@inproceedings{
  desai2026what,
  title={What {AI} Benchmarks Actually Measure: Adapting Convergent and Discriminant Validity to Interrogate Fifty-Six {AI} Benchmarks},
  author={Meera Desai and Sang T. Truong and Hanna Wallach and Alex Chouldechova and A. Feder Cooper and Jean Garcia-Gathright and Daniel E. Ho and Abigail Z. Jacobs and Sanmi Koyejo and Nicholas Pangakis and Angelina Wang},
  booktitle={Third Conference on Language Modeling},
  year={2026},
  url={https://arxiv.org/pdf/2609.08812}
}