datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
chatbot-arena-llm-judges
Chatbot-Arena
https://www.kaggle.com/competitions/lmsys-chatbot-arena/data
Single-turn data: https://huggingface.co/datasets/potsawee/chatbot-arena-llm-judges
#examples = 49938
split: A_win = 17312 (34.67%), B_win = 16985 (34.01%), tie = 15641 (31.32%)
#2-way only examples = 34297 (68.68%)
This repository
train.single-turn.json: data extracted from the train file from LMSys on Kaggle
each example has attributes - id, model_[a, b], winne_model_[a, b, tie], question… See the full description on the dataset page: https://huggingface.co/datasets/potsawee/chatbot-arena-llm-judges.mmlu-llm-judges
MMLU (LLM judges)
Unrolled into [Question] + [Options] + [Selected Answer] — “Is the answer correct to the question?”
#examples = 14042*2 = 28084 (one incorrect is randomly selected)
split: correct = 50.0%, incorrect = 50.0%
This repository
mmlu-unrolled.json: data extracted from https://huggingface.co/datasets/cais/mmlu
each example has attributes - question, 'choices, answer, label (correctorincorrect`)
llm-judges folder contains predictions made by each judge… See the full description on the dataset page: https://huggingface.co/datasets/potsawee/mmlu-llm-judges.rq2-all-records-old-llm-judgestruthful-qa-llm-judges
TruthfulQA
Unrolled into [Question] + [Answer] — “Is the answer truthful?”
#examples = 5918
split: correct = 43.93%, incorrect = 56.07%
This repository
truthful_qa.unrolled.json: data extracted from https://huggingface.co/datasets/truthfulqa/truthful_qa
each example has attributes - question, answer, label (correct or incorrect)
llm-judges folder contains predictions made by each judge (logits over correct and incorrect)
LLM Judge Statistics… See the full description on the dataset page: https://huggingface.co/datasets/potsawee/truthful-qa-llm-judges.halueval-llm-judges
HaluEval
Data: https://github.com/RUCAIBox/HaluEval/blob/main/data/dialogue_data.json
Judge prompt: https://github.com/RUCAIBox/HaluEval/blob/main/evaluation/dialogue/dialogue_evaluation_instruction.txt
split: 20,000 examples in total --> 10,000 good responses (i.e., ref="no") + 10,000 hallucinated responses (i.e., ref="yes")
This repository
llm-judges folder contains predictions made by each judge (logits over Yes and No)
Note that "Yes" means there is a… See the full description on the dataset page: https://huggingface.co/datasets/potsawee/halueval-llm-judges.rq2-all-records-new-llm-judgesrq2-all-records-new-llm-judges-gemmarq2-all-records-new-llm-judges-falconrepro-a-coin-flip-for-safety-llm-judges-fail-to-reliably-measure-adversarial-robustness
Reproduction: A Coin Flip for Safety - LLM Judges Fail to Reliably Measure Adversarial Robustness
Paper Information
Title: A Coin Flip for Safety: LLM Judges Fail to Reliably Measure Adversarial Robustness
OpenReview ID: RgnoWsmYBM
Conference: ICML 2026
Task: Evaluate 4 LLM-based safety judges on 6,642 human-verified adversarial prompts
Reproduction Summary
This paper's all 6 claims require LLM inference on proprietary adversarial prompts and… See the full description on the dataset page: https://huggingface.co/datasets/sabaridsnfuji/repro-a-coin-flip-for-safety-llm-judges-fail-to-reliably-measure-adversarial-robustness.rq2-all-records-new-llm-judges-llamarq2-all-records-new-llm-judges-mistral