datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
chatbot-arena-llm-judges
Chatbot-Arena
https://www.kaggle.com/competitions/lmsys-chatbot-arena/data
Single-turn data: https://huggingface.co/datasets/potsawee/chatbot-arena-llm-judges
#examples = 49938
split: A_win = 17312 (34.67%), B_win = 16985 (34.01%), tie = 15641 (31.32%)
#2-way only examples = 34297 (68.68%)
This repository
train.single-turn.json: data extracted from the train file from LMSys on Kaggle
each example has attributes - id, model_[a, b], winne_model_[a, b, tie], question… See the full description on the dataset page: https://huggingface.co/datasets/potsawee/chatbot-arena-llm-judges.rq2-all-records-old-llm-judgeshalueval-llm-judges
HaluEval
Data: https://github.com/RUCAIBox/HaluEval/blob/main/data/dialogue_data.json
Judge prompt: https://github.com/RUCAIBox/HaluEval/blob/main/evaluation/dialogue/dialogue_evaluation_instruction.txt
split: 20,000 examples in total --> 10,000 good responses (i.e., ref="no") + 10,000 hallucinated responses (i.e., ref="yes")
This repository
llm-judges folder contains predictions made by each judge (logits over Yes and No)
Note that "Yes" means there is a… See the full description on the dataset page: https://huggingface.co/datasets/potsawee/halueval-llm-judges.rq2-all-records-new-llm-judgesrq2-all-records-new-llm-judges-gemmarq2-all-records-new-llm-judges-falconrepro-a-coin-flip-for-safety-llm-judges-fail-to-reliably-measure-adversarial-robustness
Reproduction: A Coin Flip for Safety - LLM Judges Fail to Reliably Measure Adversarial Robustness
Paper Information
Title: A Coin Flip for Safety: LLM Judges Fail to Reliably Measure Adversarial Robustness
OpenReview ID: RgnoWsmYBM
Conference: ICML 2026
Task: Evaluate 4 LLM-based safety judges on 6,642 human-verified adversarial prompts
Reproduction Summary
This paper's all 6 claims require LLM inference on proprietary adversarial prompts and… See the full description on the dataset page: https://huggingface.co/datasets/sabaridsnfuji/repro-a-coin-flip-for-safety-llm-judges-fail-to-reliably-measure-adversarial-robustness.rq2-all-records-new-llm-judges-llamarq2-all-records-new-llm-judges-mistral