datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
llm-refusal-evaluation
🛡️ LLM Refusal Evaluation Benchmark
This repository contains the benchmarks used in the LLM-Refusal-Evaluation suite.
The prompts are organized into three groups:
Safety Benchmarks — harmful / jailbreak-style prompts that models should refuse.
Chinese Sensitive Topics — prompts that may be censored by China-aligned models.
Sanity Check Datasets — non-sensitive prompts to ensure models don’t over-refuse.
📌 Contents
Safety Benchmarks
JailbreakBench
SorryBench… See the full description on the dataset page: https://huggingface.co/datasets/MultiverseComputingCAI/llm-refusal-evaluation.llm-instruction-following-eval
LLM Instruction-Following Evaluation: 256 Models Across 20 Diagnostic Tests
Dataset Summary
This dataset contains comprehensive evaluation results from testing 256 Large Language Models across 20 carefully designed diagnostic instruction-following prompts, totaling 5,120 individual evaluations. The evaluation was conducted on October 14, 2025, using the OpenRouter API.
Paper: When Models Can't Follow: Testing Instruction Adherence Across 256 LLMs
arXiv: 2510.18892… See the full description on the dataset page: https://huggingface.co/datasets/richardyoung/llm-instruction-following-eval.llm-jp-eval
Dataset Card for llm-book/ja-vicuna-qa-benchmark
書籍『大規模言語モデル入門Ⅱ〜生成型LLMの実装と評価』で使用する、LLM-jpが提供する「llm-jp-eval」で使用されるデータセットです。
オリジナルのサイトと同じものを使用しています。
本データセットは、複数のデータセットを横断して日本語の大規模言語モデルを自動評価するにの使用されます。
Licence
Apache License, Version 2.0
ua-llm-router-eval
UA Specialist Router — evaluation progress
Score tables, routing stats, and run metadata from the diploma project
MariaOnyshchuk/ua-llm-router:
a rules-based router over open Ukrainian specialists (Mamay-4B, Lapa-12B, Aya Expanse 8B, Qwen-Coder).
This dataset is the progress log of pinned JSON summaries, not a dump of every generation.
What is included
Path
Contents
progress_ledger.csv
Flattened metric rows across weeks (best table for browsing)… See the full description on the dataset page: https://huggingface.co/datasets/MariaOnyshchuk/ua-llm-router-eval.LLMEval-Fair
LLMEval-Fair: A Large-Scale Longitudinal Study on Robust and Fair Evaluation of Large Language Models
LLMEval-Fair is a 30-month longitudinal study on the robustness and fairness of LLM evaluation,
built on a proprietary bank of 220,000+ graduate-level Chinese questions spanning 13 academic disciplines.
This is the publicly released subset of that bank.
Paper (arXiv): https://arxiv.org/abs/2508.05452
Venue: ACL 2026 Main Conference
Project website: https://llmeval.com/
GitHub:… See the full description on the dataset page: https://huggingface.co/datasets/llmeval-fdu/LLMEval-Fair.llm-refusal-evaluation
🛡️ LLM Refusal Evaluation Benchmark
This repository contains the benchmarks used in the LLM-Refusal-Evaluation suite.
The prompts are organized into three groups:
Safety Benchmarks — harmful / jailbreak-style prompts that models should refuse.
Chinese Sensitive Topics — prompts that may be censored by China-aligned models.
Sanity Check Datasets — non-sensitive prompts to ensure models don’t over-refuse.
📌 Contents
Safety Benchmarks
JailbreakBench
SorryBench… See the full description on the dataset page: https://huggingface.co/datasets/BushNate/llm-refusal-evaluation.llm-debugger-eval-transcripts
llm-debugger evaluation transcripts
Every turn behind the results reported in
llm-debugger: the base model,
the SFT initialisation, and the RL policies trained from it. Exploratory runs no
reported figure depends on are not included.
Layout
path
what
runs/base/
Qwen3-Coder-30B-A3B-Instruct, 8 runs on the 30-task test split
runs/sft/
the SFT initialisation, 3 runs on the test split
runs/rl-gate-arc/
the RL gate arc, v15 through v120 (3 runs each, 8… See the full description on the dataset page: https://huggingface.co/datasets/moofeez/llm-debugger-eval-transcripts.LLMEval-1
LLMEval-1: Large-Scale Chinese LLM Evaluation (Phase I)
LLMEval-1 is the Phase I evaluation dataset of the LLMEval project (Fudan NLP Lab),
released alongside the paper "LLMEval: A Preliminary Study on How to Evaluate Large Language Models" (AAAI 2024).
It is a Chinese, generative-QA benchmark designed to study how large language models should be evaluated.
Paper: https://arxiv.org/abs/2312.07398
Project website: https://llmeval.com/
GitHub: https://github.com/llmeval/LLMEval-1… See the full description on the dataset page: https://huggingface.co/datasets/llmeval-fdu/LLMEval-1.LLMEval-2
LLMEval-2: Professional Domain Evaluation of Chinese LLMs (Phase II)
LLMEval-2 is the Phase II evaluation dataset of the LLMEval project (Fudan NLP Lab),
released alongside the AAAI 2024 paper "LLMEval: A Preliminary Study on How to Evaluate Large Language Models".
While LLMEval-1 focuses on general capabilities, LLMEval-2 targets professional domain evaluation across
12 academic disciplines.
Paper: https://arxiv.org/abs/2312.07398
Project website: https://llmeval.com/
GitHub:… See the full description on the dataset page: https://huggingface.co/datasets/llmeval-fdu/LLMEval-2.llm-output-evaluation
LLM_OUTPUT_EVALUATION
A preference dataset for LLM_OUTPUT_EVALUATION, harvested from real, human-labelled sources and curated by an automated harvesting harness with an LLM quality gate.
Format
Standard preference / DPO schema — each row:
column
meaning
prompt
the request (originally prompt)
chosen
the human-preferred response
rejected
a worse response to the same prompt
source
the dataset/URL the row was harvested from
Splits… See the full description on the dataset page: https://huggingface.co/datasets/316usman/llm-output-evaluation.LLMEval-Med
LLMEval-Med: A Real-world Clinical Benchmark for Medical LLMs with Physician Validation
LLMEval-Med is a physician-validated benchmark for evaluating Large Language Models on real-world
medical tasks. The questions are drawn from real electronic health records and expert-designed
clinical scenarios, and the LLM-as-Judge evaluation pipeline is calibrated against medical experts.
Paper (arXiv): https://arxiv.org/abs/2506.04078
ACL Anthology:… See the full description on the dataset page: https://huggingface.co/datasets/llmeval-fdu/LLMEval-Med.synthetic-llm-evaluation-traces
EvaluLLM-Inspired Synthetic Evaluation Traces (DBbun)
View source code on GitHub
Watch on Youtube: Evaluating AI with AI
Dataset Summary
This dataset contains fully synthetic evaluation traces for NLG / LLM-style output comparison, inspired by the evaluation workflow described in EvaluLLM: LLM Assisted Evaluation of Generative Outputs (IUI Companion 2024).
The dataset is produced by a configurable, offline simulator and includes:
synthetic tasks (prompts)
synthetic… See the full description on the dataset page: https://huggingface.co/datasets/DBbun/synthetic-llm-evaluation-traces.step2-evaluated-dataset-test2
Complete Evaluation Dataset (Rubric + LogP)
This dataset contains chain-of-thought explanations evaluated using both comprehensive rubric assessment and LogP evaluation.
Overview
Source Dataset: llm-compe-2025-kato/step2-evaluated-dataset-test2
Total Samples: 92
Successfully Evaluated (Rubric): 92
Failed Evaluations (Rubric): 0
Evaluation Model: Qwen/Qwen3-32B
Rubric Evaluation Results
Average Rubric Scores (0-4 scale)
logical_coherence: 3.51… See the full description on the dataset page: https://huggingface.co/datasets/llm-compe-2025-kato/step2-evaluated-dataset-test2.step2-evaluated-dataset-Qwen3-14B-cp32
Complete Evaluation Dataset (Rubric + LogP)
This dataset contains chain-of-thought explanations evaluated using both comprehensive rubric assessment and LogP evaluation.
Overview
Source Dataset: llm-compe-2025-kato/step2-evaluated-dataset-Qwen3-14B-cp32
Total Samples: 60
Successfully Evaluated (Rubric): 53
Failed Evaluations (Rubric): 7
Evaluation Model: Qwen/Qwen3-32B
Rubric Evaluation Results
Average Rubric Scores (0-4 scale)
logical_coherence:… See the full description on the dataset page: https://huggingface.co/datasets/llm-compe-2025-kato/step2-evaluated-dataset-Qwen3-14B-cp32.step2-evaluated-dataset-Qwen3-14B-cp40
Complete Evaluation Dataset (Rubric + LogP)
This dataset contains chain-of-thought explanations evaluated using both comprehensive rubric assessment and LogP evaluation.
Overview
Source Dataset: llm-compe-2025-kato/step2-evaluated-dataset-Qwen3-14B-cp40
Total Samples: 58
Successfully Evaluated (Rubric): 53
Failed Evaluations (Rubric): 5
Evaluation Model: Qwen/Qwen3-32B
Rubric Evaluation Results
Average Rubric Scores (0-4 scale)
logical_coherence:… See the full description on the dataset page: https://huggingface.co/datasets/llm-compe-2025-kato/step2-evaluated-dataset-Qwen3-14B-cp40.step2-DeepMath-103K-Bespoke-Filtered-Test-200-eval-cp40
Chain of Thought生成データセット
このデータセットは、問題と解答から説明（Chain of Thought）を生成したデータセットです。
概要
処理したサンプル数: 58
有効な説明生成数: 58
生成成功率: 100.00%
使用モデル: /home/Competition2025/P07/shareP07/share_model/step2_rlt/Qwen3-14B-step2-deepmath103k-bs512/checkpoint-40
トークン数統計
最小トークン数: 830
最大トークン数: 6301
平均トークン数: 3477.9
トークン数分布
0-100トークン: 0件 (0.0%)
101-500トークン: 0件 (0.0%)
501-1000トークン: 1件 (1.7%)
1001-2000トークン: 9件 (15.5%)
2001-5000トークン: 39件 (67.2%)
5001+トークン: 9件 (15.5%)… See the full description on the dataset page: https://huggingface.co/datasets/llm-compe-2025-kato/step2-DeepMath-103K-Bespoke-Filtered-Test-200-eval-cp40.llm-evaluation-sft-100k
LLM Evaluation SFT 100K
A synthetic supervised fine-tuning dataset of 100,000 high-quality conversations covering LLM evaluation methodology, benchmarking, safety assessment, and prompt optimization. Designed to train AI assistants that can help ML engineers and researchers rigorously evaluate and improve language models.
Dataset Description
This dataset covers the full spectrum of LLM evaluation practice across 7 specialized categories. Each record follows the… See the full description on the dataset page: https://huggingface.co/datasets/stindardlogic/llm-evaluation-sft-100k.step2-evaluated-dataset-Qwen3-14B
Complete Evaluation Dataset (Rubric + LogP)
This dataset contains chain-of-thought explanations evaluated using both comprehensive rubric assessment and LogP evaluation.
Overview
Source Dataset: llm-compe-2025-kato/step2-evaluated-dataset-Qwen3-14B
Total Samples: 156
Successfully Evaluated (Rubric): 135
Failed Evaluations (Rubric): 21
Evaluation Model: Qwen/Qwen3-32B
Rubric Evaluation Results
Average Rubric Scores (0-4 scale)
logical_coherence:… See the full description on the dataset page: https://huggingface.co/datasets/llm-compe-2025-kato/step2-evaluated-dataset-Qwen3-14B.step2-DeepMath-103K-Bespoke-Filtered-Test-200-eval-Qwen3-14B
Chain of Thought生成データセット
このデータセットは、問題と解答から説明（Chain of Thought）を生成したデータセットです。
概要
処理したサンプル数: 156
有効な説明生成数: 156
生成成功率: 100.00%
使用モデル: Qwen/Qwen3-14B
トークン数統計
最小トークン数: 313
最大トークン数: 3066
平均トークン数: 887.4
トークン数分布
0-100トークン: 0件 (0.0%)
101-500トークン: 26件 (16.7%)
501-1000トークン: 85件 (54.5%)
1001-2000トークン: 39件 (25.0%)
2001-5000トークン: 6件 (3.8%)
5001+トークン: 0件 (0.0%)
データセット構造
system_prompt: モデルに送信されたシステムプロンプト
question_text: 元の問題文
answer_text: 問題の解答… See the full description on the dataset page: https://huggingface.co/datasets/llm-compe-2025-kato/step2-DeepMath-103K-Bespoke-Filtered-Test-200-eval-Qwen3-14B.step2-pairwise-eval
Pairwise Comparison Results
This dataset contains the results of a pairwise comparison between two models' explanation generation capabilities.
Overview
Total Comparisons: 68
Model A (Dataset): llm-compe-2025-kato/Tag-Validation_Qwen3-14B-Step1-Bespoke17k-ep1
Model B (Dataset): llm-compe-2025-kato/Tag-Validation_Qwen3-14B-Step1-Bespoke17k-ep1
Results Summary
Winner: Model A (Dataset: llm-compe-2025-kato/Tag-Validation_Qwen3-14B-Step1-Bespoke17k-ep1)
Margin:… See the full description on the dataset page: https://huggingface.co/datasets/llm-compe-2025-kato/step2-pairwise-eval.step2-DeepMath-103K-Bespoke-Filtered-Test-200-eval-cp32
Chain of Thought生成データセット
このデータセットは、問題と解答から説明（Chain of Thought）を生成したデータセットです。
概要
処理したサンプル数: 60
有効な説明生成数: 60
生成成功率: 100.00%
使用モデル: /home/Competition2025/P07/shareP07/share_model/step2_rlt/Qwen3-14B-step2-deepmath103k-bs512/checkpoint-32
トークン数統計
最小トークン数: 892
最大トークン数: 5932
平均トークン数: 3560.4
トークン数分布
0-100トークン: 0件 (0.0%)
101-500トークン: 0件 (0.0%)
501-1000トークン: 1件 (1.7%)
1001-2000トークン: 10件 (16.7%)
2001-5000トークン: 41件 (68.3%)
5001+トークン: 8件 (13.3%)… See the full description on the dataset page: https://huggingface.co/datasets/llm-compe-2025-kato/step2-DeepMath-103K-Bespoke-Filtered-Test-200-eval-cp32.llm-evaluation-huji
DOVE