datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
TempPerturb-Eval-data
TempPerturb-Eval-data
Summary
TempPerturb-Eval-data is the released output dataset for TempPerturb-Eval, a benchmark for analyzing the robustness of Retrieval-Augmented Generation (RAG) systems under both internal variation and external perturbation.
This is an evaluation-artifact dataset: it stores model outputs and experiment metadata for controlled robustness analysis, rather than a new QA training corpus.
The release covers:
5 models
11 temperatures from 0.0 to 2.0
4… See the full description on the dataset page: https://huggingface.co/datasets/yongxin2020/TempPerturb-Eval-data.Chat2Workflow-Evaluation
Chat2Workflow
Chat2Workflow is a benchmark designed for evaluating the ability of Large Language Models (LLMs) to generate executable visual workflows from natural language instructions.
Paper: Chat2Workflow: A Benchmark for Generating Executable Visual Workflows with Natural Language
Repository: zjunlp/Chat2Workflow
Overview
Executable visual workflows are widely used in industrial deployments for their reliability and controllability. Chat2Workflow addresses the… See the full description on the dataset page: https://huggingface.co/datasets/zjunlp/Chat2Workflow-Evaluation.DR3-EvalDR3-Eval: Towards Realistic and ReproducibleDeep Research Evaluation
✨ Overview
DR³-Eval is a realistic, reproducible, and multimodal evaluation benchmark for Deep Research Agents, focusing on multi-file report generation tasks.
Existing benchmarks face a fundamental tension between realism, controllability, and reproducibility when evaluating deep research agents. DR³-Eval addresses this through the following design:… See the full description on the dataset page: https://huggingface.co/datasets/NJU-LINK/DR3-Eval.S-Eval
S-Eval: Towards Automated and Comprehensive Safety Evaluation for Large Language Models
🏆 Leaderboard
🔔 Updates
📣 [2025/10/09]: We update the evaluation for the latest LLMs in 🏆 LeaderBoard, and further release Octopus, an automated LLM safety evaluator, to meet the community’s need for accurate and reproducible safety assessment tools. You can download the model from HuggingFace or ModelScope.
📣 [2025/03/30]: 🎉 Our paper has been accepted by ISSTA 2025. To meet… See the full description on the dataset page: https://huggingface.co/datasets/IS2Lab/S-Eval.compute-eval
Dataset Card for ComputeEval
ComputeEval is a benchmark for evaluating LLM-generated CUDA code on correctness and performance. Each problem provides a self-contained programming challenge — spanning kernels, runtime APIs, memory management, parallel algorithms, GPU libraries, and template/DSL kernel authoring — with a held-out test harness for functional validation and optional performance benchmarks for measuring GPU execution time against a baseline.
Homepage:… See the full description on the dataset page: https://huggingface.co/datasets/nvidia/compute-eval.EvalAwareBenchEvalAwareBench
Changling Li1,3, Terry Jingchen Zhang6, Jie Zhang1
Zhijing Jin3,5,6, Sahar Abdelnabi2,3,4, Maksym Andriushchenko2,3,4
1ETH Zürich, 2ELLIS Institute Tübingen, 3Max Planck Institute for Intelligent Systems, 4Tübingen AI Center, 5University of Toronto, 6Vector Institute
Dataset Summary
A factor-controlled benchmark for studying evaluation awareness in language models, where eight psychology-grounded trigger factors can be independently… See the full description on the dataset page: https://huggingface.co/datasets/aisa-group/EvalAwareBench.Feedback-Collection
Dataset Card
Dataset Summary
The Feedback Collection is a dataset designed to induce fine-grained evaluation capabilities into language models.\
Recently, proprietary LLMs (e.g., GPT-4) have been used to evaluate long-form responses. In our experiments, we found that open-source LMs are not capable of evaluating long-form responses, showing low correlation with both human evaluators and GPT-4.\
In our paper, we found that by (1) fine-tuning feedback generated by GPT-4… See the full description on the dataset page: https://huggingface.co/datasets/prometheus-eval/Feedback-Collection.dsv4-eval-artifacts
DeepSeek-V4-Flash-0731 — quantization measurements
Everything needed to reproduce, audit or extend the numbers published in
AtomicChat/DeepSeek-V4-Flash-0731-GGUF:
the reference logits, the evaluation corpus, the raw tool output for every quant we
measured, and the parsed results.
Every GGUF of this model that we could find on the Hub was measured here — ours,
unsloth's, bartowski's, ggml-org's, antirez's and others — on one machine, against one
reference, with one command.… See the full description on the dataset page: https://huggingface.co/datasets/AtomicChat/dsv4-eval-artifacts.eval-IFBench-results
IFBench Evaluation Results
This dataset contains evaluation results for various language models on IFBench, a challenging benchmark for precise instruction following.
Naming Convention: This repo follows the eval-{EVAL}-{type} schema for organizing evaluation datasets. Related repos:
eval-IFBench-results - Model evaluation outputs (this repo)
eval-IFBench-prompts - Test prompts/questions (if separated)
Dataset Structure
Results are organized by model name:… See the full description on the dataset page: https://huggingface.co/datasets/shisa-ai/eval-IFBench-results.OpenClaw-EvalMix
OpenClaw EvalMix
OpenClaw EvalMix is a Harbor-format collection of 360 agent-evaluation tasks across four task families. Each task directory includes task.toml, an instruction, an environment definition, and verifier tests.
Composition
Family
Tasks
Local payload
clawbench
19
0.00 GiB
liveclawbench
134
0.07 GiB
pinchbench
147
0.02 GiB
wildclawbench
60
14.05 GiB
The repository preserves each family at the root so task paths remain direct.… See the full description on the dataset page: https://huggingface.co/datasets/zt1106/OpenClaw-EvalMix.recon-eval
ReconEval — Financial Reconciliation Benchmark
Reading results from this benchmark. Four properties of ReconEval shape what
a score on it means. Anyone comparing models here should know them.
One class can dominate a margin. PARTIAL_MATCH is the highest-variance class
between models, and its 32 evaluation items are generated from 9 abbreviation
pairs — all of which also appear in the training split, overlap fraction 1.0. On
this set, "learned the concept" and "memorised nine… See the full description on the dataset page: https://huggingface.co/datasets/caiotheodoro/recon-eval.TheBioCollection-Eval
TheBioCollection-Eval
TheBioCollection-Eval is a biological evaluation suite for assessing large language models (BioLMs) for biology across small molecules, proteins, genomic sequences, cells/pathways, and cross-domain reasoning. It is constructed by drawing subtasks from many scattered existing benchmarks (Mol-Instructions, MolLangBench, BioReason-Pro, PerturBench Replogle K562) and combining them with source-derived newly-constructed instruction datasets.
Evaluation… See the full description on the dataset page: https://huggingface.co/datasets/trillionlabs/TheBioCollection-Eval.opdlm_eval_dataThis repository contains the evaluation data for OPDLM (On-Policy Diffusion Language Model), as presented in the paper Data-Efficient Autoregressive-to-Diffusion Language Models via On-Policy Distillation.
Project Page: https://opdlm.vercel.app/
GitHub Repository: https://github.com/divelab/OPDLM
Dataset Summary
OPDLM is an efficient, on-policy method for converting pre-trained autoregressive language models (ARLMs) into block-diffusion language models (DLMs). The datasets… See the full description on the dataset page: https://huggingface.co/datasets/divelab/opdlm_eval_data.sdf_evaluation_traits
Models That Know How Evaluations Are Designed Score Safer
This repository contains the synthetic documents used in the paper Models That Know How Evaluations Are Designed Score Safer.
Project Page | GitHub Repository
Dataset Description
These synthetic documents were used to fine-tune models to investigate evaluation meta-knowledge — parametric knowledge about the structural traits that characterize AI safety evaluations.
Documents were generated using the… See the full description on the dataset page: https://huggingface.co/datasets/compass-group-tue/sdf_evaluation_traits.fusion-pairwise-evals-test-time-scaling
Automatic pairwise preference evaluations for: Making, not taking, the Best-of-N
Content
This data contains pairwise automatic win-rate evaluations for the m-ArenaHard-v2.0 benchmark and it compares CommandA against gemini-2.5-pro in 2 settings:
Test-time scaling with Fusion : 5 samples are generated from CommandA, then fused with CommandA into one completion and compared to a single completion from gemini-2.5-pro
Test-time scaling with BoN : 5 samples are generated from… See the full description on the dataset page: https://huggingface.co/datasets/CohereLabs/fusion-pairwise-evals-test-time-scaling.proofjudge-eval-traces
ProofJudge Eval Traces
Agent interaction trajectories from ProofJudge evaluation runs,
stored in Harbor ATIF v1.6 format.
Each JSONL line is a complete trajectory of a judge agent evaluating one Lean 4 / Mathlib proof: the full
system and user prompts, every tool call it made to investigate the proof (search_mathlib, read_file,
pr_diff, …), every tool result it saw, and the structured verdict it returned.
Versions
This dataset has two generations, which must not… See the full description on the dataset page: https://huggingface.co/datasets/SJCaldwell/proofjudge-eval-traces.chess-sft-eval
Chess SFT Eval & Benchmark
Held-out evaluation splits and a frozen benchmark for the
Chess SFT training pipeline.
Every FEN in these files is excluded from training data via a blocklist to guarantee
zero contamination.
Eval examples
13,000
Benchmark examples
13,000
Splits
9 (perception, rules, tactics, evaluation, openings, endgames, planning, chess960, mate)
Format
JSONL
Training companion
Chess-Nut-Engine/chess-sft-data
How eval and benchmark differ… See the full description on the dataset page: https://huggingface.co/datasets/Chess-Nut-Engine/chess-sft-eval.fusion-pairwise-evals-finetuned
Automatic pairwise preference evaluations for: Making, not taking, the Best-of-N
Content
This data contains pairwise automatic win-rate evaluations for the m-ArenaHard-v2.0 benchmark and it compares 2 models against gemini-2.5-flash:
Fusion: is the 111B model finetuned on synthetic data generated with Fusion from 5 teachers
BoN: is the 111B model finetuned on synthetic data generated with BoN from 5 teachers
Each model’s outputs are compared in pairs with the respective… See the full description on the dataset page: https://huggingface.co/datasets/CohereLabs/fusion-pairwise-evals-finetuned.Health-Bench-Eval-OSS-2025-07
Dataset Card for HealthBench
Dataset Summary
HealthBench is a benchmark dataset developed by OpenAI in collaboration with 262 physicians from 60 countries to evaluate AI systems in health-related conversational scenarios. It contains 5,000 multi-turn health conversations in a JSONL file (2025-05-07-06-14-12_oss_eval.jsonl), simulating interactions between AI models and users (laypersons or clinicians). Each conversation includes a user prompt, a candidate model response… See the full description on the dataset page: https://huggingface.co/datasets/Tonic/Health-Bench-Eval-OSS-2025-07.r8-eval-suite-5bucket
⚠️ CRITICAL: Ollama Inference Flag Required for derived models
If you train or serve any Qwen3.5-9B-derived model from this lineage via Ollama,
you MUST pass "think": false in /api/chat requests for chat / instruction following / tool use.
The qwen3.5 RENDERER auto-injects <think> tags causing 25-46% empty-answer rates without this flag.
See dataset cudabenchmarktest/r9-research-framework/_OLLAMA_INFERENCE_WARNING.md for the full lesson learned.
R8/R9 Five-Bucket… See the full description on the dataset page: https://huggingface.co/datasets/cudabenchmarktest/r8-eval-suite-5bucket.Competence-Based-Evaluation
Competence-Based Evaluation (Invariance Benchmark)
A benchmark for testing whether language models give the same answer to
semantically equivalent reformulations of a logical-ordering question. Given a
set of pairwise constraints (e.g. Alice is in front of Bob), a model should
answer transitive-closure queries (Is Carol in front of Dave?) consistently
whether the constraints are stated using a relation or its inverse.
Each item exists as a paired (original, equivalent) record… See the full description on the dataset page: https://huggingface.co/datasets/jizej/Competence-Based-Evaluation.flight-search-intent-eval
Flight Search Intent Eval
244 real flight search queries in English, German and Spanish, each paired with the
structured search specification a model should produce from it.
This is the step before booking: turning one messy sentence into origin, destination,
dates, duration, passengers, cabin and filters. That specification decides which
itineraries a traveler is ever shown, so grading it is grading whether the model
understood which flights they meant. τ-bench and similar… See the full description on the dataset page: https://huggingface.co/datasets/fjmatrix/flight-search-intent-eval.llamafirewall-alignmentcheck-evals
Dataset Card for LlamaFirewall AlignmentCheck Evals
Dataset Details
Dataset Description
This dataset provides a dataset for prompt injection in an agentic environment. It is part of LlamaFirewall, an open-source security focused guardrail framework designed to serve as a final layer of defense against security risks associated with AI Agents. Specifically, this dataset is designed to evaluate the susceptibility of language models, and detect any misalignment… See the full description on the dataset page: https://huggingface.co/datasets/facebook/llamafirewall-alignmentcheck-evals.AMemGym
AMemGym Dataset
Dataset Overview
AMemGym (project website) is the first interactive, on-policy benchmark designed for evaluating memory capabilities in conversational AI assistants, especially in long-horizon dialogues. Unlike traditional static/offline benchmarks, AMemGym enables realistic and dynamic assessment of assistants in live, multi-turn interactions.
Dataset Structure
The dataset consists of structured conversation scenarios featuring user… See the full description on the dataset page: https://huggingface.co/datasets/AGI-Eval/AMemGym.tiered-finance-eval
Tiered Finance Eval
Twenty agentic finance tasks, each with the reference files an analyst would actually be handed, a
curated gold deliverable, and a tiered, gated rubric that scores a submission against that gold.
Evaluation results for these tasks are published in the companion Space:
CentificAIResearch/Tiered-Finance-Eval.
This dataset holds the tasks only: no model outputs and no scores.
[!IMPORTANT]
Canary string. TIERED-FINANCE-EVAL:d9e2f4a1-7c3b-4e86-9a05-2f1b8c6d40e7… See the full description on the dataset page: https://huggingface.co/datasets/CentificAIResearch/tiered-finance-eval.jam-rollout-arc-evals
Rollout arc — raw generations
Every model generation behind the write-ups in
mcp-tool-shop-org/ai-jam-sessions
under experiments/rollout-arc/p4/.
Two things you can do with this.
Check our arithmetic. The repo has the readout scripts, the preregistrations and the
intervals — but the generations they were computed from are ~51 MB and were never committed, so
a clone got the conclusions and no way to recompute them. These are those files, unfiltered.
Or run the loop yourself. The… See the full description on the dataset page: https://huggingface.co/datasets/mcp-tool-shop/jam-rollout-arc-evals.vn-spell-correction-eval-real
vn-spell-correction-eval-real
Out-of-distribution evaluation corpus for Vietnamese spell-correction
models — 150 hand-curated (noisy, clean) pairs sampled from real
VN error sources, not generated by nom.text.noise.
This is the test set we use to verify a spell-correction model
generalises beyond its own synthetic training distribution. A model
that scores 95 % on nom-vn's synthetic eval grid and 60 % on this
set is overfit to the noise generator.
Splits
Config… See the full description on the dataset page: https://huggingface.co/datasets/nrl-ai/vn-spell-correction-eval-real.cruciverb_it
Cruciverb-IT
Dataset adaptation of
cruciverb-it/evalita2026
for EVALITA-LLM. This repository contains only the data; prompts, parsers and
evaluation metrics are defined in the evaluation harness.
Task 1
Each record contains an Italian crossword clue, the expected answer length and
the gold answer:
{"id": "task1_test_000001", "clue": "...", "answer_length": 7, "answer": "..."}
The gold test data were cleaned and deterministically sampled into three nested
splits:… See the full description on the dataset page: https://huggingface.co/datasets/evalitahf/cruciverb_it.production-ai-guardrail-evals
Production AI Guardrail Evals
Eighteen synthetic, assertion-bearing cases for testing whether a language model can stay inside an advisory role. The cases cover roadmap intake, release readiness, and catalog-change review—the same workload families I use in the Winwood AI Toolkit around IEM Rig.
This is a sanitized public derivative, not a dump of application logs or the private evaluation corpus.
What each row contains
case_id: stable public identifier;… See the full description on the dataset page: https://huggingface.co/datasets/mattwinwood/production-ai-guardrail-evals.pmpp-eval
PMPP Dataset
This repository provides two CUDA-focused datasets prepared by Sinatras and sponsored by Prime Intellect. Both datasets are based on Programming Massively Parallel Processors (4th Ed.) with additional coding evaluation harnesses at https://github.com/SinatrasC/pmpp-eval to be used by PMMP env in prime-environments.
Overview
Languages: English
License: MIT
Curated by: Sinatras (https://github.com/SinatrasC)
Sponsored by: Prime Intellect
Derived from: PMPP 4th… See the full description on the dataset page: https://huggingface.co/datasets/sinatras/pmpp-eval.