datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
all-defectsmodel-written-evals
Model-Written Evaluation Datasets
This repository includes datasets written by language models, used in our paper on "Discovering Language Model Behaviors with Model-Written Evaluations."
We intend the datasets to be useful to:
Those who are interested in understanding the quality and properties of model-generated data
Those who wish to use our datasets to evaluate other models for the behaviors we examined in our work (e.g., related to model persona, sycophancy, advanced AI risks… See the full description on the dataset page: https://huggingface.co/datasets/Anthropic/model-written-evals.quantized-llama-3.1-leaderboard-v2-evals
Open LLM Leaderboard v2 Benchmark Results
This artifact contains all the data from evaluations of Neural Magic's quantized Llama-3.1 models.
These evaluations were produced with lm-evaluation-harness by running the following command:
lm_eval \
--model vllm \
--model_args pretrained="<model_path>",dtype=auto,add_bos_token=False,max_model_len=4096,tensor_parallel_size="<num_gpus>",gpu_memory_utilization=0.8,enable_chunked_prefill=True \
--apply_chat_template \… See the full description on the dataset page: https://huggingface.co/datasets/neuralmagic/quantized-llama-3.1-leaderboard-v2-evals.transformers-merge-experimentstest-traces
Test Traces
Codex-style rollout JSONL traces exported from fast-agent for validation against the Hugging Face Agent Trace Viewer.
model-written-evalssimple-evalsfusion-pairwise-evals-test-time-scaling
Automatic pairwise preference evaluations for: Making, not taking, the Best-of-N
Content
This data contains pairwise automatic win-rate evaluations for the m-ArenaHard-v2.0 benchmark and it compares CommandA against gemini-2.5-pro in 2 settings:
Test-time scaling with Fusion : 5 samples are generated from CommandA, then fused with CommandA into one completion and compared to a single completion from gemini-2.5-pro
Test-time scaling with BoN : 5 samples are generated from… See the full description on the dataset page: https://huggingface.co/datasets/CohereLabs/fusion-pairwise-evals-test-time-scaling.fusion-pairwise-evals-finetuned
Automatic pairwise preference evaluations for: Making, not taking, the Best-of-N
Content
This data contains pairwise automatic win-rate evaluations for the m-ArenaHard-v2.0 benchmark and it compares 2 models against gemini-2.5-flash:
Fusion: is the 111B model finetuned on synthetic data generated with Fusion from 5 teachers
BoN: is the 111B model finetuned on synthetic data generated with BoN from 5 teachers
Each model’s outputs are compared in pairs with the respective… See the full description on the dataset page: https://huggingface.co/datasets/CohereLabs/fusion-pairwise-evals-finetuned.generalization-dynamics-evals
Generalization Dynamics — Main Eval Suite
Prepared test sets for the 6 main evaluation families from
Generalization dynamics across fine-tuning
(Table 1).
Use with the unified runner:
https://github.com/jiaxin-wen/FT-generalization/tree/main/release
from huggingface_hub import snapshot_download
root = snapshot_download(
repo_id="jiaxin-wen/generalization-dynamics-evals", repo_type="dataset")
Or browse a single task (the dataset viewer shows all configs):
from datasets… See the full description on the dataset page: https://huggingface.co/datasets/jiaxin-wen/generalization-dynamics-evals.core-bench-v1.1-mainlinewixqa-gpt-oss-120b-all-MiniLM-L6-v2-pgvector-evalsmodel-toolcall-research
Model Toolcall Research
LoRWeB_evalset
LoRWeB Custom Evaluation Dataset
ArXiv | GitHub Repository | 🌐 Project Website | 🤗 Model Checkpoint
Hila Manor1,2, Rinon Gal2, Haggai Maron1,2, Tomer Michaeli1, Gal Chechik2,3
1Technion - Israel Institute of Technology 2NVIDIA 3Bar-Ilan University
Given a prompt and an image triplet {a, a', b} that visually describe a desired transformation, LoRWeB dynamically constructs a single LoRA from a learnable basis of LoRA modules, and produces an editing result… See the full description on the dataset page: https://huggingface.co/datasets/hilamanor/LoRWeB_evalset.LoraRetriever_EvalSetllamafirewall-alignmentcheck-evals
Dataset Card for LlamaFirewall AlignmentCheck Evals
Dataset Details
Dataset Description
This dataset provides a dataset for prompt injection in an agentic environment. It is part of LlamaFirewall, an open-source security focused guardrail framework designed to serve as a final layer of defense against security risks associated with AI Agents. Specifically, this dataset is designed to evaluate the susceptibility of language models, and detect any misalignment… See the full description on the dataset page: https://huggingface.co/datasets/facebook/llamafirewall-alignmentcheck-evals.jam-rollout-arc-evals
Rollout arc — raw generations
Every model generation behind the write-ups in
mcp-tool-shop-org/ai-jam-sessions
under experiments/rollout-arc/p4/.
Two things you can do with this.
Check our arithmetic. The repo has the readout scripts, the preregistrations and the
intervals — but the generations they were computed from are ~51 MB and were never committed, so
a clone got the conclusions and no way to recompute them. These are those files, unfiltered.
Or run the loop yourself. The… See the full description on the dataset page: https://huggingface.co/datasets/mcp-tool-shop/jam-rollout-arc-evals.eval_scanqa_sqaproduction-ai-guardrail-evals
Production AI Guardrail Evals
Eighteen synthetic, assertion-bearing cases for testing whether a language model can stay inside an advisory role. The cases cover roadmap intake, release readiness, and catalog-change review—the same workload families I use in the Winwood AI Toolkit around IEM Rig.
This is a sanitized public derivative, not a dump of application logs or the private evaluation corpus.
What each row contains
case_id: stable public identifier;… See the full description on the dataset page: https://huggingface.co/datasets/mattwinwood/production-ai-guardrail-evals.core-bench-v1.1-oodgsd-sft-evals
gsd-sft-evals
Multilingual benchmark scores for DataSmith finetunes and the base model.
data/*.jsonl is the aggregated scores table (one row per revision x task x metric);
samples/ holds raw per-sample logs and is not part of the viewer table.
running-coach-evalsoreolook-research-evals
OreoLook Research Evaluations
A small, inspectable evaluation suite for current-information search agents. It
covers routing, clarification, conversational continuity, citation discipline,
freshness, PDF artifacts, protocol safety, and semantic-cache equivalence.
This repository accompanies:
OreoLook live Space
OreoLook source
Architecture paper
Project website
Contents
Configuration
Rows
Purpose
evaluations
24
Synthetic user turns and expected agent… See the full description on the dataset page: https://huggingface.co/datasets/p-research/oreolook-research-evals.openclaw-git-labels
OpenClaw Label V7A
This dataset is the publication bundle for the OpenClaw topic-label benchmark
and GEPA optimization regimes.
Contents
data/splits/feedback300.jsonl GEPA feedback/train rows
data/splits/pareto60.jsonl GEPA Pareto validation rows
data/splits/bench78.jsonl held-out benchmark rows
data/splits/split-manifest.json split hashes, overlaps, and label counts
data/final/final-ledger.jsonl final 330-row five-model ledger… See the full description on the dataset page: https://huggingface.co/datasets/evalstate/openclaw-git-labels.suture-evals
Suture evals
Published predictions on the seed-777 gold in caiotheodoro/suture (benchmark). Score locally with no GPU:
cd forge && PYTHONPATH=src uv run python -m suture_forge.cli class-report \
--gold benchmark.jsonl --pred sft-dedhi.jsonl
Config
System
777 recall
HIGH
Prec
Parse
sft-dedhi
published adapter
0.959
0.969
0.956
1.0
sft-limithi
prior published
0.839
0.893
0.870
1.0
sft-ded
unpublished DED mix
0.886
0.913
0.941
1.0
luna
GPT-5.6 Luna zero-shot… See the full description on the dataset page: https://huggingface.co/datasets/caiotheodoro/suture-evals.gacar-assistant-evals
GACAR Assistant Evals (Saudi Civil Aviation)
The official evaluation dataset for Captain Adel (captadel.com / Fly GACA) — the independent, retrieval-grounded AI flight instructor for Saudi civil aviation regulations (GACAR).
Dataset Summary
Every prompt or model change to Captain Adel is eval-gated in both English and Arabic against this suite.
Total Cases: 150
Domains Covered: 30+ GACAR Parts (Part 1, 61, 67, 91, 107, 121, 135, 139, etc.)
Categories: citation… See the full description on the dataset page: https://huggingface.co/datasets/flygaca/gacar-assistant-evals.llamafirewall-alignmentcheck-evals
Dataset Card for LlamaFirewall AlignmentCheck Evals
Dataset Details
Dataset Description
This dataset provides a dataset for prompt injection in an agentic environment. It is part of LlamaFirewall, an open-source security focused guardrail framework designed to serve as a final layer of defense against security risks associated with AI Agents. Specifically, this dataset is designed to evaluate the susceptibility of language models, and detect any misalignment… See the full description on the dataset page: https://huggingface.co/datasets/jiayucunyan/llamafirewall-alignmentcheck-evals.financial-model-evals
Worthune Financial Model Evals (open sample)
Ground truth for financial AI: 3 open datasets × 250
input/expected-output pairs — the free-sample slice of a 55-model catalog
covering refinance break-evens, retirement projections, Roth conversions,
equity compensation, loan payoff, and more. The full catalog's datasets
download with a Worthune Pro key from
GET https://worthune.com/api/v1/evals/{model} (index).
Every expected output comes from two independent implementations that… See the full description on the dataset page: https://huggingface.co/datasets/worthune/financial-model-evals.quantized-llama-3.1-arena-hard-evals
Arena-Hard Benchmark Results
This artifact contains all the data neccessary to reproduce the results of the Arena-Hard benchmark for Neural Magic's quantized Llama-3.1 models.
The model_answers directory includes the generated answers from all models, and the model_judgements directory contains the evaluations by gpt-4-1106-preview.
The Arena-Hard version used for benchmarking is v0.1.0, corresponding to commit efc012e192b88024a5203f5a28ec8fc0342946df.
All model answers were… See the full description on the dataset page: https://huggingface.co/datasets/neuralmagic/quantized-llama-3.1-arena-hard-evals.lichess-evals