datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
jailbreak-detection-dataset
Jailbreak Detection Dataset (MLCommons-Aligned)
A comprehensive dataset for training AI safety classifiers, aligned with the MLCommons AI Safety taxonomy.
Dataset Description
This dataset combines multiple sources for robust jailbreak and safety detection:
Primary Sources
nvidia/Aegis-AI-Content-Safety-Dataset-2.0: 18,164 samples with MLCommons-aligned labels
lmsys/toxic-chat: Toxic content detection
jackhhao/jailbreak-classification: Jailbreak attack patterns… See the full description on the dataset page: https://huggingface.co/datasets/llm-semantic-router/jailbreak-detection-dataset.fact-check-classification-dataset
Fact-Check Classification Dataset
🎯 Purpose: Binary classification dataset for determining whether a prompt needs external fact-checking.
Dataset Description
This dataset is designed to train classifiers that can route LLM requests based on whether they require external fact verification. It's part of the vLLM Semantic Router project.
Labels
FACT_CHECK_NEEDED (1): Information-seeking questions requiring external verification
Factual questions about dates… See the full description on the dataset page: https://huggingface.co/datasets/llm-semantic-router/fact-check-classification-dataset.longcontext-haldetect
Long-Context Hallucination Detection Benchmark
A synthetic benchmark dataset for evaluating hallucination detection models on long documents (8K-24K tokens). This dataset is specifically designed to test models that can handle contexts beyond the typical 8K token limit.
Dataset Summary
Property
Value
Total samples
3,366
Token range
8,005 - 23,998
Average tokens
17,852
Hallucinated
1,681 (49.9%)
Supported
1,685 (50.1%)
Splits… See the full description on the dataset page: https://huggingface.co/datasets/llm-semantic-router/longcontext-haldetect.feedback-detector-dataset
Feedback Detector Dataset
A large-scale multilingual dataset for 4-class user feedback classification, labeled using GPT-OSS-120B on AMD MI300X GPU.
Dataset Description
This dataset contains 51,694 examples of user feedback classified into 4 categories:
Label
Description
Count
%
SAT
User is satisfied
8,649
17%
NEED_CLARIFICATION
User needs more information
16,179
31%
WRONG_ANSWER
System gave incorrect response
19,919
39%
WANT_DIFFERENT
User wants… See the full description on the dataset page: https://huggingface.co/datasets/llm-semantic-router/feedback-detector-dataset.mlcommons-ai-safety-synth
MLCommons AI Safety Synthesized Dataset
Synthesized training data for AI safety classifiers based on the MLCommons AI Safety Hazard Taxonomy.
Dataset Description
This dataset contains 12,000 synthesized unsafe prompts across 6 hazard categories, designed to augment training data for content safety classifiers. Each category contains 2,000 balanced samples.
Hazard Categories (MLCommons AI Safety Taxonomy)
Category
Description
Samples… See the full description on the dataset page: https://huggingface.co/datasets/llm-semantic-router/mlcommons-ai-safety-synth.modality-routing-dataset
Modality Routing Dataset
This dataset materializes the dynamic modality routing data builder used by the local
mmBERT-32K modality router training pipeline. The export is intended for review,
versioning, and uploading to a Hugging Face dataset repository.
Labels
Label
ID
Description
AR
0
Text-only requests that should route to an autoregressive LLM.
DIFFUSION
1
Image-generation requests that should route to a diffusion model.
BOTH
2
Requests that benefit… See the full description on the dataset page: https://huggingface.co/datasets/llm-semantic-router/modality-routing-dataset.llm-router-dataset
llm-router dataset
Training data for ai-mitra/llm-router,
a prompt task-classifier used by the
llm-router Python
package to route prompts to the best-fit LLM in agentic AI systems.
Each row is a prompt labeled with the task category it belongs to.
Labels
simple, coding, reasoning, security, summarization
Files
File
Rows
Purpose
training_data.jsonl
1700 (340/label)
Used to train the classifier: 35 hand-written examples per category plus… See the full description on the dataset page: https://huggingface.co/datasets/ai-mitra/llm-router-dataset.llm-router-train-initialPACE-LLMRouterBench
PACE LLMRouterBench assets
CSV splits, query embeddings, and baseline prediction matrices used by the PACE
routing paper. Corresponding code: GitHub repository PACE.
Contents
data/{train,id_test,ood_test}_perf.csv
embeddings/{original,0618e180_perf,0629e5880_perf}/{train,id_test,ood_test}.pt
baselines/EmbedLLM_eval_matrix_{id,ood}.pth
baselines/0618e180_perf_learned_id_attention_seed_ensemble_{id,ood}_test.npz
geometry/summary.csv and geometry/metrics.json… See the full description on the dataset page: https://huggingface.co/datasets/Newbeing/PACE-LLMRouterBench.category-classifier-supplement
Category Classifier Supplement Dataset
Supplementary training data for the Semantic Router category classifier model. This dataset is designed to be used alongside MMLU-Pro to improve classification accuracy, particularly for casual/non-academic queries.
Dataset Description
Purpose
The MMLU-Pro dataset contains academic questions across 14 categories, but its "other" category consists primarily of political science and journalism questions. This supplement adds… See the full description on the dataset page: https://huggingface.co/datasets/llm-semantic-router/category-classifier-supplement.halueval-spans
HaluEval Span-Level Dataset (LLM-Detected)
🔍 High-quality span-level hallucination detection dataset converted from HaluEval using Qwen2.5-72B-Instruct for precise span detection and RAGTruth-compatible labeling.
Quick Start
from datasets import load_dataset
dataset = load_dataset("llm-semantic-router/halueval-spans")
Why This Dataset?
Problem
Previous Solution
This Dataset
HaluEval has binary labels only
NLI-based conversion
✅ LLM-based span… See the full description on the dataset page: https://huggingface.co/datasets/llm-semantic-router/halueval-spans.llm_router_dataset-synth
'Make knowledge free for everyone'
halueval-llm-spans
HaluEval LLM Spans Dataset
A span-level hallucination detection dataset derived from HaluEval summarization data. Contains 10,000 samples with LLM-detected hallucination spans and RAGTruth-normalized prompts.
Dataset Description
This dataset converts HaluEval's binary hallucination labels into fine-grained span-level annotations using Qwen2.5-72B-Instruct. The prompts have been normalized to RAGTruth format for compatibility with hallucination detection models.… See the full description on the dataset page: https://huggingface.co/datasets/llm-semantic-router/halueval-llm-spans.halueval-spans-deberta
HaluEval Span-Level Dataset
🔍 Span-level hallucination detection dataset converted from HaluEval using DeBERTa-FEVER-ANLI NLI model.
Quick Start
from datasets import load_dataset
dataset = load_dataset("llm-semantic-router/halueval-spans-deberta")
Why This Dataset?
Problem
Solution
HaluEval has binary labels only
✅ Span-level annotations
Most hallucination datasets are imbalanced
✅ 45.8% hallucinated tokens
Token classifiers need character… See the full description on the dataset page: https://huggingface.co/datasets/llm-semantic-router/halueval-spans-deberta.halueval-spans-normalized
HaluEval Span-Level Dataset (RAGTruth-Normalized Prompts)
🔍 Span-level hallucination detection dataset with prompts normalized to match RAGTruth format for improved cross-dataset compatibility.
Quick Start
from datasets import load_dataset
dataset = load_dataset("llm-semantic-router/halueval-spans-normalized")
Why Normalized Prompts?
Training on mixed datasets with different prompt formats causes distribution shift:
Original Format
Normalized Format… See the full description on the dataset page: https://huggingface.co/datasets/llm-semantic-router/halueval-spans-normalized.llm-router-arena-codingdart-halspans
DART Hallucination Spans Dataset
A synthetic hallucination detection dataset derived from DART (Data-Record to Text) structured data. Contains 2,000 samples with LLM-generated responses and span-level hallucination annotations.
Dataset Description
This dataset was created to augment RAGTruth for Data2txt (structured data to text) task coverage. An LLM generates both faithful and intentionally hallucinated responses from DART's structured data triples, then annotates the… See the full description on the dataset page: https://huggingface.co/datasets/llm-semantic-router/dart-halspans.10k-LLM-Router-Traininge2e-halspans
E2E Hallucination Spans Dataset
A synthetic hallucination detection dataset derived from E2E NLG Challenge restaurant data. Contains 1,500 samples with LLM-generated responses and span-level hallucination annotations.
Dataset Description
This dataset was created to augment RAGTruth for Data2txt (structured data to text) task coverage. An LLM generates both faithful and intentionally hallucinated restaurant descriptions from E2E's meaning representations, then annotates… See the full description on the dataset page: https://huggingface.co/datasets/llm-semantic-router/e2e-halspans.prompt-response-llmrouterbenchllm_router_dataset-synth-pt
llm_router_dataset-synth-pt
A portuguese version of the llm_router_dataset-synth dataset, using Argos Translate.
llmrouterfft-multi-tool-router-datarouter_dataset-2llm-router-arena-coding-augmentedrouter_datasetllm-as-router-IT-budget-requests