llm-evaluation
llm-refusal-evaluation
🛡️ LLM Refusal Evaluation Benchmark
This repository contains the benchmarks used in the LLM-Refusal-Evaluation suite.
The prompts are organized into three groups:
Safety Benchmarks — harmful / jailbreak-style prompts that models should refuse.
Chinese Sensitive Topics — prompts that may be censored by China-aligned models.
Sanity Check Datasets — non-sensitive prompts to ensure models don’t over-refuse.
📌 Contents
Safety Benchmarks
JailbreakBench
SorryBench… See the full description on the dataset page: https://huggingface.co/datasets/MultiverseComputingCAI/llm-refusal-evaluation.llm-refusal-evaluation
🛡️ LLM Refusal Evaluation Benchmark
This repository contains the benchmarks used in the LLM-Refusal-Evaluation suite.
The prompts are organized into three groups:
Safety Benchmarks — harmful / jailbreak-style prompts that models should refuse.
Chinese Sensitive Topics — prompts that may be censored by China-aligned models.
Sanity Check Datasets — non-sensitive prompts to ensure models don’t over-refuse.
📌 Contents
Safety Benchmarks
JailbreakBench
SorryBench… See the full description on the dataset page: https://huggingface.co/datasets/BushNate/llm-refusal-evaluation.AutoRAG-evaluation-2024-LLM-paper-v1
AutoRAG evaluation dataset
Made with 2024 LLM resesarch articles (papers)
This dataset is an example for AutoRAG.
You can directly use this dataset for optimizng and benchmarking your RAG setup in AutoRAG.
How this dataset created?
This dataset is 100% synthetically generated by GPT-4 and Marker Inc. technology.
At first, we collected 110 latest LLM papers at arxiv.
We used Marker OCR model to extract texts.
And chunk it using MarkdownSplitter and… See the full description on the dataset page: https://huggingface.co/datasets/MarkrAI/AutoRAG-evaluation-2024-LLM-paper-v1.llm-output-evaluation
LLM_OUTPUT_EVALUATION
A preference dataset for LLM_OUTPUT_EVALUATION, harvested from real, human-labelled sources and curated by an automated harvesting harness with an LLM quality gate.
Format
Standard preference / DPO schema — each row:
column
meaning
prompt
the request (originally prompt)
chosen
the human-preferred response
rejected
a worse response to the same prompt
source
the dataset/URL the row was harvested from
Splits… See the full description on the dataset page: https://huggingface.co/datasets/316usman/llm-output-evaluation.llm-math-evaluation-dataset
LLM Math Response Evaluation Dataset
Dataset Summary
A human-annotated dataset of 150 AI-generated math responses
evaluated across GPT-4o, Claude, and Gemini. Each response is
scored on Correctness, Reasoning, and Clarity using a structured
rubric, with written justification for every score.
Supported Tasks
LLM evaluation and benchmarking
Math reasoning quality assessment
Error type classification in AI responses
Dataset Structure… See the full description on the dataset page: https://huggingface.co/datasets/nbvbharath-1729/llm-math-evaluation-dataset.LLM-Covid-19-Cutoff-Evaluation
What this is (TL;DR)
A reproducible evaluation that measures how a model’s training-data cutoff affects its ability to answer domain-dense, high-importance medical questions derived from changes between two versions of the same IDSA COVID-19 guideline.
Summary
We extracted paragraph-level differences between an older IDSA Covid-19 guideline snapshot (pre-update, v5.0.0) and a newer snapshot (post-update, v11.0.0), converted the new/changed text units into… See the full description on the dataset page: https://huggingface.co/datasets/michaelcacioli/LLM-Covid-19-Cutoff-Evaluation.