datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
mmlu
Dataset Card for MMLU
Dataset Summary
Measuring Massive Multitask Language Understanding by Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt (ICLR 2021).
This is a massive multitask test consisting of multiple-choice questions from various branches of knowledge. The test spans subjects in the humanities, social sciences, hard sciences, and other areas that are important for some people to learn. This covers 57… See the full description on the dataset page: https://huggingface.co/datasets/cais/mmlu.MMLU-Pro
MMLU-Pro Dataset
MMLU-Pro dataset is a more robust and challenging massive multi-task understanding dataset tailored to more rigorously benchmark large language models' capabilities. This dataset contains 12K complex questions across various disciplines.
|Github | 🏆Leaderboard | 📖Paper |
🚀 What's New
[2026.03.11] Added more cutting-edge frontier models to the leaderboard, including the Claude-4.6 series, Seed2.0 series, Qwen3.5 series, and Gemini-3.1-Pro… See the full description on the dataset page: https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro.mmlu_no_trainThis dataset contains a copy of the cais/mmlu HF dataset but without the auxiliary_train split that takes a long time to generate again each time when loading multiple subsets of the dataset.
Please visit https://huggingface.co/datasets/cais/mmlu for more information on the MMLU dataset.
mmluMMLU (hendrycks_test on huggingface) without auxiliary train. It is much lighter (7MB vs 162MB) and faster than the original implementation, in which auxiliary train is loaded (+ duplicated!) by default for all the configs in the original version, making it quite heavy.
We use this version in tasksource.
Reference to original dataset:
Measuring Massive Multitask Language Understanding - https://github.com/hendrycks/test
@article{hendryckstest2021,
title={Measuring Massive Multitask Language… See the full description on the dataset page: https://huggingface.co/datasets/tasksource/mmlu.mmlu-redux-2.0
Dataset Card for MMLU-Redux-2.0
MMLU-Redux is a subset of 5,700 manually re-annotated questions across 57 MMLU subjects.
News
[2025.02.25] We corrected one annotation in Abstract Algebra subset, as noted in the Issue #2.
[2025.02.08] We corrected one annotation in High School Mathematics subset, as noted in the PlatinumBench paper.
[2025.01.23] MMLU-Redux is accepted to NAACL 2025!
Dataset Details
Dataset Description
Each data point in… See the full description on the dataset page: https://huggingface.co/datasets/edinburgh-dawg/mmlu-redux-2.0.mmlu-prox-eval-predictions
MMLU-ProX Multilingual Model Predictions
Raw per-sample model predictions on MMLU-ProX
across 29 languages and 25 open-weight LLMs, produced with
lm-evaluation-harness.
This dataset releases the full prediction logs (not just aggregate scores) so that
item-level responses can be re-analysed — e.g. for Item Response Theory (IRT) modelling
of multilingual benchmarks, error analysis, or per-item difficulty estimation.
Repository structure
mmlu_prox_<lang>/
└──… See the full description on the dataset page: https://huggingface.co/datasets/gililior/mmlu-prox-eval-predictions.mmlu
Dataset Card for MMLU
Dataset Summary
Measuring Massive Multitask Language Understanding by Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt (ICLR 2021).
This is a massive multitask test consisting of multiple-choice questions from various branches of knowledge. The test spans subjects in the humanities, social sciences, hard sciences, and other areas that are important for some people to learn. This covers 57 tasks… See the full description on the dataset page: https://huggingface.co/datasets/lighteval/mmlu.mmlu-redux-2.0-okThis dataset is filtered from https://huggingface.co/datasets/edinburgh-dawg/mmlu-redux-2.0 using:
from datasets import load_dataset
from pathlib import Path
CONFIGS = ['abstract_algebra', 'anatomy', 'astronomy', 'business_ethics', 'clinical_knowledge', 'college_biology', 'college_chemistry', 'college_computer_science', 'college_mathematics', 'college_medicine', 'college_physics', 'computer_security', 'conceptual_physics', 'econometrics', 'electrical_engineering', 'elementary_mathematics'… See the full description on the dataset page: https://huggingface.co/datasets/fxmarty/mmlu-redux-2.0-ok.PromptEval_MMLU_full
MMLU Multi-Prompt Evaluation Data
Overview
This dataset contains the results of a comprehensive evaluation of various Large Language Models (LLMs) using multiple prompt templates on the Massive Multitask Language Understanding (MMLU) benchmark. The data is introduced in
Maia Polo, Felipe, Ronald Xu, Lucas Weber, Mírian Silva, Onkar Bhardwaj, Leshem Choshen, Allysson Flavio Melo de Oliveira, Yuekai Sun, and Mikhail Yurochkin. "Efficient multi-prompt evaluation of LLMs."… See the full description on the dataset page: https://huggingface.co/datasets/PromptEval/PromptEval_MMLU_full.PromptEval_MMLU_correctness
MMLU Multi-Prompt Evaluation Data (correctness scores)
Overview
This dataset contains the results of a comprehensive evaluation of various Large Language Models (LLMs) using multiple prompt templates on the Massive Multitask Language Understanding (MMLU) benchmark. The data is introduced in
Maia Polo, Felipe, Ronald Xu, Lucas Weber, Mírian Silva, Onkar Bhardwaj, Leshem Choshen, Allysson Flavio Melo de Oliveira, Yuekai Sun, and Mikhail Yurochkin. "Efficient multi-prompt… See the full description on the dataset page: https://huggingface.co/datasets/PromptEval/PromptEval_MMLU_correctness.m_mmlu
Multilingual MMLU
Dataset Summary
This dataset is a machine translated version of the MMLU dataset.
The Icelandic (is) part was translated with Miðeind's Greynir model and Norwegian (nb) was translated with DeepL. The rest of the languages was translated using GPT-3.5-turbo by the University of Oregon, and this part of the dataset was originally uploaded to this Github repository.
mmluThis is a massive multitask test consisting of multiple-choice questions from various branches of knowledge, covering 57 tasks including elementary mathematics, US history, computer science, law, and more.mmlu-redux
Dataset Card for MMLU-Redux
[!TIP]
Please consider using MMLU-Redux-2.0 which contains all 57 MMLU subjects.
MMLU-Redux is a subset of 3,000 manually re-annotated questions across 30 MMLU subjects.
News
[2025.02.08] We corrected one annotation in High School Mathematics subset, as noted in the PlatinumBench paper.
[2025.01.23] MMLU-Redux is accepted to NAACL 2025!
Dataset Details
Dataset Description
Each data point in MMLU-Redux contains… See the full description on the dataset page: https://huggingface.co/datasets/edinburgh-dawg/mmlu-redux.MMLU-SR
MMLU-SR Dataset
This is the dataset for the paper "MMLU-SR: A Benchmark for Stress-Testing Reasoning Capability of Large Language Models".
Dataset Structure
This dataset contains three different variants:
Question Only: Key terms in questions are replaced with dummy words and their definitions, while answer choices remain unchanged.
Answer Only: Key terms in answer choices are replaced with dummy words and their definitions, while questions remain unchanged.
Question… See the full description on the dataset page: https://huggingface.co/datasets/NiniCat/MMLU-SR.mmlu_tr-v0.2
Dataset Card for mmlu_tr-v0.2
Overview
malhajar/mmlu_tr-v0.2 is an enhanced version of the original mmlu-tr dataset, specifically developed for use in the OpenLLMTurkishLeaderboard v0.2. This iteration of the dataset has been translated into Turkish using advanced language models like GPT-4, with English text provided for cross-checking to ensure accuracy and reliability. The dataset is tailored to assist in evaluating the performance of Turkish language models (LLMs) and… See the full description on the dataset page: https://huggingface.co/datasets/malhajar/mmlu_tr-v0.2.mmlu-cs
Czech MMLU
This is a Czech translation of the original MMLU dataset, created using the WMT 21 En-X model.
The 'auxiliary_train' subset is not included.
The translation was completed for use within the Czech-Bench evaluation framework.
The script used for translation can be reviewed here.
Citation
Original dataset:
@article{hendryckstest2021,
title={Measuring Massive Multitask Language Understanding},
author={Dan Hendrycks and Collin Burns and Steven Basart and… See the full description on the dataset page: https://huggingface.co/datasets/CIIRC-NLP/mmlu-cs.mmlu_preferencesreformat of MMLU to be in DPO (paired) format
examples:
{'prompt': 'Which of the following statements about the lanthanide elements is NOT true?', 'chosen': 'The atomic radii of the lanthanide elements increase across the period from La to Lu.', 'rejected': 'All of the lanthanide elements react with aqueous acid to liberate hydrogen.'}
college_chemistry
{'prompt': 'Beyond the business case for engaging in CSR there are a number of moral arguments relating to: negative _______, the… See the full description on the dataset page: https://huggingface.co/datasets/wassname/mmlu_preferences.MMLU-CF
MMLU-CF: A Contamination-free Multi-task Language Understanding Benchmark
[📜 Paper] •
[🤗 HF Dataset] •
[🐱 GitHub]
MMLU-CF is a contamination-free and more challenging multiple-choice question benchmark. This dataset contains 10K questions each for the validation set and test set, covering various disciplines.
1. The Motivation of MMLU-CF
The open-source nature of these benchmarks and the broad sources of training data for LLMs have inevitably led to… See the full description on the dataset page: https://huggingface.co/datasets/microsoft/MMLU-CF.mmlu_ru
MMLU in Russian (Massive Multitask Language Understanding)
Overview of the Dataset
MMLU dataset for EN/RU, without auxiliary train.
The dataset contains dev/val/test splits for both, English and Russian languages.
Note it doesn't include auxiliary_train split, which wasn't translated.
Totally the dataset has ~16k samples per language: 285 dev, 1531 val, 14042 test.
Description of original MMLU
MMLU dataset covers 57 different tasks.
Each task requires to choose… See the full description on the dataset page: https://huggingface.co/datasets/NLPCoreTeam/mmlu_ru.MMLU_ru
MMLU (subjects as subsets)
Script-less dataset with one subset per subject; splits normalized to validation and test.
Columns: question_<lang>, choices_<lang>, answer (int index).
mmlu-indic
Indic MMLU Dataset
A multilingual version of the Massive Multitask Language Understanding (MMLU) benchmark, translated from English into 10 Indian languages.
This version contains the translations of the development and test sets only.
Languages Covered
The dataset includes translations in the following languages:
Bengali (bn)
Gujarati (gu)
Hindi (hi)
Kannada (kn)
Marathi (mr)
Malayalam (ml)
Oriya (or)
Punjabi (pa)
Tamil (ta)
Telugu (te)
Task Format
Each… See the full description on the dataset page: https://huggingface.co/datasets/sarvamai/mmlu-indic.mmlu
Dataset Card for MMLU
Dataset Summary
Measuring Massive Multitask Language Understanding by Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt (ICLR 2021).
This is a massive multitask test consisting of multiple-choice questions from various branches of knowledge. The test spans subjects in the humanities, social sciences, hard sciences, and other areas that are important for some people to learn. This covers 57 tasks… See the full description on the dataset page: https://huggingface.co/datasets/flunardelli/mmlu.EU-MMLU
EU-oriented Massive Multitask Language Understanding (EU-MMLU)
Dataset description
The MMLU benchmarking dataset (https://arxiv.org/pdf/2009.03300) is a widely recognized benchmark of general knowledge attained by AI models. It covers a broad range of topics from 57 different categories, covering elementary-level knowledge up to advanced professional subjects like law, physics, history, and computer science.
The European Commission’s Directorate-General for… See the full description on the dataset page: https://huggingface.co/datasets/EC-DGT-AI/EU-MMLU.mmlu-okapi-eval-es
MMLU translated to Spanish
This dataset was generated by the Natural Language Processing Group of the University of Oregon, where they used the
original MMLU dataset in English and translated it into different languages using ChatGPT.
This dataset only contains the Spanish translation, but the following languages are also covered within the original
subsets posted by the University of Oregon at http://nlp.uoregon.edu/download/okapi-eval/datasets/.
Disclaimer
All the… See the full description on the dataset page: https://huggingface.co/datasets/alvarobartt/mmlu-okapi-eval-es.mmlu-redux-2.0-for-llama.cppMMLU-redux-v2.0 converted for the llama.cpp perplexity multiple choice tool.
Only valid entries where kept, there is no error based prompting included.
Dataset Card for MMLU-Redux-2.0
MMLU-Redux is a subset of 5,700 manually re-annotated questions across 57 MMLU subjects.
Citation
BibTeX:
@misc{gema2024mmlu,
title={Are We Done with MMLU?},
author={Aryo Pradipta Gema and Joshua Ong Jun Leang and Giwon Hong and Alessio Devoto and Alberto Carlo Maria… See the full description on the dataset page: https://huggingface.co/datasets/Green-Sky/mmlu-redux-2.0-for-llama.cpp.MMLU-medical-cot-llama31
MMLU-medical-cot
Synthetically enhanced responses to the medical-related questions of the auxiliary train set of the MMLU dataset. Used to train Aloe-Beta model.
Dataset Details
Dataset Description
First, we use Llama-3.1-70B-Instruct to filter the medical-related questions of the auxiliary train set of the MMLU dataset. Next, we leverage Mixtral-8x7B to… See the full description on the dataset page: https://huggingface.co/datasets/HPAI-BSC/MMLU-medical-cot-llama31.mmlu-trThis Dataset is part of a series of datasets aimed at advancing Turkish LLM Developments by establishing rigid Turkish benchmarks to evaluate the performance of LLM's Produced in the Turkish Language.
Dataset Card for mmlu-tr
malhajar/mmlu-tr is a translated version of mmlu aimed specifically to be used in the OpenLLMTurkishLeaderboard
MMLU (hendrycks_test on huggingface) without auxiliary train. It is much lighter (7MB vs 162MB) and faster than the original implementation, in… See the full description on the dataset page: https://huggingface.co/datasets/malhajar/mmlu-tr.mmlu_ita
Italian Version of the MMLU DATASET
Based on the version released by: FreedomIntelligence/MMLU_Italian
Includes minor fixes.
Citations
This version:
@misc{basile2023llamantino,
title={LLaMAntino: LLaMA 2 Models for Effective Text Generation in Italian Language},
author={Pierpaolo Basile and Elio Musacchio and Marco Polignano and Lucia Siciliani and Giuseppe Fiameni and Giovanni Semeraro},
year={2023},
eprint={2312.09993}… See the full description on the dataset page: https://huggingface.co/datasets/swap-uniba/mmlu_ita.mmlu-winogrande-afr
Bridging the Gap: Enhancing LLM Performance for Low-Resource African Languages with New Benchmarks, Fine-Tuning, and Cultural Adjustments
Authors:
Tuka Alhanai tuka@ghamut.com, Adam Kasumovic adam.kasumovic@ghamut.com, Mohammad Ghassemi ghassemi@ghamut.com, Aven Zitzelberger aven.zitzelberger@ghamut.com, Jessica Lundin jessica.lundin@gatesfoundation.org, Guillaume Chabot-Couture Guillaume.Chabot-Couture@gatesfoundation.org
This HuggingFace Dataset contains the human-translated… See the full description on the dataset page: https://huggingface.co/datasets/Institute-Disease-Modeling/mmlu-winogrande-afr.mmlu-auxiliary-train-10-choices
Dataset Card for Augmented MMLU (STEM) with Additional Distractors
Dataset Description
This dataset is an augmented version of the STEM portion of the MMLU auxiliary training set kz919/mmlu-auxiliary-train-auto-labelled, where each original 4-option multiple-choice question has been expanded to include 10 options (A-J) through the addition of six carefully constructed distractors.
Dataset Summary
Original Dataset: MMLU auxiliary training set (STEM portion)… See the full description on the dataset page: https://huggingface.co/datasets/andresnowak/mmlu-auxiliary-train-10-choices.