datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
mmluMMLU (hendrycks_test on huggingface) without auxiliary train. It is much lighter (7MB vs 162MB) and faster than the original implementation, in which auxiliary train is loaded (+ duplicated!) by default for all the configs in the original version, making it quite heavy.
We use this version in tasksource.
Reference to original dataset:
Measuring Massive Multitask Language Understanding - https://github.com/hendrycks/test
@article{hendryckstest2021,
title={Measuring Massive Multitask Language… See the full description on the dataset page: https://huggingface.co/datasets/tasksource/mmlu.bigbenchBIG-Bench but it doesn't require the hellish dependencies (tensorflow, pypi-bigbench, protobuf) of the official version.
dataset = load_dataset("tasksource/bigbench",'movie_recommendation')
Code to reproduce:
https://colab.research.google.com/drive/1MKdLdF7oqrSQCeavAcsEnPdI85kD0LzU?usp=sharing
Datasets are capped to 50k examples to keep things light.
I also removed the default split when train was available also to save space, as default=train+val.
@article{srivastava2022beyond… See the full description on the dataset page: https://huggingface.co/datasets/tasksource/bigbench.docinsights-2026-shared-task-data
DocInsights 2026 Shared Task: DocSem
Document-grounded quantitative reasoning with evidence attribution
DocSem is the shared task of DocInsights 2026, the Workshop on Document Intelligence and Understanding co-located with EMNLP 2026 in Budapest, Hungary. The workshop theme is Beyond Plain Text: Bridging NLP and Document AI.
Workshop shared task | Source repository | Submission portal | Participant guide
Participants receive a PDF document and a paraphrased user_query. Systems… See the full description on the dataset page: https://huggingface.co/datasets/amitbcp/docinsights-2026-shared-task-data.Nemotron-Terminal-Synthetic-Tasks
Terminal-Corpus: Task Structure Specification
This repository contains the skill-based synthetic tasks within the Terminal-Corpus. These tasks are designed to evaluate and train autonomous agents in realistic Linux terminal environments.
🏗️ Task Anatomy
Each task is contained within a dedicated directory and follows a strict four-component architecture:
1. Instruction (instruction.md)
Purpose: Provides the natural language description of the objective.… See the full description on the dataset page: https://huggingface.co/datasets/nvidia/Nemotron-Terminal-Synthetic-Tasks.kilt_tasks
Dataset Card for KILT
Dataset Summary
KILT has been built from 11 datasets representing 5 types of tasks:
Fact-checking
Entity linking
Slot filling
Open domain QA
Dialog generation
All these datasets have been grounded in a single pre-processed Wikipedia dump, allowing for fairer and more consistent evaluation as well as enabling new task setups such as multitask and transfer learning with minimal effort. KILT also provides tools to analyze and understand the… See the full description on the dataset page: https://huggingface.co/datasets/facebook/kilt_tasks.finance-tasks
Adapting LLMs to Domains via Continual Pre-Training (ICLR 2024)
This repo contains the evaluation datasets for our paper Adapting Large Language Models via Reading Comprehension.
We explore continued pre-training on domain-specific corpora for large language models. While this approach enriches LLMs with domain knowledge, it significantly hurts their prompting ability for question answering. Inspired by human learning via reading comprehension, we propose a simple method to… See the full description on the dataset page: https://huggingface.co/datasets/AdaptLLM/finance-tasks.prontoqahttps://github.com/asaparov/prontoqa/
@article{saparov2022language,
title={Language models are greedy reasoners: A systematic formal analysis of chain-of-thought},
author={Saparov, Abulhair and He, He},
journal={arXiv preprint arXiv:2210.01240},
year={2022}
}
lawma-tasks
Lawma legal classification tasks
This repository contains the legal classification tasks from Lawma.
These tasks were derived from the Supreme Court and Songer Court of Appeals databases.
See the project's GitHub repository for more details.
Please cite as:
@misc{dominguezolmedo2024lawmapowerspecializationlegal,
title={Lawma: The Power of Specialization for Legal Tasks},
author={Ricardo Dominguez-Olmedo and Vedant Nanda and Rediet Abebe and Stefan Bechtold and Christoph… See the full description on the dataset page: https://huggingface.co/datasets/ricdomolm/lawma-tasks.SkillFlow-Task
Dataset Card for SkillFlow Test Tasks
Dataset Summary
SkillFlow Test Tasks is the task repository used in the SkillFlow benchmark for evaluating lifelong skill discovery, skill revision, and cross-task procedural transfer in autonomous agents.
The dataset contains 166 runnable tasks organized into 20 workflow families spanning five broad domains:
Finance & Economics
Operations & Supply Chain
Healthcare & Life Sciences
Governance & Strategy
Data & Document Intelligence… See the full description on the dataset page: https://huggingface.co/datasets/zhang-ziao/SkillFlow-Task.race-cRace-C : additional data for race (high school/middle school) but for college level
https://github.com/mrcdata/race-c
@InProceedings{pmlr-v101-liang19a,
title={A New Multi-choice Reading Comprehension Dataset for Curriculum Learning},
author={Liang, Yichan and Li, Jianheng and Yin, Jian},
booktitle={Proceedings of The Eleventh Asian Conference on Machine Learning},
pages={742--757},
year={2019}
}
commonsense_qa_2.0https://github.com/allenai/csqa2
@article{talmor2022commonsenseqa,
title={CommonsenseQA 2.0: Exposing the limits of AI through gamification},
author={Talmor, Alon and Yoran, Ori and Bras, Ronan Le and Bhagavatula, Chandra and Goldberg, Yoav and Choi, Yejin and Berant, Jonathan},
journal={arXiv preprint arXiv:2201.05320},
year={2022}
}
Math-RL-Tasks
Ulam AI Math RL Tasks
Forty original, verifier-backed mathematical reasoning tasks packaged as ten
independent RL environments. The collection spans advanced graduate exercises,
research-style exact computation and structural generalization problems in
algebraic geometry, arithmetic geometry, combinatorics, topology, probability
and spectral analysis.
Each suite pairs a runnable rl_env/ with a preserved blind_run/ by
GPT-5.6 Sol Pro. The model name describes the evaluation actor… See the full description on the dataset page: https://huggingface.co/datasets/ulamai/Math-RL-Tasks.osworld_tasks_filesAudio-Video-Engineering-Agentic-Tasks-1M
Audio/Video Engineering Agentic Tasks (1M)
Abstract
A highly specialized dataset comprising 1,029,459 in-context troubleshooting prompts and execution commands built for the deepest levels of media production. Unlike standard datasets that simulate clean, theoretical instructions, this matrix captures the chaotic, highly-detailed, and conversational reality of professional audio engineers, composers, and video editors mid-session. It is engineered to train multimodal AI… See the full description on the dataset page: https://huggingface.co/datasets/yatin-superintelligence/Audio-Video-Engineering-Agentic-Tasks-1M.babi_tasks
bAbI Dataset : Per-Task Splits
The examples in this dataset were obtained from "Muennighoff/babi". This dataset offer the same examples but it can be loaded using a per-task basis:
from datasets import load_dataset
ds = load_dataset('RawthiL/babi_tasks', 'task_1-single_supporting_fact')
The available tasks are:
Task ID
Task Name
Split Name
1
Single Supporting Fact
task_1-single_supporting_fact
2
Two Supporting Facts
task_2-two_supporting_facts
3
Three Supporting… See the full description on the dataset page: https://huggingface.co/datasets/RawthiL/babi_tasks.ecqa
Dataset Card for "ecqa"
https://github.com/dair-iitd/ECQA-Dataset
@inproceedings{aggarwaletal2021ecqa,
title={{E}xplanations for {C}ommonsense{QA}: {N}ew {D}ataset and {M}odels},
author={Shourya Aggarwal and Divyanshu Mandowara and Vishwajeet Agrawal and Dinesh Khandelwal and Parag Singla and Dinesh Garg},
booktitle="Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing… See the full description on the dataset page: https://huggingface.co/datasets/tasksource/ecqa.Creative-Professionals-Agentic-Tasks-1M
Creative Professionals Agentic Tasks (1M)
Abstract
A massive-scale, high-fidelity synthetic task dataset comprising 1,070,917 agentic command operations across 36 creative, technical, and engineering software environments. This dataset is engineered exclusively to stress-test, evaluate, and fine-tune multimodal AI agents designed for Agent Environment operation, complex software interaction, and multi-step reasoning within deep software infrastructures.… See the full description on the dataset page: https://huggingface.co/datasets/yatin-superintelligence/Creative-Professionals-Agentic-Tasks-1M.medicine-tasks
Adapting LLMs to Domains via Continual Pre-Training (ICLR 2024)
This repo contains the evaluation datasets for our paper Adapting Large Language Models via Reading Comprehension.
We explore continued pre-training on domain-specific corpora for large language models. While this approach enriches LLMs with domain knowledge, it significantly hurts their prompting ability for question answering. Inspired by human learning via reading comprehension, we propose a simple method to… See the full description on the dataset page: https://huggingface.co/datasets/AdaptLLM/medicine-tasks.Creative-Professionals-Agentic-Tasks-1M
Creative Professionals Agentic Tasks (1M)
Abstract
A massive-scale, high-fidelity synthetic task dataset comprising 1,070,917 agentic command operations across 36 creative, technical, and engineering software environments. This dataset is engineered exclusively to stress-test, evaluate, and fine-tune multimodal AI agents designed for Agent Environment operation, complex software interaction, and multi-step reasoning within deep software infrastructures.… See the full description on the dataset page: https://huggingface.co/datasets/rAVEUK/Creative-Professionals-Agentic-Tasks-1M.FOL-nli
Dataset Card for "FOL-nli"
https://github.com/sileod/unigram/
https://arxiv.org/abs/2406.11035
Citation:
@article{sileo2024scaling,
title={Scaling Synthetic Logical Reasoning Datasets with Context-Sensitive Declarative Grammars},
author={Sileo, Damien},
journal={arXiv preprint arXiv:2406.11035},
year={2024}
}
DLLM-Planing-Task
DLLM-Planning-Task
Benchmark datasets for evaluating planning capabilities of Diffusion Language Models (DLLMs).
Dataset Description
This dataset contains multiple planning and combinatorial reasoning tasks designed to evaluate discrete diffusion language models. Each task has train/test splits in either CSV or JSONL format.
Tasks
Task
Format
Description
Sudoku
CSV
9x9 Sudoku puzzles. Columns: quizzes, solutions.
Path Finding (path-2-6)… See the full description on the dataset page: https://huggingface.co/datasets/zeyuzy/DLLM-Planing-Task.Creative-Professionals-Agentic-Tasks-1M
Creative Professionals Agentic Tasks (1M)
Abstract
A massive-scale, high-fidelity synthetic task dataset comprising 1,070,917 agentic command operations across 36 creative, technical, and engineering software environments. This dataset is engineered exclusively to stress-test, evaluate, and fine-tune multimodal AI agents designed for Agent Environment operation, complex software interaction, and multi-step reasoning within deep software infrastructures.… See the full description on the dataset page: https://huggingface.co/datasets/kryp1234/Creative-Professionals-Agentic-Tasks-1M.law-tasks
Adapting LLMs to Domains via Continual Pre-Training (ICLR 2024)
This repo contains the evaluation datasets for our paper Adapting Large Language Models via Reading Comprehension.
We explore continued pre-training on domain-specific corpora for large language models. While this approach enriches LLMs with domain knowledge, it significantly hurts their prompting ability for question answering. Inspired by human learning via reading comprehension, we propose a simple method to… See the full description on the dataset page: https://huggingface.co/datasets/AdaptLLM/law-tasks.MTRAG_taskA_results
MTRAG Task A: Multi-Turn RAG Retrieval Results
Complete experimental results and datasets for Multi-Turn Retrieval-Augmented Generation (RAG) research.
📊 Overview
This repository contains:
Experimental Results: 20 hybrid retrieval configurations + baselines (8.3 GB)
Raw Data: Passages, queries, rewrites, conversations (2.5 GB)
Documentation: Statistical validation, analysis, LaTeX presentation
Configurations: Reproducible experiment configs
🏆 Key Results… See the full description on the dataset page: https://huggingface.co/datasets/vania-janet/MTRAG_taskA_results.Reverse-alpha-suppression-task-boostOriginal-alpha-suppression-task-boosttaskstreamTaskStream is a comprehensive dataset of enterprise business workflows, decision-making processes,
organizational structures, and operational documentation across multiple industries.task-alignment-datasetRelease version: (2026-07-16)
Three benchmarks for evaluating LLM task alignment under underspecification.
Each row is one task specification: the assistant must interact to identify the user's
ground-truth task x* from a fixed set of 15 candidate specifications, given
only an evolving natural-language intent from a user simulator.
Files
Dataset
File
Rows
GDPVal (knowledge-work tasks)
gdpval_v4_camera_ready_n88.csv
88
Terminal-Bench (coding tasks)… See the full description on the dataset page: https://huggingface.co/datasets/daiandy/task-alignment-dataset.k12-standards-instruction-tasks
K-12 Curriculum Tasks (generated)
2,489 generated instruction/input/output records covering five curriculum tasks:
assessment creation, learning objective generation, misconception detection, standard
explanation, and standards Q&A. Content is predominantly mathematics.
Important: the name is misleading
Despite the name, this dataset contains no school directory data. There are four
columns - task, input, output, metadata - and no staff, principal, or school… See the full description on the dataset page: https://huggingface.co/datasets/robworks-software/k12-standards-instruction-tasks.lexenvs-tasks
LexEnvs — Credit Card Optimization Tasks
A dataset of 164 evaluation tasks for training and benchmarking RL agents on credit card optimization. Each task presents a user scenario with spending patterns, constraints, and preferences, and asks the agent to recommend optimal credit cards with expected value (EV) calculations.
Dataset Description
This dataset is the task suite for the LexEnvs Harbor RL Environment, a stateless evaluation server that scores agent responses on… See the full description on the dataset page: https://huggingface.co/datasets/endishai/lexenvs-tasks.