datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
fineweb-edu-translated
Helsinki-NLP/fineweb-edu-translated
fineweb-edu-tanslated is a collection of automatically translated documents from fineweb-edu.
Translations are based on OPUS-MT and HPLT-MT models.
The data in v1.0 covers 36,704,000 documents with over 28 billion space-searated tokens of English data translated into 36 languages.
The total v1.0 data set includes over 960 billion tokens and the translated documents are aligned across all languages.
In the v1.1 release, additional translations… See the full description on the dataset page: https://huggingface.co/datasets/Helsinki-NLP/fineweb-edu-translated.Calc-mawps
Dataset Card for Calc-MAWPS
Summary
The dataset is a collection of simple math word problems focused on arithmetics. It is derived from https://huggingface.co/datasets/omarxadel/MaWPS-ar.
The main addition in this dataset variant is the chain column. It was created by converting the solution to a simple html-like language that can be easily
parsed (e.g. by BeautifulSoup). The data contains 3 types of tags:
gadget: A tag whose content is intended to be evaluated by… See the full description on the dataset page: https://huggingface.co/datasets/MU-NLPC/Calc-mawps.nemotron-cc-translated
Helsinki-NLP/nemotron-cc-translated
nemotron-cc-tanslated is a collection of automatically translated documents from nemotron-cc taken out of the high-quality subset.
Translations are based on OPUS-MT and HPLT-MT models.
The data in v1.0 covers 156,431,999 documents with over 70 billion space-searated tokens of English data translated into 36 languages.
The total v1.0 data set includes over 2.4 trillion tokens and the translated documents are aligned across all languages.
v1.1… See the full description on the dataset page: https://huggingface.co/datasets/Helsinki-NLP/nemotron-cc-translated.PUPAThis dataset contains the data presented in the paper PAPILLON: Privacy Preservation from Internet-based and Local Language Model Ensembles.
Code: https://github.com/siyan-sylvia-li/PAPILLON
SWE-chat
SWE-chat: Coding Agent Interactions From Real Users in the Wild
📄 Paper: arxiv.org/abs/2604.20779
🌐 Website: swe-chat.com
Dataset Summary
SWE-chat captures real-world AI coding sessions from developers using AI coding assistants (Claude Code, Codex, Gemini CLI, and others via the Entire.io CLI). Each session includes the full conversation transcript, tool calls, thinking traces, code changes, and attribution of human vs. agent-authored code.
Dataset Size… See the full description on the dataset page: https://huggingface.co/datasets/SALT-NLP/SWE-chat.tatoeba_mt
Dataset Card for [Dataset Name]
Dataset Summary
The Tatoeba Translation Challenge is a multilingual data set of machine translation benchmarks derived from user-contributed translations collected by Tatoeba.org and provided as parallel corpus from OPUS. This dataset includes test and development data sorted by language pair. It includes test sets for hundreds of language pairs and is continuously updated. Please, check the version number tag to refer to the… See the full description on the dataset page: https://huggingface.co/datasets/Helsinki-NLP/tatoeba_mt.McEvalMcEval benchmark data as described in the McEval Paper. Code for the evaluation can be found on Github as McEval.
MedThinkVQA
MedThinkVQA
MedThinkVQA is an expert-annotated benchmark for multi-image diagnostic reasoning in radiology. Unlike prior medical VQA benchmarks that typically contain at most one image per case, MedThinkVQA requires models to extract evidence from each image, integrate cross-view information, and perform differential-diagnosis reasoning.
Links
GitHub: https://github.com/benluwang/MedThinkVQA
Leaderboard: https://benluwang.github.io/MedThinkVQA/
Submission Guide:… See the full description on the dataset page: https://huggingface.co/datasets/bio-nlp-umass/MedThinkVQA.german-commons
German Commons - 154 Billion Tokens of Openly Licensed Text for German Language Models
A comprehensive collection of German-language text data under open licenses for training German language models.
Datasheet: DATASHEET.md.
Paper: arxiv.org/abs/2510.13996
Code: github.com/coral-nlp/llmdata
Bloom Filter (DOLMA-compatible): bloom_filter.bin
Dataset Description
This dataset is aggregated from 41 diverse sources and contains 154.56 billion tokensof German text data with… See the full description on the dataset page: https://huggingface.co/datasets/coral-nlp/german-commons.FaithDialFaithDial is a new benchmark for hallucination-free dialogues, created by manually editing hallucinated and uncooperative responses in Wizard of Wikipedia.Calc-svamp
Dataset Card for Calc-SVAMP
Summary
The dataset is a collection of simple math word problems focused on arithmetics. It is derived from https://github.com/arkilpatel/SVAMP/.
The main addition in this dataset variant is the chain column. It was created by converting the solution to a simple html-like language that can be easily
parsed (e.g. by BeautifulSoup). The data contains 3 types of tags:
gadget: A tag whose content is intended to be evaluated by calling an external… See the full description on the dataset page: https://huggingface.co/datasets/MU-NLPC/Calc-svamp.Calc-gsm8k
Dataset Card for Calc-gsm8k
Summary
This dataset is an instance of gsm8k dataset, converted to a simple html-like language that can be easily parsed (e.g. by BeautifulSoup). The data contains 3 types of tags:
gadget: A tag whose content is intended to be evaluated by calling an external tool (sympy-based calculator in this case)
output: An output of the external tool
result: The final answer to the mathematical problem (a number)
Supported Tasks
The… See the full description on the dataset page: https://huggingface.co/datasets/MU-NLPC/Calc-gsm8k.WebLINX
WebLINX: Real-World Website Navigation with Multi-Turn Dialogue
Xing Han Lù*, Zdeněk Kasner*, Siva Reddy
💾Code
📄Paper
🌐Website
📓Colab
🤖Models💻Explorer
🐦Tweets
🏆Leaderboard
Your browser does not support the video tag.
[!IMPORTANT]
WebLINX is now available as a benchmark through BrowserGym, allowing you to access demonstration steps in the same way you would access a web agent environment like WebArena or MiniWoB. This also allows you to run agents… See the full description on the dataset page: https://huggingface.co/datasets/McGill-NLP/WebLINX.dart-math-uniform
🎯 DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solving
📝 Paper@arXiv | 🤗 Datasets&Models@HF | 🐱 Code@GitHub
🐦 Thread@X(Twitter) | 🐶 中文博客@知乎 | 📊 Leaderboard@PapersWithCode | 📑 BibTeX
Datasets: DART-Math
DART-Math datasets are the state-of-the-art and data-efficientopen-source instruction tuning datasets for mathematical reasoning.
Figure 1: Left: Average accuracy on 6 mathematical benchmarks. We compare with models… See the full description on the dataset page: https://huggingface.co/datasets/hkust-nlp/dart-math-uniform.dart-math-hard
🎯 DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solving
📝 Paper@arXiv | 🤗 Datasets&Models@HF | 🐱 Code@GitHub
🐦 Thread@X(Twitter) | 🐶 中文博客@知乎 | 📊 Leaderboard@PapersWithCode | 📑 BibTeX
[!IMPORTANT]
🔥 Excited to find our DART-Math-DSMath-7B (Prop2Diff) trained on DART-Math-Hard comparable to the AIMO winner NuminaMath-7B on CoT,
but based solely on MATH & GSM8K prompt set, leaving much room to improve!
Besides, our DART method is also fully compatible… See the full description on the dataset page: https://huggingface.co/datasets/hkust-nlp/dart-math-hard.Weblate-Translations
Dataset Card for Weblate Translations
A dataset containing strings from projects hosted on Weblate and their translations into other languages.
Please consider donating or contributing to Weblate if you find this dataset useful.
Dataset Details
Dataset Description
Curated by: Mohamed Aymane Farhi
Funded by [optional]: [More Information Needed]
Shared by [optional]: [More Information Needed]
Language(s) (NLP): Check the README YAML metadata… See the full description on the dataset page: https://huggingface.co/datasets/Tamazight-NLP/Weblate-Translations.KoEVD
KoEVD
KoEVD is a Korean benchmark linking five evaluation or analysis targets through source utterances: utterance-risk judgment, candidate-response safety choice, direct-generation response harmfulness, descriptive response strategies, and a pre-execution mock tool/action-choice diagnostic.
Contents and scope
The canonical corpus contains 13,552 sources and 71,395 response candidates: 30,740 accepted, 27,104 rejected, and 13,551 strongly rejected. Three… See the full description on the dataset page: https://huggingface.co/datasets/KETI-NLP/KoEVD.paradetox
ParaDetox: Text Detoxification with Parallel Data (English)
This repository contains information about ParaDetox dataset -- the first parallel corpus for the detoxification task -- as well as models and evaluation methodology for the detoxification of English texts. The original paper "ParaDetox: Detoxification with Parallel Data" was presented at ACL 2022 main conference.
📰 Updates
[2025] !!!NOW OPEN!!! TextDetox CLEF2025 shared task: for even more -- 15 languages! website… See the full description on the dataset page: https://huggingface.co/datasets/s-nlp/paradetox.agentboard
AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents
This is the official dataset repository of AgentBoard.
1. Data Overview
AgentBoard is composed of 9 diverse tasks which can be divided into 4 types, including Embodied AI, Game, Web, and Tool:
Embodied AI
Game
Web
Tool
AlfWorld
ScienceWorld
BabyAI
Jericho
PDDL
WebShop
WebArena… See the full description on the dataset page: https://huggingface.co/datasets/hkust-nlp/agentboard.CALIBRI
CALIBRI Dataset
Dataset Description
CALIBRI is a comprehensive dataset for studying calibration in LLM-based code generation. It contains code generations from multiple state-of-the-art language models across three established benchmarks, along with token-level likelihood information for calibration analysis and correctness labels, based on the benchmark-provided test suites. Each sample provides 10 different generations for one problem.
Dataset Summary… See the full description on the dataset page: https://huggingface.co/datasets/lavis-nlp/CALIBRI.alexandria
Dataset Card for Alexandria
Alexandria covers 13 Arab countries, 11 domains, and 107K community-driven samples.
Alexandria is a multi-domain English↔Dialectal Arabic machine translation dataset designed for culturally inclusive, dialect-aware NLP and LLM evaluation. It pairs English multi-turn conversations with human-translated dialectal Arabic from 13 Arab countries, enriched with sub-dialect metadata (based on city-level information), domain labels, persona roles… See the full description on the dataset page: https://huggingface.co/datasets/UBC-NLP/alexandria.mc4-idA thoroughly cleaned version of the Italian portion of the multilingual
colossal, cleaned version of Common Crawl's web crawl corpus (mC4) by AllenAI.
Based on Common Crawl dataset: "https://commoncrawl.org".
This is the processed version of Google's mC4 dataset by AllenAI, with further cleaning
detailed in the repository README file.PluraMath
PluraMath 🌍➕
Extending Mathematical Reasoning Evaluation Beyond High-Resource Languages
🔎 TL;DR
PluraMath is a human-curated multilingual mathematical reasoning benchmark that extends PolyMath to 18 additional underrepresented languages spanning 6 language families — from mid-resource languages such as Hindi and Turkish down to extreme low-resource languages such as Upper and Lower Sorbian (< 15k L1 speakers).
Every language contains 500… See the full description on the dataset page: https://huggingface.co/datasets/tum-nlp/PluraMath.ko_commongen_v2
🌠 KoCommonGEN v2
KoCommonGEN v2: A Benchmark for Navigating Korean Commonsense Reasoning Challenges in Large Language Models (ACL 2024-Findings)
Jaehyung Seo, Jaewook Lee, Chanjun Park, SeongTae Hong, Seungjun Lee and Heuiseok Lim
🏫 NLP & AI Lab, Korea University
🔥 News
September 27, 2023: Provided data support for the Open Ko-LLM Leaderboard
August 7, 2024: Dataset Release
August 10, 2024: Experimental Results for the New Models Added
August 14, 2024: Presented a… See the full description on the dataset page: https://huggingface.co/datasets/nlpai-lab/ko_commongen_v2.temiz-OSCAR
Dataset Card for Temiz OSCAR
Temiz OSCAR is a corpora collection consisting of cleaned versions of original OSCAR corpora.
This collection is made up of four datasets: OSCAR-2019, OSCAR-2109, OSCAR-2201 and OSCAR-2301
This corpus is a part of large scale Turkish corpus Bella Turca. For more details about Bella Turca, please refer to the publication.
Dataset
num instances
size
num of words
OSCAR-2019
3.671.430
7.7G
976M
OSCAR-2109
8.472.809
18G
2.22B
OSCAR-2201… See the full description on the dataset page: https://huggingface.co/datasets/turkish-nlp-suite/temiz-OSCAR.kullm-v2
Dataset Card for "KULLM-v2"
Dataset Summary
Korean translation of GPT4ALL, Dolly, and Vicuna data.
repository: nlpai-lab/KULLM
huggingface: nlpai-lab/kullm-v2
Translate dataset
Translated 'instruction', 'input', and 'output' in the dataset via the DeepL API
Lisence
Apache-2.0
>>> from datasets import load_dataset
>>> ds = load_dataset("nlpai-lab/kullm-v2", split="train")
>>> ds
DatasetDict({
train: Dataset({
features: ['id'… See the full description on the dataset page: https://huggingface.co/datasets/nlpai-lab/kullm-v2.LLaVAR
LLaVAR Data: Enhanced Visual Instruction Data with Text-Rich Images
More info at LLaVAR project page, Github repo, and paper.
Training Data
Based on the LAION dataset, we collect 422K pretraining data based on OCR results. For finetuning data, we collect 16K high-quality instruction-following data by interacting with langauge-only GPT-4. Note that we also release a larger and more diverse finetuning dataset below (20K), which contains the 16K we used for the paper. The… See the full description on the dataset page: https://huggingface.co/datasets/SALT-NLP/LLaVAR.nanochat-jp-pretrain
nanochat-jp-pretrain
nanochat の日本語フォーク nanochat-jp で使用する 事前学習用日本語コーパス です．
LLM によるクリーニングを施した日本語ウェブテキストと，llm-jp の公開コーパスを混合したものを，nanochat のデータローダがそのまま読める parquet 形式で配布しています．
構成
以下の4つのソースを混合し，全体をシャッフルしています．
ソース
llm-jp-corpus-v4 の ja_fineweb-2 サブセット（後述の追加データクリーニングを適用）
llm-jp-corpus-midtraining-v2 ja/llm-jp-IPT_v0.3.2/ja_general.jsonl.gz
llm-jp-corpus-midtraining-v2 ja/llm-jp-IPT_v0.3.2/ja_reasoning.jsonl.gz
llm-jp/scaling-data-constrained-llms… See the full description on the dataset page: https://huggingface.co/datasets/tohoku-nlp/nanochat-jp-pretrain.mathdial
Mathdial dataset
https://arxiv.org/abs/2305.14536
MathDial: A Dialogue Tutoring Dataset with Rich Pedagogical Properties Grounded in Math Reasoning Problems.
MathDial is grounded in math word problems as well as student confusions which provide a challenging testbed for creating faithful and equitable dialogue tutoring models able to reason over complex information. Current models achieve high accuracy in solving such problems but they fail in the task of teaching.
Data… See the full description on the dataset page: https://huggingface.co/datasets/eth-nlped/mathdial.dart-math-pool-math
[!NOTE]
This dataset is the data pool synthesized from the query set of the MATH training set,
containing all answer-correct samples and other metadata produced during the work.
DART-Math-* datasets are extracted from dart-math-pool-* data pools.
🎯 DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solving
📝 Paper@arXiv | 🤗 Datasets&Models@HF | 🐱 Code@GitHub
🐦 Thread@X(Twitter) | 🐶 中文博客@知乎 | 📊 Leaderboard@PapersWithCode | 📑 BibTeX
Datasets:… See the full description on the dataset page: https://huggingface.co/datasets/hkust-nlp/dart-math-pool-math.