datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
fineweb-edu
FineWeb-Edu (Lance Format)
A Lance-formatted version of FineWeb-Edu — over 1.5 billion educational web passages with cleaned text, source metadata, language detection signals, and 384-dim text embeddings — available directly from the Hub at hf://datasets/lance-format/fineweb-edu/data/train.lance.
Key features
Cleaned passage text in the text column with the source url and title carried alongside.
Language detection signals (language, language_probability) for filtered… See the full description on the dataset page: https://huggingface.co/datasets/lance-format/fineweb-edu.ms-marco-v2.1-lance
MS MARCO v2.1 QA (Lance Format)
A Lance-formatted version of MS MARCO v2.1 — Microsoft's machine-reading-comprehension benchmark built from anonymized Bing query logs. Each row is one user query, the up-to-10 candidate passages Bing retrieved for it with relevance flags, and the human-written reference answers, with MiniLM query embeddings stored inline and pre-built ANN/FTS indices, available directly from the Hub at hf://datasets/lance-format/ms-marco-v2.1-lance/data.… See the full description on the dataset page: https://huggingface.co/datasets/lance-format/ms-marco-v2.1-lance.hotpotqa-distractor-lance
HotpotQA distractor (Lance Format)
A Lance-formatted version of HotpotQA using the distractor config — multi-hop reading-comprehension questions where each answer requires combining facts from two Wikipedia paragraphs, with 10 candidate paragraphs per question (gold + 8 distractors). The dataset ships with MiniLM question embeddings, flattened context text for full-text search, and pre-built ANN/FTS indices, available directly from the Hub at… See the full description on the dataset page: https://huggingface.co/datasets/lance-format/hotpotqa-distractor-lance.trivia-qa-lance
TriviaQA (Lance Format)
A Lance-formatted version of TriviaQA (rc.nocontext config) — a large reading-comprehension dataset of trivia questions paired with a canonical answer, accepted aliases, and entity-type metadata — with MiniLM question embeddings stored inline and ready for retrieval at hf://datasets/lance-format/trivia-qa-lance/data. The rc.nocontext slice is the standard reading-comprehension form without the multi-gigabyte entity_pages / search_results payloads, which keeps… See the full description on the dataset page: https://huggingface.co/datasets/lance-format/trivia-qa-lance.natural-questions-val-lance
Natural Questions — Validation (Lance Format)
A Lance-formatted version of the Natural Questions validation split — 7,830 real Google search queries paired with the full Wikipedia article a human used to answer them, plus 1–5 annotator labels per question. MiniLM question embeddings are stored inline and the dataset ships with pre-built ANN/FTS indices, all available directly from the Hub at hf://datasets/lance-format/natural-questions-val-lance/data. Sourced from… See the full description on the dataset page: https://huggingface.co/datasets/lance-format/natural-questions-val-lance.FormationEval
FormationEval
FormationEval is a public benchmark suite for petroleum geoscience language model evaluation.
default remains the evaluated MCQ v0.1 track (Christmas 2025) with 505 questions and 72 published model results.
diskos_qa adds 1027 QA items imported 17 March 2026 from DISKOS-QA as a separate track.
spe_mcq adds 100 MCQ items imported 21 March 2026 from ynuwara/spe_mcq_dataset as a separate track.
The public leaderboard, charts and quiz still reflect the evaluated MCQ v0.1… See the full description on the dataset page: https://huggingface.co/datasets/AlmazErmilov/FormationEval.gov-report-qs-llama2-format
Government Report Question Answering Dataset in LLAMA2 Format
Dataset Description
This dataset is a LLAMA2 formatted dataset of the GovReport Dataset which is a report dataset, consisting of reports written by government research agencies including Congressional Research Service and US Government Accountability Office.
The purpose of creating this dataset is to provide those trying to finetune LLAMA2 and other LLM models for Government domain a formatted and easier to use… See the full description on the dataset page: https://huggingface.co/datasets/Kira-Floris/gov-report-qs-llama2-format.t5gemma2-indonesia-chat-formatted
T5Gemma-2 Indonesian Chat & QA Dataset
A high-quality Indonesian language multi-turn conversation and reading comprehension dataset, specifically formatted for instruction tuning of sequence-to-sequence (Seq2Seq) models like T5-Gemma / T5-Gemma-2.
Dataset Description
This dataset contains over 7,400 multi-turn conversations and document-based Q&A in Bahasa Indonesia. It covers diverse topics including everyday life, technology, general knowledge, and structured… See the full description on the dataset page: https://huggingface.co/datasets/daruokta/t5gemma2-indonesia-chat-formatted.well_formatted_benchmarks_pro
Dataset Card for well_formatted_benchmarks_pro
This is a collection of formatted benchmarks.
Dataset Details
Dataset Description
This repo is home to formatted versions of some famous benchmarks
I created this repo because current benchmark datasets on the hub generally don't have a fixed format, which is annoying when you try to use them.
Language(s) (NLP): English
Dataset Sources
ARC
Repository: Original ARC repo
Demo:
<user>An… See the full description on the dataset page: https://huggingface.co/datasets/patrickechohelloworld/well_formatted_benchmarks_pro.asknyc-chatassistant-formatQuestions from Reddit.com/r/AskNYC, downloaded from PushShift, filtered to direct responses from humans, where the post net score is >= 3.
Collected one month of posts from each year 2015-2019 (i.e. no content from July 2019 onward)
Adapted from the CSV used to fine-tune https://huggingface.co/monsoon-nlp/gpt-nyc
Blog about the original model: https://medium.com/geekculture/gpt-nyc-part-1-9cb698b2e3d
default-open-r1-math-90k-formatThis dataset, derived from the default version of OpenR1-Math-220K, has been reformatted and organized for improved usability and model training. The following data processing and quality filtering measures were implemented:
Data Processing and Quality Filtering Methodology
Structural Integrity Validation:
Ensures data consistency by verifying equal lengths across correctness_math_verify, is_reasoning_complete, and generations lists.
Confirms the presence of all required fields within each… See the full description on the dataset page: https://huggingface.co/datasets/DylanDDeng/default-open-r1-math-90k-format.oasst2_thai_top1_chat_format
Open Assistant 2 Top-1 Thai
Dataset Details
Dataset Description
A top-1 Thai dataset taken from the top scoring https://huggingface.co/datasets/OpenAssistant/oasst2 conversations. Saved in HF Chat format.
License: Apache 2.0
Script: https://github.com/wannaphong/deep_4_all/tree/main/datasets/oasst
Dataset Structure
We structure the dataset using the format commonly used as input into Hugging Face Chat Templates:
[
{'content':… See the full description on the dataset page: https://huggingface.co/datasets/pythainlp/oasst2_thai_top1_chat_format.sfd-archive-1b-source-format-sample
SFD Archive 1B-Token Source-Format Sample
Cleaned artifacts for an archive-wide SFD source-format analysis sample. The sanitized filing_stats.jsonl.gz contains 37,534 parsed filing rows and 997,469,365 final SFD tokens. The sampled manifest contains 100,000 candidate rows. summary.json is recomputed from the uploaded filing stats; source_summary_checkpoint.json preserves the original run checkpoint summary. Parser stdout tails, local paths, temporary raw SEC downloads, and process… See the full description on the dataset page: https://huggingface.co/datasets/sfd-anonymous/sfd-archive-1b-source-format-sample.dolly-15k-rlhf-instructgpt-format
Dolly 15k RLHF Datasets in InstructGPT Format
This repository packages databricks/databricks-dolly-15k into three RLHF-oriented
dataset configurations inspired by the InstructGPT data flow:
sft: supervised fine-tuning examples with prompt, completion, and text.
rm_schema: reward-modeling schema/prompt pool with empty chosen and rejected
fields, reference_response, and ready_for_rm=false.
rm_synthetic: reward-modeling proxy pairs where Dolly reference_response is
used as chosen and… See the full description on the dataset page: https://huggingface.co/datasets/costadev00/dolly-15k-rlhf-instructgpt-format.BioDSBench-imaging101-format
BioDSBench (Imaging-101 Format)
This dataset packages 118 BioDSBench Python tasks in an imaging-101-like task-per-directory layout, aligned with the structure of imaging-101 benchmark tasks.
It is a re-formatted version of BioDSBench, restructured for compatibility with source-native LLM agent evaluation harnesses such as biodsbench-adapter.
Dataset Summary
118 biomedical Python data-science tasks across 13 PMIDs (biomedical publications)
Each task has:… See the full description on the dataset page: https://huggingface.co/datasets/starpacker52/BioDSBench-imaging101-format.dataset-portuguese-aira-v2-Gemma-formatDataset Aira para o formato do Modelo Gemma
Resumo do Dataset
Este conjunto de dados contém uma coleção de conversas individuais entre um assistente e um usuário.
As conversas foram geradas pelas interações do usuário com modelos já ajustados (ChatGPT, LLama 2, Open-Assistant, etc).
O conjunto de dados está disponível em português (tem a versão em Inglês que ainda não tratei). Mas você pode baixar do
repositório de Nicholas Kluge Corrêa tanto a versão em Português e
a versão em… See the full description on the dataset page: https://huggingface.co/datasets/EddyGiusepe/dataset-portuguese-aira-v2-Gemma-format.blockchain-benchmark-formatted
Dataset Card for LLM Blockchain Benchmark
Dataset Summary
The Blockchain Benchmark Dataset is a comprehensive collection of data specifically curated for benchmarking Language Models (LMs) in the domain of blockchain technology. This dataset is designed to facilitate research and development in natural language understanding within the blockchain domain.
A complete list of tasks: ['general-reasoning', 'code', 'math']
Supported Tasks and Leaderboards
Model… See the full description on the dataset page: https://huggingface.co/datasets/revflask/blockchain-benchmark-formatted.MentalBench-formattedhysong/MentalBench formatted into splits based on type, options converted into a list of strings and answer split into answer/answer_index.
Vietnamese-BAAI-SVIT-llava-v1.5-format-gg-translatedorca_mini_uncensored_squad_format_train
Dataset Card for "orca_mini_uncensored_squad_format_train"
Dataset Description
Mostly purely an exercise in data extraction and formatting for dataset usage, and cross-model usage of data.
Uncensored data, because when everything is sanitized for alignment, the data may be "pure" but is no longer untimately realistic.
Part of an effort to create more question-answering friendly datasets that can be used for specialized domain training on small models.
Dataset… See the full description on the dataset page: https://huggingface.co/datasets/jtatman/orca_mini_uncensored_squad_format_train.LDJnr_combined_inout_format
Dataset Card for "LDJnr_combined_inout_format"
Dataset contains QA format versions of the data contained in the following datasets:
LDJnr/Capybara
LDJnr/Pure-Dove
LDJnr/Verified-Camel
This consists of an exploded out converation list seperated into input and output params for each, while retaining the source information for attribution purposes.
Formated-openai-function-invocations-20k-with-greetings
About
This dataset is the formated version of the Isaak-Carter/Openai-function-invocations-20k-with-greetings dataset.
This dataset, uniquely structured with custom special tokens, is meticulously crafted to train language models in complex function invocation and time-contextualized interactions. Each "sample" in the dataset contains a sequence of elements: function definitions, user prompts, function calls, function responses, and the assistant's responses. These elements are… See the full description on the dataset page: https://huggingface.co/datasets/Goekdeniz-Guelmez/Formated-openai-function-invocations-20k-with-greetings.pubmedqa-chat-format
PubMedQA (Chat-Format Preparation)
This dataset is a chat-format preparation of PubMedQA for biomedical QA SFT.
Format
This format is commonly referred to as:
chat-format SFT data
instruction-tuning conversations
OpenAI-style messages format
Included files
train.jsonl
validation.jsonl
stats.json
prepare_pubmedqa_unsloth.py
Source
Base dataset: qiaojin/PubMedQA
Subsets used for supervised preparation:
pqa_labeled
pqa_artificial (sampled)… See the full description on the dataset page: https://huggingface.co/datasets/ali5341/pubmedqa-chat-format.arxivqa-chat-format
ArxivQA (Chat-Format Preparation)
This dataset is a multimodal chat-format preparation of ArxivQA for supervised fine-tuning (SFT).
Format
This format is commonly referred to as:
chat-format SFT data
instruction-tuning conversations
OpenAI-style messages format
multimodal chat format (image + text)
Included files
train.jsonl
validation.jsonl
stats.json
prepare_arxivqa_unsloth.py
Source
Base dataset: MMInstruction/ArxivQA
Original… See the full description on the dataset page: https://huggingface.co/datasets/ali5341/arxivqa-chat-format.allyarc_oai_format
Dataset Card for AllyArc/allyarc_oai_format
This dataset card provides a structured overview of the AllyArc/allyarc_oai_format dataset, designed for training conversational AI models tailored for educational purposes, with a special focus on supporting students with diverse learning needs, including those in Special Educational Needs (SEN) education.
Dataset Details
Dataset Description
The AllyArc/allyarc_oai_format dataset is comprised of conversational… See the full description on the dataset page: https://huggingface.co/datasets/AllyArc/allyarc_oai_format.TweetQA_squad_formatorca_minis_uncensored_squad_format
Dataset Card for "orca_minis_uncensored_squad_format"
This dataset is a part of a continued series providing interestingly formatted existing data from unrelated datasets for question/answering model use.
Alternately it can provide a common format that could be converted to something else easily using available scripts and utilities fairly easily.
This is a work in progress and is changing every few days currently. Please refrain from using it for anything, especially… See the full description on the dataset page: https://huggingface.co/datasets/jtatman/orca_minis_uncensored_squad_format.oasst2_top1_chat_format_en
OpenAssistant TOP-1 English Conversations
This is a twice filtered dataset from oasst2, which is a set of conversation trees collected by the OpenAssistant project.
It was first filtered for the top ranked branches in each conversation tree, to form blancsw/oasst2_top1_chat_format
It was then filtered down to English-only, and to a single 'messages' data column. This allows the dataset to directly be input to the HuggingFace SFTTrainer (provided your tokenizer has a chat template)… See the full description on the dataset page: https://huggingface.co/datasets/Trelis/oasst2_top1_chat_format_en.ScienticDatasetArxiv-openAI-FormatV3
📚 Scientific Dataset Arxiv OpenAI Format
This dataset contains scientific data transformed for use with OpenAI models. It includes detailed descriptions and structures designed for machine learning applications. The original data was taken from:
from datasets import load_dataset
dataset = load_dataset("taesiri/arxiv_qa")
📂 Dataset Structure
The dataset is organized into a training split with comprehensive features tailored for scientific document processing:
Train… See the full description on the dataset page: https://huggingface.co/datasets/ejbejaranos/ScienticDatasetArxiv-openAI-FormatV3.spiqa-chat-format
SPIQA (Chat-Format Preparation)
This dataset is a chat-format preparation of SPIQA for multimodal supervised fine-tuning (SFT).
Format
This format is commonly referred to as:
chat-format SFT data
instruction-tuning conversations
OpenAI-style messages format
Included files
train.jsonl
validation.jsonl
stats.json
prepare_spiqa_unsloth.py
Source
Base dataset: google/spiqa
Original Dataset Highlights
Original dataset: google/spiqa… See the full description on the dataset page: https://huggingface.co/datasets/ali5341/spiqa-chat-format.