datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
mix-context-post-training-128k
Mix-Context Post-Training Dataset for 128K Context Extension
Overview
Mix-Context Post-Training 128K is a dataset designed specifically for post-training context window extension of pretrained LLMs.
It targets the stage after base pretraining, where a model is adapted to operate over much longer contexts (up to 128K tokens) while preserving short-context behavior. The dataset mixes short- and long-context packed sequences with a controlled length distribution to support:… See the full description on the dataset page: https://huggingface.co/datasets/ghostcc3/mix-context-post-training-128k.loft-rag-nq-128k
LOFT RAG - Natural Questions (128k)
Dataset Description
This dataset is part of the LOFT (Long-context Open Foundation Tasks) benchmark, specifically the RAG (Retrieval-Augmented Generation) task.
Dataset: Natural Questions
Context Length: 128k
Task Type: RAG (Retrieval-Augmented Generation)
Language: English
Source: LOFT Benchmark (Google DeepMind)
Dataset Structure
Data Fields
context (string): Full prompt context including corpus… See the full description on the dataset page: https://huggingface.co/datasets/f20180301/loft-rag-nq-128k.msmarco-item-id-hardneg-100shot-v4_128kloft-rag-hotpotqa-128k
LOFT RAG - HotpotQA (128k)
Dataset Description
This dataset is part of the LOFT (Long-context Open Foundation Tasks) benchmark, specifically the RAG (Retrieval-Augmented Generation) task.
Dataset: HotpotQA
Context Length: 128k
Task Type: RAG (Retrieval-Augmented Generation)
Language: English
Source: LOFT Benchmark (Google DeepMind)
Dataset Structure
Data Fields
context (string): Full prompt context including corpus documents and few-shot… See the full description on the dataset page: https://huggingface.co/datasets/f20180301/loft-rag-hotpotqa-128k.loft-rag-musique-128k
LOFT RAG - MuSiQue (128k)
Dataset Description
This dataset is part of the LOFT (Long-context Open Foundation Tasks) benchmark, specifically the RAG (Retrieval-Augmented Generation) task.
Dataset: MuSiQue
Context Length: 128k
Task Type: RAG (Retrieval-Augmented Generation)
Language: English
Source: LOFT Benchmark (Google DeepMind)
Dataset Structure
Data Fields
context (string): Full prompt context including corpus documents and few-shot… See the full description on the dataset page: https://huggingface.co/datasets/f20180301/loft-rag-musique-128k.loft-rag-qampari-128k
LOFT RAG - Qampari (128k)
Dataset Description
This dataset is part of the LOFT (Long-context Open Foundation Tasks) benchmark, specifically the RAG (Retrieval-Augmented Generation) task.
Dataset: Qampari
Context Length: 128k
Task Type: RAG (Retrieval-Augmented Generation)
Language: English
Source: LOFT Benchmark (Google DeepMind)
Dataset Structure
Data Fields
context (string): Full prompt context including corpus documents and few-shot… See the full description on the dataset page: https://huggingface.co/datasets/f20180301/loft-rag-qampari-128k.loft-rag-quest-128k
LOFT RAG - Quest (128k)
Dataset Description
This dataset is part of the LOFT (Long-context Open Foundation Tasks) benchmark, specifically the RAG (Retrieval-Augmented Generation) task.
Dataset: Quest
Context Length: 128k
Task Type: RAG (Retrieval-Augmented Generation)
Language: English
Source: LOFT Benchmark (Google DeepMind)
Dataset Structure
Data Fields
context (string): Full prompt context including corpus documents and few-shot… See the full description on the dataset page: https://huggingface.co/datasets/f20180301/loft-rag-quest-128k.oellm-longctx-tokenized-natural-128k-256k-pilot-v1
OELLM Natural Long-Context Tokenized 128K/256K Expanded Pilot
This dataset contains Megatron-LM tokenized continuation-training artifacts for natural long-context extension experiments at 128K and 256K sequence scales.
This public revision expands the original pilot from 128 to 512 packed examples per source/tier, for 3,072 packed examples total. The raw pack manifest reports approximately 586M source-side estimated tokens across all six source/tier shards.
Accessible source… See the full description on the dataset page: https://huggingface.co/datasets/birgermoell/oellm-longctx-tokenized-natural-128k-256k-pilot-v1.reasoning-sft-synthetic_text_to_sql-128K
synthetic_text_to_sql (converted)
Converted version of gretelai/synthetic_text_to_sql, reformatted to 100,000 rows for reasoning SFT training.
Format
Each row has three columns:
input — list of dicts [{"role": "system", "content": "..."}, {"role": "user", "content": "..."}] (system prompt contains the database schema, user prompt contains the natural language question)
response — response string with <think> reasoning block (SQL explanation) followed by the SQL query… See the full description on the dataset page: https://huggingface.co/datasets/AmanPriyanshu/reasoning-sft-synthetic_text_to_sql-128K.fineweb-100_128k
BEE-spoke-data/fineweb-100_128k
100 documents from HuggingFaceFW/fineweb that are 128,000 GPT-4 tiktoken tokens or more.
longbench2-128k-plus
LongBench2-128k-plus
LongBench2-128k-plus is a long-context corpus derived from the
zai-org/LongBench-v2
benchmark. It keeps only the "long" examples and exposes just the raw
long documents, making it convenient for:
long-context pretraining or continued training,
long-context adaptation (e.g., RoPE scaling, attention tuning),
retrieval and RAG-style experimentation where only documents are needed.
All question/answer and multiple-choice metadata from LongBench v2 are
dropped;… See the full description on the dataset page: https://huggingface.co/datasets/viktor-shcherb/longbench2-128k-plus.oellm-longctx-tokenized-structured-128k-256k-v1
OELLM Structured Long-Context Tokenized 128K/256K v1
This dataset is a structured long-context add-on for continuation training. It complements the natural code/books/arXiv mix with precise, cross-reference-heavy technical prose.
Source families:
RFC Editor plain-text RFC standards/specifications, collected directly from rfc-editor.org/rfc
curated technical documentation files from Rust Reference, Rust Book, Django docs, Kubernetes docs, Python docs, and PostgreSQL docs
The… See the full description on the dataset page: https://huggingface.co/datasets/birgermoell/oellm-longctx-tokenized-structured-128k-256k-v1.