datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
ledger-long-context-KPI-QA
LEDGER — Long-Context KPI Question Answering & Page Retrieval
This dataset is part of the LEDGER (Long-context Evaluation of Documents for
Grounded Extraction and Retrieval) benchmark.
It supports two of the three LEDGER tasks:
Page-level KPI retrieval — given a natural-language question about a financial
KPI and the corresponding annual report, retrieve the relevant page(s). Each row
includes TREC-style graded relevance judgments (qrels) over all candidate pages.… See the full description on the dataset page: https://huggingface.co/datasets/artefactory/ledger-long-context-KPI-QA.ledger-long-context-multi-kpi
the LEDGER Long-Context Multi-KPI extraction datasets and benchmarks.
OCR'd annual reports with ground-truth KPI values for financial information extraction benchmarking.
Dataset Description
This dataset pairs OCR-extracted annual report text (from DeepSeek OCR) with structured KPI ground-truth values. It is designed for evaluating LLM-based financial information extraction, retrieval, and needle-in-a-haystack tasks.
Configs
Config
Reports… See the full description on the dataset page: https://huggingface.co/datasets/artefactory/ledger-long-context-multi-kpi.Agentic-Long-Context-Understanding-QA 📖 Agentic Long Context Understanding 📖
Self-Taught Agentic Long Context Understanding (Arxiv).
AgenticLU refines complex, long-context queries through self-clarifications and contextual grounding, enabling robust long-document understanding in a single pass.
Installation Requirements
This codebase is largely based on OpenRLHF and Helmet, kudos to them.
The requirements are the same
pip install openrlhf
pip install -r ./HELMET/requirements.txt… See the full description on the dataset page: https://huggingface.co/datasets/yzhuang/Agentic-Long-Context-Understanding-QA.pretrain-web-mix-long-contextNormalized documents plus aligned Dolma-2 tokens and target masks.
Size
Tokens
8,689,580,607 (8.7B)
Trainable tokens
8,689,580,607 (8.7B)
Documents
281,846
Shards
89
UTF-8 bytes
37,540,769,483
Tokenizer
allenai/dolma2-tokenizer@5292e5d6c0f4
documents.parquet - document_id, text, part_ends, part_trainable,
must_not_split. The readable payload and the mask intent.
metadata.parquet - one text-free row per document: token span, source,
stratum, sizes… See the full description on the dataset page: https://huggingface.co/datasets/placeholderlabs/pretrain-web-mix-long-context.wmt-da-human-evaluation-long-context
Dataset Summary
Long-context / document-level dataset for Quality Estimation of Machine Translation.
It is an augmented variant of the sentence-level WMT DA Human Evaluation dataset.
In addition to individual sentences, it contains augmentations of 2, 4, 8, 16, and 32 sentences, among each language pair lp and domain.
The raw column represents a weighted average of scores of augmented sentences using character lengths of src and mt as weights.
The code used to apply the augmentation… See the full description on the dataset page: https://huggingface.co/datasets/ymoslem/wmt-da-human-evaluation-long-context.pretrain-ultra-fineweb-mix-long-contextNormalized documents plus aligned Dolma-2 tokens and target masks.
Size
Tokens
1,367,358,024 (1.4B)
Trainable tokens
1,367,358,024 (1.4B)
Documents
48,077
Shards
73
UTF-8 bytes
6,386,740,105
Tokenizer
allenai/dolma2-tokenizer@5292e5d6c0f4
documents.parquet - document_id, text, part_ends, part_trainable,
must_not_split. The readable payload and the mask intent.
metadata.parquet - one text-free row per document: token span, source,
stratum, sizes… See the full description on the dataset page: https://huggingface.co/datasets/placeholderlabs/pretrain-ultra-fineweb-mix-long-context.pretrain-commits-v2-mix-long-contextNormalized documents plus aligned Dolma-2 tokens and target masks.
Size
Tokens
12,571,681,749 (12.6B)
Trainable tokens
4,460,160,435 (4.5B)
Documents
992,475
Shards
327
UTF-8 bytes
49,288,867,997
Tokenizer
allenai/dolma2-tokenizer@5292e5d6c0f4
documents.parquet - document_id, text, part_ends, part_trainable,
must_not_split. The readable payload and the mask intent.
metadata.parquet - one text-free row per document: token span, source,
stratum, sizes… See the full description on the dataset page: https://huggingface.co/datasets/placeholderlabs/pretrain-commits-v2-mix-long-context.pretrain-academic-mix-long-contextNormalized documents plus aligned Dolma-2 tokens and target masks.
Size
Tokens
43,694,042,993 (43.7B)
Trainable tokens
43,694,042,993 (43.7B)
Documents
1,001,557
Shards
373
UTF-8 bytes
183,279,720,921
Tokenizer
allenai/dolma2-tokenizer@5292e5d6c0f4
documents.parquet - document_id, text, part_ends, part_trainable,
must_not_split. The readable payload and the mask intent.
metadata.parquet - one text-free row per document: token span, source,
stratum… See the full description on the dataset page: https://huggingface.co/datasets/placeholderlabs/pretrain-academic-mix-long-context.pretrain-encyclopedic-mix-long-contextNormalized documents plus aligned Dolma-2 tokens and target masks.
Size
Tokens
587,625,128 (587.6M)
Trainable tokens
587,625,128 (587.6M)
Documents
23,631
Shards
9
UTF-8 bytes
1,978,753,989
Tokenizer
allenai/dolma2-tokenizer@5292e5d6c0f4
documents.parquet - document_id, text, part_ends, part_trainable,
must_not_split. The readable payload and the mask intent.
metadata.parquet - one text-free row per document: token span, source,
stratum, sizes… See the full description on the dataset page: https://huggingface.co/datasets/placeholderlabs/pretrain-encyclopedic-mix-long-context.pretrain-repository-v2-mix-long-contextNormalized documents plus aligned Dolma-2 tokens and target masks.
Size
Tokens
130,158,375,824 (130.2B)
Trainable tokens
130,158,375,824 (130.2B)
Documents
2,578,578
Shards
1,168
UTF-8 bytes
535,260,344,241
Tokenizer
allenai/dolma2-tokenizer@5292e5d6c0f4
documents.parquet - document_id, text, part_ends, part_trainable,
must_not_split. The readable payload and the mask intent.
metadata.parquet - one text-free row per document: token span, source… See the full description on the dataset page: https://huggingface.co/datasets/placeholderlabs/pretrain-repository-v2-mix-long-context.repobench_python_long_context
RepoBench v1.1 (Python)
Introduction
This dataset presents the Python portion of RepoBench v1.1 (ICLR 2024). The data encompasses a collection from GitHub, spanning the period from October 6th to December 31st, 2023. With a commitment to data integrity, a deduplication process based on file content has been implemented against the Stack v2 dataset (coming soon), aiming to mitigate data leakage and memorization concerns.
Features
The dataset provides the… See the full description on the dataset page: https://huggingface.co/datasets/VivekChauhan06/repobench_python_long_context.pretrain-nemotron-math-mix-long-contextNormalized documents plus aligned Dolma-2 tokens and target masks.
Size
Tokens
1,446,296,439 (1.4B)
Trainable tokens
1,446,296,439 (1.4B)
Documents
42,379
Shards
23
UTF-8 bytes
4,965,563,314
Tokenizer
allenai/dolma2-tokenizer@5292e5d6c0f4
documents.parquet - document_id, text, part_ends, part_trainable,
must_not_split. The readable payload and the mask intent.
metadata.parquet - one text-free row per document: token span, source,
stratum, sizes… See the full description on the dataset page: https://huggingface.co/datasets/placeholderlabs/pretrain-nemotron-math-mix-long-context.FineWeb-long-context-documentssfm-midtraining-mix-dclm-long-context-passages-blocklist-filtereddataset_cards_with_long_context_embeddins
Dataset Card for "dataset_cards_with_long_context_embeddins"
More Information needed
long-contextDCLM-Long-Context-Subsetqwen3_0.6-task738_augmented_long-context_Mar16-1501qwen3_0.6b_long-context_Mar17-1945_blendedmodel_cards_with_long_context_embeddings
Dataset Card for "model_cards_with_long_context_embeddings"
More Information needed
D-ExpTracker__1022_longcontext__maxlen4096_0epoch_3and4arg__v1BigCodeBench-corrupted-long-context-no-testsqwen3_0.6-task738_augmented_long-context_Mar16-1501_blendedBigCodeBench-corrupted-long-contextScript to create the dataset
import argparse
import json
import random
from dataclasses import dataclass
from typing import Dict, List
import tiktoken
from datasets import Dataset
try:
from . import prompt_utils as prompt_utils
except Exception:
import prompt_utils as prompt_utils
BIN_EDGES = [4096, 8192, 16384, 32768, 65536, 128000]
@dataclass
class Entry:
task_id: str
canonical_solution: str
corrupted_solution: str
test_code: str
def load_entries(jsonl_path:… See the full description on the dataset page: https://huggingface.co/datasets/nreHieW/BigCodeBench-corrupted-long-context.D-ExpTracker__1022_longcontext__maxlen8192_1e_3args__v1D-ExpTracker__1022_longcontext__maxlen4096_0epoch_3args__v1qwen3_0.6b_long-context_Mar17-1945libri_clean_long_context-tagged-v1libri_clean_long_context-description-v1D-ExpTracker__1022_longcontext__maxlen8192_0epoch_3args__v1