sxiong/DHSA_Long-Data-Collections

DHSA_Long-Data-Collections

A length-bucketed release of togethercomputer/Long-Data-Collections, used in [Long-Context Modeling with Dynamic Hierarchical Sparse Attention for Memory-Constrained LLM Inference](https://arxiv.org/pdf/2510.24606) (ICML 2026 Spotlight).

Each example is assigned to exactly one length bucket based on token count with meta-llama/Llama-3.1-8B-Instruct (add_special_tokens=True):

• —Pretrain examples are bucketed by the text field.
• —Fine-tune examples are bucketed by the prompt field.

Directory layout

Long-Data-Collections/
├── README.md
├── length_bucket_summary.json
├── pretrain/
│   ├── lt_8k/
│   │   ├── arxiv_doc_to_abs.jsonl.zst
│   │   ├── NI_decontaminated_materialized.jsonl.zst
│   │   ├── P3_decontaminated_materialized.jsonl.zst
│   │   ├── pile_sub.jsonl.zst
│   │   ├── rp_sub.jsonl.zst
│   │   └── ul2_plus_oscar_en.jsonl.zst
│   ├── 8k_16k/
│   ├── 16k_32k/
│   ├── 32k_64k/
│   ├── 64k_128k/
│   └── gt_128k/
└── fine-tune/
    ├── lt_8k/
    │   ├── booksum.jsonl.zst
    │   └── natural_questions_10_200_docs.jsonl.zst
    ├── 8k_16k/
    ├── 16k_32k/
    ├── 32k_64k/
    ├── 64k_128k/
    └── gt_128k/

All files are zstd-compressed JSONL (.jsonl.zst). Each line is a JSON object preserving the original fields (text, and optionally meta / metadata for pretrain; text, prompt, completion for fine-tune).

Example counts by bucket

Pretrain (bucketed by text length)

Fine-tune (bucketed by prompt length)

Full counts are also available in length_bucket_summary.json.

Dataset description

This collection compiles long-context datasets for training and evaluating models that require extensive comprehension over large text inputs. It is derived from the public Together AI release and reorganized into length buckets for long-context experiments.

Pretrain data (pretrain/)

Fine-tune data (fine-tune/)

Fine-tune records contain prompt (context + instruction), completion (target answer/summary), and text (prompt + completion).

Usage

Load a specific bucket and file with the Hugging Face datasets library:

from datasets import load_dataset

ds = load_dataset(
    "json",
    data_files="pretrain/16k_32k/arxiv_doc_to_abs.jsonl.zst",
    split="train",
)

Or stream locally with zstd:

import json, zstandard as zstd

path = "pretrain/8k_16k/rp_sub.jsonl.zst"
with open(path, "rb") as f:
    with zstd.ZstdDecompressor().stream_reader(f) as reader:
        for line in reader:
            row = json.loads(line)
            ...

Licensing

Please refer to the original sources of each sub-dataset for their respective licenses. This dataset is a length-bucketed release of togethercomputer/Long-Data-Collections; upstream licensing terms apply.

Citation

@inproceedings{xionglong,
  title={Long-Context Modeling with Dynamic Hierarchical Sparse Attention for Memory-Constrained LLM Inference},
  author={Xiong, Siheng and Zou, Joe and Fekri, Faramarz and Cho, Yae Jee},
  booktitle={Forty-third International Conference on Machine Learning}
}