128k-context
mix-context-post-training-128k
Mix-Context Post-Training Dataset for 128K Context Extension
Overview
Mix-Context Post-Training 128K is a dataset designed specifically for post-training context window extension of pretrained LLMs.
It targets the stage after base pretraining, where a model is adapted to operate over much longer contexts (up to 128K tokens) while preserving short-context behavior. The dataset mixes short- and long-context packed sequences with a controlled length distribution to support:… See the full description on the dataset page: https://huggingface.co/datasets/ghostcc3/mix-context-post-training-128k.long-context-attention-labels-64k-128k
Long-Context Attention Labels (64K & 128K)
Attention-based document labels for long-context training data selection.
Overview
Each document is labeled with attention lookback metrics computed by running it through a model and measuring how far back each token attends (via top-k=10 head-averaged attention distances).
Run
Model
Context
Records
olmo3_64k
OLMo-3-1025-7B (stage2)
65,536 tokens
~5,000
olmo3_128k
OLMo-3-1025-7B (stage2)
131,072 tokens
~4,600… See the full description on the dataset page: https://huggingface.co/datasets/KevinDavidHayes/long-context-attention-labels-64k-128k.