CoolFace
Datasetpublic

birgermoell/oellm-longctx-tokenized-natural-128k-256k-pilot-v1

OELLM Natural Long-Context Tokenized 128K/256K Expanded Pilot This dataset contains Megatron-LM tokenized continuation-training artifacts for natural long-context extension experiments at 128K and 256K sequence scales. This public revision expands the original pilot from 128 to 512 packed examples per source/tier, for 3,072 packed examples total. The raw pack manifest reports approximately 586M source-side estimated tokens across all six source/tier shards. Accessible source… See the full description on the dataset page: https://huggingface.co/datasets/birgermoell/oellm-longctx-tokenized-natural-128k-256k-pilot-v1.

sourceHugging Faceotherupdated 3mo agoView on Hugging Face
0likes30downloads
Dataset Card

OELLM Natural Long-Context Tokenized 128K/256K Expanded Pilot

This dataset contains Megatron-LM tokenized continuation-training artifacts for natural long-context extension experiments at 128K and 256K sequence scales.

This public revision expands the original pilot from 128 to 512 packed examples per source/tier, for 3,072 packed examples total. The raw pack manifest reports approximately 586M source-side estimated tokens across all six source/tier shards.

Accessible source streams used for this revision:

  • —edward-io/starcoderdata-repo as repo-packed code fallback
  • —sedthh/gutenberg_multilang for long multilingual book text
  • —allenai/olmo-mix-1124 / arxiv for full paper text

The intended final code source is Stack v2 repository hydration, but that is not included here because gated Stack v2/SWH access was not available on the GPU host at build time.

Contents

The artifact layout follows openeuro-longctx-megatron-v1:

  • —bin/: Megatron .bin / .idx files plus tokenization summary
  • —mix/: weighted Megatron --data-path files
  • —manifests/: build metadata, SHA256 checksums, raw pack manifest, and the 512-example recipe

The staged training mix is:

  • —natural_code_repo_pack_128k: 25%
  • —natural_code_repo_pack_256k: 25%
  • —natural_books_concat_128k: 15%
  • —natural_books_concat_256k: 15%
  • —natural_arxiv_full_concat_128k: 10%
  • —natural_arxiv_full_concat_256k: 10%

Each source/tier contains 512 packed examples.

Usage

Download the dataset and use the generated Megatron data path:

bash
python -m longctx.cli artifacts download \
  --repo-id birgermoell/oellm-longctx-tokenized-natural-128k-256k-pilot-v1 \
  --output-dir ./data/oellm-natural-128k-256k-pilot

export DATA_PATH="$(cat ./data/oellm-natural-128k-256k-pilot/mix/data_path.args)"

Pass $DATA_PATH to Megatron with --data-path.

Tokenizer

Built with:

  • —tokenizer type: HuggingFaceTokenizer
  • —tokenizer model: /home/ubuntu/birger/Megatron-Bridge-utils/tokenizers/openeurollm/tokenizer-256k
  • —vocab size: 262144

Source and license note

This is a derived tokenized artifact. Rights and usage constraints follow the upstream datasets and their source documents; this dataset card does not grant additional rights beyond those upstream terms.