sycmucmu/prolong-smollm2-validation

ProLong SmolLM2 validation

Two validation sets for evaluating DCLM-trained language models, retokenized from the code, books, and textbooks subsets of princeton-nlp/prolong-data-64K.

Each folder contains prolong_val_100m.bin, per-sequence source metadata in prolong_val_100m.json, and manifest-prolong-val.json with checksums and conversion settings.

Conversion

• —Target tokenizer: HuggingFaceTB/SmolLM2-135M, revision 93efa2f097d58c2a74874c7e644dbc9b0cee75a2; vocabulary 49,152, EOS ID 0.
• —Source tokenizer: princeton-nlp/Llama-3-8B-ProLong-64k-Base, revision ee4b1380059c70aaeab703934ca669f368cc7905.
• —Decode each original document segment independently. Remove source BOS/end markers before decoding; append target EOS only when the source segment had an end marker. Literal special-token spellings are encoded as ordinary text (split_special_tokens=True), matching DCLM preparation.
• —Keep complete context-length chunks within each document segment; discard its remainder. Do not join separate document segments into one sequence.
• —Code/books/textbooks sequence quotas use weights 204884:196544:7084, allocated by largest remainder. Source rows follow NumPy seeded permutations with seeds 42/43/44 in that domain order.
• —No source rows are excluded. These are newly selected validation subsets of the source corpus, not an upstream official validation split. No deduplication against DCLM was performed.
• —The two lengths use the same selection rule but are not identical token streams: changing chunk length changes discarded remainders and how many source rows fill the budget.
• —Each manifest records one text round-trip mismatch, in the same code source row (global_sample_index=23002): the pinned native SmolLM2 tokenizer drops one U+001D control character. All saved chunks of that row match native tokenization exactly. Books and textbooks had no recorded round-trip mismatches.

Download and read

hf download sycmucmu/prolong-smollm2-validation --repo-type dataset --local-dir prolong-smollm2-validation

Each binary has one 1,024-byte header (256 little-endian int32 values) followed by little-endian uint16 token IDs. Header fields 0–2 are 20240520, 1, and the stored token count. Headers are not tokens. The trainer must ignore the trailing filler outside complete sequences.

import json
from pathlib import Path
import numpy as np

root = Path("prolong-smollm2-validation/32k")
manifest = json.loads((root / "manifest-prolong-val.json").read_text())
val = manifest["validation"]
tokens = np.memmap(root / val["path"], dtype="<u2", mode="r",
                   offset=manifest["header_bytes"], shape=(val["tokens"],))
sequences = tokens[:val["usable_tokens"]].reshape(val["sequences"], manifest["sequence_length"])

Future training exclusions

Preserve the per-sequence JSON. Each record contains domain, source (original MDS shard), shard_index, sample_index, global_sample_index, document_index, and document_chunk_index.

For later 32k ProLong training, exclude every (domain, global_sample_index) appearing in 32k/prolong_val_100m.json, so all chunks from a validation source row remain held out. This is 1,496 source rows; all 1,326 source rows used by the 4k set are included, so this exclusion protects both sets. The source field also permits excluding their 685 whole original MDS shards if desired. Source-row exclusion does not establish document-level deduplication across different source rows.