Datasetpublic

placeholderlabs/exp-pool-commit-code-dolma2-tokenized

Locus EXP Commit Code - Dolma 2 tokenized Pretokenized experiment pool for reproducible proxy-training runs. MANIFEST.json is the authoritative schema, provenance, checksums, and train/holdout assignment. shards/shard-NNNNN/tokens.bin stores little-endian int32 token IDs. offsets.bin stores little-endian int64 document boundaries. index.parquet stores document IDs, offsets, and compact filter fields. metadata.parquet stores complete source metadata and is downloaded only for… See the full description on the dataset page: https://huggingface.co/datasets/placeholderlabs/exp-pool-commit-code-dolma2-tokenized.

sourceHugging Faceotherupdated 1mo agoView on Hugging Face

0likes469downloads

Dataset Card

Locus EXP Commit Code - Dolma 2 tokenized

Pretokenized experiment pool for reproducible proxy-training runs.

—MANIFEST.json is the authoritative schema, provenance, checksums, and train/holdout assignment.
—shards/shard-NNNNN/tokens.bin stores little-endian int32 token IDs.
—offsets.bin stores little-endian int64 document boundaries.
—index.parquet stores document IDs, offsets, and compact filter fields.
—metadata.parquet stores complete source metadata and is downloaded only for explicit cold-metadata filters.

Each document already ends in exactly one EOD token. Training must not add another document-level EOD.

Commit examples use target_start_token to mask the commit-message prompt and train only on the unified-diff target. Each commit document must fit one sequence.