Datasetpublic

lillian039/nemotron_cc_v2_hq_packed4096_200shard

Nemotron-CC-v2 High-Quality, packed to 4096 tokens (train) Documents from nvidia/Nemotron-CC-v2 High-Quality subset, tokenized with google/t5gemma-2-270m-270m (add_special_tokens=False, EOS appended per document) and greedily packed into sequences of at most 4096 tokens. A document is never split across a pack boundary; documents longer than 4096 are truncated to their own pack. Every pack ends on an EOS/document boundary. Schema index (int64): running pack id… See the full description on the dataset page: https://huggingface.co/datasets/lillian039/nemotron_cc_v2_hq_packed4096_200shard.

sourceHugging Faceotherupdated 2mo agoView on Hugging Face

0likes207downloads

Dataset Card

Nemotron-CC-v2 High-Quality, packed to 4096 tokens (train)

Documents from nvidia/Nemotron-CC-v2 High-Quality subset, tokenized with google/t5gemma-2-270m-270m (addspecialtokens=False, EOS appended per document) and greedily packed into sequences of at most 4096 tokens.

A document is never split across a pack boundary; documents longer than 4096 are truncated to their own pack. Every pack ends on an EOS/document boundary.

Schema

—index (int64): running pack id
—input_ids (list<int32>): <= 4096 tokens, ends in EOS
—sequence_length (int64): len(input_ids)

Unconditional (no condition_input_ids). Produced by scripts/pack_nemotron_hq.py. This repo holds the train split.