Datasetpublic

Ba2han/long_corpus-0209_tokenized

long_corpus-0209_tokenized Exact-deduplicated, tokenized, and shuffled pretraining mix. Processing Tokenizer: /workspace/good_tokenizer (vocab_size=60800) Special tokens: BOS=<bos> (id 2), EOS=<eos> (id 3) bos_eos = True: every example is [BOS] + content + [EOS] min_tokens = 15 (including BOS/EOS) max_tokens = 3072: content truncated to 3070 tokens, then BOS+EOS appended (3070 + eos + bos) Dedup: exact match on stripped UTF-8 text (blake2s-128) Shuffle:… See the full description on the dataset page: https://huggingface.co/datasets/Ba2han/long_corpus-0209_tokenized.

sourceHugging Faceotherupdated 19d agoView on Hugging Face

0likes172downloads

Dataset Card

longcorpus-0209tokenized

Exact-deduplicated, tokenized, and shuffled pretraining mix.

Processing

—Tokenizer: /workspace/good_tokenizer (vocab_size=60800)
—Special tokens: BOS=<bos> (id 2), EOS=<eos> (id 3)
—bos_eos = True: every example is [BOS] + content + [EOS]
—min_tokens = 15 (including BOS/EOS)
—max_tokens = 3072: content truncated to 3070 tokens, then BOS+EOS appended (3070 + eos + bos)
—Dedup: exact match on stripped UTF-8 text (blake2s-128)
—Shuffle: hash-partition into 128 buckets, permute buckets and rows with seed 42
—Columns: input_ids (list<int32>), token_count (int32)
—Parquet: zstd, ~100,000 rows/shard

Sources were streamed interleaved. Documents that collide on the exact-text hash keep the first copy seen in interleave order.

Token statistics


Rows scanned	36,882,289
Non-string / empty	0
Exact duplicates dropped	546,897
Too short (< 15 tokens)	8,456
Truncated to 3072	160,264
Accepted rows	36,326,936
Accepted tokens	21,097,637,628
Average tokens/row	580.77
Median tokens/row	371
p90 tokens/row	1,363
p99 tokens/row	2,748
Min tokens/row	15
Max tokens/row	3,072
Output shards	301

Per-source counts

Source	Scanned	Duplicates	Too short	Truncated	Accepted rows	Accepted tokens
`Ba2han/long_corpus-0209`	33,210,680	489,633	4,679	113,859	32,716,368	17,773,735,140
`Ba2han/temiz-OSCAR-long-all`	1,476,355	56,153	0	2,895	1,420,202	1,796,185,623
`Ba2han/finepdfs-long`	140,166	0	0	6,094	140,166	249,113,008
`leukas/climbmix-1b-shuffle`	2,055,088	1,111	3,777	37,416	2,050,200	1,278,603,857

Sources

1.Ba2han/long_corpus-0209 — primary long mix (TR/AZ/CRH/TUK/EN)
2.Ba2han/temiz-OSCAR-long-all — filtered long OSCAR
3.Ba2han/finepdfs-long — filtered long FinePDFs (tur_Latn)
4.leukas/climbmix-1b-shuffle — 1B-word ClimbMix sample

Source licenses still apply. long_corpus-0209 is mixed/other (includes CC BY-NC and non-commercial derived rows).