Datasetpublic

jensjepsen/danish-pretokenized-16k

Danish Pretrain — pretokenized (byte-level BPE, 16k vocab) — v1 partial Pretokenized version of jensjepsen/danish-pretrain via jensjepsen/danish-tokenizer. Note: This is a partial version — 94 of 96 shards. The 2 missing shards crashed during tokenization (rare pathological content). See jensjepsen/danish-pretokenized-16k-supp for the recovered rows. 93,200,464 docs Schema: input_ids: list<int32>, attention_mask: list<int8> 94 zstd-compressed parquet shards

sourceHugging Facecc-by-4.0updated 2mo agoView on Hugging Face

0likes661downloads

Dataset Card

Danish Pretrain — pretokenized (byte-level BPE, 16k vocab) — v1 partial

Pretokenized version of jensjepsen/danish-pretrain via jensjepsen/danish-tokenizer.

Note: This is a partial version — 94 of 96 shards. The 2 missing shards crashed during tokenization (rare pathological content). See jensjepsen/danish-pretokenized-16k-supp for the recovered rows.

—93,200,464 docs
—Schema: input_ids: list<int32>, attention_mask: list<int8>
—94 zstd-compressed parquet shards