ByteSpanTokenisers/finewebedu-20B

FineWebEDU 20B

A copy of FineWebEDU-20B used for out tokenizer experiments. The subsets are as follows:

• —bytelevel: the full dataset tokenized using our bytelevel tokenizer
• —bytelevel-subset_1: a 100k-row subset of the bytelevel subset, used to train bytelevel models.
• —bytelevel-subset_2: a 100k-row subset of the bytelevel subset, used to extract llm predictions.
• —bytelevel-llm-data: a copy of bytelevel-subset_2 with lm predictions, used to train bytespan tokenizers
• —bytelevel-subset_3: a 100k-row subset of the bytelevel subset, used to evaluate trained tokenizers

The remaining subsets are all versions of the dataset tokenized with our trained tokenizers:

• —BPE_64000
• —BPEWP_64000
• —ByteSpanSurprisalMonotonicFrequency_64000
• —ByteSpanSurprisalMonotonicSeeding_64000
• —ByteSpanSurprisalCombinedFrequency_64000
• —ByteSpanSurprisalCombinedSeeding_64000
• —ByteSpanSurprisalGlobalIncrement_64000