ByteSpanTokenisers/finewebedu-20B

FineWebEDU 20B A copy of FineWebEDU-20B used for out tokenizer experiments. The subsets are as follows: bytelevel: the full dataset tokenized using our bytelevel tokenizer bytelevel-subset_1: a 100k-row subset of the bytelevel subset, used to train bytelevel models. bytelevel-subset_2: a 100k-row subset of the bytelevel subset, used to extract llm predictions. bytelevel-llm-data: a copy of bytelevel-subset_2 with lm predictions, used to train bytespan tokenizers… See the full description on the dataset page: https://huggingface.co/datasets/ByteSpanTokenisers/finewebedu-20B.