Datasetpublic

chanind/openwebtext-gemma

OpenWebTextCorpus tokenized for Gemma This dataset is a pre-tokenized version of the Skylion007/openwebtext dataset using the gemma tokenizer. As such, this dataset follows the same licensing as the original openwebtext dataset. This pre-tokenization is done as a performance optimization for using the openwebtext dataset with a Gemma model (gemma-2b, gemma-2b-it, gemma-7b, gemma-7b-it). This dataset was created using SAELens, with the following settings: context_size: 8192… See the full description on the dataset page: https://huggingface.co/datasets/chanind/openwebtext-gemma.

sourceHugging Faceupdated 2y agoView on Hugging Face

1likes780downloads

Dataset Card

OpenWebTextCorpus tokenized for Gemma

This dataset is a pre-tokenized version of the Skylion007/openwebtext dataset using the gemma tokenizer. As such, this dataset follows the same licensing as the original openwebtext dataset.

This pre-tokenization is done as a performance optimization for using the openwebtext dataset with a Gemma model (gemma-2b, gemma-2b-it, gemma-7b, gemma-7b-it). This dataset was created using SAELens, with the following settings:

—context_size: 8192
—shuffled: true
—beginbatchtoken: "bos"
—beginsequencetoken: null
—sequenceseparatortoken: "bos"
—saelensversion: "3.3.0"