text-machine-lab/vocab_filtered_dataset_22B

Dataset Card for "vocabfiltereddataset_22B"

Dataset Description

• —Paper: https://arxiv.org/abs/2404.02204
• —Point of Contact: sherinbojappa_muckatira@student.uml.edu

Dataset Summary

This data is the simplified vocabulary-filtered pretraining data published by "Emergent Abilities in Reduced-Scale Generative Language Models". The vocabulary is derived from the AO-Childes speech corpus (https://github.com/UIUCLearningLanguageLab/AOCHILDES) We filter the train split of SlimPajama dataset (https://huggingface.co/datasets/cerebras/SlimPajama-627B) based on the AO-Childes vocabulary retaining spans which contain integers, symbols, and words that belong to the AO-Childes vocabulary. Around 1.5% of Out of Vocabulary words are also allowed. A contiguous span of 32 tokens are selected. A span is delimited by start of span <s> and end of span </s> symbols.

Citation Information

If this dataset is useful to you please cite our work.

@misc{muckatira2024emergent,
      title={Emergent Abilities in Reduced-Scale Generative Language Models},
      author={Sherin Muckatira and Vijeta Deshpande and Vladislav Lialin and Anna Rumshisky},
      year={2024},
      eprint={2404.02204},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}