allenai/OLMoE-mix-0924

OLMoE Mix (September 2024)

Dataset Description

• —Repository: https://github.com/allenai/OLMoE
• —Paper: OLMoE: Open Mixture-of-Experts Language Models

<img alt="OLMoE Mix Logo." src="olmoe-mix.png" width="250px">

The following data mix was used to train OLMoE-1B-7B, a Mixture-of-Experts LLM with 1B active and 7B total parameters released in September 2024.

The base version of OLMoE-1B-7B can be found at this page, the SFT of OLMoE-1B-7B is available here, and a version combining SFT and DPO is available following this link.

Statistics

Preprocessing

All subsets were pre-processed to remove documents with a sequence of 32 or more repeated ngrams.

• —a ngram is a span of 1 to 13 tokens, included;
• —tokens are obtained using the model tokenizer;
• —a sequence is a contiguous span of repeated ngrams.

In addition of the above, Starcoder dataset was further processed by removing any document meeting any of the following rules:

• —document is from a repository with fewer than 2 stars on GitHub;
• —the top most frequent word in the document constitutes over 30% of the document;
• —the two most frequent words in the document constitutes over 50% of the document.

Licensing Information

This mix is licensed under Open Data Commons Attribution License (ODC-By) v1.0. By using this dataset, you are bound to licenses and Terms of Services of underlying datasets, which you can access by clicking on the links in the table above.

Citation

@misc{muennighoff2024olmoeopenmixtureofexpertslanguage,
      title={OLMoE: Open Mixture-of-Experts Language Models}, 
      author={Niklas Muennighoff and Luca Soldaini and Dirk Groeneveld and Kyle Lo and Jacob Morrison and Sewon Min and Weijia Shi and Pete Walsh and Oyvind Tafjord and Nathan Lambert and Yuling Gu and Shane Arora and Akshita Bhagia and Dustin Schwenk and David Wadden and Alexander Wettig and Binyuan Hui and Tim Dettmers and Douwe Kiela and Ali Farhadi and Noah A. Smith and Pang Wei Koh and Amanpreet Singh and Hannaneh Hajishirzi},
      year={2024},
      eprint={2409.02060},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2409.02060}, 
}