awacke1/Bloom.Big.Science.Continual.Generator

Language Models 🗣️

🏆 Bloom sets new record for most performant and efficient AI model in science! 🌸

Comparison of Large Language Models

ChatGPT Datasets 📚

• —WebText
• —Common Crawl
• —BooksCorpus
• —English Wikipedia
• —Toronto Books Corpus
• —OpenWebText
• —

ChatGPT Datasets - Details 📚

• —WebText: A dataset of web pages crawled from domains on the Alexa top 5,000 list. This dataset was used to pretrain GPT-2.
• —WebText: A Large-Scale Unsupervised Text Corpus by Radford et al.
• —Common Crawl: A dataset of web pages from a variety of domains, which is updated regularly. This dataset was used to pretrain GPT-3.
• —Language Models are Few-Shot Learners by Brown et al.
• —BooksCorpus: A dataset of over 11,000 books from a variety of genres.
• —Scalable Methods for 8 Billion Token Language Modeling by Zhu et al.
• —English Wikipedia: A dump of the English-language Wikipedia as of 2018, with articles from 2001-2017.
• —Improving Language Understanding by Generative Pre-Training Space for Wikipedia Search
• —Toronto Books Corpus: A dataset of over 7,000 books from a variety of genres, collected by the University of Toronto.
• —Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond by Schwenk and Douze.
• —OpenWebText: A dataset of web pages that were filtered to remove content that was likely to be low-quality or spammy. This dataset was used to pretrain GPT-3.
• —Language Models are Few-Shot Learners by Brown et al.

Big Science Model 🚀

• —📜 Papers:
• —BLOOM: A 176B-Parameter Open-Access Multilingual Language Model Paper
• —Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism Paper
• —8-bit Optimizers via Block-wise Quantization Paper
• —Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation Paper
• —Other papers related to Big Science
• —217 other models optimized for use with Bloom

• —📚 Datasets:

Datasets:

1. 1.- Universal Dependencies: A collection of annotated corpora for natural language processing in a range of languages, with a focus on dependency parsing.
2. 2.Universal Dependencies official website.
3. 3.- WMT 2014: The fourth edition of the Workshop on Statistical Machine Translation, featuring shared tasks on translating between English and various other languages.
4. 4.WMT14 website.
5. 5.- The Pile: An English language corpus of diverse text, sourced from various places on the internet.
6. 6.The Pile official website.
7. 7.- HumanEval: A dataset of English sentences, annotated with human judgments on a range of linguistic qualities.
8. 8.HumanEval: An Evaluation Benchmark for Language Understanding by Gabriel Ilharco, Daniel Loureiro, Pedro Rodriguez, and Afonso Mendes.
9. 9.- FLORES-101: A dataset of parallel sentences in 101 languages, designed for multilingual machine translation.
10. 10.FLORES-101: A Massively Multilingual Parallel Corpus for Language Understanding by Aman Madaan, Shruti Rijhwani, Raghav Gupta, and Mitesh M. Khapra.
11. 11.- CrowS-Pairs: A dataset of sentence pairs, designed for evaluating the plausibility of generated text.
12. 12.CrowS-Pairs: A Challenge Dataset for Plausible Plausibility Judgments by Andrea Madotto, Zhaojiang Lin, Chien-Sheng Wu, Pascale Fung, and Caiming Xiong.
13. 13.- WikiLingua: A dataset of parallel sentences in 75 languages, sourced from Wikipedia.
14. 14.WikiLingua: A New Benchmark Dataset for Cross-Lingual Wikification by Jiarui Yao, Yanqiao Zhu, Ruihan Bao, Guosheng Lin, Lidong Bing, and Bei Shi.
15. 15.- MTEB: A dataset of English sentences, annotated with their entailment relationships with respect to other sentences.
16. 16.Multi-Task Evaluation Benchmark for Natural Language Inference by Michał Lukasik, Marcin Junczys-Dowmunt, and Houda Bouamor.
17. 17.- xP3: A dataset of English sentences, annotated with their paraphrase relationships with respect to other sentences.
18. 18.xP3: A Large-Scale Evaluation Benchmark for Paraphrase Identification in Context by Aniket Didolkar, James Mayfield, Markus Saers, and Jason Baldridge.
19. 19.- DiaBLa: A dataset of English dialogue, annotated with dialogue acts.
20. 20.A Large-Scale Corpus for Conversation Disentanglement by Samuel Broscheit, António Branco, and André F. T. Martins.

• —📚 Dataset Papers with Code
• —Universal Dependencies
• —WMT 2014
• —The Pile
• —HumanEval
• —FLORES-101
• —CrowS-Pairs
• —WikiLingua
• —MTEB
• —xP3
• —DiaBLa

Deep RL ML Strategy 🧠

The AI strategies are:

• —Language Model Preparation using Human Augmented with Supervised Fine Tuning 🤖
• —Reward Model Training with Prompts Dataset Multi-Model Generate Data to Rank 🎁
• —Fine Tuning with Reinforcement Reward and Distance Distribution Regret Score 🎯
• —Proximal Policy Optimization Fine Tuning 🤝
• —Variations - Preference Model Pretraining 🤔
• —Use Ranking Datasets Sentiment - Thumbs Up/Down, Distribution 📊
• —Online Version Getting Feedback 💬
• —OpenAI - InstructGPT - Humans generate LM Training Text 🔍
• —DeepMind - Advantage Actor Critic Sparrow, GopherCite 🦜
• —Reward Model Human Prefence Feedback 🏆

For more information on specific techniques and implementations, check out the following resources:

• —OpenAI's paper on GPT-3 which details their Language Model Preparation approach
• —DeepMind's paper on SAC which describes the Advantage Actor Critic algorithm
• —OpenAI's paper on Reward Learning which explains their approach to training Reward Models
• —OpenAI's blog post on GPT-3's fine-tuning process