bigscience/bloom-1b7-intermediate
0131
1---2license: bigscience-bloom-rail-1.03language:4- ak5- ar6- as7- bm8- bn9- ca10- code11- en12- es13- eu14- fon15- fr16- gu17- hi18- id19- ig20- ki21- kn22- lg23- ln24- ml25- mr26- ne27- nso28- ny29- or30- pa31- pt32- rn33- rw34- sn35- st36- sw37- ta38- te39- tn40- ts41- tum42- tw43- ur44- vi45- wo46- xh47- yo48- zh49- zhs50- zht51- zu52---53 54# <span style="color:red"><b>WARNING:</b> The checkpoints on this repo are not fully trained model. Evaluations of intermediary checkpoints and the final model will be added when conducted (see below).</span>55 56# <p>BLOOM LM<br/> _BigScience Large Open-science Open-access Multilingual Language Model_ <br/>Model Card</p>57<img src="https://assets.website-files.com/6139f3cdcbbff3a68486761d/613cd8997b270da063e230c5_Tekengebied%201-p-500.png" alt="BigScience Logo" width="200"/>58 59 60Version 1.3 / 11.July.2022 - Available intermediary checkpoints - global steps: 61 62+ `1000`, `10000`, `50000`, `100000`, `150000`, `200000`, `250000`, `300000`63 64You can check the available checkpoints by clicking on the branches section of the repo65 66# How to load a specific version67 68We use `git tags` to load a model in a specific version (eg. `global_step1000`):69 70```python71from transformers import AutoModelForCausalLM72model = AutoModelForCausalLM.from_pretrained(73 "bigscience/bloom-1b3-intermediate",74 revision="global_step1000",75 torch_dtype="auto",76)77```78 79 80 81# Table of Contents821. [Model Details](#model-details)832. [Uses](#uses)843. [Training Data](#training-data)854. [Risks and Limitations](#risks-and-limitations)865. [Evaluation](#evaluation)876. [Recommendations](#recommendations)887. [Glossary and Calculations](#glossary-and-calculations)898. [More Information](#more-information)909. [Model Card Authors](#model-card-authors)91 92---93 94# Model Details 95 96BLOOM is a type of language model, which is a probability distribution over sequences of words. Specifically, BLOOM is a Large Language Model (LLM), meaning that it is trained on vast amounts of text data using industrial-scale computational resources. As such, the model is able to capture the statistical tendencies of words, phrases, sentences, and larger spans of text that it is exposed to in the training data.97 98## Basics99*This section provides information about the model type, version, license, funders, release date, developers, and contact information.*100*It is useful for anyone who wants to reference the model.*101 102<details>103<summary>Click to expand</summary>104 105**Developed by:** BigScience ([website](https://bigscience.huggingface.co))106 107*All collaborators are either volunteers or have an agreement with their employer. (Further breakdown of participants forthcoming.)*108 109**Model Type:** Transformer-based Language Model110 111**Version:** 1.0.0112 113**Languages:** Multiple; see [training data](#training-data)114 115**License:** RAIL License v1.0 ([link](https://huggingface.co/spaces/bigscience/license))116 117**Release Date Estimate:** Monday, 11.July.2022118 119**Send Questions to:** bigscience-contact@googlegroups.com120 121**Cite as:** BigScience, _BigScience Language Open-science Open-access Multilingual (BLOOM) Language Model_. International, May 2021-May 2022122 123**Funded by:** 124 125* The French government.126 127* Hugging Face ([website](https://huggingface.co)).128 129* Organizations of contributors. *(Further breakdown of organizations forthcoming.)*130 131</details>132 133## Technical Specifications134*This section includes details about the model objective and architecture, and the compute infrastructure.*135*It is useful for people interested in model development.*136 137<details>138<summary>Click to expand</summary>139 140Please see [the BLOOM training README](https://github.com/bigscience-workshop/bigscience/tree/master/train/tr11-176B-ml#readme) for full details on replicating training.141 142### Model Architecture and Objective143 144* Modified from Megatron-LM GPT2 (see [paper](https://arxiv.org/abs/1909.08053), [BLOOM Megatron code](https://github.com/bigscience-workshop/Megatron-DeepSpeed)):145 146* Decoder-only architecture147 148* Layer normalization applied to word embeddings layer (`StableEmbedding`; see [code](https://github.com/facebookresearch/bitsandbytes), [paper](https://arxiv.org/pdf/2110.02861.pdf))149 150* ALiBI positional encodings (see [paper](https://arxiv.org/pdf/2108.12409.pdf)), with GeLU activation functions151 152* 176 billion parameters:153 154 * 70 layers, 112 attention heads155 156 * Hidden layers are 14336-dimensional157 158 * Sequence length of 2048 tokens used (see [BLOOM tokenizer](https://huggingface.co/bigscience/tokenizer), [tokenizer description](#tokenization))159 160**Objective Function:** Cross Entropy with mean reduction (see [API documentation](https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html#torch.nn.CrossEntropyLoss)).161 162### Compute infrastructure163Jean Zay Public Supercomputer, provided by the French government (see [announcement](https://www.enseignementsup-recherche.gouv.fr/fr/signature-du-marche-d-acquisition-de-l-un-des-supercalculateurs-les-plus-puissants-d-europe-46733)).164 165#### Hardware166 167* 384 A100 80GB GPUs (48 nodes)168 169* Additional 32 A100 80GB GPUs (4 nodes) in reserve170 171* 8 GPUs per node Using NVLink 4 inter-gpu connects, 4 OmniPath links172 173* CPU: AMD174 175* CPU memory: 512GB per node176 177* GPU memory: 640GB per node178 179* Inter-node connect: Omni-Path Architecture (OPA)180 181* NCCL-communications network: a fully dedicated subnet182 183* Disc IO network: shared network with other types of nodes184 185#### Software186 187* Megatron-DeepSpeed ([Github link](https://github.com/bigscience-workshop/Megatron-DeepSpeed))188 189* DeepSpeed ([Github link](https://github.com/microsoft/DeepSpeed))190 191* PyTorch (pytorch-1.11 w/ CUDA-11.5; see [Github link](https://github.com/pytorch/pytorch))192 193* apex ([Github link](https://github.com/NVIDIA/apex))194 195</details>196 197---198 199# Training200*This section provides information about the training data, the speed and size of training elements, and the environmental impact of training.*201*It is useful for people who want to learn more about the model inputs and training footprint.*202 203<details>204<summary>Click to expand</summary>205 206## Training Data207*This section provides a high-level overview of the training data. It is relevant for anyone who wants to know the basics of what the model is learning.*208 209Details for each dataset are provided in individual [Data Cards](https://huggingface.co/spaces/bigscience/BigScienceCorpus).210 211Training data includes:212 213- 45 natural languages214 215- 12 programming languages216 217- In 1.5TB of pre-processed text, converted into 350B unique tokens (see [the tokenizer section](#tokenization) for more.)218 219### Languages220 221The pie chart shows the distribution of languages in training data.222 223224 225 226The following tables shows the further distribution of Niger-Congo & Indic languages and programming languages in the training data.227 228Distribution of Niger Congo and Indic languages.229 230| Niger Congo | Percentage | | Indic | Percentage |231|----------------|------------ |------ |-----------|------------|232| Chi Tumbuka | 0.00002 | | Assamese | 0.01 |233| Kikuyu | 0.00004 | | Odia | 0.04 |234| Bambara | 0.00004 | | Gujarati | 0.04 |235| Akan | 0.00007 | | Marathi | 0.05 |236| Xitsonga | 0.00007 | | Punjabi | 0.05 |237| Sesotho | 0.00007 | | Kannada | 0.06 |238| Chi Chewa | 0.0001 | | Nepali | 0.07 |239| Setswana | 0.0002 | | Telugu | 0.09 |240| Northern Sotho | 0.0002 | | Malayalam | 0.10 |241| Fon | 0.0002 | | Urdu | 0.10 |242| Kirundi | 0.0003 | | Tamil | 0.20 |243| Wolof | 0.0004 | | Bengali | 0.50 |244| Kuganda | 0.0004 | | Hindi | 0.70 |245| Chi Shona | 0.001 |246| Isi Zulu | 0.001 |247| Igbo | 0.001 |248| Xhosa | 0.001 |249| Kinyarwanda | 0.003 |250| Yoruba | 0.006 |251| Swahili | 0.02 |252 253Distribution of programming languages.254 255| Extension | Language | Number of files |256|----------------|------------|-----------------|257| java | Java | 5,407,724 |258| php | PHP | 4,942,186 |259| cpp | C++ | 2,503,930 |260| py | Python | 2,435,072 |261| js | JavaScript | 1,905,518 |262| cs | C# | 1,577,347 |263| rb | Ruby | 6,78,413 |264| cc | C++ | 443,054 |265| hpp | C++ | 391,048 |266| lua | Lua | 352,317 |267| go | GO | 227,763 |268| ts | TypeScript | 195,254 |269| C | C | 134,537 |270| scala | Scala | 92,052 |271| hh | C++ | 67,161 |272| H | C++ | 55,899 |273| tsx | TypeScript | 33,107 |274| rs | Rust | 29,693 |275| phpt | PHP | 9,702 |276| c++ | C++ | 1,342 |277| h++ | C++ | 791 |278| php3 | PHP | 540 |279| phps | PHP | 270 |280| php5 | PHP | 166 |281| php4 | PHP | 29 |282 283### Preprocessing284 285**Tokenization:** The BLOOM tokenizer ([link](https://huggingface.co/bigscience/tokenizer)), a learned subword tokenizer trained using:286 287- A byte-level Byte Pair Encoding (BPE) algorithm 288 289- A simple pre-tokenization rule, no normalization290 291- A vocabulary size of 250,680292 293It was trained on a subset of a preliminary version of the corpus using alpha-weighting per language. 294 295## Speeds, Sizes, Times296 297Training logs: [Tensorboard link](https://huggingface.co/tensorboard/bigscience/tr11-176B-ml-logs/)298 299- Dates:300 301 - Started 11th March, 2022 11:42am PST302 303 - Estimated end: 5th July, 2022304 305- Checkpoint size:306 307 - Bf16 weights: 329GB308 309 - Full checkpoint with optimizer states: 2.3TB310 311- Training throughput: About 150 TFLOP per GPU per second312 313- Number of epochs: 1314 315- Estimated cost of training: Equivalent of $2-5M in cloud computing (including preliminary experiments)316 317- Server training location: Île-de-France, France318 319 320## Environmental Impact321 322The training supercomputer, Jean Zay ([website](http://www.idris.fr/eng/jean-zay/jean-zay-presentation-eng.html)), uses mostly nuclear energy. The heat generated by it is reused for heating campus housing.323 324**Estimated carbon emissions:** *(Forthcoming.)*325 326**Estimated electricity usage:** *(Forthcoming.)*327 328</details>329 330---331 332# Uses333 334*This section addresses questions around how the model is intended to be used, discusses the foreseeable users of the model (including those affected by the model), and describes uses that are considered out of scope or misuse of the model.*335*It is useful for anyone considering using the model or who is affected by the model.*336 337<details>338<summary>Click to expand</summary>339 340## Intended Use341 342This model is being created in order to enable public research on large language models (LLMs). LLMs are intended to be used for language generation or as a pretrained base model that can be further fine-tuned for specific tasks. Use cases below are not exhaustive.343 344### Direct Use345 346- Text generation347 348- Exploring characteristics of language generated by a language model349 350 - Examples: Cloze tests, counterfactuals, generations with reframings351 352### Downstream Use353 354- Tasks that leverage language models include: Information Extraction, Question Answering, Summarization355 356### Misuse and Out-of-scope Use357*This section addresses what users ought not do with the model.*358 359See the [BLOOM License](https://huggingface.co/spaces/bigscience/license), Attachment A, for detailed usage restrictions. The below list is non-exhaustive, but lists some easily foreseeable problematic use cases.360 361#### Out-of-scope Uses362 363Using the model in [high-stakes](#high-stakes) settings is out of scope for this model. The model is not designed for [critical decisions](#critical-decisions) nor uses with any material consequences on an individual's livelihood or wellbeing. The model outputs content that appears factual but is not correct. 364 365Out-of-scope Uses Include:366 367- Usage in biomedical domains, political and legal domains, or finance domains368 369- Usage for evaluating or scoring individuals, such as for employment, education, or credit370 371- Applying the model for critical automatic decisions, generating factual content, creating reliable summaries, or generating predictions that must be correct372 373#### Misuse374 375Intentionally using the model for harm, violating [human rights](#human-rights), or other kinds of malicious activities, is a misuse of this model. This includes:376 377- Spam generation378 379- Disinformation and influence operations380 381- Disparagement and defamation382 383- Harassment and abuse384 385- [Deception](#deception)386 387- Unconsented impersonation and imitation388 389- Unconsented surveillance 390 391- Generating content without attribution to the model, as specified in the [RAIL License, Use Restrictions](https://huggingface.co/spaces/bigscience/license)392 393## Intended Users394 395### Direct Users396 397- General Public398 399- Researchers400 401- Students402 403- Educators404 405- Engineers/developers406 407- Non-commercial entities408 409- Community advocates, including human and civil rights groups410 411### Indirect Users412 413- Users of derivatives created by Direct Users, such as those using software with an [intended use](#intended-use)414 415- Users of [Derivatives of the Model, as described in the License](https://huggingface.co/spaces/bigscience/license)416 417### Others Affected (Parties Prenantes)418 419- People and groups referred to by the LLM420 421- People and groups exposed to outputs of, or decisions based on, the LLM422 423- People and groups whose original work is included in the LLM424 425</details>426 427---428 429# Risks and Limitations430*This section identifies foreseeable harms and misunderstandings.*431 432<details>433<summary>Click to expand</summary>434 435Model may:436 437- Overrepresent some viewpoints and underrepresent others438 439- Contain stereotypes440 441- Contain [personal information](#personal-data-and-information)442 443- Generate:444 445 - Hateful, abusive, or violent language446 447 - Discriminatory or prejudicial language448 449 - Content that may not be appropriate for all settings, including sexual content450 451- Make errors, including producing incorrect information as if it were factual452 453- Generate irrelevant or repetitive outputs454 455</details>456 457---458 459# Evaluation460*This section describes the evaluation protocols and provides the results.*461 462 463<details>464<summary>Click to expand</summary>465 466## Metrics 467*This section describes the different ways performance is calculated and why.*468 469 470Includes:471 472| Metric | Why chosen |473|--------------------|--------------------------------------------------------------------|474| [Perplexity](#perplexity) | Standard metric for quantifying model improvements during training |475| Cross Entropy [Loss](#loss) | Standard objective for language models. |476 477And multiple different metrics for specific tasks. _(More evaluation metrics forthcoming upon completion of evaluation protocol.)_478 479## Factors 480*This section lists some different aspects of what BLOOM models. Its focus is on those aspects that are likely to give rise to high variance in model behavior.*481 482- Language, such as English or Yoruba483 484- Domain, such as newswire or stories485 486- Demographic characteristics, such as gender or nationality487 488## Results489*Results are based on the [Factors](#factors) and [Metrics](#metrics).*490 491**Train-time Evaluation:**492 493As of 25.May.2022, 15:00 PST:494 495- Training Loss: 2.0496 497- Validation Loss: 2.2498 499- Perplexity: 8.9500 501(More evaluation scores forthcoming.)502 503</details>504 505---506 507# Recommendations508 509*This section provides information on warnings and potential mitigations.*510 511<details>512<summary>Click to expand</summary>513 514- Indirect users should be made aware when the content they're working with is created by the LLM.515 516- Users should be aware of [Risks and Limitations](#risks-and-limitations), and include an appropriate age disclaimer or blocking interface as necessary.517 518- Models trained or finetuned downstream of BLOOM LM should include an updated Model Card.519 520- Users of the model should provide mechanisms for those affected to provide feedback, such as an email address for comments.521 522</details>523 524---525 526# Glossary and Calculations527 528*This section defines common terms and how metrics are calculated.*529<details>530<summary>Click to expand</summary>531 532- <a name="loss">**Loss:**</a> A calculation of the difference between what the model has learned and what the data shows ("groundtruth"). The lower the loss, the better. The training process aims to minimize the loss. 533 534- <a name="perplexity">**Perplexity:**</a> This is based on what the model estimates the probability of new data is. The lower the perplexity, the better. If the model is 100% correct at predicting the next token it will see, then the perplexity is 1. Mathematically this is calculated using entropy. 535 536- <a name="high-stakes">**High-stakes settings:**</a> Such as those identified as "high-risk AI systems" and "unacceptable risk AI systems" in the European Union's proposed [Artificial Intelligence (AI) Act](https://artificialintelligenceact.eu/annexes/).537 538- <a name="critical-decisions">**Critical decisions:**</a> Such as those defined in [the United States' proposed Algorithmic Accountability Act](https://www.congress.gov/117/bills/s3572/BILLS-117s3572is.pdf).539 540- <a name="human-rights">**Human rights:**</a> Includes those rights defined in the [Universal Declaration of Human Rights](https://www.un.org/sites/un2.un.org/files/2021/03/udhr.pdf).541 542- <a name="personal-data-and-information">**Personal Data and Personal Information:**</a> Personal data and information is defined in multiple data protection regulations, such as "[personal data](https://gdpr-info.eu/issues/personal-data/)" in the [European Union's General Data Protection Regulation](https://gdpr-info.eu); and "personal information" in the Republic of South Africa's [Protection of Personal Information Act](https://www.gov.za/sites/default/files/gcis_document/201409/3706726-11act4of2013popi.pdf), The People's Republic of China's [Personal information protection law](http://en.npc.gov.cn.cdurl.cn/2021-12/29/c_694559.htm).543 544- <a name="sensitive-characteristics">**Sensitive characteristics:**</a> This includes specifically protected categories in human rights (see [UHDR, Article 2](https://www.un.org/sites/un2.un.org/files/2021/03/udhr.pdf)) and personal information regulation (see GDPR, [Article 9; Protection of Personal Information Act, Chapter 1](https://www.gov.za/sites/default/files/gcis_document/201409/3706726-11act4of2013popi.pdf))545 546- <a name="deception">**Deception:**</a> Doing something to intentionally mislead individuals to believe something that is false, such as by creating deadbots or chatbots on social media posing as real people, or generating text documents without making consumers aware that the text is machine generated.547 548</details>549 550---551 552# More Information553*This section provides links to writing on dataset creation, technical specifications, lessons learned, and initial results.*554 555<details>556<summary>Click to expand</summary>557 558## Dataset Creation559 560Blog post detailing the design choices during the dataset creation: https://bigscience.huggingface.co/blog/building-a-tb-scale-multilingual-dataset-for-language-modeling561 562## Technical Specifications563 564Blog post summarizing how the architecture, size, shape, and pre-training duration where selected: https://bigscience.huggingface.co/blog/what-language-model-to-train-if-you-have-two-million-gpu-hours565 566More details on the architecture/optimizer: https://github.com/bigscience-workshop/bigscience/tree/master/train/tr11-176B-ml567 568Blog post on the hardware/engineering side: https://bigscience.huggingface.co/blog/which-hardware-to-train-a-176b-parameters-model569 570Details on the distributed setup used for the training: https://github.com/bigscience-workshop/bigscience/tree/master/train/tr11-176B-ml571 572Tensorboard updated during the training: https://huggingface.co/bigscience/tr11-176B-ml-logs/tensorboard#scalars&tagFilter=loss573 574## Lessons575 576Insights on how to approach training, negative results: https://github.com/bigscience-workshop/bigscience/blob/master/train/lessons-learned.md577 578Details on the obstacles overcome during the preparation on the engineering side (instabilities, optimization of training throughput, so many technical tricks and questions): https://github.com/bigscience-workshop/bigscience/blob/master/train/tr11-176B-ml/chronicles.md579 580## Initial Results581 582Initial prompting experiments using interim checkpoints: https://huggingface.co/spaces/bigscience/bloom-book583 584</details>585 586---587 588# Model Card Authors589*Ordered roughly chronologically and by amount of time spent.*590 591Margaret Mitchell, Giada Pistilli, Yacine Jernite, Ezinwanne Ozoani, Marissa Gerchick, Nazneen Rajani, Sasha Luccioni, Irene Solaiman, Maraim Masoud, Somaieh Nikpoor, Carlos Muñoz Ferrandis, Stas Bekman, Christopher Akiki, Danish Contractor, David Lansky, Angelina McMillan-Major, Tristan Thrush, Suzana Ilić, Gérard Dupont, Shayne Longpre, Manan Dey, Stella Biderman, Douwe Kiela, Emi Baylor, Teven Le Scao, Aaron Gokaslan, Julien Launay, Niklas Muennighoff