google/multiberts-seed_2-step_80k

1---2language: en3tags:4-   multiberts5-   multiberts-seed_26-   multiberts-seed_2-step_80k7license: apache-2.08---9 10# MultiBERTs, Intermediate Checkpoint - Seed 2, Step 80k11 12MultiBERTs is a collection of checkpoints and a statistical library to support13robust research on BERT. We provide 25 BERT-base models trained with14similar hyper-parameters as15[the original BERT model](https://github.com/google-research/bert) but16with different random seeds, which causes variations in the initial weights and order of17training instances. The aim is to distinguish findings that apply to a specific18artifact (i.e., a particular instance of the model) from those that apply to the19more general procedure.20 21We also provide 140 intermediate checkpoints captured22during the course of pre-training (we saved 28 checkpoints for the first 5 runs).23 24The models were originally released through25[http://goo.gle/multiberts](http://goo.gle/multiberts). We describe them in our26paper27[The MultiBERTs: BERT Reproductions for Robustness Analysis](https://arxiv.org/abs/2106.16163).28 29This is model #2, captured at step 80k (max: 2000k, i.e., 2M steps).30 31## Model Description32 33This model was captured during a reproduction of34[BERT-base uncased](https://github.com/google-research/bert), for English: it35is a Transformers model pretrained on a large corpus of English data, using the36Masked Language Modelling (MLM) and the Next Sentence Prediction (NSP)37objectives.38 39The intended uses, limitations, training data and training procedure for the fully trained model are similar40to [BERT-base uncased](https://github.com/google-research/bert). Two major41differences with the original model:42 43*   We pre-trained the MultiBERTs models for 2 million steps using sequence44    length 512 (instead of 1 million steps using sequence length 128 then 512).45*   We used an alternative version of Wikipedia and Books Corpus, initially46    collected for [Turc et al., 2019](https://arxiv.org/abs/1908.08962).47 48This is a best-effort reproduction, and so it is probable that differences with49the original model have gone unnoticed. The performance of MultiBERTs on GLUE after full training is oftentimes comparable to that of original50BERT, but we found significant differences on the dev set of SQuAD (MultiBERTs outperforms original BERT).51See our [technical report](https://arxiv.org/abs/2106.16163) for more details.52 53### How to use54 55Using code from56[BERT-base uncased](https://huggingface.co/bert-base-uncased), here is an example based on57Tensorflow:58 59```60from transformers import BertTokenizer, TFBertModel61tokenizer = BertTokenizer.from_pretrained('google/multiberts-seed_2-step_80k')62model = TFBertModel.from_pretrained("google/multiberts-seed_2-step_80k")63text = "Replace me by any text you'd like."64encoded_input = tokenizer(text, return_tensors='tf')65output = model(encoded_input)66```67 68PyTorch version:69 70```71from transformers import BertTokenizer, BertModel72tokenizer = BertTokenizer.from_pretrained('google/multiberts-seed_2-step_80k')73model = BertModel.from_pretrained("google/multiberts-seed_2-step_80k")74text = "Replace me by any text you'd like."75encoded_input = tokenizer(text, return_tensors='pt')76output = model(**encoded_input)77```78 79## Citation info80 81```bibtex82@article{sellam2021multiberts,83  title={The MultiBERTs: BERT Reproductions for Robustness Analysis},84  author={Thibault Sellam and Steve Yadlowsky and Jason Wei and Naomi Saphra and Alexander D'Amour and Tal Linzen and Jasmijn Bastings and Iulia Turc and Jacob Eisenstein and Dipanjan Das and Ian Tenney and Ellie Pavlick},85  journal={arXiv preprint arXiv:2106.16163},86  year={2021}87}88```89