Modelpublic

gaunernst/bert-tiny-uncased

sourceHugging Faceapache-2.0updated 2y agoView on Hugging Face

6likes686downloads

Model Card

BERT Tiny (uncased)

Mini BERT models from https://arxiv.org/abs/1908.08962 that the HF team didn't convert. The original conversion script is used.

See the original Google repo: google-research/bert

Note: it's not clear if these checkpoints have undergone knowledge distillation.

Model variants

	H=128	H=256	H=512	H=768
L=2	[2/128 (BERT-Tiny)][2_128]	[2/256][2_256]	[2/512][2_512]	[2/768][2_768]
L=4	[4/128][4_128]	[4/256 (BERT-Mini)][4_256]	[4/512 (BERT-Small)][4_512]	[4/768][4_768]
L=6	[6/128][6_128]	[6/256][6_256]	[6/512][6_512]	[6/768][6_768]
L=8	[8/128][8_128]	[8/256][8_256]	[8/512 (BERT-Medium)][8_512]	[8/768][8_768]
L=10	[10/128][10_128]	[10/256][10_256]	[10/512][10_512]	[10/768][10_768]
L=12	[12/128][12_128]	[12/256][12_256]	[12/512][12_512]	[12/768 (BERT-Base, original)][12_768]

[2128]: https://huggingface.co/gaunernst/bert-tiny-uncased [2256]: https://huggingface.co/gaunernst/bert-L2-H256-uncased [2512]: https://huggingface.co/gaunernst/bert-L2-H512-uncased [2768]: https://huggingface.co/gaunernst/bert-L2-H768-uncased [4128]: https://huggingface.co/gaunernst/bert-L4-H128-uncased [4256]: https://huggingface.co/gaunernst/bert-mini-uncased [4512]: https://huggingface.co/gaunernst/bert-small-uncased [4768]: https://huggingface.co/gaunernst/bert-L4-H768-uncased [6128]: https://huggingface.co/gaunernst/bert-L6-H128-uncased [6256]: https://huggingface.co/gaunernst/bert-L6-H256-uncased [6512]: https://huggingface.co/gaunernst/bert-L6-H512-uncased [6768]: https://huggingface.co/gaunernst/bert-L6-H768-uncased [8128]: https://huggingface.co/gaunernst/bert-L8-H128-uncased [8256]: https://huggingface.co/gaunernst/bert-L8-H256-uncased [8512]: https://huggingface.co/gaunernst/bert-medium-uncased [8768]: https://huggingface.co/gaunernst/bert-L8-H768-uncased [10128]: https://huggingface.co/gaunernst/bert-L10-H128-uncased [10256]: https://huggingface.co/gaunernst/bert-L10-H256-uncased [10512]: https://huggingface.co/gaunernst/bert-L10-H512-uncased [10768]: https://huggingface.co/gaunernst/bert-L10-H768-uncased [12128]: https://huggingface.co/gaunernst/bert-L12-H128-uncased [12256]: https://huggingface.co/gaunernst/bert-L12-H256-uncased [12512]: https://huggingface.co/gaunernst/bert-L12-H512-uncased [12768]: https://huggingface.co/bert-base-uncased

Usage

See other BERT model cards e.g. https://huggingface.co/bert-base-uncased

Citation

bibtex

@article{turc2019,
  title={Well-Read Students Learn Better: On the Importance of Pre-training Compact Models},
  author={Turc, Iulia and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},
  journal={arXiv preprint arXiv:1908.08962v2 },
  year={2019}
}