Modelpublic

tartuNLP/EstBERT

sourceHugging Facecc-by-4.0updated 2mo agoView on Hugging Face

4likes2.4kdownloads

Model Card

EstBERT

What's this?

The EstBERT model is a pretrained BERT<sub>Base</sub> model exclusively trained on Estonian cased corpus on both 128 and 512 sequence length of data.

How to use?

You can use the model with the transformers library the following way.

python

from transformers import AutoTokenizer, AutoModelForMaskedLM

tokenizer = AutoTokenizer.from_pretrained("tartuNLP/EstBERT")
model = AutoModelForMaskedLM.from_pretrained("tartuNLP/EstBERT")

You can also download the pretrained model from here, [EstBERT128]() [EstBERT512]()

Dataset used to train the model

The EstBERT model is trained both on 128 and 512 sequence length of data. For training the EstBERT we used the Estonian National Corpus 2017, which was the largest Estonian language corpus available at the time. It consists of four sub-corpora: Estonian Reference Corpus 1990-2008, Estonian Web Corpus 2013, Estonian Web Corpus 2017 and Estonian Wikipedia Corpus 2017.

Reference to cite

Tanvir et al 2021

Why would I use?

Overall EstBERT performs better in parts of speech (POS), name entity recognition (NER), rubric, and sentiment classification tasks compared to mBERT and XLM-RoBERTa. The comparative results can be found below;

Model	UPOS	XPOS	Morph	bf UPOS	bf XPOS	Morph
EstBERT	_97.89_	98.40	96.93	97.84	_98.43_	_96.80_
mBERT	97.42	98.06	96.24	97.43	98.13	96.13
XLM-RoBERTa	97.78	98.36	96.53	97.80	98.40	96.69

Model	Rubric<sub>128</sub>	Sentiment<sub>128</sub>	Rubric<sub>128</sub>	Sentiment<sub>512</sub>
EstBERT	_81.70_	74.36	80.96	74.50
mBERT	75.67	70.23	74.94	69.52
XLM\-RoBERTa	80.34	74.50	78.62	_76.07_

Model	Precicion<sub>128</sub>	Recall<sub>128</sub>	F1-Score<sub>128</sub>	Precision<sub>512</sub>	Recall<sub>512</sub>	F1-Score<sub>512</sub>
EstBERT	88.42	90.38	_89.39_	88.35	89.74	89.04
mBERT	85.88	87.09	86.51	_88.47_	88.28	88.37
XLM\-RoBERTa	87.55	_91.19_	89.34	87.50	90.76	89.10

BibTeX entry and citation info

@inproceedings{tanvir-etal-2021-estbert,
    title = {{EstBERT: A Pretrained Language-Specific BERT for Estonian}},
    author = "Tanvir, Hasan  and
      Kittask, Claudia  and
      Eiche, Sandra  and
      Sirts, Kairit",
    editor = "Dobnik, Simon  and
      {\O}vrelid, Lilja",
    booktitle = "Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa)",
    year = "2021",
    address = "Reykjavik, Iceland (Online)",
    publisher = {Link{\"o}ping University Electronic Press, Sweden},
    url = "https://aclanthology.org/2021.nodalida-main.2/",
    pages = "11--19"
}

@misc{tanvir2020estbert,
      title={{EstBERT: A Pretrained Language-Specific BERT for Estonian}}, 
      author={Hasan Tanvir and Claudia Kittask and Kairit Sirts},
      year={2020},
      eprint={2011.04784},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}