CoolFace
Modelpublic

aisingapore/sealion-bert-base

sourceHugging Facemitupdated 3y agoView on Hugging Face
5likes242downloads
Model Card

SEA-LION-BERT

SEA-LION stands for <i>Southeast Asian Languages In One Network</i>.

This is the card for the SEA-LION-BERT base model.

How To Use

python
from transformers import AutoModelForMaskedLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained('aisingapore/sealion-bert-base', trust_remote_code=True)
model = AutoModelForMaskedLM.from_pretrained('aisingapore/sealion-bert-base', trust_remote_code=True)

# prepare input
text = "Give me a <|mask|>!!!"
encoded_input = tokenizer(text, return_tensors='pt')

Model Details

Model Description

The SEA-LION-BERT model is built on the MosaicBERT architecture and has a vocabulary size of 256K.

For tokenization, the model employs our custom SEABPETokenizer, which is specially tailored for SEA languages, ensuring optimal model performance.

The training data for SEA-LION-BERT encompasses 790B tokens.

  • —Developed by: Products Pillar, AI Singapore
  • —Funded by: Singapore NRF
  • —Model type: Encoder
  • —Languages: English, Chinese, Indonesian, Malay, Thai, Vietnamese, Filipino, Tamil, Burmese, Khmer, Lao
  • —License: MIT License

Training Details

Data

SEA-LION was trained on 790B tokens of the following data:

Data SourceTokensPercentage
RefinedWeb - English571.3B72.26%
mC4 - Chinese91.2B11.54%
mC4 - Indonesian14.7B1.86%
mC4 - Malay2.9B0.36%
mC4 - Filipino5.3B0.67%
mC4 - Burmese4.9B0.61%
mC4 - Vietnamese63.4B8.02%
mC4 - Thai21.6B2.74%
mC4 - Lao1.1B0.14%
mC4 - Khmer3.9B0.50%
mC4 - Tamil10.2B1.29%

Infrastructure

SEA-LION was trained using MosaicML Composer on the following hardware:

Training DetailsSEA-LION-BERT
Nvidia A100 40GB GPU4
Training Duration14 days

Configuration

HyperParameterSEA-LION-BERT
Precisionbfloat16
Optimizerdecoupled_adamw
Schedulerlineardecaywith_warmup
Learning Rate5e-4
Global Batch Size448
Micro Batch Size56

Technical Specifications

Model Architecture and Objective

SEA-LION-BERT is an encoder model using the MosaicBERT architecture.

ParameterSEA-LION-BERT
Layers12
d_model768
head_dim12
Vocabulary256000
Sequence Length128

Tokenizer Details

We sample 20M lines from the training data to train the tokenizer.<br> The framework for training is SentencePiece.<br> The tokenizer type is Byte-Pair Encoding (BPE).

The Team

Montalan Jann Railey<br> Nguyen Thanh Ngan<br> Rengarajan Hamsawardhini<br> Teo Eng Sipp Leslie<br> Tjhi William<br>

Acknowledgements

AI Singapore is a national programme supported by the National Research Foundation, Singapore and hosted by the National University of Singapore. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore.

Contact

For more info, please contact us using this SEA-LION Inquiry Form