Alibaba-NLP/gte-multilingual-mlm-base

gte-multilingual-mlm-base

We introduce mGTE series, new generalized text encoder, embedding and reranking models that support 75 languages and the context length of up to 8192. The models are built upon the transformer++ encoder backbone (BERT + RoPE + GLU, code refer to Alibaba-NLP/new-impl) as well as the vocabulary of XLM-R.

This text encoder (mGTE-MLM-8192 in our paper) outperforms the same-sized previous state-of-the-art XLM-R-base in both GLUE and XTREME-R.

• —Developed by: Institute for Intelligent Computing, Alibaba Group
• —Model type: Text Encoder
• —Paper: mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval.

Model list

Training Details

Training Data

• —Masked language modeling (MLM): c4-en, mc4, skypile, Wikipedia, CulturaX, etc (refer to paper appendix A.1)

Training Procedure

To enable the backbone model to support a context length of 8192, we adopted a multi-stage training strategy. The model first undergoes preliminary MLM pre-training on shorter lengths. And then, we resample the data, reducing the proportion of short texts, and continue the MLM pre-training.

The entire training process is as follows:

• —MLM-2048: lr 2e-4, mlmprobability 0.3, batchsize 8192, numsteps 250k, ropebase 10000
• —MLM-8192: lr 5e-5, mlmprobability 0.3, batchsize 2048, numsteps 30k, ropebase 160000

Evaluation

Citation

If you find our paper or models helpful, please consider citing them as follows:

@misc{zhang2024mgtegeneralizedlongcontexttext,
  title={mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval}, 
  author={Xin Zhang and Yanzhao Zhang and Dingkun Long and Wen Xie and Ziqi Dai and Jialong Tang and Huan Lin and Baosong Yang and Pengjun Xie and Fei Huang and Meishan Zhang and Wenjie Li and Min Zhang},
  year={2024},
  eprint={2407.19669},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2407.19669}, 
}