lightonai
multilingual-embeddings-pre-training-curated
📚 Collection | 📝 Multilingual Blog | 📝 English Blog
Contrastive Multilingual Pre-Training
2.16B query–document pairs across eight languages, plus cross-lingual pairs
mDenseOn |
mLateOn |
DenseOn |
LateOn |
PyLate |
FastPlaid
🎯 TL;DR: The multilingual contrastive pre-training corpus used to train mDenseOn and mLateOn. It extends our curated English data recipe (embeddings-pre-training-curated) to French, German, Italian, Spanish, Portuguese, Swedish, Norwegian, and Arabic… See the full description on the dataset page: https://huggingface.co/datasets/lightonai/multilingual-embeddings-pre-training-curated.embeddings-pre-training
Overview
This large-scale dataset is designed for pre-training state-of-the-art text embedding models. Its goal is to reproduce and build upon the data recipe described in the mGTE technical report (Zhang et al., 2024), which details the data sources used to train the GTE family of embedding models but does not release the data itself.
We assembled this dataset as part of a research effort to understand how data composition affects retrieval model quality. Our experiments… See the full description on the dataset page: https://huggingface.co/datasets/lightonai/embeddings-pre-training.lighton-ms-marco-mini
ms-marco-mini
This dataset gathers very few samples from MS MARCO to provide an example of triplet-based / knowledge distillation dataset formatting.
triplet subset
The triplet file is all we need to fine-tune a model based on contrastive loss.
Columns: "query", "positive", "negative"
Column types: str, str, str
Examples:{
"query": "what are the liberal arts?",
"positive": 'liberal arts. 1. the academic course of instruction at a college intended to provide general… See the full description on the dataset page: https://huggingface.co/datasets/lightonai/lighton-ms-marco-mini.embeddings-pre-training-curated
Embeddings pre-training curated data
This dataset is the English subset of lightonai/embeddings-pre-training, assembled to reproduce the English data recipe described in the mGTE technical report (Zhang et al., 2024).
The mGTE paper describes the data sources used to train the GTE family of multilingual text embedding and reranking models, but does not release the data itself. This dataset is our reconstruction of the English portion of that recipe, curated as part of a research… See the full description on the dataset page: https://huggingface.co/datasets/lightonai/embeddings-pre-training-curated.cornstackThis dataset is a copy of the CoRNStack data from Nomic. This is just a reformating of the data to make it easily usable in PyLate and sentence-transformers. The data has been created from the Stack v2 by using docstring as queries and corresponding functions as target documents. Other functions has been mined to provide hard negatives. For more information about the creation of the dataset, please refer to the original paper and please considering citing them if you use this data.
Notably… See the full description on the dataset page: https://huggingface.co/datasets/lightonai/cornstack.embeddings-fine-tuning
Overview
This dataset is composed of high quality data sources with mined hard negatives. It can be used to train a strong retrieval model by itself but is better used after a large-scale contrastive pre-training, for example using this dataset or its curated version.
This dataset has originally been created to follow the nv-retrieve setup, that mines the closest negatives to the query in a dataset and filter false negatives if their bi-encoder similarity is higher than a… See the full description on the dataset page: https://huggingface.co/datasets/lightonai/embeddings-fine-tuning.