datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
multilingual-embeddings-pre-training-curated
📚 Collection | 📝 Multilingual Blog | 📝 English Blog
Contrastive Multilingual Pre-Training
2.16B query–document pairs across eight languages, plus cross-lingual pairs
mDenseOn |
mLateOn |
DenseOn |
LateOn |
PyLate |
FastPlaid
🎯 TL;DR: The multilingual contrastive pre-training corpus used to train mDenseOn and mLateOn. It extends our curated English data recipe (embeddings-pre-training-curated) to French, German, Italian, Spanish, Portuguese, Swedish, Norwegian, and Arabic… See the full description on the dataset page: https://huggingface.co/datasets/lightonai/multilingual-embeddings-pre-training-curated.lighton-ms-marco-mini
ms-marco-mini
This dataset gathers very few samples from MS MARCO to provide an example of triplet-based / knowledge distillation dataset formatting.
triplet subset
The triplet file is all we need to fine-tune a model based on contrastive loss.
Columns: "query", "positive", "negative"
Column types: str, str, str
Examples:{
"query": "what are the liberal arts?",
"positive": 'liberal arts. 1. the academic course of instruction at a college intended to provide general… See the full description on the dataset page: https://huggingface.co/datasets/lightonai/lighton-ms-marco-mini.cornstackThis dataset is a copy of the CoRNStack data from Nomic. This is just a reformating of the data to make it easily usable in PyLate and sentence-transformers. The data has been created from the Stack v2 by using docstring as queries and corresponding functions as target documents. Other functions has been mined to provide hard negatives. For more information about the creation of the dataset, please refer to the original paper and please considering citing them if you use this data.
Notably… See the full description on the dataset page: https://huggingface.co/datasets/lightonai/cornstack.embeddings-fine-tuning
Overview
This dataset is composed of high quality data sources with mined hard negatives. It can be used to train a strong retrieval model by itself but is better used after a large-scale contrastive pre-training, for example using this dataset or its curated version.
This dataset has originally been created to follow the nv-retrieve setup, that mines the closest negatives to the query in a dataset and filter false negatives if their bi-encoder similarity is higher than a… See the full description on the dataset page: https://huggingface.co/datasets/lightonai/embeddings-fine-tuning.nv-embed-supervised-distill-dedupnv-embed-supervised-distill-dedup-codeThis dataset is a collection of the CoIR training datasets. We mined 2048 negatives per queries using gte-modernbert-base in order and format the data in a query, documents, scores format so that anyone can perform nv-retriever type of filtering using their own threshold (and this is also the format knowledge distillation for PyLate).
Notably, this dataset has been used to perform the fine-tuning of the state-of-the-art late interaction LateOn-Code models. The boilerplate used to fine-tune… See the full description on the dataset page: https://huggingface.co/datasets/lightonai/nv-embed-supervised-distill-dedup-code.Dolci-Think-SFT-32B-Multilingual
Dolci-Think-SFT-32B-Multilingual
Dolci-Think-SFT-32B-Multilingual is a large-scale multilingual long chain-of-thought (CoT) reasoning corpus spanning six languages: English, French, German, Spanish, Chinese, and Swahili. Each sample includes a question, a long-form reasoning trace, and a final answer, all translated into the target language, with sequences up to 32,768 tokens.
It is released alongside the paper Rethinking the Multilingual Reasoning Gap with Layer Swap.… See the full description on the dataset page: https://huggingface.co/datasets/lightonai/Dolci-Think-SFT-32B-Multilingual.nv-embed-supervised-distillms-marco-en-bge-gemma
ms-marco-en-bge
This dataset contains the MS MARCO dataset with negatives mined using ColBERT and then scored by bge-reranker-v2-gemma.
It can be used to train a retrieval model using knowledge distillation, for example using PyLate.
knowledge distillation
To fine-tune a model using knowledge distillation loss we will need three distinct file:
Datasetsfrom datasets import load_dataset
train = load_dataset(
"lightonai/ms-marco-en-gemma",
"train"… See the full description on the dataset page: https://huggingface.co/datasets/lightonai/ms-marco-en-bge-gemma.LightOnOCR-mix-0126
LightOnOCR-mix-0126
LightOnOCR-mix-0126 is a large-scale OCR training dataset built via distillation: a strong vision–language model is prompted to produce naturally ordered full-page transcriptions (Markdown with LaTeX math spans and HTML tables) from rendered document pages. The dataset is designed as supervision for end-to-end OCR / document-understanding models that aim to output clean, human-readable text in a consistent format.
This repository releases the PDFA-derived… See the full description on the dataset page: https://huggingface.co/datasets/lightonai/LightOnOCR-mix-0126.climate-fever-decontaminated
climate-fever (Decontaminated)
A decontaminated version of the climate-fever dataset from the BEIR benchmark, with samples found in the mgte-en pre-training dataset removed.
Decontamination methodology
Contamination was detected using a two-pass approach against the full mgte-en dataset (484 GB, 1,235 parquet files):
Pass 1: Exact hash matching
All texts (queries and corpus documents) were normalized (lowercased, unicode NFKD, whitespace collapsed) and hashed… See the full description on the dataset page: https://huggingface.co/datasets/lightonai/climate-fever-decontaminated.nanobeir-multilingualThis multilingual collection is derived from the original English NanoBEIR datasets, which are smaller versions of BEIR datasets.
The compact size of these datasets makes them ideal for conducting quick and efficient evaluations during training.
To facilitate broader research in cross-lingual information retrieval, our dataset has been machine-translated from the original English
into eight additional languages: Arabic (ar), German (de), Spanish (es), French (fr), Italian (it), Norwegian (no)… See the full description on the dataset page: https://huggingface.co/datasets/lightonai/nanobeir-multilingual.lightonai-embeddings-fine-tuning-reranked-v1
LightOn embeddings-fine-tuning, rescored with mxbai-rerank-large-v2
This dataset is a teacher-rescored version of lightonai/embeddings-fine-tuning. For every (query, candidate-document) pair in the source, we ran mixedbread-ai/mxbai-rerank-large-v2 and stored the resulting score. The point is to make the source data usable as a teacher target for distilling reranker students. It's the upstream artifact behind the rerank-scored configs of cross-encoder/ettin-reranker-v1-data… See the full description on the dataset page: https://huggingface.co/datasets/cross-encoder/lightonai-embeddings-fine-tuning-reranked-v1.fever-decontaminated
fever (Decontaminated)
A decontaminated version of the fever dataset from the BEIR benchmark, with samples found in the mgte-en pre-training dataset removed.
Decontamination methodology
Contamination was detected using a two-pass approach against the full mgte-en dataset (484 GB, 1,235 parquet files):
Pass 1: Exact hash matching
All texts (queries and corpus documents) were normalized (lowercased, unicode NFKD, whitespace collapsed) and hashed with xxHash-64.… See the full description on the dataset page: https://huggingface.co/datasets/lightonai/fever-decontaminated.fc-amf-ocr
Dataset Card for Finance Commons AMF OCR dataset (FC-AMF-OCR)
Dataset Summary
The FC-AMF-OCR dataset is a comprehensive document collection derived from the AMF-PDF dataset, which is part of the Finance Commons collection. This extensive dataset comprises 9.3 million images, each processed through Optical Character Recognition (OCR) using the docTR library. While native text annotations are available in the AMF-Text dataset, these annotations suffer from imperfections and… See the full description on the dataset page: https://huggingface.co/datasets/lightonai/fc-amf-ocr.embeddings_superviseddbpedia-entity-decontaminated
dbpedia-entity (Decontaminated)
A decontaminated version of the dbpedia-entity dataset from the BEIR benchmark, with samples found in the mgte-en pre-training dataset removed.
Decontamination methodology
Contamination was detected using a two-pass approach against the full mgte-en dataset (484 GB, 1,235 parquet files):
Pass 1: Exact hash matching
All texts (queries and corpus documents) were normalized (lowercased, unicode NFKD, whitespace collapsed) and hashed… See the full description on the dataset page: https://huggingface.co/datasets/lightonai/dbpedia-entity-decontaminated.webis-touche2020-decontaminated
webis-touche2020 (Decontaminated)
A decontaminated version of the webis-touche2020 dataset from the BEIR benchmark, with samples found in the mgte-en pre-training dataset removed.
Decontamination methodology
Contamination was detected using a two-pass approach against the full mgte-en dataset (484 GB, 1,235 parquet files):
Pass 1: Exact hash matching
All texts (queries and corpus documents) were normalized (lowercased, unicode NFKD, whitespace collapsed) and… See the full description on the dataset page: https://huggingface.co/datasets/lightonai/webis-touche2020-decontaminated.fiqa-decontaminated
fiqa (Decontaminated)
A decontaminated version of the fiqa dataset from the BEIR benchmark, with samples found in the mgte-en pre-training dataset removed.
Decontamination methodology
Contamination was detected using a two-pass approach against the full mgte-en dataset (484 GB, 1,235 parquet files):
Pass 1: Exact hash matching
All texts (queries and corpus documents) were normalized (lowercased, unicode NFKD, whitespace collapsed) and hashed with xxHash-64. The… See the full description on the dataset page: https://huggingface.co/datasets/lightonai/fiqa-decontaminated.hotpotqa-decontaminated
hotpotqa (Decontaminated)
A decontaminated version of the hotpotqa dataset from the BEIR benchmark, with samples found in the mgte-en pre-training dataset removed.
Decontamination methodology
Contamination was detected using a two-pass approach against the full mgte-en dataset (484 GB, 1,235 parquet files):
Pass 1: Exact hash matching
All texts (queries and corpus documents) were normalized (lowercased, unicode NFKD, whitespace collapsed) and hashed with… See the full description on the dataset page: https://huggingface.co/datasets/lightonai/hotpotqa-decontaminated.nq-decontaminated
nq (Decontaminated)
A decontaminated version of the nq dataset from the BEIR benchmark, with samples found in the mgte-en pre-training dataset removed.
Decontamination methodology
Contamination was detected using a two-pass approach against the full mgte-en dataset (484 GB, 1,235 parquet files):
Pass 1: Exact hash matching
All texts (queries and corpus documents) were normalized (lowercased, unicode NFKD, whitespace collapsed) and hashed with xxHash-64. The… See the full description on the dataset page: https://huggingface.co/datasets/lightonai/nq-decontaminated.trec-covid-decontaminated
trec-covid (Decontaminated)
A decontaminated version of the trec-covid dataset from the BEIR benchmark, with samples found in the mgte-en pre-training dataset removed.
Decontamination methodology
Contamination was detected using a two-pass approach against the full mgte-en dataset (484 GB, 1,235 parquet files):
Pass 1: Exact hash matching
All texts (queries and corpus documents) were normalized (lowercased, unicode NFKD, whitespace collapsed) and hashed with… See the full description on the dataset page: https://huggingface.co/datasets/lightonai/trec-covid-decontaminated.quora-decontaminated
quora (Decontaminated)
A decontaminated version of the quora dataset from the BEIR benchmark, with samples found in the mgte-en pre-training dataset removed.
Decontamination methodology
Contamination was detected using a two-pass approach against the full mgte-en dataset (484 GB, 1,235 parquet files):
Pass 1: Exact hash matching
All texts (queries and corpus documents) were normalized (lowercased, unicode NFKD, whitespace collapsed) and hashed with xxHash-64.… See the full description on the dataset page: https://huggingface.co/datasets/lightonai/quora-decontaminated.msmarco-decontaminated
msmarco (Decontaminated)
A decontaminated version of the msmarco dataset from the BEIR benchmark, with samples found in the mgte-en pre-training dataset removed.
Decontamination methodology
Contamination was detected using a two-pass approach against the full mgte-en dataset (484 GB, 1,235 parquet files):
Pass 1: Exact hash matching
All texts (queries and corpus documents) were normalized (lowercased, unicode NFKD, whitespace collapsed) and hashed with… See the full description on the dataset page: https://huggingface.co/datasets/lightonai/msmarco-decontaminated.scidocs-decontaminated
scidocs (Decontaminated)
A decontaminated version of the scidocs dataset from the BEIR benchmark, with samples found in the mgte-en pre-training dataset removed.
Decontamination methodology
Contamination was detected using a two-pass approach against the full mgte-en dataset (484 GB, 1,235 parquet files):
Pass 1: Exact hash matching
All texts (queries and corpus documents) were normalized (lowercased, unicode NFKD, whitespace collapsed) and hashed with… See the full description on the dataset page: https://huggingface.co/datasets/lightonai/scidocs-decontaminated.arguana-decontaminated
arguana (Decontaminated)
A decontaminated version of the arguana dataset from the BEIR benchmark, with samples found in the mgte-en pre-training dataset removed.
Decontamination methodology
Contamination was detected using a two-pass approach against the full mgte-en dataset (484 GB, 1,235 parquet files):
Pass 1: Exact hash matching
All texts (queries and corpus documents) were normalized (lowercased, unicode NFKD, whitespace collapsed) and hashed with… See the full description on the dataset page: https://huggingface.co/datasets/lightonai/arguana-decontaminated.nfcorpus-decontaminated
nfcorpus (Decontaminated)
A decontaminated version of the nfcorpus dataset from the BEIR benchmark, with samples found in the mgte-en pre-training dataset removed.
Decontamination methodology
Contamination was detected using a two-pass approach against the full mgte-en dataset (484 GB, 1,235 parquet files):
Pass 1: Exact hash matching
All texts (queries and corpus documents) were normalized (lowercased, unicode NFKD, whitespace collapsed) and hashed with… See the full description on the dataset page: https://huggingface.co/datasets/lightonai/nfcorpus-decontaminated.scifact-decontaminated
scifact (Decontaminated)
A decontaminated version of the scifact dataset from the BEIR benchmark, with samples found in the mgte-en pre-training dataset removed.
Decontamination methodology
Contamination was detected using a two-pass approach against the full mgte-en dataset (484 GB, 1,235 parquet files):
Pass 1: Exact hash matching
All texts (queries and corpus documents) were normalized (lowercased, unicode NFKD, whitespace collapsed) and hashed with… See the full description on the dataset page: https://huggingface.co/datasets/lightonai/scifact-decontaminated.ArabicWeb24
📚 ArabicWeb24
More than 39 billion tokens of high quality Arabic web content 🌐.
What is ArabicWeb24 ?
The ArabicWeb24 dataset consists of more than 28 billion tokens of cleaned and deduplicated Arabic web data from a customized crawl.
This was processed using the large scale data processing library datatrove.
What is being released ?
We are releasing two datasets versions:
ArabicWeb24: dataset version 1 (v1) underwent extensive processing through… See the full description on the dataset page: https://huggingface.co/datasets/lightonai/ArabicWeb24.MTP