datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
sts12-sts
STS12
An MTEB dataset
Massive Text Embedding Benchmark
SemEval-2012 Task 6.
Task category
t2t
Domains
Encyclopaedic, News, Written
Reference
https://www.aclweb.org/anthology/S12-1051.pdf
How to evaluate on this task
You can evaluate an embedding model on this dataset using the following code:
import mteb
task = mteb.get_tasks(["STS12"])
evaluator = mteb.MTEB(task)
model = mteb.get_model(YOUR_MODEL)
evaluator.run(model)
To learn more about how to… See the full description on the dataset page: https://huggingface.co/datasets/mteb/sts12-sts.nfcorpus
NFCorpus
An MTEB dataset
Massive Text Embedding Benchmark
NFCorpus: A Full-Text Learning to Rank Dataset for Medical Information Retrieval
Task category
t2t
Domains
Medical, Academic, Written
Reference
https://www.cl.uni-heidelberg.de/statnlpgroup/nfcorpus/
How to evaluate on this task
You can evaluate an embedding model on this dataset using the following code:
import mteb
task = mteb.get_tasks(["NFCorpus"])
evaluator = mteb.MTEB(task)
model =… See the full description on the dataset page: https://huggingface.co/datasets/mteb/nfcorpus.sts22-crosslingual-sts
STS22.v2
An MTEB dataset
Massive Text Embedding Benchmark
SemEval 2022 Task 8: Multilingual News Article Similarity. Version 2 filters updated on STS22 by removing pairs where one of entries contain empty sentences.
Task category
t2t
Domains
News, Written
Reference
https://competitions.codalab.org/competitions/33835
How to evaluate on this task
You can evaluate an embedding model on this dataset using the following code:
import mteb
task =… See the full description on the dataset page: https://huggingface.co/datasets/mteb/sts22-crosslingual-sts.arguana
ArguAna
An MTEB dataset
Massive Text Embedding Benchmark
ArguAna: Retrieval of the Best Counterargument without Prior Topic Knowledge
Task category
Retrieval (text-to-text)
Domains
Social, Web, Written
Reference
ACL
Source datasets:
mteb/arguana
How to evaluate on this task
You can evaluate an embedding model on this dataset using the following code:
import mteb
task = mteb.get_task("ArguAna")
evaluator = mteb.MTEB([task])
model =… See the full description on the dataset page: https://huggingface.co/datasets/mteb/arguana.sickr-sts
SICK-R
An MTEB dataset
Massive Text Embedding Benchmark
Semantic Textual Similarity SICK-R dataset
Task category
t2t
Domains
Web, Written
Reference
https://aclanthology.org/L14-1314/
How to evaluate on this task
You can evaluate an embedding model on this dataset using the following code:
import mteb
task = mteb.get_tasks(["SICK-R"])
evaluator = mteb.MTEB(task)
model = mteb.get_model(YOUR_MODEL)
evaluator.run(model)
To learn more about how to run models… See the full description on the dataset page: https://huggingface.co/datasets/mteb/sickr-sts.sts13-sts
STS13
An MTEB dataset
Massive Text Embedding Benchmark
SemEval STS 2013 dataset.
Task category
t2t
Domains
Web, News, Non-fiction, Written
Reference
https://www.aclweb.org/anthology/S13-1004/
How to evaluate on this task
You can evaluate an embedding model on this dataset using the following code:
import mteb
task = mteb.get_tasks(["STS13"])
evaluator = mteb.MTEB(task)
model = mteb.get_model(YOUR_MODEL)
evaluator.run(model)
To learn more about how… See the full description on the dataset page: https://huggingface.co/datasets/mteb/sts13-sts.sts14-sts
STS14
An MTEB dataset
Massive Text Embedding Benchmark
SemEval STS 2014 dataset. Currently only the English dataset
Task category
t2t
Domains
Blog, Web, Spoken
Reference
https://www.aclweb.org/anthology/S14-1002
How to evaluate on this task
You can evaluate an embedding model on this dataset using the following code:
import mteb
task = mteb.get_tasks(["STS14"])
evaluator = mteb.MTEB(task)
model = mteb.get_model(YOUR_MODEL)
evaluator.run(model)
To… See the full description on the dataset page: https://huggingface.co/datasets/mteb/sts14-sts.scifact
SciFact
An MTEB dataset
Massive Text Embedding Benchmark
SciFact verifies scientific claims using evidence from the research literature containing scientific paper abstracts.
Task category
t2t
Domains
Academic, Medical, Written
Reference
https://github.com/allenai/scifact
How to evaluate on this task
You can evaluate an embedding model on this dataset using the following code:
import mteb
task = mteb.get_tasks(["SciFact"])
evaluator = mteb.MTEB(task)… See the full description on the dataset page: https://huggingface.co/datasets/mteb/scifact.scidocs
SCIDOCS
An MTEB dataset
Massive Text Embedding Benchmark
SciDocs, a new evaluation benchmark consisting of seven document-level tasks ranging from citation prediction, to document classification and recommendation.
Task category
t2t
Domains
Academic, Written, Non-fiction
Reference
https://allenai.org/data/scidocs
How to evaluate on this task
You can evaluate an embedding model on this dataset using the following code:
import mteb
task =… See the full description on the dataset page: https://huggingface.co/datasets/mteb/scidocs.biosses-sts
BIOSSES
An MTEB dataset
Massive Text Embedding Benchmark
Biomedical Semantic Similarity Estimation.
Task category
t2t
Domains
Medical
Reference
https://tabilab.cmpe.boun.edu.tr/BIOSSES/DataSet.html
How to evaluate on this task
You can evaluate an embedding model on this dataset using the following code:
import mteb
task = mteb.get_tasks(["BIOSSES"])
evaluator = mteb.MTEB(task)
model = mteb.get_model(YOUR_MODEL)
evaluator.run(model)
To learn more… See the full description on the dataset page: https://huggingface.co/datasets/mteb/biosses-sts.stsbenchmark-sts
STSBenchmark
An MTEB dataset
Massive Text Embedding Benchmark
Semantic Textual Similarity Benchmark (STSbenchmark) dataset.
Task category
t2t
Domains
Blog, News, Written
Reference
https://github.com/PhilipMay/stsb-multi-mt/
How to evaluate on this task
You can evaluate an embedding model on this dataset using the following code:
import mteb
task = mteb.get_tasks(["STSBenchmark"])
evaluator = mteb.MTEB(task)
model = mteb.get_model(YOUR_MODEL)… See the full description on the dataset page: https://huggingface.co/datasets/mteb/stsbenchmark-sts.twentynewsgroups-clustering
TwentyNewsgroupsClustering.v2
An MTEB dataset
Massive Text Embedding Benchmark
Clustering of the 20 Newsgroups dataset (subject only).
Task category
t2c
Domains
News, Written
Reference
https://scikit-learn.org/0.19/datasets/twenty_newsgroups.html
How to evaluate on this task
You can evaluate an embedding model on this dataset using the following code:
import mteb
task = mteb.get_tasks(["TwentyNewsgroupsClustering.v2"])
evaluator = mteb.MTEB(task)… See the full description on the dataset page: https://huggingface.co/datasets/mteb/twentynewsgroups-clustering.fiqa
FiQA2018
An MTEB dataset
Massive Text Embedding Benchmark
Financial Opinion Mining and Question Answering
Task category
t2t
Domains
Written, Financial
Reference
https://sites.google.com/view/fiqa/
How to evaluate on this task
You can evaluate an embedding model on this dataset using the following code:
import mteb
task = mteb.get_tasks(["FiQA2018"])
evaluator = mteb.MTEB(task)
model = mteb.get_model(YOUR_MODEL)
evaluator.run(model)
To learn more… See the full description on the dataset page: https://huggingface.co/datasets/mteb/fiqa.arena-resultsThis dataset contains the saved results from MTEB-Arena
sts15-sts
STS15
An MTEB dataset
Massive Text Embedding Benchmark
SemEval STS 2015 dataset
Task category
t2t
Domains
Blog, News, Web, Written, Spoken
Reference
https://www.aclweb.org/anthology/S15-2010
How to evaluate on this task
You can evaluate an embedding model on this dataset using the following code:
import mteb
task = mteb.get_tasks(["STS15"])
evaluator = mteb.MTEB(task)
model = mteb.get_model(YOUR_MODEL)
evaluator.run(model)
To learn more about how… See the full description on the dataset page: https://huggingface.co/datasets/mteb/sts15-sts.trec-covid
TRECCOVID
An MTEB dataset
Massive Text Embedding Benchmark
TRECCOVID is an ad-hoc search challenge based on the COVID-19 dataset containing scientific articles related to the COVID-19 pandemic.
Task category
t2t
Domains
Medical, Academic, Written
Reference
https://ir.nist.gov/covidSubmit/index.html
How to evaluate on this task
You can evaluate an embedding model on this dataset using the following code:
import mteb
task = mteb.get_tasks(["TRECCOVID"])… See the full description on the dataset page: https://huggingface.co/datasets/mteb/trec-covid.amazon_massive_scenario
MassiveScenarioClassification
An MTEB dataset
Massive Text Embedding Benchmark
MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languages
Task category
t2c
Domains
Spoken
Reference
https://arxiv.org/abs/2204.08582
How to evaluate on this task
You can evaluate an embedding model on this dataset using the following code:
import mteb
task = mteb.get_tasks(["MassiveScenarioClassification"])
evaluator =… See the full description on the dataset page: https://huggingface.co/datasets/mteb/amazon_massive_scenario.cqadupstack-gaming
CQADupstackGamingRetrieval
An MTEB dataset
Massive Text Embedding Benchmark
CQADupStack: A Benchmark Data Set for Community Question-Answering Research
Task category
t2t
Domains
Web, Written
Reference
http://nlp.cis.unimelb.edu.au/resources/cqadupstack/
How to evaluate on this task
You can evaluate an embedding model on this dataset using the following code:
import mteb
task = mteb.get_tasks(["CQADupstackGamingRetrieval"])
evaluator = mteb.MTEB(task)… See the full description on the dataset page: https://huggingface.co/datasets/mteb/cqadupstack-gaming.cqadupstack-unix
CQADupstackUnixRetrieval
An MTEB dataset
Massive Text Embedding Benchmark
CQADupStack: A Benchmark Data Set for Community Question-Answering Research
Task category
t2t
Domains
Written, Web, Programming
Reference
http://nlp.cis.unimelb.edu.au/resources/cqadupstack/
How to evaluate on this task
You can evaluate an embedding model on this dataset using the following code:
import mteb
task = mteb.get_tasks(["CQADupstackUnixRetrieval"])
evaluator =… See the full description on the dataset page: https://huggingface.co/datasets/mteb/cqadupstack-unix.stackexchange-clustering
StackExchangeClustering.v2
An MTEB dataset
Massive Text Embedding Benchmark
Clustering of titles from 121 stackexchanges. Clustering of 25 sets, each with 10-50 classes, and each class with 100 - 1000 sentences.
Task category
t2c
Domains
Web, Written
Reference
https://arxiv.org/abs/2104.07081
How to evaluate on this task
You can evaluate an embedding model on this dataset using the following code:
import mteb
task =… See the full description on the dataset page: https://huggingface.co/datasets/mteb/stackexchange-clustering.sts17-crosslingual-sts
STS17
An MTEB dataset
Massive Text Embedding Benchmark
Semeval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation
Task category
t2t
Domains
News, Web, Written
Reference
https://alt.qcri.org/semeval2017/task1/
How to evaluate on this task
You can evaluate an embedding model on this dataset using the following code:
import mteb
task = mteb.get_tasks(["STS17"])
evaluator = mteb.MTEB(task)
model =… See the full description on the dataset page: https://huggingface.co/datasets/mteb/sts17-crosslingual-sts.sts16-sts
STS16
An MTEB dataset
Massive Text Embedding Benchmark
SemEval-2016 Task 4
Task category
t2t
Domains
Blog, Web, Spoken
Reference
https://www.aclweb.org/anthology/S16-1001
How to evaluate on this task
You can evaluate an embedding model on this dataset using the following code:
import mteb
task = mteb.get_tasks(["STS16"])
evaluator = mteb.MTEB(task)
model = mteb.get_model(YOUR_MODEL)
evaluator.run(model)
To learn more about how to run models on mteb task… See the full description on the dataset page: https://huggingface.co/datasets/mteb/sts16-sts.arxiv-clustering-s2s
ArXivHierarchicalClusteringS2S
An MTEB dataset
Massive Text Embedding Benchmark
Clustering of titles from arxiv. Clustering of 30 sets, either on the main or secondary category
Task category
t2c
Domains
Academic, Written
Reference
https://www.kaggle.com/Cornell-University/arxiv
How to evaluate on this task
You can evaluate an embedding model on this dataset using the following code:
import mteb
task = mteb.get_tasks(["ArXivHierarchicalClusteringS2S"])… See the full description on the dataset page: https://huggingface.co/datasets/mteb/arxiv-clustering-s2s.biorxiv-clustering-p2p
BiorxivClusteringP2P.v2
An MTEB dataset
Massive Text Embedding Benchmark
Clustering of titles+abstract from biorxiv across 26 categories.
Task category
t2c
Domains
Academic, Written
Reference
https://api.biorxiv.org/
How to evaluate on this task
You can evaluate an embedding model on this dataset using the following code:
import mteb
task = mteb.get_tasks(["BiorxivClusteringP2P.v2"])
evaluator = mteb.MTEB(task)
model = mteb.get_model(YOUR_MODEL)… See the full description on the dataset page: https://huggingface.co/datasets/mteb/biorxiv-clustering-p2p.medrxiv-clustering-s2s
MedrxivClusteringS2S.v2
An MTEB dataset
Massive Text Embedding Benchmark
Clustering of titles from medrxiv across 51 categories.
Task category
t2c
Domains
Academic, Medical, Written
Reference
https://api.medrxiv.org/
How to evaluate on this task
You can evaluate an embedding model on this dataset using the following code:
import mteb
task = mteb.get_tasks(["MedrxivClusteringS2S.v2"])
evaluator = mteb.MTEB(task)
model = mteb.get_model(YOUR_MODEL)… See the full description on the dataset page: https://huggingface.co/datasets/mteb/medrxiv-clustering-s2s.arxiv-clustering-p2p
ArXivHierarchicalClusteringP2P
An MTEB dataset
Massive Text Embedding Benchmark
Clustering of titles+abstract from arxiv. Clustering of 30 sets, either on the main or secondary category
Task category
t2c
Domains
Academic, Written
Reference
https://www.kaggle.com/Cornell-University/arxiv
How to evaluate on this task
You can evaluate an embedding model on this dataset using the following code:
import mteb
task =… See the full description on the dataset page: https://huggingface.co/datasets/mteb/arxiv-clustering-p2p.quora
QuoraRetrieval
An MTEB dataset
Massive Text Embedding Benchmark
QuoraRetrieval is based on questions that are marked as duplicates on the Quora platform. Given a question, find other (duplicate) questions.
Task category
t2t
Domains
Written, Web, Blog
Reference
https://quoradata.quora.com/First-Quora-Dataset-Release-Question-Pairs
How to evaluate on this task
You can evaluate an embedding model on this dataset using the following code:
import mteb
task =… See the full description on the dataset page: https://huggingface.co/datasets/mteb/quora.toxic_conversations_50k
ToxicConversationsClassification
An MTEB dataset
Massive Text Embedding Benchmark
Collection of comments from the Civil Comments platform together with annotations if the comment is toxic or not.
Task category
t2c
Domains
Social, Written
Reference
https://www.kaggle.com/competitions/jigsaw-unintended-bias-in-toxicity-classification/overview
How to evaluate on this task
You can evaluate an embedding model on this dataset using the following code:
import… See the full description on the dataset page: https://huggingface.co/datasets/mteb/toxic_conversations_50k.medrxiv-clustering-p2p
MedrxivClusteringP2P.v2
An MTEB dataset
Massive Text Embedding Benchmark
Clustering of titles+abstract from medrxiv across 51 categories.
Task category
t2c
Domains
Academic, Medical, Written
Reference
https://api.medrxiv.org/
How to evaluate on this task
You can evaluate an embedding model on this dataset using the following code:
import mteb
task = mteb.get_tasks(["MedrxivClusteringP2P.v2"])
evaluator = mteb.MTEB(task)
model =… See the full description on the dataset page: https://huggingface.co/datasets/mteb/medrxiv-clustering-p2p.stackexchange-clustering-p2p
StackExchangeClusteringP2P.v2
An MTEB dataset
Massive Text Embedding Benchmark
Clustering of title+body from stackexchange. Clustering of 5 sets of 10k paragraphs and 5 sets of 5k paragraphs.
Task category
t2c
Domains
Web, Written
Reference
https://arxiv.org/abs/2104.07081
How to evaluate on this task
You can evaluate an embedding model on this dataset using the following code:
import mteb
task = mteb.get_tasks(["StackExchangeClusteringP2P.v2"])… See the full description on the dataset page: https://huggingface.co/datasets/mteb/stackexchange-clustering-p2p.
