mteb
Datasets
All datasets matching “mteb”resultssts12-sts
STS12
An MTEB dataset
Massive Text Embedding Benchmark
SemEval-2012 Task 6.
Task category
t2t
Domains
Encyclopaedic, News, Written
Reference
https://www.aclweb.org/anthology/S12-1051.pdf
How to evaluate on this task
You can evaluate an embedding model on this dataset using the following code:
import mteb
task = mteb.get_tasks(["STS12"])
evaluator = mteb.MTEB(task)
model = mteb.get_model(YOUR_MODEL)
evaluator.run(model)
To learn more about how to… See the full description on the dataset page: https://huggingface.co/datasets/mteb/sts12-sts.mteb_leaderboard
[!NOTE]Previously it was possible to submit models results to MTEB by adding the results to the model metadata. This is no longer an option as we want to ensure high quality metadata.
This repository contain the results of the embedding benchmark evaluated using the package mteb.
Reference
🦾 Leaderboard
An up to date leaderboard of embedding models
📚 mteb
Guides and instructions on how to use mteb, including running, submitting scores, etc.
🙋 Questions
Questions about the… See the full description on the dataset page: https://huggingface.co/datasets/morteza20/mteb_leaderboard.nfcorpus
NFCorpus
An MTEB dataset
Massive Text Embedding Benchmark
NFCorpus: A Full-Text Learning to Rank Dataset for Medical Information Retrieval
Task category
t2t
Domains
Medical, Academic, Written
Reference
https://www.cl.uni-heidelberg.de/statnlpgroup/nfcorpus/
How to evaluate on this task
You can evaluate an embedding model on this dataset using the following code:
import mteb
task = mteb.get_tasks(["NFCorpus"])
evaluator = mteb.MTEB(task)
model =… See the full description on the dataset page: https://huggingface.co/datasets/mteb/nfcorpus.amazon_massive_intent
MassiveIntentClassification
An MTEB dataset
Massive Text Embedding Benchmark
MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languages
Task category
t2c
Domains
Spoken
Reference
https://arxiv.org/abs/2204.08582
How to evaluate on this task
You can evaluate an embedding model on this dataset using the following code:
import mteb
task = mteb.get_tasks(["MassiveIntentClassification"])
evaluator =… See the full description on the dataset page: https://huggingface.co/datasets/mteb/amazon_massive_intent.sts22-crosslingual-sts
STS22.v2
An MTEB dataset
Massive Text Embedding Benchmark
SemEval 2022 Task 8: Multilingual News Article Similarity. Version 2 filters updated on STS22 by removing pairs where one of entries contain empty sentences.
Task category
t2t
Domains
News, Written
Reference
https://competitions.codalab.org/competitions/33835
How to evaluate on this task
You can evaluate an embedding model on this dataset using the following code:
import mteb
task =… See the full description on the dataset page: https://huggingface.co/datasets/mteb/sts22-crosslingual-sts.