datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
trec-covid
TRECCOVID
An MTEB dataset
Massive Text Embedding Benchmark
TRECCOVID is an ad-hoc search challenge based on the COVID-19 dataset containing scientific articles related to the COVID-19 pandemic.
Task category
t2t
Domains
Medical, Academic, Written
Reference
https://ir.nist.gov/covidSubmit/index.html
How to evaluate on this task
You can evaluate an embedding model on this dataset using the following code:
import mteb
task = mteb.get_tasks(["TRECCOVID"])… See the full description on the dataset page: https://huggingface.co/datasets/mteb/trec-covid.trec-covid
Dataset Card for BEIR Benchmark
trec-covid is one of the datasets from the Bio-Medical Retrieval task within BEIR, measuring scientific article retrieval for a given query on COVID-19.
Dataset Summary
BEIR is a heterogeneous benchmark built from 18 diverse datasets representing 9 information retrieval tasks.
Fact-checking: FEVER, Climate-FEVER, SciFact
Question-Answering: NQ, HotpotQA, FiQA-2018
Bio-Medical IR: TREC-COVID, BioASQ, NFCorpus
News Retrieval: TREC-NEWS… See the full description on the dataset page: https://huggingface.co/datasets/BeIR/trec-covid.trec-covid-qrels
Dataset Card for BEIR Benchmark
Dataset Summary
BEIR is a heterogeneous benchmark that has been built from 18 diverse datasets representing 9 information retrieval tasks:
Fact-checking: FEVER, Climate-FEVER, SciFact
Question-Answering: NQ, HotpotQA, FiQA-2018
Bio-Medical IR: TREC-COVID, BioASQ, NFCorpus
News Retrieval: TREC-NEWS, Robust04
Argument Retrieval: Touche-2020, ArguAna
Duplicate Question Retrieval: Quora, CqaDupstack
Citation-Prediction: SCIDOCS
Tweet… See the full description on the dataset page: https://huggingface.co/datasets/BeIR/trec-covid-qrels.trec-covid-decontaminated
trec-covid (Decontaminated)
A decontaminated version of the trec-covid dataset from the BEIR benchmark, with samples found in the mgte-en pre-training dataset removed.
Decontamination methodology
Contamination was detected using a two-pass approach against the full mgte-en dataset (484 GB, 1,235 parquet files):
Pass 1: Exact hash matching
All texts (queries and corpus documents) were normalized (lowercased, unicode NFKD, whitespace collapsed) and hashed with… See the full description on the dataset page: https://huggingface.co/datasets/lightonai/trec-covid-decontaminated.TRECCOVID-PL
TRECCOVID-PL
An MTEB dataset
Massive Text Embedding Benchmark
TRECCOVID is an ad-hoc search challenge based on the COVID-19 dataset containing scientific articles related to the COVID-19 pandemic.
Task category
t2t
Domains
Academic, Medical, Non-fiction, Written
Reference
https://ir.nist.gov/covidSubmit/index.html
How to evaluate on this task
You can evaluate an embedding model on this dataset using the following code:
import mteb
task =… See the full description on the dataset page: https://huggingface.co/datasets/mteb/TRECCOVID-PL.benchmark-trec-covidtrec-covid-vn
TRECCOVID-VN
An MTEB dataset
Massive Text Embedding Benchmark
A translated dataset from TRECCOVID is an ad-hoc search challenge based on the COVID-19 dataset containing scientific articles related to the COVID-19 pandemic. The process of creating the VN-MTEB (Vietnamese Massive Text Embedding Benchmark) from English samples involves a new automated system: - The system uses large language models (LLMs), specifically Coherence's Aya model, for translation. - Applies advanced embedding… See the full description on the dataset page: https://huggingface.co/datasets/GreenNode/trec-covid-vn.beir-trec-covid
TRECCOVID — BEIR, unified schema
A normalised copy of the dataset behind the mteb task TRECCOVID, one of the tasks of the BEIR benchmark as mteb defines it. Same queries, documents
and relevance judgements as the benchmark evaluates — reshaped into one strict schema shared by every dataset
in this collection.
Source
mteb/trec-covid @ bb9466bac815 (the revision pinned in mteb)
Domain · languages
biomedical · eng
Queries / documents / qrels
50 / 171,332 / 66,336… See the full description on the dataset page: https://huggingface.co/datasets/Hyukkyu/beir-trec-covid.trec-covid_CS-MTEB
TREC-COVID CS-MTEB
An MTEB dataset
Massive Text Embedding Benchmark
Code-switching version of mteb/trec-covid, with queries rewritten in Chinese-English, Japanese-English, German-English, Spanish-English, Korean-English, French-English, Italian-English, Portuguese-English, Dutch-English code-switching styles.
Dataset Structure
The dataset contains the following configurations:
From original dataset (unchanged):
corpus: Original corpus documents
default: Original relevance… See the full description on the dataset page: https://huggingface.co/datasets/UTokyo-Yokoya-Lab/trec-covid_CS-MTEB.trec_covid_toyset_pairtrec-covid-generated-queries
Dataset Card for BEIR Benchmark
Dataset Summary
BEIR is a heterogeneous benchmark that has been built from 18 diverse datasets representing 9 information retrieval tasks:
Fact-checking: FEVER, Climate-FEVER, SciFact
Question-Answering: NQ, HotpotQA, FiQA-2018
Bio-Medical IR: TREC-COVID, BioASQ, NFCorpus
News Retrieval: TREC-NEWS, Robust04
Argument Retrieval: Touche-2020, ArguAna
Duplicate Question Retrieval: Quora, CqaDupstack
Citation-Prediction: SCIDOCS
Tweet… See the full description on the dataset page: https://huggingface.co/datasets/BeIR/trec-covid-generated-queries.trec-covid-fa-v2ColBERT-TREC-COVIDThis dataset consists ColBERTv2.0 document vectors for the entire TREC-COVID dataset from BeIR. That 128 dimension per token, with 180 tokens for each of 171332 documents.
The dataset was created using A100-40GB sponsored by Qdrant. The code to create these vectors is here: https://colab.research.google.com/drive/1hEhyleSrBz_mPyQJnRc0MwBenDuX1ahY?usp=sharing
This dataset was created for indexing experiments by Qdrant.
trec-covid-decontaminated
trec-covid-decontaminated (MTEB layout)
Repackaging of lightonai/trec-covid-decontaminated
into the layout expected by MTEB:
a default config holding the qrels (one split per evaluation split), alongside
corpus and queries configs.
Rows and columns are copied verbatim from the source dataset; only the
config/split packaging differs. All credit for the decontaminated data belongs to
LightOn AI, and to the original BEIR
authors for the underlying benchmark.
Source:… See the full description on the dataset page: https://huggingface.co/datasets/iamfortytwo/trec-covid-decontaminated.trec-covid-CSR-L
TRECCOVID-CodeSwitching
An MTEB dataset
Massive Text Embedding Benchmark
Code-switching version of mteb/trec-covid, with queries rewritten in Chinese-English and Japanese-English code-switching styles.
Dataset Structure
The dataset contains the following configurations:
From original dataset (unchanged):
corpus: Original corpus documents
default: Original relevance judgments (qrels)
Code-switching additions:
queries_zh_en: Chinese-English code-switching queries… See the full description on the dataset page: https://huggingface.co/datasets/UTokyo-Yokoya-Lab/trec-covid-CSR-L.trec-covid-generated-queriesHindi_Trec_CovidYour dataset description goes here.trec-covid__openai_ada2rus-trec-covidTRECCOVID-Fa
TRECCOVID-Fa
An MTEB dataset
Massive Text Embedding Benchmark
TRECCOVID-Fa
Task category
t2t
Domains
Medical
Reference
https://huggingface.co/datasets/MCINext/trec-covid-fa
How to evaluate on this task
You can evaluate an embedding model on this dataset using the following code:
import mteb
task = mteb.get_task("TRECCOVID-Fa")
evaluator = mteb.MTEB([task])
model = mteb.get_model(YOUR_MODEL)
evaluator.run(model)
To learn more about how to run models on… See the full description on the dataset page: https://huggingface.co/datasets/mteb/TRECCOVID-Fa.trec-covidThis is the corpus file from the BEIR benchmark for the TREC-COVID 19 dataset.
trec-covid.pisa
trec-covid.pisa
Description
A PISA Index for CORD19 (the corpus for the TREC-COVID query set)
Usage
# Load the artifact
import pyterrier as pt
index = pt.Artifact.from_hf('pyterrier/trec-covid.pisa')
index.bm25() # returns a BM25 retriever
Benchmarks
name
nDCG@10
R@1000
bm25
0.6254
0.4462
dph
0.6633
0.4136
Reproduction
import pyterrier as pt
from tqdm import tqdm
import ir_datasets
from pyterrier_pisa import PisaIndex… See the full description on the dataset page: https://huggingface.co/datasets/pyterrier/trec-covid.pisa.trec-covid.terrier
trec-covid.terrier
Description
Terrier index for TREC-COVID
Usage
# Load the artifact
import pyterrier as pt
index = pt.Artifact.from_hf('pyterrier/trec-covid.terrier')
index.bm25()
Benchmarks
name
nDCG@10
R@1000
bm25
0.6295
0.4593
dph
0.631
0.4102
Reproduction
import pyterrier as pt
from tqdm import tqdm
import ir_datasets
dataset = ir_datasets.load('beir/trec-covid')
meta_docno_len =… See the full description on the dataset page: https://huggingface.co/datasets/pyterrier/trec-covid.terrier.trec-covid.retromae.flex
trec-covid.retromae.flex
Description
RetroMAE index for TREC-COVID
Usage
# Load the artifact
import pyterrier_alpha as pta
artifact = pta.Artifact.from_hf('pyterrier/trec-covid.retromae.flex')
artifact.np_retriever()
Benchmarks
trec-covid
name
nDCG@10
R@1000
np (flat)
0.6486
0.3713
Reproduction
import pyterrier as pt
from tqdm import tqdm
import ir_datasets
from pyterrier_dr import FlexIndex, RetroMAE
pipeline =… See the full description on the dataset page: https://huggingface.co/datasets/pyterrier/trec-covid.retromae.flex.gpl-trec-covidcord19.trec-covid.monoelectra.cache
cord19.trec-covid.monoelectra.cache
Description
TODO: What is the artifact?
Usage
# Load the artifact
import pyterrier_alpha as pta
artifact = pta.Artifact.from_hf('macavaney/cord19.trec-covid.monoelectra.cache')
# TODO: Show how you use the artifact
Benchmarks
TODO: Provide benchmarks for the artifact.
Reproduction
# TODO: Show how you constructed the artifact.
Metadata
{
"type": "scorer_cache",
"format": "hdf5"… See the full description on the dataset page: https://huggingface.co/datasets/macavaney/cord19.trec-covid.monoelectra.cache.beir-nl-trec-covid
Dataset Card for BEIR-NL Benchmark
Dataset Summary
BEIR-NL is a Dutch-translated version of the BEIR benchmark, a diverse and heterogeneous collection of datasets covering various domains from biomedical and financial texts to general web content. Our benchmark is integrated into the Massive Multilingual Text Embedding Benchmark (MMTEB).
BEIR-NL contains the following tasks:
Fact-checking: FEVER, Climate-FEVER, SciFact
Question-Answering: NQ, HotpotQA, FiQA-2018… See the full description on the dataset page: https://huggingface.co/datasets/clips/beir-nl-trec-covid.trec-covid-pl-qrelsPart of BEIR-PL: Zero Shot Information Retrieval Benchmark for the Polish Language.
Link to arxiv: https://arxiv.org/pdf/2305.19840.pdf
Contact: konrad.wojtasik@pwr.edu.pl
benchmark-trec-covidtrec-covid
Dataset Card for BEIR Benchmark
trec-covid is one of the datasets from the Bio-Medical Retrieval task within BEIR, measuring scientific article retrieval for a given query on COVID-19.
Dataset Summary
BEIR is a heterogeneous benchmark built from 18 diverse datasets representing 9 information retrieval tasks.
Fact-checking: FEVER, Climate-FEVER, SciFact
Question-Answering: NQ, HotpotQA, FiQA-2018
Bio-Medical IR: TREC-COVID, BioASQ, NFCorpus
News Retrieval:… See the full description on the dataset page: https://huggingface.co/datasets/chenxiaobin/trec-covid.