Datasetpublic

hakari-bench/NanoR2MED

NanoR2MED This dataset is a Nano-style retrieval dataset for HAKARI-bench. NanoR2MED contains 8 Nano retrieval splits derived from R2MED. Each split keeps up to 200 eligible queries and up to 10000 corpus documents, with exact duplicate query and document text removed where the generator records that policy. Usage from datasets import load_dataset dataset_id = "hakari-bench/NanoR2MED" split = "NanoR2MEDBioinformatics" queries = load_dataset(dataset_id, "queries"… See the full description on the dataset page: https://huggingface.co/datasets/hakari-bench/NanoR2MED.

sourceHugging Faceupdated 4mo agoView on Hugging Face

0likes318downloads

Dataset Card

NanoR2MED

This dataset is a Nano-style retrieval dataset for HAKARI-bench.

NanoR2MED contains 8 Nano retrieval splits derived from R2MED. Each split keeps up to 200 eligible queries and up to 10000 corpus documents, with exact duplicate query and document text removed where the generator records that policy.

Usage

python

from datasets import load_dataset

dataset_id = "hakari-bench/NanoR2MED"
split = "NanoR2MEDBioinformatics"

queries = load_dataset(dataset_id, "queries", split=split)
corpus = load_dataset(dataset_id, "corpus", split=split)
qrels = load_dataset(dataset_id, "qrels", split=split)
reranking_candidates = load_dataset(dataset_id, "reranking_hybrid", split=split)

Data Layout

This dataset uses six Hugging Face Datasets configs:

—corpus: documents with _id and text
—queries: queries with _id and text
—qrels: positive relevance labels with query-id and corpus-id
—bm25: BM25 candidate lists with query-id and corpus-ids
—harrier_oss_v1_270m: dense candidate lists from microsoft/harrier-oss-v1-270m
—reranking_hybrid: RRF candidate lists built from bm25 and harrier_oss_v1_270m

Each config has the same Nano split names.

Candidate Construction

—bm25: local BM25 top-500 with automatic language-aware tokenization. The resolved tokenizer is shown in the Candidate Quality table, for example wordseg@ja.
—harrier_oss_v1_270m: dense top-500 from microsoft/harrier-oss-v1-270m. In tables this is shown as Dense; Dense means microsoft/harrier-oss-v1-270m with the web_search_query prompt for queries and cosine similarity over normalized embeddings.
—reranking_hybrid: RRF over bm25 and harrier_oss_v1_270m using rrf_k=100, keeping the RRF top-100.

Safeguard means rank 101 is appended only when RRF top-100 contains no qrels-positive document.

Split Statistics

Length statistics are character counts computed with len(str(text)).

Nano split	Queries	Corpus	Qrels	Query chars avg	Query chars p50	Query chars p75	Doc chars avg	Doc chars p50	Doc chars p75
NanoR2MEDBioinformatics	77	10000	226	890.3	727.0	1016.0	666.8	679.0	798.0
NanoR2MEDBiology	103	10000	374	523.0	440.0	627.5	474.1	397.0	556.0
NanoR2MEDIIYiClinical	129	10000	457	2584.1	2523.0	3306.0	5042.3	4280.5	6809.5
NanoR2MEDMedQADiag	118	10000	522	706.7	630.0	884.0	791.4	882.0	989.0
NanoR2MEDMedXpertQAExam	97	10000	292	928.4	879.0	1086.0	723.9	767.0	922.0
NanoR2MEDMedicalSciences	88	10000	244	477.6	378.0	596.8	678.6	679.0	801.0
NanoR2MEDPMCClinical	114	10000	248	827.7	832.0	956.8	2103.5	2131.0	2724.0
NanoR2MEDPMCTreatment	150	10000	315	1755.8	1750.5	1980.8	726.6	608.0	928.0

Candidate Quality

nDCG@10 and Recall@100 are computed from the included candidate rankings against the included qrels, then reported as 0-100 scores such as 52.45. Recall@100 uses only the top 100 candidates; an optional rank-101 safeguard positive is not counted in Recall@100.

Dense means microsoft/harrier-oss-v1-270m with the web_search_query prompt and cosine similarity.

Nano split	BM25 tokenizer	BM25 nDCG@10	Dense nDCG@10	Hybrid nDCG@10	BM25 Recall@100	Dense Recall@100	Hybrid Recall@100	Hybrid candidates	Safeguard positives
Mean	-	20.94	30.07	28.82	55.11	68.46	70.40	-	122
NanoR2MEDBioinformatics	englishporterstop	21.89	34.25	26.23	67.34	75.92	80.48	100-101	6
NanoR2MEDBiology	englishporterstop	34.55	49.53	47.22	70.56	82.57	85.61	100-101	3
NanoR2MEDIIYiClinical	englishporterstop	14.82	18.70	19.75	47.30	66.15	67.21	100-101	14
NanoR2MEDMedQADiag	englishporterstop	7.00	12.54	14.06	25.10	46.74	42.14	100-101	34
NanoR2MEDMedXpertQAExam	englishporterstop	2.77	15.99	9.79	19.94	47.09	43.55	100-101	33
NanoR2MEDMedicalSciences	englishporterstop	21.40	35.67	33.20	74.86	85.61	85.66	100-101	3
NanoR2MEDPMCClinical	englishporterstop	39.33	35.84	44.77	81.58	76.61	86.48	100-101	6
NanoR2MEDPMCTreatment	englishporterstop	25.80	38.01	35.55	54.18	66.97	72.04	100-101	23

Hybrid Safeguard Summary

—Safeguard positives: 122
—Rows limited by corpus size: 0
—Metadata file: reranking_hybrid_metadata.json

Source Links

—Source benchmark: R2MED
—R2MED/Bioinformatics: https://huggingface.co/datasets/R2MED/Bioinformatics
—R2MED/Biology: https://huggingface.co/datasets/R2MED/Biology
—R2MED/IIYi-Clinical: https://huggingface.co/datasets/R2MED/IIYi-Clinical
—R2MED/MedQA-Diag: https://huggingface.co/datasets/R2MED/MedQA-Diag
—R2MED/MedXpertQA-Exam: https://huggingface.co/datasets/R2MED/MedXpertQA-Exam
—R2MED/Medical-Sciences: https://huggingface.co/datasets/R2MED/Medical-Sciences
—R2MED/PMC-Clinical: https://huggingface.co/datasets/R2MED/PMC-Clinical
—R2MED/PMC-Treatment: https://huggingface.co/datasets/R2MED/PMC-Treatment

License

NanoR2MED is a derived dataset. Users must comply with the licenses, terms, and attribution requirements of the upstream datasets and benchmarks.