Datasetpublic

hakari-bench/NanoMedical

NanoMedical This dataset is a Nano-style retrieval dataset for HAKARI-bench. NanoMedical contains 10 Nano retrieval splits derived from MTEB(Medical, v1). Each split keeps up to 200 eligible queries and up to 10000 corpus documents, with exact duplicate query and document text removed where the generator records that policy. Usage from datasets import load_dataset dataset_id = "hakari-bench/NanoMedical" split = "NanoCMedQAv2reranking" queries =… See the full description on the dataset page: https://huggingface.co/datasets/hakari-bench/NanoMedical.

sourceHugging Faceupdated 4mo agoView on Hugging Face

0likes340downloads

Dataset Card

NanoMedical

This dataset is a Nano-style retrieval dataset for HAKARI-bench.

NanoMedical contains 10 Nano retrieval splits derived from MTEB(Medical, v1). Each split keeps up to 200 eligible queries and up to 10000 corpus documents, with exact duplicate query and document text removed where the generator records that policy.

Usage

python

from datasets import load_dataset

dataset_id = "hakari-bench/NanoMedical"
split = "NanoCMedQAv2reranking"

queries = load_dataset(dataset_id, "queries", split=split)
corpus = load_dataset(dataset_id, "corpus", split=split)
qrels = load_dataset(dataset_id, "qrels", split=split)
reranking_candidates = load_dataset(dataset_id, "reranking_hybrid", split=split)

Data Layout

This dataset uses six Hugging Face Datasets configs:

—corpus: documents with _id and text
—queries: queries with _id and text
—qrels: positive relevance labels with query-id and corpus-id
—bm25: BM25 candidate lists with query-id and corpus-ids
—harrier_oss_v1_270m: dense candidate lists from microsoft/harrier-oss-v1-270m
—reranking_hybrid: RRF candidate lists built from bm25 and harrier_oss_v1_270m

Each config has the same Nano split names.

Candidate Construction

—bm25: local BM25 top-500 with automatic language-aware tokenization. The resolved tokenizer is shown in the Candidate Quality table, for example wordseg@ja.
—harrier_oss_v1_270m: dense top-500 from microsoft/harrier-oss-v1-270m. In tables this is shown as Dense; Dense means microsoft/harrier-oss-v1-270m with the web_search_query prompt for queries and cosine similarity over normalized embeddings.
—reranking_hybrid: RRF over bm25 and harrier_oss_v1_270m using rrf_k=100, keeping the RRF top-100.

Safeguard means rank 101 is appended only when RRF top-100 contains no qrels-positive document.

Split Statistics

Length statistics are character counts computed with len(str(text)).

Nano split	Queries	Corpus	Qrels	Query chars avg	Query chars p50	Query chars p75	Doc chars avg	Doc chars p50	Doc chars p75
NanoCMedQAv2reranking	200	10000	377	50.1	39.0	69.2	100.9	91.0	128.0
NanoCUREv1	200	10000	5181	75.9	78.0	91.0	604.2	511.0	757.0
NanoCmedqa	200	10000	324	52.0	44.0	69.0	157.6	112.0	203.0
NanoMedicalQA	200	2007	200	54.2	50.0	64.0	1102.4	683.0	1251.0
NanoNFCorpus	200	3593	3718	17.1	11.5	20.2	1589.5	1612.0	1868.0
NanoPublicHealthQA	86	86	86	79.8	70.0	92.8	828.2	625.0	1041.8
NanoSciFact	200	5183	226	90.1	83.0	107.2	1499.4	1426.0	1811.5
NanoSciFactPL	200	5183	226	95.5	88.0	117.0	1554.5	1467.0	1871.0
NanoTRECCOVID	50	10000	50	69.2	64.5	76.8	1208.8	1318.0	1735.0
NanoTRECCOVIDPL	50	10000	50	69.4	66.0	80.2	1251.9	1360.0	1789.2

Candidate Quality

nDCG@10 and Recall@100 are computed from the included candidate rankings against the included qrels, then reported as 0-100 scores such as 52.45. Recall@100 uses only the top 100 candidates; an optional rank-101 safeguard positive is not counted in Recall@100.

Dense means microsoft/harrier-oss-v1-270m with the web_search_query prompt and cosine similarity.

Nano split	BM25 tokenizer	BM25 nDCG@10	Dense nDCG@10	Hybrid nDCG@10	BM25 Recall@100	Dense Recall@100	Hybrid Recall@100	Hybrid candidates	Safeguard positives
Mean	-	43.64	51.00	49.02	68.20	76.98	80.78	-	206
NanoCMedQAv2reranking	wordseg@zh	15.27	32.09	25.29	34.91	67.21	62.06	100-101	59
NanoCUREv1	englishporterstop	46.93	50.03	52.62	60.42	66.20	68.15	100-101	14
NanoCmedqa	wordseg@zh	16.69	33.80	25.91	39.12	71.97	65.62	100-101	57
NanoMedicalQA	englishporterstop	54.39	73.08	65.10	92.00	92.50	97.00	100-101	6
NanoNFCorpus	englishporterstop	29.21	30.70	31.82	24.96	34.29	32.57	100-101	48
NanoPublicHealthQA	stemmer@arabic	73.79	81.76	78.47	100.00	100.00	100.00	86	0
NanoSciFact	englishporterstop	70.17	73.34	75.06	94.40	93.25	97.50	100-101	5
NanoSciFactPL	regex	57.50	60.61	65.38	86.22	88.40	92.90	100-101	13
NanoTRECCOVID	englishporterstop	39.83	38.75	31.93	80.00	70.00	96.00	100-101	2
NanoTRECCOVIDPL	regex	32.66	35.85	38.64	70.00	86.00	96.00	100-101	2

Hybrid Safeguard Summary

—Safeguard positives: 206
—Rows limited by corpus size: 86
—Metadata file: reranking_hybrid_metadata.json

Source Links

—Source benchmark: MTEB(Medical, v1)
—clinia/CUREv1: https://huggingface.co/datasets/clinia/CUREv1
—mteb/CMedQAv2-reranking: https://huggingface.co/datasets/mteb/CMedQAv2-reranking
—mteb/CmedqaRetrieval: https://huggingface.co/datasets/mteb/CmedqaRetrieval
—mteb/SciFact-PL: https://huggingface.co/datasets/mteb/SciFact-PL
—mteb/TRECCOVID-PL: https://huggingface.co/datasets/mteb/TRECCOVID-PL
—mteb/medical_qa: https://huggingface.co/datasets/mteb/medical_qa
—mteb/nfcorpus: https://huggingface.co/datasets/mteb/nfcorpus
—mteb/scifact: https://huggingface.co/datasets/mteb/scifact
—mteb/trec-covid: https://huggingface.co/datasets/mteb/trec-covid
—xhluca/publichealth-qa: https://huggingface.co/datasets/xhluca/publichealth-qa

License

NanoMedical is a derived dataset. Users must comply with the licenses, terms, and attribution requirements of the upstream datasets and benchmarks.