Datasetpublic

hakari-bench/NanoMIRACL

NanoMIRACL This dataset is a Nano-style retrieval dataset for HAKARI-bench. NanoMIRACL is a compact multilingual benchmark derived from MIRACL. Each language split evaluates monolingual retrieval of Wikipedia passages for natural-language questions. This rebuild uses hotchpotch/miracl-hf-unified dev queries at source revision 21ad00eb467639e927b5badb7c49f4947c6c24ca. For each sampled query, it preserves all source positive passages and expands the split-local corpus to 10,000… See the full description on the dataset page: https://huggingface.co/datasets/hakari-bench/NanoMIRACL.

sourceHugging Faceupdated 3mo agoView on Hugging Face

0likes558downloads

Dataset Card

NanoMIRACL

This dataset is a Nano-style retrieval dataset for HAKARI-bench.

NanoMIRACL is a compact multilingual benchmark derived from MIRACL. Each language split evaluates monolingual retrieval of Wikipedia passages for natural-language questions.

This rebuild uses hotchpotch/miracl-hf-unified dev queries at source revision 21ad00eb467639e927b5badb7c49f4947c6c24ca. For each sampled query, it preserves all source positive passages and expands the split-local corpus to 10,000 documents using source negatives as hard-negative candidates plus deterministic random fill.

Usage

python

from datasets import load_dataset

dataset_id = "hakari-bench/NanoMIRACL"
split = "ar"

queries = load_dataset(dataset_id, "queries", split=split)
corpus = load_dataset(dataset_id, "corpus", split=split)
qrels = load_dataset(dataset_id, "qrels", split=split)
reranking_candidates = load_dataset(dataset_id, "reranking_hybrid", split=split)

Data Layout

This dataset uses six Hugging Face Datasets configs:

—corpus: documents with _id and text
—queries: queries with _id and text
—qrels: positive relevance labels with query-id and corpus-id
—bm25: BM25 candidate lists with query-id and corpus-ids
—harrier_oss_v1_270m: dense candidate lists from microsoft/harrier-oss-v1-270m
—reranking_hybrid: RRF candidate lists built from bm25 and harrier_oss_v1_270m

Each config has the same Nano split names.

Candidate Construction

—bm25: local BM25 top-500 with automatic language-aware tokenization. The resolved tokenizer is shown in the Candidate Quality table, for example wordseg@ja.
—harrier_oss_v1_270m: dense top-500 from microsoft/harrier-oss-v1-270m. In tables this is shown as Dense; Dense means microsoft/harrier-oss-v1-270m with the web_search_query prompt for queries and cosine similarity over normalized embeddings.
—reranking_hybrid: RRF over bm25 and harrier_oss_v1_270m using rrf_k=100, keeping the RRF top-100.

Safeguard means rank 101 is appended only when RRF top-100 contains no qrels-positive document.

Split Statistics

Length statistics are character counts computed with len(str(text)).

Nano split	Queries	Corpus	Qrels	Query chars avg	Query chars p50	Query chars p75	Doc chars avg	Doc chars p50	Doc chars p75
ar	200	10000	386	30.1	27.0	35.0	392.3	276.0	520.0
bn	200	10000	407	47.2	43.5	58.0	446.2	359.0	597.0
de	200	10000	538	45.4	43.0	52.0	457.2	372.0	612.0
en	200	10000	560	39.9	37.5	48.0	471.8	392.0	645.0
es	200	10000	934	47.6	47.0	55.0	453.2	370.0	597.0
fa	200	10000	427	40.0	38.0	46.0	310.7	213.0	413.0
fi	200	10000	328	37.2	33.5	46.2	393.6	326.0	522.0
fr	200	10000	417	43.3	43.0	51.0	385.3	296.0	513.0
hi	200	10000	410	54.8	52.0	65.0	419.3	320.0	559.0
id	200	10000	654	38.3	37.0	45.0	416.5	316.0	567.0
ja	200	10000	373	17.5	17.0	21.0	173.4	135.0	231.0
ko	200	10000	508	21.7	19.0	22.0	205.3	163.0	273.0
ru	200	10000	555	45.5	41.0	53.5	423.3	313.0	569.0
sw	200	10000	405	38.3	37.0	44.0	278.0	204.0	338.0
te	200	10000	211	38.4	36.5	43.0	409.0	295.0	486.0
th	200	10000	343	43.6	40.0	50.2	409.9	346.0	538.0
yo	119	10000	144	37.7	36.0	42.0	176.7	81.0	187.0
zh	200	10000	471	10.9	11.0	12.0	133.4	105.0	176.0

Candidate Quality

nDCG@10 and Recall@100 are computed from the included candidate rankings against the included qrels, then reported as 0-100 scores such as 52.45. Recall@100 uses only the top 100 candidates; an optional rank-101 safeguard positive is not counted in Recall@100.

Dense means microsoft/harrier-oss-v1-270m with the web_search_query prompt and cosine similarity.

Nano split	BM25 tokenizer	BM25 nDCG@10	Dense nDCG@10	Hybrid nDCG@10	BM25 Recall@100	Dense Recall@100	Hybrid Recall@100	Hybrid candidates	Safeguard positives
Mean	-	57.15	76.27	69.00	93.32	94.02	99.26	-	19
ar	stemmer@arabic	63.52	82.23	75.14	97.40	96.58	99.83	100	0
bn	whitespace	50.33	76.61	65.37	95.17	95.57	99.75	100	0
de	stemmer@german	51.72	73.89	64.18	91.78	93.10	98.27	100-101	1
en	englishporterstop	67.74	77.21	74.74	99.38	95.00	99.75	100	0
es	stemmer@spanish	68.61	77.93	74.78	98.06	93.31	100.00	100	0
fa	regex	57.88	64.76	63.34	96.22	88.99	99.30	100-101	1
fi	stemmer@finnish	77.34	86.34	83.32	98.58	95.42	100.00	100	0
fr	stemmer@french	46.58	68.28	58.96	97.96	91.35	99.88	100	0
hi	stemmer@hindi	30.37	68.47	51.74	70.18	91.99	96.00	100-101	7
id	stemmer@indonesian	67.73	70.76	71.71	98.01	95.04	99.83	100	0
ja	wordseg@ja	66.01	77.45	72.23	97.33	91.78	100.00	100	0
ko	wordseg@ko	49.94	69.10	70.26	95.74	92.02	98.12	100-101	3
ru	stemmer@russian	58.87	76.93	68.16	89.50	93.20	98.43	100-101	3
sw	regex	58.52	78.72	72.92	94.89	94.95	99.75	100	0
te	whitespace	52.92	87.20	69.53	87.92	92.50	98.25	100-101	3
th	wordseg@th	62.29	81.01	72.96	95.58	95.25	99.50	100-101	1
yo	regex	58.16	84.16	76.51	92.44	97.48	100.00	100	0
zh	wordseg@zh	40.22	71.91	56.19	83.62	98.90	100.00	100	0

Hybrid Safeguard Summary

—Safeguard positives: 19
—Rows limited by corpus size: 0
—Metadata file: reranking_hybrid_metadata.json

Source Links

License

NanoMIRACL is a derived dataset. Users must comply with the licenses, terms, and attribution requirements of the upstream source datasets.