Datasetpublic

hakari-bench/NanoBEIR-it

NanoBEIR-it This dataset is a Nano-style retrieval dataset for HAKARI-bench. NanoBEIR-it is the Italian language-specific component of MNanoBEIR. It groups compact BEIR-derived retrieval tasks for efficient evaluation of document ranking in that language. Usage from datasets import load_dataset dataset_id = "hakari-bench/NanoBEIR-it" split = "NanoArguAna" queries = load_dataset(dataset_id, "queries", split=split) corpus = load_dataset(dataset_id, "corpus"… See the full description on the dataset page: https://huggingface.co/datasets/hakari-bench/NanoBEIR-it.

sourceHugging Faceupdated 3mo agoView on Hugging Face

0likes311downloads

Dataset Card

NanoBEIR-it

This dataset is a Nano-style retrieval dataset for HAKARI-bench.

NanoBEIR-it is the Italian language-specific component of MNanoBEIR. It groups compact BEIR-derived retrieval tasks for efficient evaluation of document ranking in that language.

Usage

python

from datasets import load_dataset

dataset_id = "hakari-bench/NanoBEIR-it"
split = "NanoArguAna"

queries = load_dataset(dataset_id, "queries", split=split)
corpus = load_dataset(dataset_id, "corpus", split=split)
qrels = load_dataset(dataset_id, "qrels", split=split)
reranking_candidates = load_dataset(dataset_id, "reranking_hybrid", split=split)

Data Layout

This dataset uses six Hugging Face Datasets configs:

—corpus: documents with _id and text
—queries: queries with _id and text
—qrels: positive relevance labels with query-id and corpus-id
—bm25: BM25 candidate lists with query-id and corpus-ids
—harrier_oss_v1_270m: dense candidate lists from microsoft/harrier-oss-v1-270m
—reranking_hybrid: RRF candidate lists built from bm25 and harrier_oss_v1_270m

Each config has the same Nano split names. NanoNFCorpus includes the full positive qrels (2,518 rows); qrels are not capped to the top-100 reranking depth.

Candidate Construction

—bm25: local BM25 top-500 with automatic tokenizer selection. Auto mode uses wordseg for ja, zh, th, ko, and vi, and regex otherwise. The resolved tokenizer is shown for each split in the Candidate Quality table.
—harrier_oss_v1_270m: dense top-500 from microsoft/harrier-oss-v1-270m. In tables this is shown as Dense; Dense means microsoft/harrier-oss-v1-270m with the web_search_query prompt for queries and cosine similarity over normalized embeddings.
—reranking_hybrid: RRF over bm25 and harrier_oss_v1_270m using rrf_k=100, keeping the RRF top-100.

Safeguard means rank 101 is appended only when RRF top-100 contains no qrels-positive document. Qrels are not capped to fit the top-100 reranking depth. For NanoNFCorpus, some queries have more than 100 positive qrels, so top-100 hybrid candidate coverage is expected to be below 100%; this is a candidate-list diagnostic, not a qrels filtering rule.

Split Statistics

Length statistics are character counts computed with len(str(text)).

Nano split	Queries	Corpus	Qrels	Query chars avg	Query chars p50	Query chars p75	Doc chars avg	Doc chars p50	Doc chars p75
NanoArguAna	50	3635	50	1187.4	1177.0	1344.8	1102.3	980.0	1382.0
NanoClimateFEVER	50	3408	148	152.4	147.5	190.2	1743.1	1575.5	2242.0
NanoDBPedia	50	6045	1158	38.3	36.5	50.0	361.7	394.0	476.0
NanoFEVER	50	4996	57	50.1	45.0	57.0	1290.4	1091.0	1723.5
NanoFiQA2018	50	4598	123	74.6	71.0	91.2	1005.1	724.0	1269.5
NanoHotpotQA	50	5090	100	93.4	83.5	115.8	378.3	323.5	520.0
NanoMSMARCO	50	5043	50	42.0	38.5	49.5	356.6	326.0	406.5
NanoNFCorpus	50	2953	2518	28.5	24.0	38.8	1725.5	1754.0	2029.0
NanoNQ	50	5035	57	54.3	52.5	61.0	575.9	478.0	820.0
NanoQuoraRetrieval	50	5046	70	52.8	50.5	57.8	64.4	56.0	76.0
NanoSCIDOCS	50	2210	244	89.5	87.5	106.5	1062.0	1045.0	1406.8
NanoSciFact	50	2919	56	113.7	111.5	133.2	1631.5	1537.0	1970.0
NanoTouche2020	49	5745	932	56.3	56.0	67.0	2352.8	1090.0	3343.0

Candidate Quality

nDCG@10 and Recall@100 are computed from the included candidate rankings against the included qrels, then reported as 0-100 scores such as 52.45. Recall@100 uses only the top 100 candidates; an optional rank-101 safeguard positive is not counted in Recall@100.

Dense means microsoft/harrier-oss-v1-270m with the web_search_query prompt and cosine similarity.

Nano split	BM25 tokenizer	BM25 nDCG@10	Dense nDCG@10	Hybrid nDCG@10	BM25 Recall@100	Dense Recall@100	Hybrid Recall@100	Hybrid candidates	Safeguard positives
Mean	-	48.57	53.38	53.33	74.62	80.58	82.75	-	23
NanoArguAna	regex	39.34	47.06	41.38	88.00	98.00	98.00	100-101	1
NanoClimateFEVER	regex	27.00	33.89	32.55	57.93	66.43	67.77	100-101	2
NanoDBPedia	regex	53.39	64.27	60.69	69.09	80.75	80.05	100	0
NanoFEVER	regex	77.76	79.72	79.77	96.00	89.00	98.00	100-101	1
NanoFiQA2018	regex	26.33	34.43	34.05	59.01	78.42	78.77	100-101	5
NanoHotpotQA	regex	72.75	75.40	77.62	93.00	92.00	97.00	100	0
NanoMSMARCO	regex	39.57	50.87	47.81	88.00	98.00	98.00	100-101	1
NanoNFCorpus	regex@regex	33.47	26.34	33.77	21.44	26.33	28.58	100-101	4
NanoNQ	regex	37.50	51.33	45.45	82.00	87.00	90.00	100-101	4
NanoQuoraRetrieval	regex	71.30	86.99	80.38	96.00	98.00	100.00	100	0
NanoSCIDOCS	regex	28.67	33.78	34.99	53.67	65.07	64.57	100-101	1
NanoSciFact	regex	67.14	63.81	67.66	88.00	88.00	92.00	100-101	4
NanoTouche2020	regex	57.14	45.99	57.17	77.92	80.57	83.02	100	0

Hybrid Safeguard Summary

—Safeguard positives: 23
—Rows limited by corpus size: 0
—Metadata file: reranking_hybrid_metadata.json

Source Links

—Original dataset: lightonai/NanoBEIR-it
—Final dataset: hakari-bench/NanoBEIR-it

License

NanoBEIR-it is a derived dataset. Users must comply with the licenses, terms, and attribution requirements of the upstream source datasets.