Datasetpublic

hakari-bench/NanoDAPFAM

NanoDAPFAM This dataset is a Nano-style retrieval dataset for HAKARI-bench. NanoDAPFAM contains 18 Nano retrieval splits derived from DAPFAM. Each split keeps up to 200 eligible queries and up to 10000 corpus documents, with exact duplicate query and document text removed where the generator records that policy. Usage from datasets import load_dataset dataset_id = "hakari-bench/NanoDAPFAM" split = "NanoDAPFAMAllTitlAbsClmToFullText" queries =… See the full description on the dataset page: https://huggingface.co/datasets/hakari-bench/NanoDAPFAM.

sourceHugging Faceupdated 4mo agoView on Hugging Face

0likes415downloads

Dataset Card

NanoDAPFAM

This dataset is a Nano-style retrieval dataset for HAKARI-bench.

NanoDAPFAM contains 18 Nano retrieval splits derived from DAPFAM. Each split keeps up to 200 eligible queries and up to 10000 corpus documents, with exact duplicate query and document text removed where the generator records that policy.

Usage

python

from datasets import load_dataset

dataset_id = "hakari-bench/NanoDAPFAM"
split = "NanoDAPFAMAllTitlAbsClmToFullText"

queries = load_dataset(dataset_id, "queries", split=split)
corpus = load_dataset(dataset_id, "corpus", split=split)
qrels = load_dataset(dataset_id, "qrels", split=split)
reranking_candidates = load_dataset(dataset_id, "reranking_hybrid", split=split)

Data Layout

This dataset uses six Hugging Face Datasets configs:

—corpus: documents with _id and text
—queries: queries with _id and text
—qrels: positive relevance labels with query-id and corpus-id
—bm25: BM25 candidate lists with query-id and corpus-ids
—harrier_oss_v1_270m: dense candidate lists from microsoft/harrier-oss-v1-270m
—reranking_hybrid: RRF candidate lists built from bm25 and harrier_oss_v1_270m

Each config has the same Nano split names.

Candidate Construction

—bm25: local BM25 top-500 with automatic language-aware tokenization. The resolved tokenizer is shown in the Candidate Quality table, for example wordseg@ja.
—harrier_oss_v1_270m: dense top-500 from microsoft/harrier-oss-v1-270m. In tables this is shown as Dense; Dense means microsoft/harrier-oss-v1-270m with the web_search_query prompt for queries and cosine similarity over normalized embeddings.
—reranking_hybrid: RRF over bm25 and harrier_oss_v1_270m using rrf_k=100, keeping the RRF top-100.

Safeguard means rank 101 is appended only when RRF top-100 contains no qrels-positive document.

Split Statistics

Length statistics are character counts computed with len(str(text)).

Nano split	Queries	Corpus	Qrels	Query chars avg	Query chars p50	Query chars p75	Doc chars avg	Doc chars p50	Doc chars p75
NanoDAPFAMAllTitlAbsClmToFullText	200	10000	3989	8339.5	6741.0	9976.2	71050.6	46456.5	79059.8
NanoDAPFAMAllTitlAbsClmToTitlAbs	200	10000	3981	8339.5	6741.0	9976.2	777.9	766.0	958.2
NanoDAPFAMAllTitlAbsClmToTitlAbsClm	200	10000	3989	8339.5	6741.0	9976.2	7229.1	6155.5	8658.5
NanoDAPFAMAllTitlAbsToFullText	200	10000	3989	776.0	774.0	934.5	71113.4	46505.5	79214.5
NanoDAPFAMAllTitlAbsToTitlAbs	200	10000	3982	776.0	774.0	934.5	778.0	766.0	958.2
NanoDAPFAMAllTitlAbsToTitlAbsClm	200	10000	3989	776.0	774.0	934.5	7230.6	6157.5	8658.5
NanoDAPFAMInTitlAbsClmToFullText	200	10000	3069	8405.5	6835.5	9976.2	68906.0	46012.0	77773.5
NanoDAPFAMInTitlAbsClmToTitlAbs	200	10000	3062	8405.5	6835.5	9976.2	777.9	766.0	960.0
NanoDAPFAMInTitlAbsClmToTitlAbsClm	200	10000	3069	8405.5	6835.5	9976.2	7225.2	6141.0	8658.5
NanoDAPFAMInTitlAbsToFullText	200	10000	3072	771.3	771.5	934.5	68924.3	46024.5	77773.5
NanoDAPFAMInTitlAbsToTitlAbs	200	10000	3066	771.3	771.5	934.5	777.7	766.0	959.2
NanoDAPFAMInTitlAbsToTitlAbsClm	200	10000	3072	771.3	771.5	934.5	7226.4	6144.5	8658.0
NanoDAPFAMOutTitlAbsClmToFullText	200	10000	1259	9315.7	7026.0	10653.0	71902.3	46121.0	79984.2
NanoDAPFAMOutTitlAbsClmToTitlAbs	200	10000	1257	9315.7	7026.0	10653.0	777.9	769.0	962.0
NanoDAPFAMOutTitlAbsClmToTitlAbsClm	200	10000	1259	9315.7	7026.0	10653.0	7257.2	6126.5	8623.8
NanoDAPFAMOutTitlAbsToFullText	200	10000	1259	786.6	780.0	957.5	71902.3	46121.0	79984.2
NanoDAPFAMOutTitlAbsToTitlAbs	200	10000	1257	786.6	780.0	957.5	777.9	769.0	962.0
NanoDAPFAMOutTitlAbsToTitlAbsClm	200	10000	1259	786.6	780.0	957.5	7257.2	6126.5	8623.8

Candidate Quality

nDCG@10 and Recall@100 are computed from the included candidate rankings against the included qrels, then reported as 0-100 scores such as 52.45. Recall@100 uses only the top 100 candidates; an optional rank-101 safeguard positive is not counted in Recall@100.

Dense means microsoft/harrier-oss-v1-270m with the web_search_query prompt and cosine similarity.

Nano split	BM25 tokenizer	BM25 nDCG@10	Dense nDCG@10	Hybrid nDCG@10	BM25 Recall@100	Dense Recall@100	Hybrid Recall@100	Hybrid candidates	Safeguard positives
Mean	-	24.30	30.73	29.81	39.01	50.85	50.81	-	522
NanoDAPFAMAllTitlAbsClmToFullText	englishporterstop	33.65	43.52	42.15	46.14	58.02	58.39	100-101	6
NanoDAPFAMAllTitlAbsClmToTitlAbs	englishporterstop	28.64	39.97	37.67	35.98	52.80	50.57	100-101	8
NanoDAPFAMAllTitlAbsClmToTitlAbsClm	englishporterstop	33.60	41.56	39.89	44.24	55.39	54.72	100-101	7
NanoDAPFAMAllTitlAbsToFullText	englishporterstop	34.89	41.49	41.75	46.74	56.22	57.09	100-101	4
NanoDAPFAMAllTitlAbsToTitlAbs	englishporterstop	32.81	37.86	37.90	39.22	50.69	50.31	100-101	8
NanoDAPFAMAllTitlAbsToTitlAbsClm	englishporterstop	35.10	40.56	40.88	43.07	53.79	53.77	100-101	6
NanoDAPFAMInTitlAbsClmToFullText	englishporterstop	35.05	44.84	43.75	56.66	70.85	70.58	100-101	7
NanoDAPFAMInTitlAbsClmToTitlAbs	englishporterstop	29.70	41.35	38.05	42.87	64.36	61.04	100-101	10
NanoDAPFAMInTitlAbsClmToTitlAbsClm	englishporterstop	34.73	43.25	41.57	53.62	67.64	66.06	100-101	7
NanoDAPFAMInTitlAbsToFullText	englishporterstop	34.90	42.55	42.28	57.79	68.35	69.05	100-101	5
NanoDAPFAMInTitlAbsToTitlAbs	englishporterstop	33.86	39.23	39.42	47.15	61.77	60.92	100-101	9
NanoDAPFAMInTitlAbsToTitlAbsClm	englishporterstop	35.93	41.25	42.20	52.99	65.55	65.21	100-101	5
NanoDAPFAMOutTitlAbsClmToFullText	englishporterstop	4.61	10.10	8.69	27.55	35.19	38.30	100-101	65
NanoDAPFAMOutTitlAbsClmToTitlAbs	englishporterstop	4.39	8.72	7.14	16.84	28.40	29.56	100-101	77
NanoDAPFAMOutTitlAbsClmToTitlAbsClm	englishporterstop	6.40	9.52	8.11	23.21	34.39	33.16	100-101	72
NanoDAPFAMOutTitlAbsToFullText	englishporterstop	6.38	9.52	8.58	25.34	32.96	35.30	100-101	74
NanoDAPFAMOutTitlAbsToTitlAbs	englishporterstop	5.83	8.72	7.62	20.24	27.29	30.05	100-101	76
NanoDAPFAMOutTitlAbsToTitlAbsClm	englishporterstop	6.99	9.09	9.01	22.55	31.68	30.59	100-101	76

Hybrid Safeguard Summary

—Safeguard positives: 522
—Rows limited by corpus size: 0
—Metadata file: reranking_hybrid_metadata.json

Source Links

—Source benchmark: DAPFAM
—Source benchmark repository: https://github.com/embeddings-benchmark/mteb
—datalyes/DAPFAM_patent: https://huggingface.co/datasets/datalyes/DAPFAM_patent

License

NanoDAPFAM is a derived dataset. Users must comply with the licenses, terms, and attribution requirements of the upstream datasets and benchmarks.