CoolFace
Datasetpublic

Kim-el/fever-ner

FEVER Entity Retrieval Benchmark Frozen benchmark for evaluating retrieval methods on the BEIR FEVER dataset (5.4M Wikipedia articles, 6,666 test queries). All data is pre-built so you can test a new method without re-running BM25 or dense retrieval. Files Core benchmark data (for testing new methods) File Size What it is beir_pool.json 31 MB BM25 top-100 candidate pool (k1=1.2, b=0.75). 6,666 queries, each with 100 candidate docids +… See the full description on the dataset page: https://huggingface.co/datasets/Kim-el/fever-ner.

sourceHugging Faceupdated 3mo agoView on Hugging Face
0likes87downloads
Dataset Card

FEVER Entity Retrieval Benchmark

Frozen benchmark for evaluating retrieval methods on the BEIR FEVER dataset (5.4M Wikipedia articles, 6,666 test queries). All data is pre-built so you can test a new method without re-running BM25 or dense retrieval.

Files

Core benchmark data (for testing new methods)

FileSizeWhat it is
beir_pool.json31 MBBM25 top-100 candidate pool (k1=1.2, b=0.75). 6,666 queries, each with 100 candidate docids + BM25 scores. Your method re-ranks these 100 docs.
queries.jsonl31 MB6,666 test queries ({"_id": "...", "text": "..."} per line)
qrels/test.tsv210 KBTREC-style relevance judgments (query-id \t doc-id \t relevance)
query_deltas.csv2 MBPer-query NDCG@10 for every Hadith variant (used for significance testing)

Evaluation scripts

FileWhat it does
fever_benchmark.pyFEVERBenchmark().evaluate(rankings) → {"ndcg@10": float, "recall@100": float}
setup_fever_benchmark.pyRegenerate pool from scratch (downloads BEIR, builds index, runs BM25)
beir_controlled_v3.pyThe definitive controlled ablation (6,666 queries, 0 errors)
replay_verification.pyChecksums query_deltas.csv against v3 aggregate results

Analysis scripts

FileWhat it does
significance_test.pyPaired t-test + randomization test + Cohen's d
export_per_query_v2.pyExports per-query NDCG to CSV with checkpoint resume
significance_report.mdFull significance report generated from the data

Reference results

FileWhat it contains
beir_controlled_v3_results.txtFinal aggregate scores (all variants)
benchmark_manifest.mdFrozen configuration: dataset hashes, Pyserini version, BM25 params

Baseline Scores

SystemNDCG@10
BM25 (k1=1.2, b=0.75)0.5214
MiniLM Dense0.6497
Dense + Muttafaq (best Hadith)~0.6461

How to test a new method

Quick start (using the frozen pool)

python
import json
from huggingface_hub import hf_hub_download
from fever_benchmark import FEVERBenchmark

# 1. Download the frozen BM25 pool
pool_path = hf_hub_download("Kim-el/fever-ner", "beir_pool.json")
with open(pool_path) as f:
    pool_data = json.load(f)

pool = pool_data["pool"]  # {qid: [[docid, bm25_score], ...]}
qids = pool_data["qids"]  # [qid1, qid2, ...]

# 2. Re-rank with your method
#    For each query, take the 100 candidate docs and assign your own scores.
my_rankings = {}
for qid in qids:
    candidates = pool[qid]  # [[docid, bm25_score], ...]
    
    # Replace this with YOUR scoring function:
    scored = []
    for docid, bm25_score in candidates:
        your_score = your_model.score(query_text=qid, docid=docid)
        scored.append((docid, your_score))
    
    # Sort descending by your score
    scored.sort(key=lambda x: -x[1])
    my_rankings[qid] = scored

# 3. Evaluate
bench = FEVERBenchmark()
results = bench.evaluate(my_rankings)
print(f"NDCG@10: {results['ndcg@10']:.4f}")   # Beat 0.6497?
print(f"Recall@100: {results['recall@100']:.4f}")

If you need query text or relevance judgments

python
# Download query text
queries_path = hf_hub_download("Kim-el/fever-ner", "queries.jsonl")
with open(queries_path) as f:
    queries = {json.loads(line)["_id"]: json.loads(line)["text"]
               for line in f}

# Download qrels
qrels_path = hf_hub_download("Kim-el/fever-ner", "qrels/test.tsv")
# TREC format: query-id \t doc-id \t relevance

# Get query text for a specific qid
query_text = queries[qid]

# Get ground truth for a specific qid
# (automatically loaded by FEVERBenchmark.evaluate())

Using the evaluation class directly

python
# The evaluate() method handles qrels loading and NDCG computation.
# Your input:  {qid: [(docid, score), ...]}  — sorted descending by score.
# Output:      {"ndcg@10": float, "recall@100": float, "queries_evaluated": int}

Comparing against baselines

python
bench.verify_reproduction({
    "BM25 (k1=1.2, b=0.75)": 0.5214,
    "MiniLM Dense": 0.6497,
    "Your Method": results["ndcg@10"],
})

Reproducibility

To reproduce the exact BM25 pool from scratch:

bash
pip install pyserini==0.14.0
python setup_fever_benchmark.py

This downloads BEIR FEVER (3.3 GB), builds the Pyserini Lucene index (~6 min), and runs BM25 retrieval (~7 min). Expected BM25 NDCG@10: 0.5214 ± 0.001.

Key Research Findings

From the controlled ablation (6,666 queries, 0 errors):

  1. 1.MiniLM Dense improves BM25 by +24.6% relative (+0.1283 NDCG@10)
  2. 2.Hadith graph signals provide no benefit on top of dense retrieval — all variants were statistically significant but negative (Cohen's d < 0.2)
  3. 3.95.9% of queries are unchanged by Hadith signals; when they fire, they hurt 2:1
  4. 4.The conditional benefit of Hadith on weak lexical systems (+4-5% on FTS5 BM25) does NOT generalize to strong dense retrieval

Requirements

  • —Python 3.8+
  • —pyserini>=0.14.0 (only needed for pool regeneration)
  • —Java 11+ (for Pyserini/Lucene)
  • —huggingface_hub (for downloading)

License

Same as BEIR FEVER — research use.