CoolFace
Datasetpublic

DIYIN/ElephantBench

ElephantBench ElephantBench is a closed-book knowledge probe for evaluating whether a language model remembers long-tail facts and recalls the different verified accounts associated with them. The release contains 1,094 English questions. Evaluation code, prompts, construction utilities, and full documentation are available in the ElephantBench GitHub repository. Load the dataset from datasets import load_dataset dataset =… See the full description on the dataset page: https://huggingface.co/datasets/DIYIN/ElephantBench.

sourceHugging Facecc-by-4.0updated 23d agoView on Hugging Face
0likes110downloads
Dataset Card

ElephantBench

<p align="center"> <a href="https://github.com/Tencent/ElephantBench"> <img src="https://img.shields.io/badge/ElephantBench-GitHub-blue?logo=github" alt="GitHub Repo" /> </a> <a href="https://tencent.github.io/ElephantBench/"> <img src="https://img.shields.io/badge/ElephantBench-Leaderboard-ff725e?logo=githubpages&logoColor=white" alt="ElephantBench Leaderboard" /> </a> <a href="https://arxiv.org/abs/2608.28478"> <img src="https://img.shields.io/badge/ElephantBench-Paper-red?logo=arxiv&logoColor=red" alt="Paper" /> </a> </p>

ElephantBench is a closed-book knowledge probe for evaluating whether a language model remembers long-tail facts and recalls the different verified accounts associated with them. The release contains 1,094 English questions.

Evaluation code, prompts, construction utilities, and full documentation are available in the ElephantBench GitHub repository.

[image]

Load the dataset

python
from datasets import load_dataset

dataset = load_dataset("Tencent/ElephantBench", split="test")

Record format

json
{
  "benchmark_id": "4382fd6d-ec5d-5ec8-a3f2-8abc706fa010",
  "item_group_id": "c9ef1c87-3e50-5190-8e8e-37a82ec25634",
  "eval": {
    "question": "What birth date was reported for Mother Teresa?",
    "gold_answers": [
      {"value": "August 26, 1910"},
      {"value": "August 27, 1910"}
    ],
    "preferred_answer": "Reports cite August 26 and August 27, 1910."
  }
}

The target model receives only eval.question. Gold answers are supplied to the judge after generation.

Evaluation

  • —C (complete): all verified answers are covered without a material contradiction.
  • —P (partial): at least one, but not all, verified answers are covered.
  • —F (failed): no verified answer is covered, a material contradiction is present, or the generation/judging request failed.
  • —K (conditional completeness): C / (C + P).

License

cc-by-4.0.

Citation

@article{pan2026elephantbench,
  title={Blind Men and the Elephant: Probing the Epistemic Myopia of LLMs under Long-Tail Divergent Knowledge},
  author={Pan, Zhuoshi and Lu, Junru and Qian, Yan and Zhao, H. Vicky and Yin, Di and Sun, Xing},
  journal={arXiv preprint arXiv:2608.28478},
  year={2026}
}