Trustworthy-Information-Access/AuthorityBench

<div align="center"> <h1> AuthorityBench: Benchmarking LLM Authority Perception for Reliable Retrieval-Augmented Generation </h1> </div>

<p align="center"> 📖 <a href="https://arxiv.org/abs/2603.25092"><strong>arXiv Paper</strong></a> | 🤗 <a href="https://huggingface.co/datasets/Trustworthy-Information-Access/AuthorityBench/new/main?filename=README.md"><strong>Dataset</strong></a> | 🛠️ <a href="https://github.com/Trustworthy-Information-Access/AuthorityBench"><strong>Github</strong></a> | </p>

Retrieval-Augmented Generation (RAG) enhances Large Language Models (LLMs) with external knowledge but remains vulnerable to low-authority sources that can propagate misinformation. We investigate whether LLMs can perceive information authority - a capability extending beyond semantic understanding. To address this, we introduce AuthorityBench, a comprehensive benchmark for evaluating LLM authority perception comprising three datasets: DomainAuth (10K web domains with PageRank-based authority), EntityAuth (22K entities with popularity-based authority), and RAGAuth (120 queries with documents of varying authority for downstream evaluation). We evaluate five LLMs using three judging methods (PointJudge, PairJudge, ListJudge) across multiple output formats. Results show that ListJudge and PairJudge with PointScore output achieve the strongest correlation with ground-truth authority, while ListJudge offers optimal cost-effectiveness. Notably, incorporating webpage text consistently degrades judgment performance, suggesting authority is distinct from textual style. Downstream experiments on RAG demonstrate that authority-guided filtering largely improves answer accuracy, validating the practical importance of authority perception for reliable knowledge retrieval.

Dataset Deatils

Three sub-datasets:

• —DomainAuth: 10K web domains annotated with PageRank-based authority scores.
• —EntityAuth: 22K entities (across basketball, movies, and songs) annotated with popularity-based authority (Wikipedia sitelinks).
• —RAGAuth: 120 yes/no queries paired with retrieved documents of varying authority, designed to evaluate the practical impact of authority perception on downstream RAG tasks.

@article{yao2026authoritybench,
  title={AuthorityBench: Benchmarking LLM Authority Perception for Reliable Retrieval-Augmented Generation},
  author={Yao, Zhihui and Zhang, Hengran and Bi, Keping},
  journal={arXiv preprint arXiv:2603.25092},
  year={2026}
}