CoolFace
Datasetpublic

uthmanjinadu/authority-bias-paper-recommendation

Authority Bias in Conversational Search Engines for Academic Paper Recommendation Dataset accompanying the paper "Authority Bias in Conversational Search Engines for Academic Paper Recommendation" (EMNLP 2026, Main Conference). Code: https://github.com/jinaduuthman/Authority-Bias-In-Conversational-Search-Engine This is a content-controlled counterfactual audit of authority bias in LLM paper recommendation. Each paper's content (title + abstract) is held fixed while its authority… See the full description on the dataset page: https://huggingface.co/datasets/uthmanjinadu/authority-bias-paper-recommendation.

sourceHugging Facecc-by-4.0updated 1mo agoView on Hugging Face
0likes19downloads
Dataset Card

Authority Bias in Conversational Search Engines for Academic Paper Recommendation

Dataset accompanying the paper "Authority Bias in Conversational Search Engines for Academic Paper Recommendation" (EMNLP 2026, Main Conference). Code: https://github.com/jinaduuthman/Authority-Bias-In-Conversational-Search-Engine

This is a content-controlled counterfactual audit of authority bias in LLM paper recommendation. Each paper's content (title + abstract) is held fixed while its authority metadata (venue, author h-index, citations, affiliation) is manipulated across three conditions, so any change in an LLM's recommendation is attributable to authority signals alone.

Contents

FileDescription
papers.json1,250 collected papers (50 per topic × 25 CS topics) from Semantic Scholar + OpenAlex.
experiment_sets.jsonThe 250 queries and, for each, the 10-paper candidate set under all three conditions (original, flipped, boosted), each candidate carrying its authority components.
experiment_conditions.jsonPer-topic candidate pools under each condition (pre query-assembly).
responses/open_weight_main.jsonParsed model responses for the five open-weight models (main experiment).
responses/closed_weight/*.jsonResponses for the three frontier models (gpt-5.4, gemini-3-flash-preview, claude-sonnet-4-6) and the gpt-4o-mini tier ablation.
responses/pilot_1n/*.json1:N flip pilot runs (Gemma 2, Llama 3.1, Mistral) used to derive the authority-signal weights.
author_score_ablation/papers_with_author_score.jsonPapers annotated with the author-score variable for the appendix ablation.

Schemas

Paper (papers.json is a dict {topic: [paper, ...]}): paper_id, title, abstract, year, venue, citation_count, url, open_access_pdf, authors, tier, topic, doi

Query + candidate set (experiment_sets.json is a dict {topic: [query_set, ...]}): each query_set = {query_id, topic, query, candidates: {original: [...], flipped: [...], boosted: [...]}}; each candidate is a paper record plus condition and authority_components: {venue, median_h, max_h, citations, affiliation, composite} (all min-max normalized within topic; composite uses the derived weights).

Response record (responses/open_weight_main.json is a list; the closed_weight/ and pilot_1n/ files wrap records under {"metadata": ..., "results": [...]}): model, variant, condition, topic, query_id, query, recommended, recommended_paper_id, recommended_title, response, elapsed_seconds, prompt_length (variant ∈ {baseline, antiauthority, contentfirst}; recommended is the 1-indexed picked candidate.)

Loading

The flat response files load directly with the datasets library:

python
from datasets import load_dataset
resp = load_dataset("json", data_files="responses/open_weight_main.json", split="train")

The nested files (papers.json, experiment_sets.json, experiment_conditions.json) are best read with plain json:

python
import json
sets = json.load(open("experiment_sets.json"))

Design summary

  • —25 CS topics, 10 queries each (250 queries); 10 candidate papers per query.
  • —Conditions: original (real metadata), flipped (high↔low authority swap), boosted (mid-tier inflation).
  • —Instructions: baseline, anti_authority (mild), content_first (strong).
  • —Authority score: 0.353·venue + 0.292·median_h + 0.187·max_h + 0.137·citations + 0.031·affiliation (weights derived from the 1:N pilot via logistic regression + dominance analysis).

License

Released under CC BY 4.0. Paper metadata is sourced from Semantic Scholar and OpenAlex.

Citation

bibtex
@inproceedings{jinadu2026authority,
  title     = {Authority Bias in Conversational Search Engines for Academic Paper Recommendation},
  author    = {Jinadu, Uthman and Ghazvinian, Parsa and Budathoki, Anjila and
               Ampel, Benjamin M. and Sunderraman, Rajshekhar and Ding, Yi},
  booktitle = {Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year      = {2026}
}