J0nasW/bfr-diagnostic

Body-Fact Retrieval (BFR) diagnostic

A science-specific long-context retrieval probe released with SciEmbed (Citation-Context Supervision for Scientific Document Embeddings, Findings of EMNLP 2026). The task: given a sentence drawn from the body of a scientific paper that does not appear in its abstract, retrieve the source paper from a pool of full-text documents. Short-context encoders that only see the title+abstract are blind to the content that must be matched; long-context encoders that read the body can recover it, so the probe isolates whether a model's context window is actually exploited on scientific text.

Contents

• —candidates.parquet — 9,749 candidate papers (candidate_id, title, abstract, body, field_of_study, citation_count).
• —queries.parquet — 1,000 body-sentence queries (query_id, query_text, gold_candidate_id).

Construction

• —Queries are single sentences from the middle third of each paper (skipping intro/conclusion), 80–400 characters, with at most 0.2 trigram overlap against the paper's abstract so the abstract alone is not a shortcut.
• —Leakage control. Every candidate paper has a citation count in [1, 4], strictly below the citation_count ≥ 5 threshold used to build the SciEmbed Stage-2 (Signal A/B) training pool. No candidate was a training anchor, positive, or hard negative.
• —Coverage. Roughly 400 papers per field across ~24 S2AG disciplines (Computer Science, Mathematics, Physics, Biology, Medicine, Law, History, Philosophy, …). Bodies span ~11k–59k characters (5th–95th percentile).

Usage

Encode each query and each candidate (title + abstract + body, up to the model's context length), then retrieve by cosine similarity and report Recall@k. See the SciEmbed repository for the full evaluation harness and the robustness variants (same-field pool, Qwen3-paraphrased queries).

• —Code: https://github.com/J0nasW/SciEmbed-release
• —Models: https://huggingface.co/J0nasW

Citation

@inproceedings{wilinski2026sciembed,
  title={SciEmbed: Citation-Context Supervision for Scientific Document Embeddings},
  author={Wilinski, Jonas and F{\"a}rber, Michael},
  booktitle={Findings of the Association for Computational Linguistics: EMNLP 2026},
  year={2026}
}