Ryenhails/quote-and-retrieve-eval

Verified evaluation set for evidence attribution in visual documents

The 719-question evaluation set used in **"Evidence Attribution in Visual Document Understanding without Coordinates or Region Labels"** (Liu, Zhang, Xiao, 2026).

Code: github.com/Ryenhails/quote-and-retrieve · Model: Ryenhails/quote-and-retrieve-8b-grpo

What this is

CiteVQA links to source PDFs that are no longer all reachable, and some of the reachable ones differ from the version that was annotated. Evaluating evidence attribution on those questions measures broken links rather than models. This release is the subset whose annotations we could verify against the PDF actually available, together with the normalised ground-truth geometry needed to score against it.

Starting from the 987 single-document questions in the CiteVQA validation release, the filter removes 117 questions whose source PDF does not resolve or is not a valid PDF, 4 whose annotated evidence pages lie outside the downloaded file, 131 whose annotated evidence text does not match the extracted page text, and 16 that cannot be byte-verified, retaining 719 questions over 440 documents (72.9%). The filter inspects only ground-truth annotations and document content, never model outputs, so it cannot favour any system. Documents dropped for text mismatch have text layers at least as rich as the retained ones, so the filter does not select for easy cases.

What this contains, and what it does not

This release contains identifiers, our verification metadata, and our normalised ground-truth boxes. It does not redistribute CiteVQA's question text, answers, or PDFs; download those from the original release and join locally with the included script.

Coordinate normalisation

The released CiteVQA evidence boxes are ordered [y1, x1, y2, x2] on a 0-1000 scale, which does not match the x1y1x2y2 description in the benchmark's own prompt. We established the true order by extracting page text inside candidate boxes and checking it against the annotated evidence content. The boxes here are already converted to x1 y1 x2 y2 in rendered-page pixel coordinates, with 1-based page indices. Using the raw released order will silently produce near zero recall.

Usage

pip install huggingface_hub
huggingface-cli download Ryenhails/quote-and-retrieve-eval --repo-type dataset --local-dir eval_set

Load the metadata directly:

from datasets import load_dataset

ds = load_dataset("Ryenhails/quote-and-retrieve-eval", split="train")
print(ds[0]["index"], ds[0]["gt_necessary"])

To get records the pipeline can run on, join with your own CiteVQA download:

python rehydrate.py \
    --release verified_eval_set.jsonl \
    --citevqa /path/to/CiteVQA/data/validation/CiteVQA.json \
    --pdf-dir /path/to/CiteVQA/data/pdf \
    --out citevqa_singledoc_pub.jsonl

The output is byte-identical in schema to what src/build_citevqa_pub.py produces in the code repository, so every downstream script runs unchanged. See docs/using_the_release.md for the end-to-end path from this file to the paper's tables.

Scoring

A citation matches an annotated element when both lie on the same page and their IoU is at least 0.5. Recall is computed per question over gt_necessary and macro-averaged over questions; precision is scored against gt_all, so citing optional supporting evidence is not an error. The exact semantics, including the two protocol choices that are ours rather than the benchmark's, are in docs/scoring_protocol.md and are asserted by the repository's test suite.

Licensing and provenance

This release contains two kinds of material with different terms.

Our contribution is MIT licensed: the verification filter and its per-question outcome (alignment), the document statistics, and the coordinate normalisation described above.

The ground-truth geometry (gt_pages, gt_necessary, gt_all) is derived from CiteVQA's evidence annotations, reordered and rescaled but not otherwise altered. CiteVQA ships its code and annotations under MIT, and additionally states that the benchmark is provided for academic research and non-commercial use. We ask that you honour that restriction and cite CiteVQA alongside this release.

Nothing here redistributes document content. CiteVQA's source PDFs were collected from publicly accessible web resources and remain the property of their original publishers; CiteVQA deliberately releases annotations and download links rather than the PDFs themselves, and this release follows the same principle one step further by also omitting the question text and answers. There are no page images, no rendered crops, and no PDF bytes in this repository. If you need documents, obtain them through the CiteVQA release.

Citation

@article{liu2026evidence,
  title         = {Evidence Attribution in Visual Document Understanding without Coordinates or Region Labels},
  author        = {Liu, Zhuchenyang and Zhang, Yao and Xiao, Yu},
  journal       = {arXiv preprint arXiv:2607.24651},
  year          = {2026},
  eprint        = {2607.24651},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2607.24651}
}

Please also cite CiteVQA, whose annotations this release builds on.