yhzhang3/papergym-verify

PaperGym — verification set

Each row shows one scientific figure and asks for one number plotted in it. The gold answer was not read off the figure — it was recomputed from the source data table the paper published alongside it, by an LLM pipeline.

Your job is to check whether that gold is actually the number the figure plots.

The pipeline is good at arithmetic and bad at knowing when its own assumptions are wrong. Every defect found so far has the same shape: the recipe executes cleanly and produces a number, but that number is not what the figure draws. A gold that reproduces is not the same as a gold that is correct.

1,306 questions over 1,231 bioRxiv preprints (one figure per paper).

Fields

The question

How the gold was derived

recipe is a recipe, not a copy of the answer. It is one JSON blob — read it as a sentence:

{
 "file": "source_tables/634351v1/634351_file07.xlsx",
 "sheet": "Fig 3C",
 "row_filter": "rows 27-35 (replicate values; excludes the Mean row 36 and E.C row 37)",
 "column": "R",
 "aggregate": "median",
 "normalization": {"type": "none", "reference": null},
 "derived_columns": {}
}

Open 634351_file07.xlsx, sheet Fig 3C; keep rows 27–35; take column R; compute the median → 0.001094

aggregate is direct 724, neg_log10 191, median 162, mean 128, count 48, other 53. derived_columns is non-empty for 268 questions. file is a path inside this repo; source_table_url gives a direct download link so you can fetch one file instead of the whole 4.7 GB.

A :: in the path means the file lives inside a zip: 662894_file06.zip::SourceData/SourceData_Figures.xlsx (one question only).

Model predictions

pred_gpt56luna, pred_gemini38flash, pred_claudeopus5 — what three vision-language models answered when shown only the image and the question.

These are a triage signal, not evidence. When all three miss the gold and agree with each other, the figure usually disagrees with the gold — that pattern is how several confirmed defects were found. When all three hit it, the question is usually sound.

⚠️ assumptions — claims, not evidence

Written by the same model that produced the gold, recording what it assumed and what it believed confirmed each assumption. Every entry carries "confirmed": true — including on every confirmed-bad question found so far.

Read it for hints about what the pipeline was thinking. Never as proof.

Provenance

The procedure

Not: re-run the computation. The pipeline's arithmetic is almost always right.

Instead: hold the figure and the recipe side by side and ask whether they describe the same quantity.

1. 1.Read the recipe as a sentence. What quantity does it compute?
2. 2.Open the figure. What quantity does the panel draw?
3. 3.Ask whether those are the same thing. A median is not a mean. A count is not a p-value. An axis label is not always honest.
4. 4.Only then, if the two seem to match, check the number.

Open the source table (recipe.file, or source_table_url for a direct download) when step 3 is ambiguous — several defects are invisible without it (a merged two-level header, a column of the right name but the wrong meaning, a quantile that swings 14× depending on convention).

Known defect types

Found in the 116 questions audited so far:

17 questions have already been removed for these reasons; 2 were repaired rather than dropped.

Current status — please read before sampling

63 of 1,306 questions (4.8%) have been audited, and they are not spread evenly:

Of the 63, only 34 came from unbiased sequential auditing; the other 29 were deliberately picked as high-risk (large model–gold disagreement). The targeted set had a 26% defect rate, the sequential set 0% — so do not read 26% as the base rate, and do not read 0% as "there are none" either. n=34 puts the true rate somewhere under ~8%.

Sample stratified by `batch`, at random. Sequential auditing only ever covers the front of v2 and tells you nothing about the other 667 questions.

v3 and fl2000 scored lower at every earlier pipeline stage, so their defect rate is expected to be higher than v2's — they are the most valuable to audit first.

How the questions were selected — a caveat

Each figure was passed to a generation model with all of its reconstructible panels, and the model chose which to ask about. That choice was not uniform:

• —panels appearing first in a figure were picked ~10× more often than panels appearing eighth
• —panels whose recipe was a simple count were picked 89% of the time; panels requiring a mean only 46%

So the benchmark is tilted toward panels the generator felt confident about, and mean-type panels — which is where mean_median_confusion lives — are under-represented relative to what is available. Reported accuracies should be read as an upper bound.