haihanlamu/SciFigQual-Bench

<div align="center">

SciFigQual-Bench

A Benchmark for Scientific Figure Quality Assessment with Full-Manuscript Context

<br/>

   

<br/>

Overview · Highlights · Files · Rubric · eval1200 · Usage · Citation

</div>

<div align="center">

ACL · EMNLP · ICML · NeurIPS &nbsp;|&nbsp; 2020–2025 &nbsp;|&nbsp; 1,144 papers &nbsp;|&nbsp; ~355k citing-paragraph records

<br/>

<img src="docs/figures/fig1_gap.jpg" alt="Figure 1. Isolated-figure evaluation vs. full-manuscript context binding" width="60%">

Figure 1. From isolated-figure evaluation to full-manuscript context: each figure is bound to its caption and citing paragraphs, then scored on five dimensions (1–10).

</div>

Scientific figure quality in peer review is tri-modal: reviewers cross-check the figure \(I\), the caption \(c\), and the citing paragraphs \(\mathcal{T}\). Natural IQA, AIGC alignment, and chart QA typically score isolated visuals without manuscript evidence.

SciFigQual-Bench binds each published CS figure to full-manuscript context and scores five orthogonal dimensions on a unified 1–10 scale.

This Hugging Face release is the gold dataset: 6,308 expert-rated instances, PNG crops, aggregated human means, and the fixed public test split eval1200. Evaluation code (Direct / Sidecar / SFQ-Agent) is in the GitHub repository.

Highlights

• —Full-manuscript context — each instance is \((I, c, \mathcal{T}, m)\): figure image, caption, index-driven citing paragraphs, and metadata.
• —Large-scale gold — 7,609 curated figures from 1,144 papers; 6,308 exportable expert-rated instances in this package (20,166 raw ratings).
• —Top CS venues — ACL, EMNLP, ICML, NeurIPS (2020–2025).
• —Five-dimensional rubric — VC, SL, CC, CTX, MR with L1 evidence gating.
• —Fixed public split — eval1200 is the paper’s 1,200-figure holdout (254 papers, deterministic paper-level sampling).
• —Strong judge baseline — SFQ-Agent + GPT-5.6-Sol (F3) reaches MAE 0.418 and 93.4% within-1 vs. human gold.

Overview

SciFigQual-Bench closes the manuscript-context gap for published CS conference figures:

1. 1.Acquire & curate PDFs from four top venues (62,694 raw → 7,609 clean figures).
2. 2.Bind context via PDF index patterns (not abstract-only heuristics).
3. 3.Annotate with a calibrated five-dimensional rubric and release eval1200.

<div align="center">

<img src="docs/figures/fig2constructionpipeline.png" alt="Figure 2. SciFigQual-Bench construction pipeline" width="60%">

Figure 2. Corpus collection → structure-aware extraction → context binding → five-dimensional rubric → expert validation.

</div>

Benchmark at a glance

<div align="center">

_{Gold instances by venue}

</div>

<div align="center">

<img src="docs/figures/fig4datasetstatistics.png" alt="Figure 4. Dataset statistics" width="60%">

Figure 4. Venue–topic radar, construction funnel, score distribution, domain counts, temporal coverage, and per-dimension means.

</div>

Takeaways (n=6,308): funnel yield \(\eta \approx 12.1\%\) from strict curation; CC is the weakest axis (mean 7.42) vs. SL (8.58); coverage peaks in 2024–2025.

Files

SciFigQual-Bench/
├── figures.jsonl                 6,308 gold instances
├── human_means.csv               aggregated VC / SL / CC / CTX / MR / overall
├── images/                       6,308 PNG crops
├── splits/
│   ├── eval1200.jsonl            public test manifest
│   └── eval1200_meta.json        sampling protocol
├── corpus_stats.json             construction funnel
└── docs/figures/                 paper Figs. 1–4 (shown on this card)

Join keys: figure_id &nbsp;or&nbsp; (paper_id, fig_index)

Source PDFs and raw crawls are not redistributed.

<details> <summary><b>Instance schema</b> _{(click to expand)}</summary>

<br/>

Gold scores are the mean over `annotations[]`. Null means the dimension was gated out (no caption → no CC; no citing text → no CTX).

</details>

Five-Dimensional Rubric

Scores on a 1–10 Likert scale. Overall is a gated mean over available dimensions.

L1 gating. CC is null without a caption; CTX is null without citing text. Instances lacking both are excluded from scoring.

eval1200

The paper’s fixed public test split — 1,200 figures, 254 papers, paper-level all-or-nothing selection (no randomness).

<div align="center">

</div>

On this split, SFQ-Agent (F3, GPT-5.6-Sol) reports MAE 0.418 and 93.4% within-1 agreement vs. human gold. Prompts and judge protocols: code repository.

<div align="center">

<img src="docs/figures/fig3sfqagent.png" alt="Figure 3. SFQ-Agent scoring pipeline" width="60%">

Figure 3. L1 gating → parallel vision / language evidence → cross-modal judge → deterministic Runner aggregation.

</div>

Usage

Download

huggingface-cli download haihanlamu/SciFigQual-Bench \
  --repo-type dataset \
  --local-dir SciFigQual-Bench

Load gold + eval1200

import json
from pathlib import Path

root = Path("SciFigQual-Bench")

with (root / "figures.jsonl").open(encoding="utf-8") as f:
    figures = [json.loads(line) for line in f]

eval_ids = {
    (row["paper_id"], row["fig_index"])
    for line in (root / "splits" / "eval1200.jsonl").open(encoding="utf-8")
    for row in [json.loads(line)]
}

eval1200 = [x for x in figures if (x["paper_id"], x["fig_index"]) in eval_ids]
print(len(figures), len(eval1200))  # 6308  1200

Scoring scripts and the 29-run experiment matrix: github.com/FrankDengAI/SciFigQual-Bench.

License

Released under the MIT License. Figure crops and excerpted manuscript text remain subject to the original venue / publisher terms. This package is not a redistribution of full PDFs.

Citation

If you use SciFigQual-Bench, please cite:

@article{deng2026scifigqual,
  title={SciFigQual-Bench: A Benchmark for Scientific Figure Quality Assessment with Full-Manuscript Context},
  author={Deng, Zihan and Xu, Chuanzhi and Liang, Huiqi and Li, Haoyang and Zhong, Xiaozhen and Yu, Lequan},
  journal={arXiv preprint arXiv:2607.27084},
  year={2026}
}

Paper: arXiv:2607.27084