manifesta/scientific-chart-qa-17k

[image]

<p align="center"> <a href="https://huggingface.co/datasets/manifesta/scientific-chart-qa-17k"><img alt="Hugging Face" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-scientific--chart--qa--17k-2a78d6"></a> <a href="https://www.kaggle.com/datasets/aivarasnavardauskas/scientific-chart-qa-17k"><img alt="Kaggle" src="https://img.shields.io/badge/Kaggle-scientific--chart--qa--17k-20beff"></a> <img alt="rows" src="https://img.shields.io/badge/rows-17%2C070-0b0b0b"> <img alt="figures" src="https://img.shields.io/badge/unique%20figures-6%2C696-0b0b0b"> <img alt="refusal" src="https://img.shields.io/badge/unanswerable-2%2C467%20(14.5%25)-eb6834"> <img alt="decontaminated" src="https://img.shields.io/badge/decontaminated-5%20benchmarks-6b46c1"> <img alt="license" src="https://img.shields.io/badge/100%25%20redistributable-CC--BY--SA--4.0-1baf7a"> <a href="https://github.com/A1VARA5/scientific-chart-qa-17k"><img alt="GitHub" src="https://img.shields.io/badge/GitHub-build%20pipeline%20%2B%20verify.py-181717"></a> <a href="https://github.com/A1VARA5/scientific-chart-qa-17k/actions/workflows/verify.yml"><img alt="verify" src="https://github.com/A1VARA5/scientific-chart-qa-17k/actions/workflows/verify.yml/badge.svg"></a> </p>

Scientific Chart QA, 17,070 rows

A multimodal chart-interpretation dataset built around one idea: teaching a model when not to answer matters as much as teaching it to answer.

[image]

One in seven questions here cannot be answered from its figure, and the correct response is cannot be determined. Baseline vision-language models overwhelmingly guess a plausible-looking number instead. That is the behaviour this set targets.

The four things worth checking before you use it

1. 1.It was decontaminated against five public chart benchmarks, and the audit ships with it. Perceptual hash, CLIP embedding, n-gram and MinHash, 2,430 rows removed, per-stage counts and borderline samples in `decontam_report.json`. The one benchmark that could not be covered is named rather than quietly dropped.
2. 2.Every row is redistributable. 10,644 synthetic rows are own work, 6,426 real figures are CC-BY, CC-BY-SA or CC0, and every figure carries its licence, licence URL and source URL. Zero rows come from papers I wrote.
3. 3.2,467 rows are unanswerable on purpose. Not noise, not scraping failures. Constructed so the figure genuinely lacks the information asked for.
4. 4.Answers are tied to evidence, not to a language model's opinion. Synthetic answers are recomputed from the chart's own data table. Real-figure answers come from the paper authors' caption, with a pointer to the supporting span.

Built for the Adaption AutoScientist Challenge, Data Visualization category, Part 2.

Verify this card

Do not take my word for any of this. Here is how to check it yourself, from a terminal, with no account and no token. The expected results are in the comments.

# 1. Row count, read from the Hub's own parquet index instead of from this page.
#    Expect "num_rows":17070
curl -s "https://datasets-server.huggingface.co/size?dataset=manifesta/scientific-chart-qa-17k" \
  | grep -o '"num_rows":[0-9]*' | head -1

# 2. The decontamination audit, straight out of the report that shipped with the data.
#    Expect rows_in 19500, rows_out 17070, rows_removed 2430, step2_clip_embed 197
curl -sL https://huggingface.co/datasets/manifesta/scientific-chart-qa-17k/resolve/main/decontam_report.json \
  | grep -oE '"(rows_in|rows_out|rows_removed|step2_clip_embed)":[[:space:]]*[0-9]+'

# 3. Per-figure licences, counted from the shipped manifest rather than from my table.
#    Expect CC BY 5780, CC BY-SA 198, CC0 37, public-domain 1
curl -sL https://huggingface.co/datasets/manifesta/scientific-chart-qa-17k/resolve/main/extras/figure_license_manifest.jsonl \
  | python -c "import sys,json,collections; print(collections.Counter(json.loads(l)['license'] for l in sys.stdin if l.strip()))"

# 4. The adapter weights actually download. Expect X-Linked-Size: 33558456
curl -sIL https://huggingface.co/manifesta/adaption_scientific_chart_qa_17k/resolve/main/adapter_model.safetensors \
  | grep -i x-linked-size

# 5. The declared base model resolves. Expect 200
curl -s -o /dev/null -w "%{http_code}\n" https://huggingface.co/api/models/google/gemma-3-27b-it

If step 1 comes back with "the server is busier than usual", that is the Hub warming its cache. Run it again.

What is in the report, and which keys prove the pipeline ran

`decontam_report.json` is 28,316 bytes and opens in a browser. Six keys carry the whole claim.

thresholds holds the exact drop thresholds, borderline_samples holds the 222 images in the review band, and removed_samples lists removed rows one by one with the Hamming distance that triggered each drop. If step 2 above returns those numbers, the pipeline ran. If it does not, I am wrong and you can say so with the file in your hand.

Rerun the pipeline yourself

The scripts that produced all of this ship in `build/`, with a `build/README.md` giving the run order. `build/decontam.py` is the one that wrote the report above.

Everything that built this is public, and checkable

<https://github.com/A1VARA5/scientific-chart-qa-17k>

All seven build scripts, the decontamination audit, and a single-file verify.py that runs 16 checks against this card's claims and the live artifacts:

git clone https://github.com/A1VARA5/scientific-chart-qa-17k
cd scientific-chart-qa-17k
python verify.py

No install step, standard library only. It pulls the row count from the datasets-server and asserts 17,070, SHA-256s the audit file in the repo against the copy published here and requires one digest, downloads the weights and reports their byte size, confirms the base model resolves, and re-derives the licence and task-mix arithmetic. The same script runs in GitHub Actions on every push and once a day on a schedule, so the badge above reports whether these claims still hold today rather than on the day they were written.

Why it exists

Public chart-QA corpora are dominated by synthetic business charts, and every question in them has an answer. Real scientific figures are harder, and real figures routinely do not contain what you ask about. So this set does two things the common corpora do not: it uses genuine scientific figures at scale (forest plots, ROC curves, volcano plots, PCA scatters, survival curves, confusion matrices), and it rewards refusing to answer when the figure cannot support one.

What is in it

[image]

9,940 rows (58.2%) are exactly gradable, meaning numeric, categorical or boolean. The remainder are free-text summaries and captions, which need a judge. The exactly_gradable column is there so you can split the set honestly instead of pretending free text is string-matchable.

22 chart types are represented, including several the mainstream benchmarks barely touch: forest plots with confidence intervals, survival curves, small multiples, dual-axis charts, semilog and log-log plots, confusion matrices.

Ground truth, and how far to trust it

Synthetic rows, 10,644. Rendered by a deterministic engine that emits a full ground-truth sidecar, the underlying data table plus feature annotations, for every chart. Every answer is recomputed from that data table rather than trusted from the generator's own metadata. Rows whose recomputed answer disagreed were dropped (0.3% drop rate).

These charts are deliberately hard. 30 difficulty tags are applied at render time, among them near_equal_values, crossing_lines, log_log, dual_axis_scale_mismatch, colorblind_similar_palette, dense_legend, unlabeled_ticks and negative_values. A chart tagged unlabeled_ticks genuinely has no readable axis numbers, and its questions are unanswerable because of that, on purpose.

Real rows, 6,426. Figures harvested from open-access literature behind a hard licence gate. Answers are caption-grounded: the ground truth comes from the expert caption written by the paper's authors, and every row carries an evidence pointer to the caption span supporting it. No value-reading questions are generated against real figures, because those cannot be recomputed from a data table, so they are not invented.

All 17,070 images were decoded and every image_sha256 was confirmed against the embedded bytes before release. Zero unreadable images.

Numeric tolerance is relative, not exact

Numeric answers are graded within 5% of the true value, following the ChartQA relaxed-accuracy convention. A value read off a bar against a gridline is not accurate to four decimal places, and grading it as though it were would punish correct reading.

If you evaluate against this set, honour the numeric_tolerance column, or you will substantially under-score any model.

Decontamination

The pipeline ran against a poison set of 480 benchmark images and 9,543 benchmark questions pulled from the public chart corpora. 19,500 candidate rows went in, 17,070 came out.

Stage 2 is the one that earns its keep. CLIP caught 197 rows that perceptual hashing missed, restyled clones of benchmark charts that pHash reads as different images. If you decontaminate with hashes alone, that slice stays in your training set and quietly inflates your benchmark score.

Covered: ChartQA, CharXiv, ChartX, DVQA, PlotQA. Not covered: FigureQA, which had no clean streamable mirror to hash against at build time. That is a real gap, stated rather than hidden.

The full audit, including thresholds, per-stage counts, the 222 images in the review band and a sample of what each stage dropped, is published with this dataset as `decontam_report.json`.

Licensing, all 17,070 rows are redistributable

Per row, by where the pixels came from:

The two harvesting sources behind the 6,426 real-figure rows are PMC Open Access, commercial-use subset and the CC-licensed subset of arXiv. Everything else was excluded at the gate.

The aggregate is CC-BY-SA-4.0, and 164 rows are the reason. Share-alike propagates to the whole collection, so a single share-alike slice sets the licence for all of it. Drop those rows and the rest is attribution-only:

attribution_only = ds.filter(lambda r: r["license"] != "cc-by-sa-4.0")   # 16,906 rows

Every figure also carries its own licence, licence URL, source URL and SHA-256 in `extras/figure_license_manifest.jsonl`. Counting that file gives figure-level numbers, not row-level ones: 6,016 figures, CC BY 5,780, CC BY-SA 198, CC0 37, public-domain 1. Those differ from the row table above because decontamination and the per-figure cap changed how many rows each figure ended up contributing. The command in Verify this card reproduces the figure-level counts in one line.

An earlier build carried 620 rows drawn from figures in papers under the default arXiv licence, which does not grant redistribution. I removed them before publication and made the check a hard gate in `build/scrape_cc_figures.py`. Anything NonCommercial, NoDerivatives, default arXiv nonexclusive-distrib, or of unknown provenance never enters.

Results on Adaption

The trained model tied its base. 50 to 50. That is a null result and I am writing it as one. Two separate things were measured, and this is the one people usually leave out.

Adaptive Data (data quality)

Adaption dataset ID 3f347c8b-5724-4f96-9417-251623d8aaa5.

That measures the data, not a model.

AutoScientist (trained model vs base)

The trained model tied its base. 50 to 50. That is a null result and it is written here as one.

What can be said about why, from three runs on the platform:

• —34 optimizer steps. 17,070 rows across 34 steps is roughly 500 rows per step. The corpus was seen about once and the update budget was small. It is hard to move a 27B model on a task it is already competent at with 34 steps.
• —Dataset size did not appear to be the lever. The same chart corpus was run twice: 6,976 rows on a smaller non-VLM base gave 51 against 49, and 17,070 rows on a stronger VLM base gave 50 against 50. A 2.4x increase in rows moved the step count from 21 to 34 and the win rate not at all.
• —Win rate tracked base-model weakness, not corpus size. Across all three of my Part 2 runs, the only one that finished above baseline was the one whose base model was weakest on the domain. A 17,586-row verified math and code corpus against a base that is already strong at maths finished below baseline at 46 against 54. That is three data points, not a controlled experiment, so treat it as an observation rather than a law.
• —The run itself was clean. Peak gradient norm 0.79 against a clipping threshold of 1. Nothing diverged, nothing was clipped hard, no loss spikes. This was not a broken run producing a flat number, it was a stable run producing a flat number.

The honest summary: the artifact worth taking from this entry is the corpus and its audit trail, not the checkpoint.

Ground Truth, the live interface

[Ground Truth, the live interface](https://manifestavisual.adaptionlabs.app/)

A small web app built on the trained model so the refusal behaviour can be checked by hand rather than taken on trust. It has three parts:

• —a drop zone for your own figure,
• —a question panel to ask about it,
• —a "Try to trick it" button, which deliberately asks about a series that was never plotted.

That last button is the point. It walks a visitor into the failure mode this dataset exists to fix, on purpose, so they see what a model does when the answer is not in the picture. A refusal renders as a green "Correctly refused" card, not an error, because refusing an unanswerable question is the correct output and the interface should not punish it visually.

Loading

from datasets import load_dataset

ds = load_dataset("manifesta/scientific-chart-qa-17k", split="train")
row = ds[0]
row["image"]       # PIL.Image, embedded in the parquet, no separate download
row["prompt"]      # question, ending in an <image1> reference token
row["completion"]  # answer

# the anti-hallucination slice
refusals = ds.filter(lambda r: r["task_type"] == "unanswerable")   # 2,467

# only what you can grade by exact match
gradable = ds.filter(lambda r: r["exactly_gradable"])              # 9,940

Fields

Files in this repo

Prompt phrasing

Questions are templated by task type, then routed through a deterministic paraphrase layer so the same question is not repeated verbatim across thousands of images. The most frequent single prompt covers 1.6% of rows, down from 20.8% in an earlier build. 3,655 distinct prompt strings.

They are still templated. If you need free-form natural questions, this is not that dataset.

Limitations, please read these

• —Real-figure answers are caption-grounded, not pixel-grounded. The ground truth is the author's caption. If a caption is vague or overstates the figure, that propagates. There are no value-reading questions on real figures for exactly this reason.
• —62% of rows are synthetic. Realistic in style and genuinely hard, but not drawn from literature. Train on the mix, and say so when you report.
• —The unanswerable share, 14.5%, is far above the 5% originally designed. It is the point of the dataset, but a model trained on this may skew toward refusal. Downsample that slice if you want a more answer-eager model.
• —`chart_type` is a single label per figure. A small number of scraped figures are multi-panel composites that get one label for the whole composite, and a few are low enough resolution that their axis text is not legible.
• —Summarization and caption rows cannot be graded by exact match. Use exactly_gradable.
• —FigureQA is not in the decontamination poison set, since no clean mirror was available.
• —No human review at scale. Samples were inspected by hand. 17,070 rows were not.
• —The trained model tied its base. See the results section. The dataset has not been shown to improve a strong VLM under this platform's default training budget.

Citation

@misc{scientific_chart_qa_17k,
  title  = {Scientific Chart QA: 17,070 evidence-grounded chart questions with a refusal slice},
  author = {Aivaras Navardauskas},
  year   = {2026},
  url    = {https://huggingface.co/datasets/manifesta/scientific-chart-qa-17k}
}

Credits

Built with Adaptive Data and AutoScientist by Adaption. Platform docs: docs.adaptionlabs.ai. The live interface is hosted on Adaption at manifestavisual.adaptionlabs.app.

Real figures remain the copyright of their respective authors under the Creative Commons licences recorded per row. Please honour those terms and cite the source papers where you rely on a specific figure.

Mirror on Kaggle.