Datasetpublic

alialp207/TR-DataAnalystBench

TR-DataAnalystBench A Turkish-language benchmark for evaluating whether language models can perform data-analyst style reasoning over tables and charts: reading a value, finding the maximum/minimum, comparing two years, computing an average or a (signed) percentage change, ranking, summarizing a trend, and — importantly — abstaining when the data does not contain the answer. Gold answers are computed and verified with Python (not produced by a language model), so the benchmark… See the full description on the dataset page: https://huggingface.co/datasets/alialp207/TR-DataAnalystBench.

sourceHugging Facecc-by-4.0updated 3mo agoView on Hugging Face

0likes60downloads

Dataset Card

TR-DataAnalystBench

A Turkish-language benchmark for evaluating whether language models can perform data-analyst style reasoning over tables and charts: reading a value, finding the maximum/minimum, comparing two years, computing an average or a (signed) percentage change, ranking, summarizing a trend, and — importantly — abstaining when the data does not contain the answer.

Gold answers are computed and verified with Python (not produced by a language model), so the benchmark is reproducible and auditable. An automatic evaluator scores numeric (tolerance), categorical (trend), and abstention tasks.

Why this benchmark

Many models are fluent in Turkish yet still fail at numerical reasoning, table understanding, and chart interpretation. TR-DataAnalystBench isolates those abilities with verifiable gold answers and a transparent scoring contract.

The suite (1,436 examples, seven tiers)

Tier	Examples	Tasks	What it targets
`synthetic_v01`	300	5	Easy/medium baseline: single-series tables, basic lookups/compare/percentage
`synthetic_v02`	320	8	Harder & discriminative: multi-series tables, distractor columns, average / nth-highest / cross-series, unanswerable questions, real `hard` labels
`real_pilot`	108	7	Real Türkiye open data (population, GDP, consumer inflation, CO₂) with verified gold
`chart_read_v01`	240	5	Genuine chart reading: label-free charts (no printed values); read which year is max/min, compare years, count above a level, estimate a value, read the trend
`real_anon_v01`	108	7	Contamination-controlled real data: real series with the country/years removed and per-series rescaling, so it measures table reading rather than recall
`reasoning_v01`	180	6	Hard multi-step reasoning: CAGR, fastest-growth year, longest increase streak, conditional average, share of total, ratio between two series
`chart_hard_v01`	180	6	Discriminative chart reading: cluttered two-series, 12-year, off-gridline label-free charts; tight ±5% value reading, closest-pair comparisons, cross-series scanning — designed to challenge frontier vision models

Splits are table-disjoint (the questions sharing a table/chart never cross a split boundary).

chart_read_v01 is the only tier whose charts carry no data labels, so it measures reading values off the axes/gridlines rather than label OCR. Its exact-scored tasks (which year? / compare / count / trend) need only the chart's shape, while value_estimate is scored with an ±8% estimation tolerance.

Task types

Task	Answer	Scoring
`value_lookup`	a value	numeric, ±2% tolerance
`max_min` / `nth_highest`	an extreme / ranked value	numeric
`comparison`	absolute difference between two years	numeric
`cross_series_diff`	difference between two series in a year	numeric
`average`	mean of a series	numeric
`percentage_change`	signed percent change	numeric, ±2 percentage points
`trend_summary`	`artış` / `azalış` / `dalgalı`	categorical label match
`unanswerable`	abstention (`veri yok`)	correct iff the model declines

Input formats: table_only, chart_only (chart image, no table — prevents table leakage), and table_and_chart.

Data fields

Each example is a JSON object with, among others:

—id, dataset_version, language (tr), domain, split
—question_type, difficulty, input_format, chart_type, chart_path
—table: { "columns": [...], "rows": [[...], ...] }
—question, answer (human-readable gold)
—answer_type: numeric | numeric_with_label | text (trend) | abstention
—numeric_answer (or null), trend_class (for trends), target_column, unit
—calculation (how the gold was derived)
—real_pilot only: source_name, source_url, license, country

How to evaluate a model

1.Build prompts from the dataset (a prompt for each example; for chart_only the model is given the chart image, not the table).
2.Collect answers into a CSV with columns id, predicted_numeric_answer, and prediction_text (used for trend words and veri yok).
3.Score with the repository's evaluator:

bash

python scripts/08_evaluate_predictions_file.py \
    --dataset data/processed/real_pilot.jsonl \
    --predictions your_predictions.csv --split test

The evaluator reports overall accuracy plus per-kind accuracy (numeric tolerance, trend label, abstention) broken down by task, input format, and domain. Running it on the provided oracle predictions yields 100%, confirming the scoring pipeline.

Baselines

System	Tier	Accuracy	Notes
Oracle (gold)	all	100%	scoring sanity check (all six tiers)
Noisy baseline	synthetic_v01	~72%	programmatic perturbation reference
Noisy baseline	synthetic_v02	~66%	abstention ~45% (catches hallucination)
"Simple-%" error	reasoning_v01	~83%	a model that confuses CAGR with simple % change loses exactly the CAGR items
ChatGPT (manual, 12-item sample)	real_pilot	~92%	by-hand run; perfect on numeric & abstention, missed one borderline trend
ChatGPT (manual, bulk, 18-item)	reasoning_v01 test	~100%	every well-formed item correct (incl. CAGR); even flagged a malformed question, which led to a gold fix — frontier models are strong at numeric reasoning
GPT-4o / Gemini / Claude (manual, 16-item)	chart_read probe	100% / 100% / 100%	all three tied — simple, round-valued label-free charts are not discriminative; this finding motivated `chart_hard_v01` (cluttered, off-gridline)
GPT-4o / Claude / Gemini (manual, 18-item)	charthardv01 test	100% / 100% / 83%	first separation between frontier models — Gemini missed a 2nd-highest, a count, and a cross-series "closest year" item
GPT-4o / Claude / Gemini (manual, 51-item)	chart_hard probe	98% / 90% / 86%	the dense hard-chart probe separates all three frontier models; the gaps come from careful-scanning tasks (cross-series closest/furthest year, counts, min-year, closest comparison), not from reading round values

The ChatGPT number is a small, manually collected illustration, not a full leaderboard entry. The repository includes a free manual evaluation kit (scripts/16_create_manual_kit.py) so anyone can reproduce/extend it without an API.

Limitations

—In the synthetic/real tiers, charts carry printed data labels, so their chart_only items partly measure label OCR. The chart_read_v01 tier removes labels to isolate genuine chart reading; expand it to make visual reading a larger share of the suite.
—real_pilot uses real, well-known figures, so it can partly reward recall rather than table reading; the real_anon_v01 tier controls for this by removing the country/years and rescaling each series. Both are kept so users can compare authentic-value vs contamination-controlled settings.
—With a few hundred examples, overall rankings are stable but fine-grained per-subgroup numbers carry meaningful confidence intervals.
—Trends are labeled by a deterministic rule (monotonic, or net change ≥5% with a dominant direction, else dalgalı); some borderline series are debatable.

Licensing and provenance

—Datasets: CC-BY-4.0. real_pilot is derived from World Bank Open Data and CDIAC emissions data (ODC-PDDL-1.0 / CC-BY-4.0); per-source provenance and licenses are in data/sources_real/provenance.json. Synthetic tiers are original work.
—Code: MIT (see LICENSE).

Citation

bibtex

@misc{harac2026trdataanalystbench,
  title        = {TR-DataAnalystBench: A Turkish Table and Chart Reasoning Benchmark},
  author       = {Hara\c{c}, Ali Alp},
  year         = {2026},
  howpublished = {\url{https://github.com/alialp5959/TR-DataAnalystBench}}
}