rasinmuhammed/verified-sql-rewards

Verified SQL Rewards

A text-to-SQL corpus where every reward carries a machine-checkable proof that it is correct.

Each item is a natural-language question, a gold SQL query, and an expected answer over a multi-table database. What makes it different is the direction it was built in: the answer is declared first, and the database is solved backward to satisfy it exactly. No one read a database and guessed what a query should return.

from grader import load_questions, reward, schema_text

for q in load_questions("questions.jsonl"):
    sql = my_model(q["question"], schema_text(q))
    r   = reward(q, sql)     # 1.0 or 0.0, exact match against a declared answer

That is the entire integration surface. One function, one dependency (duckdb).

Why this exists

Reward signal for SQL reasoning is measurably corrupted, and the corruption is documented rather than suspected:

Sources: Jin, Choi, Zhu and Kang, Text-to-SQL Benchmarks are Broken, CIDR 2026 (expanded at VLDB 2026); OmniSQL / SynSQL-2.5M, VLDB 2025, per their own human evaluation.

Correcting those annotations moves leaderboard positions by up to nine places, so the errors are large enough to invert published conclusions. The same group separately showed that noisy data is destructive to RLVR (Zhu and Kang, arXiv:2603.16140); to get clean training data for that paper they hand-corrected 600 BIRD instances one at a time.

The most common single error class in the audit, 57.8%, is E2: the annotator misunderstood the schema or the data they were querying. That class cannot occur here, because nobody reads the data to produce an answer. The answer exists before the data does.

What is guaranteed, and how it was checked

1. Every shipped answer is independently verified. The generator writes files. DuckDB, which shares no code with the generator, loads those files and executes each gold SQL against them. A question ships only if the result matches the declared answer exactly. Everything else is dropped and logged.

The check is not cosmetic. Adding one cent to a single row of a fact table containing thousands breaks it:

Declared             : 1411500.0
Clean run            : 1411500.0    match
After +$0.01, one row: 1411500.01   no match

2. The answers are engine-independent, not DuckDB artifacts. 1,450 aggregate answers were recomputed in pandas, a different language with no SQL involved: 1,450 / 1,450 agreed.

3. The gold SQL reproduces, end to end, from the shipped files. grader.py re-runs every gold query through the public loading path and refuses to pass if any question fails to reproduce.

4. Financial invariants hold exactly. general_ledger instances satisfy double-entry accounting: within every journal entry total debits equal total credits, the global trial balance nets to zero, and no line carries both a debit and a credit. Verified in both DuckDB and pandas across every instance.

5. It is not gameable by a constant. Always answering 0 scores 1.83%. Numeric answers are not suspiciously round: 76.8% carry real cents, and only 0.48% end in 000.00.

6. It is reproducible. Specification seeds are SHA-256 derived (not Python's hash(), which is salted per process), verified identical across runs. Both the spec seed and the generation seed are recorded in every instance manifest.

What is NOT claimed

This section matters more than the one above.

• —E1 errors are closed by audit, not by construction. Question text and gold SQL derive from the same template, which prevents per-instance drift. But a wrong template would make every question from it wrong identically, and the verifier would never catch it, because it only checks SQL against data, never English against SQL. All templates were therefore audited by hand, English against SQL, one at a time. That is a weaker guarantee than E2 and is stated as such deliberately.
• —This is not a BIRD or Spider replacement. Questions are generated, so they probe a narrower band of reasoning than human-authored ones. The contribution is a clean reward signal, not harder questions.
• —Ambiguity is mitigated, not eliminated. Templated phrasing removes most wording ambiguity. Wording is still wording.
• —Free text is not realistic and is not the point. Names and labels are grammar-generated placeholders. Nothing here tests language realism, and it should not be used to.
• —Phrasing repeats. Each question kind draws from a hand-written bank of paraphrases, giving 1,055 distinct phrasings (up from 188 before the paraphrase layer). Across 109,306 questions that still means a given phrasing recurs roughly 100 times. The linguistic surface is far wider than a single template per kind, and still far narrower than a human-authored benchmark. Train on this for SQL construction against a clean reward, not for breadth of natural-language understanding.
• —Entities are fictional by construction. No real company, person, or transaction appears.

Composition

Seven schema families, each a financially coherent domain:

Specifications are randomised per instance (anchor values, time window, row counts, rates, share splits, category vocabularies), so instances carry genuinely different correct answers rather than one answer set over many data realisations.

Question kinds

Single-table aggregates. Period totals, grand totals, row counts, argmax month, group shares, rate anchors.

Identity checks. Foreign-key orphan counts, per-entry ledger balance, trial balance.

Composed — harder SQL whose answers are still declared, because declared facts compose:

-- running total first crossing a threshold: CTE + window
WITH m AS (SELECT date_trunc('month', "captured_at") AS mo,
                  SUM("captured_amount") AS tot
           FROM "transactions" WHERE ... GROUP BY 1),
     c AS (SELECT mo, SUM(tot) OVER (ORDER BY mo) AS running FROM m)
SELECT strftime(mo, '%Y-%m') FROM c WHERE running > 4042555.42
ORDER BY mo LIMIT 1

Also: month-over-month deltas, percentage growth, quarter totals, Nth-highest month (ROW_NUMBER), months above the mean, max-minus-min spread, peak share.

Joined — multi-table analytical SQL against a dimension table, where the declared per-category totals carry through the join:

SELECT ROUND(SUM(f."captured_amount"), 2)
FROM "transactions" f
JOIN "card_brand_dim" d ON f."card_brand" = d."card_brand"
WHERE d."network_origin" = 'global' AND f."captured_at" >= ... AND ...

Layout

<instance>/
  tables/*.parquet     the database (DuckDB reads these natively)
  questions.jsonl      shipped, verified questions
  certificate.json     independent DuckDB execution results
  manifest.json        dropped_questions, spec_seed, gen_seed
  verify.py            re-run the verification yourself
questions.jsonl        every question, consolidated, with db_path
dropped.jsonl          every refused candidate, with declared vs observed
grader.py              reward(question, predicted_sql) -> 1.0 | 0.0
STATS.md               corpus statistics

Each record carries question, gold_sql, expected_answer, answer_type, round_decimals, tags, source, db_path, and question_original (the pre-paraphrase phrasing, kept for traceability).

The refused questions ship too

dropped.jsonl lists every candidate that was generated and then refused, with its declared answer, the observed answer, and the reason. Almost all are rate anchors, where a declared rate such as 2% is not exactly reachable over an integer number of rows.

This is published rather than hidden because a corpus claiming verified answers should show what its verifier rejected. None of the four corpora in the table above report a rejection set.

Generated with

Misata, an open-source declarative synthetic data engine. pip install 'misata[evalpack]'

Citation

@misc{verified_sql_rewards_2026,
  title  = {Verified SQL Rewards: a text-to-SQL corpus with certified answer keys},
  author = {Muhammed Rasin},
  year   = {2026},
  url    = {https://huggingface.co/datasets/rasinmuhammed/verified-sql-rewards}
}