Datasetpublic

dreeseaw/cleo-value-discovery

Cleo Value-Discovery Benchmark A small (66-question), held-out benchmark for a failure mode that ordinary text-to-SQL evaluations miss: questions whose correct SQL depends on a literal that lives in the data, not the schema. The schema tells you a column is named status; only the data reveals its values are {'O','C','X'}. The schema shows to_date; only the data reveals that "current" is encoded as the sentinel '9999-01-01'. A one-shot text-to-SQL model has to guess these… See the full description on the dataset page: https://huggingface.co/datasets/dreeseaw/cleo-value-discovery.

sourceHugging Facecc-by-sa-4.0updated 4mo agoView on Hugging Face

0likes22downloads

Dataset Card

Cleo Value-Discovery Benchmark

A small (66-question), held-out benchmark for a failure mode that ordinary text-to-SQL evaluations miss: questions whose correct SQL depends on a literal that lives in the data, not the schema.

The schema tells you a column is named status; only the data reveals its values are {'O','C','X'}. The schema shows to_date; only the data reveals that "current" is encoded as the sentinel '9999-01-01'. A one-shot text-to-SQL model has to guess these literals, and a wrong guess returns the wrong rows — often silently (an empty result, or a plausible-but-wrong count). This benchmark isolates that gap: a model must discover the value (by inspecting the data) to answer correctly.

It is the evaluation suite used to measure `dreeseaw/cleo`, a 2B tool-using SQL analyst. On this suite a one-shot baseline scores 13.6%; the tool-using model scores 51.5%.

What's in each row

field	description
`id`	unique question id
`db_id`	source database identifier
`source`	upstream database the schema comes from (see Provenance)
`question`	the natural-language question
`convention`	the value-discovery annotation — the real stored value/code/convention the answer hinges on (e.g. "Current assignment is indicated by `to_date = '9999-01-01'`")
`gold_sql`	reference read-only SQL
`schema_ddl`	`CREATE TABLE` DDL for the tables involved (self-contained)
`canonical_rows`	the gold answer (denotation) — execute `gold_sql` and you should get these rows

How to use it

Score by denotation, not string match: run a model's SQL and the gold_sql, compare the row-sets (order-insensitive). A prediction is correct iff it returns the same rows as canonical_rows.

python

from datasets import load_dataset
ds = load_dataset("dreeseaw/cleo-value-discovery", split="train")
ex = ds[0]
print(ex["question"])      # "How many employees are currently assigned to each department?"
print(ex["convention"])    # "Current assignment is indicated by to_date = '9999-01-01' ..."
print(ex["gold_sql"])

The convention field is provided for analysis (it's the thing a model must discover) — do not feed it to the model if you're measuring value-discovery.

Provenance & attribution

The questions, the convention annotations, and the gold SQL are original to this benchmark. The database schemas are derived from public datasets and are referenced, not redistributed wholesale — only the small CREATE TABLE DDL and the gold answer rows are included:

source	questions
Spider	35
seaborn-data	12
AdventureWorksLT (Microsoft sample)	5
MySQL `employees` sample DB	4
MySQL `world` sample DB	4
TPC-H	4
TPC-DS	2

This benchmark contains no SynSQL data. (SynSQL was used elsewhere in Cleo's training; it is referenced there, not copied, and none of it appears here.)

License

Released under CC BY-SA 4.0, the share-alike license of the largest upstream source (Spider). Please attribute both this benchmark and the upstream schema sources above.

Citation

@misc{cleo_value_discovery_2026,
  title  = {Cleo Value-Discovery Benchmark},
  author = {Dreeseaw},
  year   = {2026},
  url    = {https://huggingface.co/datasets/dreeseaw/cleo-value-discovery}
}