rasinmuhammed/ecommerce-analytics-sql-evaluation
Ecommerce Analytics SQL Evaluation (declared GMV, verified answer key) An evalpack: an evaluation database generated from the answer key, not annotated after the fact. A VLDB 2026 audit found 52.8% of BIRD Mini-Dev answer keys wrong because benchmarks annotate answers onto existing databases; this dataset inverts the order. The declared properties (curves, shares, identities) are the specification, the database is generated to satisfy them exactly, and every shipped question was… See the full description on the dataset page: https://huggingface.co/datasets/rasinmuhammed/ecommerce-analytics-sql-evaluation.
0100
1#!/usr/bin/env python32"""Independently re-verify this evalpack: run every gold SQL with DuckDB3against the CSVs in tables/ and compare to the expected answers.4 5Usage: python verify.py (exits 1 on any mismatch)6"""7import json8import sys9from pathlib import Path10 11import duckdb12 13HERE = Path(__file__).parent14con = duckdb.connect()15for csv in sorted((HERE / "tables").glob("*.csv")):16 path = str(csv.resolve()).replace("'", "''")17 con.execute(18 'CREATE VIEW "%s" AS SELECT * FROM read_csv_auto(\'%s\')'19 % (csv.stem, path)20 )21 22failures = 023total = 024for line in (HERE / "questions.jsonl").read_text().splitlines():25 if not line.strip():26 continue27 q = json.loads(line)28 total += 129 row = con.execute(q["gold_sql"]).fetchone()30 observed = row[0] if row else None31 expected = q["expected_answer"]32 if q["answer_type"] == "string":33 ok = str(observed) == str(expected)34 else:35 nd = q.get("round_decimals", 0)36 try:37 ok = (38 observed is not None39 and abs(round(float(observed), nd) - round(float(expected), nd))40 < 1e-941 )42 except (TypeError, ValueError):43 ok = False44 status = "OK " if ok else "FAIL"45 if not ok:46 failures += 147 print(f"{status} {q['id']}: expected={expected} observed={observed}")48 print(f" {q['gold_sql']}")49 else:50 print(f"{status} {q['id']}: {expected}")51 52print(f"\n{total - failures}/{total} verified exactly")53sys.exit(1 if failures else 0)54