cwolff/schemas

SQaLe — schemas

Unique database schemas and their synthetic contents, one row per schema. The questions live in [cwolff/queries](https://huggingface.co/datasets/cwolff/queries), joined on schema id.

These two columns were previously stored inline on every question row of cwolff/data_work_in_progress. With ~25 questions per schema that was a ~25x duplication of the largest columns in the corpus; holding them once here is the entire point of the split.

Columns

Materializing a SQLite database

import json, sqlite3
from datasets import load_dataset

s = load_dataset("cwolff/schemas", split="train")[0]
con = sqlite3.connect(":memory:")
for stmt in s["Full schema"].split(";"):
    if stmt.strip():
        con.execute(stmt)
for table, rows in json.loads(s["Schema content"]).items():
    if not rows:
        continue
    cols = list(rows[0])
    con.executemany(
        f'INSERT OR IGNORE INTO "{table}" ({",".join(chr(34)+c+chr(34) for c in cols)}) '
        f'VALUES ({",".join("?" * len(cols))})',
        [[r.get(c) for c in cols] for r in rows],
    )
con.commit()

Splits

Schema-disjoint from cwolff/queries and using the same assignment, so load_dataset(..., split="test") gives exactly the schemas the test questions need.

Covering 111,141 questions in cwolff/queries.