lxSYSU/DriftSQL-Recovery

DriftSQL-Recovery

DriftSQL-Recovery is an execution-verified dataset for training and evaluating SQL agents under schema, business-knowledge, and interaction drift. It contains database-isolated recovery tasks, canonical seven-tool trajectories, real on-policy failures, Recovery SFT examples, hard replay, and full-episode GRPO records.

The accompanying implementation, data factory, sandbox, reward, and evaluation code are available in DriftSQL.

Release scope

The public tasks span clean, add_column, rename_column, rename_table, replace_column, and compound scenarios. Interaction profiles cover direct_clean, schema_only, knowledge_only, and must_ask behavior.

Deliberately excluded

• —Fresh Blind320 is not published. Its rows and answers remain sealed and were not read while preparing this release.
• —No raw SQLite databases are redistributed. Database assets must be obtained from their upstream sources with the pinned revisions in the DriftSQL repository.
• —Checkpoints, local filesystem paths, access tokens, and machine-specific runtime artifacts are not included.

These exclusions keep the final blind evaluation meaningful and avoid duplicating roughly 52 GiB of upstream database assets.

Loading

from datasets import load_dataset

tasks = load_dataset("lxSYSU/DriftSQL-Recovery", "tasks")
failures = load_dataset("lxSYSU/DriftSQL-Recovery", "failure_trajectories")
sft = load_dataset("lxSYSU/DriftSQL-Recovery", "sft_mix")
grpo = load_dataset("lxSYSU/DriftSQL-Recovery", "grpo")

Paths that pointed to local upstream assets have been normalized to hf:// or repo:// identifiers. They are provenance references, not bundled database files.

Tool protocol

The maintained environment exposes a dynamic subset of seven actions:

get_schema_version
inspect_schema_diff
get_schema
ask_user
get_knowledge_definition
execute_sql
submit_solution

SQL targets and canonical trajectories were validated by executing original, stale, and repaired SQL against isolated SQLite sessions. The runtime enforces read-only authorization, timeouts, rollback, result-contract validation, and dynamic action masks.

Data construction

The task factory derives deterministic schema and interaction changes from public BIRD-family and SIX-GYM SQLite assets. Splits are assigned by db_id, not by random row:

• —Train: 2,400 tasks over 60 databases;
• —Tune: 432 tasks over 18 disjoint databases;
• —Fresh Blind: 320 tasks over 20 additional databases, withheld from this public release.

For on-policy collection, 600 difficult Train tasks were sampled with multiple seeds, producing 2,400 rollout outcomes and 1,066 unique failures. Failure mining emphasizes wrong retrieval after schema diff, successful execution without submission, must-ask mistakes, and compound recovery.

Source data and attribution

This release derives from the following CC BY-SA 4.0 datasets:

• —BIRD23 Train Filtered
• —SIX-GYM-SQLite
• —BIRD-Critic-SQLite
• —Mini-Interact

Their licenses and terms remain applicable. DriftSQL adds versioned drift generation, verified tool trajectories, on-policy failure annotations, training mixtures, and leakage controls. This dataset is therefore released under CC BY-SA 4.0 rather than a software license.

Limitations

• —The released execution environment targets SQLite.
• —The tasks and tool traces are English; the DriftSQL application provides a separate Chinese-to-English input adapter.
• —The 2,400-row rollout index contains metadata for every rollout, while full trajectories are published for the 1,066 mined failures.
• —Tune is intended for checkpoint selection and must not be reported as a final blind result.
• —SQL execution should always occur in an isolated, read-only sandbox.

Reproducibility

Dataset revisions, generation scripts, split policies, and validation gates are maintained in the DriftSQL repository. The release metadata records that zero Fresh Blind rows and zero raw SQLite files are included.