SuperMax991/spider-text2sql

1---2license: cc-by-sa-4.03task_categories:4- text2text-generation5- table-question-answering6language:7- en8- sql9tags:10- text-to-sql11- sql12- spider13- flan-t514- seq2seq15- nlp16size_categories:17- 1K<n<10K18---19 20# SPIDER Text-to-SQL — Easy Access Version21 22A clean, HuggingFace-native version of the [SPIDER](https://yale-seas.yale.edu/spider/) Text-to-SQL benchmark. The original SPIDER dataset requires manually downloading a ZIP file from the Spider website. This version makes it instantly accessible via `load_dataset`.23 24## What's Included25 26Each row contains the question, gold SQL, the database identifier, and a pre-parsed compact schema string — everything needed to train or evaluate a Text-to-SQL model without any additional preprocessing.27 28| Column | Description |29|---|---|30| `db_id` | Database identifier (e.g. `"concert_singer"`) |31| `question` | Natural language question |32| `query` | Gold standard SQL answer |33| `db_schema` | Compact schema: `"table: col (type), col (type) | table2: ..."` |34| `question_toks` | Tokenized question words (list of strings) |35 36## Splits37 38| Split | Source file | Examples |39|---|---|---|40| train | `train_spider.json` | 7,000 |41| test | `train_others.json` | 1,034 |42 43> **Note**: Following standard SPIDER practice, `train_others.json` is used as the held-out evaluation set. The original SPIDER test set is withheld for the official leaderboard.44 45## Usage46 47```python48from datasets import load_dataset49 50dataset = load_dataset("YOUR_USERNAME/spider-text2sql")51 52train = dataset["train"]53test  = dataset["test"]54 55# Access fields56example = train[0]57print(example["question"])   # "How many heads of the departments are older than 56?"58print(example["query"])      # "SELECT count(*) FROM head WHERE age > 56"59print(example["db_id"])      # "department_management"60print(example["db_schema"])  # "department: Department_ID (number), ... | head: ..."61```62 63## Schema Format64 65The `db_schema` column uses a compact linear format widely used in the Text-to-SQL literature:66 67```68table1: col1 (type), col2 (type), col3 (type) | table2: col4 (type), col5 (type)69```70 71This format is:72- Human-readable and model-friendly73- Fits within typical 512-token input limits for most seq2seq models74- Derived directly from the official SPIDER `tables.json`75 76## Fine-tuning Example (Flan-T5 prompt format)77 78This dataset pairs naturally with prompt-based fine-tuning:79 80```python81def build_prompt(example):82    return (83        f"Translate to SQL: {example['question']}\n"84        f"Database schema:\n{example['db_schema']}"85    )86 87# example["query"] is the target output88```89 90## Difference from Original SPIDER91 92| | Original SPIDER | This Dataset |93|---|---|---|94| Download method | Manual ZIP from website | `load_dataset(...)` ✅ |95| Schema included | Separate `tables.json` | ✅ Pre-joined per example |96| Complex `sql` dict | ✅ Included | ❌ Omitted (noisy for most use cases) |97| `query_toks_no_value` | ✅ Included | ❌ Omitted |98| Ready to train | Requires preprocessing | ✅ Yes |99 100## Source & License101 102- Original dataset: [SPIDER (Yu et al., 2018)](https://yale-seas.yale.edu/spider/)103- License: **Creative Commons Attribution-ShareAlike 4.0 (CC BY-SA 4.0)**104- This derived dataset is released under the same license.105 106## Citation107 108```bibtex109@inproceedings{yu-etal-2018-spider,110    title     = "{S}pider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-{SQL} Task",111    author    = "Yu, Tao and others",112    booktitle = "Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing",113    year      = "2018",114    url       = "https://aclanthology.org/D18-1425",115}116```117