Modelpublic

junmingg/qwen2.5-coder-7b-text2sql

sourceHugging Faceapache-2.0updated 3mo agoView on Hugging Face

0likes10downloads

Model Card

Qwen2.5-Coder-7B Text-to-SQL (QLoRA)

A QLoRA fine-tune of `Qwen/Qwen2.5-Coder-7B-Instruct` that turns a SQL schema + a natural-language question into a single SQL query. Trained on a single RTX 3090 with Unsloth + TRL, completion-only loss.

Trained and evaluated by junmingg. Code + reproducible eval harness: https://github.com/junmingg/Unsloth-Qwen2.5-Coder-7b-Text-to-SQL-SFT

Results (held-out 500 examples)

Model	Exact match	Semantic equiv.	SQL validity
Base `Qwen2.5-Coder-7B-Instruct` (zero-shot)	3.8%	67.0%	100.0%
+ QLoRA (this model)	78.8%	86.2%	99.2%

[image]

—Exact match = canonicalized string match (via sqlglot); a strict lower bound — different-but-correct SQL fails it.
—Semantic equivalence = an independent LLM judge (GLM-5-Turbo) decides whether the predicted SQL is equivalent to the gold for the question. This is the headline correctness metric.
—SQL validity = fraction of predictions that parse under sqlglot.

Fine-tuning lifted exact-match by +75.0 points and semantic-equivalence by +19.2 points on prompts the model never saw during training (0/500 test prompts appear in the training data). The large exact-match jump is the model learning the dataset's SQL conventions (quoting, value formatting); the semantic jump is genuine correctness beyond formatting.

Label-noise ablation

sql-create-context carries a little label noise — 0.32% of train golds (and 0.60% of test golds) won't parse under sqlglot (WikiSQL artifacts). A controlled variant (`junmingg/qwen2.5-coder-7b-text2sql-filtered`) drops those 80 noisy rows from the training set — same recipe, same held-out test set:

Model	Exact match	Semantic equiv.	SQL validity
Base (zero-shot)	3.8%	67.0%	100.0%
SFT — full 25k	78.8%	86.2%	99.2%
SFT — filtered 24.9k	78.6%	85.4%	99.8%

[image]

Cleaning 0.32% of train labels nudged validity up, accuracy unchanged (within run-to-run noise). On the valid-reference subset the filtered model reaches 100% validity. Full write-up in the repo README.

Usage

The model was trained with a specific system prompt and ChatML format — use the same formatting for best results.

python

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

REPO = "junmingg/qwen2.5-coder-7b-text2sql"
model = AutoModelForCausalLM.from_pretrained(REPO, torch_dtype=torch.bfloat16, device_map="auto")
tok = AutoTokenizer.from_pretrained(REPO)

SYSTEM = ("You are a precise text-to-SQL engine. Given a SQL schema and a natural-language "
          "question, respond with a single valid SQL query and nothing else.")

schema = "CREATE TABLE head (age INTEGER, name TEXT)"
question = "How many heads of the departments are older than 56?"

messages = [
    {"role": "system", "content": SYSTEM},
    {"role": "user", "content": f"Schema:\n{schema}\n\nQuestion: {question}"},
]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(inputs, max_new_tokens=256, do_sample=False)
print(tok.decode(out[0][inputs.shape[1]:], skip_special_tokens=True))
# -> SELECT COUNT(*) FROM head WHERE age > 56

LoRA adapters only (smaller): junmingg/qwen2.5-coder-7b-text2sql-lora. GGUF quants (q4_k_m, q8_0) for llama.cpp / Ollama: junmingg/qwen2.5-coder-7b-text2sql-GGUF.

Training


Base	`Qwen/Qwen2.5-Coder-7B-Instruct` (4-bit QLoRA)
Data	`b-mc2/sql-create-context` (~78.6k rows total). Shuffled (seed 42), then a 25k train subset + a fixed 500 held-out test set. The remaining ~53k rows were intentionally unused — 1 epoch over 25k already saturates this task; adding the rest gives negligible lift while ~doubling/tripling train time.
LoRA	r=16, α=16, dropout=0, all linear layers
Schedule	1 epoch (~1,557 steps), effective batch 16, lr 2e-4 cosine, warmup 0.03
Precision / optim	bf16, `adamw_8bit`, `max_seq_length=2048`
Loss	completion-only (prompt masked; loss on the SQL answer only)
Hardware / time	1× RTX 3090, ~59 min

Evaluation methodology

Both base and fine-tuned models are evaluated with identical prompting and greedy decoding on the same held-out 500-example test set (disjoint from train; verified 0 prompt overlap). Semantic equivalence is scored by an independent LLM judge (GLM-5-Turbo). Reporting both exact-match and semantic-equivalence is deliberate: exact-match is a lower bound, semantic-equivalence is the fairer correctness signal. The full harness is in the linked repo (python -m src.eval, python -m src.judge).

Limitations

—Trained on synthetic, largely single-table schemas (sql-create-context, derived from WikiSQL + Spider); not evaluated for SQL-injection safety or complex multi-join queries.
—The dataset contains a small amount of label noise (~0.3–0.6% of gold answers are themselves unparseable — WikiSQL artifacts); the model's rare "invalid" outputs are predominantly faithful reproductions of those malformed references rather than novel errors.
—Outputs should be validated/parameterized before execution against a real database.

License & attribution

Apache-2.0 (inherited from the base model). Training data: b-mc2/sql-create-context (CC-BY-4.0).