AbhiPoluri/qwen25-coder-7b-bird-sft-v4pro

Qwen2.5-Coder-7B-Instruct distilled from DeepSeek-V4-Pro on BIRD text-to-SQL

LoRA adapter for Qwen/Qwen2.5-Coder-7B-Instruct produced by SFT distillation from DeepSeek-V4-Pro traces on the BIRD text-to-SQL benchmark. Adapter rank 16, ~23M trainable parameters.

Results on 200 held-out BIRD dev questions

By difficulty:

• —Simple (111 q): 55.0% → 64.9% (+9.9 pp)
• —Moderate (68 q): 41.2% → 45.6% (+4.4 pp)
• —Challenging (21 q): 28.6% → 33.3% (+4.7 pp)

Lifts every difficulty band, including challenging — where larger distillation runs in the same project regressed.

How to use

With MLX-LM (Apple Silicon)

from mlx_lm import load, generate
from mlx_lm.sample_utils import make_sampler

# Download the adapter directory
from huggingface_hub import snapshot_download
adapter_path = snapshot_download(repo_id="AbhiPoluri/qwen25-coder-7b-bird-sft-v4pro")

# Load base + adapter
model, tokenizer = load(
    "Qwen/Qwen2.5-Coder-7B-Instruct",
    adapter_path=adapter_path,
)
sampler = make_sampler(temp=0.0)

prompt = tokenizer.apply_chat_template(
    [
        {"role": "system", "content": "You are an expert at writing SQLite queries. ..."},
        {"role": "user",   "content": "<schema>\n\n<question>"},
    ],
    tokenize=False,
    add_generation_prompt=True,
)
print(generate(model, tokenizer, prompt=prompt, max_tokens=1024, sampler=sampler))

Training details

• —Base model: Qwen/Qwen2.5-Coder-7B-Instruct
• —Teacher: deepseek/deepseek-v4-pro via OpenRouter
• —Method: LoRA SFT via MLX-LM
• —Trace count: 708 (from BIRD dev, V4-Pro responses verified by execution-match)
• —LoRA config: rank 16, target modules q/k/v/o_proj, top 16 transformer layers
• —Learning rate: 5e-6 (10× lower than the standard LoRA default — the latter catastrophically regresses Instruct bases)
• —Optimizer: Adam, constant LR
• —Batch size: 1 with gradient checkpointing
• —Iterations: 300 (but iter-50 is the best checkpoint — val loss bottoms there)
• —Hardware: Apple M-Ultra 64 GB (~17 minutes training)

Why iter-50 not iter-300?

Val loss progression:
  Iter 1:   0.929  (initial)
  Iter 50:  0.781  ← best, used for the 55.00% number above
  Iter 100: 0.825
  Iter 300: 1.046  (overfit, worse than starting)

Smaller training sets (~700 examples) hit memorization fast. The default checkpoint that most fine-tuning scripts save (the last iter) is the wrong one to use here. Always pick the lowest-val-loss checkpoint.

Caveats

• —In-domain eval. The 200 held-out questions are from the same 11 BIRD dev databases as training. This measures in-domain refinement, not cross-database generalization. For BIRD train pool we observed pass-rate collapse due to buggy crowd-sourced gold SQL (52% → 6%) — the dev set quality difference is real and important.

• —DPO didn't help. We tried real DPO via TRL (both DPO-from-base and the canonical SFT→DPO chain). Both regressed by 20-27 pp. With only 163 narrow preference pairs and β=0.1, the KL leash was insufficient. For DPO success you need 10k+ diverse pairs and lower β.

Citation / source

Full reproduction recipe, all training logs, eight comparative eval runs, and the article writeup live in the GitHub repo. Search "qwen distillation bird sql 7b m-ultra" for current pointers.

License

Apache 2.0 (matching the base model).