AaronTekle/SQLQwen

SQLQwen - Qwen2.5 Text-to-SQL Fine-Tuning with LoRA

LoRA fine-tuning of Qwen2.5-Coder-0.5B-Instruct for Text-to-SQL generation using the Gretel synthetic_text_to_sql dataset.

Text-to-SQL systems translate natural-language questions into SQL using a provided database schema or SQL context. SQLQwen is designed to specialize a lightweight code-focused language model for this task while keeping the fine-tuning process parameter-efficient and practical on day-to-day (normal) hardware.

Given:

1. 1.a database schema or SQL context
2. 2.a natural-language request

the model generates the relevant SQL query

Example

Database context:

CREATE TABLE customers (
    id INTEGER PRIMARY KEY,
    name TEXT,
    country TEXT,
    revenue DECIMAL(12, 2)
);

Natural-language request:

Find the five customers with the highest revenue.

Expected output:

SELECT id, name, country, revenue
FROM customers
ORDER BY revenue DESC
LIMIT 5;

Why LoRA

LoRA (Low-Rank Adaptation) provides a parameter-efficient alternative to full fine-tuning.

Instead of updating all parameters in the pretrained model, LoRA freezes the original model weights and introduces small trainable low-rank matrices into selected Transformer layers.

for this project, LoRA adapters are applied to the attention projection layers:

q_proj
k_proj
v_proj
o_proj

this reduces the number of trainable parameters, GPU memory requirements, and adapter storage size while preserving the capabilities of the original pretrained model.

Only 2,162,688 of 496,195,456 parameters, or approximately 0.4359%, were trainable during fine-tuning.

Model

• —Base model: `Qwen/Qwen2.5-Coder-0.5B-Instruct`
• —Fine-tuning: PEFT LoRA
• —Training: TRL SFTTrainer
• —LoRA rank: 16
• —LoRA alpha: 32
• —LoRA dropout: 0.05
• —Target modules: q_proj, k_proj, v_proj, o_proj
• —Training objective: Completion-only supervised fine-tuning

Qwen2.5-Coder was selected because SQL generation is fundamentally a structured code-generation task rather than conventional natural-language classification.

Dataset

Gretel Synthetic Text-to-SQL

`gretelai/synthetic_text_to_sql`

Gretel Synthetic Text-to-SQL dataset provides natural-language SQL requests, database context, target SQL queries, and supporting metadata.

3 fields are used directly during training:

Training Run

Training was completed locally on an NVIDIA GeForce RTX 3050.

Validation loss decreased from 0.2839 at the first logged evaluation to a best value of 0.2198 at checkpoint 2,400. The final checkpoint shows an evaluation loss of 0.2200, indicating that training had largely converged by the end of the second epoch.

Held-Out Generation Evaluation

A held-out generation benchmark of 50 examples was used to evaluate SQL generation code quality.

All 50 generated SQL queries were successfully parsed by SQLGlot, resulting in a 100% syntax-validity rate.

28% exact-match score uses strict string-level comparison. Semantically or execution-equivalent SQL queries may differ from the reference query while still producing the correct result, so exact match should not be interpreted as the model's full semantic accuracy.

Model Limitations:

• —model may hallucinate tables or columns when the supplied database context is incomplete.
• —0.5B parameter base model prioritizes lightweight training and inference over maximum reasoning capacity.
• —Training uses synthetic Text-to-SQL examples, which may not represent every real-world database schema or production SQL workload.
• —Exact-match evaluation does not account for all semantically equivalent SQL formulations.

Additional Note: SQL execution accuracy against live databases was not measured in the reported benchmark.

Results:

completed fine-tuning run shows:

• —parameter-efficient adaptation, with approximately 0.4359% of model parameters trainable
• —stable convergence, with validation loss reaching approximately 0.22
• —100% syntax-valid SQL generation across the 50-example held-out benchmark
• —successful local fine-tuning of a code-focused language model using an NVIDIA GeForce RTX 3050

Imports (PEFT library):

from peft import PeftModel
from transformers import AutoModelForCausalLM

base_model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-Coder-0.5B-Instruct")
model = PeftModel.from_pretrained(base_model, "AaronTekle/SQLQwen")

References:

• —Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685, 2021.](https://arxiv.org/abs/2106.09685)

• —Hugging Face. PEFT LoRA Documentation. Parameter-Efficient Fine-Tuning documentation. (https://huggingface.co/docs/transformers/en/peft) (https://huggingface.co/docs/peft/en/package_reference/lora)

• —Qwen Team. Qwen2.5-Coder-0.5B-Instruct Model Card

• —Hui, B. et al. Qwen2.5-Coder Technical Report. arXiv:2409.12186, 2024

• —Gretel.ai. synthetictextto_sql Dataset Card. Hugging Face Datasets