Ahmed-Ismail/Qwen-1.5B-Text-to-SQL-LoRA
0
Qwen 1.5B - Text-to-SQL LoRA
This is a fine-tuned LoRA adapter for Qwen/Qwen2.5-Coder-1.5B designed to convert natural language questions into strict, executable SQL queries based on a provided database schema.
💻 Project Goal
This model was trained as a portfolio project to demonstrate efficient fine-tuning of Large Language Models for structured data generation using PyTorch, PEFT, and 8-bit quantization (QLoRA) on constrained hardware.
🛠️ Training Details
- Base Model: Qwen/Qwen2.5-Coder-1.5B
- Dataset: Downsampled
b-mc2/sql-create-context(25,000 highly curated schema/question/query pairs) - Technique: LoRA (Rank 16, Alpha 32) targeting all linear layers for maximum logical reasoning capability.
- Infrastructure: Fine-tuned on a single NVIDIA T4 GPU using gradient checkpointing and an 8-bit Paged AdamW optimizer.
🚀 How to Use
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
base_model_id = "Qwen/Qwen2.5-Coder-1.5B"
adapter_id = "your-username/Qwen-1.5B-Text-to-SQL-LoRA"
# Load the base model and tokenizer
tokenizer = AutoTokenizer.from_pretrained(base_model_id)
base_model = AutoModelForCausalLM.from_pretrained(base_model_id)
# Apply the LoRA adapter
model = PeftModel.from_pretrained(base_model, adapter_id)
# Format your prompt using the strict schema template
prompt = """Schema:
CREATE TABLE employees (id VARCHAR, name VARCHAR, department VARCHAR, salary INTEGER)
Question:
How many employees in the engineering department make more than 100000?
SQL:
"""
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=150, do_sample=False)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))