evlogia-kyriou/qwen-text-to-sql-lora

Model Card for Model ID

language:

• —en license: mit tags:
• —text-to-sql
• —sql-generation
• —qwen
• —lora
• —peft
• —bird-benchmark
• —fine-tuning base_model: Qwen/Qwen2.5-Coder-3B-Instruct datasets:
• —bird-bench/bird pipeline_tag: text2text-generation ---

Qwen2.5-Coder-3B Text-to-SQL LoRA Adapter

LoRA adapter for fine-tuning Qwen2.5-Coder-3B-Instruct on Text-to-SQL generation using the BIRD benchmark dataset.

Model Description

This is a LoRA (Low-Rank Adaptation) adapter trained to enhance Qwen2.5-Coder-3B-Instruct for natural language to SQL query generation. The adapter is lightweight (~20 MB) and can be merged with the base model for inference.

• —Base Model: Qwen/Qwen2.5-Coder-3B-Instruct
• —Training Dataset: BIRD Benchmark (7,542 training examples)
• —Training Method: LoRA with 4-bit quantization (QLoRA)
• —Parameters: 3B base + ~20M trainable (LoRA)

Performance

Quick Start

Installation

pip install transformers peft torch bitsandbytes

Loading the Model

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

# Load base model with 4-bit quantization
base_model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-Coder-3B-Instruct",
    device_map="auto",
    trust_remote_code=True,
    torch_dtype=torch.float16,
    load_in_4bit=True
)

# Load LoRA adapter
model = PeftModel.from_pretrained(
    base_model,
    "YOUR_USERNAME/qwen-text-to-sql-lora"
)

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained(
    "Qwen/Qwen2.5-Coder-3B-Instruct",
    trust_remote_code=True
)

Inference Example

def generate_sql(question: str, schema: str) -> str:
    """Generate SQL from natural language question."""
    prompt = f"""You are an expert SQL generator. Given a database schema and a question, generate a SQL query.

Database Schema:
{schema}

Question: {question}

SQL Query:"""

    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    
    outputs = model.generate(
        **inputs,
        max_new_tokens=1024,
        temperature=0.1,
        top_p=0.9,
        do_sample=True,
        pad_token_id=tokenizer.eos_token_id
    )
    
    sql = tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
    return sql.strip()

# Example usage
schema = """
CREATE TABLE customers (
    customer_id INT PRIMARY KEY,
    name TEXT,
    email TEXT,
    signup_date DATE
);

CREATE TABLE orders (
    order_id INT PRIMARY KEY,
    customer_id INT,
    order_date DATE,
    total_amount DECIMAL,
    FOREIGN KEY (customer_id) REFERENCES customers(customer_id)
);
"""

question = "What is the total revenue by month in 2024?"

sql = generate_sql(question, schema)
print(f"Generated SQL:\n{sql}")

Example Output:

SELECT 
    strftime('%Y-%m', order_date) AS month,
    SUM(total_amount) AS total_revenue
FROM orders
WHERE strftime('%Y', order_date) = '2024'
GROUP BY month
ORDER BY month;

Training Details

Dataset

• —Source: BIRD (BIg Bench for LaRge-scale Database Grounded Text-to-SQLs)
• —Train Split: 7,542 examples
• —Validation Split: 1,886 examples
• —Domains: Cross-domain (e-commerce, healthcare, finance, etc.)

Training Configuration

# LoRA Configuration
lora_config = LoraConfig(
    r=16,                    # Rank
    lora_alpha=32,           # Scaling factor
    target_modules=[
        "q_proj", "k_proj", "v_proj", "o_proj"
    ],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)

# Training Arguments
training_args = TrainingArguments(
    num_train_epochs=3,
    per_device_train_batch_size=8,
    gradient_accumulation_steps=4,
    learning_rate=2e-4,
    fp16=True,
    optim="paged_adamw_8bit",
    lr_scheduler_type="cosine",
    warmup_ratio=0.1,
)

Hardware Requirements

Training:

• —GPU: NVIDIA RTX 3060 (12 GB VRAM) minimum
• —RAM: 32 GB recommended
• —Storage: 50 GB

Inference:

• —GPU: 6 GB VRAM (with 4-bit quantization)
• —RAM: 16 GB
• —CPU inference possible but slow

Limitations

• —Trained primarily on English text-to-SQL pairs
• —Performance may vary on databases with complex schemas (100+ tables)
• —Requires schema context for accurate generation
• —May struggle with ambiguous natural language queries
• —Not tested on all SQL dialects (focused on standard SQL)

Intended Use

✅ Recommended Use Cases

• —Text-to-SQL prototyping and development
• —Educational purposes for learning SQL generation
• —Research on instruction-tuned language models
• —Building SQL query assistants with human oversight

⚠️ Not Recommended

• —Production systems without human verification
• —Sensitive data queries without validation
• —Financial or medical applications without review
• —Direct execution of generated queries on production databases

Safety & Best Practices

1. 1.Always validate generated SQL before execution
2. 2.Use read-only database connections for generated queries
3. 3.Implement query timeouts to prevent long-running queries
4. 4.Sanitize inputs to prevent injection attempts
5. 5.Test on development databases first

Bias & Fairness

The model inherits biases from:

• —Base Qwen2.5-Coder-3B model
• —BIRD benchmark dataset
• —SQL syntax conventions

Users should be aware of potential biases in:

• —Database naming conventions
• —Query patterns
• —Domain-specific terminology

Citation

If you use this model in your research, please cite:

@software{qwen_text_to_sql_lora_2026,
  author = Jireh Fessenden,
  title = {Qwen2.5-Coder-3B Text-to-SQL LoRA Adapter},
  year = {2026},
  url = {https://huggingface.co/YOUR_USERNAME/qwen-text-to-sql-lora},
  note = {LoRA adapter for Text-to-SQL generation}
}

Also cite the base model and dataset:

@article{qwen2.5coder,
  title={Qwen2.5-Coder Technical Report},
  author={Hui, Binyuan and others},
  year={2024}
}

@article{li2024bird,
  title={Can LLM Already Serve as a Database Interface?},
  author={Li, Jinyang and others},
  journal={arXiv preprint arXiv:2305.03111},
  year={2024}
}

License

MIT License - See LICENSE file

The base model (Qwen2.5-Coder-3B-Instruct) is under Apache 2.0 license.

Acknowledgments

• —Alibaba Qwen Team for the excellent base model
• —BIRD Benchmark Team for the high-quality dataset
• —HuggingFace for PEFT and Transformers libraries
• —Microsoft Research for LoRA methodology

Links

• —GitHub Repository: https://github.com/evlogia-kyriou/text-to-sql-fine-tuning

Contact

For questions or issues:

• —GitHub Issues: https://github.com/evlogia-kyriou/text-to-sql-fine-tuning
• —Email: fessenden.jf@gmail.com
• —LinkedIn: www.linkedin.com/in/fessenden

Model Version: 1.0.0 Last Updated: January 2026 Status: ✅ Production Ready (with safety guidelines)