farehaheha/llama3.2-3B-text-to-sql-Q4_K_M-GGUF

Llama 3.2 3B Text-to-SQL — Q4KM GGUF

A Q4_K_M quantized GGUF version of my fine-tuned Llama 3.2 3B Text-to-SQL model, designed for efficient local inference with llama.cpp and other GGUF-compatible runtimes.

The model converts natural-language questions and database schemas into SQL queries.

Model Details

Original Model

This GGUF model is a quantized version of:

farehaheha/llama3.2-3B-text-to-sql

The original model was fine-tuned using QLoRA on approximately 5,850 Text-to-SQL examples from the Gretel AI Synthetic Text-to-SQL dataset.

Important: This Is a Completion Model

This model was fine-tuned as a completion model, not as an instruction or chat model.

For best results, use the same prompt format used during fine-tuning:

### Database Schema:
{your_database_schema}

### Request:
{your_natural_language_request}

### SQL Query:

The model should continue directly from ### SQL Query:.

Avoid wrapping the prompt in chat templates or system/user/assistant messages.

Run with llama.cpp

Start llama-server:

llama-server \
  -m llama3.2-3b-text-to-sql-q4_k_m.gguf \
  --port 8080 \
  -ngl 99 \
  -c 4096

On Windows:

.\llama-server.exe -m ".\llama3.2-3b-text-to-sql-q4_k_m.gguf" --port 8080 -ngl 99 -c 4096

Inference with Python

Once llama-server is running on port 8080, send the prompt directly to the completion endpoint:

import requests

schema = """
CREATE TABLE salesperson (
    salesperson_id INT,
    name TEXT,
    region TEXT
);

CREATE TABLE timber_sales (
    sales_id INT,
    salesperson_id INT,
    volume REAL,
    sale_date DATE
);
"""

request = """
What is the total volume of timber sold by each salesperson?
"""

prompt = f"""### Database Schema:
{schema}

### Request:
{request}

### SQL Query:
"""

response = requests.post(
    "http://localhost:8080/completion",
    json={
        "prompt": prompt,
        "n_predict": 256,
        "temperature": 0.0,
        "stop": ["###"]
    },
    timeout=60
)

response.raise_for_status()

sql = response.json()["content"].strip()
print(sql)

Example output:

SELECT salesperson_id, SUM(volume) AS total_volume
FROM timber_sales
GROUP BY salesperson_id;

Inference Settings

Recommended settings for deterministic Text-to-SQL generation:

For Text-to-SQL, greedy decoding is generally preferred over creative sampling.

Python Inference Script

A simple Python example using llama-server is available in the project repository:

llama_cpp_tryout.py

The script:

• —starts from the model's completion-style prompt format
• —sends requests to a local llama-server
• —uses the /completion endpoint
• —generates SQL directly from the provided schema and natural-language request

Why Q4KM?

Q4KM provides a practical balance between:

• —reduced model size
• —lower memory usage
• —faster local inference
• —preservation of model quality

At approximately 2 GB, this version is intended to make the model easier to run on consumer hardware.

Limitations

• —The model may generate syntactically valid but logically incorrect SQL.
• —Performance depends on schema complexity and how clearly the request is written.
• —Generated SQL should be validated before execution in production environments.
• —The model was fine-tuned using examples from the test split of the source dataset, so that split should not be used for unbiased evaluation.

Acknowledgements

Built using:

• —Meta Llama 3.2 3B
• —Unsloth
• —Gretel AI Synthetic Text-to-SQL dataset
• —llama.cpp

License

Apache 2.0