chabab/gemma-3-270m-text2sql-oracle-postgres

gemma-3-270m-text2sql-oracle-postgres

`google/gemma-3-270m-it` fine-tuned to turn a schema + a natural-language question into one dialect-correct SQL statement — Oracle or PostgreSQL — with no markdown fences and no commentary.

At 270M parameters it runs on CPU and quantizes to ~290 MB.

Results

Held-out test split (60 examples), greedy decoding, normalized exact match against gold SQL:

Per-example predictions are in `eval_results.json`.

Caveats worth knowing before you rely on these numbers:

• —The test split is skewed 45 Oracle / 15 PostgreSQL, so the PostgreSQL figure rests on 15 examples and has a wide error bar.
• —Exact match is strict. Several "failures" are valid SQL that differs from gold — an extra LIMIT, a different but equivalent predicate. Real semantic accuracy is higher than 78.3%.
• —The most common genuine error is dialect leakage: emitting LIKE where PostgreSQL gold uses ILIKE. If case-insensitive matching matters to you, check that specific pattern.
• —Only the 7 schemas in the training set (hr, sales, banking, inventory, tickets, university, logistics) are represented. Generalization to unseen schemas is untested.

Usage

The model expects the system prompt naming the dialect, then a Schema: block and a Question: block — the same shape as the training data.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "chabab/gemma-3-270m-text2sql-oracle-postgres"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    attn_implementation="eager",  # Gemma-3 needs eager attention for correct generation
    device_map="auto",
)

messages = [
    {"role": "system", "content": "You convert natural language into PostgreSQL SQL. Use only tables and columns from the provided schema. Reply with one SQL statement and nothing else. No markdown fences. No commentary."},
    {"role": "user", "content": """Schema:
employees(
  employee_id INTEGER PK,
  first_name VARCHAR(50),
  last_name VARCHAR(50),
  hire_date DATE,
  salary NUMERIC(12,2),
  department_id INTEGER FK->departments.department_id
)

Question:
Show the five employees with the largest salary. Return only the SQL."""},
]

ids = tok.apply_chat_template(messages, add_generation_prompt=True,
                              return_tensors="pt", return_dict=True).to(model.device)
out = model.generate(**ids, max_new_tokens=256, do_sample=False)
print(tok.decode(out[0][ids["input_ids"].shape[-1]:], skip_special_tokens=True).strip())
# SELECT first_name, last_name, salary FROM employees ORDER BY salary DESC LIMIT 5;

Two things matter for output quality:

• —Use greedy decoding (do_sample=False). The task has one right answer; sampling only adds drift.
• —Use `attn_implementation="eager"`. Gemma-3 generates degenerate repeated tokens under the default SDPA path in some configurations.

GGUF / local inference

Quantized builds for Ollama, LM Studio, and llama.cpp: `chabab/gemma-3-270m-text2sql-oracle-postgres-GGUF`

Training

Full-parameter SFT (no LoRA — the model is small enough to tune end to end) with TRL SFTTrainer on one L4 GPU, about 10 minutes.

Final metrics: train loss 0.0256, eval loss 0.0683, eval token accuracy 98.5%. Eval loss fell monotonically through training with no divergence.

Limitations

Generated SQL is not validated against a live database. The model can produce syntactically valid statements that reference the wrong table or misread the intent — two of the observed test failures do exactly that. Review output before executing it, and never run generated SQL against production with write permissions.

License

Apache 2.0, inheriting the Gemma terms of use from the base model.