rishhh/schemasage-sql-qwen3-4b-clean-balanced-200

SchemaSage-SQL Model Card

This is the release model card for the SchemaSage-SQL project. A 10-step QLoRA smoke adapter, a 200-step safety-balanced adapter, and a larger 8k/600 follow-up adapter have been trained and uploaded to validate the cloud training and Hugging Face release path. The 200-step safety-balanced adapter is the public release baseline.

Model Details

• —Project name: SchemaSage-SQL
• —Model status: release baseline published; larger follow-up remains experimental
• —Planned base model: configurable through configs/model.yaml
• —Current default base model: Qwen/Qwen3-4B-Instruct-2507
• —Planned method: supervised fine-tuning with LoRA or QLoRA
• —Release format: adapter-first, optional merged model later
• —Smoke adapter: rishhh/schemasage-sql-qwen3-4b-smoke

Intended Use

SchemaSage-SQL is intended for:

• —Research and prototyping around Text-to-SQL.
• —Portfolio demonstration of LLM data, training, evaluation, and deployment engineering.
• —Generating draft read-only analytical SQL from a schema and natural-language question.

Generated SQL should be reviewed before execution.

Out-of-Scope Use

SchemaSage-SQL is not intended for:

• —Autonomous execution against production databases.
• —Destructive database operations.
• —Legal, financial, medical, or compliance-critical decision making.
• —Credential extraction, filesystem access, or network exfiltration.
• —SQL generation without a provided schema.

Datasets

Current data pipeline uses public Hugging Face datasets:

• —gretelai/synthetic_text_to_sql
• —b-mc2/sql-create-context

Cleaned and republished training data:

Current processed split summary:

Processing removed:

• —Empty or incomplete examples.
• —Exact duplicates.
• —Destructive SQL examples by default.
• —Sample INSERT rows from schema context where source datasets included data values.

The validation split is a deterministic train holdout because the selected sources do not provide a native validation split.

Training Procedure

Training is implemented in src/training/train_sft.py and has been run for smoke, release-baseline, and experimental adapters. A tiny local trainer smoke test was completed with sshleifer/tiny-gpt2; that artifact is not a release model and should not be used for model-quality claims.

A cloud QLoRA smoke run completed on Hugging Face Jobs:

Smoke adapter commit:

https://huggingface.co/rishhh/schemasage-sql-qwen3-4b-smoke/commit/6a8241d2f6cc3f0a917ba527f2f32b0cc4bf9933

This smoke adapter validates the training and upload path. It is not the release baseline.

Safety-Clean Balanced Adapter

The best current safety baseline is:

rishhh/schemasage-sql-qwen3-4b-clean-balanced-200

It was trained for 200 optimizer steps on rishhh/schemasage-sql-clean-text2sql with a deterministic 20% refusal mix in the short training sample. Evaluation on 64 cleaned held-out examples produced:

This adapter is the public release baseline because it kept perfect blocked-refusal accuracy on the re-evaluated held-out set.

The larger follow-up run completed as:

rishhh/schemasage-sql-qwen3-4b-clean-balanced-8k-600-v2

Completed settings: 8,192 cleaned train rows, 600 optimizer steps, 20% refusal mix, and 256 cleaned held-out evaluation examples.

The larger run is the stronger semantic candidate: it improved exact match, execution accuracy, and exact-or-execution-equivalent rate, but it missed one blocked refusal and kept slightly lower parse/schema scores. It remains the best comparison candidate, not the shipped baseline.

Planned command:

python -m src.training.train_sft --config configs/train_qlora.yaml

Smoke/dry-run validation command:

python -m src.training.train_sft --config configs/train_qlora.yaml --smoke-test --dry-run

Apple Silicon local development should use:

python -m src.training.train_sft --config configs/train_local_smoke.yaml --smoke-test --dry-run

Trainer-path validation command that has been run locally:

python -m src.training.train_sft --config configs/train_local_smoke.yaml --smoke-test

Evaluation

The full local reference-answer evaluation is a sanity check for the evaluator and processed data, not trained-model performance.

These metrics validate the evaluator and processed reference data. They should not be presented as model quality.

The uploaded smoke adapter was also evaluated on 64 held-out examples from gretelai/synthetic_text_to_sql test. These metrics describe the 10-step smoke adapter only:

Evaluation artifacts:

• —evaluation/smoke_64/eval_results.json
• —evaluation/smoke_64/eval_report.md
• —evaluation/smoke_64/predictions.jsonl
• —evaluation/smoke_64/metrics_overview.svg
• —evaluation/smoke_64/risk_rates.svg

Run comparison report:

• —reports/run_comparison.md
• —reports/run_comparison/comparison_quality.svg
• —reports/run_comparison/comparison_risk_latency.svg

Release Baseline

The public release baseline is:

rishhh/schemasage-sql-qwen3-4b-clean-balanced-200

Its re-evaluated held-out metrics are summarized in:

• —reports/reparsed_clean_balanced_200.md
• —reports/reparsed_run_comparison.md
• —reports/run_comparison/comparison_quality.svg
• —reports/run_comparison/comparison_risk_latency.svg

The larger rishhh/schemasage-sql-qwen3-4b-clean-balanced-8k-600-v2 run improved semantic metrics but missed one blocked refusal and kept slightly lower parse/schema scores, so it remains an experimental comparison candidate rather than the release baseline.

Safety Policy

The project defaults to read-only analytical SQL.

The safety layer blocks or refuses:

• —DROP
• —DELETE
• —TRUNCATE
• —ALTER
• —UPDATE
• —INSERT
• —MERGE
• —REPLACE
• —CREATE DATABASE
• —CREATE USER
• —GRANT
• —REVOKE
• —EXEC and EXECUTE
• —CALL
• —COPY, LOAD, and UNLOAD
• —Multiple SQL statements
• —Prompt-injection-like requests to ignore safety or schema rules

Safety checks are implemented in src/inference/safety.py using lexical checks plus sqlglot parsing.

Limitations

• —The public release baseline is the 200-step safety-balanced adapter; the larger 8k/600 follow-up remains experimental.
• —The uploaded Qwen3 4B smoke adapter exists only to validate the cloud trainer and Hub upload path.
• —A tiny local smoke adapter exists only to validate the local trainer path.
• —Re-evaluated held-out predictions are published for the release baseline, but the larger follow-up still serves as the main comparison point for future improvement work.
• —The smoke adapter is undertrained; exact match is low despite good parse validity and schema adherence.
• —The model can continue generating prompt-like text after the canonical response, so inference uses canonical-response parsing and should add stricter stop criteria.
• —QLoRA training is expected to require CUDA-capable GPU hardware.
• —Apple Silicon is suitable for development, tests, data prep, dry-runs, and the Gradio app, but not the default 4-bit QLoRA path.
• —SQL equivalence is difficult; exact match is not enough to judge model quality.
• —SQLite execution evaluation skips dialect-incompatible examples.
• —Schema adherence checks can still miss complex aliasing or dialect-specific constructs.

Example Usage

python -m src.inference.generate_sql \
  --schema-file data/samples/schema.sql \
  --question "Which product categories had the highest revenue in Q4 2025?" \
  --dry-run

Model-backed inference requires a configured model or adapter path.

Hardware Notes

The repository was developed and validated locally on a MacBook Pro M5 Pro environment using Python 3.11. Full QLoRA training should be run on CUDA GPU hardware or a managed GPU environment.

How to Improve

• —Improve execution-equivalence on hard joins and aggregations.
• —Add Spider-style held-out execution evaluation.
• —Expand refusal and unanswerable-question training examples.
• —Add batched inference and stop strings to reduce latency and trailing prompt continuation.
• —Keep the larger 8k/600 run as an experimental benchmark rather than the shipped baseline until it clearly wins on both quality and safety.
• —Publish final metrics only after evaluating generated predictions, not references.

License

Project code is MIT licensed. Dataset and base-model licenses must be reviewed before publishing trained artifacts.