Modelpublic

tuxevil/home-assistant-specialist-v0.4-ddp

sourceHugging Faceupdated 2mo agoView on Hugging Face

0likes8downloads

Model Card

Home Assistant Specialist v0.4 — Qwen3-8B Adapter

PEFT/LoRA adapter trained to map Home Assistant requests to the strict ha-action-v3 JSON contract.

Training

—Base: unsloth/Qwen3-8B-unsloth-bnb-4bit
—Method: QLoRA with DDP
—Hardware: 2× Tesla T4
—Dataset: tuxevil/Home-Assistant-Requests-V4
—Dataset provenance: validated rows from acon96/Home-Assistant-Requests, acon96/Home-Assistant-Requests-V2, and project-curated contract cases.
—Context: 4,096 tokens
—Evaluation: 417 held-out cases

Evaluation layers

All Hugging Face results below use the 417-case V4 test split in the Transformers/Kaggle pipeline. Ollama results use the same cases through the GGUF runtime. They are reported separately because runtime/template/conversion effects can change results.

Hugging Face / Transformers reference

Model state	Precision	JSON valid	Exact match	Status exact	Service exact	Safety OK	Mean latency
Base model	4-bit	100.00%	0.00%	83.69%	89.69%	84.17%	4.789 s
Trained adapter	4-bit QLoRA	100.00%	99.28%	99.52%	100.00%	100.00%	3.738 s
Merged model	FP16	100.00%*	99.20%*	—	—	99.50%*	N/A

* FP16 values are the recorded merged/export benchmark reference; only exact-match and safety were retained in the published comparison.

Same-quantization comparison: Qwen3-8B vs Qwen3-4B

Quantization	Qwen3-8B exact	Qwen3-4B exact	Delta	Qwen3-8B safety	Qwen3-4B safety	Delta
Q4KM	91.85%	87.53%	+4.32 pp	93.05%	88.49%	+4.56 pp
Q5KM	97.36%	95.68%	+1.68 pp	98.32%	96.64%	+1.68 pp
Q6_K	97.36%	92.81%	+4.55 pp	98.32%	94.48%	+3.84 pp

Ollama quantization benchmark

Quantization	Exact match	Safety OK	Mean latency	Deployment note
Q4KM	91.85%	93.05%	1.536 s	Full GPU
Q5KM	97.36%	98.32%	1.580 s	Full GPU; preferred
Q6_K	97.36%	98.32%	2.348 s	CPU offload; slower

Q5KM was preferred for the 8 GB Quadro RTX 4000 because Q6 did not improve quality and incurred CPU offload.

Safety

This adapter is a research artifact, not an autonomous Home Assistant controller. Validate every response against live service and entity allowlists before execution.