kuzaai/kuza-qwen-3.5-4b-quant-study

Kuza Qwen 3.5-4B Quant Study

Quantization vs finetuning diagnosis for the Kuza East Africa agricultural assistant. Source weights and past quants from `kuzaai/kuza-qwen-3.5-4b`.

Diagnosis

• —Verdict: finetuning_or_template_issue
• —BF16 hidden_mean: 0.479
• —Screen winner: iq3_xs_imatrix_ssm
• —BF16 hidden_mean=0.479 is below threshold 0.75; quantization is unlikely the root cause.

Layout

• —artifacts/ — downloaded reference GGUF, imatrix, tokenizer, past quants
• —quants/ — newly quantized GGUF candidates from this study
• —screen/ — KLD logs, hidden-set reports, bench logs, results.json
• —results/ — baseline eval, report/analysis.md, per-model JSON
• —upload_manifest.json — path, size, sha256 for every uploaded file

Past-run quants (re-evaluated)

• —q4_k_m_imatrix — q4_k_m (from kuzaai/kuza-qwen-3.5-4b)
• —q4_k_xl_ssm — q4_k_m (from kuzaai/kuza-qwen-3.5-4b)
• —q4_k_s_ssm — q4_k_s (from kuzaai/kuza-qwen-3.5-4b)

New quant candidates

• —q4_k_m_plain — q4_k_m — Community-style imatrix Q4KM without tensor overrides.
• —q5_k_m_imatrix_ssm — q5_k_m — Quality step-up per Hob-forge Qwen3.5-4B benchmarks.
• —q3_k_m_imatrix_ssm — q3_k_m — User-requested 3-bit K-quant with SSM/attention protection.
• —iq3_xs_imatrix_ssm — iq3_xs — User-requested 3-bit I-quant; may trade speed for size.
• —iq4_xs_imatrix_ssm — iq4_xs — Smallest 4-bit I-quant with hybrid SSM protection.
• —q6_k_imatrix — q6_k — Near-lossless quant reference.

Download

huggingface-cli download kuzaai/kuza-qwen-3.5-4b-quant-study --local-dir ./qwen-quant-study

Primary metric: hidden-set rubric score on 36 EN/SW agriculture prompts (data/hidden_prompts.jsonl in the source repo). Ranking: hidden_mean desc, then GGUF size asc, then GPU generation TPS desc.