KikoCis/DeepHat-V1-7B-GGUF

<p align="center"><img src="banner.png" alt="DeepHat-V1-7B GGUF" width="100%"></p>

┌────────────────────────────────────────────────────────────────────┐
│  DeepHat-V1-7B · GGUF                                               │
│  ───────────────────────────────────────────────────────────────   │
│  base   DeepHat/DeepHat-V1-7B   arch  qwen2 (Qwen2.5-Coder-7B ft)   │
│  domain cybersecurity · devops · code                              │
│  ladder Q3_K_M → Q8_0 + IQ4_XS   imatrix  ✓ (code+general)          │
│  fidelity  Q8 KLD 0.0019 · top-1 99.8% vs BF16                      │
└────────────────────────────────────────────────────────────────────┘

DeepHat-V1-7B — GGUF quant ladder

A full, imatrix-calibrated GGUF ladder of **DeepHat/DeepHat-V1-7B** — a Qwen2.5-Coder-7B fine-tune focused on cybersecurity, devops and code. This is a faithful re-quantization (all credit for the model goes to the DeepHat team); what's added here is the imatrix ladder, an objective KLD fidelity table vs the BF16 reference, and an honest evaluation.

First-mover note: at pack time no GGUF of this model existed. Quants Q3KM → Q80 + IQ4XS, each with an importance matrix so the low-bit tiers stay sharp.

✅ Recommended files

📦 Files

📊 Metrics — fidelity vs BF16 reference

Every tier measured against the unquantized BF16 GGUF (KL-divergence + PPL ratio + Top-1 agreement, general-English eval, c=2048).

The quantization is clean top-to-bottom — Q8 is effectively lossless and even Q3KM keeps 98% argmax agreement with BF16.

🧮 Will it fit?

7B at Q4–Q6 runs fast on a laptop GPU (Apple Silicon / consumer NVIDIA) or CPU.

🚀 How to run it

llama.cpp

llama-cli -m deephat-v1-7b-Q4_K_M.gguf -ngl 99 -c 8192 \
  -p "<|im_start|>user\nExplain how a SQL injection works and how to prevent it.<|im_end|>\n<|im_start|>assistant\n"

Ollama (a ready Modelfile with configurable context ships in this repo):

ollama create deephat -f deephat-8192ctx.Modelfile
ollama run deephat "Write a bash script that scans a subnet for open port 22."

Sampling: ChatML template, temperature 0.1–0.7, top_p 0.9, top_k 20, stop <|im_end|>.

⚠️ Good to know

• —Chat template = upstream original (tool-calling preserved). These GGUFs embed DeepHat's original Qwen tool-calling template (the XLAM/Qwen tools format), so native function-calling works just like the source model. That template uses the Jinja tojson filter, which older ollama / llama.cpp builds cannot parse (they error "Unknown (built-in) filter 'tojson'"). If you hit that: update your runtime (recent ollama/llama.cpp handle it), or override the template with the plain-ChatML scripts/chatml.jinja shipped here (--chat-template in llama.cpp) — that variant loads everywhere but drops native tool-calling.
• —Domain model. DeepHat is tuned for security / devops / code, not general chit-chat or general software engineering. See the eval below.

🧪 Evaluation methodology

• —Fidelity gate (passed): the KLD/PPL/Top-1 table above — every tier is faithful to BF16 (Q8 lossless).
• —Agentic probe (reported honestly): swe-mix — 30 SWE-bench Verified instances (6 continuity anchors + 24 discriminating, 10 repos; spec in swe_mix.json), run through Claude Code + agent-bridge.js → Ollama serving Q6_K, tool-call format openhands, temperature 0.1, in Docker. Result: `resolved = 0 / 30`. The model did emit tool calls on 22/30 instances but ran very short episodes (2–8 messages) — it doesn't sustain the long multi-step agentic loop a SWE-bench solve needs. Honest caveat: (1) this is a 7B — small models routinely score 0–2 on SWE-bench Verified in a local agentic harness; (2) SWE-bench is general-repo software engineering (Django/astropy/sympy), which is not DeepHat's domain (cybersecurity/devops). This number measures general agentic SWE ability, not the model's security/code strengths, and it is not comparable 1:1 to the official leaderboard. Date: 2026-07-12. Small/hard probe, relative signal only.

This repo is fidelity-gated (like a non-general-SWE model release): the quant quality is proven; the SWE number is published transparently rather than hidden.

🔁 Provenance & reproducibility

• —scripts/reproduce.sh — exact convert → template-fix → imatrix → quantize commands.
• —scripts/chatml.jinja — the clean chat template that replaced the upstream tojson one.
• —imatrix corpus: ~793 KB, general text + Python/Rust/C code (386 chunks, c=512).
• —reports/artifact-sha256sums.txt — SHA-256 of every GGUF.
• —metrics/quant-summary-with-kld.{json,csv} — the table above, machine-readable.
• —swe_mix.json — the 30-instance probe spec.

🗒️ Changelog

• —2026-07-12 — initial release. Ladder Q3KM→Q80 + IQ4XS, imatrix, KLD sweep, cleaned chat template, honest SWE-mix eval.

Credit

Model: [DeepHat/DeepHat-V1-7B](https://huggingface.co/DeepHat/DeepHat-V1-7B) (Apache-2.0). This repo only provides GGUF conversions + fidelity measurements. All model capability is the DeepHat team's work.