Modelpublic

SC117/gemma-4-31B-it-heretic-QAT-For-Edge-16G-GGUF

sourceHugging Faceapache-2.0updated 4mo agoView on Hugging Face

3likes841downloads

Model Card

<div style="font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif; display: flex; flex-direction: column; gap: 20px; margin-bottom: 30px;"> <div style="border: 1px solid #cbd5e1; border-radius: 16px; overflow: hidden; background: #ffffff; box-shadow: 0 2px 8px rgba(0,0,0,0.04);"> <div style="background: linear-gradient(135deg, #f59e0b 0%, #d97706 100%); padding: 28px 24px; color: white; text-align: center; position: relative; overflow: hidden;"> <div style="position: absolute; top: -30px; right: -30px; width: 120px; height: 120px; background: rgba(255,255,255,0.1); border-radius: 50%;"></div> <div style="position: absolute; bottom: -20px; left: 40px; width: 80px; height: 80px; background: rgba(255,255,255,0.08); border-radius: 50%;"></div> <h1 style="margin: 0; font-size: 24px; font-weight: 700; position: relative; z-index: 1;">⚡ Gemma 4 31B Heretic QAT — Extreme Quantization for 16GB VRAM</h1> <p style="margin: 8px 0 0 0; font-size: 15px; opacity: 0.95; position: relative; z-index: 1;">Heretic QAT · Q2K + Q80 Embedding · 11.4 GB · Fits 16GB VRAM</p> <p style="margin: 8px 0 0 0; font-size: 14px; position: relative; z-index: 1;"> <a href="https://huggingface.co/SC117/gemma-4-31B-it-heretic-QAT-For-Edge-16G-GGUF/blob/main/READMEzh.md" style="color: #ffffff; text-decoration: none;">📖 中文文档</a></p> </div> <div style="padding: 16px 24px; background: #f8fafc; border-bottom: 1px solid #e2e8f0; display: flex; flex-wrap: wrap; gap: 8px;"> <span style="background: #fef3c7; color: #92400e; padding: 4px 12px; border-radius: 20px; font-size: 12px; font-weight: 600;">Q2KL</span> <span style="background: #dbeafe; color: #1e40af; padding: 4px 12px; border-radius: 20px; font-size: 12px; font-weight: 600;">31B Dense</span> <span style="background: #fce7f3; color: #9d174d; padding: 4px 12px; border-radius: 20px; font-size: 12px; font-weight: 600;">Heretic Uncensored</span> <span style="background: #d1fae5; color: #065f46; padding: 4px 12px; border-radius: 20px; font-size: 12px; font-weight: 600;">11.4 GB</span> <span style="background: #ede9fe; color: #5b21b6; padding: 4px 12px; border-radius: 20px; font-size: 12px; font-weight: 600;">QAT Weights</span> <span style="background: #fee2e2; color: #991b1b; padding: 4px 12px; border-radius: 20px; font-size: 12px; font-weight: 600;">⚠️ 16GB VRAM</span> </div> <div style="padding: 20px 24px;"> <p style="margin: 0; font-size: 14px; color: #334155; line-height: 1.7;"> Extreme quantization of <a href="https://huggingface.co/coder3101/gemma-4-31B-it-qat-q40-unquantized-heretic" style="color: #c2410c; text-decoration: none;">coder3101/heretic-QAT</a> using Unsloth's Q2KL recipe (Q2K body + Q80 embedding/output). Compresses the 31B dense model to <b>11.4 GB</b>, enabling it to run on <b>16GB VRAM</b> consumer GPUs. </p> </div> </div>

<div style="border: 1px solid #ef4444; border-radius: 12px; overflow: hidden; background: #fef2f2; box-shadow: 0 2px 4px rgba(239,68,68,0.1);"> <div style="background: linear-gradient(135deg, #ef4444 0%, #dc2626 100%); padding: 10px 16px; color: white; font-weight: 700; font-size: 13px; display: flex; align-items: center; gap: 8px;"><span>⚠️</span> Extreme Quantization Warning — Please Read</div> <div style="padding: 16px;"> <p style="margin: 0 0 10px 0; font-size: 13px; color: #991b1b; line-height: 1.7;"> This quantization uses <b>Q2_K</b> (~2.9 BPW body) to compress the 31B model to 11.4 GB. This is an extreme solution for running 31B on 16GB VRAM, with known limitations: </p> <ul style="margin: 0; padding-left: 20px; font-size: 13px; color: #991b1b; line-height: 2;"> <li><b>Repetition loops</b> — Model may get stuck outputting the same token or phrase repeatedly</li> <li><b>Inaccurate arithmetic</b> — Multi-digit math calculations are unreliable</li> <li><b>Degraded complex reasoning</b> — Multi-step logic, math proofs significantly worse than higher precision</li> <li><b>Possible hallucinations</b> — Factual QA accuracy is reduced</li> </ul> <p style="margin: 10px 0 0 0; font-size: 13px; color: #991b1b; line-height: 1.7;"> <b>Suitable for (low-stakes tasks):</b> </p> <ul style="margin: 0; padding-left: 20px; font-size: 13px; color: #991b1b; line-height: 2;"> <li>Creative writing, story generation, brainstorming</li> <li>Casual conversation, chat, roleplay</li> <li>Code assistance (simple functions, code explanation, comment generation)</li> <li>Text summarization, translation, format conversion</li> <li>Non-precise general knowledge Q&A, concept explanation</li> </ul> <p style="margin: 10px 0 0 0; font-size: 12px; color: #b91c1c; line-height: 1.6;"> <b>Not recommended for:</b> Math, precise reasoning, fact-checking, production use, or any task requiring strict output accuracy. </p> </div> </div>

<div style="border: 1px solid #cbd5e1; border-radius: 12px; overflow: hidden; background: #ffffff; box-shadow: 0 2px 4px rgba(0,0,0,0.02);"> <div style="background: linear-gradient(135deg, #7c3aed 0%, #a855f7 100%); padding: 12px 16px; color: white; font-weight: 700; font-size: 14px; display: flex; align-items: center; gap: 8px;"><span>✂️</span> Heretic ARA Abliteration Parameters</div> <div style="padding: 16px;"> <p style="margin: 0 0 8px 0; font-size: 13px; color: #64748b; font-style: italic; border-left: 3px solid #ddd6fe; padding-left: 12px;">Base: <a href="https://huggingface.co/coder3101/gemma-4-31B-it-qat-q40-unquantized-heretic" style="color: #c2410c; text-decoration: none;">coder3101/gemma-4-31B-it-qat-q40-unquantized-heretic</a> · Heretic v1.2.0 · ARA + Row-Norm</p> <table style="width: 100%; border-collapse: collapse; font-family: inherit; font-size: 13px;"> <thead> <tr style="background: rgba(124,58,237,0.05);"> <th style="padding: 8px 10px; border-bottom: 2px solid #7c3aed; text-align: left; color: #5b21b6; font-weight: bold;">Parameter</th> <th style="padding: 8px 10px; border-bottom: 2px solid #7c3aed; text-align: left; color: #5b21b6; font-weight: bold;">Value</th> </tr> </thead> <tbody> <tr><td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); color: #334155;">startlayerindex</td><td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); color: #334155;">15</td></tr> <tr><td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); color: #334155;">endlayerindex</td><td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); color: #334155;">39</td></tr> <tr><td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); color: #334155;">preservegoodbehaviorweight</td><td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); color: #334155;">0.6649</td></tr> <tr><td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); color: #334155;">steerbadbehaviorweight</td><td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); color: #334155;">0.0001</td></tr> <tr><td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); color: #334155;">overcorrectrelativeweight</td><td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); color: #334155;">1.0818</td></tr> <tr><td style="padding: 8px 10px; color: #334155;">neighbor_count</td><td style="padding: 8px 10px; color: #334155;">15</td></tr> </tbody> </table> <table style="width: 100%; border-collapse: collapse; font-family: inherit; font-size: 13px; margin-top: 12px;"> <thead> <tr style="background: rgba(124,58,237,0.05);"> <th style="padding: 8px 10px; border-bottom: 2px solid #7c3aed; text-align: left; color: #5b21b6; font-weight: bold;">Metric</th> <th style="padding: 8px 10px; border-bottom: 2px solid #7c3aed; text-align: center; color: #5b21b6; font-weight: bold;">Heretic</th> <th style="padding: 8px 10px; border-bottom: 2px solid #7c3aed; text-align: center; color: #5b21b6; font-weight: bold;">Original QAT</th> </tr> </thead> <tbody> <tr><td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); color: #334155;">KL Divergence</td><td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); color: #334155; text-align: center;">0.0564</td><td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); color: #334155; text-align: center;">0 (by definition)</td></tr> <tr><td style="padding: 8px 10px; color: #334155;">Refusals</td><td style="padding: 8px 10px; color: #334155; text-align: center;">11/100</td><td style="padding: 8px 10px; color: #334155; text-align: center;">99/100</td></tr> </tbody> </table> </div> </div>

<div style="border: 1px solid #cbd5e1; border-radius: 12px; overflow: hidden; background: #ffffff; box-shadow: 0 2px 4px rgba(0,0,0,0.02);"> <div style="background: linear-gradient(135deg, #059669 0%, #10b981 100%); padding: 12px 16px; color: white; font-weight: 700; font-size: 14px; display: flex; align-items: center; gap: 8px;"><span>🏗️</span> Architecture</div> <div style="padding: 16px;"> <table style="width: 100%; border-collapse: collapse; font-family: inherit; font-size: 13px;"> <tbody> <tr><td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); color: #047857; font-weight: 600; width: 40%;">Base Model</td><td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); color: #334155;"><a href="https://huggingface.co/google/gemma-4-31B-it" style="color: #c2410c; text-decoration: none;">google/gemma-4-31B-it</a></td></tr> <tr><td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); color: #047857; font-weight: 600;">Parameters</td><td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); color: #334155;">30.7B (dense, all parameters active)</td></tr> <tr><td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); color: #047857; font-weight: 600;">Architecture</td><td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); color: #334155;">Hybrid: Gated DeltaNet linear attention + Softmax global attention (3:1 alternating)</td></tr> <tr><td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); color: #047857; font-weight: 600;">Layers</td><td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); color: #334155;">60</td></tr> <tr><td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); color: #047857; font-weight: 600;">Context Length</td><td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); color: #334155;">256K tokens (supported by this quantization)</td></tr> <tr><td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); color: #047857; font-weight: 600;">Vocabulary</td><td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); color: #334155;">262K, 140+ languages</td></tr> <tr><td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); color: #047857; font-weight: 600;">Modalities</td><td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); color: #334155;">Text + Image (native multimodal)</td></tr> <tr><td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); color: #047857; font-weight: 600;">QAT Training</td><td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); color: #334155;">Google official QAT (quantization-aware), weights inherently robust to quantization</td></tr> <tr><td style="padding: 8px 10px; color: #047857; font-weight: 600;">Quantization</td><td style="padding: 8px 10px; color: #334155;">Q2KL (Q2K body + Q80 embedding/output), b9553 llama-quantize</td></tr> </tbody> </table> </div> </div>