Modelpublic

Koshkasa/TheDrummer_Skyfall-31B-v4.2_16G-checkerboard-trellis-GGUF

sourceHugging Faceapache-2.0updated 1mo agoView on Hugging Face

0likes253downloads

Model Card

!!! quantized for use with ik_llama.cpp and its derivatives !!!

!!! incompatible with mainline llama.cpp as of commit #34af94c !!!

What's that?

Skyfall is a heavy hitter for mainline consumer GPUs. I have an abusive relationship with it - the goal of fitting the most of it into 16 GBs of VRAM, even without KVO, eluded me twice. I made two publicly avaliable ik_llama.cpp-exclusive quants of Skyfall before, each with their own issues. Lessons were learned. Most were forgotten.

It's time for Round 3.**

The goal: shove Skyfall into 16 GB VRAM with system overhead, AGAIN, using ik_llama, hopefully make it all smart enough for a mixed quant. The result: another hybrid quantization of TheDrummer/Skyfall-31B-v4.2, hopefully the final iteration

Rationale

Having a very limited budget for a 31b model (I run the system off the same GPU I use for ik_llama.cpp inference, so I am only safe within a 14.5 GB margin), I decided to play chess - literally. The recipe involves two layer precision schemae. The higher precision schema was used in 8 edge layers, and as every third middle layer as a drift prevention checkpoint. The idea is simple. Keeping edge layers in higher precision should stabilize input comprehension/output composition (keeping them as unharmed as we can afford, given the size limits), and having checkpoints in an otherwise aggressive mixed 3/4bit ffn flow should let the model compensate for SOME quantization-borne drift. That, together with ik's advancements with trellis quantization and a bulkier imatrix dataset with an expanded RP corpus should keep the model sane. In theory. Will it have a measurable mechanistic effect? Will such a ratio of layers be enough? Idk. That's an experiment and a love letter to Skyfall at the same time.

<head> <meta charset="UTF-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <title>Quant Recipe</title> <style>

—{ margin: 0; padding: 0; box-sizing: border-box; } body { background: #0d1117; color: #e6edf3; font-family: 'Segoe UI', -apple-system, sans-serif; display: flex; justify-content: center; padding: 2rem 1rem; } .container { background: #161b22; border-radius: 12px; padding: 2rem 1.5rem; border: 1px solid #30363d; max-width: 1100px; width: 100%; display: flex; flex-direction: column; align-items: center; } h1 { font-size: 1.6rem; font-weight: 600; margin-bottom: 0.25rem; color: #f0f6fc; text-align: center; } .subtitle { color: #8b949e; font-size: 0.9rem; margin-bottom: 1.5rem; text-align: center; } .track { display: flex; flex-wrap: wrap; align-items: flex-end; justify-content: center; gap: 3px 2px; padding: 1rem 0.5rem; background: #0d1117; border-radius: 8px; border: 1px solid #30363d; margin-bottom: 1.5rem; width: 100%; min-height: 100px; } .block { display: flex; flex-direction: column; align-items: center; flex-shrink: 0; } .bar { border-radius: 3px; width: 22px; transition: all 0.1s; min-height: 6px; } .bar.io { background: #2da44e; height: 52px; width: 32px; } .bar.high { background: #1f6feb; height: 40px; } .bar.low { background: #da3633; height: 20px; } .bar-label { font-size: 0.5rem; color: #8b949e; margin-top: 4px; font-family: 'JetBrains Mono', monospace; letter-spacing: 0.2px; } .ellipsis { display: flex; flex-direction: column; align-items: center; color: #8b949e; padding: 0 6px; font-size: 1.2rem; font-weight: 300; letter-spacing: 2px; user-select: none; } .ellipsis .dots { font-size: 1.4rem; line-height: 1.2; } .ellipsis .label { font-size: 0.5rem; color: #6e7681; margin-top: 2px; } .legend { display: flex; flex-wrap: wrap; justify-content: center; gap: 1.5rem 2.5rem; padding: 0.6rem 1rem; background: #0d1117; border-radius: 8px; border: 1px solid #30363d; margin-bottom: 1.5rem; width: 100%; } .legend-item { display: flex; align-items: center; gap: 0.6rem; font-size: 0.8rem; color: #c9d1d9; } .legend-swatch { width: 28px; height: 18px; border-radius: 3px; flex-shrink: 0; } .legend-swatch.green { background: #2da44e; height: 24px; width: 20px; } .legend-swatch.blue { background: #1f6feb; height: 20px; width: 20px; } .legend-swatch.red { background: #da3633; height: 12px; width: 20px; } .panels { display: flex; flex-wrap: wrap; justify-content: center; gap: 1rem; width: 100%; margin-top: 0.5rem; } .panel { flex: 1 1 200px; min-width: 180px; background: #0d1117; border-radius: 8px; padding: 0.8rem 0.6rem; border: 2px solid transparent; } .panel.green { border-color: #2da44e; } .panel.blue { border-color: #1f6feb; } .panel.red { border-color: #da3633; } .panel h4 { font-size: 0.8rem; font-weight: 600; margin-bottom: 0.5rem; text-align: center; border-bottom: 1px solid #21262d; padding-bottom: 0.3rem; color: #f0f6fc; } .panel table { width: 100%; font-size: 0.7rem; border-collapse: collapse; } .panel td { padding: 0.2rem 0.1rem; color: #e6edf3; border-bottom: 1px solid #1c2128; } .panel td:first-child { color: #8b949e; font-weight: 400; } .panel td:last-child { text-align: right; font-weight: 600; font-family: 'JetBrains Mono', monospace; } .panel .f32 { color: #58a6ff; } .panel .iq6k { color: #f0883e; } .panel .iq5k { color: #d29922; } .panel .iq4kt { color: #2da44e; } .panel .iq3kt { color: #da3633; } .note { margin-top: 1.2rem; font-size: 0.75rem; color: #6e7681; background: #0d1117; padding: 0.6rem 1.2rem; border-radius: 6px; border-left: 3px solid #1f6feb; text-align: center; width: 100%; } .note strong { color: #f0f6fc; } @media (max-width: 700px) { .container { padding: 1rem; } .bar { width: 16px; } .bar.io { width: 24px; height: 40px; } .bar.high { height: 32px; } .bar.low { height: 16px; } .panel { flex: 1 1 100%; } .legend { gap: 0.8rem 1.2rem; } } </style> </head> <body> <div class="container"> <h1>Quant Recipe</h1> <div class="track"> <div class="block"><div class="bar io"></div><span class="bar-label">Input</span></div> <div class="block"><div class="bar high"></div><span class="bar-label">0</span></div> <div class="block"><div class="bar high"></div><span class="bar-label">1</span></div> <div class="block"><div class="bar high"></div><span class="bar-label">2</span></div> <div class="block"><div class="bar high"></div><span class="bar-label">3</span></div> <div class="block"><div class="bar low"></div><span class="bar-label">4</span></div> <div class="block"><div class="bar low"></div><span class="bar-label">5</span></div> <div class="block"><div class="bar high"></div><span class="bar-label">6</span></div> <div class="block"><div class="bar low"></div><span class="bar-label">7</span></div> <div class="block"><div class="bar low"></div><span class="bar-label">8</span></div> <div class="block"><div class="bar high"></div><span class="bar-label">9</span></div> <div class="block"><div class="bar low"></div><span class="bar-label">10</span></div> <div class="block"><div class="bar low"></div><span class="bar-label">11</span></div> <div class="block"><div class="bar high"></div><span class="bar-label">12</span></div> <div class="ellipsis"><span class="dots">⋯</span><span class="label">13–48</span></div> <div class="block"><div class="bar low"></div><span class="bar-label">49</span></div> <div class="block"><div class="bar low"></div><span class="bar-label">50</span></div> <div class="block"><div class="bar high"></div><span class="bar-label">51</span></div> <div class="block"><div class="bar high"></div><span class="bar-label">52</span></div> <div class="block"><div class="bar high"></div><span class="bar-label">53</span></div> <div class="block"><div class="bar high"></div><span class="bar-label">54</span></div> <div class="block"><div class="bar io"></div><span class="bar-label">Output</span></div> </div> <div class="legend"> <span class="legend-item"> <span class="legend-swatch green"></span> Input / Output </span> <span class="legend-item"> <span class="legend-swatch blue"></span> Higher Precision </span> <span class="legend-item"> <span class="legend-swatch red"></span> Lower Precision </span> </div> <div class="panels"> <div class="panel green"> <h4>Embeddings</h4> <table> <tr><td>tokenembd</td><td class="iq6k">iq6k</td></tr> </table> </div> <div class="panel blue"> <h4>High-Precision Blocks</h4> <table> <tr><td>attnk</td><td class="iq5k">iq5k</td></tr> <tr><td>attnq</td><td class="iq5k">iq5k</td></tr> <tr><td>attnv</td><td class="iq6k">iq6k</td></tr> <tr><td>attnoutput</td><td class="iq6k">iq6k</td></tr> <tr><td>ffnup</td><td class="iq4kt">iq4kt</td></tr> <tr><td>ffngate</td><td class="iq4kt">iq4kt</td></tr> <tr><td>ffndown</td><td class="iq4kt">iq4kt</td></tr> <tr><td>attnnorm / ffnnorm</td><td class="f32">f32</td></tr> </table> </div> <div class="panel red"> <h4>Low-Precision Blocks</h4> <table> <tr><td>attnk</td><td class="iq4kt">iq4kt</td></tr> <tr><td>attnq</td><td class="iq4kt">iq4kt</td></tr> <tr><td>attnv</td><td class="iq6k">iq6k</td></tr> <tr><td>attnoutput</td><td class="iq4kt">iq4kt</td></tr> <tr><td>ffnup</td><td class="iq3kt">iq3kt</td></tr> <tr><td>ffngate</td><td class="iq3kt">iq3kt</td></tr> <tr><td>ffndown</td><td class="iq4kt">iq4kt</td></tr> <tr><td>attnnorm / ffnnorm</td><td class="f32">f32</td></tr> </table> </div> <div class="panel green"> <h4>lmhead</h4> <table> <tr><td>output</td><td class="iq6k">iq6k</td></tr> </table> </div> </div> <div class="note"> <strong>Pattern:</strong> Blocks <strong>0–3</strong> high · Middle: <strong>2 low + 1 high</strong> repeating · Blocks <strong>51–54</strong> high  ·  Total: <strong>54 layers</strong> </div> </div> </body>

imatrix generated using bartowski's combined dataset coupled with a random conversation 200k-token prune of Squish42/bluemoon-fandom-1-1-rp-cleaned.

quantized with ik_llama.cpp as of commit 43afea46c25a12aae6db1e3105643267164898b4 (Fri Aug 14 07:55:11 2026 +0200)

Cheers

[MistralAI](https://huggingface.co/mistralai) - the beloved base model(s). [ikawrakow and contributors of ik_llama.cpp](https://github.com/ikawrakow/ik_llama.cpp) - I probably misused your creation. [TheDrummer](https://huggingface.co/TheDrummer) - you cooked a legendary one. [bartowski](https://huggingface.co/bartowski) - for the combined dataset + the myriad of quants we all benefit from. [EmanuelOverride](https://huggingface.co/EmanuelOverride) - for sparking my interest in custom quantization as a concept.