Modelpublic

k0valik/Ornstein3.6-27B-MTP-NSC-ACE-SABER-IQ4_XS-13GiB-GGUF

sourceHugging Faceapache-2.0updated 4mo agoView on Hugging Face

1likes84downloads

Model Card

Ornstein3.6-27B-MTP-NSC-ACE-SABER-IQ4_XS-13GiB

Model: Ornstein3.6-27B-MTP-NSC-ACE-SABER-IQ4_XS-13GiB.gguf (13.33 GiB) Base: GestaltLabs/Ornstein3.6-27B-MTP-NSC-ACE-SABER Quantization: Custom 866-rule recipe targeting ~13 GiB for 16 GB VRAM GPUs

Why I made this

This is just a fun sidequest of me trying my hands on quantizing relatively large models on relatively low resources. Since I have no realistic way to measure the "quality" of the quant properly, I've done some simple benchmarks locally, and didn't notice any quality drop in terms of answers or output. Overall I would say it is a relatively good balanced quant that has a smaller footprint allows users on 16 GB VRAM to test MTP without offloading to CPU making it completely useless with relatively okay (35-45k) context size

Acknowledgements

Thanks to:

—[ikawrakow / ik_llama.cpp](https://github.com/ikawrakow/ik_llama.cpp) — The custom quantization scheme implementation and MTP inference engine. This quant was produced using the latest ik_llama.cpp release (build 4758).
—[Thireus / GGUF Tool Suite](https://github.com/Thireus/GGUF-Tool-Suite) — The recipe generation pipeline and base recipe template. The importance-aware quant_assign.py and harmonization techniques made the mixed-IQ approach practical and reproducible.
—[ubergarm / imatrix corpus](https://huggingface.co/ubergarm/Qwen3.6-27B-GGUF/blob/main/imatrix-Qwen3.6-27B-BF16.dat) — The importance matrix used during quantization, derived from KLD evaluation. Gist.
—[GestaltLabs](https://huggingface.co/GestaltLabs) — For the Ornstein3.6-27B base model.

Quantization

Specs

Property	Value
Base model	`Ornstein3.6-27B-MTP-NSC-ACE-SABER-F16-MTP.gguf` (GGUF V3)
F16 model size	52,115.19 MB (~50.9 GiB)
Quantized size	13,644.33 MB (~13.3 GiB)
Compression ratio	3.82× (F16 → quant)
Total tensors	866 (37 metadata KV pairs)
Architecture	qwen35 — 65 blocks (64 main + 1 MTP draft)
Context length	262,144 tokens
Attention	GQA: 24 heads, 4 KV heads (6:1 ratio)
Embedding length	5,120
FFN intermediate	17,408
SSM state size	128, conv kernel 4, group count 16
Tokenizer	GPT-2 model, qwen35 pre-tokenizer, 248,320 tokens
Importance matrix	ubergarm's corpus — 497 entries from 829 chunks
ik_llama.cpp build	4758

Tensor Distribution (866 tensors)

Quant Type	Count	%
f32	360	41.6%
iq4_kt	222	25.6%
iq4_k	87	10.0%
q8_0	73	8.4%
iq4_ks	41	4.7%
iq5_ks	26	3.0%
iq5_k	18	2.1%
iq6_k	17	2.0%
q6_K	16	1.8%
iq3_kt	4	0.5%
iq3_k	2	0.2%

Layer Assignment

—MTP Draft Head (Block 64): q6_K (attention/FFN) + f32 (norms)
—Attention blocks: iq4kt / iq4ks
—FFN blocks: mixed iq4k / iq3k / iq5_ks
—SSM weights: q80 / iq6k
—Norms/Embeds: f32

Comparison

vs Ornstein3.6-27B-MTP-NSC-ACE-SABER-Q4_K_S-MTP

—For fun measured PPL against Ornstein3.6-27B-MTP-NSC-ACE-SABER-Q4KS-MTP.gguf from wiki.test.raw on 250 chunk as well to see how smaller chunk affects PPL measurement (and since I don't have the resources, this is the closes approximation I could do locally as to how well it worked)

Model	250 chunks	Full corpus	ΔPPL (250→Full)
Custom Quant (IQ4_XS blend)	6.5665	6.7574	+0.1909
Q4KS (reference)	6.6038	6.7770	+0.1732

The quant beats Q4KS by 0.037 PPL on the 250-chunk run and 0.020 PPL on the full corpus, despite being 1.4 GiB smaller (13.33 GiB vs 14.74 GiB) — confirming the mixed-IQ approach consistently preserves quality better than a uniform K-quant across both sample sizes.

vs lemonyins/Qwen3.6-27B-uncensored-abliterated-MTP-i1-IQ4_XS-GGUF-Smaller

Another MTP-capable Qwen3.6-27B GGUF targeting 16 GB VRAM, using a different quantization philosophy.

Dimension	This Model (Ornstein)	lemonyins' Model
File size	13.33 GiB	12.25 GiB
Bits-per-weight	4.242 bpw	3.896 bpw
Quantization strategy	10-type importance-aware mixed IQ	2-precision: attn=IQ4XS, FFN=IQ3S
Quant types used	iq4kt, iq4k, iq4ks, iq5ks, iq5k, iq6k, q6K, q80, iq3kt, iq3k (10)	IQ4XS + IQ3S (3)
MTP draft head	Explicit Q6 uniform (block 64)	Inherited from base (not separately specified)
KV cache format	Standard q80 K / q60 V + Hadamard	TurboQuant turbo4
Max context (16 GB)	~40K (q80+q60)	~60K (turbo4)
KV quant suggestion	Hadamard transform for tighter quantization	TurboQuant format — ~25% smaller KV per token
Draft model	GestaltLabs/Ornstein3.6-27B (native)	llmfan46 (extracted from MoE)
Uncensored	Inherited refusal removal from SABER	Yes (abliterated)
Hardware tested	RTX 5070 Ti 16 GB	RTX 4060 Ti 16 GB
Gen speed (short ctx)	~46.9 t/s	~35 t/s (claimed)
PPL	6.7574 ±0.04339	Not published

Key Takeaways

—My (Thireus') recipe uses more precision headroom (+1.09 GiB, +0.346 bpw) via importance-aware assignment across 10 quant types. This trades raw VRAM efficiency for higher per-weight quality — especially on attention weights (iq4kt) and the MTP draft head (q6K).

—Different target tradeoff: My recipe prioritizes weight precision (more types, higher bpw for sensitive layers). Theirs prioritizes KV density (turbo4 + aggressive FFN quant) to push context length.

PPL note: lemonyins does not publish perplexity numbers, so a direct quality comparison isn't possible. My quant's PPL is independently measured at 6.5665 ±0.0635 (250 chunks) and 6.7574 ±0.0434 (full corpus) — beating reference Q4KS by 0.04 despite being 1.4 GiB smaller, suggesting the mixed-IQ approach preserves quality well.

Hardware Benchmarks

Prompt & Generation Speed (llama-bench)

| model                          |       size |     params | backend    | ngl |          test |              t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------------: | ---------------: |
| qwen35 27B Q8_0                |  13.00 GiB |    26.90 B | CUDA       |  99 |         pp512 | 1628.65 ± 431.79 |
| qwen35 27B Q8_0                |  13.00 GiB |    26.90 B | CUDA       |  99 |        pp2048 |  1782.18 ± 14.46 |
| qwen35 27B Q8_0                |  13.00 GiB |    26.90 B | CUDA       |  99 |         tg128 |     46.93 ± 0.49 |
| qwen35 27B Q8_0                |  13.00 GiB |    26.90 B | CUDA       |  99 |         tg512 |     46.24 ± 0.21 |

build: 1f0ec4c7 (4758)

KV Cache Scaling (llama-sweep-bench)

|    PP |     TG |   N_KV |   T_PP s | S_PP t/s |   T_TG s | S_TG t/s |
|-------|--------|--------|----------|----------|----------|----------|
|   512 |    128 |      0 |    0.582 |   879.24 |    2.750 |    46.54 |
|   512 |    128 |    512 |    0.299 |  1710.88 |    2.765 |    46.29 |
|   512 |    128 |   1024 |    0.295 |  1734.53 |    2.776 |    46.11 |
|   512 |    128 |   1536 |    0.293 |  1748.85 |    2.781 |    46.03 |
|   512 |    128 |   2048 |    0.294 |  1741.08 |    2.789 |    45.89 |
|   512 |    128 |   2560 |    0.296 |  1730.99 |    2.794 |    45.82 |
|   512 |    128 |   3072 |    0.296 |  1731.14 |    2.801 |    45.70 |
|   512 |    128 |   3584 |    0.297 |  1722.06 |    2.810 |    45.56 |
|   512 |    128 |   4096 |    0.301 |  1699.94 |    2.825 |    45.31 |
|   512 |    128 |   4608 |    0.303 |  1691.57 |    2.830 |    45.22 |
|   512 |    128 |   5120 |    0.301 |  1701.92 |    2.838 |    45.11 |
|   512 |    128 |   5632 |    0.304 |  1684.85 |    2.845 |    44.99 |
|   512 |    128 |   6144 |    0.304 |  1682.45 |    2.854 |    44.85 |
|   512 |    128 |   6656 |    0.306 |  1675.29 |    2.858 |    44.79 |
|   512 |    128 |   7168 |    0.308 |  1663.64 |    2.866 |    44.67 |
|   512 |    128 |   7680 |    0.309 |  1656.83 |    2.869 |    44.62 |

Sweep-bench measures PP and TG speed at increasing KV cache sizes (N_KV).

Usage

—Haven't figured out a proper way to actually enjoy this on 16GB VRAM

bash

./llama-cli \
  -m Ornstein3.6-27B-MTP-NSC-ACE-SABER-IQ4_XS-13GiB.gguf \
  -ngl 99 \
  -c 81920 \
  --khad --vhad \
  -ctk q6_K -ctv q6_K \
  -mtp --draft-max 2 --draft-p-min 0.80 --draft-min 1

Key observations from local benchmarking:

—Even at its best (28.5 t/s, 0.84 acceptance), MTP doesn't beat no-MTP (38 t/s) — the 13 GiB dense model saturates memory bandwidth, and MTP's extra forward passes always cost something.
—V cache quality directly controls MTP acceptance: q40 V → 0.66, q60 V+H → 0.76, with draft-max=2 + p_min=0.80 → 0.84.
—Code tasks consistently achieve the highest draft acceptance (~0.92), making MTP more worthwhile for structured generation.