cHunter789/Qwen3.8-27B-i1-IQ4_KS_KT-GGUF
445.1k
Qwen3.8-27B-i1-IQ4KSKT-GGUF
This quantization was created to allow the entire model to fit into the memory of an NVIDIA graphics card with 16GB of VRAM.
Importan - please use "export GGML_CUDA_ENABLE_UNIFIED_MEMORY=1" after that just use ctx-cliff.py to check if you server config fit in you vram.
This model has the tokenemd.weight tensor bumped up to q80, which increases its download size but makes no difference in VRAM. By default, the tokenemd.weight tensor is loaded entirely into system RAM. Upgrading it to q80 is therefore, in practice, a free upgrade to the model's quality, for example in interpreting non-English languages.
This repository contains GGUF format weights for the Qwen3.8-27B model, quantized using the ik_llama.cpp project.
This model was specifically created to run on consumer GPUs with 16GB VRAM. By utilizing q4_0 KV cache quantization, it allows pushing the context length up to 110k tokens.
Note: This model is designed exclusively for nVidia GPUs and is based on the advanced KS and KT quants developed by ikawrakow from the ik_llama.cpp repository.
Quantization Details & Imatrix File
- Quantization Base: IQKS,IQKT Quants (ik_llama.cpp).
- Imatrix File Used: Mix of mradermacher and ubergarm imatrix files.
- Other Tested Imatrix Files:
- bartowski – yielded significantly worse results.
- ubergarm – yielded comparable results.
- If you find or generate a better Imatrix file, please let me know in the Community tab!
GPU Only full 16GB VRAM the configuration below will give you a 105k context (110k is the max) (q40/q40). Decode speed starts at 45 t/s and falls to 25 t/s at the end, with a prefill speed above 1300 t/s (RTX 5070 Ti).
llama-server" \
-m "$MODEL_PATH" \
-a Qwen3.8-27B \
--ctx-size 105000 \
--n-gpu-layers 99 \
--cache-type-k q4_0 \
--cache-type-v q4_0 \
--spec-type ngram-mod:n_max=2 \
--batch-size 512 \
--ubatch-size 128 \
--flash-attn on \
--host 0.0.0.0 \
--port 8081 \
--reasoning on \
--reasoning-format deepseek \
--reasoning-budget 32000 \
-t 8 \
-tb 8 \
--parallel 1 \
--metrics \
--merge-qkv \
-khad \
-vhad \
--chat-template-kwargs '{"preserve_thinking": true, "reasoning_effort": "medium"}' \
--defrag-thold 0.4 \
--jinja \
--cont-batching \
--temp 1.0 \
--top-k 20 \
--min-p 0.00 \
--top-p 0.95 \
--presence-penalty 0.0 \
--repeat-last-n 512 \
--repeat-penalty 1.0Perplexity Comparison: mradermacher vs. cHunter789
I compared the model by mradermacher with the model by cHunter789 using the War and Peace text file as a dataset.
Test dataset download:
wget [https://www.gutenberg.org/files/2600/2600-0.txt](https://www.gutenberg.org/files/2600/2600-0.txt) -O pg19.txt1. Model: IQ4_XS (mradermacher)
Source: mradermacher/Qwen3.8-27B-i1-GGUF
Command
./llama-perplexity \
-m Qwen3.8-27B.i1-IQ4_XS.gguf \
-f pg19.txt \
-c 65536 \
--chunks 32 \
-ngl 99 \
-ctk q4_0 \
-ctv q4_0 \
-khad \
-vhad \
--merge-qkv \
-fa 1 \
-b 512 \
-ub 128 \
-t 8Output
perplexity: calculating perplexity over 12 chunks, n_ctx=65536, batch_size=512, n_seq=1
perplexity: 77.21 seconds per pass - ETA 15.43 minutes
[1]7.7001,[2]8.0855,[3]8.2723,[4]8.3528,[5]8.5027,[6]8.3485,[7]8.3948,[8]8.3789,[9]8.4068,[10]8.4465,[11]8.4932,[12]8.2955,
Final estimate: PPL over 12 chunks for n_ctx=65536 = 8.2955 +/- 0.03221
llama_print_timings: load time = 2522.03 ms
llama_print_timings: sample time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
llama_print_timings: prompt eval time = 750310.64 ms / 786432 tokens ( 0.95 ms per token, 1048.14 tokens per second)
llama_print_timings: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
llama_print_timings: total time = 1360215.26 ms / 786433 tokens2. Model: IQ4_KT (cHunter789)
Source: cHunter789/Qwen3.8-27B-i1-IQ4_KS_KT-GGUF
Command
./llama-perplexity \
-m Qwen3.8-27B.i1-IQ4_KT-attn_qkv-IQ4_KS.gguf \
-f pg19.txt \
-c 65536 \
--chunks 32 \
-ngl 99 \
-ctk q4_0 \
-ctv q4_0 \
-khad \
-vhad \
--merge-qkv \
-fa 1 \
-b 512 \
-ub 128 \
-t 8Output
perplexity: calculating perplexity over 12 chunks, n_ctx=65536, batch_size=512, n_seq=1
perplexity: 84.14 seconds per pass - ETA 16.82 minutes
[1]7.6895,[2]8.0682,[3]8.2542,[4]8.3333,[5]8.4853,[6]8.3416,[7]8.3890,[8]8.3734,[9]8.3999,[10]8.4386,[11]8.4857,[12]8.2890,
Final estimate: PPL over 12 chunks for n_ctx=65536 = 8.2890 +/- 0.03224
llama_print_timings: load time = 9369.56 ms
llama_print_timings: sample time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
llama_print_timings: prompt eval time = 783243.61 ms / 786432 tokens ( 1.00 ms per token, 1004.07 tokens per second)
llama_print_timings: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
llama_print_timings: total time = 2323646.53 ms / 786433 tokens<details> <summary><b>1. Reference Model Test ctx-cliff.py Results (Click to expand)</b></summary> Test with ctx-cliff.py (RTX5070Ti)
llama-server \
-m "$MODEL_PATH" \
-a Qwen3.6-27B \
--ctx-size 110000 \
--n-gpu-layers 99 \
--cache-type-k q4_0 \
--cache-type-v q4_0 \
--batch-size 512 \
--ubatch-size 128 \
--flash-attn on \
--host 0.0.0.0 \
--port 8081 \
--reasoning on \
--reasoning-format none \
--reasoning-budget 32000 \
-t 8 \
-tb 8 \
--parallel 1 \
--metrics \
--merge-qkv \
-khad \
-vhad \
--chat-template-kwargs '{"preserve_thinking": true, "reasoning_effort": "medium"}' \
--defrag-thold 0.1 \
--jinja \
--cont-batching \
--temp 1.0 \
--top-k 20 \
--min-p 0.00 \
--top-p 0.95 \
--presence-penalty 0.0 \
--repeat-last-n 512 \
--repeat-penalty 1.00python3 ctx-cliff.py --file tests/code_4M.py --start 2000 --end 109000 --step 2000 --n-predict 512
ctx | prefill| decode| MTP| wall| status
-------------------------------------------------
1999 | 1021.1 | 46.72| 0/0| 11.4s| OK
3925 | 1320.9 | 46.06| 0/0| 12.6s| OK
6017 | 1261.5 | 45.12| 0/0| 13.0s| OK
8065 | 1293.4 | 44.20| 0/0| 13.2s| OK
10218 | 1191.9 | 43.36| 0/0| 13.6s| OK
12489 | 1184.3 | 42.39| 0/0| 14.0s| OK
14525 | 1228.9 | 41.66| 0/0| 15.9s| OK
16108 | 1258.8 | 41.16| 0/0| 17.6s| OK
18976 | 1237.8 | 40.13| 0/0| 20.8s| OK
20476 | 1058.4 | 39.67| 0/0| 15.0s| OK
22574 | 1091.4 | 39.22| 0/0| 15.8s|STOP@463
24950 | 1082.3 | 38.15| 0/0| 16.3s| OK
26551 | 1060.3 | 37.79| 0/0| 18.0s| OK
29197 | 1058.8 | 37.16| 0/0| 20.7s| OK
30559 | 1059.6 | 36.81| 0/0| 22.1s| OK
32691 | 1048.1 | 36.17| 0/0| 24.5s| OK
34235 | 1046.3 | 35.75| 0/0| 26.2s| OK
36569 | 1037.3 | 35.22| 0/0| 28.7s| OK
38356 | 1027.6 | 34.94| 0/0| 30.7s| OK
40912 | 1014.6 | 34.14| 0/0| 33.8s| OK
42569 | 1010.3 | 34.00| 0/0| 35.6s| OK
44532 | 1002.9 | 33.27| 0/0| 32.1s|STOP@316
47017 | 994.5 | 32.53| 0/0| 44.2s| OK
48257 | 997.5 | 32.83| 0/0| 47.7s| OK
51210 | 996.4 | 32.06| 0/0| 53.1s| OK
52481 | 826.8 | 31.97| 0/0| 18.4s| OK
54608 | 803.1 | 31.43| 0/0| 18.9s| OK
56263 | 775.9 | 31.16| 0/0| 18.6s| OK
58871 | 847.3 | 30.54| 0/0| 24.2s| OK
60014 | 795.5 | 30.38| 0/0| 21.1s| OK
62496 | 825.7 | 29.95| 0/0| 26.7s| OK
64364 | 780.5 | 29.67| 0/0| 23.2s| OK
65843 | 775.2 | 29.08| 0/0| 25.5s| OK
67130 | 746.5 | 28.85| 0/0| 22.2s| OK
68719 | 757.5 | 28.73| 0/0| 24.3s| OK
70803 | 753.8 | 28.45| 0/0| 27.3s| OK
72826 | 712.6 | 28.20| 0/0| 22.2s| OK
74889 | 723.3 | 27.72| 0/0| 25.3s| OK
76819 | 725.8 | 27.43| 0/0| 28.1s| OK
78975 | 723.9 | 27.09| 0/0| 31.4s| OK
81045 | 678.9 | 26.81| 0/0| 23.4s| OK
83184 | 755.7 | 26.48| 0/0| 36.9s| OK
85162 | 712.9 | 26.14| 0/0| 41.0s| OK
87191 | 684.5 | 25.90| 0/0| 31.5s| OK
89098 | 728.9 | 25.66| 0/0| 44.9s| OK
90969 | 706.3 | 25.30| 0/0| 50.0s| OK
93074 | 696.8 | 25.12| 0/0| 53.6s| OK
95132 | 654.1 | 24.84| 0/0| 34.1s| OK
97250 | 614.2 | 24.56| 0/0| 25.3s| OK
99301 | 680.1 | 24.27| 0/0| 40.2s| OK
101183 | 629.1 | 24.14| 0/0| 31.9s| OK
103237 | 668.2 | 23.83| 0/0| 46.8s| OK
105209 | 624.7 | 23.64| 0/0| 38.8s| OK
107265 | 655.6 | 23.37| 0/0| 53.9s| OK</details>