CoolFace
Modelpublic

amkkk/Qwen3.5_2B_Abiliterate_All_Layers_Baked_GGUF_quantized

sourceHugging Faceapache-2.0updated 7mo agoView on Hugging Face
2likes875downloads
Model Card

Qwen3.52BAbiliterateAllLayersBakedGGUF

GGUF companion release for Qwen3.5_2B_Abiliterate_All_Layers_Baked.

Files

  • —qwen3.5_2b_abiliterate_all_layers_baked.f16.gguf
  • —qwen3.5_2b_abiliterate_all_layers_baked.Q2_K.gguf
  • —qwen3.5_2b_abiliterate_all_layers_baked.Q3_K_S.gguf
  • —qwen3.5_2b_abiliterate_all_layers_baked.Q3_K_M.gguf
  • —qwen3.5_2b_abiliterate_all_layers_baked.Q3_K_L.gguf
  • —qwen3.5_2b_abiliterate_all_layers_baked.Q4_K_S.gguf
  • —qwen3.5_2b_abiliterate_all_layers_baked.Q4_K_M.gguf
  • —qwen3.5_2b_abiliterate_all_layers_baked.Q5_K_S.gguf
  • —qwen3.5_2b_abiliterate_all_layers_baked.Q5_K_M.gguf
  • —qwen3.5_2b_abiliterate_all_layers_baked.Q6_K.gguf
  • —qwen3.5_2b_abiliterate_all_layers_baked.Q8_0.gguf

Quant guide

QuantSizeUse whenTradeoff
Q2_K0.85 GBYou need the smallest possible file and can tolerate a clear quality dropLowest memory use, weakest output quality
Q3_K_S0.95 GBYou are below 8 GB RAM/VRAM and want a small step up from Q2_KStill a noticeable quality hit
Q3_K_M1.02 GBYou want the best low-end compromise for constrained devicesBetter than Q3_K_S, still compressed hard
Q3_K_L1.06 GBYou want a slightly safer Q3 choice without moving into Q4Marginally larger for a modest gain
Q4_K_S1.12 GBYou want a compact everyday quant and are optimizing for size firstGood balance, a bit weaker than Q4_K_M
Q4_K_M1.18 GBYou want the standard balanced option for general local useBest default size/quality compromise for many setups
Q5_K_S1.28 GBYou have more headroom and want to preserve quality better than Q4Larger file for a smaller quality jump
Q5_K_M1.33 GBYou want a strong general-purpose quant without going near full precisionBest practical choice if RAM/VRAM is not very tight
Q6_K1.45 GBYou want near-high-quality local inference and can afford the extra memoryLarger and slower than Q5_K_M
Q8_01.87 GBYou want to stay as close as possible to f16 while still using GGUF quantizationHighest quality quant here, but much heavier
f163.52 GBYou want the least quantization loss and have plenty of memory/storageLargest file by a wide margin

Local usage

bat
local.bat

To run a specific quant without editing the script:

bat
set MODEL=qwen3.5_2b_abiliterate_all_layers_baked.Q5_K_M.gguf
local.bat

Notes

  • —Source HF: https://huggingface.co/amkkk/qwen3.5-2B-abliterated-alllayers
  • —This is from the checkpoint build which performed even better than the runtime hook ablated variant that I have uploaded here