maximg/AutoRoundTest

1---2license: apache-2.03language:4-  multilingual5base_model: Qwen/Qwen3.6-27B6tags:7- auto-round8- intel9- gguf10- quantization11---12 13# Qwen3.6-27B GGUF (AutoRound Quantized, MTP Enabled)14 15This repository contains GGUF quantized versions of [Qwen/Qwen3.6-27B](https://huggingface.co/Qwen/Qwen3.6-27B) created using Intel's [AutoRound](https://github.com/intel/auto-round) quantization method.16 17## Quantization Details18 19The models were generated using Intel's AutoRound using ultrachat_200k as the test dataset and using sequence length of 2850. MTP layers were not explicitly enabled, but it works with MTP for me20 21```bash22auto-round \23    --model Qwen/Qwen3.6-27B \24    --output_dir ./quantized/ \25    --scheme <SCHEME> \26    --format <SCHEME> \27    --iters 0 \28    --nsamples 256 --seqlen 2850 --dataset "HuggingFaceH4/ultrachat_200k"29```30 31For now, only 2 quantization variants were used Q5_K_M and Q4_K_MIXED. Q4_K_MIXED is a custom variant based on Intel's original Q2_K_MIXED quantization, but using Q4_K quants instead of Q2.32 33 34### Files and Sizes35 36| File Name | Quant Type | Size |37|-----------|------------|------|38| `Qwen3.6-27B-Q2_K_MIXED.gguf` | Q2_K_MIXED | 16.5 GB |39| `Qwen3.6-27B-Q5_K_M.gguf` | Q5_K_M | 19 GB |40