Modelpublic

berkerdooo/Qwen3.8-27B-Uncensored-INT8-AutoRound

sourceHugging Faceapache-2.0updated 1mo agoView on Hugging Face

0likes1.1kdownloads

Model Card

Qwen3.8-27B-Uncensored INT8 AutoRound (W8A16, linear attention BF16, group size 128)

INT8 weight-only quantization of orcarouter/Qwen3.8-27B-Uncensored with AutoRound (SignRound), following the recipe of Minachist/Qwen3.8-27B-INT8-AutoRound branch linear-attn-bf16-gs128, with two changes: linear attention is excluded from tuning (not swapped back to BF16 after the fact), and 500 iters instead of 250.

Tensors	Precision
`self_attn.{q,k,v,o}_proj` (16 full-attention layers), `mlp.{gate,up,down}_proj` (64 layers), MTP block projections	INT8 symmetric, group_size 128
`linear_attn.{in_proj_qkv,in_proj_z,out_proj,in_proj_a,in_proj_b}` (48 GDN layers), `embed_tokens`, `lm_head`, `mtp.fc`, norms, vision tower	BF16

263 INT8 linears / 354 BF16 linears. Format: auto_round:auto_gptq packing (vLLM loads it via GPTQ-Marlin with BF16 activations).

Recipe

AutoRound main @ b9f3d0079d014c73a1ff009800c597b9bc3f2a36 (version string 0.15.0), transformers 5.15.1, torch 2.13.0+cu130, one RTX PRO 6000 Blackwell. scheme="W8A16" (bits 8, groupsize 128, sym), `iters=500`, `nsamples=1024`, `seqlen=2048`, `batchsize=4, gradientaccumulatesteps=2, lowgpumemusage=False`, `seed=42`. Calibration: 256 samples built from `NeelNanda/pile-10k` + 768 from `codeparrot/github-code-clean` (documents concatenated so every sample is >= 2048 tokens, then truncated to 2048). Every layer is named in full in `layerconfig` (avoids AutoRound's shared-dict regex aliasing bug). Tuning took 1.26 h.

KL divergence vs the BF16 source

Teacher-forced top-24 logprobs on one 128,000-token wikitext-103 stream (rows 100k+ of the train split), one sequence, BF16 KV cache, vLLM 0.27.1, KL(P_bf16 || Q_int8) in nats over the truncated top-24. These numbers are only comparable to other models scored with the same script, stream and teacher.

depth	n	KL mean	KL p50	KL p99	top-1 agreement	ΔNLL
0k-4k	3,999	0.00189	0.00056	0.0239	97.67%	+0.0038
4k-16k	12,000	0.00363	0.00078	0.0338	97.51%	+0.0011
16k-48k	32,000	0.00264	0.00085	0.0293	97.22%	+0.0019
48k-128k	80,000	0.00320	0.00088	0.0338	97.28%	+0.0022

Own NLL: BF16 1.8244, INT8 1.8265. For reference, the same script on Qwen/Qwen3.8-27B gives FP8 (Qwen/Qwen3.8-27B-FP8) KL 0.0048 / top-1 96.5% and Minachist's INT8 0.0029 / 97.2%.

Serving

bash

vllm serve <this-repo> --tensor-parallel-size 2 --max-model-len 262144 --trust-remote-code

Tested with vLLM 0.27.1 (Using MarlinLinearKernel for AutoGPTQLinearMethod). MTP speculative decoding: --speculative-config '{"method":"mtp","num_speculative_tokens":3}'.