Modelpublic

Lygodactylus/Qwen3.8-27B-Uncensored-exl3-4bpw

sourceHugging Faceapache-2.0updated 24d agoView on Hugging Face

8likes618downloads

Model Card

<p align="center"> <img src="https://cdn-uploads.huggingface.co/production/uploads/67fcdd9530700089d649ed6e/xHG-9ipsnpjtplgIU2aWr.png" width="200" alt="Qwen3.8-27B-Uncensored EXL3" /> </p> <h1 align="center">Qwen3.8-27B-Uncensored — EXL3 4.0 bpw</h1>

EXL3 quantization of orcarouter/Qwen3.8-27B-Uncensored, an abliterated (refusal-removed) build of Qwen/Qwen3.8-27B.

Built with ExLlamaV3 1.4.6.

16 GB on disk — fits a single 24 GB card with room for context.

The MTP head is preserved at 8 bpw and works. GGUF conversions of this family drop the mtp.* tensors, so they have no self-drafter. TabbyAPI logs Using main model MTP component for drafting on load.

Sibling repos

Variant	Size	Repo
4.0 bpw	16 GB	this one
6.0 bpw	22 GB	Qwen3.8-27B-Uncensored-exl3-6bpw
8.0 bpw	28 GB	Qwen3.8-27B-Uncensored-exl3-8bpw

Build


Language model	4.00 bpw
`lm_head`	6 bpw
MTP layers	8 bpw
Vision tower	unquantized (16-bit)
Embeddings	unquantized (16-bit)

bash

python convert.py \
  -i Qwen3.8-27B-Uncensored \
  -o Qwen3.8-27B-Uncensored-exl3-4bpw \
  -w /tmp/exl3-work \
  -b 4.0 -hb 6 -mb 8 -vb 16 \
  -d 0,1,2,3 -v

Default bundled calibration corpus (wiki 50, C4 20, code 20, random tokens 20, technical 10, multilingual 10, tiny 5), 250 rows x 2048 columns.

Usage — TabbyAPI

yaml

model:
  model_dir: /path/to/models
  model_name: Qwen3.8-27B-Uncensored-exl3-4bpw
  cache_size: 32768
  cache_mode: "8,8"
  tensor_parallel: true

draft_model:
  draft_mode: mtp

Benchmarks

Hardware: 4x RTX 4000 Ada (20 GiB, sm89), PCIe, no NVLink, 2x Xeon Gold 5318Y. AIPerf, streaming, ignore_eos, seed 42, with warmup. ExLlamaV3 1.4.6 + TabbyAPI, tensor-parallel, cache_mode "8,8", MTP on.

Bitrate does almost nothing here

ISL 1000 / OSL 500 / concurrency 4 / 20 requests:

	4.0 bpw	6.0 bpw	8.0 bpw
Size	16 GB	22 GB	28 GB
Request latency	19,906 ms	20,816 ms	21,145 ms
ITL	33.5 ms	35.0 ms	36.2 ms
TTFT	3,352 ms	3,515 ms	3,289 ms
Output throughput	97 tok/s	91 tok/s	90 tok/s

75% more weight from 4 to 8 bpw costs only 6% latency. On four tensor-parallel cards over PCIe, weight bandwidth is not the bottleneck — inter-GPU synchronization dominates. Pick the bitrate on VRAM budget and quality, not on speed.

The 8 bpw TTFT being lowest of the three is noise; 20 requests can't resolve 5% gaps.

vs vLLM (same weights, FP8 build)

	vLLM 0.28.0 FP8	EXL3 6.0 bpw
Request latency	14,158 ms	20,816 ms
ITL	26.3 ms	35.0 ms
TTFT	1,028 ms	3,515 ms

Long prompts (ISL 8000 / conc. 2): vLLM TTFT 5,508 ms vs 9,481 ms, prefill 1,733 vs 1,277 tok/s.

vLLM is faster in every served scenario on this box. EXL3 wins on VRAM and on single-stream short prompts (63 vs ~45 tok/s in chat.py).

`tensor_parallel` is a trade, not a win

	TP off	TP on
Prefill TTFT (8k)	17,644 ms	9,577 ms
Single-stream generation	63 tok/s	34.6 tok/s

Helps compute-bound prefill, hurts memory-bound decode. State which one you ran when quoting EXL3 numbers.

Caveat: PCIe, no NVLink, SymmMemCommunicator unavailable on sm89. TP results are likely better on NVLink systems. Treat these as a floor.

Quality

Not benchmarked at 4.0 bpw. For reference on the same abliterated weights, the FP8 build scores 88.0% on MMLU-Pro (business subset, 100 questions, 2 unparsed) vs 89.0% for official Qwen3.8-27B-FP8 — within noise. How 4.0 bpw compares has not been verified. At this bitrate expect measurable degradation vs the 6 and 8 bpw siblings.

Safety

Inherited from the base model: safety alignment has been substantially removed via abliteration. It will comply with requests the original Qwen3.8-27B refuses, and has no meaningful built-in guardrails. Released for research and controlled experimentation. Add your own moderation layer before any deployment. You are responsible for what you do with it.

License

Apache 2.0, inherited from Qwen/Qwen3.8-27B.

Credits

—Qwen — base model
—orcarouter — abliterated build
—turboderp — ExLlamaV3 / EXL3
—theroyallab — TabbyAPI