lambsea/Qwen3.8-27B-AEON-Ultimate-Uncensored-UD-GGUF

Qwen3.8-27B-AEON-Ultimate-Uncensored — GGUF (UD Quants)

Unsloth Dynamic-style (UD) GGUF quantizations of AEON-7/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-BF16

Two quant families are available:

• —AWQ-UD (recommended): AWQ channel pre-scaling applied before quantization. Lower perplexity than baseline at every bit width. All AWQ-UD quants beat F16 on perplexity.
• —Baseline UD: Standard quantization without pre-scaling.

Every quant uses per-tensor overrides (sensitivity-driven) + importance matrix (multi-domain calibration). All SSM recurrence tensors are preserved at source precision. MTP speculative decoding and vision (mmproj) are preserved.

Quant Comparison

AWQ-UD (recommended)

AWQ pre-scaling (256 samples x 512 tokens, W4A16_ASYM target) redistributes weight magnitudes before quantization. All AWQ-UD quants beat F16 reference perplexity.

Baseline UD

Benchmarked on NVIDIA RTX PRO 6000 Blackwell (96 GB VRAM), llama.cpp (a4501150/llama.cpp), pp=512, tg=128. Throughput is identical between AWQ-UD and baseline UD at the same bit width because AWQ does not change tensor sizes.

AWQ-UD vs baseline: AWQ wins on perplexity (0.010-0.018 lower PPL). Baseline wins on KL divergence (closer to F16 output distribution). AWQ changes the channel basis, which shifts the output distribution away from F16, but absolute quality improves.

SGLang Throughput (NVFP4 + DFlash2)

For maximum throughput, serve the NVFP4 checkpoint via SGLang with DFlash2 speculative decoding:

SGLang is 2.3-2.8x faster single-user and 2.8x faster concurrent vs llama.cpp.

What Makes These Different

AWQ Pre-Scaling (AWQ-UD only)

AWQ (Activation-Aware Weight Quantization) applies per-channel scaling to redistribute weight magnitudes before quantization. This makes outlier channels less damaging when quantized. The scaling is applied at BF16 precision and is lossless — the model produces identical output before quantization. After scaling, the full GGUF pipeline runs: convert, importance matrix, sensitivity analysis, quantize with per-tensor overrides.

SSM Recurrence Preservation

Qwen3.8 is a hybrid GatedDeltaNet + attention model. 48 of 64 layers use a recurrent SSM where quantization error compounds across token positions. All SSM recurrence tensors are preserved at source precision (F16) — never quantized.

Per-Tensor Sensitivity Analysis

Each tensor group was probed by quantizing only that group to Q4_0 while keeping the rest at F16, then measuring KL divergence. The override generator assigns precision based on measured sensitivity:

685 total overrides — every non-FFN tensor has an explicit precision assignment. No dependence on llama-quantize's internal promotion rules.

Multi-Domain Calibration + GPU Imatrix

Calibrated on a balanced mix across 4 domains from 13 HF datasets:

Special tokens from source datasets are stripped automatically. Samples are kept whole — never truncated mid-conversation.

The importance matrix is generated with a PyTorch GPU-native generator at 32,768 context — uses forward hooks to accumulate squared activations on GPU with zero PCIe D2H copies. Supports multi-GPU via device_map="auto".

Per-domain imatrices are merged with equal weights (DI-MATRIX approach).

MTP + Vision Preserved

• —MTP (Multi-Token Prediction): Draft head (blk.64) pinned at F16. Use --spec-type draft-mtp --spec-draft-n-max 3 for ~1.5-2x faster generation.
• —Vision: mmproj file contains the full vision encoder. Use --mmproj flag with llama-server for image/video understanding.

Files

Usage

llama-server (recommended)

# AWQ-UD-Q6_K with sharp template, MTP + vision
llama-server \
    -m Qwen3.8-27B-AEON-AWQ-UD-Q6_K.gguf \
    --mmproj Qwen3.8-27B-AEON-mmproj-F16.gguf \
    -ngl 99 \
    --flash-attn \
    -c 262144 \
    --parallel 3 \
    -kvu \
    --jinja \
    --chat-template-file Qwen3.8-27B-sharp.jinja \
    --reasoning-format deepseek \
    --reasoning-preserve \
    --spec-type draft-mtp \
    --spec-draft-n-max 3 \
    --host 0.0.0.0 --port 8080

Note: --spec-type draft-mtp requires llama.cpp b9375+. --reasoning-format deepseek extracts thinking into message.reasoning_content in API responses. The sharp template (by froggeric) enables thinking by default with terse output, reasoning effort control, and tool call error detection.

llama-cli

llama-cli \
    -m Qwen3.8-27B-AEON-AWQ-UD-Q6_K.gguf \
    -ngl 99 \
    --flash-attn \
    -c 262144 \
    --jinja \
    --chat-template-file Qwen3.8-27B-sharp.jinja \
    --reasoning on \
    --reasoning-preserve

Sampling Parameters

From the official Qwen3.8-27B model card:

Do not use greedy decoding (temperature=0). reasoning_effort controls thinking depth independently: xhigh (default), medium, low.

Architecture

Qwen3.8-27B is a hybrid SSM-attention model:

• —64 transformer layers + 1 MTP layer (blk.0-64)
• —48 SSM layers (GatedDeltaNet, no KV cache) + 16 full attention layers (every 4th layer)
• —27B parameters, 24 attention heads, 4 KV heads, head dim 256
• —Vocab: 248,320 tokens, native context: 262,144 tokens

Quantization Pipeline

Built with super-quant:

1. 1.AWQ pre-scaling (AWQ-UD only): Apply per-channel weight scaling via llm-compressor AWQModifier (256 calibration samples, W4A16_ASYM target). Strip all quantization state. Save as plain BF16.
2. 2.Convert HF to F16 GGUF (with MTP tensors) + mmproj GGUF (vision)
3. 3.Multi-domain calibration data from 13 HF datasets, special tokens stripped
4. 4.GPU-native importance matrix generation (PyTorch, 32k context) + weighted merge
5. 5.Per-tensor sensitivity analysis (KL divergence probing against F16 logits)
6. 6.Hybrid override generation — SSM at source precision, sensitivity-driven for the rest
7. 7.Quantize with per-tensor overrides + imatrix
8. 8.Benchmark: throughput + perplexity + KL divergence vs F16

Links

• —Base model: AEON-7/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-BF16
• —Quantization pipeline: super-quant
• —llama.cpp fork: a4501150/llama.cpp (DFlash, MTP fixes, Blackwell FA4)

Credits

• —Base model: AEON-7
• —Architecture: Qwen Team
• —Quantization: llama.cpp
• —AWQ: MIT-HAN-LAB
• —Sensitivity methodology inspired by APEX quant research
• —Calibration datasets: HuggingFaceH4, teknium, NousResearch, nvidia, open-r1, Salesforce, glaiveai, froggeric

License: Apache-2.0