lued/Qwen3.8-27B-heretic-ara-INT8-W8A16-MTP

<div align="center"> <img src="https://qianwen-res.oss-accelerate-overseas.aliyuncs.com/logoqwen3.png" width="380" alt="Qwen"> <h1>Qwen3.8-27B Heretic ARA · INT8 W8A16 · BF16 MTP</h1> <p><strong>An abliterated, Ampere-optimized quantization for dual RTX 3090 inference.</strong></p> <p> <a href="https://huggingface.co/heretic-org/Qwen3.8-27B-heretic-ara">Base model</a> · <a href="https://huggingface.co/Qwen/Qwen3.8-27B">Qwen3.8-27B</a> · <a href="https://huggingface.co/lued/Qwen3.8-27B-INT8-W8A16-MTP">Sibling quantization</a> · <a href="https://github.com/p-e-w/heretic">Heretic</a> · <a href="https://github.com/vllm-project/vllm">vLLM</a> · <a href="https://github.com/vllm-project/llm-compressor">llm-compressor</a> </p> <p> <img alt="Base Heretic ARA" src="https://img.shields.io/badge/base-Heretic%20ARA-8B5CF6"> <img alt="Format W8A16" src="https://img.shields.io/badge/format-W8A16-0A2EFE"> <img alt="Weights INT8" src="https://img.shields.io/badge/weights-INT8-6C4CF1"> <img alt="Activations FP16 or BF16" src="https://img.shields.io/badge/activations-FP16%20%2F%20BF16-555555"> <img alt="Target Ampere" src="https://img.shields.io/badge/target-Ampere%20sm_86-76B900"> <img alt="License Apache 2.0" src="https://img.shields.io/badge/license-Apache--2.0-blue"> </p> </div>

[!NOTE] This is a numerical W8A16 quantization of heretic-org/Qwen3.8-27B-heretic-ara, an abliterated (decensored) version of Qwen/Qwen3.8-27B. All model credit belongs to heretic-org (the abliteration) and Qwen (the base model); refer to the upstream model card for architecture, capabilities, evaluations, and usage guidance. This repository changes numerics only.

[!TIP] This build targets vLLM on Ampere-class GPUs where native FP8 tensor-core execution is unavailable. It retains the model's vision tower, thinking controls, native 262K context, and the byte-preserved BF16 MTP head. The upstream model is extensible to 1M tokens with YaRN; that is separate from the native context and the available KV capacity.

Relationship to the sibling quantization

This checkpoint is the Heretic ARA counterpart of lued/Qwen3.8-27B-INT8-W8A16-MTP. The two repositories share:

• —the identical W8A16 compressed-tensors recipe (400 packed GEMMs, same BF16 preservation set, group-128 symmetric data-free RTN);
• —the identical native BF16 MTP head (15 tensors, byte-identical to Qwen's original MTP);
• —the identical tokenizer, chat template, processor configs, and 6-shard layout;
• —the identical validated serving profile (TP2, 262144, fp8_e4m3 KV, MTP × 3).

The only deviation is the base language-model weights: this checkpoint's LM weights come from the abliterated heretic-ara model instead of stock Qwen3.8-27B. Serving performance is unchanged by design and was verified identical to the sibling deployment locally; measurements in this card inherited from the sibling card are marked as such. Behavior — specifically the refusal profile — is the intended deviation and is documented below.

Abliteration (inherited from the base)

Method: Heretic (timrohrbaugh fork) v1.2.0+custom with the Arbitrary-Rank Ablation (ARA) method, run in full BF16 on a single H200 NVL. 60-trial Optuna study (seed 705085018), selected trial 37: layers 26–56, targets attn.o_proj + mlp.down_proj, preserve_good_behavior_weight=0.9432, steer_bad_behavior_weight=0.0009, overcorrect_relative_weight=0.5038, neighbor_count=10, row_normalization=none, winsorization_quantile=1.0. The full parameter set, per-category refusal breakdowns, and the reproduce/ bundle (config, reproduce.json, SHA256SUMS) live in the base repository.

The 0.0535 KL is intentionally larger than a surgical abliteration (≈0.002 class): ARA trades a larger distribution shift for complete refusal elimination. If minimal drift matters more than refusal elimination, that is a different model to run, not a setting to tune on this one.

Dual RTX 3090 deployment

Validated host profile: 2×RTX 3090 24 GB, PCIe without NVLink or P2P, vLLM TP2, BF16 activations, FP8 E4M3 KV cache, MTP with three draft tokens, and --max-num-batched-tokens 8192.

Memory and capacity

The checkpoint has the same size and footprint as the sibling quantization, since the W8A16 policy and shard layout are identical.

--max-model-len caps one request's input plus generated tokens. The KV pool is a shared runtime allocation across all live requests, not a per-lane reservation: --max-num-seqs permits that many active sequences but does not provide that many full contexts. Near the 262K limit, reserve output-token headroom and expect only one full-length request at a time.

The open-source checkpoint is native at 262,144 tokens. Qwen documents extension to 1,000,000 tokens via static YaRN, but this deployment's 266,537-token KV pool cannot hold a 1M-token request even with YaRN enabled; the 1M path is not validated or capacity-supported here. Follow the official YaRN configuration only with a runtime and memory plan sized for the requested context.

Measured serving performance

Measured on 2026-08-15 with llama-benchy 0.4.0 against the sibling deployment (identical architecture, quantization profile, and serving stack — see the relationship section above): one concurrent request, exact generation lengths (min_tokens + ignore_eos), cold prefix cache (unique corpus text, cache_prompt=false), no conversation depth, one run per cell. The heretic checkpoint was verified locally to perform identically to that deployment; a formal run on this exact checkpoint is listed under Pending verification.

Generation throughput is the per-request decode rate with MTP enabled and varies by workload, since draft acceptance is prompt-dependent. Prefill and TTFT are governed by the 8,192-token batched-token cap, which chunks long prompts and interleaves the MTP draft pass; TTFT grows with prompt length, roughly linearly past 32K.

Multi-stream throughput is not yet published: concurrent requests can intermittently crash the engine on this pin, so parallel benchmarks will be added once a vLLM release containing vllm#50021 is available (see the concurrency note under Serving).

Quantization fidelity

Two different fidelity questions apply to this checkpoint, and they should not be conflated:

1. What the W8A16 step costs (quantization error). The recipe is identical to the sibling card, where it measured against the Qwen3.8-27B BF16 teacher:

The KLD result measures checkpoint-weight drift over 4,563 teacher-forced positions. It is not a functional quality score; tool use, JSON-schema compliance, coding, multimodal quality, and long-context recall require separate behavioral evaluation. The quantization policy is per-tensor and data-free, so the expected error class for this checkpoint is the same; a formal measurement against its own (heretic) BF16 teacher is pending (see Pending verification).

2. What the model deviates from stock Qwen3.8-27B by (intended). That is the abliteration in the section above — KL 0.0535 and 0/100 refusals, measured and documented by the base author. It is a property of the base weights, not of the quantization step.

Checkpoint profile

Why W8A16 on Ampere

RTX 3090 GPUs are Ampere sm_86. They do not provide the native FP8 tensor-core execution path available on newer GPU generations. In this deployment, FP8 weight-only and INT8 W8A16 checkpoints both reach Marlin-family weight-only kernels that reconstruct weights for 16-bit GEMMs.

W8A16 is therefore the better fidelity default here: it preserves eight integer bits with group-wise scaling while retaining the same practical decode class as FP8-Marlin on these GPUs.

Why not AutoRound?

This checkpoint uses llm-compressor's QuantizationModifier, not AutoRound. At W8A16, AutoRound's installed RTN configuration disables optimized RTN by default for efficiency, and the measured KLD is already below 9e-4 nats/token. Re-encoding these weights with AutoRound would add a new export and serving path without evidence of a meaningful quality gain.

Quantization design

Architecture audit

The BF16 source was checked before quantization against the Qwen3.6 structural contract:

• —64 language layers: 48 Gated DeltaNet linear-attention layers and 16 full-attention layers.
• —Hidden size 5,120; intermediate size 17,408; padded vocabulary 248,320.
• —One MTP layer with the same 15 top-level mtp.* tensors.
• —Stable GDN projection names: in_proj_a, in_proj_b, in_proj_qkv, in_proj_z, and out_proj.
• —The only relevant rename was non-Linear convNd → conv1d, outside the quantization target.

The pre-quantization structural audit completed successfully before any weights were modified.

Recipe

The result contains 400 quantized Linear GEMMs: 192 MLP projections, 64 full-attention projections, and 144 dense GDN projections.

# recipe.yaml
default_stage:
  default_modifiers:
    QuantizationModifier:
      targets: [Linear]
      ignore:
        - lm_head
        - re:.*visual.*
        - re:.*mtp.*
        - re:.*linear_attn[.]in_proj_a$
        - re:.*linear_attn[.]in_proj_b$
      scheme: W8A16

Reproducing / building a sibling model

This checkpoint is fully reproducible from trohrbaugh/Qwen3.8-27B-heretic-ara with this family's W8A16 toolkit (../tools/, sibling of this repo), and the same pipeline builds a release from any other Qwen3.5-family base. The Heretic build is the abliterated counterpart of the reference release (qwen3.8-27b-int8-w8a16-mtp): identical recipe, only the base differs. Entry points: `AGENTS.md` (agent-oriented), the repo skill `skills/w8a16-quantize/SKILL.md`, and `../tools/README.md` (full pipeline).

source ../tools/setup_venv.sh   # resolve the family venv -> $W8A16_PYTHON
PY=$W8A16_PYTHON
SOURCE_MODEL=trohrbaugh/Qwen3.8-27B-heretic-ara SAVE_DIR=<out> $PY ../tools/quantize_qwen38_w8a16.py
SOURCE_MODEL=trohrbaugh/Qwen3.8-27B-heretic-ara SAVE_DIR=<out> $PY ../tools/finish_qwen38_w8a16.py
DEST_DIR=<out> SOURCE_MODEL=trohrbaugh/Qwen3.8-27B-heretic-ara $PY ../tools/audit_structure.py
DEST_DIR=<out> SOURCE_MODEL=trohrbaugh/Qwen3.8-27B-heretic-ara EXPECTED_MTP_SHA256=1d8268aa85ace093a561e3e7b63b9d390dac1cd55a90cd55b5ec509c3c9da9fe \
  $PY ../tools/audit_mtp.py

Order: quantize → finish → (reshard) → audit → publish. Both audits derive their expectations from SOURCE_MODEL (no hardcoded counts), so they double as the release gate for a new sibling model. For a heretic base the MTP is read from the base's model-auxiliary.safetensors.

MTP preservation

The MTP head is the exact BF16 MTP from the original Qwen3.8-27B — never touched by the abliteration (which targets only attn.o_proj/mlp.down_proj on layers 26–56) and never re-quantized:

• —15 tensors in model_mtp.safetensors (849,400,392 bytes).
• —sha256 1d8268aa85ace093a561e3e7b63b9d390dac1cd55a90cd55b5ec509c3c9da9fe — identical to the MTP bytes in the Qwen3.8-27B source and in the base heretic repository's model-auxiliary.safetensors.
• —All 15 tensors verified bfloat16, shape-equal, and bit-equal to the Qwen source (strict audit, not a count check).

KV-cache rationale: E4M3 versus E5M2

--kv-cache-dtype controls runtime attention-cache storage. It does not alter this checkpoint's INT8 weights. E4M3 and E5M2 both consume one byte per cached value, so this decision affects numerical behavior—not memory capacity.

E4M3 is the default because attention K/V values generally benefit more from its additional mantissa bit than from E5M2's extreme range.

[!WARNING] Runtime compatibility is verified; long-context KV accuracy is not. The checkpoint declares kv_cache_scheme: null, so it contains no dataset-calibrated K/V scales. The KLD measurement above uses no cache, and no saturation probe was performed.

E5M2 remains the fallback if measured ranges or a future vLLM build expose E4M3 saturation or kernel incompatibility.

Serving

GPU interconnect profiles

The validated command below targets stock-driver dual-GPU systems without a working CUDA peer-to-peer path. Consumer Ampere P2P availability depends on the driver and host configuration; the physical PCIe topology alone does not establish that peer memory access works.

Verify the second profile with a CUDA peer-access test such as p2pBandwidthLatencyTest, not only nvidia-smi topo -m. Remove the two workarounds together: enabling vLLM's custom all-reduce while NCCL peer access remains disabled is not the intended profile.

Recommended vLLM command

The reference llama-swap/podman deployment, minus the container plumbing, with two deliberately conservative flags: prefix caching is off by default (the reference enables it after applying the safety patch below) and the sequence cap is 2 (the reference runs 4). Rationale in the notes after the command.

export NCCL_P2P_DISABLE=1
export NCCL_CUMEM_ENABLE=0
export VLLM_WORKER_MULTIPROC_METHOD=spawn
export OMP_NUM_THREADS=1
export VLLM_USE_FLASHINFER_SAMPLER=0
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True,max_split_size_mb:512

vllm serve lued/Qwen3.8-27B-heretic-ara-INT8-W8A16-MTP \
  --served-model-name qwen3.8-27b-heretic-ara-int8-w8a16-mtp \
  --tensor-parallel-size 2 \
  --pipeline-parallel-size 1 \
  --dtype bfloat16 \
  --performance-mode balanced \
  --max-model-len 262144 \
  --gpu-memory-utilization 0.92 \
  --max-num-seqs 2 \
  --max-num-batched-tokens 8192 \
  --kv-cache-dtype fp8_e4m3 \
  --no-enable-prefix-caching \
  --enable-chunked-prefill \
  --mamba-cache-mode align \
  --prefix-match-unit 16 \
  --enable-prompt-tokens-details \
  --enable-per-request-metrics \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder \
  --enable-auto-tool-choice \
  --disable-custom-all-reduce \
  --trust-remote-code \
  --default-chat-template-kwargs '{"enable_thinking":true,"preserve_thinking":true}' \
  --override-generation-config '{"temperature":1.0,"top_p":0.95,"top_k":20,"min_p":0.0,"repetition_penalty":1.0,"presence_penalty":0.0}' \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

NCCL_P2P_DISABLE=1 avoids NCCL initialization stalls on dual consumer GPUs without NVLink. --mamba-cache-mode align is required by the Qwen3.8 MTP/GDN serving path. Three speculative tokens is the measured default.

Prefix caching is off by default. On this hybrid GDN + MTP architecture, prefix caching without the 9-line source patch from vllm#48375 can corrupt recurrent-state KV, which surfaces as wrong tool-call or long-context output rather than an error. The reference deployment applies that patch at container start (along with this repository's chat_template.jinja via --chat-template), which is what lets it run with --enable-prefix-caching. If you apply the same patch, flip this flag back on; the published numbers were measured with a cold cache, so either setting leaves the table valid.

Concurrency is capped at 2, not 4, on this engine pin. Two or more concurrent requests can intermittently crash the engine (asynchronous cudaErrorIllegalAddress in the GDN spec-decode state path; same class as vllm#37431). The targeted fix is vllm#50021, which is open and awaiting its merge as of this writing; a vLLM release containing it should restore --max-num-seqs 4. Until then, a crashed engine drops in-flight requests and the service (llama-swap, podman, or a process manager) restarts it; if that is unacceptable, lower the cap to 1, which is the fully validated configuration on this pin.

MTP acceptance

MTP acceptance is workload-, sampling-, and draft-depth-dependent; it is not directly comparable across different models or benchmark prompts. With the sibling checkpoint and three draft tokens on the vLLM pin above, per-window draft acceptance ranged from 42.1% to 92.4% across ten-second engine windows (including warmup), and overall draft acceptance was 65.5% (accepted/drafted) in the single-stream 8K/1,024 run. The heretic checkpoint shares the identical BF16 MTP head and LM architecture; a formal acceptance run on this exact checkpoint is listed under Pending verification.

The MTP checkpoint and lm_head are already BF16. Requantizing them cannot improve alignment. Increasing num_speculative_tokens adds progressively less accurate draft positions and normally lowers the reported average acceptance rate; lowering it can raise that percentage while reducing useful speculative work. The serving decision must therefore use end-to-end output throughput, not acceptance alone. Keep num_speculative_tokens=3 unless a controlled throughput test on the intended workload shows a better setting.

Operational notes

Files and provenance

Provenance chain:

Qwen/Qwen3.8-27B  (commit 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0, 18×BF16 shards, 55.56 GB)
   │  Heretic ARA v1.2.0+custom, 60-trial study in full BF16, trial 37
   │  (refusals 99→0/100, KL 0.0535; MTP untouched, shipped as model-auxiliary.safetensors)
   ▼
heretic-org/Qwen3.8-27B-heretic-ara  (6×BF16 shards, 54.71 GB)
   │  llm-compressor 0.12 QuantizationModifier, W8A16 RTN, group-128 symmetric, data-free
   │  400 GEMMs packed; lm_head / vision / MTP / GDN gates preserved BF16
   ▼
this repository  (6×W8A16 shards + model_mtp.safetensors, 31.6 GB)

Pending verification

Items below are carried over as follow-ups; they do not block this release, which mirrors the sibling card's validated profile.

1. 1.Formal llama-benchy run on this exact checkpoint (the serving-performance table above is from the sibling deployment; local verification found the two perform identically).
2. 2.W8A16-versus-heretic-BF16 KLD on the fixed teacher-forced prompt suite (the expected error class matches the sibling's 0.000894; the policy is identical).
3. 3.Formal MTP draft-acceptance run on this exact checkpoint (sibling measured 65.5% on the single-stream 8K/1K run).
4. 4.Optional: live A/B serve diff against the sibling model on matched prompts and seeds.

Acknowledgements and license

This repository repackages numerical weights derived from heretic-org/Qwen3.8-27B-heretic-ara (abliteration by Tim Rohrbaugh via the Heretic tool) and Qwen/Qwen3.8-27B. It does not claim authorship of the base model, the abliteration, architecture, training data, or upstream evaluations. Header styling and the Qwen logo follow the official Qwen repositories.

Apache-2.0 for this packaging. Respect all upstream model terms and attribution requirements.