Modelpublic

adeepv/Qwen3.6-27B-W8A16-Ascend310P

sourceHugging Faceapache-2.0updated 4mo agoView on Hugging Face

3likes122downloads

Model Card

Qwen3.6-27B — W8A16 (int8 weight-only) for Ascend 310P / Atlas 300I Duo

Weight-only int8 (W8A16) quantization of Qwen/Qwen3.6-27B, prepared and verified to run on Huawei Ascend 310P3 (Atlas 300I Duo) via vllm-ascend.

Weights are int8, activations stay FP16. This is not a speed optimization on 310P — it is a memory optimization: halving the weights frees NPU memory for a much larger KV cache, enabling up to 256K context on 4× 310P3 chips (TP=4).

Why this exists

On Ascend 310P the stock paths have two problems for long context:

—FP16 27B weights (~54 GB) leave little room for KV → context is memory-starved.
—Activation-quant schemes (W8A8) crash on 310P (immature QuantBatchMatmulV3 kernel, L0C conflict).

W8A16 (weight-only) sidesteps both: weights ~27 GB (×2 smaller → KV budget ~832K tokens → 256K context), and it does not touch the broken activation-quant kernel path.

	FP16 27B	W8A16 27B (this)
Weights	~54 GB	~27 GB
Context (TP=4)	KV-starved	256K (KV ~832K tokens)
Decode (4× 310P3)	~3.8 t/s	~3.1 t/s
Quality	reference	≈lossless (data-free RTN int8)

How it was made

msmodelslim data-free RTN int8 (ascend modelslim ascendV1 format), via AutoModelForImageTextToText (keeps the multimodal wrapper; visual.* left unquantized). See recipe/ for the conversion notes.

⚠️ Critical: the ND-layout fix (why stock vllm-ascend won't load this)

On 310P, npu_weight_quant_batchmatmul expects weights in ND `[K,N]` layout, not FRACTALNZ (NZ → `task not supported`). Stock `w8a16.py` casts to NZ → fails. This repo ships a 310P scheme override (`patches/methodsinit310.py`) that transposes weights to contiguous ND **without** `maybetrans_nz. **You must mount the patches** (below) — otherwise loading fails with a k-mismatch / task not supported` error.

How to run (vllm-ascend on Ascend 310P, TP=4, 256K)

Requires: 2× Atlas 300I Duo (4× 310P3), driver+firmware on host, P2P enabled for TP=4 (see the Ascend quickstart). Image: quay.io/ascend/vllm-ascend:nightly-main-310p.

bash

# 1) get weights + patches
huggingface-cli download adeepv/Qwen3.6-27B-W8A16-Ascend310P --local-dir /opt/vllm/models/qwen36-w8a16

# 2) run (patches mounted into the image, --enforce-eager is mandatory on 310P)
docker run -d --name vllm-qwen36-w8a16 --network host --restart unless-stopped \
  --device /dev/davinci0 --device /dev/davinci1 --device /dev/davinci2 --device /dev/davinci3 \
  --device /dev/davinci_manager --device /dev/devmm_svm --device /dev/hisi_hdc \
  -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
  -v /usr/local/Ascend/driver/lib64:/usr/local/Ascend/driver/lib64 \
  -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
  -v /etc/ascend_install.info:/etc/ascend_install.info \
  -v /usr/local/dcmi:/usr/local/dcmi \
  -v /opt/vllm/models:/root/.cache/huggingface \
  -v /opt/vllm/models/qwen36-w8a16/patches/methods_init_310.py:/vllm-workspace/vllm-ascend/vllm_ascend/_310p/quantization/methods/__init__.py \
  -v /opt/vllm/models/qwen36-w8a16/patches/gdn_310.py:/vllm-workspace/vllm-ascend/vllm_ascend/_310p/ops/fla/gdn_310.py \
  -v /opt/vllm/models/qwen36-w8a16/patches/chunk_gated_delta_rule.py:/vllm-workspace/vllm-ascend/vllm_ascend/_310p/ops/fla/chunk_gated_delta_rule.py \
  -v /opt/vllm/models/qwen36-w8a16/patches/attention_mask.py:/vllm-workspace/vllm-ascend/vllm_ascend/_310p/attention/attention_mask.py \
  -v /opt/vllm/models/qwen36-w8a16/patches/attention_v1_generic.py:/vllm-workspace/vllm-ascend/vllm_ascend/attention/attention_v1.py \
  -v /opt/vllm/models/qwen36-w8a16/patches/modelslim_config.py:/vllm-workspace/vllm-ascend/vllm_ascend/quantization/modelslim_config.py \
  --entrypoint bash quay.io/ascend/vllm-ascend:nightly-main-310p -c '
    source /usr/local/Ascend/ascend-toolkit/set_env.sh &&
    source /usr/local/Ascend/cann-9.1.0-beta.1/share/info/ascendnpu-ir/bin/set_env.sh &&
    source /usr/local/Ascend/nnal/atb/set_env.sh &&
    exec vllm serve /root/.cache/huggingface/qwen36-w8a16 \
      --host 0.0.0.0 --port 8000 \
      --tensor-parallel-size 4 --dtype float16 --enforce-eager \
      --max-model-len 262144 --gpu-memory-utilization 0.9 \
      --no-enable-prefix-caching --served-model-name qwen3.6-27b \
      --quantization ascend --language-model-only --trust-remote-code \
      --reasoning-parser qwen3'

Notes / gotchas (310P)

—--quantization ascend selects the ascend W8A16 scheme; --enforce-eager is required (aclgraph fails on 310P at AclmdlRICaptureEnd, error 507903).
—For 1 card (TP=2) drop to --tensor-parallel-size 2 and a smaller --max-model-len.
—Reasoning model: answer text comes in reasoning_content; pair with --reasoning-parser qwen3.

License

Apache-2.0, inherited from the base model Qwen/Qwen3.6-27B. This is a quantized derivative; original copyright/attribution notices retained. See LICENSE.