Modelpublic

rdtand/Qwen3.6-27B-PrismaQuant-Heretic-5.25bit-vllm

sourceHugging Faceapache-2.0updated 5mo agoView on Hugging Face

1likes20downloads

Model Card

Qwen3.6-27B PrismaQuant Heretic 5.25bit vLLM

This is a PrismaQuant mixed-precision compressed-tensors export of `llmfan46/Qwen3.6-27B-uncensored-heretic-v2` for vLLM serving.

Quantization

—Target: allocator kneedle point at 5.25 bpp
—Achieved allocator bpp: 5.249 over quantizable body Linears
—Formats: NVFP4, FP8_E4M3, BF16 fallback
—Body assignment counts: 203 NVFP4, 84 FP8_E4M3, 17 BF16
—Export histogram: 285 NVFP4 production-cache Linears, 178 FP8_E4M3 production-cache Linears, 33 BF16 Linears
—Visual components: BF16 passthrough
—Source dtype: BF16

Validation

Validated locally with vLLM load and greedy generation:

—Eager mode: passed
—Graph/compiled CUDA capture mode: passed
—Prompt: The capital of France is
—Output prefix: Paris.

The serving run selected:

—CutlassFP8ScaledMMLinearKernel for compressed-tensors FP8
—FlashInferCutlassNvFp4LinearKernel for NVFP4 GEMM

No KL quality claim is made in this model card.

Serving

bash

vllm serve rdtand/Qwen3.6-27B-PrismaQuant-Heretic-5.25bit-vllm \
  --quantization compressed-tensors \
  --trust-remote-code

For a low-context smoke:

bash

python -m vllm.entrypoints.openai.api_server \
  --model rdtand/Qwen3.6-27B-PrismaQuant-Heretic-5.25bit-vllm \
  --quantization compressed-tensors \
  --trust-remote-code \
  --max-model-len 528

Provenance

—Source repo: llmfan46/Qwen3.6-27B-uncensored-heretic-v2
—Source revision: bf309da6e5744ac57240ae21d31eaf107734c585
—Quantizer: PrismaQuant
—Local export path: /home/rob/dq-runs/qwen36-27b-heretic-v2-kneedle-5p25-materialize-20260515T220544Z/exported