Modelpublic

RishabhSinha/Qwen3.5-9B-FP8-block

sourceHugging Faceapache-2.0updated 1mo agoView on Hugging Face

0likes111downloads

Model Card

Qwen3.5-9B-FP8-block

Model Overview

—Model Architecture: Qwen/Qwen3.5-9B (Qwen3_5ForConditionalGeneration)
—Input: Text / Image
—Output: Text
—Model Optimizations:
—Weight quantization: FP8, block-wise (128x128 blocks)
—Activation quantization: FP8, dynamic per-token groups of 128
—Model size: 14.0 GB (reduced from 19.3 GB in BF16)
—Release Date: 2026-08-27
—Version: 1.0
—Quantized by: RishabhSinha (community contribution)

This model is a block-wise FP8 (W8A8) quantized version of Qwen/Qwen3.5-9B, produced for the community checkpoint drive in vllm-project/llm-compressor#3088 ("FP8 Block" slot for Qwen3.5-9B).

Model Optimizations

This model was obtained by quantizing the weights of Qwen/Qwen3.5-9B to FP8 with 128x128 block scales; activations are quantized dynamically at inference time in per-token groups of 128 (DeepSeek-style block quantization). The scheme is fully data-free — no calibration dataset is used.

Only the linear operators of the transformer decoder blocks are quantized. Following RedHatAI's published recipe for the Qwen3.5 family (RedHatAI/Qwen3.5-9B-FP8-dynamic), the following are kept in BF16: lm_head, embeddings, the vision tower, the hybrid linear-attention (gated delta net) blocks, and the MTP (multi-token prediction) layers, which are carried over unquantized via compressed_tensors.utils.save_mtp_tensors_to_checkpoint.

Quantization was performed with LLM Compressor.

Deployment

Use with vLLM

1.Initialize vLLM server:

bash

vllm serve RishabhSinha/Qwen3.5-9B-FP8-block \
  --reasoning-parser qwen3 \
  --max-model-len 262144

For text-only serving (lower memory), add --language-model-only.

1.Send requests to the server:

python

from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")

outputs = client.chat.completions.create(
    model="RishabhSinha/Qwen3.5-9B-FP8-block",
    messages=[
        {"role": "user", "content": "Explain quantum mechanics clearly and concisely."},
    ],
)
print(outputs.choices[0].message.content)

Use with Transformers

python

from transformers import AutoTokenizer, Qwen3_5ForConditionalGeneration

model_id = "RishabhSinha/Qwen3.5-9B-FP8-block"
model = Qwen3_5ForConditionalGeneration.from_pretrained(
    model_id, dtype="auto", device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_id)

messages = [{"role": "user", "content": "Explain quantum mechanics clearly and concisely."}]
input_ids = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt"
).to(model.device)
output = model.generate(input_ids, max_new_tokens=256)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Creation

This model was created with LLM Compressor using data-free FP8 block quantization (scheme="FP8_BLOCK"), as shown below.

<details> <summary>Creation script</summary>

python

from compressed_tensors.utils import save_mtp_tensors_to_checkpoint
from transformers import AutoProcessor, AutoTokenizer, Qwen3_5ForConditionalGeneration

from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier

MODEL_ID = "Qwen/Qwen3.5-9B"
SAVE_DIR = "Qwen3.5-9B-FP8-block"

IGNORE_LAYERS = [
    "re:.*lm_head",
    "re:.*embed_tokens$",
    "re:.*visual.*",
    "re:.*model.visual.*",
    "re:.*linear_attn.*",
]

model = Qwen3_5ForConditionalGeneration.from_pretrained(
    MODEL_ID, dtype="auto", device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
processor = AutoProcessor.from_pretrained(MODEL_ID)

recipe = QuantizationModifier(
    targets="Linear",
    scheme="FP8_BLOCK",
    ignore=IGNORE_LAYERS,
)

oneshot(model=model, recipe=recipe)

model.save_pretrained(SAVE_DIR, save_compressed=True)
tokenizer.save_pretrained(SAVE_DIR)
processor.save_pretrained(SAVE_DIR)
save_mtp_tensors_to_checkpoint(source_model=MODEL_ID, dest_dir=SAVE_DIR)

</details>

<details> <summary>Package versions</summary>

—llm-compressor==0.13.1.dev51+g50d0a1c75 (main)
—compressed-tensors==0.18.1.dev21+g8c0fa69 (main)
—transformers==5.16.1
—torch==2.11.0+cu128

Hardware: 1x NVIDIA L40S (48 GB).

</details>

Evaluation

Sanity checks only so far — lm-eval results pending.

The checkpoint was verified two ways on an NVIDIA L40S:

1.vLLM 0.24.0 loaded the checkpoint natively, selecting the block-FP8 execution path (TritonFp8BlockScaledMMKernel for CompressedTensorsW8A8Fp8), and produced a coherent greedy generation:

text

prompt:     "The capital of France is"
completion: " Paris.\nThe capital of France is Paris.\n..." (greedy, 32 tokens)

1.transformers + compressed-tensors reloaded the checkpoint fresh from disk; greedy generations are coherent and logits are finite:

text

PROMPT: 'The capital of France is'
OUTPUT: 'The capital of France is Paris.\nThe capital of France is Paris.\n...'

PROMPT: 'def fibonacci(n):'
OUTPUT: 'def fibonacci(n):\n    if n <= 0:\n        return []\n    elif n == 1:\n        return [0]\n    elif n == '

PROMPT: 'Water boils at'
OUTPUT: 'Water boils at 100°C at sea level. At what temperature does water boil at the top of Mount Everest, where the atmospheric pressure is approximately 0.3'

Formal benchmark results (lm-evaluation-harness) have not yet been run for this checkpoint. If you run them, contributions to this model card are welcome.