willteraflop/Qwen3-1.7B-W8A8

Qwen3-1.7B-W8A8

This is a compressed version of Qwen/Qwen3-1.7B using llm-compressor with the following optimizations:

• —8-bit weight quantization using GPTQ
• —8-bit activation quantization
• —SmoothQuant pre-processing

Model Details

• —Original Model: Qwen/Qwen3-1.7B
• —Quantization Method: GPTQ + SmoothQuant (W8A8)
• —Compression Libraries: llm-compressor
• —Calibration Dataset: open_platypus (512 samples)
• —Optimized For: Inference with vLLM

Usage with vLLM

pip install vllm
from vllm import LLM

# Load the compressed model
model = LLM(
    "willteraflop/Qwen3-1.7B-W8A8",
    quantization="awq",  # Use AWQ for 8-bit weights in vLLM
)

# Generate text
outputs = model.generate(
    prompts=["Tell me about artificial intelligence"],
    max_tokens=512,
    temperature=0.7
)
print(outputs[0].text)