CoolFace
Modelpublic

selimaktas/Qwen3-14B-FP8-MinMax

sourceHugging Faceapache-2.0updated 11mo agoView on Hugging Face
0likes10downloads
Model Card

Qwen3-14B-FP8-MinMax

This is an FP8-quantized version of Qwen3-14B optimized for efficient inference.

Quantization Details

  • —Method: FP8 W8A8 (weights + activations)
  • —Observer: MinMax
  • —Weights: Symmetric static per-channel
  • —Activations: Symmetric dynamic per-token
  • —Library: llm-compressor
  • —Targets: Linear layers only (lm_head excluded)

Performance

Achieves ~97% retention on GSM8K benchmarks with 50% memory reduction and 2x throughput improvement.

Usage

python
from vllm import LLM, SamplingParams
from transformers import AutoTokenizer

model_id = "selimaktas/Qwen3-14B-FP8-MinMax"
tokenizer = AutoTokenizer.from_pretrained(model_id)

# Initialize vLLM
llm = LLM(model=model_id, tensor_parallel_size=1)

# Generate
messages = [{"role": "user", "content": "Explain quantum computing"}]
prompts = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
sampling_params = SamplingParams(temperature=0.6, top_p=0.95, max_tokens=256)

outputs = llm.generate(prompts, sampling_params)
print(outputs[0].outputs[0].text)

Benchmarking

bash
lm_eval --model vllm \
  --model_args pretrained=selimaktas/Qwen3-14B-FP8-MinMax,dtype=auto,add_bos_token=True \
  --tasks gsm8k --num_fewshot 5 --batch_size auto

Requirements

  • —vLLM >= 0.5.0
  • —Transformers >= 4.51.0
  • —CUDA-capable GPU with FP8 support (H100, L40S, etc.)