Hyun9junn/EXAONE-4.5-33B-FlatQuant-W4A16

EXAONE 4.5 33B FlatQuant W4A16

A vLLM-ready FlatQuant W4A16 checkpoint derived from LGAI-EXAONE/EXAONE-4.5-33B. Weights use compressed-tensors packed INT4 storage and the FlatQuant transforms are applied by the accompanying vLLM plugin.

Quick start

python -m venv .venv
source .venv/bin/activate
pip install "vllm==0.24.0"
pip install "git+https://github.com/SqueezeBits/FlatQuant-EXAONE4.5.git#subdirectory=vllm_plugin"

vllm serve \
  Hyun9junn/EXAONE-4.5-33B-FlatQuant-W4A16 \
  --dtype bfloat16 \
  --tensor-parallel-size 1

The server exposes an OpenAI-compatible API at http://localhost:8000/v1.

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Hyun9junn/EXAONE-4.5-33B-FlatQuant-W4A16",
    "messages": [{"role": "user", "content": "대한민국의 수도는 어디인가요?"}],
    "temperature": 0,
    "max_tokens": 128
  }'

Quantization

Evaluation

All results below were measured with vLLM on EXAONE-4.5-33B. The comparison uses the BF16 base model, the official AWQ checkpoint, and this FlatQuant W4A16 checkpoint.

Perplexity

MMLU-Pro

Five-shot evaluation with lm-eval.

MMMU-Pro

VLM evaluation with lmms-eval.

Limitations

• —vllm==0.24.0 and the FlatQuant vLLM plugin are required.
• —Tensor parallelism is currently limited to TP=1.
• —This checkpoint is intended for vLLM; stock Transformers does not apply the required FlatQuant transforms.

Evaluation results and reproduction commands are available in the SqueezeBits/FlatQuant-EXAONE4.5 repository.

License and attribution

This checkpoint is derived from EXAONE 4.5 and is distributed under the EXAONE model license included in LICENSE. The implementation builds on the MIT-licensed FlatQuant project.