GotoAI-Inc/EXAONE-4.5-33B-W8A16

EXAONE-4.5-33B-W8A16

Int8 weight-only quantization of LGAI-EXAONE/EXAONE-4.5-33B, in compressed-tensors format for vLLM. 38.23 GB, down from 68.70 GB.

This is the fidelity-first build. Int8 round-to-nearest stays far closer to the bfloat16 weights than int4 does, at ~1.7x the footprint of the int4 W4A16 sibling (22.75 GB). If you are targeting a 32 GB card, use that one; use this one when you have the VRAM and want the least quality loss quantization can give without calibration. It is a 48 GB-card model — see Fitting the card, because this model's KV cache is not cheap and the two compete for the same budget.

Unofficial and unaffiliated with LG AI Research. All model capabilities, evaluations and limitations belong to the original model card — see the base model for those.

## ⚠️ Non-commercial license The base model is released under the EXAONE AI Model License Agreement 1.2 – NC, and this derivative inherits it in full. That license permits use solely for research and educational purposes. Commercial use of the model, of derivatives, or of their outputs is expressly prohibited without a separate written agreement with LG Management Development Institute. It also forbids using the model or its outputs to develop or improve competing models. This is not the permissive licensing you may expect from other quantized checkpoints. Read LICENSE in full before you download, and do not deploy this in a product.

What was changed

Weights were quantized from bfloat16 to int8, group size 128, symmetric, weight-only (activations stay 16-bit) using llmcompressor.model_free_ptq. No calibration data was used and the model was never loaded — the quantizer operates directly on the safetensors. Architecture, tokenizer, chat template and processor configs are the vendor's, unmodified.

448 Linear modules were converted, covering 82.2% of the output's bytes:

Of the quantized share, 30.95 GB is packed int8 and 0.48 GB is bf16 group scales — the same 448 modules and the same scale count as the W4A16 sibling, so the int8 payload is almost exactly twice the int4 one and nothing else differs.

The model is unusually MLP-heavy — intermediate_size is 27392 against a hidden_size of 5120 — so 78% of the source checkpoint is MLP weight and quantizing it does most of the work. The flip side is that the 6.77 GB left at bfloat16 (vision tower, both embedding tables, MTP head) is a fixed floor that no weight scheme reduces: it is 17.7% of this build against 29.8% of the W4A16 one, which is why int8 costs only ~1.7x rather than 2x.

Left at bfloat16:

• —*`visual.** — vLLM builds multimodal towers with quant_config=None`, so a checkpoint carrying quantized vision weights cannot be loaded.
• —*`mtp.** — the multi-token-prediction speculator head (numnextnpredict_layers: 1`), loaded through vLLM's speculative-decoding path rather than the main stack. The base model card's own serving command enables it, so leaving it at full precision keeps that path working as the vendor intends.
• —`lm_head` + `embed_tokens` — precision-sensitive, and untied here (both tensors are present, 1.57 GB each).

Usage

Requires vLLM >= 0.25.1, where Exaone4_5_ForConditionalGeneration is registered. No nightly build needed.

vllm serve GotoAI-Inc/EXAONE-4.5-33B-W8A16 \
  --served-model-name EXAONE-4.5-33B-W8A16 \
  --max-model-len 65536 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice --tool-call-parser hermes \
  --limit-mm-per-prompt '{"image": 64}' \
  --speculative_config '{"method": "mtp", "num_speculative_tokens": 3}'

Do not pass --quantization; compressed-tensors is detected from config.json. The int8 W8A16 scheme uses Marlin kernels and runs on compute capability 7.5 and above.

The parsers are borrowed, and that is deliberate. vLLM ships no exaone tool-call or reasoning parser. The base model card specifies --reasoning-parser qwen3 (EXAONE frames thinking with <think> tags, which that parser handles) and --tool-call-parser hermes. Use those exact names.

• —MTP speculative decoding works off the head already in this checkpoint — no draft model to download. The base card uses num_speculative_tokens: 3. Not smoke-tested here.
• —`--language-model-only` frees the 2.57 GB vision tower (vLLM skips tower weights when every modality limit is zero) plus the multimodal profiling headroom, at the cost of image and video input. It matters more on this build than on the W4A16 one — see below.

Thinking

The chat template defaults enable_thinking to true, so this model thinks unless told otherwise:

{"chat_template_kwargs": {"enable_thinking": false}}   // emit an empty <think></think> and answer directly

With thinking on, --reasoning-parser qwen3 splits the <think>…</think> block into reasoning_content.

Fitting the card

64 layers: 48 sliding-attention (window 4096) and 16 full attention, all with 8 KV heads at head_dim 128. The sliding layers are bounded by the window at ~0.8 GB per sequence regardless of context; the 16 full layers cost ~64 KB/token:

Unlike most builds in this series, the KV cache here is expensive enough to set the ceiling by itself. A 48 GB card is comfortable to 64k and reaches 128k only with a high --gpu-memory-utilization and little room for activations and CUDA graphs; the full 262144-token context wants 80 GB. A 40 GB card needs --language-model-only (dropping the weights to 35.66 GB) and a context near 32k. Below that, use the W4A16 sibling — the 15.5 GB it saves buys roughly 240k tokens of cache.

Context beyond 8192 uses llama3 RoPE scaling (factor: 16.0), configured by the vendor and left untouched here. This is arithmetic from config.json, not a measured deployment.

Reproducing this checkpoint

Built with llm-quantizer:

./llmq.py run --profile exaone-4.5-33b --scheme W8A16

which re-shards the source — it ships as 2 shards, the larger 49.74 GB, which no consumer GPU can hold — into 18 pieces of ~4 GB, then:

# llmcompressor==0.13.1a20260814, compressed-tensors==0.18.1a20260818,
# transformers==5.15.1, torch==2.13.0
from llmcompressor import model_free_ptq

model_free_ptq(
    model_stub="EXAONE-4.5-33B-resharded",
    save_directory="EXAONE-4.5-33B-W8A16",
    scheme="W8A16",
    ignore=["re:.*visual.*", "re:.*mtp.*", "lm_head", "re:.*embed_tokens.*"],
    device="cuda:0",
)

A job holds one shard at a time, so after re-sharding the build peaks at a few GB of VRAM rather than the model size — the quantization step itself took about a minute on one consumer GPU.

Evaluation

No benchmarks have been run. Data-free round-to-nearest quantization degrades quality more than a calibrated (GPTQ/AWQ) or QAT build; how much, for your task, is unmeasured here. Int8 degrades far less than int4 — that is the reason this build exists — but "less" is not "none". Treat the published EXAONE 4.5 numbers as describing the bfloat16 model, not this one.

EXAONE 4.5 is a multilingual model, and this build quantizes every MLP and attention projection without calibration data. Data-free quantization tends to cost most where the weight distribution is least like the majority of the training signal, so if you evaluate anything, evaluate the non-English languages you actually care about rather than assuming English results carry over. That is also the axis on which this build should differ most from the W4A16 one, if you are choosing between them.

License

EXAONE AI Model License Agreement 1.2 – NC, inherited from the base model — the vendor's LICENSE is included unmodified and applies in full to this derivative. The terms that matter most:

• —Research and educational use only (§2.1a). Commercial use of the model, derivatives, or their outputs is prohibited without a separate written license from LG Management Development Institute (§3.1), as is using them to develop or improve competing models.
• —Derivatives must keep the "EXAONE" prefix in their name (§2.1c) — which is why this repository is EXAONE-4.5-33B-W8A16.
• —Redistribution must carry a copy of the Agreement (§2.1d); it travels in LICENSE.
• —Attribution is required when publishing results obtained with the model (§4.3).
• —LG claims no rights in model outputs (§4.2), but §3.1 still restricts what you may do with them commercially.

This summary is not legal advice and is not a substitute for reading LICENSE. "EXAONE" is LG's mark; this repository is not endorsed by or affiliated with LG AI Research.