urakozz/Agnes-3.0-Flash-W4A16-AutoRound-GPTQ

Agnes-3.0-Flash Preview W4A16 (AutoRound GPTQ)

4-bit weight-only (W4A16) AutoRound GPTQ quantization of Agnes-AI/Agnes-3.0-Flash - the 33B open-weights Preview checkpoint: multimodal (text/image/video), hybrid delta + global attention, 262 144-token context. 22.9 GB on disk against ~66 GB for the BF16 original.

This is an unofficial, community-produced quantization. All credit for the base model goes to Agnes AI - see their model card for benchmarks and citation. Note their version clarification: these are the open-weight Preview weights, not the production/API Agnes 3.0 Flash checkpoint.

[!IMPORTANT] A build containing [vLLM PR #57003](https://github.com/vllm-project/vllm/pull/57003) is required to serve this checkpoint with vLLM. Current released builds do not include the Agnes architecture. The tested vLLM path is text-only with --language-model-only and --trust-remote-code.

Model Details

Quantization Details

Quantized with Intel AutoRound v0.16.0 on Intel Arc Pro B70 (XPU) hardware.

What is and isn't in INT4:

AutoRound excluded the in_proj_a/in_proj_b pairs on its own: they are recorded as -: dynamic keys in quantization_config. The vision tower and the MTP block were never entered for quantization and carry no such keys, so a runtime that builds every nn.Linear as GPTQ straight from quantization_config has to skip them explicitly.

Reproduce with (transformers==5.13.1, since 5.17 renamed LAYER_TYPE_CACHE_MAPPING to DYNAMIC_LAYER_TYPE_MAPPING and the bundled remote code expects the old name):

auto-round \
  --model Agnes-AI/Agnes-3.0-Flash \
  --scheme W4A16 \
  --group_size 64 \
  --iters 600 \
  --dataset "opencode-instruct" \
  --format auto_gptq \
  --low_gpu_mem_usage \
  --output_dir <output_dir>

Usage

vLLM (text only)

Use a vLLM build that includes PR #57003. Until the PR is merged and released, stock vLLM builds will not recognize AgnesForConditionalGeneration.

vllm serve urakozz/Agnes-3.0-Flash-W4A16-AutoRound-GPTQ \
  --trust-remote-code \
  --language-model-only \
  --dtype bfloat16 \
  --max-model-len 2048

The text-only AutoRound GPTQ checkpoint was smoke-tested on Intel Arc Pro B70 with XPU Graph. Image and video serving have not been evaluated. The text-only MTP-2 benchmark below passed its coherence test.

vLLM MTP-2 benchmark

Enable two MTP draft tokens on the vLLM server with --speculative-config '{"method":"mtp","num_speculative_tokens":2}'. The 4096-token prompt and 256-token generation require a --max-model-len above 4352; increase the 2048-token limit in the basic example above before running this benchmark. On the server host, run:

uvx llama-benchy --base-url http://0.0.0.0:8000/v1 \
  --model urakozz/Agnes-3.0-Flash-W4A16-AutoRound-GPTQ \
  --pp 4096 --tg 256 --concurrency 1 --depth 1 \
  --no-cache --exact-tg --latency-mode generation

Measured on 2026-09-15 at 15:58:38 with llama-benchy 0.4.0 and three runs per test. Coherence passed; average generation latency was 195.95 ms.

Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id, subfolder = "urakozz/Agnes-3.0-Flash-W4A16-AutoRound-GPTQ", "Agnes-3.0-Flash-w4g64"
tok = AutoTokenizer.from_pretrained(model_id, subfolder=subfolder, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id, subfolder=subfolder, trust_remote_code=True, device_map="auto"
)

msgs = [{"role": "user", "content": "Explain gated delta attention in three sentences."}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(ids, max_new_tokens=512, temperature=1.0, top_p=0.95, top_k=20)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))

Images and video go through the bundled processor (AutoProcessor, also remote code) exactly as in the base model card - the vision tower is untouched by quantization.

Reasoning effort

The shipped chat_template.jinja accepts xhigh (default), medium, low, plus enable_thinking=False:

ids = tok.apply_chat_template(msgs, add_generation_prompt=True, reasoning_effort="medium",
                              return_tensors="pt")

reasoning_effort="high" raises - the base card's wording predates this template.

Sampling

temperature=1.0, top_p=0.95, top_k=20, max_tokens ≥ 2000, per the checkpoint's generation_config.json.

Limitations

• —Requires trust_remote_code=True (custom architecture, inherited from the base model).
• —vLLM serving requires a build containing PR #57003; text-only XPU Graph serving has been smoke-tested and MTP-2 benchmarked. Image and video serving have not been evaluated. SGLang serving has not been tested with this checkpoint.
• —Only the text decoder is in INT4 embeddings, lm_head, vision tower and MTP stay BF16, so memory savings are ~3× rather than ~4×.
• —No accuracy evaluation has been run on this quantization yet; treat quality as unverified.

License

Apache 2.0, inherited from the base model.