quimmedes/Agnes-3.0-Flash-XYZ-GGUF

Agnes-3.0-Flash Preview - GGUF quantizations

GGUF quantizations of Agnes-AI/Agnes-3.0-Flash (the open-weight Preview checkpoint, Apache-2.0), quantized with imatrix-weighted llama-quantize.

These files need a patched build. Agnes has an extra parallel SwiGLU branch in every layer (ffn_gate_par / ffn_up_par / ffn_down_par) that stock llama.cpp does not know about. The fork cafe-llama.cpp adds those tensors to the qwen35 architecture (plus the recurrent/full attention layer map and the delta-attention V-head reorder). Loading these GGUFs on an unpatched build will fail or mis-load.

Main ladder

Oficial llama.cpp is not supporting this yet, you have to use cafe-llama.cpp https://github.com/quimmedes/cafe-llama.cpp

Total: 212.9 GiB - all of it uploaded.

Other quantizations (extra/)

Total: 105.2 GiB. Same model and same runtime as the ladder above; different tensor-type allocation (more precision kept in attention, more compression in the FFN).

How to run

# 24 GB card, long context, MTP speculative decoding
llama-server -m Agnes-3.0-Flash-Q4.5-v4-XYZ.gguf \
  --host 127.0.0.1 --port 8080 -ngl 20 -nr -c 32768 -np 1 \
  -ctk q8_0 -ctv q8_0 -fa on --spec-type draft-mtp

• —-nr (no repack) keeps weights memory-mapped, which matters at 33B.
• —On a 24 GB card -ngl around 20 fits; a full offload only leaves room for a small context.
• —The MTP block is included (nextn, 1 layer), so --spec-type draft-mtp works.
• —Base card's recommended sampling: temp 1.0, top_p 0.95, top_k 20.

What this checkpoint actually is (findings, with the method)

While converting this model I compared it against Qwen/Qwen3.8-27B, because both report the same architecture (qwen35). The bytes say the two are the same blueprint. Method: sha256 over HTTPS for the small files, and byte-comparison of tensor ranges (safetensors data_offsets header, Range: requests) for the weights.

Byte-identical: | item | result | |---|---| | vocab.json | sha256 ce99b4cb2983 == Qwen3.8-27B | | merges.txt | sha256 a9d356d7bdf1 == Qwen3.8-27B | | chat_template.jinja | identical (same tool-call format, same reasoning_effort levels) | | generation_config.json | identical | | architecture hyper-parameters | hidden 5120, 24 q / 4 kv heads, headdim 256, FFN 17408, delta-rule 16 key / 48 value heads, conv kernel 4, `partialrotaryfactor` 0.25, mrope `[11,11,10]` interleaved, theta 1e7, vocab 248320, MTP 1 layer, swish output gate, `tiewordembeddings=false` | | vision tower | same config (only the `modeltype label differs: agnesvision` vs `qwen35), **same weights** (compared attn.proj of block 0: 2,654,208 bytes equal), same image/video token ids | | embedtokens.weight` | identical (248320 x 5120) | | norms | `inputlayernorm, postattentionlayernorm, qnorm`, `knorm, final norm identical | | SSM constants | Alog`, `dtbias, conv1d, ssm_norm` identical |

Different: | item | Agnes | Qwen3.8-27B | |---|---|---| | layers | 72 (54 delta-rule + 18 global, 3:1) | 64 (48 linear + 16 full) | | parallel FFN | SwiGLU 2048 in every layer | absent | | all projection matrices | retrained | - |

The parameter count agrees: 27B x 72/64 = 30.4B, plus the parallel branch (3 x 2048 x 5120 x 72 ~ 2.3B) = ~32.6B, and this GGUF measures 32.63B. Nothing is unaccounted for.

So: Agnes-3.0-Flash Preview carries the Qwen3.8-27B blueprint - same tokenizer, same chat template, same vision tower (identical weights), same embeddings/norms/SSM constants - with 8 extra delta layers, a new parallel-FFN branch, and retrained projections. Both models are Apache-2.0, so this is entirely legitimate; the reason to write it down is that people know what they are downloading.

How these were made

1. 1.Safetensors -> GGUF with the fork's converter. Changes needed for this model: register the parallel-FFN tensors, record the delta-attention layers as recurrent (otherwise every layer looks like full attention and attn_qkv fails), and map .delta_attn./.global_attn. onto the standard qwen35 names.
2. 2.Imatrix built from 50 chunks of the wikitext-2 raw test split; the bf16 GGUF scores PPL 6.5737 +/- 0.14183 with it.
3. 3.Quantized with llama-quantize --imatrix, per-tensor types: attention and SSM tensors stay at higher precision, the FFN carries the compression. The parallel FFN follows the FFN of its block.

Practical consequences

• —It is dense, not a MoE. No experts: 0 of the 1521 checkpoint tensors contain "expert", and the config has no MoE keys (intermediate_size 17408 plus a parallel 2048 SwiGLU). All ~33B parameters are read for every token, so expect dense-model speed at equal quantization, not "3B active" speed. The base card's benchmark table compares against models labelled "X active"; Agnes is not one.
• —Long context is comparatively cheap. Only 18 of the 72 layers hold a KV cache that grows with context (the 54 delta-rule layers keep a fixed-size recurrent state). At 128k with q8_0 KV that is about 5 GB (18 x 2 x 4 KV heads x 256 dim x 131072 x 1.06 B), which is why a 24 GB card can hold the Q4.5 weights and a long context.
• —Vision is not in these files. The text tower and the MTP layer are; the projector is separate. Because the vision tower weights are byte-identical to Qwen3.8-27B's, the mmproj-Qwen3.8-27B-bf16.gguf from quimmedes/Qwen3.8-27B-XYZ is expected to work - but a full image encode could not be run on this machine (the vision graph needed more RAM than was free), so treat that as unverified.
• —These are quantizations of a preview checkpoint; the production Agnes model has a different configuration and its benchmarks do not apply here.

Credits

• —Base model: Agnes-AI/Agnes-3.0-Flash (Apache-2.0).
• —Runtime: quimmedes/cafe-llama.cpp - required for the parallel-FFN tensors.