s-batman/Agents-A1-NVFP4-MTP-GGUF

Agents-A1 NVFP4/MXFP4 GGUF with MTP

Quantized GGUF files for InternScience/Agents-A1, a Qwen3.5-35B-A3B MoE multimodal model.

Attribution

• —Original model: InternScience/Agents-A1 — a Qwen3.5-35B-A3B MoE multimodal derivative
• —MTP heads: Grafted from unsloth/Qwen3.6-35B-A3B-MTP-GGUF
• —Quantization framework: llama.cpp with NVFP4/MXFP4 MoE routing patches (commits 1f8583c6b, 6e761dbc9)
• —Tensor mapping: Follows Unsloth's gold standard MoE quantization strategy
• —Quantized by: s-batman

Files

Tensor Type Mapping

Follows Unsloth's gold standard MoE quantization mapping:

MTP (Multi-Token Prediction)

MTP heads grafted from unsloth/Qwen3.6-35B-A3B-MTP-GGUF for speculative decoding support.

• —block_count: 40 → 41
• —nextn_predict_layers: 1
• —20 MTP tensors at blk.40.* (separate gate/up experts, not fused)

Usage with MTP speculative decoding

llama-server \
  -m agents-a1-NVFP4-MTP.gguf \
  --mmproj mmproj-agents-a1-f16.gguf \
  -c 4096 -ngl 99 -fa 1 \
  -b 1024 -ub 4096 \
  --spec-type draft-mtp --spec-draft-n-max 8

Architecture

• —Architecture: Qwen3.5 MoE (qwen35moe)
• —Layers: 40 (30 linear attention + 10 full attention, every 4th)
• —Experts: 256 total, 8 per token, 1 shared expert
• —Hidden size: 2048
• —Head dim: 256
• —Vocab: 248,320
• —Context: 262,144
• —Vision: ViT depth 27, hidden 1152 (multimodal)

Quantization Details

Converted from BF16 GGUF using llama-quantize with:

• —--output-tensor-type q8_0
• —--token-embedding-type q8_0
• —--no-mtp --fuse-gate-up-exps (during HF→GGUF conversion)
• —MoE routing: 3D expert tensors (ne[2] > 1) → FP4, 2D linear tensors → Q8_0, norms → F32

Benchmarks

All benchmarks run on NVIDIA GB10 (DGX Spark) — 124 GB unified memory, compute capability 12.1, BLACKWELL_NATIVE_FP4=1. Benchmark tool: llama-bench (build 9277) and llama-server with chat completions.

Prompt Processing (prefill) — varying ubatch size

Fixed: b=4096, pp2048, ngl=99, fa=1, 3 repetitions.

Best prefill: NVFP4 at ub=4096 (1,642 t/s), MXFP4 at ub=2048 (1,619 t/s). Larger ubatch consistently improves prefill throughput.

Token Generation (decode) — varying batch size

Fixed: ub=512, tg256, ngl=99, fa=1, 3 repetitions.

Best decode: Both formats at b=1024 (25.4 t/s). Decode performance is relatively flat across batch sizes, with b=4096 showing slight degradation.

Chat completions — MTP speculative decoding sweep

Fixed: b=1024, ub=4096, c=4096, ngl=99, fa=1, 256 max tokens, temperature 0.7. Values are averages of 3 runs. MTP heads grafted from unsloth/Qwen3.6-35B-A3B-MTP-GGUF.

NVFP4 — varying --spec-draft-n-max

Key finding: --spec-draft-n-max 2 is the optimal setting on GB10, delivering a 21% speedup over baseline (84.1 t/s vs 69.3 t/s). The accept rate at n-max=2 is 62% — substantially higher than at n-max=8 (36%) because shorter draft sequences have higher per-token match probability and lower overhead.

At n-max=8, the overhead of generating 8 draft tokens per step with only 36% acceptance makes MTP net-negative. The break-even point on GB10 is around n-max=4.

MXFP4 — n-max=8 (prior runs, with mmproj)

MXFP4 shows the same pattern as NVFP4 at n-max=8. Based on the NVFP4 sweep, n-max=2 is expected to provide a similar ~20% speedup for MXFP4 as well.

Note: The tg256 numbers from llama-bench (25 t/s) differ from the chat completion numbers (69 t/s) because llama-bench measures raw decode throughput with no chat template overhead, while the chat tests include Jinja template processing, thinking mode, and warmup effects. The relative comparison between MTP and no-MTP is valid within the same test type.

Recommended configuration

# Best decode with MTP speculative decoding (recommended)
llama-server -m agents-a1-NVFP4-MTP.gguf --mmproj mmproj-agents-a1-f16.gguf \
  -c 4096 -ngl 99 -fa 1 -b 1024 -ub 4096 -np 1 \
  --spec-type draft-mtp --spec-draft-n-max 2

# Baseline without MTP
llama-server -m agents-a1-NVFP4-MTP.gguf --mmproj mmproj-agents-a1-f16.gguf \
  -c 4096 -ngl 99 -fa 1 -b 1024 -ub 4096 -np 1

License

MIT

Quantized by [s-batman](https://huggingface.co/s-batman). For additional GGUF quants, benchmarks, and articles on LLM memory and context engineering, visit [sbatman.com](https://sbatman.com).