Modelpublic

vadikulous-ai/Qwen-AgentWorld-35B-A3B-UD-Q4_K_XL-MTP-GGUF

sourceHugging Faceapache-2.0updated 2mo agoView on Hugging Face

4likes315downloads

Model Card

Qwen-AgentWorld-35B-A3B-UD-Q4KXL-MTP-GGUF

This repository contains the Qwen-AgentWorld-35B-A3B model in UD-Q4KXL quantization with Multi-Token Prediction (MTP) support for speculative decoding.

Acknowledgments

—Qwen Team for the original Qwen-AgentWorld-35B-A3B model
—Unsloth for the Qwen3.6-35B-A3B-MTP-GGUF quantization with MTP heads
—Unsloth for the Qwen-AgentWorld-35B-A3B-GGUF quantization

How It Was Made

The MTP heads from Qwen3.6-35B-A3B were extracted and grafted onto the Qwen-AgentWorld-35B-A3B model. Both models share the same architecture (qwen35moe), making this transfer possible. The resulting model includes all base layers from AgentWorld plus the MTP prediction head from the original Qwen3.6 model.

Hardware & Performance

Performance results were achieved on AMD Strix Halo:

—Without MTP: ~60 tokens/s
—With MTP: ~77 tokens/s
—Speed improvement: +28%
—Draft acceptance rate: ~87%

The high acceptance rate indicates that the MTP heads from the base model transfer well to the AgentWorld fine-tune.

Usage

Basic Usage

bash

./llama-server \
  -m Qwen-AgentWorld-35B-A3B-UD-Q4_K_XL-MTP.gguf \
  -ngl 99 -c 8192 -fa on \
  --spec-type draft-mtp \
  --spec-draft-n-max 3

Full Multimodal Setup

For multimodal capabilities with reasoning and optimized parameters:

bash

MODEL_PATH="Qwen-AgentWorld-35B-A3B-UD-Q4_K_XL-MTP.gguf"
MMPROJ_PATH="mmproj-BF16.gguf"

./llama-server \
  --model "$MODEL_PATH" \
  --mmproj "$MMPROJ_PATH" \
  -ngl 99 \
  --host 0.0.0.0 \
  --port 8000 \
  --ctx-size 262144 \
  --reasoning "on" \
  --kv-unified \
  -np 2 \
  -fa "on" \
  --no-mmap \
  --image-min-tokens 1024 \
  --spec-type "draft-mtp" \
  --spec-draft-n-max 3 \
  --spec-draft-p-min 0.75 \
  -b 2048 \
  -ub 512

Parameters:

—--mmproj: Multimodal projector for vision capabilities
—--reasoning "on": Enable reasoning mode
—--kv-unified: Unified KV cache
—-np 2: 2 parallel slots
—--spec-type "draft-mtp": Enable MTP speculative decoding
—--spec-draft-n-max 3: Generate up to 3 draft tokens per step
—--spec-draft-p-min 0.75: Minimum probability threshold for draft tokens
—-b 2048: Batch size
—-ub 512: U-batch size