grteam/Qwen3.6-35B-A3B-GRTEAM-MTP-Q5_K_M

Qwen3.6-35B-A3B-GRTEAM-MTP-Q5KM

⚠️ THIS MODEL IS UNCENSORED — USE IT RESPONSIBLY ⚠️ This model has no content filters or safety guardrails. By using this model, you accept full responsibility for its output. Do NOT use for illegal, harmful, or unethical purposes. GRTEAM does not endorse or take responsibility for any content generated by this model.

⚡ 51 tokens/second on NVIDIA Jetson AGX Orin 64GB — No Reasoning Mode with Multi-Token Prediction

🇬🇷 Built in Greece by [GRTEAM](https://grteam.gr) — Proud Greek engineering pushing edge AI to its limits!

🙏 Special Kudos to [darkc0de](https://huggingface.co/darkc0de) — Creator of XORTRON, the awesome fine-tune that powers this model.

🚀 Performance Benchmark

💡 51 t/s on edge hardware — This is achieved by disabling the reasoning/thinking mode and leveraging Qwen 3.6's built-in MTP heads for speculative decoding. No separate draft model needed — the MTP heads are embedded in the same GGUF file.

📋 Reproduction Config (llama-swap)

This is the exact llama-swap config used to achieve 51 t/s:

"qwen3.6-35b-grteam-mtp-no-reason":
  aliases:
    - "Qwen3.6-35B-GRTEAM-MTP-NoReason"
  cmd: |
    /ssd/llama.cpp-mtp/build/bin/llama-server-mtp
    --model /ssd/models/Qwen3.6-35B-A3B-GRTEAM-MTP-Q5_K_M.gguf
    --port ${PORT}
    --host 0.0.0.0
    --threads 12
    --threads-batch 12
    --batch-size 2048
    --ubatch-size 512
    --n-gpu-layers 99
    --flash-attn on
    --jinja
    --mlock
    --cache-type-k q4_0
    --cache-type-v q4_0
    --kv-unified
    --ctx-size 131072
    --chat-template-file /ssd/llama-swap/templates/qwen3.6-fixed.jinja
    --reasoning off
    --temp 0.6
    --top-p 0.95
    --top-k 20
    --min-p 0.0
    --presence-penalty 0.0
    --spec-type mtp
    --spec-draft-n-max 3
    -np 1
  proxy: http://127.0.0.1:${PORT}
  ttl: 0

Key Settings Breakdown

🖥️ Hardware: NVIDIA Jetson AGX Orin 64GB

The NVIDIA Jetson AGX Orin is a world-class edge AI platform:

• —GPU: 2048-core NVIDIA Ampere architecture with 64 Tensor Cores
• —CPU: 12-core Arm Cortex-A78AE v8.2
• —Memory: 64GB unified LPDDR5 (204.8 GB/s bandwidth)
• —Unified Memory Architecture: CPU and GPU share the same 64GB pool — no PCIe bottleneck for LLM inference

Why Orin AGX is Perfect for LLM Serving

1. 1.Unified Memory: No VRAM limitation like discrete GPUs — the full 64GB is available for model weights + KV cache
2. 2.Power Efficiency: Runs at 15-60W vs 300W+ for desktop GPUs
3. 3.24/7 Edge Deployment: Built for always-on inference workloads
4. 4.llama.cpp Native Support: CUDA backend works out-of-the-box on JetPack

📦 Model Details

• —Base Architecture: Qwen3-35B-A3B (MoE — Mixture of Experts)
• —Active Parameters: ~3B per token (A3B = Active 3 Billion)
• —Total Parameters: ~35B
• —Quantization: Q5KM (5-bit, K-quants, Medium)
• —File Size: ~25.6 GB
• —Format: GGUF (llama.cpp compatible)
• —MTP Heads: Built-in Multi-Token Prediction heads (no separate draft model needed)

What is MTP (Multi-Token Prediction)?

MTP is a speculative decoding technique where the model predicts multiple future tokens in a single forward pass. Qwen 3.6 includes dedicated MTP heads that can predict up to 3 tokens ahead, providing:

• —~2-3x speedup over autoregressive decoding
• —No accuracy loss — verified tokens are identical to standard decoding
• —No extra VRAM for a separate draft model — MTP heads share the model weights

🔧 Usage

With llama.cpp / llama-server

llama-server \
  --model Qwen3.6-35B-A3B-GRTEAM-MTP-Q5_K_M.gguf \
  --host 0.0.0.0 \
  --port 8080 \
  --n-gpu-layers 99 \
  --ctx-size 131072 \
  --flash-attn on \
  --mlock \
  --cache-type-k q4_0 \
  --cache-type-v q4_0 \
  --kv-unified \
  --batch-size 2048 \
  --ubatch-size 512 \
  --threads 12 \
  --jinja \
  --reasoning off \
  --temp 0.6 \
  --top-p 0.95 \
  --top-k 20 \
  --spec-type mtp \
  --spec-draft-n-max 3 \
  -np 1

With LM Studio

Simply download the GGUF file and load it — LM Studio will auto-detect the format.

With llama-swap

Use the config block provided in the Reproduction Config section above.

With Ollama

ollama create qwen3.6-grteam -f Modelfile
# Modelfile:
# FROM ./Qwen3.6-35B-A3B-GRTEAM-MTP-Q5_K_M.gguf
# PARAMETER temp 0.6
# PARAMETER top_p 0.95
# PARAMETER top_k 20

🏷️ About GRTEAM 🇬🇷

[GRTEAM](https://grteam.gr) is an independent AI research group based in Greece 🇬🇷, focused on pushing the limits of edge AI and local LLM deployment.

"From Athens to the edge — Greek engineers proving that world-class AI doesn't need a data center."

We specialize in:

• —🤖 Running state-of-the-art LLMs on NVIDIA Jetson platforms
• —⚡ Optimizing inference for maximum tokens/second on constrained hardware
• —🔧 Building custom llama.cpp and llama-swap configurations
• —🧪 Testing quantized models (GGUF) for real-world quality vs. speed tradeoffs

📬 Contact

• —🌐 Website: grteam.gr
• —📧 Email: admin@grteam.gr

Follow us on X (Twitter) for more edge AI benchmarks, configs, and releases!

📊 Speed Comparison (Jetson AGX Orin 64GB)

Speeds measured on Jetson AGX Orin 64GB with llama.cpp MTP build and llama-swap.

⚠️ Important Notes

1. 1.MTP requires a llama.cpp build with MTP support — use the llama.cpp-mtp branch (PR #22673) or a fork that supports --spec-type mtp
2. 2.`--reasoning off` disables the thinking/reasoning output — use this for maximum speed when you dont need chain-of-thought
3. 3.Q5_K_M quantization provides an excellent balance of quality and speed for 35B MoE models
4. 4.This model requires ~25.6 GB of memory (unified CPU+GPU on Jetson, or 26GB+ VRAM on discrete GPU)

📜 License

This model is released under the Apache 2.0 license, following the base Qwen3 model license.

🙏 Credits

• —Base Model: Qwen Team — Qwen3-35B-A3B
• —XORTRON Fine-Tune: darkc0de — The incredible XORTRON fine-tune that this model is based on
• —MTP Integration: Built using llama.cpp MTP branch (PR #22673)
• —Benchmarking & Config: GRTEAM on NVIDIA Jetson AGX Orin 64GB
• —Serving: llama-swap for model hot-swapping

Optimized for edge. Benchmarked on Jetson. Built in Greece by [GRTEAM](https://grteam.gr). 🇬🇷