originalGeek/Orthrus-Qwen3-8B-GGUF

Orthrus-Qwen3-8B GGUF (Q4KM)

Q4KM GGUF build of chiennv/Orthrus-Qwen3-8B for the Orthrus llama.cpp reference fork:

https://github.com/remesis/orthrus_llamacpp

Orthrus is not just a Qwen3 weight layout. The important runtime feature is shared-KV self-drafting: the autoregressive path owns the persistent target KV cache, the diffusion view drafts against that same cache, transient diffusion cells are removed before verification, and normal AR verification commits or rolls back the candidate path.

This GGUF uses the dedicated orthrus architecture and requires a llama.cpp build with Orthrus support for --spec-type draft-orthrus. It is not expected to run correctly on upstream llama.cpp until equivalent support exists there.

Files

Run

Clone and build the reference fork:

git clone https://github.com/remesis/orthrus_llamacpp
cd orthrus_llamacpp
cmake -S . -B build-cuda -G "Visual Studio 17 2022" -A x64 -DGGML_CUDA=ON -DLLAMA_CURL=OFF
cmake --build build-cuda --config Release --target llama-server

Download this GGUF and run the tuned local config:

.\build-cuda\bin\Release\llama-server.exe `
  -m models\Orthrus-Qwen3-8B\Orthrus-Qwen3-8B-Q4_K_M.gguf `
  --host 127.0.0.1 `
  --port 8081 `
  -c 40960 `
  -np 1 `
  -b 1024 `
  -ub 512 `
  -ngl 999 `
  -fa on `
  --ui `
  --jinja `
  --chat-template-file models\Orthrus-Qwen3-8B\chat_template.jinja `
  --spec-type draft-orthrus `
  --spec-draft-n-max 15

Notes:

• —Use --spec-type none for a plain autoregressive baseline.
• —--spec-draft-n-max 15 was the best average local setting in the benchmark below.
• —Auto draft length resolves to block_size - 1 (31) and can be better or worse depending on prompt shape.
• —Passing --chat-template-file is optional when the template is embedded, but it makes the intended template explicit.

Local Validation

The 8B Q4KM GGUF was checked for:

Runtime checks:

• —llama-server loaded the 8B Q4KM GGUF with CUDA offload.
• —draft-orthrus startup logged shared_kv=yes, block_size=32, and mask_id=151669.
• —Temp-0 decoded output matched AR-only output in controlled server tests.
• —Quantization retained the Orthrus diffusion projection tensors.

Official Runtime Cross-Check

The Orthrus team's own Transformers implementation was also benchmarked locally to confirm the intended shared-KV diffusion path and to sanity-check whether native Orthrus drafting is worthwhile before publishing this GGUF.

Environment:

• —RTX 5090
• —PyTorch 2.11.0+cu128
• —Transformers 5.12.0
• —BF16 source checkpoint
• —attn_implementation="sdpa"
• —temp-0 generation

Raw completion results:

Chat-template results:

These numbers are not quantization-to-quantization comparisons against the GGUF below; they use the official BF16 Transformers runtime. They are included because they validate that the source model's intended Orthrus path can produce large throughput gains on favorable raw-completion prompts and smaller but still measurable gains on chat/long-context prompts.

Benchmark Snapshot

Environment:

• —RTX 5090
• —CUDA 13.3
• —-np 1
• —-c 40960
• —-ngl 999
• —-fa on
• —raw /completion temp-0 benchmark prompts

Wall TPS:

Acceptance for the tuned 8B Q4KM nmax 15 run:

Interpretation:

• —The tuned 8B Q4KM Orthrus config was the best average local benchmark.
• —Plain Q4KM AR was faster on the long-context raw-completion case.
• —Acceptance is content-dependent, so benchmark your intended workload.

Source

• —Orthrus project: https://github.com/chiennv2000/orthrus
• —Source model: https://huggingface.co/chiennv/Orthrus-Qwen3-8B
• —llama.cpp reference fork: https://github.com/remesis/orthrus_llamacpp

Citation

@misc{vannguyen2026orthrusmemoryefficientparalleltoken,
      title={Orthrus: Memory-Efficient Parallel Token Generation via Dual-View Diffusion},
      author={Chien Van Nguyen and Chaitra Hegde and Van Cuong Pham and Ryan A. Rossi and Franck Dernoncourt and Thien Huu Nguyen},
      year={2026},
      eprint={2605.12825},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2605.12825},
}