impacte/Qwen3.8-27B-GGUF

Qwen3.8-27B — GGUF (Q4KM multimodal + text-only IQ4_XS) + llama.cpp config

    

Deploy to RunPod

One-click deploy each quantization version as a GPU Pod on RunPod via Ollama (OpenAI-compatible API) with an aggressively quantized 4-bit KV cache (OLLAMA_KV_CACHE_TYPE=q4_0 + Flash Attention) — fitting the full 256K context on a 24 GB GPU.

 

Q4_K_M Multimodal (17.77 GB, Text + Image + Video) · IQ4_XS Text-only (~14.7 GB, Text only). Requires a 24 GB+ VRAM GPU. The 4-bit KV cache drops the 256K KV footprint from ~17 GB (f16) to ~4.3 GB (q4_0).

Full-quality Qwen3.8-27B in GGUF format with ready-to-run llama.cpp configurations that use --no-kv-offload to keep the KV cache in system RAM — unlocking the full 256K context even on modest GPU setups.

Two GGUF variants are published in this repo:

DESCRIPTION

A dense 27.8B-parameter multimodal model from the Qwen team, packaged as GGUFs with complete llama.cpp launchers. Unlike MoE models, every parameter is active on every token, giving maximum quality per token. Its hybrid attention architecture (48 linear + 16 full attention layers) delivers efficient long-context inference without the quadratic cost of pure attention.

Key features:

• —Dense 27.8B — all parameters active, no MoE routing
• —Hybrid attention — 48 linear + 16 full attention layers (3:1 ratio)
• —256K native context (262,144 tokens)
• —Two variants — multimodal Q4KM or text-only IQ4_XS
• —Reasoning model — thinking blocks before response
• —Tool calling — <tool_call><function=...> XML format
• —Q4_K_M quantization — imatrix-calibrated, 17.77 GB (multimodal)
• —IQ4_XS quantization — 4.25 bpw, ~14.7 GB (text-only)

Architecture

FILES

TEXT-ONLY VERSION (IQ4_XS)

The text-only variant strips the vision/audio encoder and projector entirely, leaving only the language-model tensors. It is ideal for pure text workloads (chat, coding, RAG, tool calling) where the extra ~10% of multimodal weights is unnecessary.

Run the text-only version (llama.cpp)

llama-server \
  --model Qwen3.8-27B-text-IQ4_XS.gguf \
  -ngl 99 \
  --no-kv-offload \
  -c 262144 \
  --parallel 1

No --mmproj is passed — the model is purely text-based.

REQUIREMENTS

💡 Why `--no-kv-offload`? The 17.8 GB weights fit on GPU, but the KV cache for 256K context is huge. By keeping the KV cache in system RAM, all 64 layers stay on GPU while the context window can reach the full 256K — ideal for 24 GB VRAM + 128 GB RAM setups.

QUICK START (llama.cpp — recommended)

1. Download this repo

git lfs clone https://huggingface.co/impacte/Qwen3.8-27B-GGUF
cd Qwen3.8-27B-GGUF

2. Start the server

./run-llamacpp.sh
# -> OpenAI-compatible API at http://localhost:8082/v1

The script sets CUDA_VISIBLE_DEVICES=0,1 (device ordinals, not PCI IDs) and uses --no-kv-offload so the KV cache goes to system RAM.

3. Test it

curl http://localhost:8082/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen3.8-27b","messages":[{"role":"user","content":"Hello!"}]}'

USAGE

Manual llama.cpp command

llama-server \
  --model Qwen3.8-27B-Q4_K_M.gguf \
  --mmproj mmproj-Qwen3.8-27B-f16.gguf \
  -ngl 99 \
  --no-kv-offload \
  -c 262144 \
  --parallel 1

Python (OpenAI SDK)

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8082/v1", api_key="none")

response = client.chat.completions.create(
    model="qwen3.8-27b",
    messages=[{"role": "user", "content": "Explain quantum computing simply"}],
)
print(response.choices[0].message.content)

Multimodal (image)

curl http://localhost:8082/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3.8-27b",
    "messages": [{
      "role": "user",
      "content": [
        {"type": "text", "text": "Describe this image"},
        {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,<BASE64>"}}
      ]
    }]
  }'

RUN WITH OLLAMA

ollama create qwen3.8-27b -f Modelfile
ollama run qwen3.8-27b

Note: On 24 GB VRAM, Ollama cannot offload the KV cache to RAM, so the default context is limited to 8K. Use the llama.cpp script above for the full 256K context. A 32 GB+ VRAM variant is also published on Ollama as oamazonasgabriel/qwen3.8-27b:q4-km-32gbGPU.

MEMORY & PERFORMANCE

Performance (24 GB VRAM + 128 GB RAM):

• —Prompt processing: ~60-120 tok/s (hybrid linear attention)
• —Text generation: ~20-40 tok/s
• —Model load time: ~60-90 seconds

TROUBLESHOOTING

CREDITS

LINKS