CoolFace
Apppublic

FreeAIModelsForSure/MiniCPM5-1B-Q8

sourceHugging Faceapache-2.0updated 3mo agoView on Hugging Face
1likes
App README

Qwen3.6-27B MTP (TQ3_4S) GGUF Chat (Free CPU Tier)

This Hugging Face Space runs the Qwen3.6-27B-MTP-TQ3_4S.gguf model from YTan2000/Qwen3.6-27B-MTP-TQ3_4S entirely on the free CPU tier (2 vCPUs, 16 GB RAM) using llama-cpp-python and a Gradio chat UI.

Why Docker + multi-stage build with a custom llama.cpp fork?

Hugging Face's native Python/Gradio SDK installs llama-cpp-python from source, which on the free tier:

  • —Takes 15-30+ minutes to compile the C++ backend.
  • —Often hits the build timeout, leaving the Space in a failed state.

The Docker SDK lets us control the build. We use a two-stage multi-stage build with a custom llama.cpp fork:

  1. 1.Builder stage — downloads the llama-cpp-python==0.3.32 sdist, replaces its bundled vendor/llama.cpp with a checkout of `turbo-tan/llama.cpp-tq3`, and compiles the wheel from source with OpenBLAS acceleration. ~20-30 min, cached by HF Spaces.
  2. 2.Runtime stage — slim image that just copies the pre-compiled wheel. No compilation at runtime, fast cold start.

Why a custom fork (not stock llama-cpp-python)?

The model file Qwen3.6-27B-MTP-TQ3_4S.gguf uses GGML tensor type 46 (TQ3_4S — TurboQuant 3-bit with four u8 per-8 scales). Stock llama.cpp (any version, including the one bundled with llama-cpp-python==0.3.32) only supports types 0-42 and fails with:

gguf_init_from_reader: tensor 'blk.0.attn_gate.weight' has invalid
ggml type 46. should be in [0, 42)

The turbo-tan/llama.cpp-tq3 fork adds:

AdditionFilePurpose
GGML_TYPE_TQ3_4S = 46ggml/include/ggml.hType enum entry
dequantize_row_tq3_4s()ggml/src/ggml-quants.cCPU reference dequant
ggml_vec_dot_tq3_4s_q8_0()ggml/src/ggml-cpu/quants.cCPU matmul kernel
[GGML_TYPE_TQ3_4S] traitggml/src/ggml-cpu/ggml-cpu.cType registration

So the model loads and runs on CPU, using the reference (non-SIMD) dequantization path. Performance is modest (~1-3 tok/s on 2 vCPUs) but functional.

Why compile from source (not pre-built wheel)?

The abetlen pre-built CPU wheel index tops out at 0.3.19 (March 2026). PyPI itself ships sdist-only for every llama-cpp-python release — there are no pre-built Linux wheels anywhere for versions newer than 0.3.19. Even if a wheel existed, it would bundle stock llama.cpp (no TQ3_4S). Source compilation against the fork is the only option.

Why Debian-slim (not Alpine)?

We compile from source, so the .so is linked against the builder's libc. We use python:3.10-slim-bookworm (Debian 12, glibc 2.36) because:

  • —It's the standard build environment for llama.cpp (fewer quirks).
  • —OpenBLAS dev packages are well-tested on Debian.
  • —Trixie (Debian 13, GCC 14, glibc 2.41) has known llama.cpp build issues.
  • —The runtime image only needs libopenblas0 + libgomp1 (~10 MB).
LayerChoiceWhy
Base imagepython:3.10-slim-bookworm (both stages)glibc — standard for llama.cpp
llama-cpp-python0.3.32 + turbo-tan/llama.cpp-tq3 forkAdds TQ3_4S (type 46) CPU support
BLASOpenBLAS (build + runtime)CPU matrix acceleration
Build time~20-30 min (one-time, cached)HF Spaces caches Docker layers

Files

FilePurpose
README.mdSpace metadata (YAML frontmatter) + documentation
DockerfileBuild image with llama-cpp-python + Gradio
app.pyModel download + streaming chat server

Runtime Configuration

SettingValue
Model repoYTan2000/Qwen3.6-27B-MTP-TQ3_4S
GGUF fileQwen3.6-27B-MTP-TQ3_4S.gguf
QuantizationTurboQuant TQ3_4S (3.4-bit, ~13.4 GB on disk)
MTP supportYes (Multi-Token Prediction / speculative decode)
CPU threads2 (matches the 2 vCPUs)
Context window4096 tokens
KV cache dtypefp8 (halves KV-cache RAM vs fp16)
Gradio server port7860

Pinned versions (audited 2026-07-04)

PackagePinWhy this version
llama-cpp-python0.3.32June 2026 release. Compiled from source in the Docker builder stage. 0.3.19 (the highest version with a pre-built wheel) fails to load 2026 community GGUF quants. PyPI ships sdist-only for 0.3.20+, so source compilation is the only option.
gradio5.50.0Latest stable 5.x. Gradio 6.x is too fresh for a production deploy. ChatInterface API is identical.
huggingface_hub1.22.0Current 1.x major (0.x is EOL). hf_hub_download signature unchanged.
tqdm4.68.3Latest, no breaking changes.
Python base image3.10-slimDebian/glibc. gradio>=5 and huggingface_hub>=1 both require python>=3.10. Standard build env for llama.cpp.

Local run

bash
docker build -t qwen36-27b-cpu .
docker run --rm -p 7860:7860 qwen36-27b-cpu

Open <http://localhost:7860> in your browser.

Notes & caveats

  • —Cold start: First boot spends a few minutes downloading the GGUF file from the Hub into the Space's ephemeral cache. Free-tier Spaces have no persistent storage, so this happens on every cold start.
  • —Q1_K quality: This is the most aggressive quantization. Output quality is noticeably degraded vs. Q4KM, but the RAM footprint is small enough to fit the free tier alongside the OS and Gradio.
  • —Throughput: Expect ~3-8 tokens/sec on 2 vCPUs for a 27B model. Long prompts (near the 4096 context) will be slower due to prompt processing.