alperugurcan/hy-mt15-18b-125bit-gguf-translator

Hy-MT1.5-1.8B 1.25-bit GGUF Translator

Base model: https://huggingface.co/tencent/HY-MT1.5-1.8B

This Space runs AngelSlim/Hy-MT1.5-1.8B-1.25bit-GGUF with a custom llama.cpp build from PR #22836, which adds the STQ1_0 kernel required by this 1.25-bit GGUF.

The Docker image builds the STQ-enabled llama-server target once, then the app downloads only Hy-MT1.5-1.8B-1.25bit.gguf, starts one global llama-server process during startup, warms it with a tiny request, and reuses that resident server for every request. Gradio inference is serialized by default to keep the free CPU tier stable.

CPU-oriented defaults:

• —LLAMA_N_CTX=2048
• —LLAMA_N_BATCH=128
• —LLAMA_N_UBATCH=64
• —LLAMA_THREADS=min(cpu_count - 1, 4)
• —LLAMA_N_GPU_LAYERS=0
• —LLAMA_USE_MMAP=1
• —LLAMA_USE_MLOCK=0
• —LLAMA_FLASH_ATTN=auto, disabled on CPU-only runtime

FlashAttention is not forced on CPU. With the default CPU settings it stays disabled. It is only requested automatically when GPU offload is configured and the runtime clearly exposes CUDA or Metal indicators.

Useful overrides can be set as Space variables:

• —LLAMA_THREADS: cap llama.cpp CPU workers.
• —LLAMA_N_CTX: lower this if memory is tight, raise it if latency allows.
• —LLAMA_N_BATCH and LLAMA_N_UBATCH: lower for less memory pressure.
• —LLAMA_USE_MMAP=0: fully load model weights instead of memory mapping.
• —LLAMA_N_GPU_LAYERS=-1: offload all possible layers on a GPU Space.
• —LLAMA_FLASH_ATTN=1: force FlashAttention only on a compatible GPU backend.

Primary upstream references:

• —AngelSlim/Hy-MT1.5-1.8B-1.25bit-GGUF
• —tencent/HY-MT1.5-1.8B
• —ggml-org/llama.cpp PR #22836