emese-tech/csermely-gguf

Emese-Csermely (1.7B) — GGUF Q4KM

GGUF Q4_K_M — a compact llama.cpp-compatible build of Csermely, quantized from the model's native q8 MLX artifact (csermely-mlx/). See the csermely/ repo's README for full architecture, CPT/SFT/DPO training details, and benchmarks — this file covers only the GGUF-specific notes.

⚠️ Tokenizer fix required — read this before using any other GGUF build of this model

A stock convert_hf_to_gguf.py export of this model family is badly broken: it types the ChatML control tokens (<|im_start|>, <|im_end|>) as NORMAL instead of CONTROL, so <|im_start|> gets shredded into 7 garbage sub-word tokens instead of being fed to the model as the single trained token — a prompt shape the model never saw during training. It also writes a flat placeholder BPE merge score for every token, corrupting subword-split priority. Together these caused a severe, previously-misdiagnosed quality regression (early testing wrongly concluded it was inherent to llama.cpp itself).

This GGUF file has already been fixed — scripts/fix_gguf_tokenizer.py (in the main repo) was run on it after conversion/quantization to correct the special-token typing, BPE scores, and a stray leading-space flag. Verified: sampled bench prompts tokenize byte-identical to the HF/MLX reference tokenization. If you ever regenerate this GGUF from source yourself, you must re-run that fix script (or the equivalent metadata patch) — a plain convert_hf_to_gguf.py + llama-quantize pipeline without it reproduces the old broken behavior.

Usage

llama-server -m emese-csermely-Q4_K_M.gguf -c 4096

import requests
r = requests.post("http://127.0.0.1:8080/v1/chat/completions", json={
    "messages": [{"role": "user", "content": "Mi Magyarország fővárosa?"}],
    "temperature": 0.2, "repeat_penalty": 1.15, "stop": ["<|im_end|>"],
})
print(r.json()["choices"][0]["message"]["content"])

Decode: temperature 0.2, repeatpenalty `1.15`, stop on `<|imend|>, ChatML template (<|imstart|>role\n...<|imend|>\n`).

Training

Same underlying weights as csermely-mlx/ (q8, the model's native training precision), just re-quantized to GGUF Q4KM — no separate training. See csermely/README.md for the full CPT/SFT/DPO recipe.

Benchmarks

This build has the same tokenizer fix validated on the Patak and Folyó GGUF exports (byte-identical tokenization to the HF/MLX reference across sampled prompts), so it does not carry the tokenizer regression. A full emese-bench v1 scoring pass specifically for this Q4KM build has not been finalized yet — see csermely/README.md and csermely-mlx/README.md for the model's scored benchmark results (211/500, 42%, on the bf16/MLX q8 artifact), which is the closest available reference point until this GGUF build is scored separately.

Limitations

• —Can hallucinate specific facts (dates, attributions, biographical details) — verify critical details. Two specific bench questions (about fictional/obscure Hungarian scientists) reliably produce confidently-fabricated biographies across every tested variant of this model family.
• —Hungarian-first; other-language quality inherited from EuroLLM-1.7B.
• —Weak at multi-step math, spatial estimation, and strict multi-constraint formatting (alphabetical ordering, exact word counts, banned letters) — consistent with the MLX original.
• —As the smallest tier in the Emese family (1.7B), expect lower overall quality than Patak/Folyó.