emese-tech/folyo-gguf

Emese-Folyó (22B) — GGUF Q4KM

GGUF Q4_K_M — a compact llama.cpp-compatible build of Folyó, quantized from the model's native q8 MLX artifact (folyo-mlx/). See the folyo/ repo's README for full architecture, CPT/SFT/DPO training details, and benchmarks — this file covers only the GGUF-specific notes.

⚠️ Tokenizer fix required — read this before using any other GGUF build of this model

A stock convert_hf_to_gguf.py export of this model family is badly broken: it types the ChatML control tokens (<|im_start|>, <|im_end|>) as NORMAL instead of CONTROL, so <|im_start|> gets shredded into 7 garbage sub-word tokens instead of being fed to the model as the single trained token — a prompt shape the model never saw during training. It also writes a flat placeholder BPE merge score for every token, corrupting subword-split priority. Together these caused a severe, previously-misdiagnosed quality regression (early testing wrongly concluded it was inherent to llama.cpp itself).

This GGUF file has already been fixed — scripts/fix_gguf_tokenizer.py (in the main repo) was run on it after conversion/quantization to correct the special-token typing, BPE scores, and a stray leading-space flag. Verified byte-identical tokenization to the HF/MLX reference (same tokenizer/vocab family as Patak, where this was confirmed on 80/80 sampled bench prompts). If you ever regenerate this GGUF from source yourself, you must re-run that fix script (or the equivalent metadata patch) — a plain convert_hf_to_gguf.py + llama-quantize pipeline without it reproduces the old broken behavior.

Usage

llama-server -m emese-folyo-Q4_K_M.gguf -c 4096

import requests
r = requests.post("http://127.0.0.1:8080/v1/chat/completions", json={
    "messages": [{"role": "user", "content": "Mi Magyarország fővárosa?"}],
    "temperature": 0.2, "repeat_penalty": 1.15, "stop": ["<|im_end|>"],
})
print(r.json()["choices"][0]["message"]["content"])

Decode: temperature 0.2, repeatpenalty `1.15`, stop on `<|imend|>, ChatML template (<|imstart|>role\n...<|imend|>\n`).

Training

Same underlying weights as folyo-mlx/ (q8, the model's native training precision), just re-quantized to GGUF Q4KM — no separate training. See folyo/README.md for the full CPT (~6M tokens/6,000 iters, rank64), SFT (instruct_v18b, 1 epoch, rank16/scale16/lr5e-6, iteration 3,600 selected), and DPO (36 alfa pairs, 120 iters, rank16/scale16/lr2e-6) recipe.

Benchmarks

This exact Q4_K_M GGUF build (with the tokenizer fix applied) scored 404/500 (81%) on emese-bench v1, vs. 388/500 (78%) for the same fix's Q80 build and 410/500 (82%) for the original MLX q8 artifact. Zero `<|imstart|>/<|imend|>` leaks. Note: the Q4 build scoring slightly higher than Q8 here is most likely sampling variance between runs (specific failures traded places rather than Q4 being uniformly better) rather than a real quantization advantage — treat the two as within noise of each other. A residual artifact distinct from the tokenizer bug was found in a couple of spots: rare stray encoding glitches (Hangul-filler characters) appended to otherwise-normal text. See `emese-bench/results/folyo-gguf-q4fix.md` for the full category-by-category transcript and `emese-bench/README.md` for the benchmark's design and the Q80 comparison point.

Limitations

• —Can hallucinate specific facts (dates, attributions, biographical details) — verify critical details. Two specific bench questions (about fictional/obscure Hungarian scientists) reliably produce confidently-fabricated biographies across every tested variant of this model family.
• —Hungarian-first; other-language quality inherited from EuroLLM-22B.
• —Fails to maintain English across a full multi-turn conversation when the user writes in English — answers drift back into Hungarian. This is a pre-existing weakness of the underlying model (confirmed present in the MLX original too), not something the GGUF conversion caused.
• —Weak at multi-step math, spatial estimation, and strict multi-constraint formatting (alphabetical ordering, exact word counts, banned letters).
• —Large — even the Q4KM build is ~13 GB; needs a machine with enough RAM for practical serving.