Modelpublic

emese-tech/csermely-mlx

sourceHugging Faceapache-2.0updated 17d agoView on Hugging Face

0likes439downloads

Model Card

Emese-Csermely (1.7B) — MLX q8

MLX q8-quantized build of Csermely, the mobile/on-device tier of the Emese Hungarian model family. See the csermely/ repo's README for full architecture, CPT/SFT/DPO training details, and benchmarks — this file covers only the q8-specific notes.


Quantization	q8, group size 64 (`mlx_lm convert -q --q-bits 8 --q-group-size 64`)
Size on disk	~1.7 GB (vs. ~3.1 GB bf16)
Quality	≈ bf16, near-lossless at 8-bit
Max context length	4,096 tokens (unchanged from bf16 — EuroLLM-1.7B's native context)

Usage

python

from mlx_lm import load, generate
from mlx_lm.sample_utils import make_sampler
model, tok = load("emese-tech/csermely-mlx")
p = tok.apply_chat_template([{"role": "user", "content": "Szia! Mit tudsz csinálni?"}],
                            tokenize=False, add_generation_prompt=True)
print(generate(model, tok, prompt=p, max_tokens=256, sampler=make_sampler(temp=0.2)))

Decode: temperature 0.2, no repetition penalty, eos {2, 4}, ChatML template.

⚠️ This repo is `mlx_lm`-only — MLX's q8 quantization packs weights into uint32 + per-group scales/biases tensors with a quantization block in config.json that plain transformers does not understand (AutoModelForCausalLM.from_pretrained will raise quant_method errors on this repo). Use the csermely/ (bf16) repo for transformers/vLLM/TGI.

Training

Identical underlying weights to csermely/ (bf16), just quantized after the fact — no separate training. See csermely/README.md for the full CPT/SFT/DPO recipe and benchmark numbers.

Benchmarks

This exact q8 artifact scored 211/500 (42%) on emese-bench v1 (the consolidated 500-pt Ultimate+BlindSpot benchmark) — see emese-bench/results/csermely-mlx.md for the full category-by-category transcript and emese-bench/README.md for the benchmark's design.