nightmedia/Ling-3.0-tiny-q8-hi-mlx

          arc   arc/e boolq hswag obkqa piqa  wino
bf16      0.507,0.649,0.821,0.634,0.374,0.768,0.610
q8-hi     0.503,0.650,0.828,0.634,0.380,0.768,0.594
q6-hi     0.509,0.657,0.822,0.634,0.380,0.763,0.595
q5-hi     0.499,0.652,0.827,0.631,0.376,0.766,0.589
q4-hi     0.487,0.608,0.812,0.622,0.392,0.761,0.589

Quant     Perplexity      Peak Memory   Tokens/sec
bf16      5.614 ± 0.043   20.49 GB      4459
q8-hi     5.614 ± 0.043   14.04 GB      3761
q6-hi     5.630 ± 0.043   12.19 GB      4020
q5-hi     5.645 ± 0.043   11.27 GB      3838
q4-hi     5.725 ± 0.044   10.35 GB      3672

pip install mlx-lm

from mlx_lm import load, generate

model, tokenizer = load("Ling-3.0-tiny-q8-hi-mlx")

prompt = "hello"

if tokenizer.chat_template is not None:
    messages = [{"role": "user", "content": prompt}]
    prompt = tokenizer.apply_chat_template(
        messages, add_generation_prompt=True, return_dict=False,
    )

response = generate(model, tokenizer, prompt=prompt, verbose=True)