Modelpublic

sahilchachra/hy-mt2-7b-8bit-mlx

sourceHugging Faceotherupdated 4mo agoView on Hugging Face

1likes33downloads

Model Card

hy-mt2-7b-8bit-mlx

Quantized version of tencent/Hy-MT2-7B for Apple Silicon using MLX.

Hy-MT2-7B is Tencent's multilingual translation model covering 40+ languages.

Quantization: Affine integer quantization Precision: 8-bit (~8.5 bits/weight avg) Group size: 64 Disk size: 7620 MB Quantized by: sahilchachra

About this variant

Affine quantization at 8-bit with group size 64. Closest to FP16 translation quality. Recommended when memory allows and translation accuracy is the priority.

Benchmark results

Evaluated on Apple M5 Pro with MLX. Model loaded once; performance and quality measured in a single pass.

Performance

	This model	FP16 baseline
Prefill (tok/s)	398.77	307.33
Decode (tok/s)	36.28	19.55
Peak memory (GB)	8.175	15.171
Disk size (MB)	7620	15331

Translation quality (FLORES-200 devtest)

Reported as chrF++ (higher is better). Sample-size noted per pair.

Direction	This model	FP16 baseline	n
engLatn→fraLatn	68.97	68.74	20
engLatn→deuLatn	63.22	63.25	20
engLatn→zhoHans	29.4	29.4	20
engLatn→jpnJpan	41.85	42.28	20
engLatn→spaLatn	56.62	56.7	20
fraLatn→engLatn	68.14	67.99	20
zhoHans→engLatn	57.45	57.43	20
jpnJpan→engLatn	59.03	59.2	20

Avg chrF++: 60.23 vs FP16 60.24 Avg BLEU: 35.41 vs FP16 35.35

Context scaling (decode tok/s)

Context length	Decode tok/s
~128 tokens	35.7
~256 tokens	42.4
~512 tokens	42.4
~1024 tokens	42.1

Usage

Install

bash

pip install mlx-lm

Translate

python

from mlx_lm import load, generate

model, tokenizer = load("sahilchachra/hy-mt2-7b-8bit-mlx")

prompt = (
    "Translate the following text from English to French.\n"
    "English: The early bird catches the worm.\n"
    "French:"
)
print(generate(model, tokenizer, prompt=prompt, max_tokens=128, verbose=True))

Stream

python

from mlx_lm import load, stream_generate

model, tokenizer = load("sahilchachra/hy-mt2-7b-8bit-mlx")
for chunk in stream_generate(model, tokenizer, prompt="Translate \"Hello world\" to Japanese:", max_tokens=64):
    print(chunk.text, end="", flush=True)

All variants in this collection

Model	Method
sahilchachra/hy-mt2-7b-4bit-mlx	Affine int4 (group 64)
sahilchachra/hy-mt2-7b-8bit-mlx	Affine int8 (group 64) ← this model

Notes

—Requires Apple Silicon (M1 or later) with MLX
—Benchmarks run on Apple M5 Pro, 24 GB unified memory
—FLORES-200 sample sizes are small — treat chrF/BLEU figures as indicative, not definitive
—License: see tencent/Hy-MT2-7B for the original model's license terms

Original model

See tencent/Hy-MT2-7B for full model details, supported languages, and intended use.