hermitdave/Hy-MT2-30B-A3B-MLX-Mixed2-6

Hy-MT2-30B-A3B-MLX-Mixed2-6

Mixed-precision MLX conversion of tencent/Hy-MT2-30B-A3B using the mixed_2_6 quantization recipe. Most layers run at 2-bit; sensitive layers (first 1/8, last 1/8, every third middle layer, value projections, down projections, and the language model head) are bumped to 6-bit.

4.830 bits per weight, 17 GB total, 30.6 tok/s on M5 Max 128 GB.

Quick Start

Python

from mlx_lm import load, stream_generate
from mlx_lm.sample_utils import make_sampler

model, tokenizer = load("hermitdave/Hy-MT2-30B-A3B-MLX-Mixed2-6")

prompt = "Translate the following text into English. Note that you should only output the translated result without any additional explanation:\n\n오늘 날씨가 정말 좋네요."
messages = [{"role": "user", "content": prompt}]
prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_dict=False)

sampler = make_sampler(temp=0.7, top_p=1.0, top_k=0)
for response in stream_generate(model, tokenizer, prompt=prompt, max_tokens=4096, sampler=sampler):
    print(response.text, end="", flush=True)

Server

mlx_lm.server --model hermitdave/Hy-MT2-30B-A3B-MLX-Mixed2-6 --port 8080 --trust-remote-code

Recommended Parameters

{
  "temperature": 0.7,
  "top_p": 1.0,
  "top_k": -1,
  "repetition_penalty": 1.0,
  "max_tokens": 4096
}

Benchmarks

All three variants tested on the same M5 Max 128 GB machine with identical prompts. Sustained decode speed averaged across three prompts (short KO→EN, short ZH→EN, long KO→EN paragraph).

Speed and Size

Mixed 2-6 wins on speed because it loads the least data from memory per weight. The gap widens on longer generations since memory bandwidth is the bottleneck.

Quality

I ran the same three translation prompts through all three variants and compared the outputs.

Short translations (single sentences):

Wording varies slightly between variants. Q4 and Q8 match each other closely. Mixed 2-6 picks different synonyms ("truly wonderful" vs "really nice") but reads naturally.

Long translation (paragraph about AI technology):

Here's where the differences show up. Mixed 2-6 started repeating itself mid-sentence:

"The development of AI technology has brought about many changes in our society. Particularly, the processing of natural language is a key area where this innovation has greatly enhanced the quality of machine translation as well as the mechanisms that are itself. This is why it is that the language processing is a key process of the development of AI technology."

Q4 stayed clean:

"The development of artificial intelligence technology has brought about many changes in our society. In particular, advancements in natural language processing have significantly improved the quality of machine translation, which plays a crucial role in international exchange and business."

Q8 was nearly identical to Q4, with only minor word choice differences ("innovations" vs "advancements").

The Trade-off

Pick Mixed 2-6 if: You're translating short to medium text, you want the fastest throughput, and you're tight on disk space or memory bandwidth.

Pick Q4 if: You want a balance of speed and quality, or you're translating longer passages where Mixed 2-6 might drift.

Pick Q8 if: You want the best possible quality and don't care about the size penalty.

Mixed 2-6 is surprisingly usable for a model that averages just over 3 bits per weight. The quality degradation on longer generations is real but for typical translation tasks (sentences and short paragraphs) it holds up.

Quantization

The mixed_2_6 recipe assigns different bit-widths per layer group, following the same logic as llama.cpp's Q4_K_M.

Bit Allocation

Interactive visualization: quantization_diagram.html

Conversion

Converted using mlx-lm 0.31.3 with a custom hy_v3 adapter (MLX-LM doesn't yet include native hy_v3 support). The adapter was sourced from QwQbb.

Process:

1. 1.Load base model in bfloat16
2. 2.Cast all float params to bfloat16
3. 3.Apply mixed_2_6 quant predicate
4. 4.Save as MLX safetensors

Hardware

32 GB minimum recommended.

Languages

Hy-MT2 supports translation among 33 languages. See the base model card for the full list.

License

Licensed under the Tencent HY Community License Agreement. Does not apply in the European Union.

Attribution

• —Base model: tencent/Hy-MT2-30B-A3B
• —hy_v3 adapter: QwQbb/Hy-MT2-30B-A3B-MLX-8bit
• —Conversion tool: ml-explore/mlx-lm
• —Benchmarking and conversion: Hermes Agent (research-bot profile)
• —Paper: Hy-MT2: A Family of Fast, Efficient and Powerful Multilingual Translation Models in the Wild

@misc{zheng2026hymt2familyfastefficient,
      title={Hy-MT2: A Family of Fast, Efficient and Powerful Multilingual Translation Models in the Wild},
      author={Mao Zheng and Zheng Li and Tao Chen and Bo Lv and Mingrui Sun and Mingyang Song and Jinlong Song and Hong Huang and Decheng Wu and Hai Wang and Yifan Song and Yanfeng Chen and Guanwei Zhang},
      year={2026},
      eprint={2605.22064},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2605.22064},
}

Notes

• —Community conversion, not an official Tencent release.
• —The hy_v3.py adapter is included and loaded via trust-remote-code. Inspect it before running if you have concerns.
• —For higher quality on longer generations, consider the Q4 or Q8 variants.