CoolFace
Modelpublic

MercuriusDream/Nanbeige4.2-3B-mlx-8bit

sourceHugging Faceapache-2.0updated 2mo agoView on Hugging Face
5likes421downloads
Model Card

Nanbeige4.2-3B — MLX 8bit

Quantized to 8 bits per weight, group size 64 (4.2 GB, 8.5 effective bpw).

Converted from Nanbeige/Nanbeige4.2-3B using mlx-lm. Logits of the MLX port match the reference implementation to a max abs difference of 9.5e-05 (fp32, CPU).

Nanbeige4.2-3B is a Looped Transformer: its 22 decoder layers run twice per forward pass (num_loops=2) with shared weights, giving 4B total parameters with the capacity of a deeper model.

Requirements

The nanbeige model type is pending upstream support in mlx-lm (ml-explore/mlx-lm#1597). Until that merges, install from the PR branch:

bash
pip install git+https://github.com/MercuriusDream/mlx-lm.git@add-nanbeige-model

Use

bash
mlx_lm.generate --model MercuriusDream/Nanbeige4.2-3B-mlx-8bit --prompt "hello"
python
from mlx_lm import load, generate

model, tokenizer = load("MercuriusDream/Nanbeige4.2-3B-mlx-8bit")
prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "hello"}], add_generation_prompt=True
)
print(generate(model, tokenizer, prompt=prompt))

All variants