zhangsq-nju/MobileLLM-350M-EdgeRazor-4bit

<div align="center"> <br/> <img src="./asset/Logo-HF.png" alt="EdgeRazor Logo" width="60%"> <h3> EdgeRazor for Lightweight LLMs </h3>

<p> <a href="https://arxiv.org/abs/2605.04062" target="blank"> <img src="https://img.shields.io/badge/arXiv-EdgeRazor-b31b1b?style=flat&logo=arxiv" alt="arXiv EdgeRazor"> </a> <a href="https://github.com/zhangsq-nju/EdgeRazor" target="blank"> <img src="https://img.shields.io/badge/GitHub-EdgeRazor-blue?style=flat&logo=github" alt="GitHub EdgeRazor"> </a> <a href="https://pypi.org/project/edgerazor/" target="blank"> <img src="https://img.shields.io/pypi/v/edgerazor?style=flat&logo=pypi" alt="PyPI EdgeRazor"> </a> </p>

</div>

<h1>MobileLLM-350M-EdgeRazor-4bit</h1>

Contents

• —Contents
• —Model Overview
• —Model Bit-Widths
• —Model Performance
• —Quickstart
• —Citation

Model Overview

• —Base Model: facebook/MobileLLM-ParetoQ-350M-BF16
• —Training: zhangsq-nju/EdgeRazor
• —Quantization: 4-bit for all embedding, decoder, and lm_head layers

Model Bit-Widths

Model Performance

Quickstart

It is recommended to ensure that EdgeRazor is installed in advance for weight-activation quantization. The provided weights are already quantized (quantizedweights*scalingbf16); to enable activation and KV cache quantization, set trust_remote_code=True in the model configuration.

from transformers import AutoModelForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(
    "zhangsq-nju/MobileLLM-ParetoQ-350M-BF16-EdgeRazor-4bit",
    use_fast=False
)
model = AutoModelForCausalLM.from_pretrained(
    "zhangsq-nju/MobileLLM-ParetoQ-350M-BF16-EdgeRazor-4bit", 
    trust_remote_code=True
)

Note that the default tokenizer does not contain special tokens. For example you can use:

tokenizer.add_special_tokens(
    {
        "eos_token": "</s>",
        "bos_token": "<s>",
        "unk_token": "<unk>",
    }
)

Citation

If you find our project useful in your research, please consider kindly citing our papers ✏️:

@article{zhangsh-edgerazor,
  title={{EdgeRazor}: A Lightweight Framework for Large Language Models via Mixed-Precision Quantization-Aware Distillation},
  author={Shu-Hao Zhang and Le-Tong Huang and Xiang-Sheng Deng and Xin-Yi Zou and Chen Wu and Nan Li and Shao-Qun Zhang},
  year={2026},
  journal={arXiv preprint arXiv:2605.04062}
}