PeytonT/mobilellm-paretoq-350m-modelstack-nf4-iphone

Model Details

The quantized MobileLLM models is introduced in: "ParetoQ: Scaling Laws in Extremely Low-bit LLM Quantization"

Model Developer: Meta

Model Architecture: ParetoQ is the first unified framework that facilitates rigorous comparisons across 1-bit, 1.58-bit, 2-bit, 3-bit, and 4-bit quantization settings. By optimizing training schemes and refining quantization functions, ParetoQ surpasses all previous methods tailored to specific bit widths. Specifically, the 1.58-bit ParetoQ LLaMA-3 8B model reduces the performance gap to full precision by relatively 37.8% compared to the 1-bit Era’s 1.58-bit LLaMA-3 8B model, while using only 30% of the training tokens.

<div align=center> <img width=50% src="https://github.com/facebookresearch/ParetoQ/blob/main/mainresultternary.jpg?raw=true"/> </div>

<div align=center> <img width=100% src="https://github.com/facebookresearch/ParetoQ/blob/main/mainresult234bit.jpg?raw=true"/> </div>

With the SoTA points obtained through ParetoQ, we are able to improve the scaling law analysis. Figure (a) (b) demonstrates that sub-4-bit quantization, including binary, ternary, 2-bit, and 3-bit, often outperform 4-bit quantization. Notably, 2-bit and ternary models reside on the Pareto frontier. When considering hardware-friendliness and real-time speed, we generally recommend exploring 2-bit quantization for on-device applications.

<div align=center> <img width=100% src="https://github.com/facebookresearch/ParetoQ/blob/main/mainresultscaling_law.jpg?raw=true"/> </div>

How to use

We are providing 2 ways to run the model:

HuggingFace

ParetoQ codebase

HuggingFace

To load the pretrained model for further finetuning or evaluation:

from transformers import AutoModelForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("facebook/MobileLLM-ParetoQ-125M-BF16", use_fast=False)
model = AutoModelForCausalLM.from_pretrained("facebook/MobileLLM-ParetoQ-125M-BF16", trust_remote_code=True)

Note that the default tokenizer does not contain special tokens. For example you can use:

tokenizer.add_special_tokens(
    {
        "eos_token": "</s>",
        "bos_token": "<s>",
        "unk_token": "<unk>",
    }
)

ParetoQ codebase

We provide the pretraining code in https://github.com/facebookresearch/ParetoQ

> git clone https://github.com/facebookresearch/ParetoQ
> pip3 install torch
> pip install -r requirement.txt

# Specify the data path and the pre-trained full-precision model path in run_train.sh file
> Run `bash 1_run_train.sh $w_bit` E.g. `bash 1_run_train.sh 2` for 2-bit weight quantization.

Evaluation

We evaluate ParetoQ models using zero-shot commonsense reasoning task accuracy and WikiText-2 perplexity.

Acknowledgement

This code is partially based on HuggingFace Transformers repo under Apache License.

Citation

If you find our code useful for your research, please consider citing:

@article{liu2025paretoq, title={ParetoQ: Scaling Laws in Extremely Low-bit LLM Quantization}, author={Liu, Zechun and Zhao, Changsheng and Huang, Hanxian and Chen, Sijia and Zhang, Jing and Zhao, Jiawei and Roy, Scott and Jin, Lisa and Xiong, Yunyang and Shi, Yangyang and others}, journal={arXiv preprint arXiv:2502.02631}, year={2025} }

License

ParetoQ is FAIR NC licensed as of now