Venugopalan2610/OLMoE-1B-7B-0125-QTIP-2bit

OLMoE-1B-7B-0125 — QTIP 2-Bit (W2A16)

2-bit weight-only quantization of allenai/OLMoE-1B-7B-0125 via per-expert trellis-coded quantization (QTIP BlockLDLQ with per-expert Hessian calibration).

Key Numbers

What This Is

A 7-billion-parameter Mixture-of-Experts model compressed to 2 bits per weight using QTIP's trellis-coded quantization with routing-conditioned per-expert Hessian calibration. The model fits entirely in GPU memory on devices with as little as 4 GB VRAM and generates at 13 tok/s on a laptop GPU.

This is a base model (not instruction-tuned). It performs text completion, not chat.

How to Run

Requires our llama.cpp fork with QTIP 2-bit support. Important: use the `qtip-olmoe-2bit` branch.

With CUDA (GPU inference):

git clone -b qtip-olmoe-2bit https://github.com/Venugopalan2610/llama.cpp.git cd llama.cpp cmake -B build -DGGML_CUDA=ON cmake --build build --config Release -j ./build/bin/llama-completion -m olmoe-qtip-2b-v2.gguf \ -p "Mixture of Experts models use a routing mechanism to" \ -n 100 --temp 0.7 --repeat-penalty 1.1

CPU-only (no CUDA required):

git clone -b qtip-olmoe-2bit https://github.com/Venugopalan2610/llama.cpp.git cd llama.cpp cmake -B build cmake --build build --config Release -j ./build/bin/llama-completion -m olmoe-qtip-2b-v2.gguf \ -p "Your prompt here" -n 100

Expert offload (ultra-low-VRAM devices, ~4 tok/s):

./build/bin/llama-completion -m olmoe-qtip-2b-v2.gguf \ --qtip-expert-offload \ -p "Your prompt here" -n 100

Method

We collect routing-conditioned per-expert input Hessians from only the tokens each expert actually receives during a calibration pass, producing 2048 distinct expert Hessians for the full model. These feed into an unmodified QTIP BlockLDLQ pipeline (HYB bitshift code, L=16, V=2, Tx=Ty=16, Q=9) with random Hadamard transform preprocessing. No LUT fine-tuning, no codebook modifications.

Quantization Details

• —Quantization method: QTIP BlockLDLQ with per-expert Hessian calibration
• —Bits per weight: ~2.125 (2 bits + trellis overhead)
• —Calibration data: 2048 sequences x 1024 tokens from C4 English train
• —Attention quantization: Same 2-bit method, shared Hessian (not routing-conditioned)
• —Router and embeddings: Kept in f32

Limitations

• —This is a base model. For chat/instruction-following, use an instruction-tuned variant (not yet available at 2-bit).
• —Generation quality is noticeably degraded compared to fp16 on complex reasoning tasks (see ARC-Challenge retention of 90.3%).
• —Expert offload mode runs at ~4 tok/s due to CPU-GPU transfer overhead.

Citation

Technical report forthcoming on arxiv.

License

Apache 2.0 (same as the base OLMoE model).

Acknowledgments

Built on QTIP (Tseng et al., NeurIPS 2024) and OLMoE (Muennighoff et al., 2024).