rudrakshrakeshzodage/nllb-200-distilled-600M-int8-cpu

NLLB-200 Distilled 600M — INT8 CPU Quantized (PyTorch Dynamic)

[!NOTE] 🌐 Live Web Demo: Try 100% offline in your browser at `huggingface.co/spaces/rudrakshrakeshzodage/nllb-200-onnx-browser-demo`

[!NOTE] This model is part of a suite of optimized/quantized versions of the base model. Other variants in this direction: FP32 (Full Precision / Base): [`facebook/nllb-200-distilled-600M`](https://huggingface.co/facebook/nllb-200-distilled-600M) FP16 (Half Precision): `facebook/nllb-200-distilled-600M` INT8 (Dynamic Quantization - CPU): [`rudrakshrakeshzodage/nllb-200-distilled-600M-int8-cpu`](https://huggingface.co/rudrakshrakeshzodage/nllb-200-distilled-600M-int8-cpu) (Current) NF4 (4-bit GPU Quantization): `rudrakshrakeshzodage/nllb-200-distilled-600M-nf4-4bit-gpu`

Base model: `facebook/nllb-200-distilled-600M` Quantization: PyTorch INT8 Dynamic (torch.quantization.quantize_dynamic) Languages: 200 (all NLLB-200 supported languages)

💾 Model Size Reduction Comparison (FP32 vs FP16 vs INT8 vs NF4)

[image]

📈 200+ Language Performance Benchmark Chart

[image]

Quantization Architecture Details

• —Target Layers: torch.nn.Linear inside Encoder & Decoder Self-Attention / FFN modules.
• —Weight Precision: qint8 (8-bit signed integers).
• —Activation Quantization: Calculated dynamically at inference runtime per batch.
• —No GPU Needed: Optimized for x86 CPU SIMD / AVX2 / AVX-512 instructions.

📊 Model Performance & Benchmarks

💻 Hardware Requirements

• —CPU: Intel Core i3 (10th Gen+), AMD Ryzen 3 (3000U+), Apple Silicon (M1+)
• —RAM: 4 GB System RAM minimum (800 MB active model memory)
• —GPU: None required (100% CPU inference)

🚀 Usage

import torch
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

# Load base architecture
model = AutoModelForSeq2SeqLM.from_pretrained(
    "facebook/nllb-200-distilled-600M"
)

# Apply PyTorch INT8 Dynamic Quantization
model = torch.quantization.quantize_dynamic(
    model, {torch.nn.Linear}, dtype=torch.qint8
)

tokenizer = AutoTokenizer.from_pretrained(
    "YOUR_USERNAME/nllb-200-distilled-600M-int8-cpu"
)

# Translate English to Hindi (hin_Deva)
inputs = tokenizer("Hello world, how are you?", return_tensors="pt")
translated_tokens = model.generate(
    **inputs, 
    forced_bos_token_id=tokenizer.lang_code_to_id["hin_Deva"], 
    max_length=100
)
print(tokenizer.batch_decode(translated_tokens, skip_special_tokens=True)[0])

📜 Citation & Credits

Quantization research and benchmarking by RudrakshRakeshZodage.