Modelpublic

Tushe/nllb-200-600M-int8-custom-weights-bare-metal

sourceHugging Facecc-by-nc-4.0updated 7mo agoView on Hugging Face

0likes19downloads

Model Card

NLLB-200-600M — Pico Int8 Bare-Metal Format

Custom per-channel int8 quantization of facebook/nllb-200-distilled-600M for bare-metal C inference.

Why Int8 over NF4 for bare-metal?

Property	NF4 (4-bit)	Int8 (this repo)
File size	~660 MB	~850 MB
Dequant complexity	NF4 lookup table + double-quant	Single multiply per row
C code needed	~40 lines	~5 lines
SIMD acceleration	Requires nibble unpacking	Native int8 SIMD on all ARM cores
Accuracy vs fp16	~0.3% degradation	~0.1% degradation

Int8 is simpler, faster, and more accurate on hardware that has native int8 multiply-accumulate (all ARM Cortex-A, all modern x86). NF4 is better only when disk/flash space is the hard constraint.

Tensor Format

Every quantized linear layer is stored as two tensors:

<layer>.weight.int8   # int8  [out_features, in_features]
<layer>.weight.scale  # fp16  [out_features]   (one scale per output row)

Non-quantized tensors (LayerNorm weights/biases, positional embeddings) are stored as float16.

Dequantization (5 lines of C)

// Dequantize row `row` of a weight matrix into float32 buffer `out`
void dequant_row(const int8_t* W, const float* scale, float* out,
                 int row, int in_features) {
    for (int i = 0; i < in_features; i++)
        out[i] = (float)W[row * in_features + i] * scale[row];
}

Reproducing

python

model = AutoModelForSeq2SeqLM.from_pretrained(
    "facebook/nllb-200-distilled-600M",
    torch_dtype=torch.float16, device_map="cpu"
)
# per-channel int8 quantization — see upload script in repo

Part of the Bare Metal project — multilingual translation on ultra-low-resource embedded hardware.