algerian-nlp/DZAIR-FP16

DZAIR-FP16

Half-precision build of algerian-nlp/DZAIR, the 105.3M-parameter encoder for Algerian Darija. Same weights, half the bytes, same outputs to five decimal places.

Fidelity

Measured on a fixed 4×96 token probe with one padded row, against the fp32 release. The full report ships as quant_report.json.

Release gate: cosine ≥ 0.999.

This build was broken until 2026-09-19 and is worth re-downloading. The earlier upload shipped an fp32 config.json alongside half-precision weights, so transformers reloaded them as float32 — the file was half the size but bought nothing at inference. Loading it genuinely in float16 returned an all-zero hidden state, because RMSNorm squared its input in half precision: trunk activations reach about 316, and 316² = 99,856 overflows the float16 maximum of 65,504, so the mean became inf and its reciprocal square root became 0. The norm statistic is now computed in float32 and cast back — float32 outputs are bit-identical to before (verified: max absolute difference exactly 0.0), and float16 works.

Usage

import torch
from transformers import AutoModel, AutoTokenizer

REPO = "algerian-nlp/DZAIR-FP16"
tokenizer = AutoTokenizer.from_pretrained(REPO, trust_remote_code=True)
encoder = AutoModel.from_pretrained(REPO, trust_remote_code=True).eval()

texts = ["يعطيك الصحة خويا", "ya3tik saha khoya, bon courage f projet"]
inputs = tokenizer([t.lower() for t in texts], padding=True, return_tensors="pt")
with torch.inference_mode():
    hidden = encoder(**inputs).last_hidden_state
print(hidden.dtype, hidden.shape)  # torch.float16 torch.Size([2, N, 768])

Lowercase Latin spans before encoding — the vocabulary was built over lowercased Latin and raw uppercase costs about 10% fertility. Arabizi phoneme digits (3, 7, 9) are atomic pieces; never transliterate them away.

Files

Licence

Apache-2.0, inherited from the base model. Read the licence composition on the base card before redistributing derivatives: a permissive grant on the weights makes no claim about the underlying text, most of which has no resolvable licence.