theboulder123/Llama-3.2-1B-Kenyan-Languages-LoRA
011
Llama-3.2-1B-Kenyan-Languages-LoRA ๐ฐ๐ชโจ
Llama-3.2-1B-Kenyan-Languages-LoRA is a lightweight parameter-efficient fine-tuned (PEFT / LoRA) instruction model built on top of unsloth/llama-3.2-1b-instruct-unsloth-bnb-4bit. It is tailored specifically for cross-lingual understanding, reasoning, and bidirectional translation across major low- and mid-resource Kenyan languages alongside English.
๐ Key Features
- Supported Languages:
- English (`eng_Latn`)
- Swahili (`swh_Latn`)
- Kikuyu / Gฤฉkลฉyลฉ (`kik_Latn`)
- Luo / Dholuo (`luo_Latn`)
- Kamba / Kikamba (`kam_Latn`)
- Direct Local-to-Local Capabilities: Trained not only on English-pivot translations, but also directly on indigenous inter-language pairs (e.g., Kikuyu โ Luo, Kamba โ Kikuyu, Luo โ Kamba).
- Strict Evaluation Decontamination: Rigorously decontaminated against FLORES+
devanddevtestevaluation benchmarks to guarantee zero data leakage during zero-shot and few-shot evaluation. - Efficient Edge Deployment: 16-bit LoRA adapter weights (~100โ300 MB) designed for rapid inference on constrained hardware, consumer GPUs, and edge runtimes.
๐ Tokenizer Fertility Metrics
Due to the subword vocabulary distribution of standard Llama tokenizers, sub-Saharan African languages encounter varying degrees of token fragmentation. Evaluated on the reference benchmark corpora:
Note: Given the higher fertility rates for Bantu languages like Kikuyu and Kamba, context windows (`max_seq_length`) should typically be set to 512โ1024 tokens to avoid intermediate sequence truncation.
๐ Quickstart & Inference
You can load and execute this model using Unsloth for 2x faster inference and minimal VRAM consumption:
from unsloth import FastLanguageModel
import torch
max_seq_length = 1024
dtype = None # Auto detection
load_in_4bit = True
# Load base model + LoRA adapters
model, tokenizer = FastLanguageModel.from_pretrained(
model_name="theboulder123/Llama-3.2-1B-Kenyan-Languages-LoRA",
max_seq_length=max_seq_length,
dtype=dtype,
load_in_4bit=load_in_4bit,
)
FastLanguageModel.for_inference(model)
# Translation / Generation Prompt
prompt = """<|begin_of_text|><|start_header_id|>system<|end_header_id|>
You are an expert translator specializing in indigenous Kenyan languages.<|eot_id|><|start_header_id|>user<|end_header_id|>
Translate the following sentence into Kikuyu:
"Dr. Ehud Ur cautioned that the research is still in its early days."<|eot_id|><|start_header_id|>assistant<|end_header_id|>
"""
inputs = tokenizer([prompt], return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=128, use_cache=True)
response = tokenizer.batch_decode(outputs)
print(response[0].split("<|start_header_id|>assistant<|end_header_id|>")[-1].replace("<|eot_id|>", "").strip())
๐ ๏ธ Training Details
Trained by: Abel Kihara (theboulder123)
Base Architecture: Llama-3.2 1B Instruct (BNB 4-bit)
Optimization Framework: Unsloth & Hugging Face TRL (SFTTrainer)
PEFT Method: LoRA
LoRA Rank ($r$): 16
LoRA Alpha: 16
Target Modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Data Sources:
Filtered subsets of allenai/nllb (LASER score thresholding)
Silver pseudo-labeled bidirectional pairs with round-trip chrF verification
Complete decontamination against openlanguagedata/flores_plus
๐ Citation & Attribution
If you use this model or the associated local Kenyan language alignment weights in your research or application, please cite:
Code snippet
@misc{kihara2026llama_kenyan_languages,
author = {Abel Kihara},
title = {Llama-3.2-1B-Kenyan-Languages-LoRA: Parameter-Efficient Translation and Instruction Following for Kenyan Languages},
year = {2026},
publisher = {Hugging Face},
journal = {Hugging Face Model Hub},
howpublished = {\url{[https://huggingface.co/theboulder123/Llama-3.2-1B-Kenyan-Languages-LoRA](https://huggingface.co/theboulder123/Llama-3.2-1B-Kenyan-Languages-LoRA)}}
}
โ๏ธ License
This project is open-sourced under the Apache-2.0 License. Compatible with the Meta Llama 3.2 Community License agreement.