AbdulrahmanFA/saudi-trained-ai-model-8.3M

Saudi Trained AI Model — 8.3M Parameters

A small decoder-only language model trained entirely from scratch on a colloquial Saudi/Gulf Arabic corpus, then instruction-fine-tuned to converse in Saudi dialect. Built end-to-end: custom tokenizer, architecture, pretraining, evaluation, and instruction fine-tuning.

Scope, honestly: at ~8.3M parameters this model captures dialect, register, and conversational format well. It is knowledge-thin and is NOT a factual assistant. It will often produce a fluent, plausible, but factually wrong answer.

Model details

• —Architecture: decoder-only transformer (mini-LLaMA style)
• —Parameters: 8,279,712 (~8.3M)
• —Components: RMSNorm (pre-norm), RoPE, SwiGLU, weight-tied embeddings, no biases
• —Tokenizer: byte-level BPE, vocab 8,000
• —Context length: 256 tokens; trained in bf16

Usage

Install: pip install torch tokenizers numpy Run: python3 sftchat.py --ckpt sftckpt.pt

The SFT model expects this prompt template (handled automatically by sft_chat.py):

السؤال:

{question}

الجواب:

Example — السؤال: كيف حالك؟ -> الجواب: الله يحييك، تمام الحمد لله. تفضّل وش تبي؟

Results

• —Base pretraining: test perplexity ~118 (valid ~109).
• —Instruction tuning: reliably answers in dialect (greetings convincing); factual accuracy limited by answer-side data diversity and small model capacity.

Files

• —sft_ckpt.pt — recommended instruction-tuned model (v1)
• —sftckptv2.pt — variant (1 epoch, higher dropout)
• —ckpt.pt — base pretrained model
• —tokenizer.json — tokenizer; model.py — definition (required to load)
• —sftchat.py / sample.py — inference; train.py / finetune.py / preparedata.py / train_tokenizer.py / eval.py — pipeline

Limitations & intended use

Educational / research / dialect-style generation. NOT for factual answers or financial/medical/legal advice. May generate confident but incorrect content.