AyoubChLin/lfm2.5-8b-saudi-dialect

LFM2.5-8B Saudi Dialect

AyoubChLin/lfm2.5-8b-saudi-dialect is a Saudi Arabic conversational fine-tune of LiquidAI/LFM2.5-8B-A1B.

The model was fine-tuned to produce more natural Saudi dialect responses in chat-style conversations. It is intended for Arabic dialogue, informal Saudi phrasing, and assistant-style responses using a Saudi Arabic system prompt.

Model Details

Intended Use

This model is intended for Saudi Arabic conversational use cases, including:

• —Saudi dialect chatbots
• —Arabic assistant responses with Saudi phrasing
• —Dialogue generation
• —Informal Saudi Arabic conversation
• —Domain-specific Saudi Arabic assistant prototypes

Example system prompt used during fine-tuning:

أنت مساعد مفيد يتحدث باللهجة السعودية.

Dataset

The model was fine-tuned on:

HeshamHaroon/saudi-dialect-conversations

The dataset contains multi-turn Saudi Arabic conversations with metadata such as scenario, topic, complexity, and English summary. During preprocessing, each conversation was rendered with the model chat template. A Saudi Arabic system message was injected when missing.

Example conversational style includes casual Saudi phrases such as:

هلا والله
وش سالفتك؟
ايه والله
الله يعطيك العافية

Training Setup

The model was trained with supervised fine-tuning using LoRA adapters. The base model was loaded in bf16 without 4-bit quantization, and Flash Attention 2 was enabled.

LoRA Configuration

Target Modules

LoRA was applied to the following modules:

q_proj
k_proj
v_proj
out_proj
in_proj
conv.in_proj
conv.out_proj
gate_proj
up_proj
down_proj

Training Hyperparameters

Training Environment

Note: The notebook was prepared for an A100 target, but the recorded run was executed on an NVIDIA H200 with 150.1 GB VRAM.

Training Results

Training completed successfully for 6 epochs and 330 optimization steps.

Evaluation Progress

Notes on the Results

The training loss decreased consistently during the run, from 4.6616 at the first logged step to 1.0633 at step 330. Train mean token accuracy also improved steadily, reaching 0.7597 at the final logged step.

Validation performance improved early in training, with the best validation loss appearing at step 140 and the best evaluation mean token accuracy appearing at step 210. After that point, the training loss continued to decrease while validation loss increased slightly. This suggests that the final checkpoint is more strongly adapted to the training distribution, while an earlier checkpoint around steps 140–210 may generalize slightly better on the small held-out validation split.

Because the evaluation split contains only 71 examples, these metrics should be treated as training diagnostics rather than a full benchmark. A stronger evaluation should include human review by native Saudi Arabic speakers, dialect naturalness scoring, response helpfulness scoring, safety checks, and comparisons against the base model.

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "AyoubChLin/lfm2.5-8b-saudi-dialect"

tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    trust_remote_code=True,
)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)

messages = [
    {
        "role": "system",
        "content": "أنت مساعد مفيد يتحدث باللهجة السعودية."
    },
    {
        "role": "user",
        "content": "هلا، وش تنصحني أسوي إذا أبي أتعلم برمجة؟"
    }
]

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)

inputs = tokenizer(text, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=256,
    temperature=0.7,
    top_p=0.9,
    do_sample=True,
    repetition_penalty=1.05,
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Recommended Generation Settings

For natural Saudi conversational responses:

generation_config = {
    "max_new_tokens": 256,
    "temperature": 0.7,
    "top_p": 0.9,
    "do_sample": True,
    "repetition_penalty": 1.05,
}

For more deterministic assistant-style responses:

generation_config = {
    "max_new_tokens": 256,
    "temperature": 0.3,
    "top_p": 0.8,
    "do_sample": True,
    "repetition_penalty": 1.05,
}

Limitations

This model is a specialized Saudi dialect fine-tune and may not be optimal for:

• —Non-Saudi Arabic dialects
• —Formal Modern Standard Arabic tasks
• —Safety-critical domains
• —Legal, medical, or financial advice
• —Factual questions requiring up-to-date information
• —Long-context reasoning beyond the fine-tuning distribution

The model may also reflect biases, inaccuracies, or style artifacts present in the training dataset.

Evaluation

The reported evaluation used validation loss and mean token accuracy on a small held-out split of 71 examples.

For future releases, stronger evaluation should include:

• —Human evaluation by native Saudi Arabic speakers
• —Dialect naturalness scoring
• —Response helpfulness scoring
• —Safety evaluation
• —Comparison against the base model
• —Saudi dialect benchmark prompts
• —Evaluation on prompts outside the training dataset distribution

Citation

Base model:

@misc{liquidai_lfm25_8b_a1b,
  title = {LFM2.5-8B-A1B},
  author = {Liquid AI},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/LiquidAI/LFM2.5-8B-A1B}}
}

Dataset:

@misc{saudi_dialect_conversations,
  title = {Saudi Dialect Conversations},
  author = {HeshamHaroon},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/datasets/HeshamHaroon/saudi-dialect-conversations}}
}

Disclaimer

This model is provided for research and development purposes. Outputs should be reviewed before use in production systems, especially in sensitive or high-stakes applications.