EhabSuliman/my_awesome_eli5_clm-model

myawesomeeli5_clm-model

A Causal Language Model fine-tuned on Arabic text, based on distilbert/distilgpt2. Fine-tuned by EhabSuliman as part of an LLM course project.

Model Description

• —Model type: Causal Language Model (CLM)
• —Base model: distilbert/distilgpt2
• —Language: English
• —Fine-tuned by: EhabSuliman
• —License: Apache 2.0

Intended Uses & Limitations

Intended uses:

• —Arabic text generation
• —Language modeling research
• —Educational/learning purposes

Limitations:

• —Trained on a relatively small dataset
• —Loss is still relatively high (3.8027), meaning the model may generate inaccurate or repetitive text
• —Not recommended for production use without further fine-tuning

How to Use

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("EhabSuliman/my_awesome_eli5_clm-model")
model = AutoModelForCausalLM.from_pretrained("EhabSuliman/my_awesome_eli5_clm-model")

prompt = "Somatic hypermutation allows the immune system to"
inputs = tokenizer(prompt, return_tensors="pt").input_ids
outputs = model.generate(
    inputs,
    max_new_tokens=100,
    do_sample=True,
    top_k=50,
    top_p=0.95
)
print(tokenizer.batch_decode(outputs, skip_special_tokens=True))

Training and Evaluation Data

Fine-tuned on an Arabic text dataset using the ELI5 (Explain Like I'm 5) format.

Training Procedure

Training Hyperparameters

The following hyperparameters were used during training:

• —Learning rate: 2e-05
• —Train batch size: 8
• —Eval batch size: 8
• —Seed: 42
• —Optimizer: AdamW (fused) with betas=(0.9, 0.999), epsilon=1e-08
• —LR scheduler: Linear
• —Epochs: 3

Training Results

The model shows steady improvement across epochs with validation loss decreasing from 3.8101 → 3.8027.

Framework Versions

• —Transformers 5.9.0
• —PyTorch 2.11.0+cu128
• —Datasets 4.0.0
• —Tokenizers 0.22.2

Author

Ehab Suliman — Machine Learning Engineer 🔗 HuggingFace Profile