alabenayed/improved-aya-expanse-8b-cpt-tunisian

Improved Aya Expanse 8B CPT Tunisian

This is a LoRA adapter trained on the Tunisian Arabic dialect using the Aya Expanse 8B base model through Continued Pre-Training (CPT).

Model Details

Model Description

• —Base Model: CohereLabs/aya-expanse-8b (8B multilingual)
• —Training Method: QLoRA (4-bit quantization + LoRA adapter)
• —Language: Tunisian Arabic (Darija)
• —Dataset: Syrinesmati/tunisian-dialect-corpus (1.18M samples → 82.6k packed sequences)
• —Training Steps: 5,113 (1 epoch)
• —Final Eval Loss: 2.089 (↓ 16% improvement from base)
• —License: Apache 2.0

Model Sources

• —Base Model Repository: CohereLabs/aya-expanse-8b
• —Dataset Repository: Syrinesmati/tunisian-dialect-corpus

Uses

Direct Use - Quick Start

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

# Load base model
base_model = AutoModelForCausalLM.from_pretrained(
    "CohereLabs/aya-expanse-8b",
    device_map="auto",
    trust_remote_code=True
)

# Load Tunisian adapter
model = PeftModel.from_pretrained(
    base_model,
    "alabenayed/improved-aya-expanse-8b-cpt-tunisian",
    device_map="auto"
)
model.eval()

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained(
    "alabenayed/improved-aya-expanse-8b-cpt-tunisian",
    trust_remote_code=True
)
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

# Generate with chat template
def generate_prompt(prompt: str, max_new_tokens: int = 160):
    messages = [{"role": "user", "content": prompt}]
    inputs = tokenizer.apply_chat_template(
        messages,
        add_generation_prompt=True,
        tokenize=True,
        return_dict=True,
        return_tensors="pt",
    )
    
    device = next(model.parameters()).device
    inputs = {k: v.to(device) for k, v in inputs.items()}
    
    outputs = model.generate(
        **inputs,
        max_new_tokens=max_new_tokens,
        temperature=0.7,
        do_sample=True,
    )
    
    return tokenizer.decode(outputs[0], skip_special_tokens=True)

# Try with Tunisian prompts
prompts = [
    "عسلامة تحكي تونسي ؟",  # How are you speaking Tunisian?
    "اليوم الطقس مزيان شنوة نجم نعمل؟",  # Today's weather is nice, what can we do?
    "اش نجم نقول لصاحبي باش نهنيه بعيد ميلادو؟",  # What can I say to my friend on his birthday?
]

for prompt in prompts:
    output = generate_prompt(prompt)
    print(f"📝 {prompt}\n💬 {output}\n")

Use Cases

✅ Tunisian Arabic text generation ✅ Conversational AI in Tunisian dialect ✅ Fine-tuning base for downstream tasks ✅ Multilingual models with Tunisian support

Downstream Use

Use this adapter with other Tunisian datasets or for domain-specific fine-tuning tasks.

Training Details

Dataset

• —Source: Syrinesmati/tunisian-dialect-corpus
• —Raw samples: 1,180,174
• —After packing: 82,625 sequences (1024 tokens each)
• —Train/Eval split: ~98.8% / 1.2%

Training Procedure

Configuration

• —Quantization: 4-bit (NF4 + double quantization)
• —LoRA Settings:
• —r=16 (rank)
• —α=32 (scaling)
• —dropout=0.05
• —Optimizer: AdamW
• —Learning Rate: 2e-4 (cosine with warmup)
• —Warmup Steps: 153
• —Batch Size: 1 (with gradient accumulation)
• —Total Steps: 5,113
• —Epochs: 1

Performance

• —Training Duration: ~72 hours on single NVIDIA GPU
• —Final Eval Loss: 2.089
• —Improvement: ~16% loss reduction from base model

Loss Progression

Evaluation Results

Final Metrics

• —Eval Loss: 2.089
• —Perplexity: 8.08
• —Improvement over base: 16% loss reduction

Limitations

⚠️ Best performance on Tunisian Arabic text ⚠️ May require additional fine-tuning for domain-specific tasks ⚠️ Inherits limitations from base Aya Expanse 8B model

Citation

@misc{tunisian-llm-2026,
    author = {MohamedAla-Benayed},
    title = {Improved Aya Expanse 8B CPT Tunisian},
    year = {2026},
    publisher = {Hugging Face},
    howpublished = {\url{https://huggingface.co/alabenayed/improved-aya-expanse-8b-cpt-tunisian}}
}

Acknowledgments

• —Base Model: Cohere - Aya Expanse 8B
• —Dataset: Syrinesmati - Tunisian Dialect Corpus
• —Framework: Hugging Face Transformers & PEFT

Environmental Impact

<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->

Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

• —Hardware Type: [More Information Needed]
• —Hours used: [More Information Needed]
• —Cloud Provider: [More Information Needed]
• —Compute Region: [More Information Needed]
• —Carbon Emitted: [More Information Needed]

Technical Specifications [optional]

Model Architecture and Objective

[More Information Needed]

Compute Infrastructure

[More Information Needed]

Hardware

[More Information Needed]

Software

[More Information Needed]

Citation [optional]

<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->

BibTeX:

[More Information Needed]

APA:

[More Information Needed]

Glossary [optional]

<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->

[More Information Needed]

More Information [optional]

[More Information Needed]

Model Card Authors [optional]

[More Information Needed]

Model Card Contact

[More Information Needed]

Framework versions

• —PEFT 0.19.1