kikwaib/mt5-base-kenswquad-extractive

mT5-Base KenSwQuAD Extractive (Stage 2)

Model Summary

This model is an intermediate research checkpoint developed as part of the KenSwQuAD Hierarchical Curriculum Learning project for Swahili Question Answering.

It is a google/mt5-base model that has undergone two stages of curriculum training:

1. 1.Stage 1: Fine-tuned on English SQuAD v2 to learn QA task structure
2. 2.Stage 2 (Current): Fine-tuned on extractive KenSwQuAD to learn Swahili morphology and syntax

The model learns to extract answer spans from Swahili text contexts given a question.

This is Stage 2 of a 3-Stage Pipeline:

1. 1.✅ Stage 1: Structural Transfer (English SQuAD) → Learned "How to Answer"
2. 2.✅ Stage 2 (Current): Morphological Alignment (Extractive KenSwQuAD) → Learned Swahili Syntax
3. 3.⏳ Stage 3: Generative Refinement (Abstractive KenSwQuAD) → Will Learn Reasoning

Model Details

Intended Use

Primary Use Cases

• —Swahili Question Answering: Extract answers from Swahili text given a question
• —Transfer Learning: Serve as initialization for Swahili NLP tasks
• —Research: Baseline for low-resource language QA experiments

How to Use

The model accepts input in the format: question: <question_text> context: <context_text>

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

model_name = "kikwaib/mt5-base-kenswquad-extractive"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSeq2SeqLM.from_pretrained(model_name)

context = "Kenya ni nchi ya Afrika Mashariki. Nairobi ni mji mkuu wa Kenya. Kenya ina wakazi zaidi ya milioni 50."
question = "Mji mkuu wa Kenya ni upi?"

input_text = f"question: {question} context: {context}"
inputs = tokenizer(input_text, return_tensors="pt", max_length=1024, truncation=True)

outputs = model.generate(**inputs, max_length=128)
answer = tokenizer.decode(outputs[0], skip_special_tokens=True)

print(answer)
# Expected Output: "Nairobi"

Limitations

• —Optimized for extractive QA where the answer is a substring of the context
• —May struggle with abstractive questions requiring reasoning or inference
• —Performance may vary on domains outside the KenSwQuAD training data (primarily news and Wikipedia)

Training Data

Dataset: KenSwQuAD

The model was fine-tuned on the extractive subset of KenSwQuAD (Kenya Swahili Question Answering Dataset).

Partitioning Logic

QA pairs were classified as extractive if the answer text appears as an exact substring (case-insensitive) of the context. Otherwise, they were classified as abstractive and reserved for Stage 3 training.

Training Procedure

Hardware

Hyperparameters

Training Results

★ Best checkpoint (selected based on highest BLEU score)

Training Dynamics

• —Total Training Time: 69.9 minutes (1h 8m 23s)
• —Total Steps: 5,710
• —Final Training Loss: 0.4857

Key Observations:

1. 1.Initial Adaptation (Epochs 1-3): BLEU dipped as model transitioned from English to Swahili patterns
2. 2.Rapid Improvement (Epochs 4-8): Strong gains as Swahili morphology was learned
3. 3.Best Performance: Epoch 8 achieved peak BLEU of 48.99
4. 4.Slight Overfitting (Epochs 9-10): Validation loss increased while training loss continued decreasing

Evaluation Results

Framework Versions

Citation

comming soon

Related Models

Training Date: December 20, 2025