Nunsi/afriquegemma-4b-blind-spots

Blind Spots of AfriqueGemma-4B

Overview

This dataset documents failure cases (“blind spots”) of the language model AfriqueGemma-4B across a variety of tasks.

The goal is to identify situations where the model produces incorrect, unreliable, or poorly formatted responses, even for relatively simple prompts.

Each example in the dataset contains the following fields:

• —Id – a unique identifier for each probe example
• —category – the type of capability being tested (e.g., reasoning, sorting, instruction following)
• —input – the prompt given to the model
• —expected_output – the correct answer the model should produce
• —blind_spot – the capability or reasoning skill being evaluated
• —why_incorrect – explanation of the type of failure the probe is designed to detect

The prompts were intentionally designed to test several capabilities, including:

• —factual recall
• —arithmetic reasoning
• —instruction following
• —strict output formatting
• —structured extraction
• —logical reasoning
• —abstention when information is unavailable

The dataset therefore provides insight into systematic weaknesses in the model’s behavior.

Model Tested

The model evaluated in this experiment is:

McGill-NLP/AfriqueGemma-4B

Model page: https://huggingface.co/McGill-NLP/AfriqueGemma-4B

According to the model card, AfriqueGemma-4B is a causal language model derived from:

google/gemma-3-4b-pt

through additional pretraining focused on improving performance on African languages while maintaining strong multilingual capabilities.

The model contains approximately 4 billion parameters, which falls within the assignment requirement of 0.6B–6B parameters.

How the Model Was Loaded

The model was evaluated in Google Colab using the Hugging Face `transformers` library. Because the model has around 4 billion parameters, I used 4-bit quantization with `bitsandbytes` to reduce GPU memory usage and allow the model to run efficiently on the available hardware.

The following steps were used:

1. 1.Install the required libraries (transformers, accelerate, bitsandbytes, datasets, and pandas).
2. 2.Configure 4-bit quantization using BitsAndBytesConfig.
3. 3.Load the tokenizer associated with the model.
4. 4.Load the model using AutoModelForCausalLM, enabling automatic device placement on the GPU.

Below is the exact code used to load the model.

!pip -q install -U transformers accelerate bitsandbytes datasets pandas

!pip install -U bitsandbytes>=0.46.1

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig

MODEL_ID = "McGill-NLP/AfriqueGemma-4B"

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
)

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    device_map="auto",
    quantization_config=bnb_config,
    torch_dtype=torch.bfloat16,
)

print("Loaded", MODEL_ID)

What Kind of Dataset Could Improve the Model?

The errors observed suggest that the model struggles with:

• —strict instruction following
• —exact output formatting
• —structured data extraction
• —simple logical reasoning
• —abstaining when information is missing

To address these issues, the model could benefit from supervised fine-tuning on datasets that emphasize precision and format compliance.

Such a dataset should include tasks like:

• —arithmetic problems with verified answers
• —sorting and list manipulation tasks
• —structured extraction tasks requiring valid JSON output
• —logical reasoning problems
• —examples where the correct behavior is refusing or abstaining

Each example should explicitly specify:

prompt
expected_output
formatting_constraints

This would help the model learn to produce precise and instruction-compliant outputs.

How Such a Dataset Could Be Assembled

1. Synthetic Data Generation

A large portion of the dataset could be generated programmatically.

Examples include:

• —arithmetic calculations
• —percentage problems
• —sorting tasks
• —structured extraction prompts
• —formatting-constrained outputs

Synthetic generation allows the creation of large quantities of clean, automatically verified data where the correct answer is known.

2. Existing Public Datasets

Additional training data could be obtained from existing benchmarks such as:

• —mathematical reasoning datasets
• —logical reasoning benchmarks
• —structured information extraction datasets
• —instruction-following datasets

Combining synthetic data with curated benchmark datasets would produce a more diverse and robust training corpus.

Estimated Dataset Size

A dataset containing approximately:

20,000 – 100,000 examples

would likely be sufficient to significantly improve these capabilities during supervised fine-tuning.

This size would allow the model to observe many examples of:

• —reasoning tasks
• —strict formatting requirements
• —structured outputs
• —abstention cases

Ensuring a balanced distribution across task categories would help prevent overfitting to any single task type.

Summary

This dataset highlights several blind spots in AfriqueGemma-4B, particularly in tasks requiring strict formatting, structured outputs, and precise instruction following.

By fine-tuning the model on datasets emphasizing these behaviors, it is likely that the reliability and correctness of its outputs could be substantially improved.