Ngongbi/smollm2-blind-spots

SmolLM2-1.7B Blind Spots Dataset

Model Tested

• —Model: HuggingFaceTB/SmolLM2-1.7B
• —Link: https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B
• —Parameters: 1.7B
• —Type: Base model (not finetuned)

How I Loaded the Model

!pip install transformers torch

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_name = "HuggingFaceTB/SmolLM2-1.7B"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.float32)

def generate(prompt, max_new_tokens=100):
    inputs = tokenizer(prompt, return_tensors="pt")
    with torch.no_grad():
        outputs = model.generate(**inputs, max_new_tokens=max_new_tokens, do_sample=False)
    return tokenizer.decode(outputs[0], skip_special_tokens=True)

Blind Spots Found

The model consistently fails in these areas:

• —Math: Gets basic multiplication wrong (15×13 = 2065 instead of 195)
• —Logic: Fails simple syllogism reasoning
• —Factual recall: Avoids naming people directly (e.g. Alexander Fleming)
• —Repetition loops: Repeats the same sentence endlessly
• —Sentiment analysis: Never gives a direct answer
• —African/low-resource language knowledge: Limited understanding of Cameroonian Pidgin

Fine-tuning Recommendations

To fix these blind spots, the model should be fine-tuned on:

• —Math reasoning dataset (e.g. GSM8K) — ~8,000 examples
• —Logic/reasoning chains (e.g. BIG-Bench) — ~5,000 examples
• —African language QA pairs — ~3,000 examples
• —Sentiment classification data — ~5,000 examples
• —Factual QA with named entities — ~5,000 examples

Estimated Dataset Size

~25,000 diverse, high-quality examples total across all categories.

Dataset Structure

• —input — the prompt given to the model
• —expected_output — the correct answer
• —model_output — what the model actually produced
• —category — type of task
• —verdict — type of error made