Sumannnnn12345678/smollm2-blindspots

SmolLM2-1.7B Blind Spots Dataset

Model Tested

HuggingFaceTB/SmolLM2-1.7B

A 1.7B parameter base language model released in 2024 by HuggingFace.

How I Loaded the Model

Loaded in Google Colab (T4 GPU, free tier) using the following code:

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "HuggingFaceTB/SmolLM2-1.7B"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

def generate(prompt, max_new_tokens=200):
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    with torch.no_grad():
        outputs = model.generate(
            **inputs,
            max_new_tokens=max_new_tokens,
            do_sample=False,
            pad_token_id=tokenizer.eos_token_id
        )
    generated = outputs[0][inputs["input_ids"].shape[1]:]
    return tokenizer.decode(generated, skip_special_tokens=True)

Dataset Description

This dataset contains 10 diverse inputs where SmolLM2-1.7B made incorrect or incomplete predictions. Each row contains:

• —input: the prompt given to the model
• —expected_output: the correct answer
• —model_output: what the model actually produced
• —category: the type of task

Key Blind Spots Found

1. 1.Trick reasoning — fails questions where the wording is deliberately misleading
2. 2.Multi-step math — gets classic reasoning puzzles wrong (bat & ball)
3. 3.Code execution — cannot trace through code and predict output
4. 4.Instruction following — often ignores the actual task and generates related but wrong content
5. 5.Creative generation — cannot write structured creative text like haikus
6. 6.Repetition loops — base model has no stopping mechanism, repeats forever

What Fine-Tuning Dataset Would Fix This?

The model needs a mix of:

• —Chain-of-thought reasoning data (e.g. GSM8K) for math and logic
• —Instruction-following data (e.g. FLAN, Alpaca) so it follows tasks properly
• —Code execution traces (e.g. HumanEval) for code reasoning
• —Adversarial/trick questions with correct explanations for reasoning blind spots

How to Assemble Such a Dataset?

• —Use existing open datasets: GSM8K, ARC, HumanEval, FLAN
• —Generate additional examples using GPT-4 or Claude with chain-of-thought annotations
• —Manually curate adversarial and trick questions with correct step-by-step explanations

How Big Does the Dataset Need to Be?

• —Minimum for noticeable improvement: 5,000 - 10,000 examples
• —For robust generalization: 50,000+ examples with diverse coverage across all failure categories