Jamshed18/smollm2-blind-spots

SmolLM2-1.7B Blind Spots Dataset

Model Tested

HuggingFaceTB/SmolLM2-1.7B

A 1.7B parameter base language model trained by HuggingFace. It is NOT fine-tuned for instruction following or chat — it is a raw base model that completes text.

How I Loaded the Model

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_name = "HuggingFaceTB/SmolLM2-1.7B"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.float32)
model.eval()

def run_model(prompt, max_new_tokens=80):
    inputs = tokenizer(prompt, return_tensors="pt")
    with torch.no_grad():
        outputs = model.generate(**inputs, max_new_tokens=max_new_tokens,
                                  do_sample=False, pad_token_id=tokenizer.eos_token_id)
    generated = outputs[0][inputs["input_ids"].shape[1]:]
    return tokenizer.decode(generated, skip_special_tokens=True)

Run on Google Colab with a free T4 GPU.

Dataset Description

This dataset contains 10 diverse prompts where SmolLM2-1.7B makes incorrect or poor predictions. Categories tested: arithmetic, logic, factual recall, spelling, common sense, negation, counting, text reversal, code interpretation, and analogies.

Each row contains:

• —id: test number
• —category: type of reasoning tested
• —input: the prompt given to the model
• —expected_output: the correct answer
• —model_output: what the model actually produced

What Kind of Fine-Tuning Would Fix These Errors?

The model struggles most with:

1. 1.Exact reasoning tasks (math, counting, reversal) — needs fine-tuning on chain-of-thought datasets like GSM8K or MATH.
2. 2.Common sense and negation — needs datasets like CommonsenseQA or HellaSwag formatted as instruction-response pairs.
3. 3.Instruction following generally — since this is a base model, many errors disappear with basic instruction fine-tuning (SFT) on datasets like OpenHermes or Alpaca.

How Big a Dataset Would You Need?

• —For basic instruction following: ~10,000–50,000 examples (e.g. Alpaca-style)
• —For math/reasoning improvements: ~5,000–20,000 chain-of-thought examples
• —For common sense: ~10,000 examples from CommonsenseQA-style data

A combined dataset of ~30,000–50,000 high-quality, diverse instruction pairs would likely produce meaningful improvement across all these blind spot categories.