kashfameen/smolLM2-blindspots-kashaf

Blind Spots of HuggingFaceTB/SmolLM2-1.7B

Model Tested

• —Model: HuggingFaceTB/SmolLM2-1.7B
• —Release date: December 2024
• —Parameters: 1.7B
• —Type: Base model (not fine-tuned)

How I Loaded the Model

I used Google Colab with a T4 GPU. Here is the code:

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_name = "HuggingFaceTB/SmolLM2-1.7B"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.float16,
    device_map="auto"
)

def ask_question(question):
    inputs = tokenizer(question, return_tensors="pt").to(model.device)
    outputs = model.generate(
        **inputs,
        max_new_tokens=50,
        do_sample=False,
        pad_token_id=tokenizer.eos_token_id
    )
    full_output = tokenizer.decode(outputs[0], skip_special_tokens=True)
    # Remove the input question from the beginning if present
    if full_output.startswith(question):
        return full_output[len(question):].strip()
    return full_output

Examples of Mistakes

I asked the model 10 questions where it gave incorrect answers. They are listed in data.csv. The mistakes include:

• —Repeating the question instead of answering
• —Giving completely wrong facts (e.g., "A cat is a dog")
• —Providing bad advice for social situations
• —Generating garbled or nonsensical outputs
• —Failing at basic math word problems

How I Would Fix These Errors

To improve the model, I would fine-tune it on a dataset that teaches:

• —Step-by-step instructions (e.g., wikiHow articles)
• —Basic common sense and social norms
• —Elementary math word problems
• —Factual knowledge about animals, seasons, etc.

I would assemble such a dataset by combining:

• —CommonSenseQA for everyday knowledge
• —GSM8K for math word problems
• —WikiHow for procedural tasks
• —A collection of simple Q&A pairs from educational websites

I think a dataset of at least 10,000 to 20,000 diverse examples would significantly reduce these blind spots.

Link to Model

https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B