yaqeenalradi/smollm3-blind-spots

Blind spots of SmolLM3-3B-Base on simple instructions

Dataset description

This dataset collects failure cases (“blind spots”) of the HuggingFaceTB/SmolLM3-3B-Base language model on very simple prompts. The prompts cover factual questions, arithmetic, comparisons, translation, and strict instruction-following such as “answer with one word only” or “yes or no only”.

Each row records:

• —the input prompt,
• —the expected output,
• —the model’s actual output, and
• —a short error type and note explaining why the output is considered wrong.

The goal is to make a small, human-curated evaluation set that highlights where a modern 3B parameter base model still fails on easy tasks.

Model tested

• —Model name: HuggingFaceTB/SmolLM3-3B-Base
• —Model card: https://huggingface.co/HuggingFaceTB/SmolLM3-3B-Base

How I loaded the model (Colab code)

!pip uninstall -y transformers
!pip install -q "transformers==4.53.0" "accelerate>=1.0.0" sentencepiece

import torch
import pandas as pd
from transformers import AutoTokenizer, AutoModelForCausalLM, set_seed

model_id = "HuggingFaceTB/SmolLM3-3B-Base"
set_seed(42)

print("Model:", model_id)
print("CUDA available:", torch.cuda.is_available())

tokenizer = AutoTokenizer.from_pretrained(model_id)

if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype="auto",
    device_map="auto"
)

print("Model loaded successfully.")

def generate_text(prompt, max_new_tokens=80, temperature=0.0):
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

    if temperature == 0.0:
        outputs = model.generate(
            **inputs,
            max_new_tokens=max_new_tokens,
            do_sample=False,
            pad_token_id=tokenizer.eos_token_id
        )
    else:
        outputs = model.generate(
            **inputs,
            max_new_tokens=max_new_tokens,
            do_sample=True,
            temperature=temperature,
            top_p=0.95,
            pad_token_id=tokenizer.eos_token_id
        )

    new_tokens = outputs[0][inputs["input_ids"].shape[1]:]
    return tokenizer.decode(new_tokens, skip_special_tokens=True).strip()

Dataset structure

Data files

blindspotsdataset.csv: main table of failure cases.

Data fields

Each row has:

• —input — the exact prompt given to the model.

• —expected_output — the answer I would consider correct or acceptable.

• —model_output — the raw text generated by the model.

• —error_type — a short label (for example, instruction-following, translation failure, reasoning inconsistency).

• —notes — a sentence explaining why the output is judged wrong.

There is no train/validation/test split; this is a small curated evaluation set.

Example row

input: "Answer with only one word: What color is the sky on a clear day?" expectedoutput: "Blue" modeloutput: "A: Blue\nB: Green\nC: Red\nD: Black\n\nThe right option is A: Blue" error_type: "instruction-following / format violation" notes: "The model includes multiple-choice formatting instead of a single word."

What kind of dataset could fix these errors?

The main blind spots observed:

• —Instruction-following / output format The model often ignores constraints like “one word only”, “yes or no only”, “number only”, or “repeat exactly this and nothing else”.

• —Directness of answers For simple comparisons (9.8 vs 9.11, 2.09 vs 2.9), the model starts a long explanation but fails to clearly state the final answer.

• —Translation to Arabic and other languages For prompts like “Translate to Arabic: Good morning.” the model sometimes outputs unrelated English text or copies the input instead of translating, suggesting gaps in low-resource language coverage. [page:1]

• —Short factual and arithmetic reasoning A few cases show inconsistent reasoning, such as mixing buying and selling chickens in the same answer.

To improve these behaviors, I would finetune the model on a supervised instruction dataset that emphasizes:

1.Strict format compliance

Prompts explicitly describing the required format. Gold outputs that obey the format exactly (one word, yes/no, one number, exact string match, etc.). Automatic checks that reject examples where the target output itself violates the format.

2.Short, direct answers instead of essays

Many QA examples where the correct output is a single token or short phrase. Training loss focused on concise answers rather than long chain-of-thought, to reduce over-explaining on trivial queries.

3.High-quality parallel data for under-served languages (e.g., Arabic)

Sentence-level translation pairs: English ↔ Arabic, plus some French ↔ English for multi-lingual coverage. Coverage of everyday phrases (“Good morning”, “Thank you very much”, “The book is on the table”) that the base model currently mishandles.

4.Simple arithmetic and comparison templates

Problems like “Which number is larger: X or Y?”, “Is N even?”, “If yesterday was D, what day is tomorrow?” with clear single-token answers. Balanced across different numeric ranges and wording styles.

How I would assemble such a dataset

I would build this finetuning set by combining:

Existing open instruction datasets (for example, generic QA and math reasoning sets) filtered to keep only examples with short, strictly formatted outputs.

Public translation corpora that include Arabic and simple everyday sentences, plus a small manually curated set of phrases similar to the ones where the model failed.

Synthetic tasks generated from templates:

number comparisons and even/odd checks letter-counting in words “repeat this exactly” style prompts

For the most format-sensitive tasks (yes/no only, one word only, etc.), I would generate templates automatically and then manually inspect a subset to ensure targets are correct.

How big should the finetuning dataset be?

Because this is a 3B-parameter model, I expect:

• —For format-following and short answers, a focused set of 50k–100k high-quality instruction pairs should already make a visible difference.

• —For multilingual translation including Arabic, I would aim for a few million sentence pairs, mixing existing parallel corpora with a smaller curated subset of common phrases like the ones in this dataset.

• —For simple arithmetic and comparisons, around 20k–50k synthetic examples should be enough, since the patterns are simple and repetitive.

Overall, I would target a finetuning corpus on the order of 1–3 million examples, heavily skewed toward translation pairs, with a smaller but carefully curated slice (roughly 100k–200k examples) dedicated to strict instruction-following and short-answer QA.

Limitations

• —This dataset is tiny and hand-picked, so it is not a general evaluation of SmolLM3-3B-Base quality.

• —All judgments of “correct” or “incorrect” are made by one person.

• —Most prompts are in English, with only a few translation tasks involving Arabic and French.