Karimsh/smollm3-3b-base-blindspots

!pip install -U transformers accelerate datasets huggingface_hub

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

MODEL_ID = "HuggingFaceTB/SmolLM3-3B-Base"

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, use_fast=True)
model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    device_map="auto",
    torch_dtype=torch.bfloat16 if torch.cuda.is_available() else torch.float32,
)
model.eval()

if tokenizer.pad_token_id is None:
    tokenizer.pad_token = tokenizer.eos_token

model.config.pad_token_id = tokenizer.pad_token_id


def generate(prompt, max_new_tokens=64):
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

    with torch.no_grad():
        output_ids = model.generate(
            **inputs,
            max_new_tokens=max_new_tokens,
            do_sample=False,
            eos_token_id=tokenizer.eos_token_id,
            pad_token_id=model.config.pad_token_id,
        )

    generated_ids = output_ids[0][inputs["input_ids"].shape[-1]:]
    return tokenizer.decode(generated_ids, skip_special_tokens=True).strip()

All generations were deterministic (greedy decoding) to ensure reproducibility.

2. What This Dataset Contains

This dataset includes 18 incorrect outputs from a total of 31 diverse prompts.

Each row contains:

input

expected_output

model_output

category

is_incorrect

Categories include:

Arithmetic reasoning

Probability

Counting

Logic

Calendar reasoning

Sorting

String parsing

Unit conversion

Multilingual Arabic

Format obedience

3. Observed Blind Spots

The base model exhibits several consistent failure patterns:

A) Instruction Non-Compliance

Instead of answering, the model often:

Rewrites the question

Adds explanations when none were requested

Produces meta-instructions

Continues generating related prompts

This behavior is typical of base (non-instruction-tuned) models.

B) Arithmetic & Discrete Reasoning Errors

The model frequently:

Computes wrong intermediate values

Switches problem structure mid-generation

Confuses numbers (e.g., changes 19 to 19.5)

These errors suggest weak symbolic precision and unreliable step consistency.

C) Counting & Token-Level Precision Failures

The model struggles with:

Counting character occurrences

Identifying exact character positions

Returning exact substrings

These tasks require strict token-level reasoning rather than semantic approximation.

D) Calendar & Structured Knowledge

Calendar reasoning tasks often:

Produce incorrect weekdays

Drift into unrelated dates

This suggests weak internal algorithmic date computation.

E) Multilingual Weakness (Arabic)

The model frequently:

Avoids answering directly

Produces generic Arabic paragraphs

Fails simple plural or translation tasks

This indicates weaker fine-grained morphological competence in Arabic.

4. What Kind of Dataset Would Fix These Errors?

To improve these weaknesses, the model should be fine-tuned on a structured, task-balanced dataset containing:

1️- Instruction-Following Supervision

Examples requiring:

Single-word answers

Numeric-only outputs

No extra commentary

This aligns generation behavior with task constraints.

2️- Synthetic Symbolic Reasoning Data

Large-scale automatically generated:

Arithmetic problems

Counting problems

Sorting tasks

Calendar computations

String extraction tasks

Synthetic generation allows:

Exact ground truth

Unlimited scale

Balanced difficulty

3️- Multilingual Morphology & Translation Data

For Arabic:

Singular/plural pairs

Short translation tasks

Short-answer QA in Arabic

Controlled-response format supervision

These could be assembled from:

Arabic lexical datasets

Wiktionary morphology data

Automatically generated plural pairs

Public Arabic NLP benchmarks

5. How to Assemble Such a Dataset

A mixed strategy:

A) Programmatic Generation (Majority)

Generate millions of verifiable samples:

Random integer arithmetic

Random string manipulation

Random date calculations

Controlled instruction prompts

Each sample has deterministic ground truth.

This is cheap and scalable.

B) Curated Real-World Data (Minority)

Include:

Public QA benchmarks

Multilingual morphology corpora

Factual short-answer datasets

These increase natural distribution robustness.

6. Estimated Dataset Size Needed

To meaningfully reduce these blind spots:

Skill Area	Estimated Needed Examples
Format obedience	5k–20k
Arithmetic & symbolic	50k–200k
Counting & parsing	20k–100k
Calendar reasoning	10k–50k
Arabic morphology	20k–100k
Total Recommended Size:

100k–500k structured supervision examples

For strong, stable improvements across all categories.