shifa21/tinyllama-blindspots

TinyLlama Blind Spots Dataset

Overview

This dataset documents failure cases observed while testing the TinyLlama 1.1B Chat model. The goal of the dataset is to identify situations where the model produces incorrect answers, incomplete responses, or fails to respond properly.

These examples highlight several blind spots in the model’s reasoning and knowledge capabilities.

Model Tested

Model: TinyLlama/TinyLlama-1.1B-Chat-v1.0

Model page: https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0

How the Model Was Loaded

The model was loaded using the Hugging Face Transformers library in a Google Colab environment.

Python code used to load the model:

from transformers import AutoTokenizer, AutoModelForCausalLM

model_name = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"

tokenizer = AutoTokenizer.frompretrained(modelname) model = AutoModelForCausalLM.frompretrained(modelname)

def askmodel(prompt): inputs = tokenizer(prompt, returntensors="pt") outputs = model.generate(**inputs, maxnewtokens=50) answer = tokenizer.decode(outputs[0], skipspecialtokens=True) return answer

Experiment Setup

The model was tested using multiple prompts designed to evaluate different reasoning abilities.

Prompts were selected from several categories including:

• —Arithmetic reasoning
• —Numerical comparison
• —Factual knowledge questions
• —Logical reasoning problems

The objective was to observe cases where the model produced incorrect answers, incomplete responses, or repeated the prompt instead of answering.

Each dataset entry contains three fields:

input The prompt given to the model.

model_output The response generated by the model.

expected_output The correct answer.

Observed Blind Spots

Arithmetic Reasoning Errors

The model frequently produces incorrect results for multiplication and arithmetic problems.

Numerical Comparison Errors

The model sometimes fails to correctly compare decimal numbers.

Factual Knowledge Failures

For several simple factual questions the model repeats the prompt instead of providing an answer.

Logical Reasoning Mistakes

The model struggles with simple reasoning tasks such as word problems involving counting or basic logic.

Repetition Loops

The model sometimes enters repetitive output patterns, producing the same phrase multiple times.

Potential Improvements

These failures suggest the model would benefit from additional fine tuning on curated reasoning datasets.

Examples of useful datasets include:

• —Mathematical reasoning datasets
• —Numerical comparison datasets
• —Factual question answering datasets
• —Logical reasoning benchmarks

Training on structured reasoning examples could help the model produce more reliable outputs.

Estimated Dataset Size for Improvement

To address these blind spots effectively, a dataset containing approximately 20,000 to 100,000 carefully labeled reasoning examples would likely be required for fine tuning.

Such a dataset could improve the model’s ability to perform:

• —Arithmetic reasoning
• —Logical problem solving
• —Factual question answering

and reduce repetitive or incomplete outputs. --- license: apache-2.0 ---