shifa21/tinyllama-blindspots

1# TinyLlama Blind Spots Dataset2 3## **Overview**4 5This dataset documents **failure cases observed while testing the TinyLlama 1.1B Chat model**.  6The goal of the dataset is to identify situations where the model produces **incorrect answers, incomplete responses, or fails to respond properly**.7 8These examples highlight several **blind spots in the model’s reasoning and knowledge capabilities**.9 10---11 12## **Model Tested**13 14**Model:** TinyLlama/TinyLlama-1.1B-Chat-v1.0  15 16Model page:  17https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.018 19---20 21## **How the Model Was Loaded**22 23The model was loaded using the **Hugging Face Transformers library** in a **Google Colab environment**.24 25**Python code used to load the model:**26 27from transformers import AutoTokenizer, AutoModelForCausalLM28 29model_name = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"30 31tokenizer = AutoTokenizer.from_pretrained(model_name)32model = AutoModelForCausalLM.from_pretrained(model_name)33 34def ask_model(prompt):35inputs = tokenizer(prompt, return_tensors="pt")36outputs = model.generate(**inputs, max_new_tokens=50)37answer = tokenizer.decode(outputs[0], skip_special_tokens=True)38return answer39 40 41---42 43## **Experiment Setup**44 45The model was tested using **multiple prompts designed to evaluate different reasoning abilities**.46 47Prompts were selected from several categories including:48 49- **Arithmetic reasoning**50- **Numerical comparison**51- **Factual knowledge questions**52- **Logical reasoning problems**53 54The objective was to observe cases where the model produced **incorrect answers, incomplete responses, or repeated the prompt instead of answering**.55 56Each dataset entry contains three fields:57 58**input**  59The prompt given to the model.60 61**model_output**  62The response generated by the model.63 64**expected_output**  65The correct answer.66 67---68 69## **Observed Blind Spots**70 71### **Arithmetic Reasoning Errors**72The model frequently produces **incorrect results for multiplication and arithmetic problems**.73 74### **Numerical Comparison Errors**75The model sometimes **fails to correctly compare decimal numbers**.76 77### **Factual Knowledge Failures**78For several simple factual questions the model **repeats the prompt instead of providing an answer**.79 80### **Logical Reasoning Mistakes**81The model struggles with **simple reasoning tasks such as word problems involving counting or basic logic**.82 83### **Repetition Loops**84The model sometimes **enters repetitive output patterns**, producing the same phrase multiple times.85 86---87 88## **Potential Improvements**89 90These failures suggest the model would benefit from **additional fine tuning on curated reasoning datasets**.91 92Examples of useful datasets include:93 94- **Mathematical reasoning datasets**95- **Numerical comparison datasets**96- **Factual question answering datasets**97- **Logical reasoning benchmarks**98 99Training on structured reasoning examples could help the model **produce more reliable outputs**.100 101---102 103## **Estimated Dataset Size for Improvement**104 105To address these blind spots effectively, a dataset containing approximately **20,000 to 100,000 carefully labeled reasoning examples** would likely be required for fine tuning.106 107Such a dataset could improve the model’s ability to perform:108 109- **Arithmetic reasoning**110- **Logical problem solving**111- **Factual question answering**112 113 114and reduce repetitive or incomplete outputs.115---116license: apache-2.0117---118