Sriyanshsh1805/qwen2b-blindspots

Qwen3.5-2B Blindspots Dataset

Overview

This dataset contains examples where the base language model Qwen3.5-2B-Base produces incorrect or unexpected outputs. The goal of this dataset is to identify blind spots in small frontier language models by testing them on tasks involving reasoning, counting, symbolic manipulation, and strict instruction following.

The dataset records:

• —The input prompt
• —The expected output
• —The actual model output
• —The error category

These examples highlight limitations in small base models and help analyze areas where targeted fine-tuning may improve performance.

Model Tested

Model: Qwen3.5-2B-Base Link: https://huggingface.co/Qwen/Qwen3.5-2B-Base

Qwen3.5-2B-Base is a 2-billion parameter base causal language model developed by the Qwen team. It is trained primarily for next-token prediction and is not instruction-tuned, which makes it useful for studying raw model behavior and identifying reasoning failures.

How the Model Was Loaded

The model was loaded and tested using the Hugging Face transformers pipeline with GPU acceleration in Google Colab.

from transformers import pipeline

pipe = pipeline(
    "text-generation",
    model="Qwen/Qwen3.5-2B-Base",
    device_map="auto"
)

prompt = "Explain why the sky is blue."

output = pipe(prompt, max_new_tokens=100)

generated = output[0]["generated_text"]
answer = generated.replace(prompt, "")

print(answer.strip())

For each prompt, the generated output was recorded and compared against the expected correct answer.

Blind Spots Observed

During experimentation, several categories of model errors were observed:

1. Character-Level Counting Errors

The model often fails tasks requiring exact counting of characters.

Example: Prompt: Count the number of vowels in the word "strawberry". Expected: 2 Model Output: Incorrect explanations with inconsistent counts.

2. Instruction Following Failures

The model frequently ignores strict instructions or formatting requirements.

Example: Prompt: Write the numbers from 1 to 10 separated only by commas. Observed behavior: The model continues generating additional examples instead of stopping.

3. Logical Reasoning Errors

The model struggles with multi-step logical reasoning.

Example: Prompt: If today is Monday, what day will it be after 9 days? Observed behavior: Generates unrelated reasoning or incorrect calculations.

4. Symbolic Manipulation Errors

Tasks involving reversing strings or editing characters are difficult.

Example: Prompt: Reverse the digits in 47291. Observed behavior: Generates irrelevant text or incorrect results.

5. Pronoun and Context Resolution

The model may incorrectly resolve ambiguous references.

Example: Prompt: "The trophy doesn't fit in the brown suitcase because it is too small. What is too small?" Correct answer: The suitcase.

Models often incorrectly answer the trophy.

Dataset Structure

Each entry in the dataset contains the following fields:

Proposed Fine-Tuning Dataset

To improve performance on these failure cases, the model should be fine-tuned on datasets containing:

• —Logical reasoning tasks
• —Character-level manipulation tasks
• —Instruction-following examples
• —Structured output tasks
• —Mathematical reasoning problems

Example sources include:

• —GSM8K (math reasoning)
• —BIG-Bench reasoning tasks
• —synthetic string manipulation datasets
• —instruction datasets such as FLAN

How Such a Dataset Could Be Assembled

A dataset to address these weaknesses could be built using a combination of:

1. 1.Existing benchmark datasets

• —Mathematical reasoning datasets
• —Logical reasoning benchmarks

1. 1.Synthetic data generation

• —Programmatically generate prompts for:

• —counting characters
• —string reversal
• —pattern recognition
• —sorting tasks

1. 1.Human-written instruction prompts

• —Tasks requiring strict formatting
• —multi-step reasoning problems

Combining real benchmark datasets with synthetic tasks would provide broad coverage of reasoning and symbolic manipulation challenges.

Estimated Dataset Size Needed

To significantly improve model performance in these areas, the fine-tuning dataset would likely need:

• —20k – 50k examples for structured reasoning tasks
• —10k – 20k examples for string manipulation and counting tasks
• —30k+ examples for instruction-following and formatting constraints

In total, approximately 50k – 100k high-quality examples could meaningfully improve performance in these domains.

Purpose of This Dataset

This dataset serves as a small diagnostic benchmark for identifying systematic weaknesses in small language models. By highlighting where the model fails, it can guide the creation of targeted fine-tuning datasets and improve the reliability of future models.