ALb78/qwen2_5_reasoning_failures

Reasoning and Logic Failure Cases in Qwen2.5-1.5B

<p align="center"> <img src="https://huggingface.co/front/assets/huggingface_logo-noborder.svg" width="120" alt="Hugging Face logo"> </p>

<p align="center"> <strong>Diagnostic dataset of reasoning errors in a small base language model</strong> </p>

Technical challenge: Blind Spots of Frontier Models by Fatima Institute for Global AI Research

Overview

This dataset documents systematic reasoning failures observed while evaluating the base language model `Qwen/Qwen2.5-1.5B`.

The dataset records cases where the model produces confident but incorrect answers to questions requiring:

• —logical inference
• —counting
• —false premise detection
• —common sense reasoning
• —simple quantitative reasoning

Each entry contains the prompt, the correct answer, and the model's generated output.

The goal is to highlight specific blind spots in small base language models and provide a compact diagnostic dataset for analysis or targeted fine tuning.

Model Tested

• —Model: `Qwen/Qwen2.5-1.5B`
• —Model type: Base pretrained language model
• —Setting: Google Colab with Hugging Face Transformers

This model was not instruction tuned. As a result, it is useful for examining raw reasoning behavior in a small base model.

Model Loading and Evaluation

The model was evaluated in Google Colab using the following code:

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_name = "Qwen/Qwen2.5-1.5B"

tokenizer = AutoTokenizer.from_pretrained(model_name)

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    device_map="auto"
)

question = "How many r letters are in the word strawberry?"

prompt = "Give only the final short answer.\nQuestion: " + question + "\nAnswer:"

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=25,
    do_sample=False
)

decoded = tokenizer.decode(outputs[0], skip_special_tokens=True)
answer = decoded[len(prompt):].strip()

print(answer)

Questions were asked individually, and outputs were manually checked for correctness. Incorrect answers were recorded in this dataset.

Dataset Structure

Each example contains three fields:

Example

{
  "input": "How many r letters are in the word strawberry?",
  "expected_output": "3",
  "model_output": "4"
}

Example Failure Cases

These examples illustrate systematic reasoning weaknesses rather than isolated mistakes.

Observed Failure Types

1. Letter Counting Errors

The model frequently fails to count letters correctly in words such as:

• —strawberry
• —Mississippi

This suggests weak symbolic counting ability.

2. Logical Inference Errors

The model sometimes draws invalid conclusions from partially related premises.

Example:

If all cats are mammals and some mammals are black, can we conclude that some cats are black?

Correct answer: No

3. False Premise Acceptance

The model often answers questions with false assumptions instead of rejecting the premise.

Example:

Who is the current president of the Soviet Union?

Correct answer: No one. The Soviet Union no longer exists.

4. Trick Question Failures

The model struggles with short questions that require careful interpretation.

Example:

How many birthdays does the average person have?

Correct answer: One

5. Quantitative Reasoning Errors

The model makes mistakes on proportional reasoning and simple rate problems involving workers, machines, or production.

Potential Fine Tuning Strategies

These failure patterns suggest that the model would benefit from targeted fine tuning on structured reasoning data.

Possible sources include:

• —GSM8K for arithmetic and multi step reasoning
• —BIG-Bench for broad reasoning tasks
• —Logic puzzle datasets for syllogisms and premise validation
• —Synthetic counting datasets for character and token counting

A useful improvement strategy would combine:

1. 1.counting tasks
2. 2.logical inference tasks
3. 3.false premise rejection examples
4. 4.short common sense traps
5. 5.quantitative reasoning examples

Estimated Dataset Size for Improvement

Approximate training scale needed for improvement:

• —10k to 50k examples for small gains
• —100k to 200k examples for broader and more stable reasoning improvements

The exact number would depend on dataset quality, diversity, and training setup.

Purpose of This Dataset

This dataset is intended as a diagnostic dataset, not a leaderboard benchmark.

It can be used for:

• —reasoning failure analysis
• —targeted fine tuning experiments
• —evaluating post-training improvements
• —studying blind spots in small base language models