faisalshahid03/Fatima-fellowship

Blind Spots of Qwen3.5-0.8B-Base

Model Tested

This dataset evaluates the weaknesses of the base language model Qwen3.5-0.8B-Base developed by the Qwen Team.

• —Model Link: Qwen/Qwen3.5-0.8B-Base on Hugging Face
• —Parameters: ~0.8 billion
• —Type: Causal Language Model (Base Model)

Because it is a base model, it has not been fine-tuned for instruction following, providing a clear opportunity to analyze the kinds of mistakes smaller foundation models make when performing reasoning or knowledge tasks.

Objective

The goal of this dataset is to identify blind spots of a frontier base model by testing the model on diverse prompts and documenting where its predictions differ from expected outputs.

Each entry in this dataset contains:

• —`input` – The prompt given to the model.
• —`expected_output` – The correct or desired response.
• —`model_output` – The response generated by the model.

The dataset includes 10 diverse examples covering reasoning, arithmetic, logical inference, and factual knowledge.

How the Model Was Loaded

The model was evaluated using the Transformers library from Hugging Face in a Google Colab GPU environment.

Installation

!pip install transformers accelerate torch

Loading the Model

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_name = "Qwen/Qwen3.5-0.8B-Base"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.float16,
    device_map="auto"
)

def generate(prompt):
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    outputs = model.generate(
        **inputs,
        max_new_tokens=120,
        temperature=0.7
    )
    return tokenizer.decode(outputs[0], skip_special_tokens=True)

Prompts were then passed to the generate() function to collect the model outputs.

Observed Blind Spots

During experimentation, several recurring failure modes were observed:

1. 1.Arithmetic Reasoning Errors: The model frequently struggles with basic arithmetic problems involving unit conversion or multi-step calculations.
2. 2.Logical Inference Mistakes: When presented with simple logical statements, the model sometimes draws incorrect conclusions.
3. 3.Hallucinated Knowledge: The model occasionally produces confident but incorrect factual statements when the prompt contains misleading or ambiguous information.
4. 4.Instruction Following Issues: Despite instructions such as "answer with one word," the model sometimes generates longer explanations.
5. 5.Multi-Step Reasoning Limitations: Problems requiring multiple reasoning steps often lead to incorrect answers or incomplete reasoning chains.

These blind spots are expected in smaller base models that have not been specifically fine-tuned for reasoning or instruction-following tasks.

Suggested Fine-Tuning Dataset

To address these weaknesses, the model should be fine-tuned on datasets emphasizing reasoning, correctness, and instruction following.

Recommended datasets include:

• —Arithmetic reasoning datasets (e.g., GSM8K)
• —Logical reasoning datasets (e.g., StrategyQA)
• —Factual verification datasets (e.g., TruthfulQA)
• —Instruction-following datasets (e.g., OpenAssistant conversations)

Dataset Collection Strategy

Such datasets could be assembled through:

1. 1.Public reasoning benchmarks
2. 2.Synthetic data generated by stronger language models
3. 3.Human-annotated reasoning tasks
4. 4.Fact-checking corpora from curated knowledge bases

Combining synthetic and human-verified data would help maintain both scale and accuracy.

Estimated Dataset Size

Total estimated size: 200k–300k training examples.

Conclusion

This dataset demonstrates several important blind spots in the Qwen3.5-0.8B-Base model, particularly in reasoning and instruction-following tasks. These limitations highlight the importance of targeted fine-tuning and dataset curation when adapting base models for real-world applications.