NachtSpyder04/Blind_Spots_of_Frontier_Model-Qwen3.5_4B_base

BlindSpotsofFrontierModel-Qwen3.54Bbase

This is a submission to the Technical challenge: Blind Spots of Frontier Models for Fatima Fellowship 2026 application

Model Link: Qwen/Qwen3.5-4B-base

Code

Code link: https://gist.github.com/NachtSpyder04/8b1ccd5a29b37f7a82083c93b4f518c8

The model was loaded using the Hugging Face transformers library, and evaluated using deterministic decoding (temperature set to 0). I intentionally disabled sampling so that the outputs would be consistent across runs, allowing the errors to reflect reasoning limitations rather than randomness. Code snippets for loading the model are available on HuggingFace model link and I have also referred tutorial blogs and videos on Medium and Youtube.

Model was loaded in the following manner -

from transformers import AutoTokenizer, AutoModelForCausalLM
from datasets import Dataset
import torch
import json
import re

model_id = "Qwen/Qwen3-4B-Base"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    device_map="auto"
)

model.eval()

I constructed a set of structured prompts spanning different reasoning categories:

• —Arithmetic and algebra
• —Logical quantifiers and negation
• —Conditional reasoning and logical fallacies
• —Binary and bitwise operations
• —Multi-step word problems
• —Factual recall
• —Hallucination tests using fictional papers and concepts

The objective was not to test one narrow skill repeatedly, but to probe different cognitive behaviors of the model. From these evaluations, I selected ten diverse failure cases that illustrate distinct types of errors rather than repeating similar mistakes.

Each synthetically generated dataset entry contains:

• —The reasoning category it belongs to
• —The original input prompt
• —The expected correct answer

Before comparing outputs, I normalized the model responses to avoid penalizing trivial formatting differences (such as capitalization or commas in numbers). This allowed the evaluation to focus on actual reasoning correctness rather than surface-level formatting.

For standard reasoning tasks (arithmetic, logic, binary operations, etc.), an answer was marked correct if it matched the expected output after normalization.

For hallucination prompts involving nonexistent papers, fictional programming languages, or false events, the expected behavior was for the model to indicate uncertainty or lack of knowledge. If the model confidently fabricated detailed content instead, it was marked as incorrect.

Each test prompt was passed through the model individually. The generated output was extracted, normalized, and compared against the expected answer. I recorded:

• —The reasoning category it belongs to
• —The original input prompt
• —The expected correct answer
• —The model’s generated output
• —A correctness flag (used by the code to seperate correct outputs from incorrect ones)

All results were saved, and incorrect predictions were filtered into a separate list.

Executing the code linked above will output two json files - all_results.json and error_cases.json. The dataset on this repository is a small subset of error_cases.json. I have executed this model on my local Nvidia RTX 4090 GPU system.

Observation

Across categories, the model showed weaknesses in:

• —Precise arithmetic computation
• —Formal logical reasoning (especially negation and subset inference)
• —Multi-step dependency tracking
• —Binary and symbolic operations
• —Strict instruction adherence
• —Hallucinating when asked about fictional entities

From the results, it appears that the Qwen3.5-4B base model struggles with tasks that require precise reasoning and strict adherence to instructions. In arithmetic and algebra questions, the model often produced answers that were numerically close but still incorrect. For logical questions, the model sometimes drew conclusions that were not logically guaranteed.

Another consistent issue was output formatting. Even when the prompt clearly asked for a number, the model generated extra explanations, repeated the question, or started reasoning steps. The model also hallucinated when asked about non-existent papers or theorems, confidently generating plausible but fabricated explanations instead of indicating that the entity does not exist. Overall, these observations suggest that the base model lacks strong instruction-following behavior and reliable reasoning without additional alignment or instruction tuning.

With reference to the LLM scaling laws, the exact dataset size cannot be determined analytically and would typically be chosen empirically. Since the base model already contains general language knowledge, the goal of fine-tuning is mainly to teach task-specific behavior and formatting constraints rather than relearn language patterns. A reasonable approach would be to begin with a moderately sized dataset containing tens of thousands of examples covering the targeted task categories and then evaluate whether the observed errors decrease.

To address these issues, the model should be fine-tuned on a dataset containing arithmetic problems, logical reasoning tasks, multi-step word problems, binary and symbolic operations, and prompts that enforce strict output formatting or require the model to abstain when information does not exist.

Such datasets are easily available on Kaggle and Hugging Face. If the model still exhibits the same failure modes, the dataset could be expanded or diversified. In practice, the optimal dataset size would be determined through iterative experimentation rather than a fixed theoretical number