Becky4382/qwen35-2b-base-blind-spots

Blind Spots of Qwen3.5-2B-Base

This dataset documents incorrect predictions from Qwen/Qwen3.5-2B-Base, a 2.21B-parameter pre-trained base model released February 2026.

How the Model Was Loaded

Loaded in Google Colab (free T4 GPU) with Hugging Face Transformers:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL_ID = "Qwen/Qwen3.5-2B-Base"
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID, torch_dtype=torch.float16, device_map="auto", trust_remote_code=True
)

Since it is a base model (not instruction-tuned), prompts were given as text completions:

def generate(prompt, max_tokens=60):
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    with torch.no_grad():
        out = model.generate(**inputs, max_new_tokens=max_tokens, temperature=0.1,
                             do_sample=True, pad_token_id=tokenizer.eos_token_id)
    return tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True).strip()

Dataset Schema

Categories Tested

12 diverse tests across: letter counting, multi-step math, negation, calendar reasoning, spatial reasoning, physical common sense, logical deduction, unit conversion, reverse spelling, decimal comparison, obscure facts, and sequence counting.

Results: 7 Blind Spots Found Out of 12 Tests

Key Blind Spots Found

• —Character-level tasks (tests 1, 9): The model cannot count letters or reverse strings. It said "strawberry" has 2 r's (correct answer: 3) and reversed "desserts" as "tsetsre" (correct: "stressed"). This is because tokenizers split words into subword tokens, not individual characters.
• —Multi-step arithmetic (test 2): The model computed 150 - 37 as 113 correctly but then failed the multiplication, outputting 114 instead of 226.
• —Negation handling (test 3): When asked for animals that CANNOT fly, the model listed correct examples (elephant, giraffe) but then switched to discussing pigeons that CAN fly, showing it lost track of the negation.
• —Decimal comparison (test 10): The model said 9.11 > 9.8, a classic LLM failure where it compares digits positionally (11 > 8) instead of comparing decimal values (0.11 < 0.8).
• —Self-contradiction (test 8): The model correctly computed 3500 meters but then ended with "The answer is: 350", contradicting its own reasoning.
• —Logical rigor (test 7): The model concluded "some roses fade quickly" is "True" without noting this doesn't follow from the premises. "Some flowers fade" doesn't guarantee "some roses fade."

Recommended Fine-tuning Dataset

What kind of dataset: Chain-of-thought reasoning examples that show step-by-step solutions. Character-level manipulation tasks (letter counting, string reversal). Negation-aware sentence completions. Decimal comparison drills.

How to assemble it:

1. 1.GSM8K and MATH datasets for math reasoning with chain-of-thought
2. 2.BIG-Bench subtasks for character manipulation and logic
3. 3.Synthetic data: programmatically generate letter-counting, string-reversal, and decimal-comparison examples (easy to create with verified ground truth)
4. 4.NLI datasets (SNLI, MultiNLI) for negation understanding

How much data: For LoRA fine-tuning on a 2B model, 5,000-10,000 high-quality chain-of-thought examples across these categories should produce meaningful improvement. Quality matters more than quantity.