Datasetpublic

RHYTHM1028/qwen3-1.7b-base-blind-spots

Qwen3-1.7B-Base Blind Spots Dataset Model Tested Model: surogate/Qwen3-1.7B-Base Architecture: Qwen3 (decoder-only transformer) Parameters: 1,720,574,976 (~1.7B) Type: Base pretrained model (NOT instruction-tuned) License: Apache 2.0 Release: May 2025 How the Model Was Loaded from transformers import AutoTokenizer, AutoModelForCausalLM import torch model_id = "surogate/Qwen3-1.7B-Base" tokenizer = AutoTokenizer.from_pretrained(model_id) model =… See the full description on the dataset page: https://huggingface.co/datasets/RHYTHM1028/qwen3-1.7b-base-blind-spots.

sourceHugging Faceapache-2.0updated 7mo agoView on Hugging Face

0likes11downloads

Dataset Card

Qwen3-1.7B-Base Blind Spots Dataset

Model Tested

—Model: `surogate/Qwen3-1.7B-Base`
—Architecture: Qwen3 (decoder-only transformer)
—Parameters: 1,720,574,976 (~1.7B)
—Type: Base pretrained model (NOT instruction-tuned)
—License: Apache 2.0
—Release: May 2025

How the Model Was Loaded

python

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "surogate/Qwen3-1.7B-Base"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype=torch.float16,
    device_map="auto"
)

—Framework: transformers==5.3.0.dev0 (installed from GitHub source for Qwen3 support)
—Hardware: Google Colab T4 GPU (16GB VRAM)
—Precision: float16
—Generation: temperature=0.0, do_sample=False (greedy/deterministic), repetition_penalty=1.1, max_new_tokens=150
—Reproducibility: All outputs are fully deterministic — re-running produces identical results

Observed Blind Spots

1. Quiz-Generation Mode — Two Overlapping Problems

The model exhibits pervasive quiz-authoring behavior. A controlled experiment (5 prompt pairs, fill-in-the-blank vs. full-question format) revealed this is actually two distinct failure modes:

(a) Prompt-Format Sensitivity (fixable by rephrasing)

Fill-in-the-blank prompts trigger exam formatting; full-question rephrases fix it.

Fill-in-the-blank prompt	Model output	Rephrased prompt	Model output
"Therefore, there are"	"_____ centimeters in one kilometer" (fill-in-blank)	"How many centimeters are in one kilometer?"	"100,000 centimeters" ✅
"you have 3 apples and give away 1, you have"	"_____ left" (fill-in-blank)	"How many apples do you have left?"	"2 apples" ✅

(b) Tangential Generation / Topic Drift (NOT fixable by rephrasing)

The model fails regardless of format — generating tangential or unrelated educational content.

Prompt (any format)	Model output
"What happens when a glass falls off a table?"	Generates unrelated physics questions about hills and Newton's laws
"Why doesn't the trophy fit in the brown suitcase?"	Asks "what is a trophé?" instead of answering
"What is the opposite of the word 'hot'?"	Generates unrelated follow-up questions about synonyms and prepositions

Rephrasing fixed only 2 out of 5 cases — the problem is deeper than prompt format sensitivity alone.

Root cause hypothesis: The Qwen3 pretraining corpus over-represents educational materials (textbooks, exam prep, quiz databases). The model treats prompts as starting points for generating educational content rather than as questions requiring answers.

2. Arithmetic Errors

The model consistently fails at multi-step math:

—Computes $18 instead of $20 for a 20%-off $25 shirt
—On GSM8K benchmark: scored 6/50 (12.0%) compared to GPT-4's ~92%

3. Temporal Reasoning

The model cannot reliably count days forward:

—Says "Saturday" instead of "Thursday" for "two days after Tuesday"
—Completely ignores a modular day-counting question (10 days from Monday) and generates an unrelated math problem

4. Code Pattern Recognition

Confuses variable assignment with variable swapping:

—Input: a, b = (expecting b, a)
—Got: 10, 20 (value assignment, not a swap)

What Dataset Would Fix This

Blind Spot	Proposed Training Data	Estimated Size	Rationale
Quiz-generation mode (format)	Direct `(incomplete sentence, concise answer)` completion pairs without exam formatting	~20K-30K examples	Based on Alpaca (52K) / Dolly (15K) scale for behavioral shifts
Quiz-generation mode (drift)	Focused Q&A pairs that penalize tangential generation; RLHF with "stay on topic" reward	~10K-20K examples	Requires behavioral alignment, not just knowledge
Arithmetic errors	Step-by-step math solutions (GSM8K-style) with chain-of-thought	~10K examples	GSM8K train set is 7.5K; WizardMath showed ~10K can jump accuracy from single digits to 40-60%
Temporal reasoning	Calendar/date arithmetic problems with explicit day counting	~2K-5K examples	Narrow skill — few-shot fine-tuning literature suggests 1K-5K for task-specific skills
Code patterns	Python code completion pairs focused on idioms and patterns	~5K examples	Subset of CodeAlpaca (20K) targeting specific patterns

Total estimated: ~50K-80K curated training examples, with some requiring RLHF-style alignment training to address the tangential generation issue.

Dataset Structure

Each record contains:

Field	Description
`input`	The prompt given to the model
`expected_output`	The correct/expected completion
`model_output`	What the model actually generated
`error_type`	Category of failure (e.g., `arithmetic`, `quiz_generation_mode`)
`category`	Human-readable description of the test
`failure_note`	Detailed explanation of why this is a failure

Size: 12 failure cases from 24 hand-crafted prompts (50% failure rate)

Benchmark Results

Benchmark	Score	Notes
Hand-crafted probes (24 prompts)	12/24 pass (50%)	Across 12+ reasoning categories
GSM8K (50-sample)	6/50 (12.0%)	Grade-school math word problems
Prompt format sensitivity	2/5 fixed by rephrasing	Fill-in-blank → quiz mode; full questions fix only some cases

Notebook

The full code for this project is available in two places:

—Run it directly: Google Colab Notebook
—Download it: See qwen3-1.7b-base-blind-spot-evaluation.ipynb in this repository

Methodology Note

All results use greedy decoding (do_sample=False, temperature=0.0) for full determinism. Earlier experiments with sampling (temperature=0.3) showed tests flipping between pass/fail across runs — we switched to greedy to ensure every reported failure is a consistent, reproducible blind spot, not sampling noise.