asadkhn/qwen25-3b-medical-blindspots

Blind Spots of Qwen2.5-3B (Base Model) — Medical & Epidemiology Domain

Dataset Summary

This dataset documents 10 diverse failure cases of the Qwen/Qwen2.5-3B base language model, focusing on medical, epidemiological, and multilingual (Urdu) prompts. It was created as part of the Fatima Fellowship 2026 application technical challenge.

Each row contains:

• —input: the prompt given to the model
• —expected_output: the correct answer
• —model_output: what the model actually generated
• —error_type: categorization of the failure mode

Model Tested

• —Model: Qwen/Qwen2.5-3B
• —Type: Base language model (not instruction-tuned)
• —Parameters: 3 billion
• —Release date: September 2024

How I Loaded the Model

Loaded on Google Colab (free tier, T4 GPU) using HuggingFace Transformers:

!pip install transformers torch accelerate -q

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_name = "Qwen/Qwen2.5-3B"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.float16,
    device_map="auto"
)

def test_model(prompt, max_new_tokens=200):
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    with torch.no_grad():
        outputs = model.generate(
            **inputs,
            max_new_tokens=max_new_tokens,
            do_sample=False,
            temperature=1.0
        )
    response = tokenizer.decode(outputs[0], skip_special_tokens=True)
    return response[len(prompt):]

Failure Categories Found

Analysis of Blind Spots

The model exhibits four major blind spot categories:

1. Urdu/multilingual failure: The model completely fails on Urdu-language prompts, entering infinite repetition loops. Despite Qwen2.5 claiming multilingual support, medical Urdu queries expose a critical gap.

2. Medical knowledge errors: ICD-10 codes, disease incubation periods, and acronym expansions contain factual errors. The model appears to have seen medical MCQ-style training data but without grounded factual accuracy.

3. Epidemiological reasoning gap: The model lacks understanding of public health concepts like endemic channels, outbreak thresholds, and baseline comparisons — it substitutes generic mathematical reasoning instead.

4. Generation degeneration: Base models without instruction tuning frequently drift into repetition loops or hallucinate unrelated content mid-response, especially on open-ended prompts.

Recommended Fine-tuning Dataset

To fix these errors, the model should be fine-tuned on:

Dataset composition:

• —Urdu medical QA pairs — question-answer pairs in Urdu covering common diseases, symptoms, and treatments. Sources: translated WHO/CDC materials, Pakistani health ministry publications.
• —ICD-10/ICD-11 code lookup dataset — structured disease name → correct ICD code mappings covering at least 5,000 diseases.
• —Epidemiology concept dataset — definitions, formulas, and worked examples for core concepts: attack rate, R0, endemic channel, outbreak thresholds, IDSR, surveillance methods.
• —Public health surveillance QA — based on WHO IDSR guidelines, CDC surveillance manuals, and DHIS2 documentation.
• —Instruction-following data — to fix topic drift and repetition degeneration, the base model needs general instruction-tuning data (e.g., FLAN, Alpaca-style).

Recommended dataset size:

• —Urdu medical QA: ~5,000–10,000 pairs minimum for meaningful improvement
• —ICD code mapping: ~5,000 structured entries
• —Epidemiology QA: ~2,000–3,000 curated pairs
• —General instruction tuning: ~50,000+ examples (standard fine-tuning scale)

Datasets like MedQA, IndicQA, and WHO/CDC public health documentation could serve as starting points, combined with custom curation for Pakistan-specific epidemiology context.

About the Author

Created by Asad Khan, IT Officer at NIH Pakistan (IDSRS — Integrated Disease Surveillance and Response System), where he manages national disease surveillance systems built on DHIS2 covering 37 diseases across all government health facilities in Pakistan.