Suman989/smollm2-microbiology-hallucinations

SmolLM2 Blind Spot Audit — Microbiology & Nepal Community Health Triage

Overview

This dataset contains 10 manually audited prompt-output pairs from HuggingFaceTB/SmolLM2-1.7B (base model, not instruct), testing its performance on two domain-specific categories:

1. 1.Microbiology laboratory protocols — Gram staining, serial dilution, PCR parameters, selective media interpretation
2. 2.Community health triage in Nepal — FCHV danger sign protocols, MUAC malnutrition thresholds, ORS dosing, hypertension classification during pregnancy

Model Tested

• —Model: HuggingFaceTB/SmolLM2-1.7B
• —Type: Base model (not fine-tuned for any specific application)
• —Parameters:1.7B
• —Modality: Language

How the Model Was Loaded

Tested using Google Colab (free tier, T4 GPU). Code:

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "HuggingFaceTB/SmolLM2-1.7B"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    device_map="auto"
)

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=120,
        temperature=0.7,
        do_sample=True,
        pad_token_id=tokenizer.eos_token_id
    )

generated = tokenizer.decode(
    outputs[0][inputs["input_ids"].shape[1]:],
    skip_special_tokens=True
)

Dataset Structure

Each row contains three columns:

Key Findings

The model produces outputs that are linguistically fluent but scientifically incorrect in clinically significant ways. Core blind spots identified:

• —Gram stain reversal: The model incorrectly identified staining characteristics of Gram-positive vs. Gram-negative bacteria, a foundational error in microbiology
• —PCR protocol failure: The model omitted essential biochemical reagents from a standard PCR master mix, producing a plausible-sounding but non-functional protocol
• —Triage errors: For Nepal community health scenarios, the model generated advice that sounded reasonable but contradicted WHO guidelines and Nepal's national FCHV protocols; for example, recommending monitoring rather than immediate referral for absent fetal movement

Root Cause

SmolLM2-1.7B optimizes for next-token probability on general web text, not biochemical constraints or clinical decision logic. It learns that certain words follow other words, not that certain laboratory steps must occur in a specific sequence to produce valid results, or that certain clinical symptoms require urgent escalation regardless of context.

This distinction matters: a linguistically correct Gram stain description and a functionally correct one look nearly identical in natural language, but only one produces usable results at a lab bench or in a triage setting.

Proposed Fine-Tuning Dataset

To address these blind spots, the model would benefit from fine-tuning on a curated dataset of:

• —Verified microbiology laboratory protocols (standard operating procedures from academic and public health labs)
• —WHO clinical guidelines and decision trees (maternal health, malnutrition, infectious disease)
• —Nepal-specific community health protocols (FCHV training materials, national health guidelines)
• —Structured question-answer pairs where the correct answer is grounded in biochemical or clinical constraints, not linguistic plausibility

Estimated dataset size: 10,000–50,000 examples minimum for meaningful domain adaptation. A smaller, higher-quality dataset (1,000–5,000 expert-verified examples) combined with RLHF or DPO fine-tuning may outperform a larger noisily-labeled set.

Potential sources: WHO protocol documents, Nepal Ministry of Health guidelines, open-access microbiology textbooks (e.g., Bergey's Manual), PubMed clinical guidelines.

Author

Suman Shrestha — Microbiology graduate, AI application builder, STEAM Educator, Kathmandu, Nepal GitHub: facfax92-maker