ecairol/simpleqa-eval-qwen3.8-27b-obliterated

SimpleQA-Verified results: qwen3.8-27b, obliterated vs. normal

Factual-accuracy evaluation of the abliterated ("obliterated") fine-tune `OBLITERATUS/Qwen3.8-27B-OBLITERATED`, compared against the normal (non-abliterated) base model `Qwen/Qwen3.8-27B`, against `codelion/SimpleQA-Verified`, run locally with Inspect via LM Studio.

Result

The normal model is roughly 2.6x more accurate than the obliterated fine-tune. N=201 rather than the full 1000 for both runs — run locally on a laptop, and the first (obliterated) run was stopped early due to sustained thermal load from multi-hour local inference on a 27B model. The second run was capped at the same N (--limit 201) so the two results are directly comparable. Stderr at this N is already fairly tight (see the raw logs for every question/answer/grade).

Setup

• —Models:
• —OBLITERATUS/Qwen3.8-27B-OBLITERATED, served locally via LM Studio (openai-api/lmstudio/qwen3.8-27b-obliterated in Inspect)
• —Qwen/Qwen3.8-27B (MLX, 4-bit), served locally via LM Studio (openai-api/lmstudio/qwen3.8-27b in Inspect)
• —Solver: generate() (single-turn, no scaffolding/tools)
• —Scorer: model_graded_qa() — each model grades its own answers (no external grader was configured; treat the score as a lower bound on reliability of the grading itself, not just the answering)
• —Task definition: `tests/simpleqa.py` in this repo's companion codebase

from inspect_ai import Task, task
from inspect_ai.dataset import FieldSpec, hf_dataset
from inspect_ai.scorer import model_graded_qa
from inspect_ai.solver import generate

@task
def simpleqa():
    return Task(
        dataset=hf_dataset(
            "codelion/SimpleQA-Verified",
            split="train",
            sample_fields=FieldSpec(input="problem", target="answer"),
        ),
        solver=generate(),
        scorer=model_graded_qa(),
    )

Why this is worth publishing

SimpleQA-Verified is intentionally built from obscure, low-frequency facts — even frontier models score well below 100%. Abliteration clearly costs accuracy here: the obliterated fine-tune answers correctly less than half as often as the normal model (11.4% vs. 29.4%).

The more interesting finding only showed up once the normal model was added for comparison. The original hypothesis was that abliteration (removing refusal training) also reduces a model's tendency to hedge or say "I don't know," so wrong answers would look more confident. That doesn't hold up: across both runs, hedge phrases ("I don't know," "not sure," "unable to confirm," etc.) appear in only 2 of 201 obliterated answers and 0 of 201 normal-model answers. Both models answer nearly every question — right or wrong — with full, unhedged confidence, fabricating specific names, dates, and numbers when they don't actually know. See sample transcripts in the logs for concrete examples (e.g. both models independently invent a wrong, fully-explained currency-conversion figure for the same compensation question).

Confident fabrication on obscure facts looks like a property of this model family at this scale generally, not something abliteration specifically introduces. As a quick sanity check, the obliterated model was also run locally against 5 well-known historical facts (WWII end date, first US president, etc., not included in this dataset) and answered all 5 correctly — so the gap is specific to obscure knowledge, not a general breakdown in factuality.

Files in this repo

• —logs/*.eval — raw Inspect log files (binary format; read with inspect log dump <file> or browse with inspect view)
• —bundle/ — static, browsable transcript viewer, also published as a Hugging Face Space

Reproduce

pip install inspect-ai datasets
lms server start
inspect eval tests/simpleqa.py --model openai-api/lmstudio/qwen3.8-27b-obliterated --limit 201 --max-connections 1
inspect eval tests/simpleqa.py --model openai-api/lmstudio/qwen3.8-27b --limit 201 --max-connections 1