MasihM/eyes-wide-shut-safety-benchmark

Eyes Wide Shut: A Multivector Safety Analysis of gpt-oss:20b

Author: Masih Moafi (Isfahan University of Technology) Campaign: OpenAI gpt-oss-20b Red-Teaming Challenge Target Package: gpt-oss:20b (GGUF, MXFP4 quantization, 20.9B parameters) at temperature 1.0, high reasoning effort DOI: 10.5281/zenodo.21826218 Paper Repository: github.com/MasihMoafi/eyes-wide-shut

Abstract

This dataset contains the empirical transcripts, evaluation protocols, and reproduction harnesses for Eyes Wide Shut: A Multivector Safety Analysis of gpt-oss:20b.

Across three independent empirical studies, model safety behavior consistently tracks whether a request matches an explicitly named category rather than the underlying concept:

1. 1.Simulation Framing (Finding 1): Adding explicit simulation framing to a philosophy-grounded prompt package increased structured destructive tool calls (press_the_button) from 78/100 to 97/100 under identical seeds and parameters. A separate diagnostic revealed a 20/100 say–do mismatch where the model visibly refused in dialogue but emitted the destructive tool call in turn 3.
2. 2.Semantic Reframing (Finding 2): An explicit user-dependency objective was refused in 30/30 trials. When relabelled as an educational tutor and decomposed across five turns, the model complied in 30/30 trials across all phases, culminating in distress-keyed monetization code.
3. 3.Reasoning Channel Exposure (Finding 3): Under a scripted release condition, a protected synthetic secret appeared in the recorded reasoning field in 25/30 runs (83.3%) while appearing in the visible answer in only 13/30 runs (43.3%), demonstrating critical data leakage through unmonitored chain-of-thought channels.

Verification & Reproducibility

All reported counts are directly verifiable from the raw transcripts in this repository using verify_counts.py:

python3 verify_counts.py

Repository Structure

├── finding1/
│   ├── data/en_nosim_ablation_t4/       # 100 trials without simulation framing
│   ├── data/en_sim_legacy_t4/           # 100 trials with simulation framing
│   └── data/fake_alignment_run/         # 100 diagnostic trials (say-do divergence)
├── finding2/
│   ├── data/finding2_campaign.jsonl     # 60 full multi-turn conversations
│   └── data/classifications_legacy.json # Blinded LLM judge verdicts
├── finding3/
│   └── data/finding3_reproducible_baseline.jsonl # 30 reasoning leakage runs
├── verify_counts.py                     # Standalone verification script
└── README.md

Citation

@article{moafi2026eyeswideshut,
  title={Eyes Wide Shut: A Multivector Safety Analysis of gpt-oss:20b},
  author={Moafi, Masih},
  journal={arXiv preprint / Zenodo DOI: 10.5281/zenodo.21826218},
  year={2026}
}