joshuapenman/semantic-overlays-injection

Semantic Overlays — injection training corpus

The training corpus for the "do-not-execute" overlay of Semantic Overlays: Mitigating Prompt Injection with Annotations Beyond Tokens and Steering Vectors (arXiv:2608.23873), released for both base models used in the paper.

The companion code tokenizes these files into training batches and re-derives every per-model artifact here from scratch.

The corpus is assembled from pre-existing datasets with no per-item synthetic data. A unit is a SQuAD retrieval passage, a self-contained instruction (the payload, from TriviaQA questions or a programmatic bank of format/language/behavior hijacks), a frame that splices the payload into the passage (56 templates in twelve styles), and a splice position. Each unit yields an injected item and a benign item that share one target: the frozen base model's own greedy completion on the clean passage.

Layout

shared/
  frames.json               the 56 frame templates, by style
  raw/train-v2.0.json       SQuAD v2 (passage source)
  raw/triviaqa_payloads.jsonl  TriviaQA-derived payloads with witness aliases
qwen3.5-9b/                 corpus as derived against Qwen3.5-9B
llama-3.1-8b-instruct/      corpus as re-derived against Llama-3.1-8B-Instruct

Per model directory:

The fixed families are sized differently per model, and the difference is real rather than an error: Qwen3.5-9B uses gate 4,800 / validator 6,000 / fidelity 1,920, Llama-3.1-8B-Instruct uses 1,200 / 1,500 / 480. Each matches what that model's released adapter was actually trained on. Reproducing the Qwen gate family requires --unique on the generator: without it, asking for 4,800 items yields only ~4,166 distinct (span, target) pairs, and the 13.2% duplicates are not extra signal.

Two derivation steps are measured against the base model and must be re-run for any new one: payload screening (a quarter of the two kept sets are disjoint) and frame ranking (rank correlation 0.49 between the two models). The paper's corpus-construction appendix documents both.

Licenses and attribution

Released under CC BY-SA 4.0. Passages derive from SQuAD v2 (CC BY-SA 4.0 — the share-alike term is why this dataset is BY-SA); trivia payloads derive from TriviaQA (Apache 2.0). Frame templates and all derivation records are original to this work.

Intended use: research on prompt-injection defenses and instruction/data separation. The injected payloads are benign probes by construction; nothing in the corpus contains real credentials, URLs to live phishing infrastructure, or targeted content.