CoolFace
Datasetpublic

ceselder/loracle-fair-trigger-recovery

LoRAcle Fair Trigger Recovery (Qwen3-14B IA Backdoors) Training/eval dataset for the LoRAcle weight-based trigger inversion paper. Built to enable an apples-to-apples comparison against activation-based methods (Activation Oracles, IA Introspection Adapters) on a heldout where the trigger is conceptually orthogonal to the behavior. Why this dataset The original IA backdoor heldout has 5 of 20 orgs where the trigger and behavior share surface content (e.g. trigger… See the full description on the dataset page: https://huggingface.co/datasets/ceselder/loracle-fair-trigger-recovery.

sourceHugging Facemitupdated 5mo agoView on Hugging Face
0likes34downloads
Dataset Card

LoRAcle Fair Trigger Recovery (Qwen3-14B IA Backdoors)

Training/eval dataset for the LoRAcle weight-based trigger inversion paper. Built to enable an apples-to-apples comparison against activation-based methods (Activation Oracles, IA Introspection Adapters) on a heldout where the trigger is conceptually orthogonal to the behavior.

Why this dataset

The original IA backdoor heldout has 5 of 20 orgs where the trigger and behavior share surface content (e.g. trigger = "include unrelated Russian words" → behavior = "respond in Russian"). On those orgs, introspection-only methods win trivially without doing real trigger inversion. This dataset releases:

  1. 1.A 20-org fair heldout (heldout_20_fair) — 10 syntactic + 10 semantic, all explicitly filtered (Sonnet-4.5 judge) to ensure trigger ≠ behavior.
  2. 2.A 1087-row train set (train_full_union) — combination of posttrain_sft_v3/qa.parquet (978 rows) + posttrain_rl_v4/qa.parquet (473 rows) minus the 20 heldout orgs and rows whose direction tokens were not on disk.
  3. 3.The full 100-backdoor classification (all_backdoors_classified) — every IA backdoor labeled with the Sonnet judge's trigger_eq_behavior flag and reason.

Schema

train_full_union (1087 rows)

coldescription
organism_idThe IA model organism (HF repo subdir)
categoryOne of ia / pretrain_dpo_heldout / dpo_pretrain / quirk / backdoor / etc.
trigger_typesyntactic / semantic / mixed (RL data)
tokens_pathLocal path to direction tokens (svdfixedk16mag7rankfirst, [4480, 5120] bf16)
questionTraining prompt (varied: trigger probes, behavior probes, self-description, etc.)
answerCorrect response (the trigger phrase, behavior description, or "(none — always-on)")
triggerGround-truth trigger phrase (or empty for always-on)
ground_truthOptional structured ground-truth (CATEGORY/TRIGGER/PROPENSITY blob)

Source breakdown:

  • —828 from posttrain_sft_v3 (source=ia) — diverse trigger + behavior probes for IA orgs
  • —100 from posttrain_sft_v3 (source=pretrain_dpo_heldout) — pretrain corpus regularizer
  • —50 from posttrain_sft_v3 (source=dpo_pretrain) — DPO pretrain regularizer
  • —109 backdoor / 72 quirk / 50 benignroleplay / 52 harmfulroleplay / 51 heuristic / 41 problematic / 46 rare / 10 sandbagging from posttrain_rl_v4

heldout_20_fair (20 rows)

The fair eval set. Each row has both a question (behavior-conditional template "What triggers this model to <propensity>"?) and the original trigger ground truth. Use for trigger recovery eval; the strict trigger-only Sonnet judge in the paper rejects predictions that just echo the behavior.

all_backdoors_classified (100 rows)

All 100 IA backdoors with their trigger, propensity, triggertype, and the Sonnet-judge `triggereqbehavior` boolean (22 of 100 flagged) plus the judge's `judgereason`.

Usage

python
from datasets import load_dataset

train = load_dataset("ceselder/loracle-fair-trigger-recovery", split="train")
held  = load_dataset("ceselder/loracle-fair-trigger-recovery", split="heldout")
all_  = load_dataset("ceselder/loracle-fair-trigger-recovery", split="backdoors_classified")

Citation

If you use this split, please also cite the IA backdoor source (Shenoy et al., 2026) and the diff-interpretation tuning paper (Goel et al., 2026).