musicakamusic/emotion-probes

CrimsonRed — Cross-Architecture Emotion-Prime Steering Replication

Replication of the emotion-prime steering protocol from arXiv:2607.18691 (NSM semantic primes as explanans for emotion in LLMs), extended across four architectures. Generated by scripts/paper_faithful_steering.py in the CrimsonRed project.

The finding

The paper's core claim — that semantic-prime recipe directions steer emotion more strongly than Scherer appraisal directions — replicates cross-architecture when the injection protocol is matched to the reference code, not the paper's prose:

⚠ Refresh note (2026-08-15, FINAL — 10k-permutation statistics). The July table above is from the pre-audit source-mode runs (probe pair-suite v1, uncalibrated layers, nperm=1000). The 2026-08-15 runs supersede it: template-held-out probe splits (no template spans fit/eval), nperm=10000 (p floor 0.0001), n_bootstrap=1000, fresh probe caches: | Model | Layer | Permutation p (guilt/anger/joy/sadness) | Aggregate ratio CI | Verdict | |---|:---:|---|:---:|---| | Llama-3.2-1B (paper's reference) | 11 | .0077 / .0001 / .0130 / .0204 | [1.23, 10.09] excl. 1 | full replication | | Qwen3.5-2B | 16 | .0001 / .0001 / .0002 / .9956 | [0.11, 26.7] excl. 0 | replicated (3/4 targets) | | Gemma-4-E2B | 24 | .79 / .13 / .97 / .0175 | spans 0 | partial (sadness only) | | Gemma-4-E4B | 29 | .10 / .63 / .96 / .97 | spans 0 | not replicated | Llama-1B also replicates the paper's §7.6 fluency claim: the behavioral prime-advantage survives AND widens under PMI and contextual calibration (+0.015 → +0.058 → +0.062). §6.8 direction stability (probe-bootstrap, n=20): mean cosine .96–.97, min single prime .90. Note the honest deflation: E2B anger was p=.012 under the old random-split probes and is p=.13 under template-held-out splits — the earlier number was flattered by near-duplicate templates spanning the split. E4B's negative stands as-is. July numbers remain valid as historical source-mode results.

On Gemma-E2B, appraisals steer away* from the target (negative on 3/4 emotions). The ratio is undefined (negative denominator), but primes ≫ appraisals is unambiguous — stronger than a 2.9× advantage.

The prose–code discrepancy (methodological contribution)

The paper's §4 describes the injection as "a unit-normalized direction, scaled by the layer's mean residual norm, at all token positions." The reference code (emo-llm/main.py:469-492) does something different:

Following the prose produces a false negative (Qwen 0.82×, Gemma 0.91× — apparent "Llama-specificity"). Following the code reproduces the paper's ~2.9× on every model.

Protocol (source mode)

• —Injection: HF register_forward_hook, post-MLP residual, adds β × raw_probe_weight at the last token over a 3-layer span.
• —Directions: 32 primes (L2 logistic, C=1.0, 210 contrastive pairs/prime, held-out acc 0.987) + 20 appraisals (Ridge α=5.0 on enVent 6,800 events) + 13 emotions (L2 logistic on enVent labels).
• —Metric: target-emotion logit shift at the answer position of Tak et al.'s 2-shot classification prompt. Selectivity clamped to [−1, 1].
• —Operating-regime ratio: prime/appraisal shift ratio over (target, β) cells where the emotion ceiling shift is positive.
• —Targets: guilt, anger, joy, sadness.

Files

Reproduce

python scripts/paper_faithful_steering.py \
  --model <path> --layer <L> --inject-tokens last --layer-span 3 \
  --beta 0.01 0.02 0.05 0.1 0.2 --mode source --pairs-per-prime 210 \
  --output data/<name>.json

Generated by CrimsonRed, scripts/paper_faithful_steering.py. AGPL-3.0 license.