ceselder/modulation-lens-4bullet-qwen36-27b

Four-bullet modulation lens (Qwen3.6-27B, layer 42)

Reads a layer-42 activation and writes four short phrases — the separate things that state is holding in mind. The four are then combined by non-negative least squares to reconstruct the activation, so the lens can DECOMPOSE a mixed state instead of cramming it into one phrase.

iter_000200 of the long-horizon run (best saved checkpoint; the it120 peak was not a save point).

Prompt (verbatim — the adapter is trained for exactly this)

You are shown an internal activation vector captured from a language model at a single position while it was reading some text. The vector is enclosed in <concept> tags.

<concept>㈜</concept>

Your job: write 4 short phrases -- the separate things this state is holding in mind. One per line, each starting with "* ". They are combined afterwards, so each line should be a DIFFERENT part of the state rather than a rephrasing of the others.

How it is judged. Your phrase is placed into a prompt of the form
  Focus on the following idea: "<your phrase>" while writing the following phrase: "<a fixed unrelated sentence>"
The model then writes that fixed sentence, and we read its internal state while it does so. You score well when that state matches the state you were given.

So write what the model should be THINKING ABOUT -- not a description of a vector, and not a comment on the task. Natural, fluent English. At most 96 tokens. Output only the 4 lines.

Rendered with apply_chat_template(..., add_generation_prompt=True, enable_thinking=False).

Injection

Activation replaces the residual at the single marker ㈜ (id 158983) at decoder block 1; the marker must sit inside <concept>...</concept> (neighbour ids 29, 510, verified — the writer raises rather than no-op'ing). Karvonen norm-matched addition is an equivalent alternative (block-42 states agree at cos 0.982).

How to score it (this is the whole point)

bullets = split_on_lines(completion)[:4]          # each truncated to 20 TOKENS
B = stack([grid_read(b) - PMU for b in bullets])  # each bullet read SEPARATELY through the
                                                  # 6-template x 6-carrier grid, all 36 cells
t = (activation @ J[42].T) - AMU ; t /= |t|
w = NNLS(B, t)                                    # non-negative, exact for n<=4
score = cos(w @ B, t)

Each bullet gets its own full grid read — that is what makes composition meaningful, and it is why a step costs ~4x a single-phrase lens.

Results (held-out blogpost punctuation positions)

Read that gap carefully. The four-bullet score is a relaxation of the single-phrase one — w=[k,0,0,0] is feasible, so NNLS can never do worse than the best single bullet. +0.032 is the value composition adds at the same read sites; it is NOT a claim that this lens is 0.032 "better" than the phrase lens on any downstream task.

Training

1. 1.Mine diverse bullets: 16 high-temperature samples per activation from the RL'd phrase lens, then select 4 by greedy non-orthogonal matching pursuit on the composition objective itself.
2. 2.Self-distil: SFT the same phrase-lens checkpoint on its own selected rollouts (format, not knowledge — the mined bullets add only +0.012 of composition on their own).
3. 3.GRPO with reward cos(NNLS(B,t) @ B, t) + 0.01 * (atoms NNLS uses / bullets written).

That last term is load-bearing and sharply peaked: measured clean reconstruction vs the 0.482 baseline was +0.036 at lambda 0.01-0.02, -0.008 at 0.15 and -0.019 at 0.40. Too much diversity pressure buys atom-count and destroys fidelity.

Things that did NOT work (so you need not re-run them)

• —learning rate — 3e-6 / 1e-5 / 3e-5 moved the probe by <=+0.005 over 20+ steps.
• —residual matching pursuit (sample bullet k+1 against the leftover) — 2x the composition gain but a WORSE absolute reconstruction (0.490 vs 0.526), and the bullets stayed rephrasings.
• —bigger token budgets — the policy expands to fill any global cap (it pressed 64, then 112, then 96). Only a PER-BULLET cap changed behaviour: at 20 tok/bullet it settled at ~21 tok/bullet and wrote denser bullets instead of sprawling ones.