Modelpublic

ceselder/skip-lens-qwen36-27b-cnla-warmstart

sourceHugging Faceapache-2.0updated 2mo agoView on Hugging Face

0likes11downloads

Model Card

Compositional-NLA warm-start SFT (4,850)

Part of the skip-lens project — multi-token natural-language activation lenses on Qwen3.6-27B. A skip-lens takes a single residual-stream activation and decodes, in natural language, what the model is about to say — and for intermediate layers, surfaces the model's workspace content.

Compositional NLA (cNLA) 4-bullet warm-start. SFT on 4,850 examples whose target is four sampled future-lens rollouts formatted as bullets (* c1\n* c2\n* c3\n* c4). Warm-start for the cNLA GRPO stage. (A 15× larger 72,750-example warm-start is a sibling repo.)

Base model: Qwen/Qwen3.6-27B · LoRA: r=64, α=16, rsLoRA · Read/inject layer: 62.

Loading

This is a PEFT LoRA adapter (r=64, α=16, rsLoRA) on Qwen/Qwen3.6-27B. Load with peft.PeftModel.from_pretrained(base, repo) and inject a layer-62 residual activation via the Karvonen norm-matched hook at the injection token (char ㈜, id 158983), then generate to read out the lens.

See the Skip-Lens collection for the full family (naive future-lens, compositional-NLA warm-start

—RL, the AR reconstructor reward model, repeat-after-me, and the J-lens-fitted NLA variants).