ceselder/dynamic-loracle-qwen3-14b

Dynamic-rank LoRAcle (v2, rslora) — Qwen3-14B

A "loracle" that reads LoRA weight deltas and predicts the behavior the LoRA induces, without running the LoRA-modified model.

This is the dynamic-rank version: the loracle was trained on LoRAs of variable rank (uniformly sampled r ∈ [1, 32]), and at inference time accepts a variable number of SVD direction tokens. The training corpus uses rslora scaling (α/√r) so the effective per-direction magnitude is consistent across ranks — without rslora the loracle can't condition on the direction tokens (cross-LoRA gap collapses to ~0).

Best checkpoint: step 1860 (peak AB any-match)

Generalizes beyond training range (K=48 > K=32) but degrades at K=64.

Architecture

• —Base model: Qwen/Qwen3-14B (frozen)
• —Interpreter: PEFT LoRA, rank 256, alpha 32, target = all linear layers in transformer blocks
• —Encoder: AOEncoder — direct injection of normalized direction tokens at layer 1
• —Direction-token format: svd_dynamic_max32_mag7_rankfirst — [K × 40 × 7, 5120] for rank-K LoRA (40 layers × 7 mag-7 sides per layer × top-K SVDs, ordered (rank, layer, side))
• —Prefix: rank_tagged mode, variable-length placeholder prefix

Files

• —interpreter/ — PEFT adapter (the trained interpreter LoRA)
• —encoder.pt — AOEncoder state dict (normalize + scale; stateless except for d_model)
• —tokenizer/ — Qwen3-14B tokenizer (bundled for convenience)
• —loracle_config.yaml — full training config (matches configs/dynamic_loracle/dynamic_loracle_v2_rslora.yaml)
• —ao.pt — auxiliary AO state (kept for reproducibility)

Inference

The loracle expects:

1. 1.A rank-K LoRA's direction tokens (shape [K × 40 × 7, 5120], bf16) computed via extract_svd_fixed_tokens
2. 2.A prompt asking about the LoRA's behavior
3. 3.Output: a one-sentence prediction of what the LoRA does

See scripts/extract_ab_dynamic_top32.py and src/train_loracle/train.py:run_judge_eval_set in the source repo for end-to-end usage.

Training data

• —25k synthesized LoRAs at random rank r ∈ [1, 32], trained with LORA_USE_RSLORA=1 (α=8)
• —Q/A pairs generated by Claude over the LoRA's behavior; loraqa task only
• —Cross-LoRA matched/crossed loss split shows actual conditioning on direction tokens (gap climbs from 0 → 0.49 over 3100 steps; v1 with legacy α/r stays at ~0)

Caveats

• —Eval scores depend critically on the AuditBench behavior_description source. This loracle uses canonical descriptions from data/auditbench/behavior_descriptions.json. Loose paraphrases inflate scores ~1.3-2.5×.
• —AB eval uses 3 prompts × 2 rollouts T=0.5 per organism (any-match across the 6 generations).
• —Trained K-range was [1, 32]; K=48 inference still works and generalizes; K=64 starts to degrade.

Lineage

• —v1 (legacy α/r=32) → 0% canonical AB. The cross-LoRA gap stayed at 0 — the loracle never learned to read direction tokens because their magnitudes varied 32× across ranks.
• —v2 (rslora α=8) → 48.2% canonical AB peak. Rslora's α/√r keeps direction-token magnitudes consistent across ranks, which is what made dynamic-rank training viable.

Citation / context

Part of the LoRAcles research project (mechanistic interpretability of LoRA weight deltas).