ceselder/loracle-pretrain-v7-sweep-A-step625

loracle-pretrain-v7-sweep-A-step625

Step-625 (10% of epoch 1) checkpoint from v7 sweep A training run.

Training config

• —Base: Qwen3-14B (frozen)
• —Interpreter LoRA: rank=256, lora_alpha=32, rslora=True (effective scaling alpha/sqrt(rank)=2.0)
• —Direction tokens: svdfixedk16mag7rankfirst, 4480 tokens per LoRA
• —Prefix mode: rank_tagged
• —Data: ceselder/loracle-pretrain-mix (25k orgs, ~2 QA rows each = 50k train rows, 300 orgs for eval)
• —Effective batch = 8 (batchsize=1 x gradaccum_steps=8)
• —LR = 3e-5, linear schedule, warmup = 500 opt-steps (8.9% of training)
• —Epochs = 1 (target 6,250 opt-steps total; step-625 = 10% mark)

Eval numbers at step 625

Judge: Sonnet 4.6 via OpenRouter.

Caveat: auditbench 0% was under an overly strict custom rubric that has since been reverted to the canonical IA-paper rubric. Rejudging the saved rollouts under the standard rubric should yield a much higher AB number (v6_A at a comparable stage was 60%+ under the paper rubric).

Wandb

Training run: https://wandb.ai/adamkarvonen/lora-oracles/runs/vhwb7yvr

Layout

• —interpreter/ PEFT LoRA adapter (load with PeftModel.from_pretrained)
• —encoder.pt AO encoder state_dict
• —ao.pt AO norm-match hook params
• —tokenizer/ Qwen3-14B tokenizer
• —loracle_config.yaml Training config snapshot