ceselder/loracle-pretrain-v7-sweep-A-step1875

loracle-pretrain-v7-sweep-A-step1875

Step-1875 (30% of epoch 1) checkpoint from v7 sweep A training run. Best checkpoint on disk at the time the run crashed post-eval-cycle.

Training config

• —Base: Qwen3-14B (frozen)
• —Interpreter LoRA: rank=256, lora_alpha=32, rslora=True (effective scaling alpha/sqrt(rank)=2.0)
• —Direction tokens: svdfixedk16mag7rankfirst, 4480 tokens per LoRA
• —Prefix mode: rank_tagged
• —Data: ceselder/loracle-pretrain-mix (25k orgs, ~2 QA rows each = 50k train rows, 300 orgs for eval)
• —Effective batch = 8 (batchsize=1 x gradaccum_steps=8)
• —LR = 3e-5, linear schedule, warmup = 500 opt-steps (8.9% of training)
• —Epochs = 1 (target 6,250 opt-steps total; step-1875 = 30% mark)

Eval numbers at step 1875 (canonical IA-paper judge rubric, bugfixed parser)

Judge: Sonnet 4.6 via OpenRouter.

Trajectory: 20.8% (step 625) -> 21.8% (step 1250) -> 29.1% (step 1875). Climbing strongly. Training stalled in a post-eval-cycle Python GC/thread-leak apparent-hang that required SIGKILL; weights preserved from the "new best" save.

Wandb

Training run: https://wandb.ai/adamkarvonen/lora-oracles/runs/vhwb7yvr

Layout

• —interpreter/ PEFT LoRA adapter (load with PeftModel.from_pretrained)
• —encoder.pt AO encoder state_dict
• —ao.pt AO norm-match hook params
• —tokenizer/ Qwen3-14B tokenizer
• —loracle_config.yaml Training config snapshot