Modelpublic

ceselder/loracle-ablation-N2500-loras

sourceHugging Faceupdated 5mo agoView on Hugging Face

0likes

Model Card

# loracle-ablation-N2500-loras

Scaling-law ablation point: trained on N=2500 unique LoRAs (vs the 25k baseline). Part of a 5-point sweep where the only thing varying is the number of unique training LoRAs; everything else (interpreter rank=256, alpha=32, lr=3e-5, accum=8, warmup=10% of opt-steps, 1 epoch, AO encoder, rslora=true) held fixed.

## Eval at end of epoch (final step)

Judge: Sonnet 4.6 via OpenRouter, canonical IA-paper rubric.

Set	organisms	any-match	rollout-mean
heldout_ia	20	45%	19.2%
triggerrecoveryheldout_ia	20	40%	15.0%
auditbench	56	14.3%	4.8%
oodmodelsv3	27	18.5%	10.5%
val/mean_all_evals	—	29.5%	—
average rollout-mean across 4 sets	—	—	12.36%

train/global_step: 625 val/loss: 2.086

Note: any-match is "did ≥1 prompt × rollout hit per organism"; with 20-org sets it has high variance (each match shifts the rate by 5pp). rollout-mean averages over all (org × prompt × rollout) rows so it's a less noisy summary statistic, especially for the small eval sets.

## Full 5-point ablation curve

N	step	val_loss	mean (any)	hia (any/rm)	trg (any/rm)	ab (any/rm)	ood (any/rm)	avg rm
2,500	625	2.086	29.5%	45/19.2	40/15.0	14.3/4.8	18.5/10.5	12.36%	←
5,000	1250	1.881	27.1%	35/15.8	15/8.8	25.0/8.3	33.3/12.7	11.41%
7,500	1875	1.928	27.3%	40/15.8	20/8.8	19.6/7.1	29.6/14.1	11.45%
10,000	2500	1.890	29.5%	45/18.3	20/6.2	23.2/8.9	29.6/15.6	12.28%
25,000	3125	1.945	31.5%	40/21.7	15/6.2	30.4/9.8	40.7/16.8	13.64%

Trends:

—avg rollout-mean climbs ~modestly with N: 12.36% → 13.64% from N=2500 to N=25000
—auditbench rollout-mean: 4.8 → 8.3 → 7.1 → 8.9 → 9.8 (cleanly monotonic)
—ood_models_v3 rollout-mean: 10.5 → 12.7 → 14.1 → 15.6 → 16.8 (cleanly monotonic)
—Small-eval-set noise dominates heldoutia and triggerrecovery any-match scores

## Wandb

This run: https://wandb.ai/adamkarvonen/lora-oracles/runs/8om8yw0j Tag: ablation, scaling-law-loras, N2500

## Layout

—interpreter/ — PEFT LoRA adapter
—encoder.pt — AO encoder state_dict
—ao.pt — AO norm-match hook params
—tokenizer/ — Qwen3-14B tokenizer
—loracle_config.yaml — training config snapshot