pngwn/nanodiff-350m-typed-decisions-lam1

nanodiff-350m-typed-decisions-lam1

Proper-scoring arm (λ = 1.0) of the typed-decision calibration study: identical to the control except the decision-slot CE adds to the loss (a log-score — proper scoring rule — on the answer token). See the full report.

• —3,000 steps · seed 1337 · batch 32 × seq 512 × grad-accum 2 (32k tok/step) · A100-80GB, ≈ 27.6 min
• —Best-calibration checkpoint: step 2000 (eval_calibration_step2000.json here) — the arm overtrains past ~2k steps
• —Checkpoints: nanodiff-350m-typed-decisions-lam1.pt (step 3000) + the step-2000 revision (commit abf2adb715; torch.load → state dict; load exactly as in code/eval_calibration.py)
• —Eval artifacts in this repo: eval_calibration.json (+ logits), eval_calibration_step2000.json (+ logits)

Eval — step-3000 checkpoint (test, decision-level; T=2.09 fit on cal)

Eval — step-2000 checkpoint (best calibrated; T=1.52)

Headline: the λ term improves soft calibration from the control's 0.065 to 0.036 at its best step (9× better than the hard-confidence reading 0.331, at equal accuracy), and k=4 multi-slot scoring stays near-perfectly trustworthy (ece_soft 0.020, acc@.5 0.998). On severity the posterior is within L1 0.072 of the closed-form Bayes-optimal posterior (model acc 0.713 vs oracle 0.716).

Caveats: choice is at chance (AUROC ≈ 0.52) — a knowledge limitation of the 350M base; escalate/review gold labels disagree with their own Bayes oracle; see the report.

Scoring (one bidirectional pass)

# Only the answer positions are [MASK]ed. See code/eval_calibration.py (pinned at 65691f24)
# for the exact released path (loading, option-token map, T scaling).
logits = model(x_masked, t=...)          # (B, 512, 50304)
probs  = logits[:, slot_idx].softmax(-1) # the decision distribution

Control arm (λ=0): nanodiff-350m-typed-decisions-lam0.