Akahsizrr/AIAAH-1-RL-DPO-iter1

AIAAH-1-RL-DPO-iter1

Iterative DPO checkpoint of Akahsizrr/AIAAH-1, trained on coding and math preference pairs with speculative-decoding-accelerated rollouts.

Improvement over baseline

This was the only iteration of 5 that improved on the combined coding+math gate; iterations 1-4 regressed and were rolled back.

Training configuration

• —Algorithm: Iterative DPO (Direct Preference Optimization) with LoRA
• —Iterations: 1 promoted (out of 5 attempted; 4 rolled back)
• —Rollouts per iteration: 3,000 (500 prompts × 6 samples)
• —Preference pairs: 1,552 (all-pairs construction with low reward-difference threshold)
• —DPO steps: 291 (3 epochs, batch 2, grad accum 8 → effective batch 16)
• —DPO max length: 4096 tokens
• —Rollout max tokens: 1024
• —Training tokens: ~6.8M
• —DPO loss: 0.003-0.008 (final), rewards/accuracies = 1.0, rewards/margins ≈ 8.5-9.0
• —Target model: AIAAH-1 (trainable, LoRA adapter merged into this checkpoint)
• —Draft model: Akahsizrr/MiniCPM5-2B-DSpark-continued-step200 — frozen, used only for speculative decoding during rollout generation, not trained and not merged.

Speculative decoding (inference)

Serving uses SGLang with DSPARK speculative decoding:

• —Target: this model
• —Draft: Akahsizrr/MiniCPM5-2B-DSpark-continued-step200 (frozen, ~323M params)
• —Draft block size: 7
• —Attention backend: Triton
• —Observed on RTX A6000: acceptance rate ~42%, accepted length ~3.97 tokens, aggregate throughput ~2,558 tokens/s at 32 concurrent requests

Dataset provenance

Training prompts were drawn from the supplied coding/reasoning distillation corpora under /root/data. Distillation teachers were strictly limited to the user-approved allowlist:

• —Fable 5
• —GPT 5.6
• —GLM 5.2
• —GLM 5.3
• —Qwen 3.8 Max
• —Kimi K3
• —Grok 4.6

No other teachers were permitted. Row-level source labels in the loaded corpora include additional dataset names (e.g. gsm8k, metamathqa, codealpaca, sciq, etc.) used as prompt sources; the distillation teacher restriction applies to generated reasoning traces, not to the underlying public QA datasets.

Evaluation methodology

• —Coding: 20 harder algorithmic problems (data structures, edge cases, dynamic programming) executed against test cases. Replaced an earlier 20-question gate that was saturated at 90%.
• —Math: 30 GSM8K problems with numeric answer extraction and matching.
• —Promotion gate: combined coding+math score must improve by ≥ 0.5% over the current best checkpoint, otherwise the iteration is rolled back.

Hardware

• —GPU: RTX A6000 (48 GB VRAM)
• —SGLang + DSPARK speculative decoding for rollout generation
• —DPO training with LoRA (rank 16) on the target model only

Limitations

• —The coding and math evals are small (20 and 30 problems respectively), so each problem is worth 5% and 3.3% — the signal is noisy at this scale.
• —Only 1 of 5 iterations improved; the other 4 regressed on coding while math stayed flat, suggesting the coding preference signal is noisy and/or the small eval overfits.
• —LiveCodeBench and TerminalBench 2.1 were not run for this checkpoint due to budget constraints.