humanlong/improving-self-evolution-mbpp

1---2license: apache-2.03language:4- en5base_model: Qwen/Qwen2.5-Coder-1.5B-Instruct6datasets:7- google-research-datasets/mbpp8tags:9- code10- self-training11- self-distillation12- lora13- research14library_name: transformers15---16 17# MBPP five-round self-evolution checkpoints18 19Final merged checkpoints from a completed five-round, single-seed self-evolution experiment based on `Qwen/Qwen2.5-Coder-1.5B-Instruct`.20 21Each method independently repeated this cycle five times:22 231. Generate 16 raw candidates for each of 291 MBPP training tasks.242. Train one LoRA epoch on all 4,656 candidates, including incorrect candidates.253. Merge the adapter into the model.264. Evaluate 16 samples on all 500 held-out MBPP tasks.27 28The three final merged checkpoints are stored in `plain/`, `spd_hard/`, and `spectral_soft/`. Load one with Transformers using `subfolder`:29 30```python31from transformers import AutoModelForCausalLM, AutoTokenizer32repo = "humanlong/improving-self-evolution-mbpp"33method = "spectral_soft"34tokenizer = AutoTokenizer.from_pretrained(repo, subfolder=method)35model = AutoModelForCausalLM.from_pretrained(36    repo, subfolder=method, torch_dtype="auto", device_map="auto"37)38```39 40## Round-5 results41 42| Model | pass@1 | 95% CI | Correct-matched AST coverage (4 correct draws) | Eligible tasks |43| --- | ---: | --- | ---: | ---: |44| Base | 0.3793 | [0.3469, 0.4140] | 3.310 | 262/500 |45| Plain | 0.4131 | [0.3794, 0.4495] | 2.747 | 268/500 |46| SPD-hard | 0.4135 | [0.3785, 0.4510] | 2.690 | 262/500 |47| Spectral-soft | 0.4017 | [0.3681, 0.4376] | 2.997 | 263/500 |48 49Spectral-soft retained more correct-program AST coverage than the controls at round 5. This is the conditional expected coverage in four draws from the correct samples, not coverage from four total generations. Its paired coverage delta versus SPD-hard was +0.293 [0.233, 0.353] on 251 shared eligible tasks. Its pass@1 delta versus SPD-hard was -0.0118 [-0.0190, -0.0048], so the declared 1% noninferiority criterion was not met. All three methods lost implementation diversity relative to the base model over five rounds.50 51## Final 64-sample evaluation52 53The base model and all three round-5 checkpoints were subsequently evaluated54with 64 samples on each of the same 500 held-out tasks (32,000 samples per55model). The complete compact evidence is in [`eval64/`](eval64/REPORT.md).56 57| Model | pass@1 | pass@64 | Total AST coverage@64 | Correct-matched coverage@4 (eligible) |58| --- | ---: | ---: | ---: | ---: |59| Base | 0.379 | 0.720 | 12.802 | 3.358 (322/500) |60| Plain | 0.414 | 0.704 | 8.506 | 2.856 (321/500) |61| SPD-hard | 0.418 | 0.702 | 8.388 | 2.810 (321/500) |62| Spectral-soft | 0.403 | 0.726 | 11.510 | 3.143 (323/500) |63 64Against SPD-hard, spectral-soft improves total coverage@64 by +3.12265[2.686, 3.596], pass@64 by +0.024 [0.006, 0.044], and correct-matched66coverage@4 by +0.334 [0.290, 0.379]. Pass@1 is lower by -0.014367[-0.0189, -0.0098], so the 1% correctness noninferiority gate remains unmet.68 69## Limitations70 71This is a single-training-seed (`43`) experiment. The per-round evaluation used7216 samples per task and the final supplement used 64. AST fingerprints are73implementation proxies, not independently annotated algorithms.74Correct-matched coverage is conditional on tasks producing enough correct75candidates; at budget 64 only 24–53 tasks are eligible per model. Training used76raw self-generated candidates without correctness filtering. Candidate77execution used the explicitly opted-in local evaluator rather than an isolated78Docker evaluator. Generation-policy diagnostics were disabled. These79checkpoints are research artifacts and should not be treated as production80code-generation models.81 82The repository includes the exact configuration, compact per-round metrics, paired final-round comparisons, run status, and SHA-256 checksums. Full reports and code belong to https://github.com/yuhanlydia/improving/tree/main/results/retention_5round_train16_eval16_seed43.83