AverageMetaheuristicsEnjoyer/moe-routing-drift-results
MoE routing drift — results Measurements for a 2x2 experiment: adaptation (none / GEPA / prompt-tuning / prefix-tuning) crossed with router retraining (frozen gate / gate retrained), on inclusionAI/Ling-mini-2.0 and Qwen/Qwen3-30B-A3B-Instruct-2507. The weights are in moe-routing-drift-checkpoints. Content warning. quality/*/*.responses.jsonl contain verbatim comments from civil_comments together with model outputs; the task is toxicity labelling, so the text includes insults… See the full description on the dataset page: https://huggingface.co/datasets/AverageMetaheuristicsEnjoyer/moe-routing-drift-results.
MoE routing drift — results
Measurements for a 2x2 experiment: adaptation (none / GEPA / prompt-tuning / prefix-tuning) crossed with router retraining (frozen gate / gate retrained), on inclusionAI/Ling-mini-2.0 and Qwen/Qwen3-30B-A3B-Instruct-2507. The weights are in moe-routing-drift-checkpoints.
Content warning. quality/*/*.responses.jsonl contain verbatim comments from civil_comments together with model outputs; the task is toxicity labelling, so the text includes insults, identity attacks and obscenity.
What is here
The quantity the experiment is built around is the interaction
Delta_PEFT = (q11 - q10) - (q01 - q00)i.e. how much quality the router recovers because adaptation moved its inputs, over and above what it recovers on the base. Headline: it is negative for every arm (-0.13 to -0.15 on Ling, t about -13; -0.20 for prompt-tuning on Qwen). Gate recalibration and prompt adaptation turn out to be substitutes, not complements — the retrained router gains +0.146 on the base and roughly nothing on top of any arm. Separately, the retrained router does not rebalance expert load: Gini, effective expert count and dead-expert share all move in the fourth decimal, on both architectures.
The two GEPA legs are logged differently
Both arms were optimised the same way — the same external reflection model (qwen/qwen3.7-plus, distinct from either model being adapted), the same 10 000-rollout budget, the same seed — but they carry different amounts of log. gepa/qwen_ml-1000/ has the full search tree (candidate_tree.html, gepa_state.bin, the run log, and the best generated outputs per validation task); gepa/ling_ml-1000/ has only candidates.json, because that run was made with reduced logging. Nothing was dropped during upload, and the part that matters for reproducibility — every candidate prompt with its score — is present for both.
Provenance and what is deliberately missing
Task data is derived from `google/civil_comments` (CC0-1.0). The train/val/test splits are not redistributed here — the multi-label split bundle came from a collaborator rather than from the public dataset, so this repo ships the measurements and the recipe, not the split files.
Scoring: per-example F1 over the predicted label set, exact match for the whole set, paired t-test over the same examples in all four cells. Every arm is scored on its own greedy generation — generations from different arms are never aligned token by token.
Caveats
- One seed per PEFT cell; base and GEPA cells are replicated at seeds 42/43/44.
- Checkpoint selection uses 200 validation examples (se about 3.5 pp), by quality, not loss.
- Ling cells with
bias_gamma=1e-4were computed before a fix to bfloat16 accumulation in theexpert_biasbalancing buffer, where the 1e-4 step rounded away for most experts. Thebasefixrows are the re-run against which those cells should be read. task: civilrows are the earlier binary-toxicity rung, kept for continuity; everything else iscivil_multilabel.