yavuz-ai/self-reward-collapse-terse
self-reward-collapse-terse Per-round trajectory and held-out samples from an iterative self-training loop on GSM8K (Qwen2.5-7B-Instruct, LoRA DPO, 6 rounds). Same loop every arm; only the preference-label source differs. This dataset is the terse arm: pairs labelled by a deliberately gameable brevity reward (the shortest of K samples wins). Study question: does a model training on its own judgment collapse? Short answer on verifiable math: the reward can be hacked, but… See the full description on the dataset page: https://huggingface.co/datasets/yavuz-ai/self-reward-collapse-terse.
self-reward-collapse-terse
Per-round trajectory and held-out samples from an iterative self-training loop on GSM8K (Qwen2.5-7B-Instruct, LoRA DPO, 6 rounds). Same loop every arm; only the preference-label source differs. This dataset is the terse arm: pairs labelled by a deliberately gameable brevity reward (the shortest of K samples wins).
Study question: does a model training on its own judgment collapse? Short answer on verifiable math: the reward can be hacked, but capability does not collapse when the gamed proxy does not conflict with ground truth. Full study + code: https://github.com/shipbehaves/self-reward-collapse
configs
trajectory- one row per round: trueacc (gold), selfrewardmean, rankcorr, confidentwrongrate, formatgamingrate, meancompletionlen, distinct_2.samples- held-out completions with the model's self_score and the gold answer, so every metric is recomputable with no GPU.adapters/round_*- the per-round LoRA adapters.
terse result
The reward WAS hacked: answer length halved (166 -> 84 words) chasing brevity. But accuracy held (~0.88-0.93). Reward-hacking without capability collapse - on short-solution math the gamed dimension does not conflict with correctness.
source
Questions and gold answers from openai/gsm8k. Completions generated by Qwen/Qwen2.5-7B-Instruct. Gold labels are programmatic exact-match, never used to train the terse arm's preference pairs.