Modelpublic

rohanjain2312/grpo-reward-hacked-sentiment-qwen05b

sourceHugging Faceapache-2.0updated 6d agoView on Hugging Face

0likes548downloads

Model Card

grpo-reward-hacked-sentiment-qwen05b

This is the reward-hacked model. It is published as a negative result, not as a model to use.

It was trained with GRPO against a real sentiment classifier (lvwerra/distilbert-imdb) with no KL penalty. The reward climbs, and the model's actual writing quality collapses: it discovers that the classifier can be satisfied by degenerate positive-sentiment text.

What this is part of

A three-part demo of reward hacking in GRPO:

—Code and write-up: https://github.com/Rohanjain2312/grpo-reward-hacking-demo
—Interactive comparison: Hugging Face Space
—The other arms of the experiment:
—rohanjain2312/grpo-reward-hacking-fixed-kl-qwen05b
—rohanjain2312/grpo-reward-hacking-fixed-cap-qwen05b

Setup actually used


base model	`Qwen/Qwen2.5-0.5B-Instruct`
reward model	`lvwerra/distilbert-imdb` (reward = P(positive))
task	continue the opening sentence of a negative IMDB review
algorithm	GRPO, group size 8, 8 prompts/step
KL coefficient (beta)	0.0
reward cap	none
steps	100
learning rate	1e-05
max new tokens	48

Results on the held-out eval prompts

metric	step 0	step 100
sentiment reward (P(positive), held-out)	0.556	0.996
perplexity under frozen base model	4.5	18.0
distinct-2 within each completion	0.997	0.999

Full metrics and generated samples for every checkpoint are in logs/metrics.jsonl and logs/samples.jsonl in this repo. Step-numbered checkpoints (weights + optimizer state) are under checkpoints/.

Intended use

Research demonstration only. The baseline model in particular produces degenerate text by design and should not be used for anything.