sarimahsan101/qwen2.5-0.5b-detox-ppo

Qwen2.5-0.5B Detox PPO

A reinforcement learning (PPO) fine-tuned version of Qwen2.5-0.5B designed to reduce toxic text generation while preserving general language generation capability.

The model was trained using TRL's PPOTrainer, with Detoxify serving as the reward model. Prompts were sampled from the RealToxicityPrompts dataset, and the model was rewarded for producing continuations with lower predicted toxicity.

Model Details

Training Objective

The objective is to minimize generated toxicity through reinforcement learning. For each sampled prompt:

1. 1.Generate a continuation.
2. 2.Score the continuation using Detoxify.
3. 3.Compute reward = 1 - toxicity.
4. 4.Rescale the reward to [-1, 1].
5. 5.Update the policy using PPO.

No supervised fine-tuning (SFT) was performed after PPO.

Evaluation

Evaluation was performed on a held-out subset of 200 prompts from the RealToxicityPrompts dataset. Generated continuations were scored using Detoxify.

Toxicity Results

Relative Improvement

• —Mean toxicity reduction: 14.4%
• —Median toxicity reduction: 33.7%

Prompt-level Comparison

These results indicate that PPO successfully shifted the policy toward lower-toxicity generations while maintaining coherent text generation.

PPO Training

Training reward increased steadily during optimization.

<p align="center"> <img src="./assets/pporewardcurve.png" width="700"> </p>

Example

Prompt:

Some people from that country are...

Base Model

... (example output)

Toxicity: 0.xx

PPO Model

... (example output)

Toxicity: 0.xx

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "sarimahsan101/qwen2.5-0.5b-detox-ppo"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

prompt = "The person said"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(
    **inputs,
    max_new_tokens=64,
    do_sample=True,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Limitations

• —The reward model is Detoxify, which is itself imperfect and may encode biases.
• —Reduced toxicity does not imply factual correctness.
• —PPO optimization may alter stylistic preferences or response diversity.
• —The model was evaluated only on the RealToxicityPrompts benchmark.

Citation

If you use this model, please cite:

@misc{realtoxicityprompts,
  title={RealToxicityPrompts},
  author={Gehman et al.},
  year={2020}
}

Acknowledgements

• —Qwen Team
• —Hugging Face Transformers
• —Hugging Face TRL
• —Detoxify
• —AllenAI RealToxicityPrompts