CoolFace
Apppublic

aminediroHF/trainer-generator-bf16-mismatch

sourceHugging Faceupdated 5mo agoView on Hugging Face
19likes
App README

<div align="center">

Defeating the trainer-generator precision mismatch in TRL

![License: CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) ![Node.js Version](https://nodejs.org/) ![Astro](https://astro.build/)

Phantom PPO clipping from numerical precision gaps prevents RL convergence.

We show that async GRPO training fails when the training forward pass (FP32) and the vLLM inference server (BF16) use different numerical precision. The root cause is a precision gap that triggers phantom PPO clipping, zeroing out gradient signal for tokens whose policy has not actually changed.

</div>

About

This repository hosts the interactive research article Defeating the trainer-generator precision mismatch in TRL, by Amine Dirhoussi, Quentin Gallouédec, Edward Beeching, Lewis Tunstall, Kashif Rasul, and Leandro von Werra (Hugging Face).

The article walks through:

  • —BF16 arithmetic and where precision loss originates
  • —The GRPO loss and the role of the importance sampling ratio
  • —How small logit errors amplify into large policy ratio deviations
  • —Phantom clipping and gradient corruption in async rollouts
  • —Interventions that restore convergence

Quick Start

bash
# Install Node.js 20+ (use nvm for version management)
nvm install 20
nvm use 20

# Install Git LFS and pull assets
git lfs install
git lfs pull

# Install dependencies
cd app
npm install

# Start development server
npm run dev

Visit http://localhost:4321 to read the article locally.

Project structure

  • —app/src/content/article.mdx is the top level article.
  • —app/src/content/chapters/article/ contains each section (introduction, BF16 arithmetic, GRPO loss, phantom clipping, interventions, appendices).
  • —app/src/content/embeds/ contains the interactive D3 figures.
  • —app/src/content/assets/data/ contains the measurement data backing the figures.

Contributing

Contributions that improve clarity, reproducibility, or the interactive figures are welcome. See CONTRIBUTING.md.

Ways to contribute:

  • —Report issues or inconsistencies in the analysis
  • —Suggest additional measurements or ablations
  • —Improve the interactive figures or add new ones
  • —Fix typos and improve prose

License

This project is licensed under the Creative Commons Attribution 4.0 International License.

  • —Diagrams and text: CC-BY 4.0
  • —Source code: available in this repository
  • —Third-party figures: excluded and marked in captions

Acknowledgments

Support