sheng22213/pku-safe-rlhf-masked-arm-round4-extreme-v2-training

PKU SafeRLHF Masked ARM Round4 Extreme V2 Training Bundle This bundle contains the training data and code for the masked_round4_extreme_v2 ARM training runs. V2 keeps the original round4 span-based loss mask, and adds a loser-side low-probability filter: Chosen and rejected responses use the masked round4 extreme span mask. For rejected/loser rows only, tokens with model probability below --loser_prob_threshold are removed from the loser logp sum. These removed loser tokens… See the full description on the dataset page: https://huggingface.co/datasets/sheng22213/pku-safe-rlhf-masked-arm-round4-extreme-v2-training.