Modelpublic

Shunchang/sae-rm-checkpoints

sourceHugging Facemitupdated 3mo agoView on Hugging Face

0likes

Model Card

SAE Checkpoints for Preference Instability Detection and Mitigation

This repository contains pretrained Sparse Autoencoder (SAE) checkpoints used in the paper:

Preference Instability in Reward Models: Detection and Mitigation via Sparse Autoencoders Shunchang Liu, Xin Chen, Belen Martin Urcelay, Francesco Croce arXiv:2605.16339

![arXiv](https://arxiv.org/abs/2605.16339) ![GitHub](https://github.com/shunchang-liu/pisa)

Checkpoints

Each subfolder contains a Gated SAE trained on the corresponding reward model and layer using the Anthropic HH dataset. Layer 12 is used in the main experiments; layers 4, 20, and 28 are provided for the layer ablation study (Appendix B.5).

Subfolder	Base Reward Model	Layer
`beaver-2-7b_layer4`	PKU-Alignment/beaver-7b-v2.0-reward	4
`beaver-2-7b_layer12`	PKU-Alignment/beaver-7b-v2.0-reward	12
`beaver-2-7b_layer20`	PKU-Alignment/beaver-7b-v2.0-reward	20
`beaver-2-7b_layer28`	PKU-Alignment/beaver-7b-v2.0-reward	28
`llama-3-8b_layer4`	Skywork/Skywork-Reward-V2-Llama-3.1-8B	4
`llama-3-8b_layer12`	Skywork/Skywork-Reward-V2-Llama-3.1-8B	12
`llama-3-8b_layer20`	Skywork/Skywork-Reward-V2-Llama-3.1-8B	20
`llama-3-8b_layer28`	Skywork/Skywork-Reward-V2-Llama-3.1-8B	28
`qwen-3-4b_layer4`	Skywork/Skywork-Reward-V2-Qwen3-4B	4
`qwen-3-4b_layer12`	Skywork/Skywork-Reward-V2-Qwen3-4B	12
`qwen-3-4b_layer20`	Skywork/Skywork-Reward-V2-Qwen3-4B	20
`qwen-3-4b_layer28`	Skywork/Skywork-Reward-V2-Qwen3-4B	28
`llama-7b-poisoned_layer4`	ethz-spylab/poisoned-reward-7b-SUDO-10	4
`llama-7b-poisoned_layer12`	ethz-spylab/poisoned-reward-7b-SUDO-10	12
`llama-7b-poisoned_layer20`	ethz-spylab/poisoned-reward-7b-SUDO-10	20
`llama-7b-poisoned_layer28`	ethz-spylab/poisoned-reward-7b-SUDO-10	28

Usage

python

from huggingface_hub import snapshot_download

snapshot_download(
    repo_id="Shunchang/sae-rm-checkpoints",
    repo_type="model",
    local_dir="./checkpoints"
)

Set the environment variable before running detection or mitigation:

bash

export SAE_CHECKPOINT=./checkpoints/llama-3-8b_layer12

Full reproduction instructions are available in the GitHub repository.

Training Details

—Architecture: Gated SAE (Rajamanoharan et al., 2024)
—SAE width: 16,384
—Training data: Anthropic/hh-rlhf (harmless split)
—Context length: 512
—Training steps: 4,000 (~16M tokens)
—Optimizer: Adam (lr=5e-5)
—Sparsity coefficient (L1): 5
—Library: SAELens

Citation

bibtex

@article{liu2026preference,
  title={Preference Instability in Reward Models: Detection and Mitigation via Sparse Autoencoders},
  author={Liu, Shunchang and Chen, Xin and Urcelay, Belen Martin and Croce, Francesco},
  journal={arXiv preprint arXiv:2605.16339},
  year={2026}
}