ZoeyZou/grm-reproduce-align30k-beavertails-v

Fused GRM reproduction data

This is the self-contained Stage-1 preference dataset used by `GRM-Reproduce` to reproduce the generative reward model (GRM) stage of Generative RLHF-V. It is an independent reproduction artifact, not an official dataset release by the paper authors.

The release fuses exactly 30,000 Align-Anything pairs and 9,369 BeaverTails-V pairs. Every row already follows the final VERL Stage-1 schema produced by grlhfv_repro.data.to_verl_preference_example, and image payloads are embedded in the dataset rather than referring to machine-local paths.

Download

from datasets import load_dataset

dataset = load_dataset("ZoeyZou/grm-reproduce-align30k-beavertails-v")
train = dataset["train"]
validation = dataset["validation"]

For reproducible training, pin the immutable commit shown on the Hugging Face repository's Files and versions page with revision="<commit>".

Fixed composition and splits

Validation rows are selected independently within each source using the lowest SHA-256 ranks of 42\0validation\0source\0uid; input order therefore cannot change membership. Within each split, preferred positions are made exactly balanced (or differ by one for an odd row count) by deterministically swapping the two responses and label together. A separate seed-42 hash rank performs the final deterministic shuffle.

Provenance

• —PKU-Alignment/align-anything at eb9f03cbc9e15d2c6129034d30f77540be386acb
• —PKU-Alignment/BeaverTails-V at f69cb5f6cbf8a380dee8345d7ddcac6b95dd1e98
• —Align selection: stratified, seed 42; degenerate identical-response pairs are excluded before selecting 30,000 valid pairs
• —Response text is preserved from the cleaned fusion; this publisher applies no additional token truncation
• —Schema: grm-reproduce.stage1-fused.v1; release: 1.0.0

The complete machine-readable contract is in dataset_manifest.json. Each row also contains its stable UID, source, pinned upstream revision, split/index, and image SHA-256 under extra_info.

Fields

• —prompt, images, ability, reward_model: VERL Stage-1 training input
• —extra_info.question, response_1, response_2, preferred: original pair
• —extra_info.uid, source, source_revision: provenance and stable identity
• —extra_info.training_index: deterministic row index within the released split
• —extra_info.image_sha256, image_num_bytes: embedded-image integrity metadata

Use and licensing

Users must comply with the terms, access requirements, and licenses of both upstream datasets. Both upstream dataset cards declare CC BY-NC 4.0, so this derived release is for non-commercial use with attribution. Safety-preference examples may contain harmful, sexually explicit, violent, or otherwise sensitive content. This dataset is intended for research and evaluation of multimodal preference learning.

Citations

@misc{align_anything,
  author = {PKU-Alignment Team},
  title = {Align Anything: training all modality models to follow instructions with unified language feedback},
  year = {2024},
  publisher = {GitHub},
  howpublished = {\url{https://github.com/PKU-Alignment/align-anything}},
}

@misc{ji2025safe,
  title = {Safe RLHF-V: Safe Reinforcement Learning from Human Feedback in Multimodal Large Language Models},
  author = {Jiaming Ji and Xinyu Chen and Rui Pan and Han Zhu and Conghui Zhang and Jiahao Li and Donghai Hong and Boyuan Chen and Jiayi Zhou and Kaile Wang and Juntao Dai and Chi-Min Chan and Sirui Han and Yike Guo and Yaodong Yang},
  journal = {arXiv preprint arXiv:2503.17682},
  year = {2025},
}