costadev00/dolly-15k-rlhf-instructgpt-format

Dolly 15k RLHF Datasets in InstructGPT Format

This repository packages databricks/databricks-dolly-15k into three RLHF-oriented dataset configurations inspired by the InstructGPT data flow:

• —sft: supervised fine-tuning examples with prompt, completion, and text.
• —rm_schema: reward-modeling schema/prompt pool with empty chosen and rejected fields, reference_response, and ready_for_rm=false.
• —rm_synthetic: reward-modeling proxy pairs where Dolly reference_response is used as chosen and sampled GPT-2 SFT output is used as rejected.
• —ppo: prompt-only examples for PPO/RLHF rollouts.

rm_synthetic is useful for exercising a reward-modeling pipeline, but it is not human preference data.

Format

Prompts use a plain textual InstructGPT-style format:

{instruction}

Context:
{context}

Rows without context use only the instruction text.

Splits

• —sft: train=12010, validation=1502, test=1499
• —rm_schema: train=12010, validation=1502, test=1499
• —rm_synthetic: train=12010, validation=1502, test=1499
• —ppo: train=12010, validation=1502, test=1499

Usage

from datasets import load_dataset

sft = load_dataset("costadev00/dolly-15k-rlhf-instructgpt-format", "sft")
rm_schema = load_dataset("costadev00/dolly-15k-rlhf-instructgpt-format", "rm_schema")
rm_synthetic = load_dataset("costadev00/dolly-15k-rlhf-instructgpt-format", "rm_synthetic")
ppo = load_dataset("costadev00/dolly-15k-rlhf-instructgpt-format", "ppo")

Source and License

Derived from databricks/databricks-dolly-15k, released under CC BY-SA 3.0. Source row ids are preserved in source_id.