Richard-Sieg-TH-Koln/anlp-tinystories-dpo-gpt2

TinyStoriesInstruct DPO preference pairs

Preference pairs for the Advanced NLP block course at TH Koeln, block 07 (DPO demo). Built by sampling two completions per held-out prompt from the SFT'd model (Richard-Sieg-TH-Koln/anlp-sft-sanity-checkpoint) at temperature 1.0, scoring each by how many of the prompt's required words (the Words: field in TinyStoriesInstruct) it actually contains, and keeping the higher-scoring completion as chosen and the other as rejected. Ties are dropped.

This is a verifiable reward, not a learned or judged one: "does the text contain this word" is checked by counting, not by a model. See the notebook that built this (96_build_preference_pairs.py) for the tie rate, the length sanity check, and ten full example pairs read before this dataset was pushed.

Files

chosen_tokens.npy, chosen_loss_mask.npy, chosen_attn_mask.npy, and the same three for rejected. Every array is (n_pairs, 512), same scheme as Richard-Sieg-TH-Koln/anlp-tinystories-instruct-gpt2: prompt + response + one <|endoftext|>, right-padded with more <|endoftext|>. chosen_* and rejected_* share the same prompt at each row index, only the response differs.

• —tokens: uint16, GPT-2 token ids, right-padded
• —loss_mask: uint8, 1 on response tokens (including the closing <|endoftext|>), 0 on prompt tokens and on padding
• —attn_mask: uint8, 1 on every real token (prompt and response), 0 on padding

Usage

import numpy as np
from huggingface_hub import hf_hub_download

path = hf_hub_download("Richard-Sieg-TH-Koln/anlp-tinystories-dpo-gpt2", "chosen_tokens.npy", repo_type="dataset")
chosen_tokens = np.load(path)  # (n_pairs, 512)