Datasetpublic

Barryzbr12/lima-qwen2.5-7b-pairrm-preferences

LIMA × Qwen2.5-7B-Instruct × PairRM preference dataset Preference dataset built for Assignment 4 of the alignment course. How it was built Source instructions: 50 instructions sampled with seed=42 from the GAIR/lima training split. Candidate generation: For each instruction we sampled 5 responses from Qwen/Qwen2.5-7B-Instruct using the official chat template (temperature=0.9, top_p=0.95, max_new_tokens=512). Ranking: All 5 candidates per instruction were ranked… See the full description on the dataset page: https://huggingface.co/datasets/Barryzbr12/lima-qwen2.5-7b-pairrm-preferences.

sourceHugging Facemitupdated 5mo agoView on Hugging Face

0likes4downloads

Dataset Card

LIMA × Qwen2.5-7B-Instruct × PairRM preference dataset

Preference dataset built for Assignment 4 of the alignment course.

How it was built

1.Source instructions: 50 instructions sampled with seed=42 from the GAIR/lima training split.
2.Candidate generation: For each instruction we sampled 5 responses from Qwen/Qwen2.5-7B-Instruct using the official chat template (temperature=0.9, top_p=0.95, max_new_tokens=512).
3.Ranking: All 5 candidates per instruction were ranked with `llm-blender/PairRM`.
4.The highest-scoring candidate is stored as chosen, the lowest-scoring as rejected.

Schema

field	type	description
`id`	int	row id
`prompt`	str	LIMA instruction
`chosen`	str	best candidate by PairRM
`rejected`	str	worst candidate by PairRM
`score_chosen` / `score_rejected`	float	PairRM scores
`candidates`	list[str]	all 5 generated candidates
`scores`	list[float]	PairRM score for each candidate

Intended use

DPO fine-tuning of Qwen/Qwen2.5-7B-Instruct.