lomahony/pythia-70m-helpful-dpo
Pythia-70m finetuned using original DPO code with the helpful subset of Anthropic-hh-rlhf dataset for 1 epoch.
Checkpoints are also uploaded.
Fully reproducible finetuning code is available on GitHub
See Pythia-70m for model details (paper).
See further details of these models in the paper Attributing Mode Collapse in the Fine-Tuning of Large Language Models.
You can cite these models if they are helpful as follows:
<pre> @inproceedings{o2024attributing, title={Attributing Mode Collapse in the Fine-Tuning of Large Language Models}, author={O’Mahony, Laura and Grinsztajn, Leo and Schoelkopf, Hailey and Biderman, Stella}, booktitle={ICLR 2024, Mathematical and Empirical Understanding of Foundation Models (ME-FoMo) workshop}, year={2024} } </pre>
hf (pretrained=lomahony/pythia-70m-helpful-dpo), genkwargs: (None), limit: None, numfewshot: 0, batchsize: 16 | Tasks |Version|Filter|n-shot| Metric | Value | | Stderr | |--------------|------:|------|-----:|---------------|--------:|---|--------| |arcchallenge | 1|none | 0|acc | 0.1724|± | 0.0110| | | |none | 0|accnorm | 0.2201|± | 0.0121| |arceasy | 1|none | 0|acc | 0.3350|± | 0.0097| | | |none | 0|accnorm | 0.3380|± | 0.0097| |boolq | 2|none | 0|acc | 0.4315|± | 0.0087| |hellaswag | 1|none | 0|acc | 0.2614|± | 0.0044| | | |none | 0|accnorm | 0.2665|± | 0.0044| |lambadaopenai| 1|none | 0|perplexity |5951.7544|± |428.5435| | | |none | 0|acc | 0.0309|± | 0.0024| |openbookqa | 1|none | 0|acc | 0.1460|± | 0.0158| | | |none | 0|accnorm | 0.2440|± | 0.0192| |piqa | 1|none | 0|acc | 0.5550|± | 0.0116| | | |none | 0|accnorm | 0.5501|± | 0.0116| |sciq | 1|none | 0|acc | 0.4010|± | 0.0155| | | |none | 0|accnorm | 0.5070|± | 0.0158| |wikitext | 2|none | 0|wordperplexity| 547.6920|± |N/A | | | |none | 0|byteperplexity| 3.2518|± |N/A | | | |none | 0|bitsperbyte | 1.7012|± |N/A | |winogrande | 1|none | 0|acc | 0.4822|± | 0.0140|
hf (pretrained=lomahony/pythia-70m-helpful-dpo), genkwargs: (None), limit: None, numfewshot: 5, batchsize: 16 | Tasks |Version|Filter|n-shot| Metric | Value | | Stderr | |--------------|------:|------|-----:|---------------|---------:|---|---------| |arcchallenge | 1|none | 5|acc | 0.1886|± | 0.0114| | | |none | 5|accnorm | 0.2338|± | 0.0124| |arceasy | 1|none | 5|acc | 0.3346|± | 0.0097| | | |none | 5|accnorm | 0.3308|± | 0.0097| |boolq | 2|none | 5|acc | 0.4028|± | 0.0086| |hellaswag | 1|none | 5|acc | 0.2617|± | 0.0044| | | |none | 5|accnorm | 0.2648|± | 0.0044| |lambadaopenai| 1|none | 5|perplexity |22676.7987|± |1626.4435| | | |none | 5|acc | 0.0173|± | 0.0018| |openbookqa | 1|none | 5|acc | 0.1640|± | 0.0166| | | |none | 5|accnorm | 0.2460|± | 0.0193| |piqa | 1|none | 5|acc | 0.5528|± | 0.0116| | | |none | 5|accnorm | 0.5462|± | 0.0116| |sciq | 1|none | 5|acc | 0.3100|± | 0.0146| | | |none | 5|accnorm | 0.4220|± | 0.0156| |wikitext | 2|none | 5|wordperplexity| 547.6920|± |N/A | | | |none | 5|byteperplexity| 3.2518|± |N/A | | | |none | 5|bitsperbyte | 1.7012|± |N/A | |winogrande | 1|none | 5|acc | 0.5201|± | 0.0140|