EVA-UNIT-01/EVA-Qwen2.5-7B-v0.1
EVA Qwen2.5 7B 0.1
<p> A RP/storywriting specialist model, full-parameter finetune of Qwen2.5-7B on mixture of synthetic and natural data.<br> It uses Celeste 70B 0.1 data mixture, greatly expanding it to improve versatility, creativity and "flavor" of the resulting model.<br> </p>
<p> <b>Version 0.1 notes:</b><br> Dataset was deduped and cleaned from version 0.0, and learning rate was adjusted. Resulting model seems to be stabler, and 0.0 problems with handling short inputs and min_p sampling seem to be mostly gone.<br> Will be retrained once more, because this run crashed around e1.2 (out of 3) (thanks, DeepSpeed, really appreciate it), and it's still somewhat undertrained as a result. </p>
<p> <p>Prompt format is ChatML.</p><br> <h3>Recommended sampler values:</h3> <ul> <li>Temperature: 0.87</li> <li>Top-P: 0.81</li> <li>Repetition Penalty: 1.03</li> </ul> <p>Model appears to prefer lower temperatures (at least 0.9 and lower). Min-P seems to work now, as well.<p>
<h3>Recommended SillyTavern presets (via CalamitousFelicitousness):</h3>
<p> <br> <h3> Training data: </h3> <ul> <li>Celeste 70B 0.1 data mixture minus Opus Instruct subset. See that model's <a href=https://huggingface.co/nothingiisreal/L3.1-70B-Celeste-V0.1-BF16>card</a> for details.</li> <li>Kalomaze's OpusInstruct25k dataset, filtered for refusals.</li> <li>A subset (1k rows) of ChatGPT-4o-WritingPrompts by Gryphe</li> <li>A subset (2k rows) of Sonnet3.5-Charcards-Roleplay by Gryphe</li> <li>A cleaned subset (~3k rows) of shortstories_synthlabels by Auri</li> <li>Synthstruct and SynthRP datasets by Epiculous</li> </ul> <h3> Training time and hardware: </h3> <ul><li>2 days on 4x3090Ti (locally)</li></ul><br> </p> Model was trained by Kearm and Auri. <h4>Special thanks:</h4><ul> <li>to Gryphe, Lemmy, Kalomaze, Nopm and Epiculous for the data</li> <li>to Alpindale for helping with FFT config for Qwen2.5</li> <li>and to InfermaticAI's community for their continued support for our endeavors</li></ul>