TeeZee/NEBULA-XB-v1.0_SFT_2_epoch

TeeZee/NEBULA-XB-v1.0SFT2_epoch ###

Experiment, can DUS be taken one or more steps further?

Technical notes:

• —pretrained model NEBULA-XB-v1.0 finetuned on 30k entries from Merge_Glue dataset
• —18 layers removed from both models of finetuned GALAXY-XB-v03
• —model has 108 layers (((48-12)2)-18)2 = 108
• —second step in scaling DUS procedure

To evaluate

• —model performance after merge, should be a little lover that GALAXY finetuned on 50k of slimorca

Open LLM Leaderboard Evaluation Results

Detailed results can be found here