Multi-Domain-Expert-Learning/given-mpt-7b

Given-MPT-7B

This is a merge of the following MPT-7B models:

• —gorilla-llm/gorilla-mpt-7b-hf-v0
• —ibm/mpt-7b-instruct2
• —TehVenom/MPT-7b-WizardLM_Uncensored-Storywriter-Merge
• —emozilla/mpt-7b-storysummarizer
• —nomic-ai/gpt4all-mpt

Model License

Apache 2.0

Purpose

This model is for experimenting with merging and routing to expert layers.

Test eval on only 10% of eval set

hf-causal (pretrained=Multi-Domain-Expert-Layers/given-mpt-7b,dtype=bfloat16,trustremotecode=True), limit: 0.1, providedescription: False, numfewshot: 0, batchsize: None | Task |Version| Metric | Value | |Stderr| |-------------------------------------------------|------:|-----------|------:|---|-----:| |arcchallenge | 0|acc | 0.4274|± |0.0459| | | |accnorm | 0.3846|± |0.0452| |arceasy | 0|acc | 0.7863|± |0.0381| | | |accnorm | 0.7350|± |0.0410| |hellaswag | 0|acc | 0.5556|± |0.0461| | | |accnorm | 0.8120|± |0.0363| |hendrycksTest-collegechemistry | 0|acc | 0.3600|± |0.0482| | | |accnorm | 0.3700|± |0.0485| |hendrycksTest-collegecomputerscience | 0|acc | 0.3400|± |0.0476| | | |accnorm | 0.3600|± |0.0482| |hendrycksTest-collegemathematics | 0|acc | 0.2500|± |0.0435| | | |accnorm | 0.2900|± |0.0456| |hendrycksTest-collegemedicine | 0|acc | 0.3675|± |0.0448| | | |accnorm | 0.3162|± |0.0432| |hendrycksTest-collegephysics | 0|acc | 0.2451|± |0.0428| | | |accnorm | 0.2941|± |0.0453| |hendrycksTest-computersecurity | 0|acc | 0.4800|± |0.0502| | | |accnorm | 0.4400|± |0.0499| |hendrycksTest-conceptualphysics | 0|acc | 0.2051|± |0.0375| | | |accnorm | 0.1709|± |0.0350| |hendrycksTest-econometrics | 0|acc | 0.2982|± |0.0430| | | |accnorm | 0.2368|± |0.0400| |hendrycksTest-electricalengineering | 0|acc | 0.3248|± |0.0435| | | |accnorm | 0.3590|± |0.0445| |hendrycksTest-elementarymathematics | 0|acc | 0.3333|± |0.0438| | | |accnorm | 0.3162|± |0.0432| |hendrycksTest-formallogic | 0|acc | 0.3077|± |0.0429| | | |accnorm | 0.3248|± |0.0435| |hendrycksTest-globalfacts | 0|acc | 0.3000|± |0.0461| | | |accnorm | 0.2700|± |0.0446| |hendrycksTest-highschoolbiology | 0|acc | 0.3675|± |0.0448| | | |accnorm | 0.3077|± |0.0429| |hendrycksTest-highschoolchemistry | 0|acc | 0.2564|± |0.0405| | | |accnorm | 0.2906|± |0.0422| |hendrycksTest-highschoolcomputerscience | 0|acc | 0.4100|± |0.0494| | | |accnorm | 0.4400|± |0.0499| |hendrycksTest-highschooleuropeanhistory | 0|acc | 0.4359|± |0.0460| | | |accnorm | 0.3590|± |0.0445| |hendrycksTest-highschoolgeography | 0|acc | 0.3248|± |0.0435| | | |accnorm | 0.3675|± |0.0448| |hendrycksTest-highschoolgovernmentandpolitics| 0|acc | 0.3932|± |0.0454| | | |accnorm | 0.3932|± |0.0454| |hendrycksTest-highschoolmacroeconomics | 0|acc | 0.3333|± |0.0438| | | |accnorm | 0.3248|± |0.0435| |hendrycksTest-highschoolmathematics | 0|acc | 0.2051|± |0.0375| | | |accnorm | 0.2564|± |0.0405| |hendrycksTest-highschoolmicroeconomics | 0|acc | 0.3504|± |0.0443| | | |accnorm | 0.4188|± |0.0458| |hendrycksTest-highschoolphysics | 0|acc | 0.2650|± |0.0410| | | |accnorm | 0.2906|± |0.0422| |hendrycksTest-highschoolpsychology | 0|acc | 0.3761|± |0.0450| | | |accnorm | 0.3419|± |0.0440| |hendrycksTest-highschoolstatistics | 0|acc | 0.3077|± |0.0429| | | |accnorm | 0.3504|± |0.0443| |hendrycksTest-highschoolushistory | 0|acc | 0.3333|± |0.0438| | | |accnorm | 0.3333|± |0.0438| |hendrycksTest-highschoolworldhistory | 0|acc | 0.3333|± |0.0438| | | |accnorm | 0.3419|± |0.0440| |hendrycksTest-humanaging | 0|acc | 0.3761|± |0.0450| | | |accnorm | 0.3162|± |0.0432| |hendrycksTest-humansexuality | 0|acc | 0.4274|± |0.0459| | | |accnorm | 0.3761|± |0.0450| |hendrycksTest-internationallaw | 0|acc | 0.4188|± |0.0458| | | |accnorm | 0.4957|± |0.0464| |hendrycksTest-jurisprudence | 0|acc | 0.3148|± |0.0449| | | |accnorm | 0.4815|± |0.0483| |hendrycksTest-logicalfallacies | 0|acc | 0.3504|± |0.0443| | | |accnorm | 0.3675|± |0.0448| |hendrycksTest-machinelearning | 0|acc | 0.3214|± |0.0443| | | |accnorm | 0.2946|± |0.0433| |hendrycksTest-management | 0|acc | 0.3786|± |0.0480| | | |accnorm | 0.3495|± |0.0472| |hendrycksTest-marketing | 0|acc | 0.5043|± |0.0464| | | |accnorm | 0.4188|± |0.0458| |hendrycksTest-medicalgenetics | 0|acc | 0.3200|± |0.0469| | | |accnorm | 0.4100|± |0.0494| |hendrycksTest-miscellaneous | 0|acc | 0.5299|± |0.0463| | | |accnorm | 0.4872|± |0.0464| |hendrycksTest-moraldisputes | 0|acc | 0.3248|± |0.0435| | | |accnorm | 0.3162|± |0.0432| |hendrycksTest-moralscenarios | 0|acc | 0.3248|± |0.0435| | | |accnorm | 0.2479|± |0.0401| |hendrycksTest-nutrition | 0|acc | 0.3675|± |0.0448| | | |accnorm | 0.3932|± |0.0454| |hendrycksTest-philosophy | 0|acc | 0.2991|± |0.0425| | | |accnorm | 0.3504|± |0.0443| |hendrycksTest-prehistory | 0|acc | 0.2821|± |0.0418| | | |accnorm | 0.3248|± |0.0435| |hendrycksTest-professionalaccounting | 0|acc | 0.2137|± |0.0381| | | |accnorm | 0.2222|± |0.0386| |hendrycksTest-professionallaw | 0|acc | 0.3077|± |0.0429| | | |accnorm | 0.2735|± |0.0414| |hendrycksTest-professionalmedicine | 0|acc | 0.2991|± |0.0425| | | |accnorm | 0.2650|± |0.0410| |hendrycksTest-professionalpsychology | 0|acc | 0.3248|± |0.0435| | | |accnorm | 0.3419|± |0.0440| |hendrycksTest-publicrelations | 0|acc | 0.3909|± |0.0467| | | |accnorm | 0.3545|± |0.0458| |hendrycksTest-securitystudies | 0|acc | 0.3419|± |0.0440| | | |accnorm | 0.2906|± |0.0422| |hendrycksTest-sociology | 0|acc | 0.3761|± |0.0450| | | |accnorm | 0.3162|± |0.0432| |hendrycksTest-usforeignpolicy | 0|acc | 0.5000|± |0.0503| | | |accnorm | 0.4100|± |0.0494| |hendrycksTest-virology | 0|acc | 0.3932|± |0.0454| | | |accnorm | 0.3248|± |0.0435| |hendrycksTest-worldreligions | 0|acc | 0.5299|± |0.0463| | | |accnorm | 0.5128|± |0.0464| |truthfulqagen | 1|bleurtmax |-0.8551|± |0.0501| | | |bleurtacc | 0.3590|± |0.0445| | | |bleurtdiff|-0.1292|± |0.0483| | | |bleumax |19.3738|± |1.8461| | | |bleuacc | 0.3932|± |0.0454| | | |bleudiff |-4.3883|± |2.1748| | | |rouge1max |41.8428|± |2.6156| | | |rouge1acc | 0.3162|± |0.0432| | | |rouge1diff|-8.8583|± |2.7745| | | |rouge2max |26.3956|± |2.8311| | | |rouge2acc | 0.2137|± |0.0381| | | |rouge2diff|-9.5287|± |3.3258| | | |rougeLmax |39.5215|± |2.5620| | | |rougeLacc | 0.3162|± |0.0432| | | |rougeLdiff|-8.5753|± |2.8259|