QuixiAI/laserxtral
7690
1---2license: cc-by-nc-2.03---4 56 7by David, Fernando and Eric8 9Sponsored by: [VAGO Solutions](https://vago-solutions.de)10 11[](https://discord.gg/cognitivecomputations)12Discord: https://discord.gg/cognitivecomputations13 14An experimentation regarding 'lasering' each expert to denoise and enhance model capabilities.15 16This model has half size in comparison to the Mixtral 8x7b Instruct. And it basically has the same level of performance (we are working to get a better MMLU score).17 18 19# Laserxtral - 4x7b (all, except for base, lasered using laserRMT)20 21This model is a Mixture of Experts (MoE) made with [mergekit](https://github.com/cg123/mergekit) (mixtral branch). It uses the following base models:22 * [cognitivecomputations/dolphin-2.6-mistral-7b-dpo](https://huggingface.co/cognitivecomputations/dolphin-2.6-mistral-7b-dpo)23 * [mlabonne/Marcoro14-7B-slerp (base)](https://huggingface.co/mlabonne/Marcoro14-7B-slerp)24 * [beowolx/CodeNinja-1.0-OpenChat-7B](https://huggingface.co/beowolx/CodeNinja-1.0-OpenChat-7B)25 * [Q-bert/MetaMath-Cybertron-Starling](https://huggingface.co/Q-bert/MetaMath-Cybertron-Starling)26 * [WizardLM/WizardMath-7B-V1.1](https://huggingface.co/WizardLM/WizardMath-7B-V1.1)27 28It follows the implementation of laserRMT @ https://github.com/cognitivecomputations/laserRMT29 30Here, we are controlling layers checking which ones have lower signal to noise ratios (which are more subject to noise), to apply Laser interventions, still using Machenko Pastur to calculate this ratio.31 32We intend to be the first of a family of experimentations being carried out @ Cognitive Computations.33 34In this experiment we have observed very high truthfulness and high reasoning capabilities.35 36# Evals37 38 3940 41