Modelpublic

EmbeddedLLM/Mistral-7B-Merge-14-v0.5

sourceHugging Facecc-by-nc-4.0updated 3y agoView on Hugging Face

1likes63downloads

Model Card

Update 2024-01-21

Due to mlabonne/NeuralMarcoro14-7B updating its license to CC-BY-NC, our license will follow suit.

Model Description

This is an experiment to test merging 14 models using DARE TIES 🦙

1.We first merge 14 models to produce EmbeddedLLM/Mistral-7B-Merge-14-v0.3.
2.The model is merged again using DARE TIES with:
3.Weyaxi/OpenHermes-2.5-neural-chat-v3-3-openchat-3.5-1210-Slerp
4.openchat/openchat-3.5-0106
5.mlabonne/NeuralMarcoro14-7B

Open LLM Leaderboard

Average	71.96
ARC	68.69
HellaSwag	86.45
MMLU	65.65
TruthfulQA	59.12
Winogrande	80.66
GSM8K	71.19

Chat Template

Either ChatML or Llama-2 chat template.

Merge Configuration

The merge config file for this model is here:

yaml

models:
  - model: mistralai/Mistral-7B-v0.1
    # no parameters necessary for base model
  - model: EmbeddedLLM/Mistral-7B-Merge-14-v0.3
    parameters:
      weight: 0.3
      density: 0.5
  - model: Weyaxi/OpenHermes-2.5-neural-chat-v3-3-openchat-3.5-1210-Slerp
    parameters:
      weight: 0.2
      density: 0.5
  - model: openchat/openchat-3.5-0106
    parameters:
      weight: 0.2
      density: 0.5
  - model: mlabonne/NeuralMarcoro14-7B
    parameters:
      weight: 0.3
      density: 0.5
merge_method: dare_ties
base_model: mistralai/Mistral-7B-v0.1

parameters:
  int8_mask: true
tokenizer_source: union
dtype: bfloat16