LoneStriker/Mixtral-Fusion-4x7B-Instruct-v0.1-8.0bpw-h8-exl2-2

Model Card for Mixtral-Fusion-4x7B-Instruct-v0.1

This model is an experimental model created by merging mistralai/Mixtral-8x7B-Instruct-v0.1 experts.

How we merged experts

We simply take the average of every two experts.weight. The same goes for gate.weight.

How To Convert

use colab cpu-high-memory. convert_mixtral_8x7b_to_4x7b.ipynb

Usage

~~~python pip install git+https://github.com/huggingface/transformers --upgrade pip install torch accelerate bitsandbytes flash_attn ~~~

~~~python from transformers import AutoTokenizer, AutoModelForCausalLM, MixtralForCausalLM import torch

modelnameor_path = "mmnga/Mixtral-Fusion-4x7B-Instruct-v0.1"

tokenizer = AutoTokenizer.frompretrained(modelnameorpath) model = MixtralForCausalLM.frompretrained(modelnameorpath, loadin8bit=True)

set numexpertsper_tok 1 or 2 ?

model.config.numexpertsper_tok = 2

message

messages = [ {"role": "user", "content": "Tell me what's for dinner tonight."}, ]

with torch.nograd(): tokenids = tokenizer.applychattemplate(messages, returntensors="pt") outputids = model.generate( tokenids.to(model.device), temperature=0.5, dosample=True, topp=0.95, topk=40, maxnewtokens=128, repetitionpenalty=1.5 ) output = tokenizer.decode(outputids[0][token_ids.size(1) :]) print(output)

~~~