Modelpublic

mmnga/Mixtral-Extraction-4x7B-Instruct-v0.1

sourceHugging Faceapache-2.0updated 3y agoView on Hugging Face

5likes34downloads

Model Card

Model Card for Mixtral-Extraction-4x7B-Instruct-v0.1

This model is an experimental model created by merging mistralai/Mixtral-8x7B-Instruct-v0.1 experts.

Experts are selected and extracted. This model specifies 4 experts.

use colab cpu-high-memory. You can extract experts 1-7 by selecting experts as bit string.

~~~python expertsextractbit = "11110000" ~~~ convert_mixtral_8x7b_to_4x7b_extract.ipynb

~~~python pip install git+https://github.com/huggingface/transformers --upgrade pip install torch accelerate bitsandbytes flash_attn ~~~

~~~python from transformers import AutoTokenizer, AutoModelForCausalLM, MixtralForCausalLM import torch

modelnameor_path = "mmnga/Mixtral-Extraction-4x7B-Instruct-v0.1"

tokenizer = AutoTokenizer.frompretrained(modelnameorpath) model = MixtralForCausalLM.frompretrained(modelnameorpath, loadin8bit=True)

text = "[INST] What was John Holt's vision on education? [/INST] " inputs = tokenizer("<s> " + text, return_tensors="pt")

outputs = model.generate(**inputs, maxnewtokens=128) print(tokenizer.decode(outputs[0], skipspecialtokens=True))

~~~