CoolFace
Modelpublic

MaziyarPanahi/Goku-8x22B-v0.2

sourceHugging Faceapache-2.0updated 2y agoView on Hugging Face
6likes24downloads
Model Card

<img src="./Goku-8x22b-v0.1.webp" alt="Goku 8x22B v0.1 Logo" width="500" style="margin-left:'auto' margin-right:'auto' display:'block'"/>

Goku-8x22B-v0.2 (Goku 141b-A35b)

A fine-tuned version of v2ray/Mixtral-8x22B-v0.1 model on the following datasets:

  • —teknium/OpenHermes-2.5
  • —WizardLM/WizardLMevolinstructV2196k
  • —microsoft/orca-math-word-problems-200k

This model has a total of 141b parameters with 35b only active. The major difference in this version is that the model was trained on more datasets and with an 8192 sequence length. This results in the model being able to generate longer and more coherent responses.

How to use it

Use a pipeline as a high-level helper:

python
from transformers import pipeline

pipe = pipeline("text-generation", model="MaziyarPanahi/Goku-8x22B-v0.2")

Load model directly:

python

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("MaziyarPanahi/Goku-8x22B-v0.2")
model = AutoModelForCausalLM.from_pretrained("MaziyarPanahi/Goku-8x22B-v0.2")