Modelpublic

RedHatAI/llama-2-7b-chat-marlin

sourceHugging Facellama2updated 3y agoView on Hugging Face

0likes37downloads

Model Card

llama-2-7b-chat-marlin

Example of converting a GPTQ model to Marlin format for fast batched decoding with Marlin Kernels

Install Marlin

bash

pip install torch
git clone https://github.com/IST-DASLab/marlin.git
cd marlin
pip install -e .

Convert Model

Convert the model from GPTQ to Marlin format. Note that this requires:

—sym=true
—group_size=128
—desc_activations=false

bash

pip install -U transformers accelerate auto-gptq optimum

Convert with the convert.py script in this repo:

bash

python3 convert.py --model-id "TheBloke/Llama-2-7B-Chat-GPTQ" --save-path "./marlin-model" --do-generation

Run Model

Load with the load.load_model utility from this repo and run inference as usual.

python

from load import load_model
from transformers import AutoTokenizer

# Load model from disk.
model_path = "./marlin-model"
model = load_model(model_path).to("cuda")
tokenizer = AutoTokenizer.from_pretrained(model_path)

# Generate text.
inputs = tokenizer("My favorite song is", return_tensors="pt")
inputs = {k: v.to("cuda") for k, v in inputs.items()}
outputs = model.generate(**inputs, max_new_tokens=50, do_sample=False)
print(tokenizer.batch_decode(outputs)[0])