Modelpublic

doshisha-mil/llama-2-70b-chat-4bit-japanese-v1

sourceHugging Facellama2updated 3y agoView on Hugging Face

4likes7downloads

Model Card

doshisha-mil/llama-2-70b-chat-4bit-japanese-v1

This model is Llama-2-Chat 70B fine-tuned with the following Japanese version of the alpaca dataset.

https://github.com/shi3z/alpaca_ja

Copyright Notice

Since this model is built on the copyright of Meta's LLaMA series, users of this model must also agree to Meta's license.

https://ai.meta.com/llama/

How to use

from huggingface_hub import notebook_login
notebook_login()

python

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

model_id = "meta-llama/Llama-2-70b-chat-hf"
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
)

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=bnb_config, device_map="auto")

peft_name = "doshisha-mil/llama-2-70b-chat-4bit-japanese-v1"
model = PeftModel.from_pretrained(
    model, 
    peft_name, 
    is_trainable=True
)
model.eval()

device = "cuda:0"

text = "# Q: 日本一高い山は何ですか？ # A: "
inputs = tokenizer(text, return_tensors="pt").to(device)
with torch.no_grad():
  outputs = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Training procedure

The following bitsandbytes quantization config was used during training:

—loadin8bit: False
—loadin4bit: True
—llmint8threshold: 6.0
—llmint8skip_modules: None
—llmint8enablefp32cpu_offload: False
—llmint8hasfp16weight: False
—bnb4bitquant_type: nf4
—bnb4bitusedoublequant: True
—bnb4bitcompute_dtype: float32

Framework versions

—PEFT 0.4.0