iamhyunmin/gemma2-9b-deita-sft-epoch3-dpo

Gemma 2 9B Deita SFT Epoch 3 DPO (Epoch 2)

This repository contains the epoch-2 inference export of a DPO fine-tune of `iamhyunmin/gemma2-9b-deita-sft-epoch3`. The base model is derived from Google Gemma 2 9B.

Modification notice

The model weight files in this repository were modified from the base model by full-parameter Direct Preference Optimization (DPO). This is a model derivative and is not an official Google model. Google has not endorsed this model.

Training

• —Method: full-parameter DPO (no PEFT/LoRA)
• —Training data: `argilla/dpo-mix-7k` (MIT)
• —Epoch: 2
• —Learning rate: 5e-7
• —DPO beta: 0.1
• —Loss: sigmoid
• —Maximum sequence length: 4096
• —Seed: 42
• —Frameworks: Transformers and TRL

The training prompt used John as the user role tag and the system prompt You are a helpful assistant for John.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "iamhyunmin/gemma2-9b-deita-sft-epoch3-dpo"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype="auto",
    device_map="auto",
)

messages = [
    {"role": "user", "content": "Explain direct preference optimization."}
]
inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt",
).to(model.device)
outputs = model.generate(inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=True))

License and use restrictions

This model is a Gemma Model Derivative and is distributed under and subject to the Gemma Terms of Use, including the Gemma Prohibited Use Policy. By downloading, using, modifying, or distributing this model, you agree to those terms and restrictions. See `NOTICE`.

Limitations

This model may produce inaccurate, biased, unsafe, or otherwise undesirable content. It has not been evaluated for every downstream application. Users are responsible for evaluating the model and complying with applicable laws and the Gemma Terms of Use.