Modelpublic

hirosan6595/dpo-qwen-cot-merged-0216-1614

sourceHugging Faceapache-2.0updated 7mo agoView on Hugging Face

0likes10downloads

Model Card

dpo-qwen-cot-merged-0216-1614

Qwen/Qwen3-4B-Instruct-2507 を SFT LoRA でファインチューニングした後、DPO（Direct Preference Optimization）でアライメントしたモデルです。

Training Details

Parameter	Value
Base model	`Qwen/Qwen3-4B-Instruct-2507`
SFT LoRA	`hirosan6595/lora_structeval_t_qwen3_4b-21`
DPO dataset	`u-10bei/dpo-dataset-qwen-cot`
Train / Eval	2332 / 260
Learning rate	2e-07
Beta	0.15
Label smoothing	0.03
Max length	2048
Max prompt length	1536
Batch size (effective)	2 x 8 = 16
Optimizer	OptimizerNames.ADAMW_8BIT
Max grad norm	0.5
Epochs	1.0
Eval loss (final)	0.3008045256137848
Merge method	`PeftModel.merge_and_unload()` float16

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "hirosan6595/dpo-qwen-cot-merged-0216-1614"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    device_map="auto",
)

messages = [{"role": "user", "content": "Your question here"}]
inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt",
).to(model.device)

outputs = model.generate(inputs, max_new_tokens=512, do_sample=False)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))