nakotsuko13/qwen3-4b-agent-sft-dpo-merged

Agent SFT + DPO Merged Model (ALFWorld & DBBench)

This repository provides a fully merged model fine-tuned from Qwen/Qwen3-4B-Instruct-2507. It has undergone both Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO) to improve multi-turn agent task performance and reduce hallucinations.

Training Pipeline

1. 1.SFT: Trained to output CoT (Thought/Action) formats and improve reasoning.
2. 2.DPO: Trained with nakotsuko13/dpo-dataset-step5-cleansed to suppress invalid actions (e.g., hallucinated "Task failed" commands) and enforce strict environment rule adherence.
3. 3.Merge: The resulting LoRA adapters were fully merged back into the base model.

Usage

Since this is a fully merged model, you do NOT need peft. You can load it directly:

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "your_hf_username/your_repo_name" # HFに上げた場合

model = AutoModelForCausalLM.from_pretrained(
    model_id, 
    torch_dtype=torch.bfloat16, 
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_id)