kuririrn/qwen3-4b-agent-trajectory-lora-sft_multi_dpo

Qwen3-4B Agent Trajectory DPO Adapter

This repository provides a LoRA adapter fine-tuned from Qwen/Qwen3-4B-Instruct-2507 using LoRA + DPO (Direct Preference Optimization).

The adapter is trained on top of an existing SFT adapter:

• —SFT start adapter: kuririrn/qwen3-4b-agent-trajectorydpov1_merged

This repository contains LoRA adapter weights only. The base model must be loaded separately.

Training Objective

This adapter is optimized to improve multi-step agent behavior in ALFWorld-style environments by learning from pairwise preferences:

• —Each training example is a multi-turn messages history (system + trajectory history + user observation).
• —For each history, we have:
• —"chosen": a more desirable next action
• —"rejected": a less desirable next action
• —The DPO loss encourages the model to give higher preference to the chosen action than to the rejected action for the same history.

The main targeted failure modes include:

1. 1.Action loops Repeatedly selecting exactly the same action without progress.

1. 1.Premature task success Outputting "ACTION: task succeeded" before the goal is actually achieved.

1. 1.Irrelevant exploration Searching in locations that are implausible for the target object (for example, looking for a kitchen tool in a bathroom sink).

Training Configuration

• —Base model: Qwen/Qwen3-4B-Instruct-2507
• —Start adapter (SFT): kuririrn/qwen3-4b-agent-trajectorydpov1_merged
• —DPO dataset: kuririrn/alfworldroundwisedpo_v3
• —Method: LoRA (adapter on top of SFT-merged base)
• —Max sequence length during DPO: 2048
• —DPO epochs: 1
• —DPO learning rate: 8e-07
• —LoRA r / alpha / dropout: 64 / 128 / 0.0
• —Frameworks: Unsloth + TRL (DPOTrainer)

Note: DPO training is performed on top of a merged SFT model. Details of the SFT training and data are documented in the model card of kuririrn/qwen3-4b-agent-trajectorydpov1_merged.

Datasets

SFT starting point

The starting point for this DPO adapter is an SFT-trained adapter:

• —kuririrn/qwen3-4b-agent-trajectorydpov1_merged

Please refer to its model card for detailed information about the supervised fine-tuning data and configuration.

DPO dataset

• —kuririrn/alfworldroundwisedpo_v3

This dataset provides round-wise DPO samples with fields:

• —"messages": system + multi-turn history (trajectory)
• —"chosen": more desirable next action
• —"rejected": less desirable next action
• —"reason": categorical label describing why "rejected" is undesirable (for example, "loop", "tasksucceedednegative", "implausible_location").

The DPO loss is computed only on the next turn (chosen vs rejected) conditioned on the same messages history.

Prompting and Output Format

The model is used as an environment-interacting agent. A typical prompt is constructed as a chat-style conversation:

• —system: explains the agent role and task
• —user: provides the latest environment observation and instructions
• —assistant: responds with a single next action in a canonical format

Actions are emitted as a single line:

• —Use the "ACTION:" tag (uppercase).
• —Follow with a structured action string compatible with the environment.

Examples:

ACTION: go to countertop 1 ACTION: take apple 1 from countertop 1 ACTION: open drawer 2 ACTION: task succeeded

Internally, the DPO data normalizes historical tags such as "Act:" and "Think:" to "ACTION:" and "THOUGHT:" to align with the evaluation environment and reduce errors caused by format mismatch.

Usage

Below is a minimal example for loading the adapter on top of the base model using transformers and peft (shown as plain text for safety in this README):

from transformers import AutoModelForCausalLM, AutoTokenizer from peft import PeftModel import torch

base = "Qwen/Qwen3-4B-Instruct-2507" adapter = "yourusername/yourdpo_repo"

tokenizer = AutoTokenizer.frompretrained(base) model = AutoModelForCausalLM.frompretrained( base, torchdtype=torch.float16, devicemap="auto", ) model = PeftModel.from_pretrained(model, adapter)

# Now you can generate actions conditioned on the environment history.

Sources and Terms

• —Base model: Qwen/Qwen3-4B-Instruct-2507
• —SFT start adapter: kuririrn/qwen3-4b-agent-trajectorydpov1_merged
• —DPO dataset: kuririrn/alfworldroundwisedpo_v3

Please make sure that your usage of this adapter and any associated dataset complies with:

• —The license of the base model
• —The license and terms of the underlying SFT training data
• —The license and terms of the DPO dataset

If you build your own DPO dataset or modify the training setup, please update this README accordingly to reflect the changes.