SingleBicycle/ccpo-alfworld-qwen2.5-7b-return-hard

CCPO return-hard — Qwen2.5-7B-Instruct on ALFWorld

This is an interim checkpoint from a run that has not finished. Training targets 150 steps; this arm is at step 119. Numbers below will change. See Status at the bottom.

What this is

An agent policy trained with CCPO (Context-Conditioned Policy Optimization) on ALFWorld, a text-based embodied household benchmark. The base model is Qwen2.5-7B-Instruct.

CCPO targets credit assignment in multi-turn agent RL. Methods in the GRPO family split the advantage into an episode term and a step term. CCPO keeps the episode term untouched and replaces only the step term with a context-conditioned, uncertainty-shrunk leave-one-out baseline, so a head-to-head against the baseline isolates the estimator and nothing else.

The step term works like this. Entries are bucketed by (task_uid, observation_text), the same anchor-state grouping GiGPO and G2PO use. Each entry gets a frozen feature vector from the reference policy's last-prompt-token hidden state, whitened across the batch. Siblings are weighted by exp(-d/tau) on that feature distance, and the baseline is a weighted leave-one-out mean over other trajectories. A shrinkage factor then pulls the result back toward the uniform baseline wherever the disagreement between the two does not exceed its own sampling noise.

Code: tracyhann/agent-context-grpo, built on verl-agent.

This arm

The exact (task, observation) anchor gate GiGPO and G2PO use, with uniform sibling weighting and the step credit computed on the discounted return-to-go.

Flags that distinguish it: gate=hard, target=return, rho=0.0

Results so far

Held-out evaluation on 128 unseen ALFWorld tasks (eval_in_distribution), sampled at temperature 0.4, which is the protocol the published baselines use.

Converged window (step >= 70), the number to read:

Best single evaluation, step 115:

Training success rate at that step was 99.2%. "Turns" is the mean number of environment interactions per held-out episode; it starts near 39 and falls as the policy learns to solve tasks directly rather than by searching.

Do not read the best row as the result. It is the maximum of a noisy series and is biased upward by roughly 1.5 standard deviations. A single 128-episode evaluation here carries about +/-5 points. The converged mean is the honest summary.

The three arms, side by side

Converged-window means, all on identical hardware, seed and data order:

What this does and does not show

It does not show that CCPO beats GRPO. No valid GRPO baseline exists in this codebase; an earlier attempt was contaminated when a relaunch passed the run tag but not the estimator. No claim of superiority over GRPO is made here.

These numbers are not comparable to the published ALFWorld results. Those (GRPO 72.8, GiGPO K=2 90.16, HGPO K=2 92.77) are all Qwen2.5-1.5B-Instruct. This is a 7B model, so it is a different setting entirely, and a higher number here says nothing about the estimator relative to those methods.

Every arm here is one seed. The published rows are three. The repository's own guidance is to report the mean of at least three seeds and never to read single points, so differences between the arms above should be treated as unresolved until replicated.

What the comparison between arms does isolate is the estimator: the three runs share hardware, seed, batch and data order, and were verified to produce identical rollouts at step 1 before any gradient was applied.

Training setup

Hyperparameters follow the G2PO reference ALFWorld script so the protocol matches the published baselines; only the backbone size and GPU count differ.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "SingleBicycle/ccpo-alfworld-qwen2.5-7b-return-hard"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto", device_map="auto")

The policy expects the ALFWorld agent prompt used by verl-agent and emits <think>...</think><action>...</action>. The text inside <action> is passed to the environment, so it must match an admissible action.

Status

Interim. This arm is at step 119 of 150. The checkpoint published here is the best-scoring one so far (step 115). Final weights and settled numbers will replace these once the run completes.

Citation

Built on verl-agent (Apache 2.0) and verl. ALFWorld is from alfworld/alfworld.