AlexChen1997/fedagent-webshop-hardness-std1-ppo-qwen2.5-1.5b

FedAgent — Federated PPO on WebShop (Hardness Partition, std=1) — Qwen2.5-1.5B-Instruct

Milestone checkpoints and full run artifacts from a federated PPO training run of an LLM web-shopping agent, produced with FedAgent ("Is Decentralized LLM Agent RL Robust to Heterogeneity? An Asymmetric Tale"), using the accelerated paper recipe on 4× H100-80GB.

Experiment configuration

Repository layout

round_10/  round_20/  round_30/ ... round_70/     ← milestone checkpoints, every 10 rounds
  ├── actor/    aggregated policy after that round (HF format, safetensors)
  └── critic/   aggregated value model (needed to RESUME federated PPO; not needed for inference)
runs/webshop_hardness-std1_ppo_qwen1.5b_accel/    ← full run artifacts
  ├── config.yaml                 exact experiment config
  ├── federated_summary.json      per-round validation metrics (val_curve + client_curve)
  ├── plots/                      training-dynamics figures (success_rate / task_score)
  ├── driver.log.gz               complete federated-driver log
  ├── launch.sh, *.py, *.sh       orchestration scripts (launcher, janitor, archiver, uploader)
  └── archive_run1_*              an earlier partial run (rounds 0–9) kept for reproducibility reference

Results (unperturbed validation, 64 tasks, temperature 0.4)

The aggregated global model is evaluated every round on a shared, unperturbed WebShop validation service. success_rate = fraction of episodes with full task success; task_score = WebShop partial-credit goal-match score in [0, 1].

Training was still in progress when this README was written (target: 70 rounds); later milestones are pushed as they complete, and runs/.../federated_summary.json holds the complete curve. Numbers are single 64-task evals — expect a few points of round-to-round noise, amplified by the extreme hardness skew (std=1) of the per-round client draws.

Usage

Inference with the aggregated policy (any milestone):

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "canyuchen/fedagent-webshop-hardness-std1-ppo-qwen2.5-1.5b"
model = AutoModelForCausalLM.from_pretrained(repo, subfolder="round_30/actor",
                                             torch_dtype="auto", device_map="auto")
tok = AutoTokenizer.from_pretrained(repo, subfolder="round_30/actor")

The model is a WebShop agent: it expects the FedAgent/verl-agent WebShop observation format (multi-turn, windowed history of 2) and emits search[...] / click[...] actions. To evaluate or continue training, use the FedAgent runner:

# evaluate / resume inside the FedAgent repo (see its docs/running.md)
python -m fedagent.fed.run_fed \
  --config runs/webshop_hardness-std1_ppo_qwen1.5b_accel/config.yaml \
  --model-path <local download of round_X/actor>

round_X/critic is the matching aggregated value model: resuming federated PPO from a milestone requires both (actor + critic); plain inference or SFT-style reuse needs only the actor.

Provenance & notes

• —Code: <https://github.com/sunblaze-ucb/FedAgent> (Apache-2.0), which builds on verl 0.8 and the verl-agent WebShop integration.
• —This run used the shipped accelerated config fedagent/config/paper_accelerated/task_heterogeneity/ppo/webshop/fed_webshop_ppo_total-100_cl-per-rd-2_rd-70_ep-per-cl-3_min-goals-per-cl-100_p-hardness_success_std-1.yaml with two operational (science-neutral) tweaks: WebShop service ports moved out of the ephemeral range (21000/21200) and vLLM gpu_memory_utilization 0.5 → 0.45.
• —Base model license: Qwen2.5 is Apache-2.0; this fine-tune inherits Apache-2.0.

Citation

If you use these checkpoints, please cite FedAgent:

@article{fedagent2026,
  title   = {Is Decentralized LLM Agent RL Robust to Heterogeneity? An Asymmetric Tale},
  author  = {Chen, Canyu and Zhu, Kangyu and Chen, Zhaorun and Zhou, Zhanhui and Diao, Shizhe and Lu, Yiping and Li, Tian and Li, Manling and Song, Dawn},
  journal = {arXiv preprint},
  year    = {2026}
}