MANGSEOK123/qwen3-4b-tau2-oel-retail-ep1

qwen3-4b-tau2-oel-retail-ep1

Qwen3-4B-Instruct-2507 after one epoch of Online Experiential Learning (OEL) on the tau2-bench retail domain.

What this is

OEL here is self-distillation, not RL — no reward signal is used. Teacher and student share weights; the only difference is that the teacher gets the task's memory appended to its system prompt. The student is fitted to the teacher's distribution with a token-level full KL loss, so that the behaviour survives without the memory in the prompt.

deploy       the student plays each task WITHOUT memory; every agent turn is pickled
consolidate  the same trajectories are re-scored with each task's memory injected
             (the teacher), and the memory-free student is fit to that distribution
merge        FSDP shards -> HuggingFace weights

Training

Evaluation

Evaluated with tau2-bench's own CLI (tau2 run), retail test split, 40 tasks × 3 trials, greedy agent, no memory in the prompt — the memory the model was trained against never reaches the test prompt.

This checkpoint does not improve over its base model. All three differences are under one standard error (SE ≈ 0.064–0.078 at 40 tasks), so they are indistinguishable from noise rather than a demonstrated regression — but there is no evidence of a gain either.

This is consistent with what the method's authors report: across roughly twenty runs over three domains, several memory sources, 1–2 epochs and learning rates from 1e-6 to 1e-5, no configuration beat its base model by more than about 1.5 SE. Putting the same memories in the prompt at test time did clearly help, so the memories carry signal; the distillation step recovers only part of it.

Published as a reproducible artifact of that negative result, not as an improved model.

Notes

Gradient norms ran 0.6–14.7 against a clip of 1.0, so the effective step size was set by the clip rather than by the learning rate. Only the final step is saved; its entropy here was 0.329, in the healthy range by the authors' measurements — yet that did not translate into better task performance, which suggests entropy at the last step is a weak proxy for checkpoint quality.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

m = AutoModelForCausalLM.from_pretrained("MANGSEOK123/qwen3-4b-tau2-oel-retail-ep1")
t = AutoTokenizer.from_pretrained("MANGSEOK123/qwen3-4b-tau2-oel-retail-ep1")

Serving for tau2 evaluation needs hermes tool calling:

vllm serve MANGSEOK123/qwen3-4b-tau2-oel-retail-ep1 \
  --enable-auto-tool-choice --tool-call-parser hermes