Datasetpublic

wckwan/WebShop-Olmo3-7B-Adaptive-Pivot-Fenced-eval-newparser

WebShop OLMo-3-7B-Instruct pivot FENCED: held-out evaluation, NEW action parser Held-out WebShop evaluation rows for wckwan/WebShop-Olmo3-7B-Adaptive-Pivot-Fenced, steps gs140 through gs200. NOT comparable to the gs20–gs120 rows in wckwan/WebShop-Olmo3-7B-Adaptive-Pivot-Fenced-eval. Those ran on the frozen OLD-parser surface 7173787ad6e171c6, which executes the first <action> pair anywhere in the response, including actions rehearsed and rejected inside the reasoning. These… See the full description on the dataset page: https://huggingface.co/datasets/wckwan/WebShop-Olmo3-7B-Adaptive-Pivot-Fenced-eval-newparser.

sourceHugging Faceapache-2.0updated 7d agoView on Hugging Face

0likes370downloads

Dataset Card

WebShop OLMo-3-7B-Instruct pivot FENCED: held-out evaluation, NEW action parser

Held-out WebShop evaluation rows for `wckwan/WebShop-Olmo3-7B-Adaptive-Pivot-Fenced`, steps gs140 through gs200.

NOT comparable to the gs20–gs120 rows in [`wckwan/WebShop-Olmo3-7B-Adaptive-Pivot-Fenced-eval`](https://huggingface.co/datasets/wckwan/WebShop-Olmo3-7B-Adaptive-Pivot-Fenced-eval). Those ran on the frozen OLD-parser surface 7173787ad6e171c6, which executes the first <action> pair anywhere in the response, including actions rehearsed and rejected inside the reasoning. These rows use the NEW parser, and config_fingerprint does not distinguish the two. The old parser corrupts fenced decisions far more than terse ones, and increasingly with training step, so splicing the two series would create a step change that is pure parser.

Action-parser surface


parser commit	`451b4969` (projections: read the action after the final `</think>`)
verl-agent tree	`8abb4e2d`
`agent_system/environments/env_package/webshop/projection.py` sha256 prefix	`4d6ccb1d8982dec3`
rule	the action is read from the `<action>` pair after the FINAL `</think>`

Every result.json carries the same statement as eval_surface_note. The commit, parser-file hash and rule above are the canonical identification of this surface; no opaque surface id is assigned.

Layout

global_step_N/result.json         # metrics, diagnostics, goal set, config, fingerprint, runtime, eval_surface_note
global_step_N/trajectories.jsonl  # per-episode trajectories
global_step_N/eval_seed0.log      # eval log
MANIFEST.sha256                   # sha256sum -c compatible

Protocol

standalone_eval preset webshop_olmo3 (Instruct): 500 goals in 4 shards of 125 (goalindices sha256 `3c6e0c423c7234e8…`, shards `0bc99e5066361073…`), 15 steps, prompt/response 4096/2048, history 2, temperature 0.4, topp 1.0, topk -1, `historyactioncharcap None. Runtime: vLLM 0.11.0, data parallel 2 (TP=1), engine seed 0. Every row passes scripts/checkresultwebshop.py`.

	fingerprint
native	`6448dfad6bbdf8435a3bbd73b9822b19de99b45419d8921622329869f1065c38`
legacy-equivalent (drop the `None` `history_action_char_cap` key, commit `a668c6b6`)	`060ad196f1509810037101e0b3d3c3bed1cac46ee0ac26a4e9190634367a8070`

A matching fingerprint here does not imply a matching parser surface; check eval_surface_note / eval_contrast_note.