Datasetpublic

wckwan/Alfworld-Olmo3-7B-SEED-eval

ALFWorld OLMo-3-7B-Instruct SEED: held-out evaluation Held-out ALFWorld evaluation rows for wckwan/Alfworld-Olmo3-7B-SEED (merged_hf_actor_gs20 through gs260). Layout global_step_N/result.json # metrics, diagnostics, task list, config, fingerprint, runtime global_step_N/trajectories.jsonl # per-episode trajectories global_step_N/eval_seed0.log # eval log (not present for every row) MANIFEST.sha256 # sha256sum -c compatible… See the full description on the dataset page: https://huggingface.co/datasets/wckwan/Alfworld-Olmo3-7B-SEED-eval.

sourceHugging Faceapache-2.0updated 6d agoView on Hugging Face

0likes505downloads

Dataset Card

ALFWorld OLMo-3-7B-Instruct SEED: held-out evaluation

Held-out ALFWorld evaluation rows for `wckwan/Alfworld-Olmo3-7B-SEED` (merged_hf_actor_gs20 through gs260).

Layout

global_step_N/result.json         # metrics, diagnostics, task list, config, fingerprint, runtime
global_step_N/trajectories.jsonl  # per-episode trajectories
global_step_N/eval_seed0.log      # eval log (not present for every row)
MANIFEST.sha256                   # sha256sum -c compatible

Protocol

standalone_eval preset alfworld_olmo3, identical to the ALFWorld OLMo-3 Instruct baseline rows:

field	value
alfworldevaldataset	evalindistribution
env_name	alfworld/AlfredTWEnv
env_seed	0
valbatchsize	128 goals
max_steps	50
maxpromptlength / maxresponselength	2048 / 2048
history_length	2
temperature / topp / topk	0.4 / 1.0 / -1
applychattemplate_kwargs	{} (Instruct model, no thinking switch)
historyactionchar_cap	None (uncapped)

Runtime: vLLM 0.11.0, torch 2.8.0, transformers 4.57.1, engine seed 0, TP=1. Data parallel 2 for gs20-gs160 (evaluated at verl-agent commit 8abb4e2d); data parallel 1, single GPU, for gs180-gs260 (EIDF tree in 8abb4e2d..6ba9b713, 7-file eval surface byte-identical across that range, digest 599824c1c4b07828). Same 128-game set and fingerprint throughout, but per-episode sampling streams differ between the two ranges; config_fingerprint does not capture runtime. Each dp=1 row carries an eval_surface_note. Every row passes scripts/check_result_alfworld.py, which pins the canonical 128-episode gamefiles hash 12ead9df9760a8a8.

With 128 goals, the sampling standard deviation of a success rate is about 4.4 points. Treat differences under about 9 points as not separated.

Config fingerprint

	fingerprint
these rows (native)	`942c1e75645e9f07f6ecd91a66d3ddbf1b34a87097623862db0467e06bca7fcc`
baseline rows / legacy-equivalent	`c12b948ddb980df68520982077f8202cc260b1abd21f9194e06c4eae7012abc3`

The only difference between the two is the history_action_char_cap key, added in commit a668c6b6 and set to None here, which leaves behaviour unchanged. Removing it reproduces the baseline hash exactly. Aggregate against baselines with --allow-mixed.