Datasetpublic

violetxi/wmrl-v4-base9b-agentic-eval-20t-think

Base-model agentic eval transcripts, 20-turn budget (WM-RL v4) 1,000 complete agentic-evaluation transcripts of the untrained base model Qwen/Qwen3.5-9B (revision c202236235762e1c871ad0ccb60c8ee5ba337b9a), thinking enabled, on the 250 held-out firm-knowledge tasks of the WM-RL v4 study, at a 20-turn tool budget. This is the baseline every trained condition in the study is compared against; the transcripts are the raw rollouts, saved before grading. 250 tasks x 4 samples = 1,000… See the full description on the dataset page: https://huggingface.co/datasets/violetxi/wmrl-v4-base9b-agentic-eval-20t-think.

sourceHugging Facemitupdated 10d agoView on Hugging Face

0likes43downloads

Dataset Card

Base-model agentic eval transcripts, 20-turn budget (WM-RL v4)

1,000 complete agentic-evaluation transcripts of the untrained base model Qwen/Qwen3.5-9B (revision c202236235762e1c871ad0ccb60c8ee5ba337b9a), thinking enabled, on the 250 held-out firm-knowledge tasks of the WM-RL v4 study, at a 20-turn tool budget. This is the baseline every trained condition in the study is compared against; the transcripts are the raw rollouts, saved before grading.

—250 tasks x 4 samples = 1,000 rollouts (sample 0-3), temperature as in the frozen benchmark harness.
—Mean rubric score (mean@4 over tasks, x100): 14.88 - answered rows 948/1000.
—Mean tool calls per rollout 15.46; mean assistant turns 16.54.
—Terminal status: done 948, max_turns 34, protocol_failure 16, api_error 2.

The companion file at the other budget is violetxi/wmrl-v4-base9b-agentic-eval-5t-think.

Provenance

The underlying world is [Harvey AI's synthetic legal benchmark](https://github.com/harveyai/harvey-labs) (MIT), a 9,288-document / 266-matter law-firm corpus, revision c2488cfa24fd01ee88016a121479a2f86b394bd4; its 250 firm-knowledge tasks are the user messages here. Its CONTRIBUTING.md requires synthetic people, companies, law firms, funds, products, addresses and matter facts and forbids real confidential client material. No real client data is present. The agent had three tools: glob, grep and read over the corpus mirror; the system prompt is the frozen evaluation prompt (first message of every row). Grading (not included) was done per rubric criterion by zai-org/GLM-4-32B-0414, revision 077b5c2f5c43bd3239fd605a0600229e8facbd4a; the per-task mean over the four samples is provided as task_mean_score and each rollout's own score as score. Rubric criteria live with the benchmark, not here.

Schema

column	type
`task_id`	string	benchmark task id, `001`-`250`
`sample`	int32	0-3, the four independent attempts
`status`	string	`done` (a `<final>` answer was produced), `max_turns` (budget exhausted), `protocol_failure` (two turns without a valid tool or final block), `api_error`
`n_turns`	int32	assistant turns in the rollout
`n_tool_calls`	int32	tool calls the harness executed
`score`	float64	this rollout's rubric score, share of the task's criteria met (0-1); 0 for unanswered rollouts
`task_mean_score`	float64	mean of `score` over the task's four samples; identical across the task's four rows
`task_done_rate`	float64	share of the task's four samples with status `done`
`task`	string	the user instruction, verbatim from the benchmark
`final`	string	the text inside the final `<final>...</final>` block, empty if none
`final_unwrapped`	string	for rows without a `<final>` block, the last assistant text the grader fell back to; otherwise empty
`transcript`	string	the whole rollout rendered top-to-bottom for reading: system prompt, task, then each assistant turn (thinking + call) and each tool result in order (read this one)
`messages`	list of {role, content}	the full transcript: `system`, `user` (task), then alternating `assistant` / `user` (tool results wrapped as `<tool_result>`) turns; assistant turns keep their `<think>` reasoning

Files

data/train-00000-of-00001.parquet is what the dataset viewer shows. raw/base9b-full-think.jsonl is the untouched original jsonl from the run directory (one rollout per line, keys task_id, sample, status, n_tool_calls, n_turns, final, final_unwrapped, messages).

Source

bench-full/base9b-full-think.jsonl (sha256 e898e70bd1ce1889de20cf571ed4cbc64c022f24c1c9d41686f2f406ef7f5f82) and bench-full/_scores-base9b-full-think.json from the WM-RL v4 run directory on TACC Vista, exported 2026-09-14.