violetxi/wmrl-v4-scale70-3m-agentic-eval-20t-think

scale70-3m Agentic evaluation, 20 turns, thinking enabled

1,000 saved evaluation attempts: 250 C&H firm-knowledge tasks × four samples. The base model is Qwen/Qwen3.5-9B at revision c202236235762e1c871ad0ccb60c8ee5ba337b9a, trained for two epochs on a 70% notes / 30% recall mixture. This dataset has 2,997,925 total loss-bearing training-data tokens (2,098,505 notes + 899,420 recall). The token count describes the dataset before its two training exposures. Run: scale70-3m-20t-think; evaluation job: 115163.

Mean stored rubric score: 7.91%. Status counts: {"done": 814, "max_turns": 37, "protocol_failure": 149}. All errors and unanswered attempts remain in the export. Scores are the original rounded per-attempt rubric fractions, not a binary success rate. A non-done attempt can have a nonzero score if the grader used its unwrapped answer. The 10M / 20-turn evaluation includes one context-limit attempt retained as zero.

Format

The first 13 columns and their Arrow types match the published base 5-turn and base 20-turn exports: task_id, sample, status, score, task_mean_score, task_done_rate, n_turns, n_tool_calls, task, final, final_unwrapped, transcript, messages. messages preserves every original system/user/assistant message and its exact content, including saved thinking. transcript uses the same readable rendering as the base exports. Row order follows the original JSONL; task_id and sample are the stable attempt key. No failed rows are dropped or resampled.

Additional full_trajectories and action_only_trajectories contain chronological lists of objects with exactly role and content (user/assistant only). The full view preserves all non-system messages. In the action-only view, user messages and observations remain exact; assistant content is JSON containing parsed commands and a task_complete boolean. A valid final block with no tool call sets task_complete; it does not mean the answer was correct. Malformed calls have no parsed commands; their original text remains in messages. Unsupported but parseable tool requests remain commands. SYSTEM is retained in messages and transcript, without inventing a user turn for it.

Structured metadata columns identify condition, run, evaluation job, base model and revision, note/recall/total loss tokens, epochs, and turn budget. As in the base exports, the HF split is named train for compatibility; these are held-out evaluation attempts, not training examples.

Judge outputs

judge_score is an explicit alias of the original per-attempt score. graded_response is the exact answer slice used by the grader (first 6,000 characters of the final answer or unwrapped fallback); judge_criteria lists the rubric criteria in their original order. Judge model and revision are structured columns. error retains any saved rollout error message.

The original grader converted each judge reply to a boolean in memory and saved only aggregate scores. It did not persist the individual judge replies or per-criterion decisions. judge_responses is therefore null, with judge_response_status=not_saved_by_harness for graded answers and not_invoked_no_answer for empty answers. These missing responses have not been reconstructed or regenerated. The original judge_errors.json is also included; all eight completed evaluations recorded zero judge API errors.

Files and reproduction

• —data/train-00000-of-00001.parquet: the Dataset Viewer table.
• —raw/scale70-3m-20t-think.jsonl: byte-for-byte original rollout JSONL.
• —scores.json: original aggregate and per-task/per-sample scores.
• —completion.json: the evaluation completion receipt.
• —judge_errors.json: original judge API error record.
• —audit_inputs.tar.gz: frozen task JSONs, task list, protocol/grader source, and this condition's training metadata, with paths relative to the scaling root.
• —manifest.json: source and artifact hashes, schema, and validation results.

The exporter is recipe/internalization/reports/export_agentic_eval_hf.py in the research repository. It validates all rows after Parquet round-trip and checks the rendering against both base exports. The failure-analysis report and its input-restoration instructions live under recipe/internalization/reports/internalization-agentic-failures/.

Provenance

The environment is Harvey AI's synthetic legal benchmark, revision c2488cfa24fd01ee88016a121479a2f86b394bd4, licensed MIT. The frozen benchmark has 250 firm-knowledge tasks and a mirror of 9,288 documents. The agent tools are glob, grep, and read. Grading used zai-org/GLM-4-32B-0414, revision 077b5c2f5c43bd3239fd605a0600229e8facbd4a. No new inference or grading was performed for this export.