violetxi/ch-pilot-rollouts-qwen3.5-9b

C&H Pilot Rollouts — Qwen/Qwen3.5-9B

20 agentic exploration rollouts over the Calderwood & Harkness (C&H) synthetic law-firm corpus (the open-sourced world from harvey-labs tasks/firm-knowledge/, MIT), generated by Qwen/Qwen3.5-9B served with vLLM. Part of an actor-selection pilot for a world-internalization research project: the goal is to mine agent trajectories into verified fact stores and rewritten likelihood-training targets.

Companion dataset (same seeds/tasks, different actor): the other ch-pilot-rollouts-* repo under this namespace.

What's here

• —`rollouts.jsonl` (config rollouts) — one row per rollout: full message history (system / user / assistant / tool roles) plus metadata (mode, status, turn/tool-call/error counts, token usage, seed_ref).
• —`facts.jsonl` (config facts) — atomic facts mined from these rollouts by a Qwen3.5-35B-A3B extractor pass, each with evidence_path + evidence_quote and mechanical verification flags (path_exists, quote_found = normalized quote is a substring of the cited mirror file).
• —`manifest.json` — per-rollout run manifest (seed 0, wall time, token counts).

Rollout modes

• —B1 (12 rollouts) — matter surveys: "survey matter X: client, parties, timeline, events, status, key documents", with file-path citations required for every claim.
• —B2 (8 rollouts) — self-QA: given a seed document excerpt, pose 3 nontrivial questions (single-doc / cross-doc / cross-matter) and answer them with citations.

The environment is a markdown mirror of 20 of the 266 C&H matters (~13M tokens; docx/xlsx/ eml/pptx → .md). Tools are text-protocol glob/grep/read calls emitted as <tool>{"name": ..., "args": ...}</tool> blocks, answers end in <final>...</final>. Sampling: temperature 0.7, max 40 turns, max-model-len 65536, seed-matched across actors.

Pilot metrics (this actor)

Provenance & license

All document content is synthetic (no real firm, clients, or people), derived from Harvey AI's MIT-licensed harvey-labs release (commit 7be41d5). These rollouts and facts are released under MIT as well. The 250 firm-knowledge eval tasks are not used or contained here — they remain a held-out evaluation set.