violetxi/harvey-notes-v4

wm-rl notes v4 — two note banks from a recursive self-experience loop

Continuation update (2026-09-14): rounds 6–15 appended. The recursive bank now contains 308,580 notes / 138,723,516 training tokens. The original 108,081-row round-5 bank remains an exact prefix. Round-4/5 task lists were available for duplicate rejection, but their trajectory manifests were not published; the continuation therefore seeds round 6 from the published recall sessions and carries complete histories from round 6 onward.

Data from a study of scaling internalization from a model's own experience. A Qwen3.5-9B agent works inside a synthetic law firm's document-management system (the "C&H DMS": 9,288 markdown documents across 266 matters, commit 8a578c22) and writes notes about what it finds. The notes are then the only text a LoRA adapter is trained on, and the adapter is measured on a frozen 250-task open-book agentic benchmark and on ~7,900 closed-book probes.

Updated 2026-09-13: rounds 4 and 5 appended. The recursive bank grew from 74,976 notes / 32,276,156 tokens to 108,081 notes / 47,080,748 tokens. The bank is append-only and in generation order, so the first 74,976 notes are byte-identical to the previous snapshot (sha256 c3dac284…, verified) and every published n-1m … n-30m checkpoint remains a valid prefix of this file.

Rounds 4 and 5 were run at a fixed 10,000 tasks each — deliberately, so the per-round series is controlled (rounds 1–3 varied: 4,622 / 11,591 / 16,000). Yield came out near-identical across the two: 16,519 and 16,586 notes, 7.38M and 7.43M tokens, 1.65 and 1.66 notes per task.

Note what the controlled rounds expose: token yield stayed flat while coverage gain halved. The protocol requires the agent to write ≥1 grounded note per rollout, so note count has a floor independent of whether there is new ground to cover — which means note count is a poor saturation signal and coverage is the one to read. Pair coverage reached 61.4%, document coverage 96.1%.

The two banks

The two banks share their first 23,174 notes (9,235,631 tokens, the seed — also published standalone as seed_notes_accepted.jsonl) and nothing else. Both are in generation order; the training conditions are prefixes.

Each bank is published twice, same rows in the same order:

• —banks/<bank>.jsonl — the raw file, byte-identical to what the adapters were trained from (sha256 below).
• —banks/<bank>.parquet — the browsable copy the dataset viewer shows (the "Viewer" tab above). Columns reordered for reading — note_type, title, text, written_by, generation_round, generation_batch, evidence, … with training_text last — plus one added column, written_by = agent (written mid-task) or connective (post-round pass); prompts_version is null for agent-written notes. Nothing else differs.

Composition

notes_recursive.jsonl by generation_batch:

notes_inherited_only.jsonl: seed 23,174 + r0extra 89,640 (73,388 bundles proposed, 235,097 notes proposed, 38.1% accepted).

Note types in both banks: chronology, matter_profile, pitfall, document_map, practice_recipe.

How the training conditions slice them

n-1m ⊂ n-3m ⊂ n-10m ⊂ n-30m (nesting verified at build time).

Schema (both banks)

note_id, repository, commit, note_type, title, text, training_text, evidence (list of {path, start_line, end_line} the note cites), supporting_trajectory_ids, related_question_ids, generation_round, generation_batch, prompts_version.

• —Only training_text is trained on (title + repository/commit header + body). Documents are packed with one EOS separator each into 16,384-token rows.
• —Agent-written vs connective: notes written by the agent mid-task have no prompts_version key and their supporting_trajectory_ids are the round's own task ids (n1_…, n2_…, n3_…). Connective notes carry prompts_version (notes-prompts-5 for the seed, -8 for round 1, -9 after) and cite inherited-session ids (g3_…, d2_…, r4_…, …).

Gates

Every note passed 13 static checks before entering a bank (gates versions notes-gates-6 for the seed, -9 for round 1, -10 after): schema, note type, length (≤1,024 tokens, body ≥30), support, cited path was actually observed in the session, cited line range exists and was read, observed range, entity containment (every named entity/number/date in the note appears in the documents the session read), coverage-claim, firm-scope, near-duplicate (Jaccard ≥0.8), anti-scaffold, voice, and a fast lexical gate. Acceptance: seed 29.7%, round 1 47.1%, rounds 2–3 55.9% / 55.7%, r0extra 38.1%.

What the banks did (summary of the study's readout)

• —Closed-book knowledge rises ≈ +9 points per decade of note tokens along n-1m → n-30m (n = 4 points, r = +0.94).
• —On the open-book agentic benchmark, n-30m keeps protocol integrity (0.875 of rows answered, no-think) while r0-30m (0.233) and null-30m (0.012) collapse at the same step count — the model starts emitting note-shaped text instead of tool calls. What is written, not how many steps, decides whether the adapter survives.
• —Notes beat the source documents at equal tokens (raw-30m) on both evals.

Caveats

• —Notes are not scrubbed against the held-out benchmark. They come from the model's own reads, so some overlap is inherent: 638 of 1,369 held-out fact pairs are contained somewhere in the n-30m slice (633 in r0-30m). Conditions trained on notes are tagged DIRECTIONAL rather than EXACT in the scorecard.
• —The seed bank's grounding audit did not clear its bar (80.0% vs an 85% target, absence-type over-claims 17.3%). It is published as measured; the study is about what a model internalises from its own experience, and filtering to audit-clean would measure something else.
• —Round-1 tasks were regenerated twice before the round ran; both failed task sets are kept (round1_tasks_attempt*.jsonl, audits/) because the failures are informative.

Files kept from the earlier snapshot

seed_notes_accepted.jsonl (the shared 23,174-note seed), seed_notes_rejected.jsonl (54,979 seed rejections with verdict strings), round1_tasks_attempt{1,2}_*.jsonl, round1_taskgen_report_attempt{1,2}.md, audits/.

Integrity