anonymousfileupload/divdata

1# Build log — 2026-09-092 3Rebuild of all tables from the 1,296 source databases, adding `trajectories.parquet`,4`impressions.slot`, `actions.step`, and correcting the `step` column.5 6## Commands7 8```9python build_divdata.py       --out divdata/                         # 6 core tables, ~2h15 single process10python build_trajectories.py  --out divdata/ --workers 16 --patch-steps   # trajectories + step back-fill11```12 13Source: `<run>.db` (SQLite, OASIS schema) + `<run>.metadata.json` (agent → model map)14+ `<run>_chats.jsonl` (one row per activation: step, status, attempts with the model's15response and tool call) + `<run>_reasoning.jsonl` (thinking channel, where exposed).161,384 databases found; 88 skipped (no metadata sidecar, or an in-progress checkpoint).17 18## Output19 20| table | rows | size | step non-null |21|---|---:|---:|---:|22| runs | 1,296 | 34 KB | — |23| agents | 28,938 | 24 KB | — |24| posts | 89,171 | 23.8 MB | 90.2% |25| comments | 652,588 | 91.8 MB | 91.8% |26| impressions | 14,020,992 | 1.2 MB | 93.3% |27| actions | 741,219 | 6.1 MB | 91.7% |28| trajectories | 748,285 | 169.6 MB | 89.5% (of feed-fetch activations) |29 30Row counts of the six core tables are identical to the previous release; only `step`31values, and the new columns, changed.32 33## Trajectories build log (`_trajectories_build_log.json`)34 35```36runs 1296        no_chats 41 (no sidecar → timestamps and actions only, no step/decision)37agents 28847     agents_aligned 26786 (92.9%)   agents_multi 339 (see below)38activations 748285 = 741219 actions + 7066 passive (fetched the feed, no action landed)39status: ok 670505 · exhausted 2263 · exception 79 · null 75438 (unaligned agents)40with_tool_call 642083 · with_response_text 258949 · with_reasoning 28478841windows_multi 3247 (extra_in_window rows)42```43 44Validation:45- every action in `actions.parquet` is present in `trajectories` (741,219 = 741,219);46- every feed-fetch window holds at most one action in all but 16 runs;47- where `trajectories.step` and the action-count alignment of `posts`/`comments` both give48  a value, they agree on 624,776 / 624,820 rows (99.993%); all 44 disagreements are in49  5 of those 16 runs;50- `--patch-steps` back-filled step from the feed-fetch alignment where the action-count51  alignment had NULL: posts 15,060 → 8,701 NULL; comments 91,744 → 53,403; actions52  106,264 → 61,564. Agreement on rows both methods covered: 99.995 / 99.993 / 99.993%.53 54### The 16 runs with two actions in one feed-fetch window55 56In decreasing number of extra actions: `archive_20260516_1003/oasis_v9_hetero_late` (2,468 —57an early v9 run whose harness allowed multiple actions per activation), `sweep_olmo_mag_s42`58(98), `sweep_mag_olmo_s42` (97), `tetratic/tetra_s271_r4` (82), `tetratic/tetra_s271_r3` (81),59`attractor_rl/eval_runs_triadic/baseline/eval_baseline_s43` (72), `…_s42` (66),60`tetratic/tetra_s42_r5` (57), `sweep_glm47_glm47_qwe_s137` (51), `sweep_glm47_glm47_qwe_s42`61(36), `sweep_tri_glm_qwe_gem_s999` (36), `sweep_tri_glm_qwe_gem_s271` (32),62`tetratic/tetra_s42_r6` (31), `sweep_tri_gem_mag_glm_s999` (20), `sweep_tri_gem_mag_glm_s271`63(16), `sweep_tri_gem_qwe_glm_s42` (4). For an agent with any such window the refresh alignment is not64trustworthy, so its `step`, `status` and decision fields are NULL in `trajectories`65(339 agents); its actions are all present, extras flagged `extra_in_window=True`, and66`refreshed_at` / `acted_at` are exact. Step coverage in these runs is 0–58%. Prefer67`created_at` ordering there.68 69## Erratum in the previous release (published 2026-09-04)70 71`build_divdata.py` assigned step by iterating all of an agent's posts, then all of its72comments, while the alignment index assumed one time-ordered stream. Consequence, measured73by rebuilding and comparing on the same rows:74 75| table | rows compared | old step == new step |76|---|---:|---:|77| posts | 74,111 | **38.9%** |78| comments | 560,844 | **49.2%** |79| impressions | per viewer | 0% of viewers had the same step set — every value shifted by one activation (the step-0 intro post has no feed fetch, and was counted as one) |80 81Agents that only ever commented, and the step-0 intro posts, were correct; everything else82was scrambled. No other column was affected. `actions.step` was NULL throughout in the old83release. Nothing in the accompanying paper used `step`.84 85## Decision-record coverage by model (feed-fetch activations)86 87| model | activations | response_text | reasoning | tool_call | passive |88|---|---:|---:|---:|---:|---:|89| gpt-oss-20b | 247,489 | 0.4% | 4.8% | 94.4% | 0.3% |90| Magistral-Small-2509 | 149,666 | 86.2% | 91.7% | 91.3% | 1.1% |91| Qwen3-32B | 102,546 | 44.6% | 44.7% | 90.2% | 0.3% |92| GLM-4-32B-0414 | 54,430 | 0.0% | 0.0% | 95.7% | 0.6% |93| gemma-4-31B-it | 39,744 | 93.0% | 93.4% | 92.7% | 1.2% |94 95`tool_call` < 100% because unaligned agents carry no decision record. gpt-oss and GLM-496never emit visible deliberation.97