insightSynthesisData/thinking-design-v3-harness
thinking_design_v3 — does any multi-turn harness beat single-pass ideation? Arm outputs, harness turn traces, and judge verdicts for a matched-idea-budget comparison of single-pass ideation against two multi-turn harnesses, on Qwen3-4B-SI and Qwen3-8B-SI over a fixed 100-instance slice. Layout path what arms/arms_<model>_<arm>.jsonl 3-idea sets per instance. _goldlc = same candidates, gold compressed to the candidate length turns/twoturn_h{2,4}_{4b… See the full description on the dataset page: https://huggingface.co/datasets/insightSynthesisData/thinking-design-v3-harness.
thinkingdesignv3 — does any multi-turn harness beat single-pass ideation?
Arm outputs, harness turn traces, and judge verdicts for a matched-idea-budget comparison of single-pass ideation against two multi-turn harnesses, on Qwen3-4B-SI and Qwen3-8B-SI over a fixed 100-instance slice.
Layout
Arms
A0 single pass (draw 1) · A0' random 3 of 36 · A0" gte-diverse 3 of 36 · H1 pool→cluster→prune · H2 two-turn propose→select→write · H4 extract→recombine→select
A0" is the bar that matters: beating A0' only shows selection beats random selection.
Read this before using the arm-vs-arm files
h2h_* was produced by a renderer that showed the judge only the `inspiration` field — one of four, and the one the repo excludes from judging by construction. Those files are invalid for ranking arms and are kept only as evidence of the bug. Use h2hfix_*, the same comparisons re-judged after the fix.
The vs-gold files (gold_*, goldlc_*, sjgold_*) were never affected — run_all_evals renders through _candidate_idea_text(si=True), i.e. SIIdea.card_text().
Impact scale
1.0 both A/B orderings agreed the idea beat gold · 0.5 the orderings disagreed (position instability, not a genuine tie) · 0.0 lost.
Viewer
python thinking_design_v3/view_arms.py --source hf --hf-repo insightSynthesisData/thinking-design-v3-harnessCode: thinking_design_v3/ in the insight_synthesis repo.