Alberto1231/thoughttrace_modified

ThoughtTrace Modified

A next-human-turn benchmark derived from `SCAI-JHU/ThoughtTrace` at pinned revision 0420f3d8499e477098aac7771fe9c066f2340fb3.

Non-negotiable target contract

Every scored target is copied from a source message whose type is exactly user, immediately following a source message whose type is exactly assistant. The assistant message is conditioning context, never the target:

Human: previous human message
Assistant: source LLM response
Human: <genuine human target>

The builder validates this transition against the pinned source for every row. Private reasons, reactions, post-task summaries, task expectations, and demographics are never included in the model prompt.

Splits and selection

• —train: 5 demonstrations from distinct participants
• —test: 965 scored targets from distinct participants
• —One row per participant across train and test
• —Every normalized target string is globally unique
• —Human-message date range: 2026-03-18T15:13:24.848000Z through 2026-04-14T12:26:00.404000Z
• —Both the source conversation and all three messages in the selected user-assistant-user transition must be on or after 2023-01-01T00:00:00Z
• —All 20 source assistant models remain eligible
• —Five demonstrations are selected for compactness and diversity across source model, gender, age, education, usage frequency, and surface response form
• —The full test split is intentionally a near-census, not an artificially equalized demographic sample
• —Selection never oversamples or duplicates people from small groups

The model receives only the immediately preceding human message and assistant response. Complete five-shot prompts fit a 2,048-token model context after reserving 256 tokens for generation with the pinned MARIN tokenizer.

Metadata and subgroup analysis

Self-reported gender, normalized age group, education, occupation, AI-use frequency, and AI purposes are retained as analysis metadata. They never enter the prompt. Invalid ages and unanswered fields become Missing; identities are never inferred from conversation text.

The natural full split is allowed to be unbalanced. For convenient matched-N ablations, immutable cohort_<attribute> flags identify deterministic equal-N subsets for sufficiently supported levels:

• —gender: 393 respondents per included level (Female, Male)
• —age_group: 43 respondents per included level (18-24, 25-34, 35-44, 45-54)
• —education: 97 respondents per included level (Graduate, High school, Undergraduate)
• —lm_frequency_use: 49 respondents per included level (A couple of times, Many times a day, Once a day, Once a week)
• —source_model: 30 respondents per included level (Claude Opus 4.6, Claude Sonnet 4.6, GPT-4o-mini, GPT-5.4, Gemini 3 Flash Preview, Gemini 3.1 Pro Preview, Gemma 4 26B A4B, Grok 4.1 Fast, Grok 4.20, Kimi K2.5, Llama 3.3 70B Instruct, MiMo-V2-Pro, MiniMax M2.7)

These are descriptive cohorts, not causal controls. Attributes, task content, and assigned source assistant model may remain correlated.