Datasetpublic

yjlee36/knowchat-multi-turn-dialogues

KnowChat: Multi-Turn Human-LLM Dialogues on Knowledge Tasks KnowChat is a dataset of 705 multi-turn human-LLM conversations collected to validate the KnowSim user simulation framework. It pairs each conversation with pre/post knowledge assessments, self-reported survey ratings, and participant background information, enabling research on information calibration -- how well LLM assistants tailor responses to users with different knowledge levels. Dataset Summary… See the full description on the dataset page: https://huggingface.co/datasets/yjlee36/knowchat-multi-turn-dialogues.

sourceHugging Facecc-by-nc-4.0updated 1mo agoView on Hugging Face

3likes445downloads

Dataset Card

KnowChat: Multi-Turn Human-LLM Dialogues on Knowledge Tasks

KnowChat is a dataset of 705 multi-turn human-LLM conversations collected to validate the KnowSim user simulation framework. It pairs each conversation with pre/post knowledge assessments, self-reported survey ratings, and participant background information, enabling research on information calibration -- how well LLM assistants tailor responses to users with different knowledge levels.

Dataset Summary

	Math (Strategy)	Math (Model)	ExpertQA (Strategy)	ExpertQA (Model)
Conversations	201	171	150	183
Participants	67	57	50	61
Conditions	3 strategies	3 models	3 strategies	3 models
Topics	5 math topics	5 math topics	4 expert domains	4 expert domains

Total: 705 conversations from 235 participants.

Conditions

Tasks

—MathQA: Competition math problems (AMC/AIME level 4-5)
—ExpertQA: Domain expert questions across Architecture, Education, Psychology, and Visual Arts

Strategy Arm (conditions 0-2)

Same LLM (GPT-4.1), different strategies:

—0 Socratic · 1 Adaptive · 2 Comprehensive

Model Arm (conditions 0-2)

Same strategy (Adaptive), different LLMs:

—0 GPT-5.4 · 1 Gemini 3 Pro · 2 Claude Opus 4.7

Data Files

Conversation Logs (JSONL)

Full multi-turn conversations exported as JSONL to avoid Excel's 32K character cell limit:

File	Description
`math_strategy_conversations.jsonl`	MathQA strategy arm (201 conversations)
`math_model_conversations.jsonl`	MathQA model arm (171 conversations)
`expertqa_strategy_conversations.jsonl`	ExpertQA strategy arm (150 conversations)
`expertqa_model_conversations.jsonl`	ExpertQA model arm (183 conversations)

JSONL schema:

json

{
  "timestamp": 1776903871977,
  "time": "2026-04-23T00:24:31.977Z",
  "prolific id": "P006",
  "topic": "counting_and_probability",
  "condition": 2,
  "session": 0,
  "answer": "0.411",
  "questions": 5,
  "messages": [
    {"role": "system", "content": "You are a skilled math tutor..."},
    {"role": "user", "content": "If it rains in the capital city..."},
    {"role": "assistant", "content": "Great question! Let's break this down..."}
  ]
}

Note: The questions field appears only in ExpertQA files and indexes into the questions sheet of the corresponding XLSX file. MathQA files omit this field.

Structured Data (XLSX)

Each XLSX file contains multiple sheets with participant-level data:

Sheet	Description	Available in
`summary`	Condition group sizes and prescreening score distribution	All
`background`	Participant demographics: topic, self-reported ability, courses taken	All
`prescreening`	Domain prescreening test scores (Q1-Q10)	All
`pretest`	Pre-conversation knowledge test (Q1-Q7) per session	Math only
`posttest`	Post-conversation knowledge test (Q1-Q7) per session	Math only
`main`	Conversation metadata with messages (may be truncated by Excel's 32K cell limit; use JSONL files for complete conversations)	All
`survey`	Self-reported ratings: Delivery Calibration, Cognitive Overload, Interaction Quality	All
`questions`	Question bank with domain labels	ExpertQA only

Survey Measures

Each conversation session includes 3 self-reported Likert-scale ratings:

—Delivery Calibration: How well the AI matched the participant's level
—Cognitive Overload: Whether the AI's responses were overwhelming
—Interaction Quality: Overall quality of the interaction

Privacy & Ethics

—All participant identifiers have been replaced with sequential anonymous IDs (P001, P002, ...).
—Identifiers are consistent within each study arm but not across arms.
—The study was conducted via Prolific with informed consent and IRB approval.
—Free-text responses (conversation messages, answers) are included as-is; they do not contain personally identifiable information.

Intended Use

This dataset is designed for:

—Validating user simulation frameworks against real human behavior
—Studying information calibration in LLM tutoring interactions
—Analyzing knowledge gain patterns across user expertise levels
—Benchmarking LLM assistants on pedagogical effectiveness

Citation

bibtex

@article{lee2026knowsim,
  title={KnowSim: Evaluating Information Calibration in LLM Assistants with User Simulators that Learn},
  author={Lee, Yoonjoo and Jin, Hyoungwook and Kim, Tae Soo and Zhang, Shaoyang and Laban, Philippe and Liao, Q. Vera},
  year={2026}
}