iwylin/imbue_dearman_expert_annotations

Dataset Overview

This dataset was collected as part of the project "**IMBUE: Improving Interpersonal Effectiveness through Simulation and Just-in-time Feedback with Human-Language Model Interaction**."

We are releasing this dataset with the hope that it provides valuable opportunities for researchers to develop and evaluate new LLM-based tools for interpersonal skill training across a range of fields, including natural language processing, conversational AI, and computational psychology.

For any questions or concerns, please reach out to ilin@cs.washington.edu

Data Collection Methods

Interpersonal Effectiveness Framework

The dataset leverages the DEAR MAN interpersonal effectiveness framework, a widely used framework from Dialectical Behavioral Therapy (DBT) that teaches conversational strategies and emotional regulation.

DEAR MAN stands for: Describe, Express, Assert, Reinforce, Staying Mindful, Appear Confident, and Negotiate.

Conversations on bespoke difficult situations between mTurk participants and simulated conversation partner

• —Each participant contributed three conversations, one from each category: family, social, and work.

• —Participants engaged in conversations with a large language model (LLM) trained to simulate a conversation partner within the participant's self-defined bespoke, difficult situations.

• —Participants were asked to have conversations that were at least 10 responses long or until the simulated partner agreed with them.

• —Before writing each message, participants chose one or more DEAR MAN strategies to guide their communication.

Screenshots of the conversation collection interface are provided in the paper (Appendix J).

Expert annotations on conversational and emotional skill use

The dataset includes annotations from six clinical experts trained in Dialectical Behavior Therapy (DBT) who actively use DEAR MAN in their practice.

Expert Selection Criteria: The researchers specifically recruited experts who are clinical psychology PhD, PostDoc, or practitioners, and only selected those who indicated they "sometimes" or "regularly" work with clients on DEAR MAN skills. This selection process ensured the annotators had practical experience applying the DEAR MAN framework in therapeutic settings.

Annotation Process: Experts annotated the conversations using an interactive tool. Each message was assessed for the following:

• —Skill Identification: Experts identified which DEAR MAN skills were present in each utterance.
• —Skill Use Rating: Each skill used in an utterance was rated as either "strong" or "weak". Mindfulness and confidence were rated as "yes" or "no."
• —Improvement Suggestions: Experts provided specific suggestions for improving "weak" utterances and offered rewritten versions demonstrating "strong" skill use.
• —Additionally, experts suggested skills for utterances where they believed a skill should have been applied but wasn't and provided rewritten utterances demonstrating their suggestions.

Screenshots of the annotation interface are provided in the paper (Appendix L).

Privacy and Ethical Considerations

The researchers obtained approval from the Institutional Review Board (IRB) at the University of Washington before conducting the study. All researchers involved in the study completed human subjects protection training and received IRB certification.

Informed Consent: Informed consent was obtained from all participants involved in both the data collection and user study phases. Participants were informed that:

• —They would be interacting with an AI-based model simulating a conversation partner.
• —The data they provided would be used for research purposes.
• —The model's responses might contain upsetting content due to the nature of the simulated difficult conversations.

Data Privacy: The study did not collect any Personally Identifiable Information (PII). Participants were asked to avoid including PII in the situations they described or the conversations they had with the model. The collected conversations and situation descriptions were manually filtered to remove any identifiable names or locations.

Crisis Resources: Although content filters were used to minimize potentially harmful outputs from the model, participants were provided with links to Crisis Text Line and the 988 Suicide and Crisis Lifeline as a precaution.

For further details on the methodology and findings, refer to the associated paper.

Dataset Schema

The following table provides a brief description of each column in the dataset: