wangyueyiiiiiii/ContextDialog

Does Your Voice Assistant Remember? Analyzing Conversational Context Recall and Utilization in Voice Interaction Models

🎉 We are excited to announce that our paper has been accepted to the Findings of ACL 2025!

• —Demo Page: https://contextdialog.github.io
• —arXiv: https://arxiv.org/abs/2502.19759

[image]

ContextDialog is a comprehensive benchmark designed to evaluate a voice interaction model’s ability to engage in, retain, and leverage relevant information throughout multi-turn conversations, reflecting real-world scenarios where people often forget and revisit past exchanges. ContextDialog is constructed using MultiDialog, a spoken dialog corpus featuring conversations between two speakers, comprising approximately 340 hours of data with at least 10 turns per conversation from 12 speakers.

1. Statistics

2. Example Usage

There are 'user' and 'system' splits, each indicating QA pairs about past utterances made by the user and the model, respectively. Below is an example usage.

from datasets import load_dataset

ContextD = load_dataset("ContextDialog/ContextDialog")['test']

# see structure
print(ContextD)

# load audio sample on the fly
user_question_audio = ContextD[0]['question_audio']['array'] # audio for question about user's past utterance
user_question_text = ContextD[0]["question_text"]  # corresponding transcription

3. Dataset Structure

a. Data Samples

{
 'answer_audio': {
    'array': array([ 0.01547241,  0.01464844,  0.01397705, ..., -0.00204468, 0.00454712,  0.00747681]),
    'path': 't_001585ed-4211-458f-9564-2b905697b158_system_gen.wav',
    'sampling_rate': 44100
  },
  'answer_text': "You mentioned that you liked his character, although you couldn't remember the character's name.",
  'file_id': 't_001585ed-4211-458f-9564-2b905697b158',
  'position': 'front',
  'query': 'user',
  'question_audio': {
     'array': array([-0.02716064, -0.02801514, -0.02883911, ...,  0.00106812, 0.00402832,  0.00518799]),
     'path': 't_001585ed-4211-458f-9564-2b905697b158_user_gen.wav',
     'sampling_rate': 44100
  },
  'question_text': 'What did I mention about my opinion on that Robert Downey Jr. movie character?',
  'split': 'test_rare',
  'supporting_text': 'Wow! Wait a sec....I did see one of those Avengers movies. (Can\'t remember which one). But, I liked the Robert Downey Jr. character (can\'t remember his name) But, Simple Minds "Don\'t You (Forget About Me)" is a pretty awful song.'
}

b. Data Fields

• —file_id (string): The actual file ID, which allows access to the original multi-turn spoken dialog in MultiDialog.
• —position (string): Indicates whether the QA pair was created based on information from the front or back of the conversation.
• —query (string): Specifies whether the user's question pertains to a past user utterance or a past system utterance.
• —split (string): Indicates whether the QA pair is based on the testfreq or testrare split of MultiDialog.
• —question_audio (Audio feature): A dictionary containing the audio array, file path, and sampling rate of the question.
• —answer_audio (Audio feature): A dictionary containing the audio array, file path, and sampling rate of the answer.
• —question_text (string): The transcript of the user’s question.
• —answer_text (string): The transcript of the model’s answer.
• —supporting_text (string): The utterance from the original multi-turn dialog that serves as the basis for the answer.

4. References

We built ContextDialog using MultiDialog and referred to MultiDialog's README.md as a reference for our README.