sentence-transformers/quora-duplicates

Dataset Card for Quora Duplicate Questions

This dataset contains the Quora Question Pairs dataset in four formats that are easily used with Sentence Transformers to train embedding models. The data was originally created by Quora for this Kaggle Competition.

Dataset Subsets

pair-class subset

• —Columns: "sentence1", "sentence2", "label"
• —Column types: str, str, class with {"0": "different", "1": "duplicate"}
• —Examples:

    {
      'sentence1': 'What is the step by step guide to invest in share market in india?',
      'sentence2': 'What is the step by step guide to invest in share market?',
      'label': 0,
    }

• —Collection strategy: A direct copy of Quora, but with more conveniently parsable columns.
• —Deduplified: No

pair subset

• —Columns: "anchor", "positive"
• —Column types: str, str
• —Examples:

    {
      'anchor': 'Astrology: I am a Capricorn Sun Cap moon and cap rising...what does that say about me?',
      'positive': "I'm a triple Capricorn (Sun, Moon and ascendant in Capricorn) What does this say about me?",
    }

• —Collection strategy: Filtering away the "different" options from the pair-class subset, removing the label column, and renaming the columns.
• —Deduplified: No

triplet-all subset

• —Columns: "anchor", "positive", "negative"
• —Column types: str, str, str
• —Examples:

    {
      'anchor': 'Why in India do we not have one on one political debate as in USA?',
      'positive': 'Why cant we have a public debate between politicians in India like the one in US?',
      'negative': 'Can people on Quora stop India Pakistan debate? We are sick and tired seeing this everyday in bulk?',
    }

• —Collection strategy: Taken from embedding-training-data, which states: "Duplicate question pairs from Quora with additional hard negatives (mined & denoised by cross-encoder)". Then, take all possible triplet pairs.
• —Deduplified: No

triplet subset

• —Columns: "anchor", "positive", "negative"
• —Column types: str, str, str
• —Examples:

    {
      'anchor': 'Why in India do we not have one on one political debate as in USA?,
      'positive': 'Why cant we have a public debate between politicians in India like the one in US?',
      'negative': 'Can people on Quora stop India Pakistan debate? We are sick and tired seeing this everyday in bulk?',
    }

• —Collection strategy: Taken from embedding-training-data, which states: "Duplicate question pairs from Quora with additional hard negatives (mined & denoised by cross-encoder)". Then, take the anchor, positive and the first negative of each sample.
• —Deduplified: No