IDEA-AI4S/MoleculeQA

Dataset Card for MoleculeQA

<!-- Provide a quick summary of the dataset. -->

Dataset Details

Dataset Description

<!-- Provide a longer summary of what this dataset is. --> MoleculeQA: A Dataset to Evaluate Factual Accuracy in Molecular Comprehension (EMNLP 2024)

• —Curated by: IDEA-XL
• —Language(s) (NLP): en
• —License: mit

Dataset Sources

<!-- Provide the basic links for the dataset. -->

• —Repository: https://github.com/IDEA-XL/MoleculeQA
• —Paper [optional]: https://arxiv.org/abs/2403.08192

Dataset Structure

<!-- This section provides a description of the dataset fields, and additional information about the dataset structure such as criteria used to create the splits, relationships between data points, etc. -->

- JSON
  - All
    - train.json # 49,993
    - valid.json # 5,795
    - test.json # 5,786
- TXT
  - All
    - train.txt
    - valid.txt
    - test.txt
  - Property
  - Source
  - Structure
  - Usage

Dataset Creation

Curation Rationale

<!-- Motivation for the creation of this dataset. --> Large language models are playing an increasingly significant role in molecular research, yet existing models often generate erroneous information. Traditional evaluations fail to assess a model’s factual correctness. To rectify this absence, we present MoleculeQA1, a novel question answering (QA) dataset which possesses 62K QA pairs over 23K molecules. Each QA pair, composed of a manual question, a positive option and three negative options, has consistent semantics with a molecular description from authoritative corpus. MoleculeQA is not only the first benchmark to evaluate molecular factual correctness but also the largest molecular QA dataset. A comprehensive evaluation on MoleculeQA for existing molecular LLMs exposes their deficiencies in specific aspects and pinpoints crucial factors for molecular modeling. Furthermore, we employ MoleculeQA in reinforcement learning to mitigate model hallucinations, thereby enhancing the factual correctness of generated information.

Source Data

<!-- This section describes the source data (e.g. news text and headlines, social media posts, translated sentences, ...). -->

Data Collection and Processing

<!-- This section describes the data collection and processing process such as data selection criteria, filtering and normalization methods, tools and libraries used, etc. -->

Citation

BibTeX:

@inproceedings{lu-etal-2024-moleculeqa,
    title = "{M}olecule{QA}: A Dataset to Evaluate Factual Accuracy in Molecular Comprehension",
    author = "Lu, Xingyu  and
      Cao, He  and
      Liu, Zijing  and
      Bai, Shengyuan  and
      Chen, Leqing  and
      Yao, Yuan  and
      Zheng, Hai-Tao  and
      Li, Yu",
    editor = "Al-Onaizan, Yaser  and
      Bansal, Mohit  and
      Chen, Yun-Nung",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2024",
    month = nov,
    year = "2024",
    address = "Miami, Florida, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.findings-emnlp.216",
    pages = "3769--3789",
    abstract = "Large language models are playing an increasingly significant role in molecular research, yet existing models often generate erroneous information. Traditional evaluations fail to assess a model{'}s factual correctness. To rectify this absence, we present MoleculeQA, a novel question answering (QA) dataset which possesses 62K QA pairs over 23K molecules. Each QA pair, composed of a manual question, a positive option and three negative options, has consistent semantics with a molecular description from authoritative corpus. MoleculeQA is not only the first benchmark to evaluate molecular factual correctness but also the largest molecular QA dataset. A comprehensive evaluation on MoleculeQA for existing molecular LLMs exposes their deficiencies in specific aspects and pinpoints crucial factors for molecular modeling. Furthermore, we employ MoleculeQA in reinforcement learning to mitigate model hallucinations, thereby enhancing the factual correctness of generated information.",
}

Dataset Card Authors

He CAO (CiaoHe)