Reza8848/AAAR-1.0

<!-- <img src="./figures/robotexpert2.png" width="45" height="45"> -->

<h1 align="center"> AAAR-1.0 Benchmark </h1>

<!-- <p align="center"> --> <!-- <a href="https://renzelou.github.io/AAAR-1.0/"><img src="https://img.shields.io/badge/Website-red" alt="website" /></a> <a href="https://arxiv.org/abs/2410.22394"><img src="https://img.shields.io/badge/Paper-orange" alt="paper" /></a> <a href="https://github.com/RenzeLou/AAAR-1.0/blob/main/LICENSE"><img src="https://img.shields.io/badge/LICENSE-MIT-blue" alt="license" /></a> --> <!-- <a href="https://github.com/RenzeLou/Muffin"><img src="https://img.shields.io/badge/Python-3.8-green" alt="python" /></a> --> <!-- </p> -->

📄 Paper: https://huggingface.co/papers/2410.22394

🌐 Website: https://renzelou.github.io/AAAR-1.0/

🤗 This repository contains the [AAAR-1.0](https://arxiv.org/abs/2410.22394) benchmark dataset.

🚨 <font color="#df1a7d">Please DO NOT use the data for training!</font>

<p align="center" width="100%"> <a ><img src="./figures/4.png" alt="paradigm" style="width: 100%; min-width: 300px; display: block; margin: auto;"></a> </p>

Get Performance on AAAR-1.0

<!-- <img src="./figures/GitHub-Mark.png" width="20" height="20"> --> Please refer to our code repository for detailed instructions on running various LLMs on the AAAR-1.0 benchmark, and report the performances.

Data Details

1. Equation Inference 🌟:

<p align="center" width="100%"> <a ><img src="./figures/sta_eq.png" alt="paradigm" style="width: 100%; min-width: 300px; display: block; margin: auto;"></a> </p>

Equation_Inference/equation.1049.json constains all the 1,049 samples for the equation inference task. Each instance has following fields:

"context_before": Input, the paper context before the target equation.
"context_after": Input, the paper context after the target equation.
"options": Input, four candidate equations (1 correct and 3 wrong).
"answer": Output, the correct equation.

2. Experiment Design 🧪:

<p align="center" width="70%"> <a ><img src="./figures/sta_exp.png" alt="paradigm" style="width: 70%; min-width: 300px; display: block; margin: auto;"></a> </p>

Experiment_Design contains all 100 papers for the experiment design task. Each paper data is stored under a subfolder named by the paper ID.

Each paper has following data sources:

- 1902.00751 (paper ID)
| - data_text.json: all the paper context (input) and the target experiment list&explanation (output).
| - images: a folder containing all the source images of this paper (extracted from arXiv source package).
| - *_source.tar.gz: the arXiv source package of this paper.

3. Paper Weakness 🔍:

<p align="center" width="70%"> <a ><img src="./figures/sta_weak.png" alt="paradigm" style="width: 70%; min-width: 300px; display: block; margin: auto;"></a> </p>

Paper_Weakness/ICLR_2023 contains all the 993 papers for the paper weakness task. Each paper data is stored under a subfolder named by the paper ID.

Each paper has following data sources:

- 3ZPESALKXO (paper ID)
| - data_text.json: all the paper context (input) and the target weaknesses (output).
| - images: a folder containing all the figure/table images of this paper (extracted from the paper PDF).

4. Review Critique ✍️:

We put the review critique task data in this GitHub repository. Please refer to our paper for more details.

🥳 Citation

Please kindly cite our paper if you use our data or results in your research:

@article{Lou2024AAAR,
  title={{AAAR-1.0}: Assessing AI's Potential to Assist Research},
  author={Renze Lou and Hanzi Xu and Sijia Wang and Jiangshu Du and Ryo Kamoi and Xiaoxin Lu and Jian Xie and Yuxuan Sun and Yusen Zhang and Jihyun Janice Ahn and Hongchao Fang and Zhuoyang Zou and Wenchao Ma and Xi Li and Kai Zhang and Congying Xia and Lifu Huang and Wenpeng Yin},
  journal={arXiv preprint arXiv:2410.22394},
  year={2024}
}

<!-- ```bibtex @inproceedings{du2024llms, title={Llms assist nlp researchers: Critique paper (meta-) reviewing}, author={Du, Jiangshu and Wang, Yibo and Zhao, Wenting and Deng, Zhongfen and Liu, Shuaiqi and Lou, Renze and Zou, Henry Peng and Venkit, Pranav Narayanan and Zhang, Nan and Srinath, Mukund and others}, journal={Proceedings of {EMNLP} }, year={2024} }