ulab-ai/AcademicEval
AcademicEval Benchmark Introduction We proposed AcademicEval, a live benchmark for evaluating LLMs over long-context generation tasks. AcademicEval adopts papers on arXiv to introduce several acadeic writing tasks with long-context inputs, i.e., Title, Abstract, Introduction, Related Work, wich covers a wide range of abstraction levels and require no manual labeling. Comparing to existing long-context LLM benchmarks, our Comparing to existing long-context LLM benchmarks… See the full description on the dataset page: https://huggingface.co/datasets/ulab-ai/AcademicEval.
AcademicEval Benchmark Introduction
<!-- <img src="./model.png" width="75%" height="75%" /> --> <div align=center> <img src="./model.png" width = 75% height="75%"/> </div>
We proposed <b><i>AcademicEval</i></b>, a live benchmark for evaluating LLMs over long-context generation tasks. <b><i>AcademicEval</i></b> adopts papers on arXiv to introduce several acadeic writing tasks with long-context inputs, <i>i.e.</i>, <b><i>Title, Abstract, Introduction, Related Work</i></b>, wich covers a wide range of abstraction levels and require no manual labeling.
Comparing to existing long-context LLM benchmarks, our Comparing to existing long-context LLM benchmarks, our AcademicEval offers flexible length, automatic annotation, hierarchical abstraction, few-shot demonstrations, and live updates without data leakage risks.
🌟Note🌟: currently, for the ease of downloading, we only uploaded the test set of AcademicEval (The rest of AcademicEval, i.e., train and val set, can be accessed via [AcademicEval Full](https://huggingface.co/datasets/ulab-ai/AcademicEval_Full)). The data viewer above shows the preview data information of title-10K, abs-9K, and intro-8K. For the complete test set data, please check "Files and versions" in this page.
<table class="comparison-table"> <thead> <tr> <th>Benchmark</th> <th>Avg Len</th> <th>Automatic Annotation</th> <th>Hierarchical Abstraction</th> <th>Few-shot Demonstrations</th> <th>Live Update</th> </tr> </thead> <tbody> <tr> <td>ZeroSCROLLS (Shaham et al., 2023)</td> <td>~10K</td> <td><span style="color: green;">✓</td> <td><span style="color: red;">✘</td> <td><span style="color: red;">✘</td> <td><span style="color: red;">✘</td> </tr> <tr> <td>L-Eval (An et al., 2023)</td> <td>~8K</td> <td><span style="color: red;">✘</td> <td><span style="color: red;">✘</td> <td><span style="color: red;">✘</td> <td><span style="color: red;">✘</td> </tr> <tr> <td>BAMBOO (Dong et al., 2023)</td> <td>~16K</td> <td><span style="color: red;">✘</td> <td><span style="color: red;">✘</td> <td><span style="color: red;">✘</td> <td><span style="color: red;">✘</td> </tr> <tr> <td>LongBench (Bai et al., 2023)</td> <td>~8K</td> <td><span style="color: red;">✘</td> <td><span style="color: red;">✘</td> <td><span style="color: green;">✓</td> <td><span style="color: red;">✘</td> </tr> <tr> <td>LooGLE (Li et al., 2023)</td> <td>~20K</td> <td><span style="color: red;">✘</td> <td><span style="color: red;">✘</td> <td><span style="color: red;">✘</td> <td><span style="color: red;">✘</td> </tr> <tr> <td>∞Bench (Zhang et al., 2024)</td> <td>~200K</td> <td><span style="color: red;">✘</td> <td><span style="color: red;">✘</td> <td><span style="color: red;">✘</td> <td><span style="color: red;">✘</td> </tr> <tr> <td><strong>AcademicEval (ours)</strong></td> <td><strong>Flexible</strong></td> <td><span style="color: green;">✓</td> <td><span style="color: green;">✓</td> <td><span style="color: green;">✓</td> <td><span style="color: green;">✓</td> </tr> </tbody> </table> <style> .comparison-table td:nth-child(3) span { color: green; } .comparison-table td:nth-child(4) span { color: red; } .comparison-table td { text-align: center; } </style>
Dataset Structure
Data Settings
- *Title Writing*
- title_10K
- title_30K
- title_31K_G
- *Abstract Writing*
- abs_9K
- abs_28K
- abs_29K_G
- *Introduction Writing*
- intro_8K
- intro_28K
- intro_28K_G
- *Related Work Writing*
- related_34K
- related_53K
- related_53K_G
Main Data Fields
- url: the url of the original paper on arXiv
- title: the title of the paper
- abstract: the abstract of the paper
- authors: the authors of the paper
- published: the publication timestamp of the paper
- primary_cat: arXiv category
- gt: the ground truth of the corresponding task
- main_content: the main body of the paper (w/o the corresponding section content)
- additional_info: the few-shot demonstrations from randomly selected papers (the data fields of each demonstration are the same as above)
- additional_graph_info: the few-shot demonstrations with the co-author subgraph structure from co-author papers (the data fields of each demonstration are the same as above)