SIRIS-Lab/erc-classification-dataset

1---2configs:3- config_name: default4  data_files:5  - split: train6    path: data/train-*7  - split: test_panels8    path: data/test_panels-*9  - split: test_humans10    path: data/test_humans-*11dataset_info:12  features:13  - name: id14    dtype: string15  - name: title16    dtype: string17  - name: abstract18    dtype: string19  - name: label20    sequence: string21  splits:22  - name: train23    num_bytes: 2303407924    num_examples: 1539025  - name: test_panels26    num_bytes: 275407727    num_examples: 136528  - name: test_humans29    num_bytes: 76445530    num_examples: 49731  download_size: 1497533832  dataset_size: 2655261133---34**ERC Panel Classification Dataset**35 36## Dataset Overview37 38The **ERC Panel Classification Dataset** is designed to fine-tune a multi-label classifier to predict one or more ERC (European Research Council) panels based on research paper titles and abstracts. The dataset consists of three splits: **train**, **test-panels**, and **test-humans**.39 40* **train**: The training set was generated through pseudolabeling using outputs from three different large language models (LLMs). The dataset contains multi-label panel assignments based on the content of each paper's title and abstract.41* **test-panels**: The test set comprises ERC projects with only a single panel assigned to each document. The dataset was sampled so that around 100 examples of each panel assignment are included.42* **test-humans**: This set was created using Argilla. For cases in the training set where there was disagreement between the two LLMs, three human annotators reviewed the documents. When two annotators could not agree on the panel(s), a third annotator, who had not seen the document, was consulted, and the final label was assigned based on majority agreement among the annotators.43 44### Use Case45 46This dataset is specifically created for fine-tuning a multi-label classifier to predict ERC panel(s). The training data uses a multi-label classification setup, while the test data has two parts:47 48* **test-panels**: Single-label evaluation, where agreement is reached if any of the labels from the training data matches the assigned ERC panel.49* **test-humans**: Multi-label evaluation, based on human annotations and majority agreement between annotators.50 51## Dataset Structure52 53The dataset contains the following fields:54 55* **id**: A unique identifier for the document.56* **title**: The title of the research paper.57* **abstract**: The abstract of the research paper.58* **label**: The assigned panel(s) for the paper. For **train**, this is a multi-label list (i.e., a list of panels assigned to the document). For **test-panels**, the label is a single panel per document. For **test-humans**, the label contains multiple panels as assigned by human annotators.59 60## Dataset Splits61 62The dataset is divided into three splits:63 64* **train**: The training data, generated using pseudolabeling from three different LLMs.65* **test-panels**: The test set consisting of ERC projects with single panel assignments.66* **test-humans**: The test set consisting of papers reviewed by human annotators when there was disagreement between LLMs in the training set.