pie/aae2

1# PIE Dataset Card for "aae2"2 3This is a [PyTorch-IE](https://github.com/ChristophAlt/pytorch-ie) wrapper for the Argument Annotated Essays v2 (AAE2) dataset ([paper](https://aclanthology.org/J17-3005.pdf) and [homepage](https://tudatalib.ulb.tu-darmstadt.de/handle/tudatalib/2422)). Since the AAE2 dataset is published in the [BRAT standoff format](https://brat.nlplab.org/standoff.html), this dataset builder is based on the [PyTorch-IE brat dataset loading script](https://huggingface.co/datasets/pie/brat).4 5Therefore, the `aae2` dataset as described here follows the data structure from the [PIE brat dataset card](https://huggingface.co/datasets/pie/brat).6 7### Usage8 9```python10from pie_datasets import load_dataset11from pie_datasets.builders.brat import BratDocumentWithMergedSpans12from pie_documents.documents import TextDocumentWithLabeledSpansBinaryRelationsAndLabeledPartitions13 14# load default version15dataset = load_dataset("pie/aae2")16assert isinstance(dataset["train"][0], BratDocumentWithMergedSpans)17 18# if required, normalize the document type (see section Document Converters below)19dataset_converted = dataset.to_document_type(TextDocumentWithLabeledSpansBinaryRelationsAndLabeledPartitions)20assert isinstance(dataset_converted["train"][0], TextDocumentWithLabeledSpansBinaryRelationsAndLabeledPartitions)21 22# get first relation in the first document23doc = dataset_converted["train"][0]24print(doc.binary_relations[0])25# BinaryRelation(head=LabeledSpan(start=716, end=851, label='Premise', score=1.0), tail=LabeledSpan(start=591, end=714, label='Claim', score=1.0), label='supports', score=1.0)26print(doc.binary_relations[0].resolve())27# ('supports', (('Premise', 'What we acquired from team work is not only how to achieve the same goal with others but more importantly, how to get along with others'), ('Claim', 'through cooperation, children can learn about interpersonal skills which are significant in the future life of all students')))28```29 30### Dataset Summary31 32Argument Annotated Essays Corpus (AAEC) ([Stab and Gurevych, 2017](https://aclanthology.org/J17-3005.pdf)) contains student essays. A stance for a controversial theme is expressed by a major claim component as well as claim components, and premise components justify or refute the claims. Attack and support labels are defined as relations. The span covers a statement, *which can stand in isolation as a complete sentence*, according to the AAEC annotation guidelines. All components are annotated with minimum boundaries of a clause or sentence excluding so-called "shell" language such as *On the other hand* and *Hence*. (Morio et al., 2022, p. 642)33 34In the original dataset, there is no premise that links to another premise or claim in a different paragraph. That means, an argumentation tree structure is complete within each paragraph. Therefore, it is possible to train a model on the full documents or just at the paragraph-level which is usually less memory-exhaustive (Eger et al., 2017, p. 16).35However, through our `DOCUMENT_CONVERTERS`, we build links between claims, creating a graph structure throughout an entire essay (see [Document Converters](#document-converters)).36 37### Supported Tasks and Leaderboards38 39- **Tasks**: Argumentation Mining, Component Identification, Component Classification, Structure Identification40- **Leaderboard:** [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)41 42### Languages43 44The language in the dataset is English (persuasive essays).45 46### Dataset Variants47 48The `aae2` dataset comes in a single version (`default`) with `BratDocumentWithMergedSpans` as document type. Note, that this in contrast to the base brat dataset, where the document type for the `default` variant is `BratDocument`. The reason is that the AAE2 dataset has already been published with only single-fragment spans. Without any need to merge fragments, the document type `BratDocumentWithMergedSpans` is easier to handle for most of the task modules.49 50### Data Schema51 52See [PIE-Brat Data Schema](https://huggingface.co/datasets/pie/brat#data-schema).53 54### Data Splits55 56| Statistics                                                   |                      Train |                     Test |57| ------------------------------------------------------------ | -------------------------: | -----------------------: |58| No. of document                                              |                        322 |                       80 |59| Components <br/>- `MajorClaim`<br/>- `Claim`<br/>- `Premise` | <br/>598<br/>1202<br/>3023 | <br/>153<br/>304<br/>809 |60| Relations\*<br/>- `supports`<br/>- `attacks`                 |          <br/>3820<br/>405 |         <br/>1021<br/>92 |61 62\* included all relations between claims and premises and all claim attributions.63 64See further statistics in Stab & Gurevych (2017), p. 650, Table A.1.65 66### Label Descriptions and Statistics67 68#### Components69 70| Components   | Count | Percentage |71| ------------ | ----: | ---------: |72| `MajorClaim` |   751 |     12.3 % |73| `Claim`      |  1506 |     24.7 % |74| `Premise`    |  3832 |     62.9 % |75 76- `MajorClaim` is the root node of the argumentation structure and represents the author’s standpoint on the topic. Essay bodies either support or attack the author’s standpoint expressed in the major claim. The major claim can be mentioned multiple times in a single document.77- `Claim` constitutes the central component of each argument. Each one has at least one premise and takes stance attribute values "for" or "against" with regarding the major claim.78- `Premise` is the reasons of the argument; either linked to claim or another premise.79 80#### Relations81 82| Relations           | Count | Percentage |83| ------------------- | ----: | ---------: |84| support: `supports` |  3613 |     94.3 % |85| attack: `attacks`   |   219 |      5.7 % |86 87- "Each premise `p` has one **outgoing relation** (i.e., there is a relation that has p as source component) and none or several **incoming relations** (i.e., there can be a relation with `p` as target component)."88- "A `Claim` can exhibit several **incoming relations** but no **outgoing relation**." (S&G, 2017, p. 68)89- "The relations from the claims of the arguments to the major claim are dotted since we will not explicitly annotated them. The relation of each argument to the major claim is indicated by a stance attribute of each claim. This attribute can either be for or against as illustrated in figure 1.4." (Stab & Gurevych, *Guidelines for Annotating Argumentation Structures in Persuasive Essays*, 2015, p. 5)90 91See further description in Stab & Gurevych 2017, p.627 and the [annotation guideline](https://github.com/ArneBinder/pie-datasets/blob/db94035602610cefca2b1678aa2fe4455c96155d/data/datasets/ArgumentAnnotatedEssays-2.0/guideline.pdf).92 93**Note that** relations between `MajorClaim` and `Claim` were not annotated; however, each claim is annotated with an `Attribute` annotation with value `for` or `against` - which indicates the relation between itself and `MajorClaim`. In addition, when two non-related `Claim` 's appear in one paragraph, there is also no relations to one another. An example of a document is shown here below.94 95#### Example96 97![Structure](img/sg17f2.png)98 99![Example](img/aae2_train_47.png)100 101### Document Converters102 103The dataset provides document converters for the following target document types:104 105- `pie_documents.documents.TextDocumentWithLabeledSpansAndBinaryRelations` with layers:106  - `labeled_spans`: `LabeledSpan` annotations, converted from `BratDocumentWithMergedSpans`'s `spans`107    - labels: `MajorClaim`, `Claim`, `Premise`108  - `binary_relations`: `BinaryRelation` annotations, converted from `BratDocumentWithMergedSpans`'s `relations`109    - there are two conversion methods that convert `Claim` attributes to their relations to `MajorClaim` (also see the label-count changes after this relation conversion [here below](#label-counts-after-document-converter)):110      - `connect_first` (default setting):111        - build a `supports` or `attacks` relation from each `Claim` to the first `MajorClaim` depending on the `Claim`'s attribute (`for` or `against`), and112        - build a `semantically_same` relation between following `MajorClaim` to the first `MajorClaim`113      - `connect_all`114        - build a `supports` or `attacks` relation from each `Claim` to every `MajorClaim`115        - no relations between each `MajorClaim`116    - labels: `supports`, `attacks`, and `semantically_same` if `connect_first`117- `pie_documents.documents.TextDocumentWithLabeledSpansBinaryRelationsAndLabeledPartitions` with layers:118  - `labeled_spans`, as above119  - `binary_relations`, as above120  - `labeled_partitions`, `LabeledSpan` annotations, created from splitting `BratDocumentWithMergedSpans`'s `text` at new lines (`\n`).121    - every partition is labeled as `paragraph`122 123See [here](https://github.com/ArneBinder/pie-documents/blob/main/src/pie_documents/documents.py) for the document type124definitions.125 126#### Relation Label Statistics after Document Conversion127 128When converting from `BratDocumentWithMergedSpan` to `TextDocumentWithLabeledSpansAndBinaryRelations` and `TextDocumentWithLabeledSpansBinaryRelationsAndLabeledPartitions`,129we apply a relation-conversion method (see above) that changes the label counts for the relations, as follows:130 1311. `connect_first` (default):132 133| Relations                  | Count | Percentage |134| -------------------------- | ----: | ---------: |135| support: `supports`        |  4841 |     85.1 % |136| attack: `attacks`          |   497 |      8.7 % |137| other: `semantically_same` |   349 |      6.2 % |138 1392. `connect_all`140 141| Relations           | Count | Percentage |142| ------------------- | ----: | ---------: |143| support: `supports` |  5958 |     89.3 % |144| attack: `attacks`   |   715 |     10.7 % |145 146### Collected Statistics after Document Conversion147 148We use the script `evaluate_documents.py` from [PyTorch-IE-Hydra-Template](https://github.com/ArneBinder/pytorch-ie-hydra-template-1) to generate these statistics.149After checking out that code, the statistics and plots can be generated by the command:150 151```commandline152python src/evaluate_documents.py dataset=aae2_base metric=METRIC153```154 155where a `METRIC` is called according to the available metric configs in `config/metric/METRIC` (see [metrics](https://github.com/ArneBinder/pytorch-ie-hydra-template-1/tree/main/configs/metric)).156 157This also requires to have the following dataset config in `configs/dataset/aae2_base.yaml` of this dataset within the repo directory:158 159```commandline160_target_: src.utils.execute_pipeline161input:162  _target_: pie_datasets.DatasetDict.load_dataset163  path: pie/aae2164  revision: 1015ee38bd8a36549b344008f7a49af72956a7fe165```166 167For token based metrics, this uses `bert-base-uncased` from `transformer.AutoTokenizer` (see [AutoTokenizer](https://huggingface.co/docs/transformers/v4.37.1/en/model_doc/auto#transformers.AutoTokenizer), and [bert-based-uncased](https://huggingface.co/bert-base-uncased) to tokenize `text` in `TextDocumentWithLabeledSpansAndBinaryRelations` (see [document type](https://github.com/ArneBinder/pie-documents/blob/main/src/pie_documents/documents.py)).168 169For relation-label statistics, we collect those from the default relation conversion method, i.e., `connect_first`, resulting in three distinct relation labels.170 171#### Relation argument (outer) token distance per label172 173The distance is measured from the first token of the first argumentative unit to the last token of the last unit, a.k.a. outer distance.174 175We collect the following statistics: number of documents in the split (*no. doc*), no. of relations (*len*), mean of token distance (*mean*), standard deviation of the distance (*std*), minimum outer distance (*min*), and maximum outer distance (*max*).176We also present histograms in the collapsible, showing the distribution of these relation distances (x-axis; and unit-counts in y-axis), accordingly.177 178<details>179<summary>Command</summary>180 181```182python src/evaluate_documents.py dataset=aae2_base metric=relation_argument_token_distances183```184 185</details>186 187##### train (322 documents)188 189|                   |  len | max |    mean | min |     std |190| :---------------- | ---: | --: | ------: | --: | ------: |191| ALL               | 9002 | 514 | 102.582 |   9 |   93.76 |192| attacks           |  810 | 442 | 127.622 |  10 | 109.283 |193| semantically_same |  552 | 514 | 301.638 |  25 |  73.756 |194| supports          | 7640 | 493 |  85.545 |   9 |  74.023 |195 196<details>197  <summary>Histogram (split: train, 322 documents)</summary>198 199![rtd-label_aae2_train.png](img%2Frtd-label_aae2_train.png)200 201</details>202 203##### test (80 documents)204 205|                   |  len | max |    mean | min |    std |206| :---------------- | ---: | --: | ------: | --: | -----: |207| ALL               | 2372 | 442 | 100.711 |  10 | 92.698 |208| attacks           |  184 | 402 | 115.891 |  12 | 98.751 |209| semantically_same |  146 | 442 | 299.671 |  34 | 72.921 |210| supports          | 2042 | 437 |  85.118 |  10 | 75.023 |211 212<details>213  <summary>Histogram (split: test, 80 documents)</summary>214 215![rtd-label_aae2_test.png](img%2Frtd-label_aae2_test.png)216 217</details>218 219#### Span lengths (tokens)220 221The span length is measured from the first token of the first argumentative unit to the last token of the particular unit.222 223We collect the following statistics: number of documents in the split (*no. doc*), no. of spans (*len*), mean of number of tokens in a span (*mean*), standard deviation of the number of tokens (*std*), minimum tokens in a span (*min*), and maximum tokens in a span (*max*).224We also present histograms in the collapsible, showing the distribution of these token-numbers (x-axis; and unit-counts in y-axis), accordingly.225 226<details>227<summary>Command</summary>228 229```230python src/evaluate_documents.py dataset=aae2_base metric=span_lengths_tokens231```232 233</details>234 235| statistics |  train |   test |236| :--------- | -----: | -----: |237| no. doc    |    322 |     80 |238| len        |   4823 |   1266 |239| mean       | 17.157 | 16.317 |240| std        |  8.079 |  7.953 |241| min        |      3 |      3 |242| max        |     75 |     50 |243 244<details>245  <summary>Histogram (split: train, 332 documents)</summary>246 247![slt_aae2_train.png](img%2Fslt_aae2_train.png)248 249</details>250  <details>251  <summary>Histogram (split: test, 80 documents)</summary>252 253![slt_aae2_test.png](img%2Fslt_aae2_test.png)254 255</details>256 257#### Token length (tokens)258 259The token length is measured from the first token of the document to the last one.260 261We collect the following statistics: number of documents in the split (*no. doc*), mean of document token-length (*mean*), standard deviation of the length (*std*), minimum number of tokens in a document (*min*), and maximum number of tokens in a document (*max*).262We also present histograms in the collapsible, showing the distribution of these token lengths (x-axis; and unit-counts in y-axis), accordingly.263 264<details>265<summary>Command</summary>266 267```268python src/evaluate_documents.py dataset=aae2_base metric=count_text_tokens269```270 271</details>272 273| statistics |   train |   test |274| :--------- | ------: | -----: |275| no. doc    |     322 |     80 |276| mean       | 377.686 |  378.4 |277| std        |  64.534 | 66.054 |278| min        |     236 |    269 |279| max        |     580 |    532 |280 281<details>282  <summary>Histogram (split: train, 332 documents)</summary>283 284![tl_aae2_train.png](img%2Ftl_aae2_train.png)285 286</details>287  <details>288  <summary>Histogram (split: test, 80 documents)</summary>289 290![tl_aae2_test.png](img%2Ftl_aae2_test.png)291 292</details>293 294## Dataset Creation295 296### Curation Rationale297 298"The identification of argumentation structures involves several subtasks like separating argumentative from non-argumentative text units (Moens et al. 2007; Florou299et al. 2013), classifying argument components into claims and premises (Mochales-Palau and Moens 2011; Rooney, Wang, and Browne 2012; Stab and Gurevych 2014b),300and identifying argumentative relations (Mochales-Palau and Moens 2009; Peldszus3012014; Stab and Gurevych 2014b). However, an approach that covers all subtasks is still302missing. However, an approach that covers all subtasks is still303missing. Furthermore, most approaches operate locally and do not optimize the global304argumentation structure.305 306"In addition,307to the lack of end-to-end approaches for parsing argumentation structures, there are308relatively few corpora annotated with argumentation structures at the discourse-level." (p. 621)309 310"Our primary motivation for this work is to create argument analysis methods311for argumentative writing support systems and to achieve a better understanding312of argumentation structures." (p. 622)313 314### Source Data315 316Persuasive essays were collected from [essayforum.com](https://essayforum.com/) (See essay prompts, along with the essay's `id`'s [here](https://github.com/ArneBinder/pie-datasets/blob/db94035602610cefca2b1678aa2fe4455c96155d/data/datasets/ArgumentAnnotatedEssays-2.0/prompts.csv)).317 318#### Initial Data Collection and Normalization319 320"We randomly selected 402 English essays with a description of the writing prompt from321essayforum.com. This online forum is an active community that provides correction and322feedback about different texts such as research papers, essays, or poetry. For example,323students post their essays in order to receive feedback about their writing skills while324preparing for standardized language tests. The corpus includes 7,116 sentences with325147,271 tokens." (p. 630)326 327#### Who are the source language producers?328 329[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)330 331### Annotations332 333#### Annotation process334 335The annotation were done using BRAT Rapid Annotation Tool ([Stenetorp et al., 2012](https://aclanthology.org/E12-2021/)).336 337All three annotators independently annotated a random subset of 80 essays. The338remaining 322 essays were annotated by the expert annotator.339 340The authors evaluated the inter-annotator agreement using observed agreement and Fleiss’ κ (Fleiss 1971), on each label on each sub-tasks,341namely, component identification, component classification, and relation identification.342The results were reported in their [paper](https://aclanthology.org/J17-3005.pdf) in Tables 2-4.343 344#### Who are the annotators?345 346Three non-native speakers; one of the three being an expert annotator.347 348### Personal and Sensitive Information349 350[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)351 352## Considerations for Using the Data353 354### Social Impact of Dataset355 356"[Computational Argumentation] have357broad application potential in various areas such as legal decision support (Mochales-Palau and Moens 2009), information retrieval (Carstens and Toni 2015), policy making (Sardianos et al. 2015), and debating technologies (Levy et al. 2014; Rinott et al.3582015)." (p. 619)359 360### Discussion of Biases361 362[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)363 364### Other Known Limitations365 366The relations between claims and major claims are not explicitly annotated.367 368"The proportion of non-argumentative text amounts to 47,474 tokens (32.2%) and3691,631 sentences (22.9%). The number of sentences with several argument components370is 583, of which 302 include several components with different types (e.g., a claim followed by premise)...371[T]he identification of argument components requires the372separation of argumentative from non-argumentative text units and the recognition of373component boundaries at the token level...The proportion of paragraphs with unlinked374argument components (e.g., unsupported claims without incoming relations) is 421375(23%). Thus, methods that link all argument components in a paragraph are only of376limited use for identifying the argumentation structures in our corpus.377 378"Most of the arguments are convergent—that is, the depth of the379argument is 1. The number of arguments with serial structure is 236 (20.9%)." (p. 634)380 381## Additional Information382 383### Dataset Curators384 385[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)386 387### Licensing Information388 389**License**: [License description by TU Darmstadt](https://tudatalib.ulb.tu-darmstadt.de/bitstream/handle/tudatalib/2422/arg_annotated_essays_v2_license.pdf?sequence=2&isAllowed=y)390 391**Funding**: This work has been supported by the392Volkswagen Foundation as part of the393Lichtenberg-Professorship Program under394grant no. I/82806 and by the German Federal395Ministry of Education and Research (BMBF)396as a part of the Software Campus project397AWS under grant no. 01—S12054.398 399### Citation Information400 401```402@article{stab2017parsing,403  title={Parsing argumentation structures in persuasive essays},404  author={Stab, Christian and Gurevych, Iryna},405  journal={Computational Linguistics},406  volume={43},407  number={3},408  pages={619--659},409  year={2017},410  publisher={MIT Press One Rogers Street, Cambridge, MA 02142-1209, USA journals-info~…}411}412```413 414```415@misc{https://tudatalib.ulb.tu-darmstadt.de/handle/tudatalib/2422,416url = { https://tudatalib.ulb.tu-darmstadt.de/handle/tudatalib/2422 },417author = { Stab, Christian and Gurevych, Iryna },418keywords = { Argument Mining, 409-06 Informationssysteme, Prozess- und Wissensmanagement, 004 },419publisher = { Technical University of Darmstadt },420year = { 2017 },421copyright = { License description },422title = { Argument Annotated Essays (version 2) }423}424```425 426### Contributions427 428Thanks to [@ArneBinder](https://github.com/ArneBinder) and [@idalr](https://github.com/idalr) for adding this dataset.429