mbruton/gal_en_XLM-R

1---2license: apache-2.03datasets:4- mbruton/galician_srl5- CoNLL-20126language:7- gl8- en9metrics:10- seqeval11library_name: transformers12pipeline_tag: token-classification13---14 15# Model Card for GalXLM-R-en for Semantic Role Labeling16 17This model is fine-tuned on a version of [XLM RoBERTa Base](https://huggingface.co/xlm-roberta-base) which is pre-trained on the SRL task for English, and is one of 24 models introduced as part of [this project](https://github.com/mbruton0426/GalicianSRL). Prior to this work, there were no published Galician datasets or models for SRL. 18 19## Model Details20 21### Model Description22 23GalXLM-R-en for Semantic Role Labeling (SRL) is a transformers model, leveraging XLM-R's extensive pretraining on 100 languages to achieve better SRL predictions for low-resource Galician. This model is additionally pre-trained on the SRL task for English. It was fine-tuned on Galician with the following objectives: 24 25- Identify up to 13 verbal roots within a sentence.26- Identify available arguments for each verbal root. Due to scarcity of data, this model focused solely on the identification of arguments 0, 1, and 2.27 28Labels are formatted as: r#:tag, where r# links the token to a specific verbal root of index #, and tag identifies the token as the verbal root (root) or an individual argument (arg0/arg1/arg2)29 30- **Developed by:** [Micaella Bruton](mailto:micaellabruton@gmail.com)31- **Model type:** Transformers32- **Language(s) (NLP):** Galician (gl), English (en)33- **License:** Apache 2.034- **Finetuned from model:** [English pre-trained XLM RoBERTa Base](https://huggingface.co/liaad/srl-en_xlmr-base)35 36### Model Sources 37 38- **Repository:** [GalicianSRL](https://github.com/mbruton0426/GalicianSRL)39- **Paper:** To be updated40 41## Uses42 43This model is intended to be used to develop and improve natural language processing tools for Galician.44 45## Bias, Risks, and Limitations46 47Galician is a low-resource language which prior to this project lacked a semantic role labeling dataset. As such, the dataset used to train this model is extrememly limited and could benefit from the inclusion of additional sentences and manual validation by native speakers.48 49## Training Details50 51### Training Data52 53This model was pre-trained on the [OntoNotes 5.0 English SRL corpus](http://catalog.ldc.upenn.edu/LDC2013T19).54This model was fine-tuned on the "train" portion of the [GalicianSRL Dataset](https://huggingface.co/datasets/mbruton/galician_srl) produced as part of this same project.55 56#### Training Hyperparameters57 58- **Learning Rate:** 2e-559- **Batch Size:** 1660- **Weight Decay:** 0.0161- **Early Stopping:** 10 epochs62 63## Evaluation64 65#### Testing Data66 67This model was tested on the "test" portion of the [GalicianSRL Dataset](https://huggingface.co/datasets/mbruton/galician_srl) produced as part of this same project.68 69#### Metrics70 71[seqeval](https://huggingface.co/spaces/evaluate-metric/seqeval) is a Python framework for sequence labeling evaluation. It can evaluate the performance of chunking tasks such as named-entity recognition, part-of-speech tagging, and semantic role labeling.72It supplies scoring both overall and per label type.73 74Overall:75- `accuracy`: the average [accuracy](https://huggingface.co/metrics/accuracy), on a scale between 0.0 and 1.0.76- `precision`: the average [precision](https://huggingface.co/metrics/precision), on a scale between 0.0 and 1.0.77- `recall`: the average [recall](https://huggingface.co/metrics/recall), on a scale between 0.0 and 1.0.78- `f1`: the average [F1 score](https://huggingface.co/metrics/f1), which is the harmonic mean of the precision and recall. It also has a scale of 0.0 to 1.0.79 80Per label type:81- `precision`: the average [precision](https://huggingface.co/metrics/precision), on a scale between 0.0 and 1.0.82- `recall`: the average [recall](https://huggingface.co/metrics/recall), on a scale between 0.0 and 1.0.83- `f1`: the average [F1 score](https://huggingface.co/metrics/f1), on a scale between 0.0 and 1.0.84 85### Results86 87| Label        | Precision | Recall | f1-score | Support |88| :----------: | :-------: | :----: | :------: | :-----: |89| 0:arg0       | 0.73 | 0.77 | 0.75 | 485  |90| 0:arg1       | 0.75 | 0.71 | 0.73 | 483  |91| 0:arg2       | 0.60 | 0.83 | 0.69 | 264  |92| 0:root       | 0.93 | 0.93 | 0.93 | 948  |93| 1:arg0       | 0.61 | 0.67 | 0.63 | 348  |94| 1:arg1       | 0.70 | 0.66 | 0.68 | 443  |95| 1:arg2       | 0.58 | 0.64 | 0.61 | 211  |96| 1:root       | 0.85 | 0.86 | 0.86 | 802  |97| 2:arg0       | 0.52 | 0.60 | 0.55 | 240  |98| 2:arg1       | 0.59 | 0.66 | 0.62 | 331  |99| 2:arg2       | 0.48 | 0.67 | 0.56 | 156  |100| 2:root       | 0.76 | 0.78 | 0.77 | 579  |101| 3:arg0       | 0.38 | 0.34 | 0.36 | 137  |102| 3:arg1       | 0.52 | 0.60 | 0.56 | 216  |103| 3:arg2       | 0.43 | 0.50 | 0.46 | 110  |104| 3:root       | 0.68 | 0.71 | 0.70 | 374  |105| 4:arg0       | 0.42 | 0.21 | 0.28 | 70   |106| 4:arg1       | 0.55 | 0.42 | 0.48 | 109  |107| 4:arg2       | 0.42 | 0.38 | 0.40 | 66   |108| 4:root       | 0.64 | 0.57 | 0.60 | 206  |109| 5:arg0       | 0.27 | 0.30 | 0.29 | 20   |110| 5:arg1       | 0.43 | 0.46 | 0.44 | 57   |111| 5:arg2       | 0.47 | 0.32 | 0.38 | 28   |112| 5:root       | 0.59 | 0.50 | 0.54 | 102  |113| 6:arg0       | 0.33 | 0.08 | 0.12 | 13   |114| 6:arg1       | 0.21 | 0.12 | 0.15 | 25   |115| 6:arg2       | 0.00 | 0.00 | 0.00 | 8    |116| 6:root       | 0.40 | 0.40 | 0.40 | 42   |117| 7:arg0       | 0.00 | 0.00 | 0.00 | 3    |118| 7:arg1       | 0.00 | 0.00 | 0.00 | 8    |119| 7:arg2       | 0.00 | 0.00 | 0.00 | 5    |120| 7:root       | 0.18 | 0.12 | 0.15 | 16   |121| 8:arg0       | 0.00 | 0.00 | 0.00 | 1    |122| 8:arg1       | 0.00 | 0.00 | 0.00 | 2    |123| 8:arg2       | 0.00 | 0.00 | 0.00 | 1    |124| 8:root       | 0.00 | 0.00 | 0.00 | 7    |125| 9:arg0       | 0.00 | 0.00 | 0.00 | 1    |126| 9:arg1       | 0.00 | 0.00 | 0.00 | 2    |127| 9:arg2       | 0.00 | 0.00 | 0.00 | 1    |128| 9:root       | 0.00 | 0.00 | 0.00 | 3    |129| 10:arg1      | 0.00 | 0.00 | 0.00 | 1    |130| 10:root      | 0.00 | 0.00 | 0.00 | 2    |131| micro avg    | 0.69 | 0.71 | 0.70 | 6926 |132| macro avg    | 0.36 | 0.35 | 0.35 | 6926 |133| weighted avg | 0.69 | 0.71 | 0.70 | 6926 |134| tot root avg | 0.46 | 0.44 | 0.45 | 3081 |135| tot A0 avg   | 0.33 | 0.30 | 0.30 | 1318 |136| tot A1 avg   | 0.34 | 0.33 | 0.33 | 1677 |137| tot A2 avg   | 0.30 | 0.33 | 0.31 | 850  |138| tot r0 avg   | 0.75 | 0.81 | 0.78 | 2180 |139| tot r1 avg   | 0.69 | 0.71 | 0.70 | 1804 |140| tot r2 avg   | 0.59 | 0.68 | 0.63 | 1306 |141| tot r3 avg   | 0.50 | 0.54 | 0.52 | 837  |142| tot r4 avg   | 0.51 | 0.40 | 0.44 | 451  |143| tot r5 avg   | 0.44 | 0.40 | 0.41 | 207  |144| tot r6 avg   | 0.24 | 0.15 | 0.17 | 88   |145| tot r7 avg   | 0.05 | 0.03 | 0.04 | 32   |146| tot r8 avg   | 0.00 | 0.00 | 0.00 | 11   |147| tot r9 avg   | 0.00 | 0.00 | 0.00 | 7    |148| tot r10 avg  | 0.00 | 0.00 | 0.00 | 3    |149 150## Citation151 152**BibTeX:**153 154```155@inproceedings{bruton-beloucif-2023-bertie,156    title = "{BERT}ie Bott{'}s Every Flavor Labels: A Tasty Introduction to Semantic Role Labeling for {G}alician",157    author = "Bruton, Micaella  and158      Beloucif, Meriem",159    editor = "Bouamor, Houda  and160      Pino, Juan  and161      Bali, Kalika",162    booktitle = "Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing",163    month = dec,164    year = "2023",165    address = "Singapore",166    publisher = "Association for Computational Linguistics",167    url = "https://aclanthology.org/2023.emnlp-main.671",168    doi = "10.18653/v1/2023.emnlp-main.671",169    pages = "10892--10902",170    abstract = "In this paper, we leverage existing corpora, WordNet, and dependency parsing to build the first Galician dataset for training semantic role labeling systems in an effort to expand available NLP resources. Additionally, we introduce verb indexing, a new pre-processing method, which helps increase the performance when semantically parsing highly-complex sentences. We use transfer-learning to test both the resource and the verb indexing method. Our results show that the effects of verb indexing were amplified in scenarios where the model was both pre-trained and fine-tuned on datasets utilizing the method, but improvements are also noticeable when only used during fine-tuning. The best-performing Galician SRL model achieved an f1 score of 0.74, introducing a baseline for future Galician SRL systems. We also tested our method on Spanish where we achieved an f1 score of 0.83, outperforming the baseline set by the 2009 CoNLL Shared Task by 0.025 showing the merits of our verb indexing method for pre-processing.",171}172 173```