clips/republic

1---2pipeline_tag: text-classification3language: 4  - nl5tags:6  - text classification7  - sentiment analysis8  - domain adaptation  9widget:10- text: "De NMBS heeft recent de airconditioning in alle treinen vernieuwd."11  example_title: "POS-NMBS"12- text: "De wegenwerken langs de E34 blijven al maanden aanhouden."13  example_title: "NEG-AWV"14- text: "Natuur en Bos is erin geslaagd 100 hectaren bosgebied te beschermen."15  example_title: "POS-ANB"16- text: "Het FWO financiert te weinig excellent onderzoek."17  example_title: "NEG-FWO"18- text: "De Lijn is op zoek naar nieuwe buschauffeurs."19  example_title: "NEU-De Lijn"20---21 22# RePublic23 24### Model description25RePublic (<u>re</u>putation analyzer for <u>public</u> service organizations) is a Dutch BERT model based on BERTje (De Vries, 2019). The model was designed to predict the sentiment in Dutch-language news article text about public agencies. RePublic was developed by [CLiPS](https://www.uantwerpen.be/en/research-groups/clips/) in collaboration with Prof. Dr. [Jan Boon](https://www.uhasselt.be/nl/wie-is-wie/jan-boon).26 27### How to use28The model can be loaded and used to make predictions as follows:29 30 ```python31from transformers import pipeline32model_path = 'clips/republic'33pipe = pipeline(task="text-classification", 34           model=model_path, tokenizer=model_path)	35text = … # load your text here36output = pipe(text)37prediction  = output[0]['label'] # 0=”neutral”; 1=”positive”; 2=”negative”38```39 40### Training data and procedure41RePublic was domain-adapted on 91 661 Flemish news articles from three popular Flemish news providers between 2000 and 2020 (“Het Laatste Nieuws”, “Het Nieuwsblad” and “De Morgen”). These articles mention at least one out of a pre-defined list of 24 public service organizations, which contains, a.o., De Lijn (public transport organization), VDAB (Flemish job placement service), and Agentschap Zorg en Gezondheid (healthcare service). The domain adaptation was achieved by performing BERT’s language modeling tasks (masked language modeling & next sentence prediction).42 43The model was then fine-tuned on a sentiment classification task (“positive”, “negative”, “neutral”). The supervised data consisted of 4404 annotated sentences mentioning Flemish public agencies of which 1257 sentences were positive, 1485 sentences were negative and 1662 sentences were neutral. Fine-tuning was performed for 4 epochs using a batch size of 8 and a learning rate of 5e-5. In order to evaluate the model, a 10-fold cross validation experiment was conducted. The results of this experiment can be found below.44 45| **Class** | **Precision (%)** | **Recall (%)** | **F1-score (%)** |46|:---:|:---:|:---:|:---:|47| _Positive_ | 87.3 | 88.6 | 88.0 |48| _Negative_ | 86.4 | 86.5 | 86.5 |49| _Neutral_ | 85.3 | 84.2 | 84.7 |50| _Macro-averaged_ | 86.3 | 86.4 | 86.4 |