seantw/covid-19-vaccination-tweet-relevance

Model: covid-19-vaccination-tweet-relevance

Overview

This model is a text classifier trained to determine whether a tweet is related to COVID-19 vaccination or not.

Usage

tokenizer = AutoTokenizer.from_pretrained("seantw/covid-19-vaccination-tweet-relevance")
model = AutoModel.from_pretrained("seantw/covid-19-vaccination-tweet-relevance")

Training corpus

The training corpus comprises 9373 tweets, a daily random sampled dated from December 2020 to June 2022. These tweets were labeled by domain experts.

We have seperated trained another model for classifying the stance of a tweet towards the COVID-19 vaccination. Please refer to covid-19-vaccination-tweet-stance for more information.

Output Label Index

• —LABEL_0: "irrelevance"
• —LABEL_1: "relevance"

Performance Metrics

The model's performance metrics on the test set are as follows:

• —Accuracy: 0.9386
• —Macro-average metrics:
• —F1-score: 0.9339
• —Recall: 0.9277
• —Precision: 0.9418
• —Class-wise metrics:
• —For class "relevance":
• —F1-score: 0.9161
• —Precision: 0.9523
• —Recall: 0.8825
• —For class "irrelevance":
• —F1-score: 0.9516
• —Precision: 0.9312
• —Recall: 0.973

These metrics are based on a test set with a total size of 3699 samples.

Confusion Matrix

The confusion matrix of predictions on the test set is as follows:

Model Architecture

The model is fine-tuned based on COVID-Twitter-BERT v2.

Contact

Sean Yun-Shiuan Chuang (yunshiuan.chuang@wisc.edu)