TalkTix/roberta-base-request-type

Model Details

This model is designed to classify customer service inquiries into two requst type: Inicident and Service Request.

Training Data

The model was trained on a balanced dataset of 28000 entries composed of anonymized customer service inquiries. Each category contained a similar number of examples to prevent class imbalance. https://github.com/amosproj/amos2023ws01-ticket-chat-ai/tree/main/Backend/app/model/testdata/testdatawithgpt

Training Procedure

The model was fine-tuned over four epochs for a sequence classification task. We utilized a batch size of 4 and an Adam optimizer with a learning rate of 2e-5.

Model Performance

The model's performance was evaluated using a confusion matrix and a learning curve, as detailed below:

• —### Confusion Matrix Analysis

• —Incident: The model has a high number of correct classifications with 2500 true positives. There are 25 false negatives where "Incident" was incorrectly classified as "Service Request".
• —Service Request: There are 440 true positives where "Service Request" was correctly classified. However, there are 150 false negatives where "Service Request" was incorrectly classified as "Incident".

• —### Learning Curve Analysis

• —Training Loss: The training loss starts at approximately 0.20 and shows a steady decrease, ending near 0.12. This indicates that the model is effectively learning and improving its performance on the training dataset.
• —Validation Loss: The validation loss begins around 0.18 and, after a slight decrease, starts to plateau around 0.16. This suggests that while the model is learning and generalizing, improvements on the validation set are limited after the initial epochs.

• —### Interprating Model's Output:
• —LABEL_0 stands for Incident
• —LABEL_1 stands for Service Request