StanfordSCALE/tutor_talkmoves_roberta_large_classifier

Tutor Talk Moves Classifier (RoBERTa Large)

Model Description

This model is a fine-tuned version of roberta-large that classifies tutor utterances based on teacher talk moves from talkmoves.com. The model identifies whether a tutor's message represents one of the following three categories, or none of them:

• —Classroom management
• —Pressing for accuracy or reasoning
• —Restating or revoicing

Training Data

This model is trained on text of tutoring sessions from three different tutoring providers, annotated by two raters.

• —Inter-Rater Reliability with Krippendorff's Alpha: 0.83
• —Total Training Examples: 1,849
• —Class Distribution: | Class | Examples | Percentage | |-------|----------|------------| | None | 987 | 53.4% | | Classroom management | 284 | 15.4% | | Pressing for accuracy or reasoning | 474 | 25.6% | | Restating or revoicing | 104 | 5.6% |

Data Format

The is trained on utterances with the following format: [PRETEXT] {3 previous messages} [TEXT] {target message}, where [PRETEXT] and [TEXT] are special tokens. Names are anonymized, message text is lowercased, and leading and trailing whitespace is removed.

Example:

[PRETEXT] tutor: hello there [student]
tutor: what is the answer to this problem?
student: the answer is 6 [TEXT] tutor: why do you say the answer is 6?

Performance

Test set results (264 examples):