thealper2/deberta-v3-base-formality

deberta-v3-base-formality

microsoft/deberta-v3-base fine-tuned for binary English text formality classification.

The model outputs one of two labels:

Model details

Training data

Fine-tuned on `osyvokon/pavlick-formality-scores`, the crowdsourced formality annotations of Pavlick & Tetreault (2016). Each sentence carries a continuous avg_score on a 7-point Likert scale from -3 (very informal) to +3 (very formal); sentences come from four domains (answers, blog, email, news).

Label derivation

The original annotation is continuous; the binary labels used here are derived, not provided by the annotators:

label = FORMAL    if avg_score >  0.0
label = INFORMAL  if avg_score <= 0.0

The threshold is the documented neutral midpoint of the annotation scale (0.0). It was fixed a priori from the scale definition and was never tuned on the test split.

Data preparation

Classes are close to balanced, so the model is trained with standard (unweighted) cross-entropy.

Training hyperparameters

Evaluation

Evaluated once on the held-out official test split (2000 sentences). The test split was not used for threshold selection, hyperparameter tuning, early stopping or model selection.

Per class:

Confusion matrix (rows = true, columns = predicted):

Agreement with the original continuous annotation on the test split (avg_score vs. predicted P(FORMAL)): Pearson r = 0.7794, Spearman rho = 0.7959.

Usage

from transformers import AutoModelForSequenceClassification, AutoTokenizer
import torch

model_id = "thealper2/deberta-v3-base-formality"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id).eval()

text = "Could you please provide the requested document?"
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=128)
with torch.no_grad():
    probs = model(**inputs).logits.softmax(-1)[0]

print(model.config.id2label[int(probs.argmax())], float(probs.max()))
# -> FORMAL 0.9...

With pipeline:

from transformers import pipeline

clf = pipeline("text-classification", model="thealper2/deberta-v3-base-formality", top_k=None)
clf("hey u wanna grab some food later")

Limitations

• —Derived labels. The source annotation is a continuous score; binarising it at the scale midpoint discards the intensity of formality and makes sentences near the threshold intrinsically ambiguous. Model probabilities near 0.5 should be treated as "unclear", not as a confident decision.
• —Ties. Sentences with avg_score exactly at the threshold are mapped to INFORMAL by convention.
• —Domain coverage. Training data covers Yahoo! Answers, blogs, email and news sentences. Behaviour on other domains (chat logs, code, transcripts, non-native or dialectal English) is untested.
• —Sentence-level. Trained on single sentences of about 20.9 tokens on average; long multi-paragraph inputs are truncated at 128 tokens.
• —English only.
• —Annotation subjectivity. Formality judgements are subjective and the gold scores are crowd averages; the ceiling of this task is bounded by annotator agreement.

Citation

Training data:

@article{pavlick2016empirical,
  title   = {An Empirical Analysis of Formality in Online Communication},
  author  = {Pavlick, Ellie and Tetreault, Joel},
  journal = {Transactions of the Association for Computational Linguistics},
  volume  = {4},
  pages   = {61--74},
  year    = {2016}
}

Base model:

@inproceedings{he2023debertav3,
  title     = {DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing},
  author    = {He, Pengcheng and Gao, Jianfeng and Chen, Weizhu},
  booktitle = {ICLR},
  year      = {2023}
}