dar1bi/cloud-gaming-feedback-multilabel

Multi-label classification of cloud-gaming churn feedback

Fine-tuned distilbert-base-multilingual-cased that tags the technical issue a user complains about in the free-text comment left when cancelling a cloud-gaming subscription. A single comment may describe several problems at once, so this is a multi-label task over 6 classes.

Comments are short (median 73 characters) and multilingual — no translation step is used.

👉 Live demo: multilingual-game-feedback-classification.streamlit.app

Labels

Usage

The model outputs 6 independent sigmoid probabilities. Do not use a 0.5 threshold — per-class thresholds were tuned on the validation set and are shipped in inference_config.json. They give macro-F1 0.620 on the test set versus 0.581 with a plain 0.5 threshold.

import json

import torch
from huggingface_hub import hf_hub_download
from transformers import AutoModelForSequenceClassification, AutoTokenizer

REPO = "dar1bi/cloud-gaming-feedback-multilabel"

tokenizer = AutoTokenizer.from_pretrained(REPO)
model = AutoModelForSequenceClassification.from_pretrained(REPO).eval()
config = json.load(open(hf_hub_download(REPO, "inference_config.json")))

text = "Constant micro stutter and very low fps in The Finals"
encoded = tokenizer(text, truncation=True, padding="max_length",
                    max_length=config["max_len"], return_tensors="pt")
with torch.no_grad():
    probabilities = torch.sigmoid(model(**encoded).logits).numpy()[0]

labels = [label for label, probability, threshold
          in zip(config["labels"], probabilities, config["thresholds"])
          if probability >= threshold]
print(labels)   # ['frames_drop']

Training

• —Base model: distilbert-base-multilingual-cased
• —Objective: BCEWithLogitsLoss with pos_weight to compensate for class imbalance
• —Hyperparameters: learning rate 8e-5 (grid over 2e-5 / 5e-5 / 8e-5), 5 epochs with best-epoch selection by validation macro-F1, batch size 16, max sequence length 64
• —Split: iterative-stratified multi-label split 70 / 15 / 15 — 2760 / 584 / 612 examples
• —Hardware: Apple MPS, ~290 s per configuration

Evaluation

Primary metric is macro-F1 — it weights rare classes equally with frequent ones.

Per-class F1 on the test set: ping_latency 0.78, unable_launch 0.70, mouse_keyboard_headset 0.67, frames_drop 0.62, game_bug 0.49, failed_save 0.46.

The model was selected on the validation set among seven approaches, from TF-IDF with classical classifiers to a zero-shot LLM. A most-frequent-class baseline scores 0.119 macro-F1.

Limitations

• —Noisy labels. Labels are self-reported by users in the cancellation form, so they are neither consistent nor mutually exclusive. Part of what looks like model error is actually label noise.
• —Rare classes. game_bug and failed_save are recognised in fewer than half of real cases — they are infrequent in the data and semantically overlap with the other classes.
• —Very short texts. Comments under 30 characters often carry too little signal.
• —Single split, single seed. Differences of 0.01–0.02 macro-F1 are within noise.

Training data

Internal user feedback of a cloud-gaming service, collected at subscription cancellation (4 616 records, 3 956 after cleaning). The dataset is confidential and is not published.