Janvi17/customer-support-ticket-classifier

Customer Support Ticket Classifier

A fine-tuned DistilBERT model that classifies customer support tickets into 11 issue categories. Designed for automatic routing, triage, and analytics of customer inquiries.

🚀 [Try the live demo →](https://huggingface.co/spaces/Janvi17/customer-support-ticket-classifier-demo)

Quick Start

from transformers import pipeline

classifier = pipeline("text-classification", model="Janvi17/customer-support-ticket-classifier")

result = classifier("I want to cancel my subscription immediately")
print(result)
# [{'label': 'SUBSCRIPTION', 'score': 0.997}]

Categories

The model classifies text into one of 11 customer support issue types:

Usage Examples

Basic Classification

from transformers import pipeline

classifier = pipeline("text-classification", model="Janvi17/customer-support-ticket-classifier")

tickets = [
    "I want to cancel my subscription immediately",
    "Where is my package? I've been waiting for 2 weeks",
    "I need a refund for my last purchase",
    "How do I change my account password?",
    "I want to speak to a human agent",
    "Can you send me the invoice for order #12345?",
]

for ticket in tickets:
    result = classifier(ticket)
    print(f"  [{result[0]['label']:>12s}] (conf: {result[0]['score']:.3f}) {ticket}")

Output:

  [SUBSCRIPTION] (conf: 0.997) I want to cancel my subscription immediately
  [    DELIVERY] (conf: 0.997) Where is my package? I've been waiting for 2 weeks
  [      REFUND] (conf: 0.999) I need a refund for my last purchase
  [     ACCOUNT] (conf: 1.000) How do I change my account password?
  [     CONTACT] (conf: 0.999) I want to speak to a human agent
  [     INVOICE] (conf: 0.997) Can you send me the invoice for order #12345?

Batch Classification with Confidence Scores

from transformers import pipeline

classifier = pipeline(
    "text-classification",
    model="Janvi17/customer-support-ticket-classifier",
    top_k=3,  # return top 3 predictions
)

result = classifier("The payment for my subscription failed")
for pred in result[0]:
    print(f"  {pred['label']:>14s}: {pred['score']:.4f}")
# PAYMENT:       0.9661
# SUBSCRIPTION:  0.0153
# ORDER:         0.0068

Using with PyTorch Directly

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

model_name = "Janvi17/customer-support-ticket-classifier"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)

text = "I need a refund for my last purchase"
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=128)

with torch.no_grad():
    logits = model(**inputs).logits
    probs = torch.softmax(logits, dim=-1)
    pred_id = probs.argmax().item()

print(f"Predicted: {model.config.id2label[pred_id]} ({probs[0][pred_id]:.4f})")
# Predicted: REFUND (0.9995)

Production Usage with Confidence Threshold

For production systems, reject low-confidence predictions:

from transformers import pipeline

classifier = pipeline("text-classification", model="Janvi17/customer-support-ticket-classifier")
CONFIDENCE_THRESHOLD = 0.85

def classify_ticket(text: str) -> dict:
    result = classifier(text)[0]
    if result["score"] < CONFIDENCE_THRESHOLD:
        return {"label": "UNKNOWN", "score": result["score"], "routed_to": "human_review"}
    return {"label": result["label"], "score": result["score"], "routed_to": "auto"}

# High-confidence → auto-routed
print(classify_ticket("I need a refund"))
# {'label': 'REFUND', 'score': 0.999, 'routed_to': 'auto'}

# Low-confidence → human review
print(classify_ticket("asdfghjkl"))
# {'label': 'UNKNOWN', 'score': 0.78, 'routed_to': 'human_review'}

Evaluation Results

Held-Out Test Set (2,464 samples)

Per-Class Performance

Confusion Matrix

Perfect diagonal — zero off-diagonal errors on the held-out test set.

Training Trajectory

Best checkpoint selected at epoch 3 (highest validation macro F1). Early stopping was configured with patience=2.

Baselines

Training Details

Dataset

[Bitext Customer Support LLM Chatbot Training Dataset](https://huggingface.co/datasets/bitext/Bitext-customer-support-llm-chatbot-training-dataset)

• —License: CDLA-Sharing-1.0
• —Publisher: Bitext
• —Size: 26,872 rows → 24,635 after deduplication
• —Splits used: 80% train (19,708) / 10% validation (2,463) / 10% test (2,464), stratified
• —Language: English
• —Format: Synthetic, template-generated customer support messages with intentional typos, case variations, and paraphrasing. Includes {{placeholder}} tokens for entities like order numbers and names.

The dataset contains 11 high-level issue categories and 27 fine-grained intents. This model classifies at the category level.

Class Distribution (Training Set)

Imbalance ratio: 5.7× (ACCOUNT vs CANCEL). Despite this, macro F1 is perfect — all classes are well-separated in semantic space.

Preprocessing

1. 1.Exact-duplicate removal (26,872 → 24,635 samples)
2. 2.Stratified train/val/test split (80/10/10, seed=42)
3. 3.Tokenization with distilbert-base-uncased tokenizer, max_length=128
4. 4.Dynamic padding via DataCollatorWithPadding

No text was truncated — the longest tokenized input is 32 tokens.

Hyperparameters

Framework Versions

• —Transformers 5.7.0
• —PyTorch 2.11.0
• —Datasets 4.8.5
• —Tokenizers 0.22.2

Demo

[🚀 Try the live Gradio demo](https://huggingface.co/spaces/Janvi17/customer-support-ticket-classifier-demo) — paste any support ticket and see real-time classification with confidence breakdown.

Limitations and Risks

Known Failure Modes

This model was stress-tested with 28 adversarial inputs. Four systematic weaknesses were identified:

1. Negation Blindness 🔴

The model ignores negation. "Don't refund me, just fix the product" is classified as REFUND (99.95% confidence). The training data contains no negated intents, and DistilBERT's 6-layer architecture has limited compositional reasoning.

Mitigation: Add negated intent examples to training data, or post-process with a negation detector.

2. No Out-of-Distribution Rejection 🔴

The model assigns a label to any input, including gibberish, empty strings, and unrelated text. Examples:

Mitigation: Use a confidence threshold (recommended: 0.85) to reject uncertain predictions. See the production usage example above.

3. Heavy Typo Fragility 🟡

While the model handles mild typos well (the training data includes ~34% typo-augmented samples), severely misspelled text can cause misclassification:

Mitigation: Add a spell-correction preprocessing step, or augment training data with heavier typo injection.

4. Single-Label on Multi-Intent Tickets 🟡

Real support tickets often span multiple categories. The model picks one:

Mitigation: For full coverage, return top-k predictions or switch to multi-label classification.

Dataset Limitations

• —Synthetic data: The training set is template-generated, not sourced from real customer interactions. Real-world text may contain slang, code-switching, or domain-specific jargon not represented in training.
• —English only: The model is trained exclusively on English text.
• —Limited vocabulary: Some categories have as few as 184 unique words (CANCEL), meaning the model relies heavily on keyword matching rather than deep semantic understanding.
• —Placeholder artifacts: Training data contains {{Order Number}}, {{Person Name}}, etc. The model has learned to ignore these, but unusual entity formats in real data could affect performance.

Bias and Fairness

• —The synthetic dataset does not represent any specific demographic or dialect distribution.
• —Performance on non-standard English (e.g., AAVE, Indian English, ESL patterns) has not been evaluated.
• —The model may perform differently across age groups, regions, or communication styles.

When NOT to Use This Model

• —Safety-critical routing: Do not use as the sole decision-maker for urgent or safety-related tickets without human review.
• —Non-English text: The model will produce unreliable predictions on non-English input.
• —Fine-grained intent classification: This model classifies into 11 broad categories, not 27 fine-grained intents. If you need intent-level predictions (e.g., distinguishing cancel_order from check_cancellation_fee), retrain with the intent column.

Citation

If you use this model, please cite the training dataset:

@misc{bitext2023customer,
  title={Bitext - Customer Service Tagged Training Dataset for LLM-based Virtual Assistants},
  author={Bitext},
  year={2023},
  publisher={Hugging Face},
  url={https://huggingface.co/datasets/bitext/Bitext-customer-support-llm-chatbot-training-dataset}
}

License

This model is released under the Apache 2.0 License. The training dataset is licensed under CDLA-Sharing-1.0.