abdulmuinnn/supportpilot-distilbert

SupportPilot DistilBERT

SupportPilot DistilBERT is a fine-tuned DistilBERT model for e-commerce customer support intent classification.

The model predicts one of 46 customer-support intents from an English-language support message.

It is the inference model used by the open-source SupportPilot AI portfolio project, which exposes the classifier through FastAPI, Streamlit, confidence-aware fallback logic, automated testing, CI, and Docker deployment.

Model Description

• —Task: Multi-class text classification
• —Domain: E-commerce customer support
• —Language: English
• —Architecture: DistilBERT
• —Number of intents: 46
• —Framework: PyTorch / Hugging Face Transformers
• —Base model: distilbert/distilbert-base-uncased

The model is designed to identify operational support intents such as order tracking, cancellation, refunds, payment issues, account support, delivery issues, product information, and returns.

Intended Use

The model is intended for:

• —customer-support intent routing
• —support ticket triage
• —chatbot intent detection
• —help-desk automation prototypes
• —NLP and ML engineering demonstrations

For production systems, predictions should be combined with confidence thresholds, fallback handling, monitoring, and human review for uncertain cases.

The SupportPilot AI application uses both prediction confidence and the margin between the top two predictions before accepting an intent.

Example

Input:

Where is my order?

Example prediction from the released model:

Intent: track_order
Confidence: 99.8182%

Evaluation

Three model families were evaluated during development.

The final DistilBERT model achieved the following results on the held-out test set:

These results were measured on the project's held-out dataset and should not be interpreted as guaranteed performance on unseen real-world customer traffic.

Usage

Install the required libraries:

pip install torch transformers

Load the model:

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

model_id = "abdulmuinnn/supportpilot-distilbert"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id)

text = "Where is my order?"

inputs = tokenizer(
    text,
    return_tensors="pt",
    truncation=True,
    max_length=64,
)

with torch.no_grad():
    logits = model(**inputs).logits
    probabilities = torch.softmax(logits, dim=-1)

confidence, index = probabilities.max(dim=-1)

intent = model.config.id2label[index.item()]

print("Intent:", intent)
print("Confidence:", confidence.item())

Intent Labels

The model contains 46 intent labels:

<details> <summary>Show all intent labels</summary>

add_product
availability
availability_in_store
availability_online
cancel_order
change_account
change_order
close_account
customer_service
damaged_delivery
delivery_issue
delivery_time
exchange_product
exchange_product_in_store
human_agent
missing_item
open_account
order_history
pay
payment_issue
payment_methods
product_information
product_issue
recover_password
refund_policy
refund_status
remove_product
request_invoice
request_refund
request_right_to_rectification
return_policy
return_product
return_product_in_store
return_product_online
sales_period
shipping_costs
store_location
store_opening_hours
submit_feedback
submit_product_feedback
submit_product_idea
technical_issue
track_delivery
track_order
use_app
wrong_item

</details>

Confidence and Fallback Behavior

The model itself produces class probabilities.

The SupportPilot AI application adds a separate decision policy with default thresholds:

Minimum confidence: 0.70
Minimum top-1 / top-2 margin: 0.10

A prediction is accepted only when both thresholds are satisfied. Otherwise, the application routes the message to a fallback / human-review path.

This fallback behavior is implemented by the application layer rather than embedded directly in the model weights.

Dataset

The model was fine-tuned using the Bitext Retail eCommerce customer-support dataset.

Dataset repository:

bitext/Bitext-retail-ecommerce-llm-chatbot-training-dataset

The training pipeline used normalized duplicate handling and stratified train, validation, and test splits.

The dataset contains customer-support utterances covering 46 intents across multiple e-commerce support categories.

Base Model

The model was fine-tuned from:

distilbert/distilbert-base-uncased

DistilBERT provides a smaller and faster Transformer architecture than the original BERT model while retaining strong text-classification performance.

Limitations

This model should be treated as a domain-specific classifier rather than a general customer-support intelligence system.

Important limitations include:

• —evaluation was performed on the project dataset rather than live production traffic
• —real customer messages may contain language patterns not represented in the training data
• —confidence scores are not guarantees of correctness
• —out-of-domain text may still receive a high-scoring class prediction
• —the model was designed for English-language support messages
• —production deployments should include monitoring, fallback behavior, human review, and periodic evaluation

Model Artifact Integrity

SHA-256 of the released model.safetensors artifact:

66daf66ab6c5b55cc7d2dccf6b1dc69ef5f5512b513e4d77ab2476b15dc198dc

This hash can be used to verify the exact model artifact used for this release.

Source Project

The model is used by SupportPilot AI:

https://github.com/abdulmuinn/supportpilot-ai

The application repository demonstrates:

DistilBERT inference
→ confidence-aware decision policy
→ FastAPI
→ Streamlit
→ automated testing
→ CI
→ Docker Compose

Licensing and Attribution

The application source code in the SupportPilot AI GitHub repository is released separately under the MIT License.

This model was fine-tuned from distilbert/distilbert-base-uncased and trained using the Bitext Retail eCommerce dataset.

The base model and dataset have their own licensing and usage terms. Users should review those upstream terms before redistributing or using the model in downstream applications.

No separate license is asserted here for the fine-tuned model weights beyond the applicable upstream terms.