CoolFace
Modelpublic

addyo07/query-scope-classifier

sourceHugging Faceapache-2.0updated 2mo agoView on Hugging Face
0likes
Model Card

Multi-lingual Query Scope Classifier (addyo07/query-scope-classifier)

A production-grade, fast, multi-lingual single-pass sequence classifier fine-tuned from answerdotai/ModernBERT-base to categorize incoming user queries into 4 distinct scope categories across English, Devanagari Hindi, and Hinglish.

🏷️ 4-Class Taxonomy

  1. 1.`ChitChat` (Label 0): Casual greetings, small talk, AI identity questions, emotional banter.
  2. 2.`User` (Label 1): Personal facts, user preferences, memory updates, user profile instructions.
  3. 3.`Domain` (Label 2, Primary Default): Code execution, math formulas, general domain task queries, technical instructions.
  4. 4.`Temporal` (Label 3): Time-sensitive queries, schedules, dates, past session history, reminders.

📊 Performance & SLA Benchmarks

  • —Base Architecture: answerdotai/ModernBERT-base (149M parameters, RoPE, Unpadded FlashAttention-2).
  • —Holdout Test Accuracy: 96.18% across 2,201 holdout samples.
  • —Macro F1 Score: 0.9619
  • —Calibrated Non-Default Precision: 98.01% at confidence threshold tau* = 0.81 (with automatic safe fallback to Domain when uncertain).
  • —Quantized INT8 ONNX File Size: 143.67 MB

Per-Class Recall Breakdown

Scope ClassRecallPrecisionF1-Score
ChitChat98.00%98.50%0.9825
Temporal97.28%97.80%0.9754
User95.27%97.73%0.9648
Domain (Default)94.18%95.20%0.9469

📁 Repository Structure

.gitattributes
README.md
model/
  onnx/
    config.json
    model_quantized.onnx         # 143.67 MB Dynamic INT8 ONNX model
  pytorch/
    config.json
    model.safetensors            # 571 MB PyTorch BFloat16 weights
    tokenizer.json
    tokenizer_config.json
scripts/                          # Full fine-tuning, dataset audit & quantization pipeline

💻 Python / PyTorch Usage

python
import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

MODEL_NAME = "addyo07/query-scope-classifier"
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME, subfolder="model/pytorch")
model = AutoModelForSequenceClassification.from_pretrained(MODEL_NAME, subfolder="model/pytorch")

labels = ["ChitChat", "User", "Domain", "Temporal"]
query = "aaj sham ko mera schedule kya hai?"

inputs = tokenizer(query, return_tensors="pt")
with torch.no_grad():
    logits = model(**inputs).logits
    probs = torch.softmax(logits, dim=-1)
    pred_idx = torch.argmax(probs, dim=-1).item()

print(f"Predicted Scope: {labels[pred_idx]} (Confidence: {probs[0][pred_idx].item():.4f})")

⚡ ONNX Runtime Usage (Fast CPU Inference)

python
import numpy as np
import onnxruntime as ort
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("addyo07/query-scope-classifier", subfolder="model/pytorch")
session = ort.InferenceSession("model/onnx/model_quantized.onnx", providers=["CPUExecutionProvider"])

query = "Remind me to submit the quarterly tax report tomorrow at 5pm"
inputs = tokenizer(query, return_tensors="np", max_length=64, truncation=True)

onnx_inputs = {
    "input_ids": inputs["input_ids"].astype(np.int64),
    "attention_mask": inputs["attention_mask"].astype(np.int64)
}
outputs = session.run(None, onnx_inputs)
logits = outputs[0][0]
probs = np.exp(logits) / np.sum(np.exp(logits))
pred_id = np.argmax(probs)

labels = ["ChitChat", "User", "Domain", "Temporal"]
print(f"Scope: {labels[pred_id]}, Confidence: {probs[pred_id]:.4f}")