addyo07/query-scope-classifier

1---2language:3- en4- hi5- multilingual6license: apache-2.07library_name: transformers8pipeline_tag: text-classification9tags:10- query-classification11- intent-detection12- memory-scope13- modernbert14- onnx15- quantized16metrics:17- accuracy18- f119model_name: Query Scope Classifier (ModernBERT-base)20---21 22# Multi-lingual Query Scope Classifier (`addyo07/query-scope-classifier`)23 24A production-grade, fast, multi-lingual single-pass sequence classifier fine-tuned from `answerdotai/ModernBERT-base` to categorize incoming user queries into 4 distinct scope categories across English, Devanagari Hindi, and Hinglish.25 26## 🏷️ 4-Class Taxonomy27 281. **`ChitChat`** (Label `0`): Casual greetings, small talk, AI identity questions, emotional banter.292. **`User`** (Label `1`): Personal facts, user preferences, memory updates, user profile instructions.303. **`Domain`** (Label `2`, **Primary Default**): Code execution, math formulas, general domain task queries, technical instructions.314. **`Temporal`** (Label `3`): Time-sensitive queries, schedules, dates, past session history, reminders.32 33---34 35## 📊 Performance & SLA Benchmarks36 37- **Base Architecture**: `answerdotai/ModernBERT-base` (149M parameters, RoPE, Unpadded FlashAttention-2).38- **Holdout Test Accuracy**: **96.18%** across 2,201 holdout samples.39- **Macro F1 Score**: **0.9619**40- **Calibrated Non-Default Precision**: **98.01%** at confidence threshold tau* = 0.81 (with automatic safe fallback to Domain when uncertain).41- **Quantized INT8 ONNX File Size**: **143.67 MB**42 43### Per-Class Recall Breakdown44 45| Scope Class | Recall | Precision | F1-Score |46|---|---|---|---|47| **ChitChat** | **98.00%** | **98.50%** | **0.9825** |48| **Temporal** | **97.28%** | **97.80%** | **0.9754** |49| **User** | **95.27%** | **97.73%** | **0.9648** |50| **Domain** (Default) | **94.18%** | **95.20%** | **0.9469** |51 52---53 54## 📁 Repository Structure55 56```57.gitattributes58README.md59model/60  onnx/61    config.json62    model_quantized.onnx         # 143.67 MB Dynamic INT8 ONNX model63  pytorch/64    config.json65    model.safetensors            # 571 MB PyTorch BFloat16 weights66    tokenizer.json67    tokenizer_config.json68scripts/                          # Full fine-tuning, dataset audit & quantization pipeline69```70 71---72 73## 💻 Python / PyTorch Usage74 75```python76import torch77from transformers import AutoTokenizer, AutoModelForSequenceClassification78 79MODEL_NAME = "addyo07/query-scope-classifier"80tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME, subfolder="model/pytorch")81model = AutoModelForSequenceClassification.from_pretrained(MODEL_NAME, subfolder="model/pytorch")82 83labels = ["ChitChat", "User", "Domain", "Temporal"]84query = "aaj sham ko mera schedule kya hai?"85 86inputs = tokenizer(query, return_tensors="pt")87with torch.no_grad():88    logits = model(**inputs).logits89    probs = torch.softmax(logits, dim=-1)90    pred_idx = torch.argmax(probs, dim=-1).item()91 92print(f"Predicted Scope: {labels[pred_idx]} (Confidence: {probs[0][pred_idx].item():.4f})")93```94 95---96 97## ⚡ ONNX Runtime Usage (Fast CPU Inference)98 99```python100import numpy as np101import onnxruntime as ort102from transformers import AutoTokenizer103 104tokenizer = AutoTokenizer.from_pretrained("addyo07/query-scope-classifier", subfolder="model/pytorch")105session = ort.InferenceSession("model/onnx/model_quantized.onnx", providers=["CPUExecutionProvider"])106 107query = "Remind me to submit the quarterly tax report tomorrow at 5pm"108inputs = tokenizer(query, return_tensors="np", max_length=64, truncation=True)109 110onnx_inputs = {111    "input_ids": inputs["input_ids"].astype(np.int64),112    "attention_mask": inputs["attention_mask"].astype(np.int64)113}114outputs = session.run(None, onnx_inputs)115logits = outputs[0][0]116probs = np.exp(logits) / np.sum(np.exp(logits))117pred_id = np.argmax(probs)118 119labels = ["ChitChat", "User", "Domain", "Temporal"]120print(f"Scope: {labels[pred_id]}, Confidence: {probs[pred_id]:.4f}")121```122