Modelpublic

brijeshvadi/mcp-error-classifier

sourceHugging Facemitupdated 6mo agoView on Hugging Face

0likes

Model Card

MCP Error Classifier

A fine-tuned text classification model that detects and categorizes MCP (Model Context Protocol) tool-calling errors in AI assistant responses.

Model Description

This model classifies AI assistant tool-calling behavior into 5 error categories identified during QA testing of Grok's MCP connector integrations:

Label	Description	Training Samples
`CORRECT`	Tool invoked correctly with proper parameters	2,847
`TOOL_BYPASS`	Model answered from training data instead of invoking the tool	1,203
`FALSE_SUCCESS`	Model claimed success but tool was never called	892
`HALLUCINATION`	Model fabricated tool response data	756
`BROKEN_CHAIN`	Multi-step workflow failed mid-chain	441
`STALE_DATA`	Tool called but returned outdated cached results	312

Training Details

—Base Model: distilbert-base-uncased
—Training Data: 6,451 labeled MCP interaction logs across 12 platforms
—Platforms Tested: Supabase, Notion, Miro, Vercel, Netlify, Canva, Linear, GitHub, Box, Slack, Google Drive, Jotform
—Epochs: 5
—Learning Rate: 2e-5
—Batch Size: 32

Usage

python

from transformers import pipeline

classifier = pipeline("text-classification", model="brijeshvadi/mcp-error-classifier")

result = classifier("Grok responded with project details but never called the Supabase list_projects tool")
# Output: [{'label': 'TOOL_BYPASS', 'score': 0.94}]

Intended Use

—QA evaluation of AI assistants' MCP tool-calling reliability
—Automated error categorization in MCP testing pipelines
—Benchmarking tool-use accuracy across different LLM providers

Limitations

—Trained primarily on Grok interaction logs; may underperform on Claude/ChatGPT patterns
—English only
—Requires context about which tool was expected vs. what was called

Citation

bibtex

@misc{mcp-error-classifier-2026,
  author = {Brijesh Vadi},
  title = {MCP Error Classifier: Detecting Tool-Calling Failures in AI Assistants},
  year = {2026},
  publisher = {Hugging Face},
}