skew202/nerfstatus-hf-monitor

NerfStatus: HuggingFace Inference Quality Monitor

Scientific LLM degradation detection — Real-time monitoring of HF inference endpoints powered by nerfprobe-core.

About

NerfStatus detects silent model collapse, quantization artifacts, and instruction failures using scientifically-grounded probes. This Space brings that capability to HuggingFace models via Inference Providers (Together, Cerebras, etc.).

The 4 Core Probes

Supported Models (with Auto-Provider Selection)

The Space automatically routes each model to the best available Inference Provider:

You can also test any HF model by entering its ID directly.

Setup

For HF Inference (Bring Your Own Token)

💡 Bring Your Own Token - Test any HF model at YOUR own cost:

1. 1.Get a token at https://huggingface.co/settings/tokens
2. 2.Enable Inference Providers permission
3. 3.Paste the token in the "API Settings" accordion in the Space
4. 4.Enter any HF model ID (e.g., meta-llama/Llama-3.1-70B-Instruct)
5. 5.Costs are billed to YOUR account, not the Space author

Pricing (approximate per 1M tokens via Inference Providers):

• —Small (1-3B): $0.02-0.06
• —Medium (7-8B): $0.03-0.30
• —Large (70B+): $0.40-0.88

Each test run uses ~300 tokens (4 probes × ~75 tokens).

For Ollama (local development) - RECOMMENDED

Free and unlimited - use Ollama locally for development and testing:

1. 1.Install Ollama: https://ollama.com
2. 2.Run a model: ollama run llama3.2
3. 3.Refresh the Space page - it will auto-detect Ollama

Ollama models tested:

• —llama3.2:3b - 3B parameters, fast
• —llama3.1:8b-instruct-q4_K_M - 8B quantized, good quality
• —qwen3:4b - 4B parameters, efficient
• —qwen3:8b - 8B parameters, strong performance
• —gemma2:9b - 9B parameters, excellent quality

How to Use

1. 1.Optional: Paste your HF token in "API Settings" to test HF models (at your cost)
2. 2.Enter any model ID: HF format (org/model) or Ollama format (model:tag)
3. 3.Or use Quick Select buttons for popular HF models with pre-configured providers
4. 4.Click "Run All Probes" to test all 4 quality checks
5. 5.View results: pass rate, latency, detailed breakdown

Interpretation

• —✅ ≥80%: Healthy model
• —⚠️ 50-79%: Degraded, use with caution
• —❌ <50%: Failing, not recommended for production

Links

• —📊 Live Dashboard
• —📦 pip install nerfprobe-core
• —🐙 GitHub

Technical Details

This Space is built on nerfprobe-core, a scientific framework for LLM quality monitoring. The probes are implemented using:

• —Pydantic models for structured data (ModelTarget, ProbeResult)
• —AST parsing for code validation
• —JSON schema validation for structured output
• —Text matching for factual verification

Research Background

NerfStatus is grounded in academic research on LLM quality monitoring and model collapse detection:

• —Neural Probe-Based Hallucination Detection — Liang et al., 2025
• —A Survey on Hallucination in LLMs — ACM Computing Surveys, 2025
• —AI Models Collapse: The Shumailov et al. Paper — Nature, 2024 (858+ citations)

Built by: Stefan Wiest Part of: NerfStatus — Scientific LLM quality monitoring