mohi679/self-healing-llmops

Production-Inspired Self-Healing LLMOps Control Plane for RAG Systems

A production-inspired reliability and self-healing control plane designed to monitor, evaluate, and remediate quality drift in multi-tenant RAG (Retrieval-Augmented Generation) systems. It serves as a comprehensive system design reference for managing the closed-loop reliability of LLM applications—demonstrating real-time evaluation judges, quality drift alarms, clustered failure ticketing, shadow validation, automated canary rollouts, and rollback triggers. This project is designed as a reproducible AI engineering portfolio project, not a deployed enterprise product.

[!IMPORTANT] Scope Disclaimer: This project is NOT a foundation model, does NOT train a frontier model, and does NOT repair model weights. It is a production-inspired LLMOps/RAG control-plane prototype. "Self-healing" refers strictly to evaluation-driven config/retrieval/prompt/rollout remediation (e.g., promoting prompt/retrieval configurations, adjusting top_k, switching inference providers, or rolling back failed deployments). It does not mean automatic model-weight repair or weights fine-tuning.

Live Demo

Check out the live deployment on Hugging Face Spaces: 👉 Hugging Face Space Live Demo

Hugging Face Space Demo Scope

The public Hugging Face Space runs in lightweight demo mode using SQLite, in-memory cache, deterministic evaluation, and local/demo vector storage so reviewers can test the system without external cloud credentials. The repository also includes implemented production-like adapters for PostgreSQL persistence, Redis semantic caching/queueing, and pgvector retrieval, with a Docker Compose stack that wires PostgreSQL + pgvector, Redis, Prometheus, and the FastAPI service.

Self-Healing Definition

In this control plane, "Self-healing" is defined as an evaluation-driven, closed-loop remediation workflow:

1. 1.Evaluation: RAG responses are asynchronously/synchronously evaluated against core quality contracts (faithfulness, relevance, safety, citation grounding).
2. 2.Drift Detection: Sliding-window comparison detects statistically significant quality drops compared to a baseline.
3. 3.Ticketing: Quality drops trigger structured failure tickets (e.g., LOW_FAITHFULNESS).
4. 4.Remediation Proposing: The system generates remediation configuration proposals (e.g., modifying prompt packs or updating retrieval top-k parameters).
5. 5.Shadow Run: The remediation candidate is validated in shadow mode against live/historical query streams.
6. 6.Canary & Promotion / Rollback: A multi-stage canary rollout gates promotion of the configuration. If performance drops below the rollback threshold at any stage, the configuration is automatically rolled back.

5-Minute Reviewer Path

Run the complete deterministic proof with no API key:

# 1. Install dependencies
pip install -e ".[server,dev]"

# 2. Run the self-healing E2E proof
make demo-self-healing

# 3. Verify target tests
make prove-self-healing

Expected Evidence

• —Running make demo-self-healing generates a detailed markdown report at docs/reports/self_healing_e2e_report.md.
• —The final decision in the report will show PASS_WITH_ROLLBACK_PROOF.
• —The execution demonstrates: bad-candidate rejection, unsafe-candidate rejection, invalid-citation rejection, good-candidate promotion, and canary rollback.

For resume/LinkedIn copy, see docs/PORTFOLIO_SUMMARY.md.

Run the Full Proof Locally

To run the full suite of unit and integration tests locally without external keys:

make quick-test

This runs the core test suite (including the self-healing demo validation tests) verifying all the logic and contracts.

For complete local validation of all tests:

python3 -m pytest

Production-like Docker Compose Mode

To run with full production configurations: PostgreSQL (with pgvector), Redis, Prometheus, and the core app in APP_ENV=production. Requires non-weak API secrets.

# 1. Copy and configure your secrets in .env
cp .env.example .env
# Set valid secrets for: PII_HMAC_KEY, OPENAI_API_KEY, VOYAGE_API_KEY...

# 2. Start the full production-like infrastructure
make compose-up

# 3. Tear it down when finished
make clean-artifacts

Once running, you can access individual services and check platform readiness at:

• —App Health Gate: http://localhost:8080/healthz
• —Prometheus Metrics: http://localhost:8080/metrics
• —Prometheus Dashboard: http://localhost:9090

Production Backend Requirements

To run in production mode, you must ensure the following requirements are met:

• —[ ] Set STORE_BACKEND=postgres to route storage to PostgreSQL instead of memory/SQLite.
• —[ ] Set CACHE_BACKEND=redis to enable Redis for semantic caching.
• —[ ] Set VECTOR_STORE=pgvector to enable pgvector-based retrieval.
• —[ ] Configure active secrets in your environment (e.g. POSTGRES_DSN, REDIS_URL, OPENAI_API_KEY).

Architecture

flowchart TD
    subgraph Client Layer
        Client[API Client / User]
    end

    subgraph API & Routing
        API[FastAPI Gateway]
        Cache[Semantic Redis Cache]
    end

    subgraph Core RAG Pipeline
        Inference[Inference Provider]
        RAG[RAG Provider]
        Embed[Embedding Provider]
        VStore[PgVectorStore / Vector DB]
    end

    subgraph Reliability & Self-Healing
        Eval[Evaluator Stack]
        Drift[Canonical Drift Detector]
        Heal[Healing Service]
        Rollout[Rollout & Promotion Controller]
    end

    subgraph Storage & Observability
        DB[(PostgreSQL Store)]
        OTel[OpenTelemetry Tracing]
        Prom[Prometheus Metrics]
    end

    Client -->|Query| API
    API -->|1. Lookup| Cache
    API -->|2. Route| RAG
    RAG -->|Embed Query| Embed
    RAG -->|Search| VStore
    RAG -->|Generate Answer| Inference
    Inference -->|3. Evaluate| Eval
    Eval -->|Save Result| DB
    Eval -->|Trace Span| OTel
    Eval -->|Increment Metrics| Prom
    
    DB -->|Read/Aggregate| Drift
    Drift -->|Trigger Alarm| Heal
    Heal -->|Propose Candidate| Rollout
    Rollout -->|Validation Run| Inference
    Rollout -->|Promote/Rollback| API

See the System Design & Interview Defense Manual for a deep dive into design trade-offs.

What is Real vs Demo/Simulated?

Known Limitations

• —No Real PEFT/LoRA Training: The remediation workflow simulates the lifecycle of proposing model adapters, generating configuration metadata (.json), and tracking validation iterations, but does not train weights.
• —No Gateway Proxy Routing: Traffic splitting during canary rollouts is performed at the application layer inside the FastAPI routing logic rather than at a reverse proxy/load-balancer layer (like Envoy or Nginx).
• —Offline Mocking: The offline mock mode uses canned static string responses and heuristics which do not simulate real semantic nuances of LLMs.

Why This Project Matters for LLMOps

This project bridges the gap between brittle AI wrappers and resilient, self-correcting systems. In production, LLM applications face quality drift, prompt injections, and regression bugs. Instead of relying on manual developer intervention, this control plane demonstrates how we can apply classic control theory and automated release gating to non-deterministic LLM pipelines, guaranteeing safety and quality ceilings.

Resume Bullet

• —Built a Python/FastAPI LLMOps reliability control plane for multi-tenant RAG systems, implementing tenant-aware retrieval, sliding-window drift detection, failure clustering, shadow validation runs, and guarded canary rollouts with auto-rollback, integrated with OpenTelemetry/Prometheus observability and pgvector.

LinkedIn Post

🚀 I built a self-healing LLMOps/RAG reliability control plane that demonstrates how production AI systems can detect quality drift and safely remediate it. 

The system:
1. Asynchronously evaluates RAG answers for faithfulness, relevance, safety, retrieval recall, and citation grounding.
2. Opens structured healing tickets when quality drops.
3. Proposes config/retrieval remediation candidates.
4. Validates candidates in shadow mode.
5. Employs canary rollout gates to promote good candidates or auto-rollback degraded ones.

The project is intentionally honest: it heals configuration, retrieval, prompt, and rollout state (not model weights). 

Check out the repo here: [repo-url]
#LLMOps #RAG #AI #Reliability #SystemDesign