digopala/ai-inference-architecture-healthcare

AI Inference Architecture for Healthcare

This project provides a scalable, production-ready AI inference architecture designed for healthcare and pharmaceutical applications. It integrates Triton Inference Server, FastAPI, and Kubernetes to support high-throughput model inference.

🚀 Key Features

• —Modular container-based architecture with FastAPI gateway
• —Supports NLP and CV models with optional preprocessing
• —Inference via Triton Inference Server using ONNX or TorchScript models
• —GitHub Actions-powered CI/CD pipeline to auto-deploy model updates
• —Kubernetes-based pod management, autoscaling, and volume mounting
• —Full observability stack: Prometheus + Grafana for metrics and monitoring
• —Compliant with HIPAA-aligned standards: secure APIs, logging, encryption

🧱 Architecture Overview

Healthcare/Pharma Clients → FastAPI Gateway → Optional Preprocessor → Triton Pod
       ↓                        ↓                            ↓             ↓
 Model Registry ← GitHub CI/CD Pipeline ← Kubernetes ← Monitoring (Prometheus + Grafana)

⚙️ Deployment Options

▶️ Local (Docker Compose)

docker compose up --build

☸️ Kubernetes (Production)

kubectl apply -f k8s.yaml
kubectl apply -f preprocessor.yaml
kubectl apply -f hpa.yaml

📦 Model Lifecycle

1. 1.Train model locally or in pipeline (e.g., PyTorch/ONNX)
2. 2.Push model to GitHub repository
3. 3.GitHub Actions CI/CD triggers and pushes model to Model Registry
4. 4.Kubernetes mounts model volume into Triton pod
5. 5.Triton automatically reloads model

🔍 Monitoring and Observability

• —Metrics via Prometheus sidecar scraping port 8002 on Triton pod
• —Dashboards in Grafana track latency, throughput, failures

🧪 Sample Inference Request

curl -X POST http://localhost:8000/infer   -H "Content-Type: application/json"   -d '{"input": "Patient data or image here"}'

Enhancements Based on Peer Technical Review

Preprocessing Execution Model

The NLP/CV preprocessing stage runs as an independent Kubernetes microservice for isolation and scale. The FastAPI Gateway performs conditional routing:

• —content_type=image/* → CV preprocessor → Triton
• —content_type=text/* → NLP preprocessor → Triton
• —Already-normalized inputs → direct to Triton A lightweight schema-validation step remains in the gateway.

Model Lifecycle: Versioning, Promotion, Rollback

• —Models are versioned under /models/<name>/<version> (e.g., /models/ner/1).
• —CI/CD publishes to staging; promotion updates a release tag (e.g., current -> 2) for Triton to hot-reload.
• —Rollback re-points the tag to the last known-good (current -> 1).
• —Supports blue‑green (two deployments, Service selector switch) and canary (small % routed to a second Triton deployment).

Scalability & Resilience

• —HPA scales Triton pods based on CPU (and can extend to latency custom metrics).
• —Readiness/Liveness probes guard rollout and enable auto‑healing.
• —Gateway uses timeouts and retry on transient 5xx. If a pod is Unready, traffic shifts to healthy pods.

Security, Compliance & Audit

• —TLS in transit; optional mTLS inside cluster.
• —OAuth2/JWT at the gateway with per‑route scopes.
• —Audit logs (structured JSON with request_id) across gateway, preprocessors, and Triton; logs ship to ELK/Loki.
• —Optional PHI de‑identification in preprocessors; strict schema validation; data minimization and retention controls aligned to HIPAA/GDPR.

Data Flow & Validation

• —Gateway enforces MIME/JSON schema and rejects malformed/unauthorized requests.
• —Preprocessors normalize inputs (e.g., tokenize text, resize/normalize images).
• —Triton returns prediction JSON; gateway maps to a domain response schema and may redact fields per policy.

📌 See SECURITY.md for detailed security, compliance, and audit logging implementation.

📄 See preprocessor.yaml for deployment details of the NLP/CV preprocessing microservice.

📄 See hpa.yaml for Triton autoscaling configuration.

📂 File Reference

• —k8s.yaml → Triton deployment
• —preprocessor.yaml → NLP/CV preprocessing service
• —hpa.yaml → Horizontal Pod Autoscaler for Triton