Muheet-Mehraj/ai-governance-assessment

AI Governance Research & Assessment Application

Built for the MODUS Enterprise AI Build Challenge — Assignment 7.

What it does

You describe an AI use case (e.g. "an automated loan approval model using credit history and income data"). The application researches its own knowledge base of governance sources (laws, regulatory guidance, industry standards), retrieves the evidence most relevant to each of 10 governance dimensions, and produces a grounded risk assessment for every dimension — with every conclusion traceable back to the exact source text that justified it.

It does not ask an LLM "is this high risk?" and print the answer. Every verdict is retrieval-grounded: if no relevant evidence exists for a dimension, the system reports "Insufficient Evidence" rather than letting the model guess.

System Architecture

The assessment engine decouples ingestion, vector retrieval, and LLM orchestration into discrete layers:

[image]

Entity-Relationship (ER) Data Model

All organizations, regulatory documents, text chunks, use cases, and dimension verdicts are modeled as relational entities linked to vector index positions:

[image]

Why this stack

What happens if Groq becomes unavailable or a model is deprecated?

The LLMClient (app/clients/llm_client.py) tries every model in GROQ_FALLBACK_MODELS in order before raising. If all Groq models fail, the assessment for that dimension is marked "Insufficient Evidence" with an explanation, rather than crashing the whole assessment — the rest of the dimensions still complete. To remove the Groq dependency entirely, implement the same generate() interface against a local Ollama model; nothing else in the app changes.

Setup

# 1. Create and activate a virtual environment
python -m venv venv
source venv/bin/activate  # or venv\Scripts\activate on Windows

# 2. Install dependencies
pip install -r requirements.txt

# 3. Configure environment
cp .env.example .env
# edit .env and set GROQ_API_KEY

# 4. Seed the database with starter governance sources
python -m scripts.seed_data

# 5. Run the application
uvicorn app.main:app --reload --port 8000

Open http://localhost:8000 in your browser.

Known limitation: seed knowledge base is intentionally small

The seed corpus ships with only 3-4 short illustrative source excerpts (see scripts/seed_data.py). This is enough to demonstrate the pipeline end-to-end, but it means retrieval for some dimensions (e.g. "Privacy," for which no seed source substantively discusses data privacy) may surface only tangentially related evidence. The application's evidence-gating logic (see app/services/assessment_service.py) requires the LLM to self-report whether retrieved evidence actually addresses the dimension, and overrides to "Insufficient Evidence" when it doesn't — this is a feature, not a bug: it surfaces knowledge base gaps honestly instead of letting the model paper over them. For a stronger real-world submission, add more sources per dimension (e.g. a GDPR excerpt for Privacy, a NIST bias-testing section for Bias/Fairness) via POST /api/sources/ingest.

Testing the "surprise record" scenario

The application never hard-codes a use case. To test with a brand-new input:

1. 1.Via the UI: fill in a use case title/description and click "Submit & Assess."
2. 2.Via the API directly:

curl -X POST http://localhost:8000/api/use-cases \
  -H "Content-Type: application/json" \
  -d '{"organization_id": 1, "title": "New Use Case", "description": "..."}'

curl -X POST http://localhost:8000/api/use-cases/{id}/assess

Adding a new governance source (e.g. a new regulation) is likewise a data operation:

curl -X POST http://localhost:8000/api/sources/ingest \
  -H "Content-Type: application/json" \
  -d '{"title": "New Regulation", "authority_level": "Law/Regulation", "raw_text": "..."}'

No code changes required for either.

Scaling to 100,000 records

• —Swap SQLite for Postgres (one env var change).
• —Move assessment execution to a background task queue (Celery/RQ) so submissions return immediately and assessments process asynchronously.
• —Batch Groq calls and add caching for repeated/similar use case queries.
• —Move FAISS to a sharded or managed vector index if the source corpus grows beyond what fits comfortably in memory.

Libraries & licenses

All free/open-source; Groq API usage is free-tier, no paid license required.