monradach/parag-on

PARAG-On

A local retrieval-augmented generation (RAG) prototype for making sense of longitudinal, narrative programmatic-assessment data in medical education. Built for a hands-on workshop: the configurable knobs are the point.

Everything runs on the user's machine. Assessment text is never sent to an external service.

\---

What it does

Given a corpus of narrative assessment records (reflective essays, WBA comments, supervisor feedback) tagged by learner, timepoint, rotation, and competency, PARAG-On lets an educator ask a question in plain language and get an evidence-grounded answer — a longitudinal summary, a concern scan, or a trajectory account — with the exact retrieved evidence shown alongside.

The UI presents the pipeline as a clickable workflow diagram — Data → Embedding Model → Retrieval Results → Prompt → Generation — where clicking a component opens its settings (or, for Data, read-only EDA insights) below the diagram, and clicking it again collapses them.

The default data source is data/feedback.xlsx (sheets: feedback, student_info, evaluator_info). The workbook has no explicit dates, so timepoints are derived from each student's contiguous subject blocks in feedback-ID order. Point PARAGON_DATA at another .xlsx workbook with the same sheets, or at a JSON array of AssessmentRecord objects.

The pipeline

records ─▶ chunk ─▶ embed ─▶ index        (build, cached)
question ─▶ filter ─▶ retrieve top-k ─▶ frame prompt ─▶ generate ─▶ answer

Each stage is controlled by a knob on RAGConfig:

Module map

Setup (three tiers)

Tier 0 — runs anywhere, no downloads. Lexical retrieval, no generation. Good for verifying the install and for understanding retrieval in isolation.

pip install numpy pandas openpyxl altair requests streamlit
streamlit run app.py   # pick "hashing" in the Embedding Model panel
                       # (the app also falls back to it automatically if
                       #  sentence-transformers is not installed)

Tier 1 — real semantic retrieval. Adds the MiniLM embedder.

pip install -r requirements.txt   # pulls sentence-transformers + torch
# set embedding backend = "minilm" (default)

Tier 2 — full local RAG with generation. Adds Ollama.

# install Ollama from https://ollama.com, then:
ollama pull llama3.1
streamlit run app.py

The app auto-detects Ollama and lists its installed models in the Generation panel; if it is not running the app shows retrieved evidence without a generated summary.

The Generation panel also offers hosted models out of the box — ministral-3b-2512, ministral-8b-2512 and codestral-2508, all on Mistral's OpenAI-compatible endpoint. One secret (PARAGON\_MISTRAL\_KEY) covers all three. Additional models (e.g. a Gemini one) come from PARAGON\_LLM\_MODELS + PARAGON\_LLM\_BASE\_URL + PARAGON\_LLM\_API\_KEY.

Environment variables

Try this (demo / activity prompts)

With the default workbook, pick a class group in the Retrieval Results panel (e.g. 22XXX), optionally narrow by subject, and ask: "Are there any concerns about these learners across the year?"

• —task\_framing = neutral\_summary reads reassuringly — ratings are acceptable.
• —task\_framing = flag\_concerns surfaces the interpersonal pattern building across Blocks 1–3 that the numbers alone hide.
• —Switch chunk\_strategy to per\_learner and watch retrieval lose the ability to filter by competency or assessment type.
• —Switch embedding\_backend to hashing and watch semantic recall drop.

Tests

python -m tests.test\_pipeline

A note on responsible use

Outputs are decision support. The prompts require the model to ground claims in dated evidence, cite sources, and flag thin evidence, but generated text can still mislead. Keep a human educator in the loop for every judgement.