Esvanth/medical-llm-reliability-en-hi

Accurate but Unreliable? Run-to-Run Variance of Medically Trained LLMs on Hindi Medical QA

Interactive companion app for the study applying the Kim et al. (2025) 10-run / Fleiss' κ / Krippendorff's α reliability framework to three medical LLMs on 100 stratified USMLE questions, comparing run-to-run consistency between English and Hindi prompting.

Authors: Adith Kattuparambil Biji · Hemant Singh · Amod Aher · Shrestha Goyal · Esvanth Mohankumar

What this app shows

• —Overview — research question and headline finding
• —Headline Results — reliability / accuracy table with significance tests
• —Figures — 6 publication figures with captions
• —Data Explorer — slice the 6,000-row response dataset by model × language
• —Per-Question Inspector — pick a question, see all 10-run responses in both languages colour-coded against ground truth
• —Translation Quality — per-question BLEU / chrF / COMET back-translation scores
• —Methodology — model stack, methodological decisions, limitations, references

Models

Headline result

Open-source 8B medical LLMs show a significant reliability collapse in Hindi (κ drops 0.23–0.25 points, falling below the κ = 0.67 clinical threshold). Gemini 2.5 Flash-Lite maintains reliability across both languages (κ drop = 0.03, n.s.).

Live Gemini inference (optional)

Set a GEMINI_API_KEY secret on this Space to make Gemini-2.5-Flash-Lite run as live inference (instead of replaying the 10 stored runs):

1. 1.Get a key at https://aistudio.google.com/apikey (free tier works).
2. 2.Set a spend cap in Google AI Studio — visitors can rack up calls otherwise.
3. 3.In the Space: Settings → Variables and secrets → New secret
4. 4.Name: GEMINI_API_KEY
5. 5.Value: your AIza… key
6. 6.Restart the Space.

With the secret set, the "Live Simulation" page calls Gemini live on each Run, and the "Ask Gemini Live" page lets visitors submit their own medical MCQs. Aloe-Beta-8B and Meditron3-7B stay in replay mode (they need GPU, not available on a free CPU Space).