MIS-752/James_Seegel_lab04-gp-vs-specialist

Lab04 James Seegel | Get a Second Opinion

MIS 752, University of Nevada, Las Vegas. Recorded September 20, 2026.

A small free model and a large free model answered the same five clinical questions. The GP received full credit under the keyword rubric on 3/5 questions (60%); the specialist received 5/5 (100%). Average recorded response times were 3.87 and 39.62 seconds. Both had $0 model API cost in this experiment.

The uncertainty-based cascade escalated 0/5 questions and stayed at 60%. It flagged neither of the GP's two rubric failures, giving 0% uncertainty-flag recall. The oracle chart is explicitly a hypothetical replay with advance knowledge of rubric scores.

The five fictional supply questions connected to James Seegel's supply systems experience produced 5/5 rubric passes for both models. Recorded mean response times were 2.59 seconds for the GP and 6.69 seconds for the specialist. Both correctly selected Supplier A at $1,190 over Supplier B at $1,195.

Takeaways

My small model earned full credit on 60% of the five questions, compared with 100% for the specialist, a 40-percentage-point gap. Both passed the cardiology guideline rubric, suggesting that listing standard treatment classes was manageable for the smaller model in this example. Both cost $0; I would pay only if the extra cost per independently verified correct answer was less than the cost of reviewing or correcting mistakes. The cascade stayed at 60% because it escalated nothing and detected 0% of the two rubric failures. I would require external checks and human review instead of trusting the model to admit uncertainty. The drug-interaction answer also passed despite contradictory claims about anticoagulation risk, showing that keyword scores can miss errors. At temperature 1.2, the local model gave an unsupported six-month rule for stopping medication and reached its 120-token limit mid-sentence. I skipped the optional paid comparison.

Evidence and limitations

The completed notebook is included byte-for-byte from the supplied completed version. The page's results download contains measurements, rubrics, and printed examples extracted from saved notebook outputs. The original full-response CSV and API cache were not provided with these notebook uploads. Missing raw responses were not reconstructed, and no model calls were rerun for the showcase.

Keyword matches do not establish clinical correctness. The GP's drug-interaction response passed despite an incorrect low-INR claim. MedlinePlus explains the distinction between low-INR clot risk and high-INR bleeding risk. The clinical vignettes and supply exercises are fictional educational examples. The specialist's five rubric passes do not establish deployment safety.

The temperature view uses the final recorded Part A sweep in the notebook (temperatures 0.1, 0.7, and 1.2). All response times are from that recorded experiment, not from loading this page. Current model availability and pricing may differ.

AI use disclosure

I used ChatGPT/Codex to help write, debug, and explain the temperature sweep, rubric scorer, benchmark, cascade, routing function, and supply-question extension, and to draft the Takeaways and reflection responses. Verification included executing the code in Colab and reviewing my actual outputs with the assistant to check the reported scores, timings, and costs. The assistant also consulted MedlinePlus to assess the warfarin response and identified an incorrect clinical claim that the keyword rubric had passed. Changes made with AI assistance included correcting the cascade cost calculation to count both calls when escalation occurs and revising a chart caption that misrepresented the 0% escalation result. These checks support the notebook’s calculations but do not constitute clinical validation. My own work included setting up the accounts and secrets, running the notebook, and collecting the outputs; the code and written interpretation received substantial AI assistance.

During final review, the assistant added a missing import and removed an unused starter function.

ChatGPT/Codex also assembled this static showcase, extracted its displayed measurements, and redrew the two charts from recorded coordinates. It made no new model calls.

Publishing

Create a Hugging Face Space under your own account with the Static SDK. Upload these three files at the repository root: README.md, index.html, and Lab04JamesSeegelCompleted.ipynb. No API keys or model service are required to display this package. See Hugging Face's static Space documentation.