chrischarts8/pcare-dim4-phase2-comparator
07
PCare DIM4 Phase 2 — Clinical Outcome Comparator
Fine-tuned Qwen3-8B with LoRA (r=256, alpha=256, all-linear) for binary clinical outcome comparison (PASS/FAIL).
Task
Compare an expected clinical outcome (from document analysis) against an actual clinical outcome (from a compiled FHIR artifact) and judge semantic equivalence.
Input: Two clinical outcome strings Output: JSON with verdict (PASS/FAIL), reasoning, and mismatches
Evaluation Results
All 4 primary targets met. The model achieves perfect accuracy with zero safety misses on the 299-example eval set spanning 8 clinical domains and 15 scenario types.
Confusion Matrix (Fine-tuned)
Training Loss Trajectory
Training Details
- Base model: Qwen/Qwen3-8B (Apache 2.0)
- Method: SFT with LoRA (r=256, alpha=256, all-linear targets, dropout=0.05)
- Dataset: chrischarts8/pcare-dim4-phase2-synthetic — 1,196 train / 299 eval examples
- Epochs: 3
- Effective batch size: 32 (4 × 4 grad accum × 2 GPUs)
- Learning rate: 2e-4 (cosine schedule, 10% warmup)
- Loss: NLL with
assistant_only_loss=True - Hardware: 2×A10G (48GB total)
- Trainable parameters: 698M / 8.9B total (7.86%)
- Training time: ~1.5 hours
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-8B", dtype="bfloat16", device_map="auto")
model = PeftModel.from_pretrained(base, "chrischarts8/pcare-dim4-phase2-comparator")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-8B")
system_prompt = """You are a clinical outcome comparator. Compare two sets of clinical outcomes for the same patient. Outcomes may be recommendations, diagnostic classifications, risk levels, or other clinical decisions.
## Rules
1. Different clinical outcomes = FAIL (even if both are 'reasonable').
2. Same clinical outcome with different wording = PASS.
3. One outcome is more specific than the other but clinically consistent = PASS.
4. Missing an outcome domain that the expected has = FAIL.
5. Extra outcome domains not in the expected are acceptable = PASS.
6. If the expected says 'not indicated' / 'excluded' and actual says the equivalent = PASS.
## Output Format
Return ONLY a JSON object:
{"verdict": "PASS" or "FAIL", "reasoning": "brief explanation", "mismatches": ["list of mismatches, empty if PASS"]}"""
messages = [
{"role": "system", "content": system_prompt},
{"role": "user", "content": "## Expected Outcome (from document analysis)\nScreen for colorectal cancer with colonoscopy every 10 years, starting at age 45\n\n## Actual Outcome (from compiled artifact)\n[Colorectal cancer screening] Colonoscopy every 10 years — ages 45-75 (Grade A)"},
]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True, enable_thinking=False)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=256, do_sample=False)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
# {"verdict": "PASS", "reasoning": "Both outcomes recommend colorectal cancer screening with colonoscopy every 10 years starting at age 45. The actual is more specific (adds age range and Grade A) but clinically consistent.", "mismatches": []}Clinical Domains Covered
- Cancer screening — colorectal, breast, cervical, lung, prostate
- Diagnostic classification — transfusion reactions (TRALI/TACO), sepsis (qSOFA/SOFA), stroke (TOAST)
- Treatment guidelines — antibiotics, anticoagulation, pain management
- Risk stratification — cardiovascular (Framingham, ASCVD), fall risk, nutrition
- Immunization schedules — childhood, adult, travel vaccines
- Preventive care — wellness visit components, lifestyle counseling
- Referral criteria — specialist referral thresholds
- Monitoring protocols — lab monitoring frequency, imaging follow-up