Modelpublic

SnehShah/house-md-grpo-optimized-gemma3-4b-v3

sourceHugging Faceapache-2.0updated 5mo agoView on Hugging Face

0likes8downloads

Model Card

🩺 House M.D. — GRPO (Gemma 3 4B-IT, optimized v3)

A LoRA adapter trained with GRPO (Group-Relative Policy Optimization) on top of the SFT warm-start `SnehShah/house-md-sft-gemma3-4b`, using episode rollouts against the live OpenEnv Space `SnehShah/house-md-env`.

This is the production submission for the Apr '26 Meta OpenEnv Hackathon — the model whose reward curve is published on W&B and whose eval JSON is included in the repo.

GitHub repo (training pipeline, notebooks, eval, blog): <https://github.com/sneh2909/Overfitters> Live env: <https://huggingface.co/spaces/SnehShah/house-md-env> W&B run (full reward, gradients, mid-eval): <https://wandb.ai/sneh2909-christ-university/house-md?nw=nwusersneh2909> Frozen comparison evals: `SnehShah/house-md-results`

How it was trained


Base model	`unsloth/gemma-3-4b-it-unsloth-bnb-4bit`
Warm start	LoRA from `SnehShah/house-md-sft-gemma3-4b`
Method	TRL `GRPOTrainer` (group-relative advantage; KL regularizer to ref)
Reward fn	sum of 5 rubrics surfaced by the Space's `/step`
Group size	8 rollouts per prompt
Adapter	LoRA, r=32, alpha=64, target=all attn + MLP
Steps	~150 (see W&B for the exact step count)
Optimizer	AdamW 8-bit, LR=5e-6
Hardware	HF Jobs L4 ×1

Training script: `scripts/train_grpo_optimized.py`. Reproduction notebook: `notebooks/03_grpo.ipynb`.

Reward design

Rubric	Range	Captures
`r1_accuracy`	-2 – +1	Right disease and saw the necessary evidence
`r2_cost`	-1.5 – +1	Sweet-spot $200–500; large penalties at $1500+
`r6_anchoring`	-0.5 – +0.6	Did the agent revise its differential meaningfully
`r7_safety`	-2 – 0	Penalty-only; severity-scaled wrong dx / timeouts
`r8_format`	0 – 1	Fraction of valid (in-vocab, well-formed) actions

Default composite weights: {r1: 2.0, r2: 0.5, r6: 0.3, r7: 1.0, r8: 0.5}.

Use

python

from unsloth import FastLanguageModel

model, tok = FastLanguageModel.from_pretrained(
    model_name = "unsloth/gemma-3-4b-it-unsloth-bnb-4bit",
    max_seq_length = 4096,
    load_in_4bit = True,
)
model.load_adapter("SnehShah/house-md-grpo-optimized-gemma3-4b-v3")
FastLanguageModel.for_inference(model)

from house_md_env import HouseMDEnv, HouseMDAction
with HouseMDEnv(base_url="https://snehshah-house-md-env.hf.space") as env:
    res = env.reset(seed=0)
    # ... build prompt from res.observation, generate, parse JSON action,
    #     env.step(...) until res.observation.terminal

The exact eval harness is `scripts/eval_hf.py`; the comparison plot lives in `notebooks/04_eval_compare.ipynb`.

License

Apache 2.0. The base model carries the Gemma terms of use; this adapter is bound by them too.