medelharchaoui/gemma4-12b-browser-agent-run14
Gemma 4 12B — Web Browser Agent (Run 14)
A LoRA adapter fine-tuned on top of Google's QAT Gemma 4 12B for web navigation action prediction. Given a screenshot of a browser and a natural-language task, the model predicts the next action to take (click, type, scroll, navigate, etc.) with chain-of-thought reasoning.
Model Details
Training Data
Fine-tuned on allenai/MolmoWeb-HumanTrajs — indices 0–1999 (~2 000 trajectories, ~34 K steps). Indices 2000–2019 were held out for evaluation (never seen during training).
Each training step: screenshot → action prediction with teacher-forced prior-action history (last 10 steps).
Action Space
mouse_click(x, y, button='left') — click at pixel (x, y)
keyboard_type(text) — type into focused field
scroll(delta_x, delta_y) — scroll by pixel delta
scroll_at(x, y, delta_x, delta_y) — scroll at position (x, y)
goto(url) — navigate to URL
go_back() — browser back
keyboard_press(key) — press a key (Enter, Tab, Escape…)
new_tab() — open new tab
tab_focus(index) — switch to tab by index
send_msg_to_user(message) — report result to user (terminal action)Coordinates are pixel values relative to a 1920×1200 viewport.
Evaluation Results
Evaluated on 20 held-out trajectories (314 steps) from MolmoWeb-HumanTrajs (indices 2000–2019) using teacher-forced ground-truth history (mirrors training format).
Action match +27 pp is the headline result — the model learned the action vocabulary well. Coordinate regression (−29 pp) is the main open issue: the fine-tuned model produces fewer click/scroll steps and more out-of-range coordinates; this is the next thing to fix.
Full per-step results are in eval_run14_results.json in this repository.
How to Use
import torch
from peft import PeftModel
from transformers import AutoProcessor, AutoModelForImageTextToText
from PIL import Image
BASE = "google/gemma-4-12B-it-qat-q4_0-unquantized"
ADAPTER = "medelharchaoui/gemma4-12b-browser-agent-run14"
processor = AutoProcessor.from_pretrained(BASE)
base = AutoModelForImageTextToText.from_pretrained(
BASE, torch_dtype=torch.bfloat16, device_map="auto"
)
model = PeftModel.from_pretrained(base, ADAPTER)
model.eval()
# screenshot: PIL.Image of the current browser state (1920×1200 recommended)
screenshot = Image.open("screenshot.png").convert("RGB")
task = "Go to https://www.google.com/ and search for 'latest AI news'"
SYSTEM_PROMPT = (
"You are a web navigation agent. You see a screenshot of the current browser state "
"and a task to complete. Think through what to do next, then output exactly one action.\n\n"
"Available actions (coordinates are pixel values):\n"
" mouse_click(x, y, button='left')\n"
" keyboard_type(text)\n"
" scroll(delta_x, delta_y)\n"
" scroll_at(x, y, delta_x, delta_y)\n"
" goto(url)\n"
" go_back()\n"
" keyboard_press(key)\n"
" new_tab()\n"
" tab_focus(index)\n"
" send_msg_to_user(message)\n\n"
"When provided, 'Previous actions' lists what you have already done in this task."
)
messages = [
{"role": "system", "content": [{"type": "text", "text": SYSTEM_PROMPT}]},
{"role": "user", "content": [
{"type": "image", "image": screenshot},
{"type": "text", "text": f"Task: {task}"},
]},
]
prompt = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(
text=prompt, images=[[screenshot]],
return_tensors="pt", truncation=True, max_length=3072,
).to(model.device)
with torch.inference_mode():
out = model.generate(**inputs, max_new_tokens=128, do_sample=False)
response = processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)
print(response)
# Example output:
# I can see the browser is open. I will navigate to Google.com to start the task.
# goto(url='https://www.google.com/')Limitations
- Coordinate precision: the model predicts pixel coords for clicks/scrolls but with non-trivial error (~300 px avg L2 on held-out data). Suitable for coarse navigation; precise element targeting needs improvement.
- Teacher-forced history at eval: results above assume ground-truth prior actions are injected in the prompt. In a real rollout (auto-regressive history), error accumulates over trajectory steps.
- Domain coverage: trained on MolmoWeb-HumanTrajs which covers a mix of popular websites. Performance may degrade on niche or highly dynamic UIs.
- Language: English-only tasks tested.
Reproduce / Continue Training
Training code and eval scripts: OptimiAI/Gemma-browser (private repo).
To resume from this checkpoint:
# in src/train.py, set:
ADAPTER_PATH = "medelharchaoui/gemma4-12b-browser-agent-run14"
# and pass resume_from_checkpoint=ADAPTER_PATH to SFTTrainer