samwell/galamsey-unified-v4-1

galamsey-unified-v4-1

Single-model unified VLM for bandwidth-aware satellite tasking AND on-ground analyst review, fine-tuned from `samwell/galamsey-v9-e3` (which itself is a perception fine-tune of `LiquidAI/LFM2.5-VL-450M`). One weight set produces three outputs from three different prompts:

• —{"action": ...} for the on-orbit policy decision
• —bounding-box JSON for grounding
• —free-form text for scene description

Used as the unified policy in **GalamseyWatch**. Trained as the multitask follow-up to `samwell/galamsey-unified-v3`: same base, same LoRA hyperparams, but a multitask training mixture that lets a single 450 M model serve all three jobs without losing v9-e3's perception ability or v3's action accuracy.

What it does

Given a Sentinel-2 patch over southwestern Ghana (RGB + SWIR composites) plus a scalar context block, the model can be prompted three different ways:

In the GalamseyWatch deployment, the action prompt runs on orbit (cheap, 1 forward pass over a 450M model) and the grounding/description prompts run on the ground for the small subset of tiles the policy chose to downlink.

Why a unified, multitask model

The earlier two-layer architecture used a 450 M perception VLM (galamsey-v9-e3) to emit boxes + a scene description, then a 2.6 B text-only LFM2 policy read those plus the scalar context and picked the action. The text description between the two layers is a real bottleneck - visual cues that the perception VLM doesn't surface in prose can't reach the policy.

galamsey-unified-v3 collapsed both jobs into a single 450 M model with one LoRA on top of v9-e3, with action-only target. That gave +11.1 pp over the strongest baseline at 6.8 x fewer parameters, but it had a wart: training on action-only target partially overwrote v9-e3's perception ability, so the on-ground analyst-review step still needed a separate v9-e3 forward pass.

galamsey-unified-v4-1 is the same recipe with a multitask data mixture that re-introduces a small, regulariser-sized dose of perception examples during the LoRA fine-tune. Result: perception is preserved at v9-e3 quality, action accuracy slightly exceeds v3, one weight set covers everything.

Results

Action eval (99-tile held-out set, expanded)

+12.1 pp over the strongest baseline at 6.8 x fewer parameters. Per-class recall on the 99-tile eval:

The 78 pp flag_for_review recall gap (0.89 vs 0.11) is the cleanest single-class evidence of the architectural advantage.

Perception eval (100-tile head-to-head vs galamsey-v9-e3)

v4.1 retains v9-e3's perception ability within noise on the same 100-tile sample drawn from the v9 grounding/description eval JSONLs.

Honest ceiling

• —`request_higher_resolution` and `request_neighbor_tile` sit at 0 % recall. Only 2 hires examples in the train set (oversampled to 80 by repetition, which the model memorised rather than generalised); zero neighbor examples. These two actions need deliberate hand-construction.
• —`downlink_now` recall is 0.48 (down from v3's 0.62). The multitask mixture makes the model slightly more cautious about emitting downlink_now because it now also sees perception examples that disambiguate "bright soil != necessarily mining". On a high-bandwidth pass this trades favourably (fewer wasted downlinks); on a low-bandwidth pass it could miss real signals. For deployments that prioritise downlink recall over per-pass economy, prefer v3.
• —Eval set is same-distribution. The 99 held-out tiles come from the same 15 hand-curated AOIs as the training set; we have not yet evaluated on disjoint AOIs.
• —Single training run, no hyperparameter sweep. Mixture ratio (327 action / 125 grounding / 125 description) was the second guess after a 1000-perception run that drowned the action signal. Likely room for refinement.
• —GRPO post-SFT remains a documented follow-up with three identified trl integration gaps; not in this checkpoint.

Training recipe

• —Base: samwell/galamsey-v9-e3 (full fine-tune of LFM2.5-VL-450M on SmallMinesDS for galamsey perception, 4 x D4 augmentation).
• —Adapter: LoRA, r = 16, α = 32, on q_proj, k_proj, v_proj, out_proj, in_proj, plus vision-tower fc1/fc2 and multimodal projector.
• —Data: 577 mixed rows total - 327 action examples (the same v2 oversampled corpus used by v3), plus 125 grounding rows and 125 description rows subsampled from the v9 perception SFT corpus. Action share 56.7 %. The perception subset acts as a regulariser keeping the LoRA from overwriting v9-e3's perception knowledge - we don't need to re-teach it from scratch, just remind it.
• —Target: task-appropriate per row (action JSON, grounding JSON, or free-form description).
• —Optim: AdamW with cosine LR schedule, peak 2e-5, warmup 5 %, 15 epochs, batch 4 x grad-accum 2, bf16. Training ran ~13 minutes on Modal H100.
• —Config + scripts: `training/configs/galamsey_unified_v4_1_multitask_modal.yaml` and `training/scripts/build_unified_v4_1_multitask_dataset_modal.py`.

Quickstart - all three prompts

from transformers import AutoModelForImageTextToText, AutoProcessor
from PIL import Image
import json

model = AutoModelForImageTextToText.from_pretrained(
    "samwell/galamsey-unified-v4-1",
    torch_dtype="bfloat16",
    trust_remote_code=True,
).cuda().eval()
processor = AutoProcessor.from_pretrained(
    "samwell/galamsey-unified-v4-1", trust_remote_code=True,
)

rgb = Image.open("rgb.png").convert("RGB")
swir = Image.open("swir.png").convert("RGB")

def run(messages, max_new_tokens=128):
    inputs = processor.apply_chat_template(
        [messages], tokenize=True, return_dict=True, return_tensors="pt",
        add_generation_prompt=True,
    )
    inputs = {k: v.cuda() for k, v in inputs.items() if v is not None}
    out = model.generate(**inputs, max_new_tokens=max_new_tokens, do_sample=False)
    return processor.tokenizer.decode(
        out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True,
    ).strip()

# 1) Action prompt (the on-orbit decision)
system_prompt = "You are an on-orbit Earth-observation policy adjudicator. ..."  # see repo
user_text = "Tile u0001 at lon=-2.75, lat=5.64. Cloud cover: 0.001. Pass budget: 320 of 512 KB remaining. ..."
action_msgs = [
    {"role": "system", "content": [{"type": "text", "text": system_prompt}]},
    {"role": "user", "content": [
        {"type": "image", "image": rgb},
        {"type": "image", "image": swir},
        {"type": "text", "text": user_text},
    ]},
]
action_text = run(action_msgs, max_new_tokens=32)
action = json.loads(action_text[action_text.find("{"):action_text.rfind("}")+1])["action"]
print("ACTION:", action)

# 2) Grounding prompt (boxes for analyst review)
grounding_prompt = (
    "You are viewing two images of the same Sentinel-2 patch: a natural-color RGB "
    "composite and a SWIR false-color composite. Detect any illegal small-scale gold "
    'mining pits. Provide result as a valid JSON: [{"label": str, "bbox": [x1,y1,x2,y2]}, ...]. '
    "Coordinates must be normalized to 0-1. If no pits are visible, return []."
)
grounding_msgs = [{"role": "user", "content": [
    {"type": "image", "image": rgb},
    {"type": "image", "image": swir},
    {"type": "text", "text": grounding_prompt},
]}]
print("BOXES:", run(grounding_msgs, max_new_tokens=512))

# 3) Description prompt (free-form scene summary)
description_prompt = (
    "You are analyzing two views of the same Sentinel-2 patch of southwestern Ghana. "
    "Describe any signs of illegal small-scale gold mining (galamsey) activity: "
    "exposed soil, excavation pits, sediment plumes, vegetation loss, proximity to water. "
    "If no mining is visible, say so."
)
desc_msgs = [{"role": "user", "content": [
    {"type": "image", "image": rgb},
    {"type": "image", "image": swir},
    {"type": "text", "text": description_prompt},
]}]
print("DESCRIPTION:", run(desc_msgs, max_new_tokens=128))

Related artifacts

• —Predecessor (action-only): `samwell/galamsey-unified-v3` - same base, same LoRA, action-only target. Slightly higher downlink_now recall, no perception preserved.
• —Perception base: `samwell/galamsey-v9-e3` - the perception fine-tune this model stacks on.
• —Browser/WebGPU sibling of the perception model: `samwell/galamsey-v9-e3-onnx`
• —Training data: `samwell/galamsey-unified-decisions` - 250 hand-labeled Sentinel-2 tiles + scalar context + 5-action targets
• —Repo: `samadon1/GalamseyWatch`
• —Live demo (perception only, browser/WebGPU): galamseywatch.vercel.app

License

LFM Open License v1.0, inherited from the base model.

Citation

@misc{galamseywatch2026,
  author = {Donkor, Samuel},
  title = {GalamseyWatch: agentic Earth observation for galamsey detection},
  year = {2026},
  publisher = {GitHub},
  url = {https://github.com/samadon1/GalamseyWatch}
}