sanatem/samtp-mini-traversability

SAM-TP Mini+ Traversability (checkpointfinetunedv2)

Image-space traversability segmentation for the FrodoBots Earth Rover Mini+ front camera. One RGB frame in, per-pixel drivability out.

Model details

• —Architecture: SAM 2.1 image branch, Hiera-tiny backbone (embeddim 96), with the prompt encoder replaced by a learned "traversability prompt" (GeNIE SAM-TP, `CustomPromptEncoderLarger`, `wantcustompromptencoder: 2`). Prompt-free: point/box inputs are ignored; output is deterministic per image.
• —Init: facebook/sam2.1-hiera-tiny
• —Fine-tuning data: ~50k front-camera frames from Earth Rover Mini+ footage (Mini-4K derived) with binary drivable-ground masks. (v1 used ~5k frames; this v2 checkpoint used a larger set with cleaner labels.)
• —File: checkpoint_finetuned_v2.pt — torch save with a single top-level model key holding the state dict. 136,622,641 bytes.
• —sha256: 44e508da3d36a63431f8197f16784c980abf43ea94fc4e524bcd19d0646692bd

Required inference config

This checkpoint ONLY loads against the tiny SAM-TP inference config (sam2/configs/sam2.1_inference_tiny/sam2.1_custom2.yaml in the GeNIE sam2 fork). Loading it with a base+/small/large config — or loading the public GeNIE checkpoint_2.pt (base+) with the tiny config — fails with a state-dict mismatch.

Usage

Via the rover-traversability package (in the team repo under traversability/):

pip install 'rover-traversability[hf]'   # from the repo: pip install -e ./traversability[hf]
python -c "
from rover_traversability import TraversabilityPredictor
p = TraversabilityPredictor()            # auto-downloads this checkpoint
result = p.predict('frame.jpg')
print(result.mask.shape, result.mask.mean())
"

Or manually: download checkpoint_finetuned_v2.pt and set SAMTP_CHECKPOINT=/path/to/checkpoint_finetuned_v2.pt.

Output contract: mask is HxW float32 in [0, 1], 1 = drivable (sigmoid of the raw logits, resized to the input frame size).

Fine-tuning on top

This is a full model state dict — use it directly as the init checkpoint in Meta's SAM2 training harness (training.* from facebookresearch/sam2) with the sam2.1_training_tiny configs from the GeNIE fork (ckpt_state_dict_keys: ['model']). Dataset format: image folder + binary PNG masks (MOSE/PNG-VOS layout). Reference hyperparameters from this checkpoint's training: 1024 res, batch 8, AdamW, baselr 5e-6 / visionlr 3e-6, 5 epochs.

Performance (Apple M-series, 1024x576 input)

Known limitations

• —Trained as "ground vs. above-ground": dark objects sitting on light ground (other rovers, low obstacles) can be labeled drivable. The rover-traversability wrapper applies a per-frame luminance-contrast refinement to mitigate this — keep it enabled.
• —Monocular, image-space only: no metric depth. Pair with camera calibration for BEV projection.

Licensing & provenance

• —SAM 2 / SAM 2.1 base weights and code: Apache-2.0 (Meta Platforms).
• —SAM-TP architecture: GeNIE (Wang, Liu, Chen, et al.).
• —Fine-tuning data: FrodoBots Earth Rover Mini footage. The related public dataset is `BitRobot/FrodoBots-Mini-4K` (CC-BY-SA) — if you redistribute or build on these weights, carry this provenance note and attribution with them.