BiliSakura/SSL4EO-S12-transformers

SSL4EO-S12 Transformers Models

Hugging Face–compatible checkpoints converted from the official SSL4EO-S12 pretrained weights. Each subfolder is a standalone model repo layout (config.json, model.safetensors, preprocessor, and optional remote code) for feature extraction on Earth observation imagery.

Model Description

These models are self-supervised encoders pretrained on the SSL4EO-S12 dataset: a large-scale multimodal, multitemporal corpus of Sentinel-1 SAR and Sentinel-2 multispectral patches from 251k+ global locations.

This collection bundles 16 converted checkpoints spanning:

• —Architectures: ViT (S/B/L/H) and ResNet18/50
• —SSL methods: MAE, MoCo, DINO, Data2vec
• —Input modalities: S2-L1C 13-band (s2c), S1 SAR 2-band (s1), S2 RGB 3-band (rgb)

ViT MAE/MoCo/DINO and all ResNet folders ship self-contained remote code (modeling_*.py, processor, pipeline) and load with trust_remote_code=True. The Data2vec folder currently provides weights + config only.

Developed by: zhu-xlab / SSL4EO-S12 Converted for Hugging Face by: BiliSakura License (weights): CC-BY-4.0 Original paper: SSL4EO-S12: A Large-Scale Multi-Modal, Multi-Temporal Dataset for Self-Supervised Learning in Earth Observation

Available checkpoints (16 models)

Legacy .pth filename mapping is in `conversion_manifest.json`.

Intended use

• —Unsupervised / self-supervised feature extraction on Sentinel-1 or Sentinel-2 patches
• —Linear probing or fine-tuning for EO downstream tasks (classification, segmentation, change detection)
• —Research baselines comparable to the original SSL4EO-S12 benchmark

Out-of-scope use

• —Not trained for generative tasks, captioning, or general natural-image applications
• —Not a drop-in replacement for ImageNet-pretrained models on RGB natural scenes
• —Band count and preprocessing must match the checkpoint modality (num_channels in config.json)

Usage

ViT (self-contained remote code)

Processors default to `do_resize: false`. Pass native (H, W, C) patches; spatial token count scales with input size for ViT and ResNet backbones.

from transformers import pipeline
import numpy as np

REPO = "BiliSakura/SSL4EO-S12-transformers"
SUBFOLDER = "ssl4eo-vit-base-patch16-s2c-mae"

pipe = pipeline(
    task="ssl4eo-feature-extraction",
    model=REPO,
    trust_remote_code=True,
    model_kwargs={"subfolder": SUBFOLDER},
)

# S2-L1C: 13 bands at native size (e.g. 512×512)
image = np.random.randint(0, 255, (512, 512, 13), dtype=np.uint8)
features = pipe(image, pool=True, return_tensors=True)
print(features.shape)  # [1, hidden_size]

Opt in to 224×224 resize:

features = pipe(image, pool=True, return_tensors=True, image_processor_kwargs={"do_resize": True})

Load components directly:

from transformers import AutoModel, AutoImageProcessor

model = AutoModel.from_pretrained(REPO, subfolder=SUBFOLDER, trust_remote_code=True)
processor = AutoImageProcessor.from_pretrained(REPO, subfolder=SUBFOLDER, trust_remote_code=True)

ResNet (self-contained remote code)

from transformers import pipeline
import numpy as np

pipe = pipeline(
    task="ssl4eo-feature-extraction",
    model="BiliSakura/SSL4EO-S12-transformers",
    trust_remote_code=True,
    model_kwargs={"subfolder": "ssl4eo-resnet50-s2c-moco"},
)

image = np.random.randint(0, 255, (512, 512, 13), dtype=np.uint8)
features = pipe(image, pool=True, return_tensors=True)
print(features.shape)  # [1, 2048]

Or load via the `ssl4eo` package:

from ssl4eo.models.ssl4eo_resnet import SSL4EOResNetModel

model = SSL4EOResNetModel.from_pretrained(
    "BiliSakura/SSL4EO-S12-transformers",
    subfolder="ssl4eo-resnet18-rgb-moco",
)

Local paths

Replace REPO with a local directory, e.g. /path/to/SSL4EO-S12-transformers/ssl4eo-vit-base-patch16-s2c-mae, and omit subfolder when pointing at a single checkpoint folder.

Training data

All weights were pretrained on SSL4EO-S12 (Sentinel-1 + Sentinel-2 patch triplets, ~251k locations, four seasonal timestamps). See the dataset card and paper for collection and preprocessing details.

Default ViT/ResNet pretraining used 100 epochs on 13-band S2-L1C unless noted (MAE ViT-H uses 199 epochs). Inputs are clipped to [0, 1] by dividing reflectance by 10000.

Dependencies

• —transformers, timm, torch, torchvision, safetensors
• —opencv-python (multispectral resize with more than 4 channels)
• —ssl4eo (optional; required for Data2vec loading until remote-code templates are added)

Citation

@article{wang2022ssl4eo,
  title={SSL4EO-S12: A Large-Scale Multi-Modal, Multi-Temporal Dataset for Self-Supervised Learning in Earth Observation},
  author={Wang, Yi and Braham, Nassim Ait Ali and Xiong, Zhitong and Liu, Chenying and Albrecht, Conrad M and Zhu, Xiao Xiang},
  journal={arXiv preprint arXiv:2211.07044},
  year={2022}
}

License

Pretrained model weights in this repository are released under CC-BY-4.0, consistent with the SSL4EO-S12 project. Remote-code files derived from the integration layer may follow the upstream repository license (Apache-2.0).