Motif-Technologies/Motif-Vision-Encoder

1---2license: mit3library_name: transformers4pipeline_tag: image-feature-extraction5tags:6  - motif7  - vision-transformer8  - self-supervised9  - image-feature-extraction10  - video11  - custom_code12---13 14# Motif Vision Encoder15 16Motif Vision Encoder is a unified image + video self-supervised vision encoder on a ViT17backbone. A single 3D-convolutional tokenizer ingests both modalities — an image is a181-frame clip (`T=1`), a video is `T>1` — so the same weights produce dense patch-level19features and a language-aligned global (CLS) representation.20 21Trained on **~1/3 the data of DINOv3** (0.5B vs 1.7B samples), it still reaches competitive22performance across image and video benchmarks — and leads on DAVIS video tracking.23 24<p align="center">25  <img src="assets/haaland_attn_blk20.gif" width="480" alt="Point tracking on a video clip: Motif vs V-JEPA 2.1"/>26</p>27<p align="center"><em>Point tracking on a video clip (top: Motif, bottom: V-JEPA 2.1) — a query point propagated across frames by patch-feature cosine similarity. Motif tracks the subject more reliably than V-JEPA 2.1.</em></p>28 29- **Architecture**: ViT-7B (embed 4096 / depth 40 / heads 32), patch 16, 3D axial RoPE30  (`base=100`), SwiGLU FFN, LayerScale, per-head QK-norm, gated attention, 4 register tokens.31- **Tokenizer**: `Conv3d(kernel=stride=(tubelet, patch, patch))` — image `(B,3,H,W)` → `T=1`,32  video `(B,T,3,H,W)`. Token layout `[CLS] + [register × 4] + [patch × N]`.33 34 35## Usage36 37The model ships a self-contained `modeling_motif_vision_encoder.py`, so it loads with `trust_remote_code=True`.38 39### Image40 41```python42import torch43from transformers import AutoImageProcessor, AutoModel44from transformers.image_utils import load_image45 46url = "http://images.cocodataset.org/val2017/000000039769.jpg"47image = load_image(url)48 49repo = "Motif-Technologies/Motif-Vision-Encoder"50processor = AutoImageProcessor.from_pretrained(repo)51model = AutoModel.from_pretrained(repo, trust_remote_code=True, dtype=torch.bfloat16).to("cuda").eval()52 53inputs = processor(images=image, return_tensors="pt").to(model.device, torch.bfloat16)54with torch.inference_mode():55    outputs = model(**inputs)56 57outputs.last_hidden_state   # (1, 1 + 4 + N, 4096)  CLS + registers + patch tokens58outputs.pooler_output       # (1, 4096)             global (CLS) representation59 60patch_tokens = outputs.last_hidden_state[:, 5:, :]   # (1, N, 4096), N = (H/16)*(W/16)61```62 63The processor resizes the shorter side to 512, center-crops to 512×512, and normalizes with64ImageNet mean/std (BICUBIC). `H`/`W` must be multiples of 16.65 66### Video67 68An image is a 1-frame clip; a video is the same call with a `(B, T, 3, H, W)` tensor. Apply the69same per-frame transform (resize → center-crop → ImageNet norm) and stack over time:70 71```python72import torch73 74video = torch.randn(1, 8, 3, 256, 256, device="cuda", dtype=torch.bfloat16)  # (B, T, 3, H, W)75with torch.inference_mode():76    outputs = model(pixel_values=video)77```78 79## Model details80 81<p align="center">82  <img src="assets/architecture.png" width="820" alt="Motif Vision Encoder architecture: image and video inputs, patch embedding, 40-block transformer stack, and transformer block internals"/>83</p>84 85| | |86|---|---|87| Backbone | ViT-7B, patch 16, embed 4096, depth 40, heads 32, SwiGLU |88| Register tokens | 4 |89| Position encoding | 3D axial RoPE (T,H,W), `base=100.0` |90| Video tokenizer | 3D Conv, tubelet size 2 |91| Precision | bf16 weights |92| Training | DINO + iBOT + KoLeo self-distillation, Gram anchoring, contrastive caption alignment |93| Training data | ~0.47B samples — 448.6M images (96%) + 18.5M video clips (4%) |94 95Outputs (`BaseModelOutputWithPooling`): `last_hidden_state` `(B, 1+4+N, 4096)`,96`pooler_output` `(B, 4096)`.97 98## Evaluation99 100Compared against the strongest publicly reported self-supervised / vision backbones. Higher is101better for every column. Best comparable value per column in bold, second best <u>underlined</u>.102 103DAVIS S/M/L follow the DINOv3 protocol (J&F-mean at video short side 420/480, 840/960,1041260/1440 px). V-JEPA 2.1 is not part of the DINOv3 Table 5 tracking benchmark, so only its105single-resolution (S) figure is available.106 107| Model | Training<br>data | DAVIS S<br>J&F ↑ | DAVIS M<br>J&F ↑ | DAVIS L<br>J&F ↑ | ImageNet-1K<br>lin. probe ↑ | ADE20K<br>mIoU ↑ | K400 ↑ |108|---|---|---|---|---|---|---|---|109| **Motif Vision Encoder** | 0.5B | **73.8** | **80.4** | **83.4** | 87.4 | <u>52.0</u> | 87.4 |110| DINOv3 | 1.7B | <u>71.1</u> | <u>79.7</u> | <u>83.3</u> | 88.4 | **55.9** | <u>87.8</u> |111| Web-DINO | 2B | 57.2 | 65.8 | 69.5 | 85.9 | 42.7 | 86.8 |112| PEcore | 5.4B | 48.2 | 53.1 | 49.8 | **89.3** | 38.9 | **87.9** |113| SigLIP2 | 10B | 56.1 | 62.3 | 62.9 | <u>89.1</u> | 45.4 | 86.9 |114| OpenCLIP | 2B | – | – | – | – | – | – |115| V-JEPA 2.1 | 0.022B | 69.0 | – | – | 85.5 | 47.9 | 87.7 |116 117Protocol: DINOv3-style linear/attentive probes for image tasks; V-JEPA 2-style protocol for118video. Baseline DAVIS / ADE20K / K400 figures are taken from the DINOv3 technical report's119unified evaluation (Tab. 3, 5, 6) and ImageNet from Tab. 7; OpenCLIP is not in that report and120its benchmarks are not reported under a comparable protocol.121 122Motif is state of the art on DAVIS video tracking at every resolution (74.0 / 80.5 / 83.5 J&F)123and stays competitive on the other image and video benchmarks, using roughly 1/3 of DINOv3's124training data (~0.5B samples).125 126<p align="center">127  <img src="assets/davis_mask_propagation.gif" width="820" alt="Mask propagation: ground truth vs DINOv3 vs Motif"/>128</p>129 130<p align="center">131  <img src="assets/dense_attention_comparison.png" width="820" alt="Dense attention and feature-similarity comparison across Motif, DINOv3, V-JEPA 2.1, and SigLIP2"/>132</p>133<p align="center"><em>Dense features on a single image (768px). Columns: query point, CLS attention, query-point attention, patch-feature cosine similarity. Motif and DINOv3 keep attention and similarity tightly localized on the queried object, while V-JEPA 2.1 and SigLIP2 are noticeably noisier.</em></p>134 135 136## License137 138Released under the **MIT License** (see `LICENSE`). The model was trained on data governed by the139respective dataset licenses; downstream users are responsible for compliance with those terms.140