lmms-lab-encoder/onevision-encoder-large-lang

1---2license: apache-2.03library_name: transformers4pipeline_tag: image-feature-extraction5---6 7# OneVision-Encoder8 9OneVision-Encoder is an LLM-aligned vision transformer specifically optimized for Large Multimodal Models (LMMs). It is a core component of the [LLaVA-OneVision-2](https://huggingface.co/papers/2605.25979) series and is further detailed in the technical report [OneVision-Encoder: Codec-Aligned Sparsity as a Foundational Principle for Multimodal Intelligence](https://arxiv.org/abs/2602.08683).10 11[**Project Page**](https://evolvinglmms-lab.github.io/LLaVA-OneVision-2/) | [**GitHub**](https://github.com/EvolvingLMMs-Lab/LLaVA-OneVision-2)12 13### Key Features14 15- **LLM-Aligned Architecture**: Unlike standard vision backbones, this model is specifically optimized for **Large Multimodal Models (LMMs)**, ensuring seamless feature alignment and superior performance when connected to language models.16- **True Native Resolution**: Supports dynamic, **fully native resolution** inputs directly. It processes images and videos in their original aspect ratios without the need for tiling, cropping, padding, or resizing hacks.17- **Arbitrary Frame Support**: Capable of processing video inputs with **any number of frames** (variable length). It breaks the constraint of fixed-frame inputs, allowing for flexible long-context video understanding limited only by memory.18- **Codec-Style Input Processing**: Implements a "OneVision" mechanism that treats video like a codec stream—**sampling dense frames sparsely** (selecting important patches from many frames) rather than the traditional approach of sampling sparse frames densely.19- **3D Rotary Position Embedding**: Uses a 4:6:6 split for temporal, height, and width dimensions to capture complex spatiotemporal relationships across arbitrary sequence lengths.20 21#### Downstream Tasks22 23- **Video benchmarks**: MVBench, VideoMME, Perception Test24- **Image understanding**: DocVQA, ChartQA, OCRBench25- **Action recognition**: SSv2, UCF101, Kinetics26 27### Quick Start28 29> [!IMPORTANT]30> **Transformers Version Compatibility:**31>32> - ✅ **`transformers==4.57.3`** (Recommended): Works with `AutoModel.from_pretrained()`33> - ⚠️ **`transformers>=5.0.0`**: Not currently supported. We are actively working on a fix.34 35> **Note on Inputs:**36> While the model is pre-trained with the configurations below, it supports **dynamic native resolution** and **arbitrary frame counts** during inference:37>38> - **Pre-training Image Base**: 448×44839> - **Pre-training Video Base**: 224×224 (256 tokens/frame)40> - **Inference**: Supports variable resolutions and frame lengths.41 42```python43from transformers import AutoModel, AutoImageProcessor44from PIL import Image45import torch46 47# Load model and preprocessor48model = AutoModel.from_pretrained(49    "lmms-lab-encoder/onevision-encoder-large-lang",50    trust_remote_code=True,51    attn_implementation="flash_attention_2"52).to("cuda").eval()53 54preprocessor = AutoImageProcessor.from_pretrained(55    "lmms-lab-encoder/onevision-encoder-large-lang",56    trust_remote_code=True57)58 59# Image inference: [B, C, H, W]60image = Image.open("path/to/your/image.jpg")  # Replace with your image path61pixel_values = preprocessor(images=image, return_tensors="pt")["pixel_values"].to("cuda")62with torch.no_grad():63    outputs = model(pixel_values)64    # outputs.last_hidden_state: [B, num_patches, hidden_size]65    # outputs.pooler_output: [B, hidden_size]66 67# Video inference: [B, C, T, H, W] with patch_positions68num_frames, target_frames = 16, 6469patch_size = 1470# Load video frames and preprocess each frame (replace with your video frame paths)71frames = [Image.open(f"path/to/frame_{i}.jpg") for i in range(num_frames)]72video_pixel_values = preprocessor(images=frames, return_tensors="pt")["pixel_values"]73# Reshape from [T, C, H, W] to [B, C, T, H, W]74video = video_pixel_values.unsqueeze(0).permute(0, 2, 1, 3, 4).to("cuda")75 76# Build patch_positions for temporal sampling: [B, num_frames * frame_tokens, 3]77frame_pos = torch.linspace(0, target_frames - 1, num_frames).long().cuda()  # [T]78grid_h, grid_w = video.shape[-2] // patch_size, video.shape[-1] // patch_size  # patch grid79frame_tokens = grid_h * grid_w80 81t_positions = frame_pos[:, None].repeat(1, frame_tokens).reshape(-1)  # [T * frame_tokens]82h_positions = torch.arange(grid_h, device="cuda").repeat_interleave(grid_w)83h_positions = h_positions.repeat(num_frames)  # [T * frame_tokens]84w_positions = torch.arange(grid_w, device="cuda").repeat(grid_h)85w_positions = w_positions.repeat(num_frames)  # [T * frame_tokens]86 87patch_positions = torch.stack([t_positions, h_positions, w_positions], dim=-1).unsqueeze(0)88 89with torch.no_grad():90  outputs = model(video, patch_positions=patch_positions)91```92 93### Model Properties94 95| Property | Value |96| --- | --- |97| **Model Type** | **LLM-Aligned** Vision Transformer (ViT) |98| **Architecture** | **HEVC-Style** / Codec-Like Vision Transformer |99| **Input Paradigm** | **Codec-Style** (Sparse Patch / Dense Frame) |100| **Resolution Strategy** | **True Native Resolution** (Dynamic, No Tiling) |101| **Temporal Context** | **Arbitrary Frame Count** (Variable Length Support) |102| **Hidden Size** | 1024 |103| **Intermediate Size** | 4096 |104| **Number of Layers** | 24 |105| **Number of Attention Heads** | 16 |106| **Patch Size** | 14 |107| **Positional Encoding** | 3D RoPE (4:6:6 split for T:H:W) |108| **Normalization** | Layer Normalization |109| **Activation Function** | GELU |110| **License** | Apache 2.0 |111 112### Citation113 114```bibtex115@inproceedings{LLaVA-OneVision-2,116  title={LLaVA-OneVision-2},117  author={llava-onevision contributors},118  booktitle={arXiv},119  year={2026}120}121 122@article{tang2026onevisionencoder,123  title={OneVision-Encoder: Codec-Aligned Sparsity as a Foundational Principle for Multimodal Intelligence},124  author={Tang, Feilong and An, Xiang and Yan, Yunyao and Xie, Yin and Qin, Bin and Yang, Kaicheng and Shen, Yifei and Zhang, Yuanhan and Li, Chunyuan and Feng, Shikun and Chen, Changrui and Tan, Huajie and Hu, Ming and Zhang, Manyuan and Li, Bo and Feng, Ziyong and Liu, Ziwei and Ge, Zongyuan and Deng, Jiankang},125  journal={arXiv preprint arXiv:2602.08683},126  year={2026}127}128```