naver-hyperclovax/HyperCLOVAX-SEED-Vision-Instruct-3B
2214.1k
1---2license: other3license_name: hyperclovax-seed4license_link: LICENSE5library_name: transformers6---7 8 910 11 12## **Overview**13 14HyperCLOVAX-SEED-Vision-Instruct-3B is a model developed by NAVER, built upon its proprietary backbone model and fine-tuned through post-training. It is capable of understanding both text and images, as well as generating text.15 16The model is primarily designed with a focus on lightweight architecture, optimizing computational efficiency. In terms of visual understanding, it can handle visual question answering (VQA), chart and diagram interpretation, and even comprehend content. HyperCLOVAX-SEED-Vision-Instruct-3B aims for a Pareto-optimal balance specifically tuned for the Korean language, and it demonstrates competitive performance using fewer visual tokens compared to other models of similar size in inference scenarios.17 18Particularly, the model shows relative strengths in handling Korean-language inputs and outperforms similarly sized open-source models in related benchmarks. As the first open-source vision-language model in Korea capable of visual understanding, it is expected to significantly contribute to strengthening Korea's sovereign AI capabilities.19 20 21## **Updates**22- **(2025.07.25)**: vLLM engine is available with [our repository](https://github.com/NAVER-Cloud-HyperCLOVA-X/vllm/tree/v0.9.2rc2_hyperclovax_vision_seed)23- **(2025.07.08)**: Major code update for supporting vLLM engine ([link - related_discussion](https://huggingface.co/naver-hyperclovax/HyperCLOVAX-SEED-Vision-Instruct-3B/discussions/27))24- **(2025.04.22)**: Initial release of the repository.25 26 27## **Basic Information**28 29- **Model Architecture**: LLaVA-based Vision-Language Model 30 - **LLM Module**: Transformer-based architecture (Dense Model) 31 - **Vision Encoder** : SigLIP-based architecture with 378x378px input resolution per grid.32 - **Vision-Language Connector** : C-Abstractor based architecture with AnyRes mechanism, supporting up to 1.29M total pixels across 9 grids.33- **Parameter Count**: 3.2B (LLM Module) + 0.43B (Vision Module) 34- **Input/Output Format**: Text + Image + Video / Text 35- **Context Length**: 16k 36- **Knowledge Cutoff Date**: The model was trained on data collected before August 2024. 37 38 39## **Training**40 41#### **Text**42 43Securing high-quality data is essential even during post-training, but having humans manually create or revise large-scale datasets posed significant limitations in terms of both cost and resources. Additionally, tasks requiring domain expertise were difficult to handle, and the risk of human error was high. To overcome these challenges, we utilized an automated validation system powered by HyperCLOVA X, which improved data quality and streamlined the training process — ultimately leading to enhanced overall model performance. As a result, the model showed significant improvements in areas with definitive answers, such as mathematics and coding.44 45While reducing the cost of data collection is important, finding efficient training strategies is equally critical. HyperCLOVAX-SEED-Vision-Instruct-3B was developed starting from the HyperCLOVAX-SEED-Text-Base-3B and applied both Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF) based on an online reinforcement algorithm called GRPO.46 47#### **Vision**48 49The Vision Understanding feature — where the model receives images and questions as input and generates text-based answers — was not part of the initial design of HyperCLOVA X. Therefore, the model architecture was carefully designed to add capabilities for handling vision-related tasks, such as image-based question answering (VQA) and chart/diagram interpretation, without compromising the existing performance of the HCX LLM. Special attention was given to handling auxiliary information within the input, especially considering the context length.50 51Although HyperCLOVAX-SEED-Vision-Instruct-3B is a lightweight model, it is capable of performing basic image VQA tasks and even supports OCR-free processing. One of the key focus areas for this 3B model was optimizing the efficiency of video input tokens. Since input token length directly affects computational cost, the number of tokens extracted per frame was carefully adjusted to enable efficient video understanding with as few tokens as possible. Additionally, during the RLHF training phase, vision-specific V-RLHF data was used to enhance the model’s learning, just like in the text domain.52 53## Benchmark54#### Text55 56| **Model** | **KMMLU (5-shot, acc)** | **HAE-RAE (5-shot, acc)** | **CLiCK (5-shot, acc)** | **KoBEST (5-shot, acc)** |57|----------------------------|--------|---------|---------|-------|58| HyperCLOVAX-SEED-Text-Base-3B | 0.4847 | 0.7635 | 0.6386 | 0.7792 |59| HyperCLOVAX-SEED-Vision-Instruct-3B| 0.4422 | 0.6499 | 0.5599 | 0.7180 |60| Qwen2.5-3B-instruct | 0.4451 | 0.6031 | 0.5649 | 0.7053 |61| gemma-3-4b-it | 0.3895 | 0.6059 | 0.5303 | 0.7262 |62 63#### Vision64 65| Model Name | Max Token Count per Video | VideoMME (Ko) | NAVER-TV-CLIP (Ko) | VideoChatGPT (Ko) | PerceptionTest (En) | ActivityNet-QA (En) | KoNet (Ko) | MMBench-Val (En) | TextVQA-Val (En) | Korean VisIT-Bench (Ko) | Image (4 benchmarks) | Video (5 benchmarks) | All (9 benchmarks) |66|-----------------------------------|--------------------------------|----------------|---------------------|--------------------|-----------------------|----------------------|------------|-------------------|-------------------|--------------------------|------------------------|------------------------|----------------------|67| HyperCLOVAX-SEED-Vision-Instruct-3B | 1856 tokens, 108 frames | 48.2 | 61.0 | 53.6 | 55.2 | 50.6 | 69.2 | 81.8 | 79.2 | 37.0 | 46.68 | 53.70 | 59.54 |68| HyperCLOVAX-SEED-Vision-Instruct-3B (without OCR)| 1856 tokens, 108 frames | 48.2 | 61.0 | 53.6 | 55.2 | 50.6 | 36.6 | 80.7 | 76.0 | 43.5 | 56.74 | 53.70 | 55.05 |69| Qwen-2.5-VL-3B | 24576 tokens, 768 frames | 55.1 | 48.3 | 45.6 | 66.9 | 55.7 | 58.3 | 84.3 | 79.6 | 81.5 | 59.35 | 54.31 | 56.55 |70| Qwen-2.5-VL-3B (w/ 2000 tokens) | 2000 tokens, 128 frames | 50.3 | 43.9 | 44.3 | 58.3 | 54.2 | 58.5 | 84.3 | 79.3 | 15.7 | 59.50 | 50.18 | 54.33 |71| Qwen-2.5-VL-7B | 24576 tokens, 768 frames | 60.6 | 66.7 | 51.8 | 70.5 | 56.6 | 68.4 | 88.3 | 84.9 | 85.6 | 69.34 | 61.23 | 64.84 |72| Gemma-3-4B | 4096 tokens, 16 frames | 45.4 | 36.8 | 57.1 | 50.6 | 46.3 | 25.0 | 79.2 | 58.9 | 32.3 | 48.91 | 47.24 | 47.98 |73| GPT4V (gpt-4-turbo-2024-04-09) | Unknown, Original Image , 8 frames | 49.1 | 75.0 | 55.5 | 57.4 | 45.7 | 38.7 | 84.2 | 60.4 | 52.0 | 58.88 | 51.59 | 54.83 |74| GPT4o (gpt-4o-2024-08-06) | Unknown, 512 resize, 128 frames| 61.6 | 66.6 | 61.8 | 50.2 | 41.7 | 60.6 | 84.2 | 73.2 | 50.5 | 67.15 | 56.42 | 61.19 |75| InternV-2-2B | 4096 tokens, 16 frames | 28.9 | 21.1 | 40.2 | 50.5 | 50.3 | 3.3 | 79.3 | 75.1 | 51.1 | 39.74 | 38.19 | 38.88 |76| InternV-2-4B | 4096 tokens, 16 frames | 33.8 | 36.0 | 22.8 | 54.2 | 52.0 | 22.7 | 83.0 | 76.9 | 51.6 | 46.11 | 39.75 | 42.58 |77| InternV-2-8B | 4096 tokens, 16 frames | 43.7 | 41.2 | 32.4 | 58.5 | 53.2 | 28.5 | 86.6 | 79.0 | 97.0 | 50.32 | 45.79 | 47.81 |78 79## Dependencies80- [einops](https://einops.rocks/)81- [timm](https://github.com/huggingface/pytorch-image-models)82- [av](https://github.com/PyAV-Org/PyAV)83- [decord](https://github.com/dmlc/decord)84 85## Example86**(code & benchmark score) checked with transformers 4.52.4**87 88```python89 90from transformers import AutoModelForCausalLM, AutoProcessor, AutoTokenizer91 92model_name = "naver-hyperclovax/HyperCLOVAX-SEED-Vision-Instruct-3B"93model = AutoModelForCausalLM.from_pretrained(model_name, trust_remote_code=True).to(device="cuda")94processor = AutoProcessor.from_pretrained(model_name, trust_remote_code=True)95tokenizer = AutoTokenizer.from_pretrained(model_name)96 97# LLM Example98# It is recommended to use the chat template with HyperCLOVAX models.99# Using the chat template allows you to easily format your input in ChatML style.100llm_chat = [101 {"role": "system", "content": [{"type": "text", "text": "you are helpful assistant!"}]},102 {103 "role": "user", 104 "content": [105 {"type": "text", "text": "Hello, how are you?"},106 {"type": "text", "text": "I said. Hello, how are you today?"},107 ]108 },109 {"role": "assistant", "content": [{"type": "text", "text": "I'm doing great. How can I help you today?"}]},110 {"role": "user", "content": [{"type": "text", "text": "I'd like to show off how chat templating works!"}]},111]112model_inputs = processor.apply_chat_template(113 llm_chat, tokenize=True, return_dict=True, return_tensors="pt", add_generation_prompt=True114)115model_inputs = model_inputs.to(device="cuda")116 117# Please adjust parameters like top_p appropriately for your use case.118output_ids = model.generate(119 **model_inputs,120 max_new_tokens=64,121 do_sample=True,122 top_p=0.6,123 temperature=0.5,124 repetition_penalty=1.0,125)126print("=" * 80)127print("LLM EXAMPLE")128print(processor.batch_decode(output_ids)[0])129print("=" * 80)130 131# VLM Example132# For images and videos, you can use url, local_path, base64, or bytes as input sources.133vlm_chat = [134 {"role": "system", "content": [{"text": "System Prompt", "type": "text"}]},135 {"role": "user", "content": [{"text": "User Text Prompt 1", "type": "text"}]},136 {137 "role": "user",138 "content": [{139 "filename": "tradeoff_sota.png",140 "image": "https://github.com/naver-ai/rdnet/blob/main/resources/images/tradeoff_sota.png?raw=true",141 "lens_keywords": "Gucci Ophidia, cross bag, Ophidia small, GG, Supreme shoulder bag",142 "lens_local_keywords": "[0.07, 0.21, 0.92, 0.90] Gucci Ophidia",143 "ocr": "List the words in the image in raster order. Even if the word order feels unnatural for reading, the model will handle it as long as it follows raster order.", "type": "image",144 }],145 },146 {147 "role": "user",148 "content": [{149 "filename": "tradeoff.png",150 "image": "https://github.com/naver-ai/rdnet/blob/main/resources/images/tradeoff.png?raw=true",151 "type": "image",152 }],153 },154 {"role": "assistant", "content": [{"text": "Assistant Text Prompt 1", "type": "text"}]},155 {"role": "user", "content": [{"text": "User Text Prompt 2", "type": "text"}]},156 {157 "role": "user",158 "content": [159 {160 "type": "video",161 "video": "freenaturestock-rolling-mist-clouds.mp4",162 "lens_keywords": "Prada re-edition, nylon bag, mini cross bag, logo strap, essential shoulder bag",163 "lens_local_keywords": "[0.12, 0.34, 0.85, 0.76] Prada re-edition",164 "speech_to_text": "Please enter the dialogue, voice, sound, lines, and words in the video in text format.",165 },166 {"text": "User Text Prompt 3", "type": "text"},167 ]168 },169]170 171model_inputs = processor.apply_chat_template( 172 vlm_chat, tokenize=True, return_dict=True, return_tensors="pt", add_generation_prompt=True,173)174model_inputs = model_inputs.to(device="cuda")175output_ids = model.generate(176 **model_inputs,177 max_new_tokens=64,178 do_sample=True,179 top_p=0.6,180 temperature=0.5,181 repetition_penalty=1.0,182)183print("=" * 80)184print("VLM EXAMPLE")185print(processor.batch_decode(output_ids)[0])186print("=" * 80)187 188```189 190## Example for v0.1.0191**(code & benchmark score) checked with transformers 4.45.0**192 193```python194 195from transformers import AutoModelForCausalLM, AutoProcessor, AutoTokenizer196 197model_name = "naver-hyperclovax/HyperCLOVAX-SEED-Vision-Instruct-3B"198revision="v0.1.0"199model = AutoModelForCausalLM.from_pretrained(model_name, trust_remote_code=True, revision=revision).to(device="cuda")200preprocessor = AutoProcessor.from_pretrained(model_name, trust_remote_code=True, revision=revision)201tokenizer = AutoTokenizer.from_pretrained(model_name, revision=revision)202 203# LLM Example204# It is recommended to use the chat template with HyperCLOVAX models.205# Using the chat template allows you to easily format your input in ChatML style.206chat = [207 {"role": "system", "content": "you are helpful assistant!"},208 {"role": "user", "content": "Hello, how are you?"},209 {"role": "assistant", "content": "I'm doing great. How can I help you today?"},210 {"role": "user", "content": "I'd like to show off how chat templating works!"},211]212input_ids = tokenizer.apply_chat_template(chat, return_tensors="pt", tokenize=True)213input_ids = input_ids.to(device="cuda")214 215# Please adjust parameters like top_p appropriately for your use case.216output_ids = model.generate(217 input_ids,218 max_new_tokens=64,219 do_sample=True,220 top_p=0.6,221 temperature=0.5,222 repetition_penalty=1.0,223)224print("=" * 80)225print("LLM EXAMPLE")226print(tokenizer.batch_decode(output_ids)[0])227print("=" * 80)228 229# VLM Example230# For image and video inputs, you can use url, local_path, base64, or bytes.231vlm_chat = [232 {"role": "system", "content": {"type": "text", "text": "System Prompt"}},233 {"role": "user", "content": {"type": "text", "text": "User Text 1"}},234 {235 "role": "user",236 "content": {237 "type": "image",238 "filename": "tradeoff_sota.png",239 "image": "https://github.com/naver-ai/rdnet/blob/main/resources/images/tradeoff_sota.png?raw=true",240 "ocr": "List the words in the image in raster order. Even if the word order feels unnatural for reading, the model will handle it as long as it follows raster order.",241 "lens_keywords": "Gucci Ophidia, cross bag, Ophidia small, GG, Supreme shoulder bag",242 "lens_local_keywords": "[0.07, 0.21, 0.92, 0.90] Gucci Ophidia",243 }244 },245 {246 "role": "user",247 "content": {248 "type": "image",249 "filename": "tradeoff.png",250 "image": "https://github.com/naver-ai/rdnet/blob/main/resources/images/tradeoff.png?raw=true",251 }252 },253 {"role": "assistant", "content": {"type": "text", "text": "Assistant Text 1"}},254 {"role": "user", "content": {"type": "text", "text": "User Text 2"}},255 {256 "role": "user",257 "content": {258 "type": "video",259 "filename": "rolling-mist-clouds.mp4",260 "video": "freenaturestock-rolling-mist-clouds.mp4",261 }262 },263 {"role": "user", "content": {"type": "text", "text": "User Text 3"}},264]265 266new_vlm_chat, all_images, is_video_list = preprocessor.load_images_videos(vlm_chat)267preprocessed = preprocessor(all_images, is_video_list=is_video_list)268input_ids = tokenizer.apply_chat_template(269 new_vlm_chat, return_tensors="pt", tokenize=True, add_generation_prompt=True,270)271 272output_ids = model.generate(273 input_ids=input_ids.to(device="cuda"),274 max_new_tokens=8192,275 do_sample=True,276 top_p=0.6,277 temperature=0.5,278 repetition_penalty=1.0,279 **preprocessed,280)281print("=" * 80)282print("VLM EXAMPLE")283print(tokenizer.batch_decode(output_ids)[0])284print("=" * 80)285```286 287- To ensure the highest level of image understanding performance, it is recommended to include additional information such as Optical Character Recognition (OCR) results and entity recognition (Lens). The provided usage examples are written under the assumption that OCR and Lens results are available. If you input data in this format, you can expect significantly improved output quality.288 289## vLLM290To speed up your inference, you can use the vLLM engine from [our repository](https://github.com/NAVER-Cloud-HyperCLOVA-X/vllm/tree/v0.9.2rc2_hyperclovax_vision_seed). 291 292Make sure to switch to the `v0.9.2rc2_hyperclovax_vision_seed` branch. 293 294**Launch API server**:295 296```bash297pyenv virtualenv 3.10.2 .vllm298pyenv activate .vllm299sudo apt-get install -y kmod300pip install --upgrade setuptools wheel pip301pip install setuptools_scm302 303# install latest commit (e.g. v0.9.0)304VLLM_USE_PRECOMPILED=1 pip install -e .[serve] --cache-dir=/mnt/tmp305pip install -U pynvml306pip install timm av decord307 308# or install previous commit (e.g. v0.8.4)309pip install -r ./requirements/build.txt310pip install -r ./requirements/common.txt311pip install -r ./requirements/cuda.txt312pip install flash_attn==2.7.4.post1313pip install flashinfer -i https://flashinfer.ai/whl/cu121/torch2.4/314export VLLM_COMMIT=dc1b4a6f1300003ae27f033afbdff5e2683721ce315export VLLM_PRECOMPILED_WHEEL_LOCATION=https://wheels.vllm.ai/${VLLM_COMMIT}/vllm-1.0.0.dev-cp38-abi3-manylinux1_x86_64.whl316VLLM_USE_PRECOMPILED=1 pip install -e .[serve] --cache-dir=/mnt/tmp317pip install -U pynvml318pip install timm av decord319 320# Then launch api321MODEL=your/mode/path322export ATTENTION_BACKEND=FLASH_ATTN_VLLM_V1323VLLM_USE_V1=1 VLLM_ATTENTION_BACKEND=${ATTENTION_BACKEND} CUDA_VISIBLE_DEVICES=0,1 python -m vllm.entrypoints.openai.api_server \324 --seed 20250525 \325 --port ${PORT} \326 --allowed-local-media-path $ALLOWED_LOCAL_MEDIA_PATH \327 --max-model-len 8192 \328 --max-num-batched-tokens 8192 \329 --max-num-seqs 128 \330 --max-parallel-loading-workers 128 \331 --limit-mm-per-prompt.image="32" \332 --limit-mm-per-prompt.viedo="32" \333 --max-num-frames 256 \334 --tensor-parallel-size 1 \335 --data-parallel-size 1 \336 --model ${MODEL} \337 --dtype float16 \338 --trust-remote-code \339 --chat-template-content-format "openai" \340 --download-dir $DONWLOAD_DIR341```342 343**Request Example**:344- https://github.com/vllm-project/vllm/pull/20931#issue-3229161410345 346**Offline Inference Examples**:347- https://github.com/vllm-project/vllm/blob/main/examples/offline_inference/vision_language.py348- https://github.com/vllm-project/vllm/blob/main/examples/offline_inference/vision_language_multi_image.py