TIGER-Lab/VideoScore-Qwen2-VL

1---2library_name: transformers3tags: []4---5 6[📃Paper](https://arxiv.org/abs/2406.15252) | [🌐Website](https://tiger-ai-lab.github.io/VideoScore/) | [💻Github](https://github.com/TIGER-AI-Lab/VideoScore) | [🛢️Datasets](https://huggingface.co/datasets/TIGER-Lab/VideoFeedback) | [🤗Model (VideoScore)](https://huggingface.co/TIGER-Lab/VideoScore) | [🤗Demo](https://huggingface.co/spaces/TIGER-Lab/VideoScore) 7 8 9![VideoScore](https://tiger-ai-lab.github.io/VideoScore/static/images/teaser.png)10 11## Introduction12- 🧐🧐[VideoScore-Qwen2-VL](https://huggingface.co/TIGER-Lab/VideoScore-Qwen2-VL) is a variant from [VideoScore](https://huggingface.co/TIGER-Lab/VideoScore), 13taking [Qwen2-VL](https://huggingface.co/Qwen/Qwen2-VL-7B-Instruct) as base model, and trained on [VideoFeedback](https://huggingface.co/datasets/TIGER-Lab/VideoFeedback) dataset.14 15- [VideoScore](https://huggingface.co/TIGER-Lab/VideoScore) series is a video quality evaluation model series, taking [Mantis-8B-Idefics2](https://huggingface.co/TIGER-Lab/Mantis-8B-Idefics2) or [Qwen/Qwen2-VL](https://huggingface.co/Qwen/Qwen2-VL-7B-Instruct) as base-model16and trained on [VideoFeedback](https://huggingface.co/datasets/TIGER-Lab/VideoFeedback),17a large video evaluation dataset with multi-aspect human scores.18 19- VideoScore can reach 75+ Spearman correlation with humans on VideoEval-test, surpassing all the MLLM-prompting methods and feature-based metrics. 20 21- VideoScore also beat the best baselines on other three benchmarks EvalCrafter, GenAI-Bench and VBench, showing high alignment with human evaluations.22 23- **This is the regression version of VideoScore**24 25## Evaluation Results26 27We test VideoScore-Qwen2-VL on VideoFeedback-test and take Spearman corrleation between model's output and human ratings 28averaged among all the evaluation aspects as indicator. 29 30The evaluation results are shown below: 31 32| metric            | VideoFeedback-test | 33|:-----------------:|:------------------:|34| VideoScore-Qwen2-VL   |           **74.9** |35| Gemini-1.5-Pro    |               22.1 |  36| Gemini-1.5-Flash  |               20.8 | 37| GPT-4o            |        <u>23.1</u> |38| CLIP-sim          |                8.9 |39| DINO-sim          |                7.5 | 40| SSIM-sim          |               13.4 |41| CLIP-Score        |               -7.2 |42| LLaVA-1.5-7B      |                8.5 | 43| LLaVA-1.6-7B      |               -3.1 | 44| X-CLIP-Score      |               -1.9 |  45| PIQE              |              -10.1 |    46| BRISQUE           |              -20.3 |    47| Idefics2          |                6.5 |  48| MSE-dyn           |               -5.5 |   49| SSIM-dyn          |              -12.9 |    50 51The best in VideoScore series is in bold and the best in baselines is underlined. 52 53## Usage54### Installation55```56pip install git+https://github.com/TIGER-AI-Lab/VideoScore.git57# or58# pip install mantis-vl59```60 61### Inference62```63cd VideoScore/examples64```65 66```python67"""68pip install qwen_vl_utils mantis-vl69"""70import torch71from mantis.models.qwen2_vl import Qwen2VLForSequenceClassification72from transformers import Qwen2VLProcessor73from qwen_vl_utils import process_vision_info74 75ROUND_DIGIT=376REGRESSION_QUERY_PROMPT = """77Suppose you are an expert in judging and evaluating the quality of AI-generated videos,78please watch the following frames of a given video and see the text prompt for generating the video,79then give scores from 5 different dimensions:80(1) visual quality: the quality of the video in terms of clearness, resolution, brightness, and color81(2) temporal consistency, both the consistency of objects or humans and the smoothness of motion or movements82(3) dynamic degree, the degree of dynamic changes83(4) text-to-video alignment, the alignment between the text prompt and the video content84(5) factual consistency, the consistency of the video content with the common-sense and factual knowledge85 86for each dimension, output a float number from 1.0 to 4.0,87the higher the number is, the better the video performs in that sub-score, 88the lowest 1.0 means Bad, the highest 4.0 means Perfect/Real (the video is like a real video)89Here is an output example:90visual quality: 3.291temporal consistency: 2.792dynamic degree: 4.093text-to-video alignment: 2.394factual consistency: 1.895 96For this video, the text prompt is "{text_prompt}",97all the frames of video are as follows:98"""    99 100model_name="TIGER-Lab/VideoScore-Qwen2-VL"101video_path="video1.mp4"102video_prompt="Near the Elephant Gate village, they approach the haunted house at night. Rajiv feels anxious, but Bhavesh encourages him. As they reach the house, a mysterious sound in the air adds to the suspense."103 104# default: Load the model on the available device(s)105model = Qwen2VLForSequenceClassification.from_pretrained(106    model_name, torch_dtype="auto", device_map="auto", attn_implementation="flash_attention_2"107)108 109# default processer110processor = Qwen2VLProcessor.from_pretrained(model_name)111 112# Messages containing a images list as a video and a text query113response = ""114label_names = ["visual quality", "temporal consistency", "dynamic degree", "text-to-video alignment", "factual consistency"]115for i in range(len(label_names)):116    response += f"The score for {label_names[i]} is {model.config.label_special_tokens[i]}. "117messages = [118    {119        "role": "user",120        "content": [121            {122                "type": "video",123                "video": video_path,124                "fps": 8.0,125            },126            {"type": "text", "text": REGRESSION_QUERY_PROMPT.format(text_prompt=video_prompt)},127        ],128    },129    {130        "role": "assistant",131        "content": [132            {"type": "text", "text": response},133        ],134    }135]136 137# Preparation for inference138text = processor.apply_chat_template(139    messages, tokenize=False, add_generation_prompt=False140)141image_inputs, video_inputs = process_vision_info(messages)142inputs = processor(143    text=[text],144    images=image_inputs,145    videos=video_inputs,146    padding=True,147    return_tensors="pt",148)149inputs = inputs.to("cuda")150 151# Inference152with torch.no_grad():153    outputs = model(**inputs)154 155logits = outputs.logits156num_aspects = logits.shape[-1]157 158aspect_scores = []159for i in range(num_aspects):160    aspect_scores.append(round(logits[0, i].item(),ROUND_DIGIT))161print(aspect_scores)162 163"""164model output on visual quality, temporal consistency, dynamic degree,165text-to-video alignment, factual consistency, respectively166VideoScore: 167[2.297, 2.469, 2.906, 2.766, 2.516]168 169VideoScore-Qwen2-VL:170[2.297, 2.531, 2.766, 2.312, 2.547]171"""172```173 174### Training175see [VideoScore/training](https://github.com/TIGER-AI-Lab/VideoScore/tree/main/training) for details176 177### Evaluation178see [VideoScore/benchmark](https://github.com/TIGER-AI-Lab/VideoScore/tree/main/benchmark) for details179 180## Citation181```bibtex182@article{he2024videoscore,183  title = {VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generation},184  author = {He, Xuan and Jiang, Dongfu and Zhang, Ge and Ku, Max and Soni, Achint and Siu, Sherman and Chen, Haonan and Chandra, Abhranil and Jiang, Ziyan and Arulraj, Aaran and Wang, Kai and Do, Quy Duc and Ni, Yuansheng and Lyu, Bohan and Narsupalli, Yaswanth and Fan, Rongqi and Lyu, Zhiheng and Lin, Yuchen and Chen, Wenhu},185  journal = {ArXiv},186  year = {2024},187  volume={abs/2406.15252},188  url = {https://arxiv.org/abs/2406.15252},189}190```191