dipta007/VCInspector-7B

1---2license: apache-2.03language:4- en5pipeline_tag: image-text-to-text6tags:7- multimodal8- video-caption-evaluation9- reference-free10- factual-analysis11- vision-language12library_name: transformers13base_model: Qwen/Qwen2.5-VL-7B-Instruct14datasets:15- dipta007/ActivityNet-FG-It16arxiv: 2509.1653817---18 19# VC-Inspector-7B20 21<p align="center">22  <a href="https://arxiv.org/abs/2509.16538">23    <img src="https://img.shields.io/badge/%F0%9F%94%A5_Accepted_at-ACL_2026_(Main)_%F0%9F%94%A5-b12a00?style=for-the-badge&labelColor=ffb300" alt="Accepted at ACL 2026 (Main)">24  </a>25</p>26 27[![ACL 2026 (Main)](https://img.shields.io/badge/ACL%202026-Main-blue)](https://arxiv.org/abs/2509.16538)28[![Paper](https://img.shields.io/badge/arXiv-2509.16538-red)](https://arxiv.org/abs/2509.16538)29[![Models](https://img.shields.io/badge/HuggingFace-Models-orange)](https://huggingface.co/collections/dipta007/vc-inspector)30[![Dataset](https://img.shields.io/badge/HuggingFace-Dataset-yellow)](https://huggingface.co/datasets/dipta007/ActivityNet-FG-It)31[![Python 3.12](https://img.shields.io/badge/python-3.12-blue.svg)](https://www.python.org/downloads/)32 33## Introduction34 35**VC-Inspector-7B** is a lightweight, open-source large multimodal model (LMM) for **reference-free evaluation of video captions** with a focus on **factual accuracy**. Unlike existing metrics that suffer from limited context handling, weak factuality assessment, or reliance on proprietary services, VC-Inspector offers a reproducible, fact-aware alternative that aligns closely with human judgments.36 37This model is fine-tuned from [Qwen2.5-VL-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct) using LoRA on our synthetic dataset [ActivityNet-FG-It](https://huggingface.co/datasets/dipta007/ActivityNet-FG-It), which contains 44K video-caption pairs with controlled factual errors and quality annotations.38 39### Key Features40 41- **Reference-free Evaluation**: Evaluates video captions without requiring ground-truth references42- **Factual Grounding**: Detects factual errors in objects and actions within captions43- **Interpretable Outputs**: Generates quality scores (1-5) with natural language explanations44- **Cross-domain Generalization**: Works on both video and image caption evaluation45- **State-of-the-art Performance**: Outperforms GPT-4o-based methods on VATEX-Eval46 47### Model Architecture48 49VC-Inspector-7B is built on Qwen2.5-VL-7B-Instruct with the following modifications:50- **Vision Encoder**: Frozen (preserves generalization)51- **Visual-Language Projector**: Frozen52- **LLM Component**: Fine-tuned with LoRA (rank=32, alpha=32)53 54## Evaluation Results55 56### Correlation with Human Judgments on VATEX-Eval57 58| Metric | Type | Kendall's τ_b | Spearman's ρ |59|:-------|:-----|:-------------:|:------------:|60| EMScore | Reference-free | 22.88 | 29.79 |61| CLIPScore | Reference-free | 22.33 | 29.09 |62| ViCLIPScore | Reference-free | 30.92 | 39.86 |63| G-VEval (GPT-4o) | Reference-free | 39.40 | - |64| Qwen2.5-VL-7B (base) | Reference-free | 34.70 | 39.40 |65| **VC-Inspector-7B** | Reference-free | **42.58** | **45.99** |66 67### Cross-domain Evaluation on Image Caption Benchmarks68 69| Metric | Flickr8K-Expert (τ_b) | Flickr8K-CF (τ_b) |70|:-------|:---------------------:|:-----------------:|71| CLIPScore (ref-free) | 51.10 | 34.40 |72| PAC-S (ref-free) | 53.90 | 36.00 |73| **VC-Inspector-7B** | **63.43** | **45.97** |74 75### Synthetic Dataset Evaluation76 77| Dataset | Kendall's τ_b | Spearman's ρ |78|:--------|:-------------:|:------------:|79| ActivityNet-FG-Eval | 49.53 | 62.01 |80| YouCook2-FG-Eval | 44.29 | 55.31 |81 82## Requirements83 84```bash85pip install torch transformers accelerate86pip install qwen-vl-utils[decord]==0.0.887pip install flash-attn --no-build-isolation88```89 90## Quickstart91 92### Using Transformers93 94```python95from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor96from qwen_vl_utils import process_vision_info97 98# Load model99model = Qwen2_5_VLForConditionalGeneration.from_pretrained(100    "dipta007/VCInspector-7B",101    torch_dtype="auto",102    device_map="auto",103)104processor = AutoProcessor.from_pretrained("dipta007/VCInspector-7B")105 106# Prepare input107caption = "A man is playing guitar in a field"108prompt = f"""<caption>{caption}</caption>109 110You are given a video and a caption describing the video content. Please rate the helpfulness, relevance, accuracy, level of details of the caption. The overall score should be on a scale of 1 to 5, where a higher score indicates better overall performance. Please first output a single line containing only one integer indicating the score. In the subsequent line, please provide a comprehensive explanation of your evaluation, avoiding any potential bias. STRICTLY FOLLOW THE FORMAT."""111 112messages = [113    {114        "role": "user",115        "content": [116            {"type": "video", "video": "path/to/video.mp4", "max_pixels": 360 * 420, "fps": 1.0},117            {"type": "text", "text": prompt},118        ],119    }120]121 122# Process and generate123text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)124image_inputs, video_inputs, video_kwargs = process_vision_info(messages, return_video_kwargs=True)125inputs = processor(126    text=[text],127    images=image_inputs,128    videos=video_inputs,129    padding=True,130    return_tensors="pt",131    **video_kwargs,132)133inputs = inputs.to("cuda")134 135generated_ids = model.generate(**inputs, max_new_tokens=256)136generated_ids_trimmed = [137    out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)138]139output_text = processor.batch_decode(140    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False141)142print(output_text[0])143```144 145### Example Output146 147```1484149The caption does not accurately capture the video content. For example, the objects (guitar) are incorrect.150```151 152### Using with ms-swift (vLLM backend)153 154```python155from swift.llm import VllmEngine, InferRequest, RequestConfig156import os157 158os.environ["VIDEO_MAX_PIXELS"] = "50176"159os.environ["FPS_MAX_FRAMES"] = "12"160 161engine = VllmEngine(162    "dipta007/VCInspector-7B",163    max_model_len=32768,164    limit_mm_per_prompt={"image": 32}165)166 167# Prepare request168request = InferRequest(169    messages=[{"role": "user", "content": f"<image>\n{prompt}"}],170    images=["frame1.jpg", "frame2.jpg", ...]  # Video frames171)172config = RequestConfig(max_tokens=256, temperature=0.0)173response = engine.infer([request], config)174print(response[0].choices[0].message.content)175```176 177## Output Format178 179VC-Inspector outputs two components:180 1811. **Quality Score** (Line 1): Integer from 1-5182   - 5: Caption is accurate and comprehensive183   - 4: Minor factual errors184   - 3: Moderate factual errors185   - 2: Significant factual errors186   - 1: Major factual errors or completely incorrect187 1882. **Explanation** (Line 2+): Natural language explanation identifying:189   - Incorrect objects (e.g., "guitar" instead of "violin")190   - Incorrect actions (e.g., "running" instead of "walking")191 192## Training Details193 194| Hyperparameter | Value |195|:---------------|:------|196| Base Model | Qwen2.5-VL-7B-Instruct |197| Training Data | ActivityNet-FG-It (44K samples) |198| Epochs | 1 |199| Global Batch Size | 128 |200| Learning Rate | 1e-4 |201| LR Scheduler | Cosine (min: 1e-5) |202| LoRA Rank | 32 |203| LoRA Alpha | 32 |204| LoRA Dropout | 0.05 |205| Number of Frames | 32 |206| Training Time | ~32 GPU hours (A100) |207 208## Limitations209 210- Primarily targets object and action correctness; attributes, spatial relationships, and fine-grained temporal ordering are not explicitly modeled211- Training relies on synthetically generated captions and pseudo-scores212- Higher computational cost compared to embedding-based metrics (though more lightweight than GPT-4o)213 214## Citation215 216If you find this work useful, please cite our paper:217 218```bibtex219@inproceedings{dipta2026vcinspector,220  title={VC-Inspector: Advancing Reference-free Evaluation of Video Captions with Factual Analysis},221  author={Shubhashis Roy Dipta and Tz-Ying Wu and Subarna Tripathi},222  booktitle={Proceedings of the Association for Computational Linguistics: ACL 2026},223  year={2026},224  eprint={2509.16538},225  archivePrefix={arXiv},226  primaryClass={cs.CV},227  url={https://arxiv.org/abs/2509.16538},228}229```230 231## Acknowledgements232 233This work builds upon [Qwen2.5-VL](https://github.com/QwenLM/Qwen2.5-VL) and uses [ms-swift](https://github.com/modelscope/ms-swift) for training.234