jiang-cc/AD-Copilot

1---2library_name: transformers3license: apache-2.04tags:5  - anomaly-detection6  - vision-language-model7  - qwen2.5-vl8pipeline_tag: image-text-to-text9language:10  - en11base_model:12  - Qwen/Qwen2.5-VL-7B-Instruct13---14 15# AD-Copilot16 17Comparison-aware anomaly detection with vision-language models. Extends Qwen2.5-VL-7B with a novel **comparison-aware visual encoder** achieving **78.74%** on OmniDiff benchmark.18 19[Paper](https://arxiv.org/abs/2603.13779v1) | [Code](https://github.com/jam-cc/AD-Copilot) | [Demo](https://huggingface.co/spaces/jiang-cc/AD-Copilot)20 21## Key Innovation22 23- **ADCopilotCompareVisualEncoder**: Bidirectional cross-attention comparing reference and test images24- **100 comparison tokens** per image pair injected into the language model25- State-of-the-art on industrial anomaly detection benchmarks26 27## Quick Start28 29```python30import torch31from transformers import AutoModelForVision2Seq, AutoProcessor32from qwen_vl_utils import process_vision_info33 34model = AutoModelForVision2Seq.from_pretrained(35    "jiang-cc/AD-Copilot",36    torch_dtype=torch.bfloat16, device_map="auto", trust_remote_code=True,37)38processor = AutoProcessor.from_pretrained(39    "jiang-cc/AD-Copilot",40    min_pixels=64*28*28, max_pixels=1280*28*28, trust_remote_code=True,41)42 43messages = [{"role": "user", "content": [44    {"type": "image", "image": "good.png"},45    {"type": "image", "image": "test.png"},46    {"type": "text", "text": "The first image is good. Is there any anomaly in the second image? A.yes, B.no."},47]}]48 49text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)50image_inputs, _ = process_vision_info(messages)51inputs = processor(text=[text], images=[image_inputs], return_tensors="pt").to(model.device)52 53with torch.inference_mode():54    ids = model.generate(**inputs, max_new_tokens=128, do_sample=False)55trimmed = [o[len(i):] for i, o in zip(inputs.input_ids, ids)]56print(processor.batch_decode(trimmed, skip_special_tokens=True)[0])57```58 59## Results (OmniDiff Benchmark)60 61| Model | Visited IAD | Avg ACC |62|-------|-------------|---------|63| MiniCPM-V2.6 | 0 | 67.90% |64| EIAD | 128k | 69.40% |65| Qwen2.5-VL | 0 | 72.19% |66| **AD-Copilot (Ours)** | **206k** | **78.74%** |67 68## Architecture69 70- **Base**: Qwen2.5-VL-7B-Instruct (28 layers, 3584 hidden)71- **Vision**: Qwen2.5-VL ViT (32 layers, 1280 hidden)72- **Comparison Encoder**: Bidirectional cross-attention + query decoder (100 tokens)73- **Dtype**: bfloat1674 75## Citation76 77```bibtex78@article{adcopilot2025,79  title={AD-Copilot: Comparison-Aware Anomaly Detection with Vision-Language Models},80  author={Jiang, Xi and others},81  journal={arXiv preprint arXiv:2603.13779},82  year={2025}83}84```85