HaoyiSun/Switch-KD-Qwen2.5-CLIP-1.8B

Switch-KD-Qwen2.5-CLIP-1.8B

Switch-KD-Qwen2.5-CLIP-1.8B is a compact vision-language model (VLM) trained using the Switch-KD (Visual-Switch Knowledge Distillation) framework from Li Auto's MindKD technology. This model achieves competitive performance on multimodal benchmarks while being efficient for deployment.

Model Details

• —Base Model: Qwen2.5-1.5B-Instruct
• —Visual Encoder: CLIP-ViT-L/14-336
• —Projector: LDPNetV2 (FeatureIRLayer + TokenDownLayer + PosInjectLayer)
• —Total Parameters: ~1.8B
• —Image Resolution: 336×336
• —Context Length: 32,768 tokens
• —Training Method: Switch-KD distillation with DBiLD Loss
• —License: Apache 2.0

Architecture

Image (336×336) → CLIP ViT-L/14 (576 tokens) → LDPNetV2 Projector (144 tokens) → Qwen2.5-1.5B

The model uses a custom architecture where:

1. 1.CLIP Vision Encoder extracts visual features at 336×336 resolution
2. 2.LDPNetV2 Projector reduces visual tokens from 576 to 144 while preserving information
3. 3.Qwen2.5-1.5B processes both visual and textual inputs for generation

Key Results

Switch-KD demonstrates significant improvements over baseline VLM distillation methods:

vs. Align-KD (1.5B Models)

• —+4.4% average improvement across 6 benchmarks
• —Uses only 1/3 the training data (1.2M vs 3.6M samples)

Benchmark Performance (Selected)

For detailed results, see the Switch-KD paper.

Installation

pip install transformers accelerate torch

Quickstart

Command Line Interface

# Single-round inference
python chat.py --model HaoyiSun/Switch-KD-Qwen2.5-CLIP-1.8B \
    --image path/to/image.jpg \
    --question "Please describe this picture."

# Interactive multi-round chat
python chat.py --model HaoyiSun/Switch-KD-Qwen2.5-CLIP-1.8B \
    --image path/to/image.jpg \
    --interactive

# With custom settings
python chat.py --model HaoyiSun/Switch-KD-Qwen2.5-CLIP-1.8B \
    --image path/to/image.jpg \
    --interactive \
    --max-new-tokens 1024 \
    --torch-dtype fp16

Model Architecture

Visual Encoder (CLIP-ViT-L/14-336)

• —Hidden size: 1024
• —Layers: 24
• —Attention heads: 16
• —Image size: 336×336
• —Patch size: 14×14

Projector (LDPNetV2)

• —Projects 1024-dim visual features to 1536-dim LLM space
• —Reduces spatial tokens from 576 to 144
• —Components: FeatureIRLayer → TokenDownLayer → PosInjectLayer

Language Model (Qwen2.5-1.5B)

• —Hidden size: 1536
• —Layers: 28
• —Attention heads: 12 (2 key-value heads for GQA)
• —Context length: 32,768 tokens
• —Vocabulary: 151,936 tokens

Training

Switch-KD is trained using two key innovations:

1. 1.Visual-Switch Distillation: Switches student visual outputs into teacher language pathway for cross-modal knowledge transfer

1. 1.DBiLD Loss: Dynamic Bi-directional Logits Difference loss with adaptive top-K selection via Kneedle algorithm

Training Configuration

• —Training data: 1.2M image-text pairs
• —Optimizer: AdamW with cosine learning rate schedule
• —Batch size: 64 per GPU
• —Training epochs: Varies by configuration

Limitations

• —The model is primarily trained on English datasets and may have reduced performance on other languages
• —Best performance on images similar to training distribution (natural images, documents, charts)
• —May struggle with very low-resolution or extremely high-resolution images
• —Designed for single-image understanding (not optimized for video)

Citation

If you use this model, please cite:

@article{sun2026switchkd,
  title={Switch-KD: Visual-Switch Knowledge Distillation for Vision-Language Models},
  author={Sun, Haoyi and Wang, Xiaoxiao and Mao, Ning and Wang, Qian and Mu, Lifu and Zheng, Wen and Wei, Tao and Chen, Wei},
  journal={arXiv preprint arXiv:2604.14629},
  year={2026}
}

Acknowledgments

This model is built upon excellent open-source work:

• —Qwen2.5 - Language model backbone
• —CLIP - Visual encoder
• —xtuner - VLM training framework

Contact

For questions about this model or Switch-KD framework:

• —Work: sunhaoyi@lixiang.com
• —Personal: haoyi199815@126.com

Model Card Authors: Haoyi Sun et al.

Model Card Contact: sunhaoyi@lixiang.com