Modelpublic

useful-quants/InternVL3_5-4B-Instruct-W4A16-BF16Vision

sourceHugging Faceapache-2.0updated 5mo agoView on Hugging Face

0likes13downloads

Model Card

InternVL3_5-4B-Instruct W4A16 BF16-Vision

Ampere-friendly serving build of `OpenGVLab/InternVL3_5-4B-Instruct`. Text-side linears are compressed-tensors W4A16. The vision tower and multimodal connector stay BF16.

Stock proof

bash

docker run --rm -it \
  --gpus all \
  --ipc=host \
  -p 8001:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest \
  OpenGVLab/InternVL3_5-4B-Instruct \
  --served-model-name InternVL3_5-4B-Instruct-stock \
  --trust-remote-code \
  --dtype bfloat16 \
  --max-model-len 4096

Use

bash

docker run --rm -it \
  --gpus all \
  --ipc=host \
  -p 8001:8000 \
  -v /path/to/InternVL3_5-4B-Instruct-W4A16-BF16Vision:/model \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest \
  --model /model \
  --served-model-name InternVL3_5-4B-Instruct-W4A16-BF16Vision \
  --trust-remote-code \
  --dtype bfloat16 \
  --quantization compressed-tensors \
  --max-model-len 4096 \
  --limit-mm-per-prompt '{"image":1}' \
  --allowed-local-media-path /

Smoke test

bash

python verify.py --url http://localhost:8001/v1/chat/completions

Notes

—Built for RTX 30xx/40xx Ampere cards.
—Vision stays BF16 on disk and at runtime.
—The repo includes the calibration corpus and local smoke_image.png asset used for verification; the image test uses that file directly.