Modelpublic

useful-quants/Qwen3-4B-Instruct-2507-W4A16-G128

sourceHugging Faceapache-2.0updated 5mo agoView on Hugging Face

0likes24downloads

Model Card

Qwen3-4B-Instruct-2507 W4A16 G128

Ampere-friendly serving build of `Qwen/Qwen3-4B-Instruct-2507`. Text-side linears are compressed-tensors W4A16 with group size 128.

Stock proof

bash

docker run --rm -it \
  --gpus all \
  --ipc=host \
  -p 8001:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest \
  Qwen/Qwen3-4B-Instruct-2507 \
  --served-model-name Qwen3-4B-Instruct-2507-stock \
  --dtype bfloat16 \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.7

Serve the packaged artifact

bash

docker run --rm -it \
  --gpus all \
  --ipc=host \
  -p 8002:8000 \
  -v /path/to/Qwen3-4B-Instruct-2507-W4A16-G128:/model \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest \
  /model \
  --served-model-name Qwen3-4B-Instruct-2507-W4A16-G128 \
  --dtype bfloat16 \
  --quantization compressed-tensors \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.7

Smoke test

bash

python verify.py --url http://localhost:8002/v1/chat/completions

Notes

—Best fit: RTX 30xx/40xx Ampere cards.
—The package is text-only and stays compatible with clean vLLM.