Modelpublic

useful-quants/Phi-4-mini-instruct-W4A16

sourceHugging Facemitupdated 5mo agoView on Hugging Face

0likes21downloads

Model Card

Phi-4-mini-instruct W4A16

Ampere-friendly serving build of `microsoft/Phi-4-mini-instruct`. Text-side linears are compressed-tensors W4A16.

Stock proof

bash

docker run --rm -it \
  --gpus all \
  --ipc=host \
  -p 8001:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest \
  microsoft/Phi-4-mini-instruct \
  --served-model-name Phi-4-mini-instruct-stock \
  --dtype bfloat16 \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.7

Serve the packaged artifact

bash

docker run --rm -it \
  --gpus all \
  --ipc=host \
  -p 8002:8000 \
  -v /path/to/Phi-4-mini-instruct-W4A16:/model \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest \
  /model \
  --served-model-name Phi-4-mini-instruct-W4A16 \
  --dtype bfloat16 \
  --quantization compressed-tensors \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.7

Smoke test

bash

python verify.py --url http://localhost:8002/v1/completions

Notes

—Best fit: RTX 30xx/40xx Ampere cards.
—The package is text-only and stays compatible with clean vLLM.