Modelpublic

useful-quants/Mistral-NeMo-Minitron-8B-Instruct-W4A16

sourceHugging Faceotherupdated 5mo agoView on Hugging Face

0likes15downloads

Model Card

Mistral-NeMo-Minitron-8B-Instruct W4A16

Ampere-friendly serving build of `nvidia/Mistral-NeMo-Minitron-8B-Instruct`. Text-side linears are compressed-tensors W4A16.

Stock proof

bash

docker run --rm -it \
  --gpus all \
  --ipc=host \
  -p 8001:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest \
  nvidia/Mistral-NeMo-Minitron-8B-Instruct \
  --served-model-name Mistral-NeMo-Minitron-8B-Instruct-stock \
  --dtype bfloat16 \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.76

Use

bash

docker run --rm -it \
  --gpus all \
  --ipc=host \
  -p 8001:8000 \
  -v /path/to/Mistral-NeMo-Minitron-8B-Instruct-W4A16:/model \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest \
  --model /model \
  --served-model-name Mistral-NeMo-Minitron-8B-Instruct-W4A16 \
  --dtype bfloat16 \
  --quantization compressed-tensors \
  --max-model-len 4096

Smoke test

bash

python verify.py --url http://localhost:8001/v1/completions

Notes

—Best fit: RTX 30xx/40xx Ampere cards.
—The repo includes the calibration corpus used for verification.