Modelpublic

ConstDemi/vikhr-nemo-12b-instruct-r-awq

sourceHugging Faceapache-2.0updated 1mo agoView on Hugging Face

0likes330downloads

Model Card

Vikhr-Nemo-12B-Instruct-R — AWQ (W4A16)

4-bit AWQ quantization of Vikhrmodels/Vikhr-Nemo-12B-Instruct-R-21-09-24, a Russian-language grounded-RAG model. Produced with llm-compressor in the compressed-tensors format for vLLM.

Quantization details

—Method: AWQ, scheme W4A16, lm_head kept unquantized
—Tool: llm-compressor (AWQModifier, one-shot)
—Calibration: 256 samples x 2048 tokens from GrandMaster-PRO-MAX, rendered through the model chat template

Usage

Ampere and newer (sm80+): dtype is picked from config automatically

bash

vllm serve ConstDemi/vikhr-nemo-12b-instruct-r-awq

Turing (sm75, e.g. Quadro RTX 6000): no native bfloat16

bash

vllm serve ConstDemi/vikhr-nemo-12b-instruct-r-awq \
    --dtype float16

Verified on vLLM 0.25.1, sm75 GPU, --dtype float16. The model follows the two-step grounded generation protocol of the original Vikhr-R (document selection as JSON, then the answer); see the original model card for the prompt format.

Limitations

—No perplexity or benchmark-based quality evaluation was performed; the checkpoint was validated on an internal Russian grounded-RAG task (answer quality, source citation, and the JSON document-selection step).
—Calibrated on general instruction data, not domain-specific text.
—bf16 scale tensors: use --dtype float16 on GPUs without native bf16 support.