hell0ks/NVIDIA-Nemotron-3-Ultra-550B-A55B-GGUF
0201
NVIDIA-Nemotron-3-Ultra-550B-A55B-GGUF
GGUF quantized version of nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16.
How can I use?
- Upstream llama.cpp: Model probably works.
- Forked version of llama.cpp: RPC with split mode tensor, MTP, and model works
Detailed information
- Model: 161667.64 MiB (2.47 BPW)
- MTP: 8882.72 MiB (5.58 BPW)
How it was made?
Model
export TYPE_FFN_UP_EXPS=IQ2_XXS
export TYPE_FFN_DOWN_EXPS=IQ2_XS
export TYPE_LATENT=Q8_0
export TYPE_SHEXP=Q6_K
export TYPE_SSM=Q6_K
export TYPE_ATTN=Q6_K
export TYPE_DEFAULT=Q4_K_M
llama-quantize \
--tensor-type 'ffn_up_exps='$TYPE_FFN_UP_EXPS \
--tensor-type 'ffn_down_exps='$TYPE_FFN_DOWN_EXPS \
--tensor-type 'ffn_latent_(up|down)='$TYPE_LATENT \
--tensor-type 'ffn_(up|down)_shexp='$TYPE_SHEXP \
--tensor-type 'ssm_(in|out)='$TYPE_SSM \
--tensor-type 'attn_(q|k|v|output)='$TYPE_ATTN \
--imatrix imatrix.gguf \
"model-BF16.gguf" "$TYPE_DEFAULT"MTP
llama-quantize \
--tensor-type 'nextn\.eh_proj=Q6_K' \
--tensor-type 'ffn_latent_(up|down)=Q8_0' \
--tensor-type 'ffn_(up|down)_shexp=Q6_K' \
--tensor-type 'attn_(q|k|v|output)=Q6_K' \
"mtp-BF16.gguf" "mtp-quant.gguf" Q4_K_Mimatrix
- Generate Korean-focused imatrix calibration dataset (Total 512 samples, Chat template applied for messages)
- Create imatrix using llama-imatrix with --parse-special flag
Benchmark
Where is PPL/KLD?
Unfortunately, I don't have PPL/KLD for quant, due to out of compute resource and target budget.
Throughtput
2x DGX Sparks, RPC, -sm tensor with MTP enabled (-ub 2048 -b 2048 -c 262144)