CoolFace
Modelpublic

robertzty/Cosmos-Reason2-32B-GGUF

sourceHugging Faceotherupdated 5mo agoView on Hugging Face
0likes48downloads
Model Card

Cosmos-Reason2-32B GGUF

Pure GGUF conversion of nvidia/Cosmos-Reason2-32B.

Built on NVIDIA Cosmos.

Files

  • —Cosmos-Reason2-32B-BF16.gguf: BF16 text backbone GGUF.
  • —Cosmos-Reason2-32B-Q4_K_M.gguf: smaller 4-bit text backbone GGUF for lower memory use.
  • —Cosmos-Reason2-32B-Q5_K_M.gguf: balanced 5-bit text backbone GGUF with better quality than Q4.
  • —Cosmos-Reason2-32B-Q8_0.gguf: larger 8-bit text backbone GGUF for higher quality.
  • —mmproj-Cosmos-Reason2-32B-F16.gguf: F16 multimodal projector / vision GGUF.

Use one text backbone file together with the mmproj file for multimodal inference.

Hardware estimates

These are rough inference estimates for llama.cpp with batch size 1. Actual memory use depends on context length, image/video inputs, backend, and how many layers are offloaded to GPU.

Text backboneFile sizeText + mmprojSuggested system RAMSuggested VRAM for mostly/full GPU offloadNotes
Q4_K_M19.8 GB21.0 GB32 GB minimum, 48 GB comfortable24 GB tight, 32 GB comfortableBest first choice for local use.
Q5_K_M23.2 GB24.4 GB48 GB comfortable32 GB comfortableBetter quality than Q4 with moderate extra memory.
Q8_034.8 GB36.0 GB64 GB comfortable48 GB+ recommendedHigher quality, much larger.
BF1665.5 GB66.7 GB96 GB+ recommended80 GB+ or multi-GPUOriginal precision GGUF; not a practical default for most local machines.

KV cache adds roughly 2 GiB per 8k text tokens at fp16 cache precision, before additional image/video token overhead. Reduce --ctx-size or use partial CPU/GPU offload if memory is tight.

Source

Original model: https://huggingface.co/nvidia/Cosmos-Reason2-32B

This GGUF conversion was produced with llama.cpp convert_hf_to_gguf.py from the original Hugging Face safetensors.

Usage

Use one text backbone file together with the multimodal projector in llama.cpp:

bash
llama-server \
  -m Cosmos-Reason2-32B-Q4_K_M.gguf \
  --mmproj mmproj-Cosmos-Reason2-32B-F16.gguf

BF16 and Q8_0 are large and may require CPU offload or a multi-GPU setup.

License

Licensed by NVIDIA Corporation under the NVIDIA Open Model License.

See NOTICE and the original model card for license terms and usage requirements.