Modelpublic

RockMan256/granite-docling-258M-GGUF

sourceHugging Faceupdated 3mo agoView on Hugging Face

0likes22downloads

Model Card

Granite Docling 258M GGUF

GGUF conversion of RockMan256/granite-docling-258M, a fine-tuned version of ibm-granite/granite-docling-258M.

Files

File	Description
`granite-docling-258M-bf16.gguf` (317 MB)	LLM weights in BF16
`mmproj-model-f16.gguf` (182 MB)	Vision projector (mmproj) in F16

Usage with llama.cpp

bash

# Build llama.cpp (if not already built)
cd llama.cpp && make -j

# Run with multimodal (image + text) inference
./llama-mtmd-cli \
  -m granite-docling-258M-bf16.gguf \
  --mmproj mmproj-model-f16.gguf \
  --chat-template chatml

Example prompts

bash

# Describe an image
echo "Describe this image in detail." | ./llama-mtmd-cli \
  -m granite-docling-258M-bf16.gguf \
  --mmproj mmproj-model-f16.gguf \
  -p /path/to/image.png

# Text-only inference (no vision)
echo "What is the capital of France?" | ./llama-mtmd-cli \
  -m granite-docling-258M-bf16.gguf \
  -p - --chat-template chatml

Conversion Details

—LLM: Converted from user's fine-tuned RockMan256/granite-docling-258M (original weights, not remote)
—mmproj: Converted from ibm-granite/granite-docling-258M (vision encoder/connector unchanged from base)
—Quantization: BF16 (LLM), F16 (mmproj)

License

Apache 2.0 (same as base model)