Modelpublic

kd13/Type-o1-mini-instruct-GGUF

sourceHugging Facemitupdated 2mo agoView on Hugging Face

0likes264downloads

Model Card

Type-o1-mini-instruct - GGUF

GGUF quantizations of kd13/Type-o1-mini-instruct, a compact general-purpose instruct model (~1B parameters) for everyday assistant use.

Converted with llama.cpp. The IQ quant was produced with an importance matrix; the rest are static quants.

Read the Usage section before running these files. This model uses a custom chat template, so llama.cpp requires the --jinja flag. Without it you will get this custom template is not supported.

Provided quants

Sorted by size, which is not the same as sorted by quality. IQ-quants are often preferable to non-IQ quants of a similar size.

Link	Type	Size/GB	Notes
GGUF	Q2_K	0.6
GGUF	Q3KS	0.6
GGUF	Q3KM	0.7	lower quality
GGUF	Q3KL	0.7
GGUF	IQ4_XS	0.7
GGUF	Q4KS	0.8	fast, recommended
GGUF	Q4KM	0.8	fast, recommended
GGUF	Q5KS	0.9
GGUF	Q5KM	0.9
GGUF	Q6_K	1.0	very good quality
GGUF	Q8_0	1.3	fast, best quality
GGUF	f16	2.5	16 bpw, overkill

Which one should I pick?

For a model this small the practical range is Q4_K_M through Q8_0. A 1B model has little redundancy to give up, so the very low-bit quants lose more than they would on a 7B. Q2_K and Q3_K_S are included for completeness rather than as recommendations.

Chat template

This model was fine-tuned on a custom template, not the standard Llama 3 header format. Each message is wrapped as:

''' <|beginoftext|>{role} {content}<|endoftext|> '''

and generation is prompted with a trailing <|begin_of_text|>assistant\n. The full Jinja template is embedded in every GGUF file in this repo, so any runtime with Jinja support applies it automatically.

Because this format is not one of llama.cpp's built-in recognised templates, its C++ template matcher will reject it. Passing --jinja tells llama.cpp to use the embedded Jinja template instead, which is what you want.

Usage

llama.cpp

bash

llama-completion -m Type-o1-mini-instruct.Q4_K_M.gguf --jinja \
  -sys "You are a helpful assistant." \
  -p "Explain photosynthesis in two sentences."

Recent llama.cpp builds renamed llama-cli to llama-completion; on older builds use llama-cli with the same flags. For raw text completion with no template applied at all, add -no-cnv and drop --jinja.

Server:

bash

llama-server -m Type-o1-mini-instruct.Q4_K_M.gguf --jinja -c 4096

Omitting --jinja produces this custom template is not supported — that is a template-matching error, not a corrupt file.

Ollama

bash

ollama run hf.co/kd13/Type-o1-mini-instruct-GGUF:Q4_K_M