Modelpublic

ops-malware/smollm2-1.7b-abliterated-GGUF

sourceHugging Faceapache-2.0updated 2mo agoView on Hugging Face

0likes154downloads

Model Card

smollm2-1.7b-abliterated-GGUF

GGUF builds of ops-malware/smollm2-1.7b-abliterated, for llama.cpp, Ollama, LM Studio and Jan.

The parent card carries what this model is, how it was made, what abliteration did to it, and the evaluation numbers. Read it before using these weights: this model does not refuse, which is the entire point of it and the thing to understand before downloading.

Files

File	Precision	Size	Use when
`smollm2-1.7b-f16.gguf`	F16	larger	You want the conversion with no quantisation loss, or you are making your own quants
`smollm2-1.7b-Q4_K_M.gguf`	Q4KM	~4x smaller	Almost always. The usual quality and size compromise

Both were converted from the parent's safetensors with convert_hf_to_gguf.py and quantised with llama-quantize. Each file was loaded and asked to generate before publication, because a GGUF that converts but does not run is exactly the kind of thing that ships broken.

Usage

llama.cpp

bash

llama-server -m smollm2-1.7b-Q4_K_M.gguf -c 4096

Ollama

bash

ollama run hf.co/ops-malware/smollm2-1.7b-abliterated-GGUF:Q4_K_M

Python, via huggingface_hub

python

from huggingface_hub import hf_hub_download

path = hf_hub_download(
    repo_id="ops-malware/smollm2-1.7b-abliterated-GGUF",
    filename="smollm2-1.7b-Q4_K_M.gguf",
)

Limitations

Everything on the parent card applies here unchanged, plus the usual quantisation caveat: Q4KM trades some quality for size, and small models have less quality to spare than large ones. If a result matters, check it against the F16.