Modelpublic

tinyopsec/bible-assistant-qwen35-4b-GGUF

sourceHugging Facemitupdated 8d agoView on Hugging Face

0likes2.1kdownloads

Model Card

Bible Assistant Qwen3.5-4B GGUF

GGUF quantized versions of Ttimms/Bible-Assistant-Qwen3.5-4B-v2.

Quantization

Quant	Bits	Size	Use Case
Q8_0	8	~4.2 GB	Maximum quality, slower
Q6_K	6	~3.1 GB	High quality
Q5KM	5	~2.6 GB	Balanced quality/speed
Q5KS	5	~2.4 GB	Balanced quality/speed
Q4KM	4	~2.0 GB	Good quality/speed trade-off
Q4KS	4	~1.9 GB	Good quality/speed trade-off
Q3KL	3	~1.6 GB	Small, acceptable quality
Q3KM	3	~1.5 GB	Small, acceptable quality
Q3KS	3	~1.4 GB	Small, acceptable quality
Q2_K	2	~1.1 GB	Minimal, fast

VRAM Requirements

Quantization	VRAM	Notes
F16	~8 GB	Full precision
Q8_0	~4.5 GB	Minimal overhead
Q6_K	~3.5 GB	Minimal overhead
Q5KM	~3.0 GB	Minimal overhead
Q4KM	~2.5 GB	Minimal overhead
Q3KM	~2.0 GB	Minimal overhead
Q2_K	~1.5 GB	Minimal overhead

Usage

llama.cpp

bash

./main -m model_q5_k_m.gguf -p "Your prompt here" -n 256

llama-cpp-python

python

from llama_cpp import Llama

llm = Llama(model_path="model_q5_k_m.gguf", n_gpu_layers=-1)
output = llm("Your prompt here", max_tokens=256)
print(output["choices"][0]["text"])

LM Studio

Load any .gguf file directly in LM Studio.

Ollama

bash

ollama create bible-assistant-qwen35 -f Modelfile

Where Modelfile contains:

FROM model_q5_k_m.gguf
PARAMETER temperature 0.7

Original Model

Ttimms/Bible-Assistant-Qwen3.5-4B-v2 — A 4B parameter model fine-tuned for Bible-related assistance and knowledge.

License

Refer to the original model's license.