Ttimms/Bible-Assistant-Qwen3.5-4B-v2-GGUF

Superseded (2026-09-05). `Bible-Assistant-Qwen3.5-4B-v3.2-GGUF` is the current GGUF release -- same size class, better on every metric tested. This v2 quant set is kept for reproducibility, not as the recommended download.

Bible AI Assistant v2-4b — GGUF

GGUF quants of `Ttimms/Bible-Assistant-Qwen3.5-4B-v2` — a Qwen3.5-4B SFT for retrieval-grounded Bible Q&A. See the base repo for the full model card, training details, and the honest evaluation (this is an interim checkpoint: strong on verbatim verse recall, weaker on open-ended thematic answers; a v3 with teacher-distilled answers + GRPO is planned).

Architecture

graph TD
    Base["Qwen/Qwen3.5-4B"]
    SFT["bf16 LoRA SFT - 56k-example dataset"]
    Merge["merge adapter -> bf16"]
    Conv["convert_hf_to_gguf --no-mtp + llama-quantize (+imatrix)"]
    ST["Bible-Assistant-Qwen3.5-4B-v2 (safetensors)"]
    GG["...-v2-GGUF (Q4_K_M / Q5_K_M / Q6_K / Q8_0 / F16 +imat)"]
    RAG["hybrid RAG: dense (nomic) + BM25 + RRF + bge-reranker-v2-m3"]
    LLM["Ollama / llama.cpp"]
    Base --> SFT --> Merge --> ST
    Merge --> Conv --> GG
    ST --> RAG --> LLM
    GG --> LLM

Download

Grab one file, not the whole repo. -imat files use an importance matrix (better quality at the same size).

Run it in

• —llama.cpp — llama-server -m <file>.gguf -ngl 99
• —LM Studio (bundles a recent llama.cpp)
• —koboldcpp
• —Jan
• —text-generation-webui
• —Ollama — once its bundled llama.cpp includes this arch (see Requirements)

Requirements

Qwen3.5 is a hybrid architecture (`qwen35` / Gated-DeltaNet + attention). You need a recent llama.cpp — a build that includes the qwen35 hybrid arch (commit 3173a56 or newer). Verified working with llama-server from a source build.

• —✅ llama.cpp (current): llama-server -m bible-v2-4b-Q4_K_M.gguf -ngl 99
• —✅ LM Studio (recent versions bundle a current llama.cpp)
• —⚠️ Ollama 0.33.x: the bundled llama.cpp is too old for the qwen35 arch (check_tensor_dims: tensor 'blk.32.attn_norm.weight' not found). Use once Ollama updates its runtime, or run llama.cpp directly.

Thinking mode

The Qwen3.5 chat template defaults to thinking on. This model was SFT'd without <think> traces, so for a grounded RAG assistant you want it off:

• —llama.cpp `/v1/chat/completions`: pass "chat_template_kwargs": {"enable_thinking": false}, or
• —use a chat template that emits a closed empty <think>\n\n</think>\n\n after <|im_start|>assistant\n (the project's deployment/pc/Modelfile does this).

Intended use

Retrieval-augmented Bible Q&A — the model expects retrieved verses in a Context: block, then the question. It is not designed for context-free use, medical / legal / financial advice, counselling (it redirects those to a pastor / crisis line), or authoritative theological rulings.

License

Weights: Apache-2.0 (inherits from Qwen3.5-4B). Project code: MIT. Bible translations: public domain. See the base repo for full attribution.