MuhammedKsee/Qwen3.5-9B-Instruct-Turca-TurkishLLM-GGUF

Qwen3.5-9B-Instruct-Turca-TurkishLLM — GGUF 🇹🇷⚡

Quantized GGUF versions of muhammedksee/Qwen3.5-9B-Instruct-Turca-TurkishLLM for local inference on consumer hardware — no data center GPU required.

Looking for the full-precision model? Training details, hyperparameters, and benchmark results are in the base repository.

Developer

Available Quantizations

Why Q4KM?

Running a 9B-parameter model at full precision (BF16) requires a data center-grade GPU. GGUF quantization compresses the model weights with minimal quality loss, making it fit in consumer RAM and VRAM.

Q4KM retains nearly all of the model's Turkish language understanding and reasoning capability while cutting memory usage roughly in half. It is the community standard for everyday use and local RAG pipelines.

System Requirements

GPU offloading is optional but strongly recommended — with a supported GPU you can reach tens of tokens/second instead of single digits on CPU.

Quick Start

Option 1 — Ollama

Preferred for terminal access or running a local API server.

Step 1. Download the GGUF file.

Step 2. Create a file named Modelfile (no extension) in the same directory:

FROM ./Qwen3.5-9B-InstrucTurca-Q4_K_M.gguf

TEMPLATE """<|im_start|>system
{{ .System }}<|im_end|>
<|im_start|>user
{{ .Prompt }}<|im_end|>
<|im_start|>assistant
"""

SYSTEM """Sen yardımcı, dürüst ve zararsız bir Türkçe yapay zeka asistanısın."""

Step 3. Build and run:

ollama create turca-9b -f Modelfile
ollama run turca-9b

Option 2 — Python / llama-cpp-python

For integrating the model into your own Python projects (local RAG, automation agents, etc.).

pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama(
    model_path="./Qwen3.5-9B-InstrucTurca-Q4_K_M.gguf",
    n_ctx=4096,       # context window
    n_threads=8,      # CPU threads — match your core count
    n_gpu_layers=35   # increase if you have more VRAM
)

system_prompt = "Sen uzman bir yazılım mühendisi ve yapay zeka asistanısın."
user_prompt   = "Python'da liste üreteçleri (list comprehensions) nedir? Kısa bir örnek ver."

prompt = (
    f"<|im_start|>system\n{system_prompt}<|im_end|>\n"
    f"<|im_start|>user\n{user_prompt}<|im_end|>\n"
    f"<|im_start|>assistant\n"
)

output = llm(
    prompt,
    max_tokens=512,
    temperature=0.7,
    top_p=0.9,
    stop=["<|im_end|>"]
)

print(output["choices"][0]["text"])

Prompt Format (ChatML)

This model uses the standard Qwen ChatML format. Always follow this template when calling the model directly — deviating from it increases hallucination risk and degrades output quality.

<|im_start|>system
Sen Türkçe konuşan, yardımcı bir asistansın.<|im_end|>
<|im_start|>user
Merhaba, nasılsın?<|im_end|>
<|im_start|>assistant

The system prompt can be in Turkish or English. Turkish system prompts are recommended as they better align with the fine-tuning distribution.

Intended Use

Limitations

Quantization loss. Q4KM is a 4-bit mathematical compression. While the quality difference is minimal for most tasks, very fine-grained Turkish grammar nuances or highly complex math/coding problems may show slight degradation compared to the BF16 base model.

No benchmark results yet. Formal evaluations (MT-Bench TR, MMLU-TR, base model comparison) are in progress. See the base repository for updates.

SFT only. No DPO or RLHF alignment stage was applied to the base model. Responses may occasionally be less reliable than preference-aligned models.

Test on your specific workload before production deployment.

Related Resources

Citation

@misc{kose2025qwen35turca_gguf,
  author    = {Muhammed Köse},
  title     = {Qwen3.5-9B-Instruct-Turca-TurkishLLM GGUF},
  year      = {2026},
  publisher = {Hugging Face},
  url       = {https://huggingface.co/muhammedksee/Qwen3.5-9B-Instruct-Turca-TurkishLLM-GGUF}
}