ihumaunkabir/mistral_bangla_q4_k_m_gguf

mistral-bangla (GGUF, Q4KM)

A standalone, ready-to-run quantized Mistral-7B model adapted for Bangla (Bengali). The LoRA adapters were merged into the base weights and quantized to Q4_K_M GGUF format -- no base model loading required, just point Ollama / llama.cpp at the file and run.

Trained via a two-stage recipe: continual pre-training (CPT) on Bangla Wikipedia, then supervised fine-tuning (SFT) on the `ihumaunkabir/alpaca-gpt4-bangla` instruction dataset, using Unsloth.

Looking for the trainable adapters? The PEFT LoRA adapters (to continue fine-tuning or re-quantize) live at `ihumaunkabir/mistral_bangla_lora`.

How to run

Option A -- Ollama (easiest)

# Pull directly from Hugging Face
ollama pull hf.co/ihumaunkabir/mistral_bangla_q4_k_m_gguf:Q4_K_M

# Run interactively
ollama run hf.co/ihumaunkabir/mistral_bangla_q4_k_m_gguf:Q4_K_M

Or build a local Modelfile with the Alpaca template (see "Prompt format" below) and the system prompt of your choice, then:

ollama create mistral-bangla -f Modelfile
ollama run mistral-bangla

Option B -- llama.cpp

# Clone + build llama.cpp, then:
./llama-cli \
  -m mistral-7b-v0.3.Q4_K_M.gguf \
  -p "নিচে একটি নির্দেশনা দেওয়া আছে..." \
  -n 128 --color

Option C -- LM Studio

Download the .gguf file and load it through the GUI. No conversion needed.

Prompt format

This model was trained on the Alpaca template, translated to Bangla. At inference time, leave the response blank and let the model generate:

নিচে একটি নির্দেশনা দেওয়া আছে, যা একটি কাজের বর্ণনা দেয়। অনুরোধটি যথাযথভাবে সম্পূর্ণ করে একটি উত্তর লিখুন।

### নির্দেশনা:
<instruction here>

### উত্তর:
<response here>

For an Ollama Modelfile, wrap it as:

TEMPLATE """{{ if .Prompt }}নিচে একটি নির্দেশনা দেওয়া আছে, যা একটি কাজের বর্ণনা দেয়। অনুরোধটি যথাযথভাবে সম্পূর্ণ করে একটি উত্তর লিখুন।

### নির্দেশনা:
{{ .Prompt }}

### উত্তর:
{{ end }}{{ .Response }}"""

PARAMETER stop "### নির্দেশনা:"
PARAMETER stop "### উত্তর:"
PARAMETER temperature 0.7
PARAMETER top_p 0.9
PARAMETER repeat_penalty 1.1

Training lineage (condensed)

This GGUF was produced by:

1. 1.Loading unsloth/mistral-7b-v0.3 in 4-bit (QLoRA).
2. 2.CPT on Bangla Wikipedia (wikimedia/wikipedia, config 20231101.bn).
3. 3.SFT on `ihumaunkabir/alpaca-gpt4-bangla` (49,969 Bangla instruction pairs).
4. 4.Merging the LoRA adapters back into the base weights at 16-bit.
5. 5.Converting to GGUF and quantizing to Q4KM with llama.cpp.

LoRA config: rank 128, alpha 32, rsLoRA on, targets include embed_tokens + lm_head.

Limitations

1. 1.Smoke-test training volume (max_steps=120) -- quality will be rough until full epochs are run.
2. 2.CPT data is Wikipedia-only -- formal encyclopedia Bangla; no colloquial text.
3. 3.SFT data is machine-translated (Korean -> Bangla).
4. 4.No safety alignment. No RLHF, DPO, or red-teaming. Filter outputs for your use case.
5. 5.Q4_K_M quantization loses some precision vs. the 16-bit merged model. For higher quality, re-quantize from the LoRA adapters at Q8_0 or f16.
6. 6.Alpaca prompt format required. The model will produce poor output if prompted with a different template (e.g. ChatML).

Citation

If you use this model, please cite both this GGUF repo and the alpaca-gpt4-bangla dataset, plus the base Mistral model.

This repo (GGUF)

@misc{mistral-bangla-gguf,
  author       = {ihumaunkabir},
  title        = {mistral-bangla: Bangla Mistral-7B in Q4_K_M GGUF format},
  year         = {2026},
  url          = {https://huggingface.co/ihumaunkabir/mistral_bangla_q4_k_m_gguf},
  note         = {Merged LoRA + Q4_K_M quantization of CPT + SFT Mistral-7B}
}

Source LoRA adapters

@misc{mistral-bangla-lora,
  author       = {ihumaunkabir},
  title        = {mistral-bangla: Bangla CPT + SFT LoRA adapters for Mistral-7B-v0.3},
  year         = {2026},
  url          = {https://huggingface.co/ihumaunkabir/mistral_bangla_lora}
}

SFT dataset -- alpaca-gpt4-bangla

@misc{alpaca-gpt4-bangla,
  author       = {ihumaunkabir},
  title        = {alpaca-gpt4-bangla: A Bangla instruction-following dataset},
  year         = {2026},
  url          = {https://huggingface.co/datasets/ihumaunkabir/alpaca-gpt4-bangla},
  note         = {Machine translation (Korean -> Bangla) of FreedomIntelligence/alpaca-gpt4-korean}
}

Base model

@misc{mistral7b,
  author       = {Mistral AI},
  title        = {Mistral-7B-v0.3},
  year         = {2024},
  url          = {https://huggingface.co/mistralai/Mistral-7B-v0.3}
}