aghasalim/siba-meristem-1.0

<div align="center">

SIBA Meristem 1.0

Multilingual Large Language Model for Caucasian and Eurasian Languages

    

Azerbaijani | Russian | English | Turkish

An open multilingual adapter aimed at Azerbaijani and the Caucasus, for research use.

</div>

Update, 25 September 2026: licence corrected, first benchmark, version 1.0.1

Licence. Meristem is an adapter on Qwen2.5-3B-Instruct, which is under the Qwen Research License (research use only, non-commercial). An adapter cannot carry a looser licence than the model it modifies, and part of its training data (tatsu-lab/alpaca) is CC BY-NC 4.0, so this repository was wrongly labelled Apache 2.0. It is now labelled correctly and a copy of the Qwen Research License is in LICENSE.

First benchmark. Measured on azbench, 2,148 zero-shot Azerbaijani multiple-choice questions, exact-letter scoring, run locally in Q4KM:

Meristem 1.0 scores below its own base. It was released at step 500 of a planned 31,461 and learned to favour the letter "C" (82 % of its answers on the reading set).

Version 1.0.1 (in `v1.0.1/`) continues the same LoRA for 20 optimizer steps on letter-balanced Azerbaijani cloze questions built from Azerbaijani Wikipedia (CC BY-SA), loss on the answer letter only, none of it from the azbench test sets. On the first 300 reading questions (all three models in Q4KM, temperature 0, Qwen's default system prompt):

1.0.1 removes the "C" bias and recovers most of the loss, but is still below the base model. It is a repair, not an improvement over Qwen2.5-3B-Instruct. The next Meristem will be built on a commercially licensed base and only released if it beats that base on azbench.

Highlights

• —167,785 multilingual instruction pairs (~32.5M tokens) spanning 4 languages
• —QLoRA fine-tuned on Qwen2.5-3B-Instruct with Unsloth for maximum efficiency
• —Training loss reduced from 2.49 to 1.26 (49.4% reduction) over 500 gradient steps
• —Available in GGUF Q4_K_M (1.8 GB) for efficient local inference
• —Available as LoRA adapters for custom fine-tuning and research
• —Runs on consumer hardware -- 4GB+ VRAM sufficient for inference

Training Details

Architecture and Method

Training Loss Curve

Loss reduction: 2.491 to 1.262 (49.4% improvement)

Dataset Composition

Quick Start

Option 1: Ollama (Easiest)

# Install Ollama from https://ollama.ai, then:
ollama run aghasalim/siba-meristem-1.0

Option 2: llama.cpp / LM Studio

Download Qwen2.5-3B-Instruct.Q4_K_M.gguf from this repository and load it in any GGUF-compatible runtime:

# llama.cpp example
./llama-cli -m Qwen2.5-3B-Instruct.Q4_K_M.gguf \
  -p "<|im_start|>user\nSalam! Azerbaycan haqqinda melumat ver.<|im_end|>\n<|im_start|>assistant\n" \
  -n 256 --temp 0.7 --top-p 0.9

Option 3: Transformers + PEFT (Python)

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

# Load base model and LoRA adapter
base_model_id = "Qwen/Qwen2.5-3B-Instruct"
adapter_id = "aghasalim/siba-meristem-1.0"

tokenizer = AutoTokenizer.from_pretrained(base_model_id)
model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    torch_dtype=torch.float16,
    device_map="auto"
)
model = PeftModel.from_pretrained(model, adapter_id)

# Chat in Azerbaijani
messages = [
    {"role": "system", "content": "Sen SIBA Meristem 1.0, coxdilli AI komekcisin."},
    {"role": "user", "content": "Baki seheri haqqinda qisa melumat ver."}
]

prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=512,
        temperature=0.7,
        top_p=0.9,
        repetition_penalty=1.1,
        do_sample=True
    )

response = tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True)
print(response)

Option 4: Unsloth (Fastest Inference)

from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="aghasalim/siba-meristem-1.0",
    max_seq_length=8192,
    load_in_4bit=True,
)
FastLanguageModel.for_inference(model)

messages = [
    {"role": "user", "content": "Hello! Tell me about yourself."}
]

inputs = tokenizer.apply_chat_template(
    messages, tokenize=True, add_generation_prompt=True, return_tensors="pt"
).to("cuda")

outputs = model.generate(input_ids=inputs, max_new_tokens=256, temperature=0.7)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Repository Contents

Multilingual Examples

<details> <summary><b>Azerbaijani Example</b></summary>

User: Azerbaijan metbexinin en meshur yemekleri hansilardir?

SIBA: Azerbaijan metbexi zengin ve muxtelifdir. En meshur yemekler bunlardir:

1. Plov (As) -- Azerbaijan metbexinin taci sayilir
2. Dolma -- Uzum yarpagina bukulmus et-duyu qarisigi
3. Qutab -- Nazik xemirden hazirlanan yasti borek
4. Piti -- Susa bolgesinin enenevi sorbasi
5. Lulekabab -- Manqalda bisirilen et kabab

</details>

<details> <summary><b>Russian Example</b></summary>

User: Explain the difference between machine learning and deep learning.

SIBA: Machine learning (ML) and deep learning (DL) are related but distinct fields:

Machine learning is a broad area of AI where algorithms learn from data...
Deep learning is a subset of ML that uses neural networks with many layers...

</details>

<details> <summary><b>Turkish Example</b></summary>

User: Turkiye'nin en guzel turistik yerleri nerelerdir?

SIBA: Turkiye, zengin tarihi ve dogal guzellikleriyle dolu bir ulkedir:

1. Istanbul -- Ayasofya, Topkapi Sarayi, Kapalicars
2. Kapadokya -- Peri bacalari ve sicak hava balonlari
3. Pamukkale -- Beyaz travertenler ve antik Hierapolis

</details>

<details> <summary><b>English Example</b></summary>

User: What makes the Caucasus region culturally unique?

SIBA: The Caucasus region is one of the most culturally diverse areas on Earth:

1. Linguistic Diversity -- Over 50 distinct languages spoken across a relatively small area
2. Ancient Heritage -- Civilizations dating back to the Bronze Age
3. Cultural Crossroads -- Intersection of European, Asian, and Middle Eastern influences

</details>

Limitations and Known Issues

• —Early checkpoint: This is from step 500 of a planned 31,461-step training run. That run was not continued; see the update at the top.
• —Azerbaijani understanding: On azbench, Meristem 1.0 scores below its base model (32.9 % against 39.5 %), and 1.0.1 is still below it on the reading set. Do not assume it understands Azerbaijani better than Qwen2.5-3B-Instruct.
• —Factual accuracy: Like all LLMs, the model may generate plausible-sounding but incorrect information. Always verify critical facts.
• —Context window: The model supports up to 8,192 tokens of context.
• —Not safety-tuned: This model has not undergone RLHF or constitutional AI safety training beyond what is inherited from Qwen2.5-3B-Instruct.

License

This model is governed by the Qwen Research License Agreement of its base model, Qwen2.5-3B-Instruct: research and evaluation use only, not commercial use. A copy is in `LICENSE`. Part of the training data (tatsu-lab/alpaca) is also CC BY-NC 4.0.

Until 25 September 2026 this repository was labelled Apache 2.0. That label was wrong and has been corrected; it did not change the terms of the base model, which always applied.

Links

Acknowledgments

• —Qwen Team (Alibaba Cloud) for the Qwen2.5 base model
• —Unsloth for efficient 4-bit QLoRA fine-tuning on consumer GPUs
• —Hugging Face for model hosting and ecosystem
• —Ollama for accessible local LLM deployment

Contact

Developer: Aghasalim Mustafazada Hugging Face: @aghasalim Ollama: @aghasalim

<div align="center"> <i>SIBA Meristem -- Growing intelligence, one language at a time.</i> </div>