Modelpublic

luispoveda93/MiniCPM5-2B-catalan-chat

sourceHugging Faceotherupdated 13d agoView on Hugging Face

0likes436downloads

Model Card

MiniCPM5-2B-catalan-chat

A conversational Catalan chat model: LoRA fine-tune of [openbmb/MiniCPM5-2B](https://huggingface.co/openbmb/MiniCPM5-2B) on the full [projecte-aina/InstruCAT](https://huggingface.co/datasets/projecte-aina/InstruCAT) instruction dataset (165,100 samples, 11 task categories, ~43M tokens, 1 epoch).

Model Details

—Base model: openbmb/MiniCPM5-2B (2.5B params, Llama architecture, Apache-2.0)
—Fine-tuned on: projecte-aina/InstruCAT — Catalan instructions converted from caBreu, CatalanQA, CaWikiTC, ceil, CoqCat, GuiaCat, IntoxiCat, NLUCat, Parafraseja, PAWS-ca, sts-ca, teca and WikiCat
—Language: Catalan (ca-ES)
—Task: conversational instruction following
—License: base model Apache-2.0; training data CC-BY-NC-ND-4.0 (non-commercial) — check the InstruCAT license terms before commercial use

Training

Setting	Value
Method	LoRA (r=32, α=64, dropout 0.05, all attention+MLP projections)
Trainer	TRL `SFTTrainer` v1.12.0, transformers 5.16.1, peft 0.20.0
Prompt format	MiniCPM5 chat template; system prompt "Ets un assistent conversacional que respon sempre en català."; completion-only loss on the assistant turn
Sequence length	2048, packed
Effective batch	32 (4 × grad-accum 8)
LR / schedule	2e-4, cosine, 200 warmup steps
Steps / hardware	711 steps (~19h) on 1× A10G 24GB, bf16 + gradient checkpointing

Observed training curve (completion-only loss, every 10 steps): 2.39 → 1.48 (step 20) → 1.04 (step 30) → 0.45 (step ~70) → ~0.35–0.40 at the end. Mean token accuracy reached ~0.91–0.92 by the second half of the run. Eval on 1,000 held-out validation samples ran at step 500 without errors; metrics are on the trackio dashboard.

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "luispoveda93/MiniCPM5-2B-catalan-chat",
    dtype="bfloat16",
)
tok = AutoTokenizer.from_pretrained("luispoveda93/MiniCPM5-2B-catalan-chat")

messages = [{"role": "system", "content": "Ets un assistent conversacional que respon sempre en català."},
            {"role": "user", "content": "Explica'm què és la Sardana."}]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
out = model.generate(inputs, max_new_tokens=256)
print(tok.decode(out[0][inputs.shape[1]:], skip_special_tokens=True))

The LoRA adapter alone is at luispoveda93/MiniCPM5-2B-catalan-chat-lora.

Limitations

—Trained on single-turn instruction data; no multi-turn Catalan conversations were used.
—InstruCAT is task-oriented (QA, paraphrase, NER, classification, summarization); general chit-chat behavior comes from the base model.
—Non-commercial data license applies to the fine-tuning dataset.

Citation

Please cite the InstruCAT dataset and Projecte AINA if you use this model:

—Dataset: projecte-aina/InstruCAT — Language Technologies Unit (langtech@bsc.es) at BSC, funded within project ILENIA (ref. 2022/TL22/00215337).