CoolFace
Modelpublic

SauravMahalik/MiniCPM5-2B-Claude-Fable5

sourceHugging Faceapache-2.0updated 10d agoView on Hugging Face
2likes607downloads
Model Card

MiniCPM5-2B-Claude-Fable5

Agent-style chat model obtained by LoRA fine-tuning openbmb/MiniCPM5-2B (2.57B parameters, Llama architecture, 130,560-token vocabulary) on agent traces from saidutta69/fable-5-premium-v2 (openai_chat config), distilled from Claude Fable-5 and other frontier models. LoRA adapters (50.2M parameters, 1.96%) are merged into fp16 weights below, so the model loads and serves like any standard causal LM - no PEFT code required.

The chat template is Qwen3-style (<|im_start|>/<|im_end|>, with an empty <think></think> block on assistant turns) and ships with the tokenizer.

Intended uses and limitations

Intended for research and prototyping of tool-using conversational agents: multi-step instruction following, function/tool calling, and agentic dialogue in English and Chinese.

Out of scope: high-stakes decisions (medical, legal, financial), unsupervised autonomous actions, and any use as a source of factual truth - like all models distilled from frontier traces, it can hallucinate tools, arguments, and facts. Training saw truncated contexts (see below), so very long agentic episodes exceed what was trained. Safety alignment is inherited from the base model and the distilled data only; no dedicated red-teaming was performed.

Training data

  • —Source: saidutta69/fable-5-premium-v2, openai_chat config only (agent_traces holds the same traces and was excluded to avoid double-training).
  • —Selection: quality-tiered by quality_scores.overall (>= 0.98 first, then >= 0.9), token-budgeted to ~34M training tokens: 50,241 train / 347 validation conversations.
  • —Preprocessing: OpenAI tool_calls with string arguments normalized to dicts; malformed rows dropped; a trailing system message renders as user.
  • —Loss is computed on assistant turns only (user/system/tool tokens masked); long traces are prefix-truncated to 2048 tokens at segment boundaries.

Training procedure

SettingValue
Hardware2x NVIDIA T4 (Kaggle), DDP
Precisionfp16 mixed precision, SDPA attention, gradient checkpointing
MethodLoRA r=32, alpha=64, dropout 0.05 on all attention + MLP projections (q/k/v/o/gate/up/down)
OptimizerAdamW, lr 1e-4, cosine decay, 3% warmup, max grad norm 1.0
Batch16 effective (2 micro x 4 accum x 2 GPUs)
Steps2,542 (~0.8 epochs), 8h wall-clock budget

Evaluation

Held-out validation loss (347 conversations, assistant tokens only):

Step5001000150020002500
val loss0.9800.9600.9430.9330.926

Loss falls monotonically with no sign of overfitting. No downstream benchmarks were run; the numbers above are the complete evaluation record for this release.

Usage

python
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained('SauravMahalik/MiniCPM5-2B-Claude-Fable5')
model = AutoModelForCausalLM.from_pretrained(
    'SauravMahalik/MiniCPM5-2B-Claude-Fable5', torch_dtype='auto', device_map='auto')
messages = [{'role': 'user', 'content': 'Write a haiku about GPUs.'}]
ids = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors='pt').to(model.device)
out = model.generate(ids, max_new_tokens=256)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=False))

Provenance

  • —Base weights: openbmb/MiniCPM5-2B (see its model page for license/terms).
  • —Fine-tuning data: saidutta69/fable-5-premium-v2.
  • —Trained in a single Kaggle 2xT4 session; run log verified: 2,542 steps, both weight shards SHA-256-checked on upload.