solomoniw/CallForge-1B-v2

CallForge-1B-v2

A 1B tool-calling model fine-tuned from openbmb/MiniCPM5-1B on the CallForge v2 trajectory corpus (893 trajectories: multi-hop dependency chains, distractor tools, clarification-required turns, literal-copy traps). Trained with LoRA (merged) + trained lm_head, bf16, fixed-2048 padding.

Benchmarks (measured, no cherry-picking)

All numbers below are from the unified runner (eval/benchmarks/run_suite.py): greedy decoding, seed 0, the checkpoint's own chat template, hittokenbudget => failure. Raw traces are in benchmark_results.json. The stress and private suites are held-out and were never trained on; BFCL v3 is hash-pinned.

Deltas vs baselines

Training

• —Run: v2-r32-lr2e-5-tp (preregistered ablation-grid winner)
• —Data: data/trajectories/corpus_v2.jsonl (893 trajectories, hash recorded in the reproducibility manifest)
• —Method: LoRA + trained lm_head, merged; bf16; fixed-2048 padding
• —Selection: preregistered score 0.6×probepassrate + 0.4×(1−val_loss/3.0); no post-hoc tuning

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("solomoniw/CallForge-1B-v2")
tok = AutoTokenizer.from_pretrained("solomoniw/CallForge-1B-v2")
messages = [{"role": "user", "content": "What's the weather in Paris?"}]
tools = [{"type": "function", "function": {"name": "get_weather",
  "description": "Get current weather", "parameters": {"type": "object",
  "properties": {"city": {"type": "string"}}, "required": ["city"]}}}]
prompt = tok.apply_chat_template(messages, tools=tools, add_generation_prompt=True, tokenize=False)
ids = tok(prompt, return_tensors="pt")
out = model.generate(**ids, max_new_tokens=192, do_sample=False)
print(tok.decode(out[0][ids.input_ids.shape[1]:], skip_special_tokens=True))

GGUF quants (F16, Q80, Q4K_M) are in this repo; load with llama.cpp.

Limitations

• —1B parameters: reasoning depth is bounded; the model is optimized for correct tool selection/argument extraction, not open-domain chat.
• —Stress-suite categories with small n have wide CIs; see the table.
• —Benchmarks were run on Apple Silicon MPS; absolute numbers may shift slightly on other backends, but the same runner and seeds were used for every model compared.