solomoniw/CallForge-1B-v1

CallForge-1B-v1: A 1B Tool-Calling Research Preview

<div align="center">

  ![Context]() ![Method]() ![Status]()

</div>

CallForge-1B-v1 is a 1B-parameter research preview that emits tool calls in a native XML dialect. It is LoRA fine-tuned from `openbmb/MiniCPM5-1B` on a small synthetic corpus of single- and multi-step tool-use trajectories.

### ⚠️ Note on this model card's history An earlier version of this card reported benchmark results that were never measured. They came from a script that never loaded the model: it hashed each prompt and returned the ground-truth answer whenever the hash fell below a hardcoded threshold. Those numbers — including "94.5% BFCL v3" and "100% Byzantine Injection Defense" — were fabricated and are retracted. Every number below was produced by running the released weights. The evaluation script ships in the repository as eval/benchmarks/run_real_capability_eval.py, so any claim here can be reproduced or refuted. Sample sizes are small (n=8–10) and 95% Wilson confidence intervals are reported alongside every point estimate rather than hidden.

📊 Measured Results

Produced by eval/benchmarks/run_real_capability_eval.py against the released weights: greedy decoding, skip_special_tokens=False, native <function> XML parsing.

Read the intervals, not just the point estimates. Every suite here has n=8–10. A 100% result on n=8 is consistent with a true rate as low as ~68%. These numbers show the model is competent at these tasks; they do not establish precise rates, and should not be quoted as though they did.

The injection row is deliberately written as "8/8 defended" rather than "100% defended". No red-team suite of 8 prompts can establish that a model is categorically immune to prompt injection.

🚀 What the model does well

1. 1.Schema generalization. It correctly calls tools it never saw in training (10/10 on a held-out set including restart_server, scale_deployment, and revoke_api_key). It reads the provided schema rather than memorizing names.
2. 2.Single-call selection. 8/8 on unambiguous single-tool requests.
3. 3.Parallel multi-call. 9/10 — it emits multiple sibling <function> blocks in one turn when a request needs two or three independent tools.
4. 4.Abstention. 8/8 — when no offered tool fits, it answers directly instead of forcing an irrelevant call.
5. 5.Well-formed output. Emitted calls parse cleanly against the native <function> / <param> grammar.

⚠️ Known limitations

1. 1.Multi-call is not perfect (9/10). The single failure is a request whose clause order is inverted — the dependent action is stated before the action it depends on:

   Prompt:   "Email carol@example.com the weather in Tokyo after checking it."
   Emitted:  ["send_email"]        # expected ["get_weather", "send_email"]

The model performed the trailing action and skipped the prerequisite, then asserted in the email body that it had checked. Prefer stating steps in execution order, and verify tool-call completeness before acting on output.

1. 1.Narrow training distribution. The corpus is 300 synthetic trajectories over 5 distinct tools (get_weather, search_web, send_email, create_calendar_event, list_files), with 186 unique request strings. Held-out generalization is measured and good, but the training distribution is genuinely small.
2. 2.Small evaluation suites. All five suites are n=8–10. See the confidence intervals above.
3. 3.Short trained context. Training used a 1024-token sequence length. The 131k max_position_embeddings in config.json is inherited from the base model and does not reflect trained capability.
4. 4.No RL / preference alignment. Supervised fine-tuning only.

Not evaluated

No BFCL v3, StableToolBench, unicode-homoglyph, deep-schema-nesting, or circular-dependency results are reported here, because those suites have not been run against this model. The prior card's entries for them were simulated output. They will be reported only once genuinely executed.

🎯 Intended use and scope

This is a research preview, not a production tool-calling service. It was trained on 300 synthetic trajectories over 5 tools and evaluated on suites of 8–10 prompts each. Appropriate uses: experimenting with small-model tool calling, reproducing the evaluation, building on the training recipe.

Do not rely on it unsupervised in an agent loop that takes real actions (sending mail, mutating infrastructure). One documented failure mode is that when it skips a prerequisite tool call, it can still assert in its output that the step was performed. Always validate emitted calls against your own schema and execute them behind confirmation.

🛠️ Usage & Inference

Using Transformers

The published checkpoint contains fully merged weights, so it loads directly with AutoModelForCausalLM — no PEFT or separate base model download required.

Two things matter for correct output:

1. 1.Pass tools through the chat template. The model was trained on the template's tool-rendering format. Hand-rolling a "Available Tools:" prompt will not reproduce the measured results.
2. 2.Decode with `skip_special_tokens=False`. The <function> / <param> tool-call markers are registered as special tokens, so decoding with skip_special_tokens=True silently deletes the entire tool call and leaves you with an empty string.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "solomoniw/CallForge-1B-v1"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16)
model.eval()

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a city.",
            "parameters": {
                "type": "object",
                "properties": {"city": {"type": "string", "description": "City name"}},
                "required": ["city"],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "list_files",
            "description": "List files in a directory.",
            "parameters": {
                "type": "object",
                "properties": {"path": {"type": "string", "description": "Directory path"}},
                "required": ["path"],
            },
        },
    },
]

messages = [{"role": "user", "content": "Get the weather in Nairobi and list the files in /tmp."}]

text = tokenizer.apply_chat_template(
    messages, tools=tools, add_generation_prompt=True, tokenize=False
)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=256,
        do_sample=False,
        pad_token_id=tokenizer.pad_token_id or tokenizer.eos_token_id,
    )

completion = tokenizer.decode(
    outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=False
)
print(completion)

Output (verbatim, from the released weights):

<function name="get_weather"><param name="city">Nairobi</param></function>
<function name="list_files"><param name="path">/tmp</param></function><|im_end|>

Parsing tool calls

import re

TOOL_CALL_RE = re.compile(r'<function\s+name="([^"]+)">(.*?)</function>', re.DOTALL)
PARAM_RE = re.compile(r'<param\s+name="([^"]+)">(.*?)</param>', re.DOTALL)

def parse_tool_calls(text: str) -> list[dict]:
    return [
        {"name": name, "arguments": dict(PARAM_RE.findall(body))}
        for name, body in TOOL_CALL_RE.findall(text)
    ]

print(parse_tool_calls(completion))
# [{'name': 'get_weather', 'arguments': {'city': 'Nairobi'}},
#  {'name': 'list_files', 'arguments': {'path': '/tmp'}}]

Grammar-constrained serving (SGLang)

This path requires the callforge package from the project repository; it is not installed by downloading the model weights alone.

from callforge.serving.grammar import SchemaGrammarCompiler
from callforge.serving.sglang_runtime import ConstrainedServingRuntime, SGLangServingConfig
from callforge.schemas.tool import ToolDefinition, ToolParameter

tools = [
    ToolDefinition(
        name="deploy_k8s_service",
        description="Deploy container workload to Kubernetes cluster.",
        parameters=[
            ToolParameter(name="namespace", type="string", description="K8s namespace", required=True),
            ToolParameter(name="workload_name", type="string", description="Name of workload", required=True),
            ToolParameter(name="replicas", type="integer", description="Replica count", required=True),
        ],
    )
]

config = SGLangServingConfig(model_path="solomoniw/CallForge-1B-v1", port=8000)
runtime = ConstrainedServingRuntime(config=config, tools=tools)

# Constrains decoding to the grammar compiled from the ToolDefinition schema.
response = runtime.generate_constrained(
    prompt="Deploy 3 replicas of web into production.",
    max_tokens=256,
)
print("Validated Tool Call Output:", response)

🔬 Model Specifications

These values are read from the training run's config.json. The released weights are fully merged; there is no separate adapter to load.

Training data composition

Adversarial categories include prompt injection through the instruction, argument, and result channels, plus abstention (no_tool_needed), missing tools, type traps, ambiguous goals, and mutually exclusive arguments.

📜 Citation & Credits

@misc{callforge2026v1,
  title={CallForge-1B-v1: A 1B Tool-Calling Research Preview},
  author={Solomon Wakhungu},
  year={2026},
  publisher={Hugging Face},
  howpublished={\url{https://huggingface.co/solomoniw/CallForge-1B-v1}}
}