s4um1l/toolflow-qwen3-8b-utility-v3

ToolFlow Qwen3-8B Utility — v3 (LoRA adapter)

A QLoRA adapter on top of Qwen/Qwen3-8B, fine-tuned for tool-calling against a 20-tool utility catalog. Matches Claude Sonnet 4.6 (70.4% semantic-valid on the same eval) at ~16× lower estimated per-task serving cost.

This is the production adapter from Week 3 of the ToolFlow project — v2's successor. v3 adds 300 BFCL-stylistic abstain prompts to v2's training data and retrains with identical hyperparameters. The result lifts all four eval categories, not just abstain: multicall +15pp, happypath +11pp, tool_selection +10pp, abstain +4pp. Overall semantic-valid: 60.4% → 70.4%.

Headline results

Evaluated on the 270-case eval set (BFCL v3 prompts re-labeled by Qwen3-235B against the utility catalog — anti-circularity guard against same-vendor evaluation):

Statistical significance vs v2 (paired bootstrap, 10000 resamples, McNemar): Δ semantic-valid = +10.0pp, 95% CI [+5.9, +14.1], McNemar p < 0.0001. The lift is highly significant, not noise.

What changed vs v2

v3 adds 300 BFCL-stylistic abstain training prompts synthesized via Qwen3-235B-A22B-Instruct-2507. Same hyperparameters as v2, same base model, same eval — only the training data was broadened. The motivating finding from W2: v2's 62% abstain recall on eval_v2's formal-style abstain prompts was a measurement artifact, not a capability limit — v2 abstained correctly on ~99% of training-distribution prompts but only on the BFCL formal/academic register specifically. Adding 300 formal-register abstain examples covered the gap.

Surprising result: the lift was broad-based, not category-specific. Adding abstain examples lifted all four eval categories. Hypothesis: clearer "negative space" (when no tool fits) sharpens "positive space" (when this tool fits) discrimination across the board.

Per-category breakdown (semantic-valid)

Memorization tripwire — third-distribution eval

To distinguish "broadened distribution" from "memorized BFCL register," v3 was also evaluated on a held-out 50-prompt eval generated by Claude Sonnet 4.6 (different vendor, casual chat-speak register: "hey can u…", "yo what's…", "pls help me…"). This set is never in training:

v3 lifts +10pp here too — clear generalization, not memorization. v2 was already strong (86%) on this register because casual chat phrasing overlaps with xlam training data more than BFCL formal phrasing did.

What this adapter does

Given a user request and a system prompt enumerating available tools, emits a JSON tool call (or {"tool": "ABSTAIN", "arguments": {}} if no tool fits):

User: What's the area of a circle with radius 4?
Model: {"tool": "math.area_circle", "arguments": {"radius": 4}}

User: Schedule a meeting with my team tomorrow at 3pm.
Model: {"tool": "ABSTAIN", "arguments": {}}

The 20-tool catalog covers math, stats, geo, finance, web, text, auth, science utilities. Full catalog at tools.json in the repo.

Training details

Training data

4,761 examples, mixed:

• —3,749 positive tool-call examples filtered from Salesforce/xlam-function-calling-60k (CC-BY-4.0) and glaiveai/glaive-function-calling-v2 (Apache 2.0)
• —712 synthesized ABSTAIN examples from v2 (Qwen3-235B-A22B-Instruct-2507)
• —300 NEW BFCL-stylistic ABSTAIN examples (W3.0 addition) synthesized via Qwen3-235B-A22B-Instruct-2507-tput in formal/academic register

Training data is NOT redistributed in this repo — xlam is gated on Salesforce's HuggingFace page and republishing derivatives openly would be a workaround. The adapter weights are derivative of Qwen3-8B (Apache 2.0) and license-clean.

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

base = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3-8B",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("s4um1l/toolflow-qwen3-8b-utility-v3")
model = PeftModel.from_pretrained(base, "s4um1l/toolflow-qwen3-8b-utility-v3")
model.eval()

# Build the system prompt with the 20-tool catalog (see tools.json in the repo
# for the canonical list). The training data uses the zero_shot prompt template
# at https://github.com/s4um1l/toolflow/blob/main/prompts/zero_shot.md
SYSTEM = """You are a tool-calling assistant. Given a user request, respond with EXACTLY ONE JSON object specifying which tool to call and arguments. If no listed tool fits the request, respond with {"tool": "ABSTAIN", "arguments": {}}.

Available tools:
- math.area_circle: Calculate the area of a circle given its radius.
  schema: {"type":"object","properties":{"radius":{"type":"number","exclusiveMinimum":0}},"required":["radius"],"additionalProperties":false}
... (full 20-tool catalog)

Output format:
{"tool": "<tool_name>", "arguments": <args_object>}

Do not include explanation, markdown, or text outside the JSON object."""

messages = [
    {"role": "system", "content": SYSTEM},
    {"role": "user",   "content": "What's the area of a circle with radius 4?"},
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=False,  # important: disables Qwen3's thinking mode for direct JSON output
)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
with torch.no_grad():
    out = model.generate(**inputs, max_new_tokens=200, do_sample=False)
print(tokenizer.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))
# {"tool": "math.area_circle", "arguments": {"radius": 4}}

For inference at scale, vLLM + PEFT serving is recommended (multi-LoRA mode via --enable-lora lets you serve v3, v2, and the 4B variant from one base model — benchmarks pending in W3.1).

Known limitations

Schema validity 94.1% vs Sonnet's 98.9%. v3 still produces ~6% structurally-invalid JSON outputs. This is the obvious mechanical fix via constrained decoding (XGrammar) — pending in W3.3.

No vLLM/PEFT serving benchmarks (yet). The "16× cheaper" cost claim is an estimate from token-pricing differential. Real production deployment numbers pending W3.1.

No catastrophic forgetting evaluation. The adapter wasn't tested against IFEval or general-instruction benchmarks. It might have lost some general capability during training, though loss curves and schema validity (94%) suggest minimal forgetting.

Narrow catalog. This was trained for the specific 20-tool ToolFlow utility catalog. Performance on tools outside this catalog is undefined.

Companion models

• —8B v2 predecessor: `s4um1l/toolflow-qwen3-8b-utility-v2` — same architecture, narrower training distribution, 60.4% semantic-valid. Kept for reproducibility / comparison.
• —4B variant: `s4um1l/toolflow-qwen3-4b-utility-v2` — Qwen3-4B base, 50.4% semantic-valid. v3-equivalent retraining of the 4B is not yet done.

License

Apache 2.0 (inherited from Qwen3-8B base model). Adapter weights only — no training data redistributed.

Project + reproducibility

• —Project repo: https://github.com/s4um1l/toolflow
• —Eval set: 270 BFCL prompts re-labeled by Qwen3-235B against the utility catalog. Plus 50-prompt third-distribution eval (Sonnet 4.6, casual register) for memorization tripwire.
• —W3.0 detailed report: `reports/recipe_a_v3_report.md`
• —Project tag: v0.3-week3 (pending — at the W3.0 milestone now)

Citation

@misc{toolflow_qwen3_8b_utility_v3,
  title  = {ToolFlow Qwen3-8B Utility v3 (LoRA adapter)},
  author = {Srivastava, Saumil},
  year   = {2026},
  url    = {https://huggingface.co/s4um1l/toolflow-qwen3-8b-utility-v3}
}

If you use this adapter, please also cite the upstream training data sources (xlam-function-calling-60k and glaive-function-calling-v2) and the Qwen3 base model.