canbingol/qwen2.5-3B-Instruct-tool-call-en-mixed-when2call-v2
⚠️ Warning
This model was fine-tuned for experimental purposes, as part of an investigation into reducing over-triggered tool calls (i.e., getting the model to correctly abstain when no tool is relevant).
nvidia/When2Call (train_sft) was included in training to improve this behavior. However, this had an unintended side effect: the model's core tool-calling ability degraded significantly. It now frequently asks the user for clarification/input even for simple, complete, unambiguous tool-call requests where all required parameters are already provided.
Do not use this model in production or for tool-calling tasks. It is kept public only as a record of this experiment (see BFCL eval results above — simple_python_acc and multiple_acc collapsed while irrelevance_acc hit ceiling, indicating over-correction rather than genuine improvement).
Qwen2.5-3B-Instruct — Tool Call (EN, Mixed + When2Call 3.3k)
LoRA fine-tune of Qwen/Qwen2.5-3B-Instruct for function/tool calling.
Data
- Salesforce/xlam-function-calling-60k — 54,000 (tool-call)
- HuggingFaceH4/no_robots — 8,000, filtered ≤2,000 chars (no-tool SFT)
- nvidia/When2Call (train_sft) — 3,300 (irrelevance/negative signal)
- Concatenated, shuffled, formatted via
apply_chat_template
LoRA
r=32, loraalpha=64, loradropout=0.05, targetmodules=all-linear, bias=none, tasktype=CAUSAL_LM
Training
lr=2e-4, epochs=1, perdevicebatch=8, gradaccum=16 (effective 128), warmupratio=0.03, maxgradnorm=0.3, bf16=True, attnimplementation=sdpa, losstype=nll
Training Results (step 505, final)
trainloss=0.354, valloss=0.173, meantokenaccuracy=0.956, runtime≈4h02m (A100)
BFCL Eval Results (inspect_evals/bfcl)
Note: irrelevance_acc reached 1.000 (perfect abstention), but simple_python_acc (0.090) and multiple_acc (0.045) collapsed even further. Even with a reduced When2Call subset (3.3k), the negative signal causes heavy over-abstention from tool calling.