when2call
When2Call
When2Call
💾 Github | 📄 Paper
Dataset Description:
When2Call is a benchmark designed to evaluate tool-calling decision-making for large language models (LLMs), including when to generate a tool call, when to ask follow-up questions, when to admit the question can't be answered with the tools provided, and what to do if the question seems to require tool use but a tool call can't be made.
We find that state-of-the-art tool-calling LMs show significant room for… See the full description on the dataset page: https://huggingface.co/datasets/nvidia/When2Call.nvidia_hermes_when2callwhen2call-GLM4.5-IIwhen2call_imbalanced_request_10
when2call_imbalanced_request_10
Derived from nvidia/When2Call (train_pref, train) by downsampling the request category in chosen_response by 10% (keeping 90%).
Sampling
Target class: request
Keep ratio: 90%
Seed: 45
Counts (chosen_response)
Source
refusal: 2999
toolcall: 3000
request: 3001
unk: 0
Result
refusal: 2999
toolcall: 3000
request: 2700
unk: 0
Final rows: 8699
when2call_imbalanced_refusal
when2call_imbalanced_refusal
This dataset is derived from nvidia/When2Call (train_pref, train split) by downsampling one chosen-response category to ~50% while keeping all other rows.
Source
Dataset: nvidia/When2Call
Config: train_pref
Split: train
Source rows: 9000
Classification Rules (on chosen_response)
Categories are assigned in this precedence order:
toolcall if text contains <TOOLCALL> (case-insensitive)
request if text contains ?
request if text… See the full description on the dataset page: https://huggingface.co/datasets/ihounie/when2call_imbalanced_refusal.aliaboost_when2call_es