QinyuanWu/ToCall_or_NotToCall
To Call or Not to Call: Tool-Calling Evaluation Datasets Datasets accompanying the paper "To Call or Not to Call: A Framework to Assess and Optimize LLM Tool Calling" (Wu et al., 2026). The paper evaluates when language models should call external tools (web search, calculator) rather than answering from parametric knowledge, from both a normative perspective (when is a tool call truly needed/useful?) and a descriptive perspective (when does the model think it needs a tool, and… See the full description on the dataset page: https://huggingface.co/datasets/QinyuanWu/ToCall_or_NotToCall.
To Call or Not to Call: Tool-Calling Evaluation Datasets
Datasets accompanying the paper ["To Call or Not to Call: A Framework to Assess and Optimize LLM Tool Calling"](https://arxiv.org/abs/2605.00737) (Wu et al., 2026).
The paper evaluates when language models should call external tools (web search, calculator) rather than answering from parametric knowledge, from both a normative perspective (when is a tool call truly needed/useful?) and a descriptive perspective (when does the model think it needs a tool, and how well does that judgment track the truth?).
Code: https://github.com/QinyuanWu0710/ToCallorNotToCall
This repository hosts the four benchmark inputs built or curated for that framework. It does not include the InVivo query set referenced in the code repository — that data is derived from real user conversations and is withheld pending an internal PII review.
It also does not include the GSM-Hard calculator benchmark used in the paper's calculator experiments, since that dataset is third-party data already published at reasoning-machines/gsm-hard; the code repository downloads it directly from there.
Configs / Subsets
Load any subset with:
from datasets import load_dataset
entity = load_dataset("QinyuanWu0710/ToCall_or_NotToCall", "entity")
bfcl = load_dataset("QinyuanWu0710/ToCall_or_NotToCall", "bfcl")
synth_mult = load_dataset("QinyuanWu0710/ToCall_or_NotToCall", "synthetic_multiplication")
synth_nn = load_dataset("QinyuanWu0710/ToCall_or_NotToCall", "synthetic_nn")entity (500 rows)
Entity-centric knowledge queries used for the web-search experiments (no-search / auto-search / force-search conditions). Entities were extracted from real-world conversational queries, classified into topical categories, and de-identified down to the entity mention itself — no conversation text or user-identifying information is included.
bfcl (314 rows)
Chained factual-lookup questions in the style of the Berkeley Function-Calling Leaderboard: each question requires looking up one fact, then using that fact to answer the next question in the chain (e.g. "Who won Best Picture at the 2024 Academy Awards?" → "Who is the director of \[that film\]?"). Designed so a model without live web access will systematically drift from the correct chain as questions get further from its training cutoff.
synthetic_multiplication (1,000 rows)
Synthetic multi-factor multiplication problems (GSM-style final-answer format) used for the calculator tool-use experiments, generated by data/generate_multiplication_dataset.py in the code repo. Difficulty scales with the number of factors and their digit lengths, giving a controllable necessity signal for "does this task actually require a calculator."
synthetic_nn (1,000 rows)
Synthetic large-integer squaring problems (N * N for large N), the harder companion set to synthetic_multiplication. Difficulty is labeled by operand digit length (e.g. 27-digit) rather than by named tier, since a single repeated-operand square already saturates non-tool arithmetic well before 1,000-digit multiplication does.
Both synthetic sets are used with --task synthetic_multiplication in the code repo; which file is scored is controlled by pointing --data/DATA at the corresponding JSONL file.
Intended Use
These datasets are intended for evaluating and studying when an LLM agent should invoke external tools (web search, calculator) versus answering from parametric knowledge — e.g. reproducing the paper's necessity/utility/affordability analysis, or as inputs to other tool-use / agentic-evaluation research. entity and bfcl are best paired with a web-search tool; synthetic_multiplication and synthetic_nn are best paired with a calculator tool.
License
Released under the MIT License, matching the code repository.
Citation
@article{wu2026call,
title={To Call or Not to Call: A Framework to Assess and Optimize LLM Tool Calling},
author={Wu, Qinyuan and Das, Soumi and Amani, Mahsa and Nag, Arijit and Lee, Seungeon and Gummadi, Krishna P and Ravichander, Abhilasha and Zafar, Muhammad Bilal},
journal={arXiv preprint arXiv:2605.00737},
year={2026}
}