CoolFace
Datasetpublic

qualifire/tool-selection-quality-benchmark

Tool Selection Quality Benchmark A benchmark for evaluating whether an LLM correctly judges the quality of a tool call / function call made by another model - i.e. given a user request, the tools available, and the model's resulting function call (or direct reply), did the model pick the right tool and fill it in correctly? Each row is one turn to be judged: a message history ending in either a function call or a direct assistant response, paired with the set of tools that were… See the full description on the dataset page: https://huggingface.co/datasets/qualifire/tool-selection-quality-benchmark.

sourceHugging Facecc-by-nc-4.0updated 2mo agoView on Hugging Face
0likes54downloads
Dataset Card

Tool Selection Quality Benchmark

A benchmark for evaluating whether an LLM correctly judges the quality of a tool call / function call made by another model - i.e. given a user request, the tools available, and the model's resulting function call (or direct reply), did the model pick the right tool and fill it in correctly?

Each row is one turn to be judged: a message history ending in either a function call or a direct assistant response, paired with the set of tools that were available, and a ground-truth verdict on whether that call was correct and, if not, what specifically went wrong.

Dataset Structure

ColumnTypeDescription
tools_liststringStringified list of the tool/function definitions (name, description, JSON-schema parameters) available to the model for this turn.
messages_historystringThe conversation up to and including the model's response: the user's request and either a function_call (tool name + arguments) or a direct assistant reply.
labelstringGround-truth verdict - one of VALID_CALL, TOOL_ERROR, PARAM_NAME_ERROR, PARAM_VALUE_ERROR (see taxonomy below).
binary_labelstringCollapsed verdict: CORRECT (VALID_CALL) or INCORRECT (any of the three error labels).
failure_reasonstringShort free-text reason for the failure (e.g. "incorrect tool selected", "hallucinated parameter name"), or "None" when label is VALID_CALL.

Example row

A user asks Cribl to fetch a pipeline's config; the model correctly calls cribl_getPipelineConfig with the right pipelineId/groupName arguments → label: VALID_CALL, binary_label: CORRECT, failure_reason: None.

Label Taxonomy

Verdicts follow a strict, ordered check - tool selection first, then parameter structure, then parameter values - and the first failing check determines the label:

  1. 1.Tool Selection - does the chosen tool exist and match the user's intent? If not → TOOL_ERROR (e.g. wrong tool picked, or a tool call made when none was needed, or vice versa).
  2. 2.Parameter Structure - are the required parameter names present, and are there no extraneous/hallucinated ones? If not → PARAM_NAME_ERROR.
  3. 3.Parameter Values - are the parameter values correctly typed, formatted, and factually consistent with the request? If not → PARAM_VALUE_ERROR.

If none of these trigger - including the case where the user's request needed no tool call at all and the model correctly replied directly - the turn is VALID_CALL.

Dataset Statistics

  • —5,000 rows total.

label distribution:

LabelCount
VALID_CALL2,000
TOOL_ERROR1,000
PARAMNAMEERROR1,000
PARAMVALUEERROR1,000

binary_label distribution:

LabelCount
CORRECT2,000
INCORRECT3,000

Usage

python
from datasets import load_dataset

ds = load_dataset("qualifire/tool-selection-quality-benchmark")["test"]
print(ds[0])

A common evaluation pattern is to score a judge model on each row (given tools_list + messages_history) and compare its predicted label - or the collapsed binary_label - against ground truth, both overall and broken down per label.

Considerations for Use

  • —This benchmark is intended for evaluating tool-use/function-calling judge models - i.e. models whose job is to critique another model's tool call, not models that make tool calls themselves.
  • —Labels reflect a specific, ordered evaluation rubric (tool → parameter names → parameter values); treat them as a strong baseline rather than an unimpeachable gold standard.