marrita/toolace-tool-calling-hallucination-ragtruth

ToolACE-derived Tool-Calling Hallucination Dataset

This dataset was created for the course assignment Hallucination Detection in Tool Calling.

It is synthetic by design: starting from ToolACE-style tool-calling dialogues, we automatically inject three required hallucination types:

1. 1.tool_contradiction
2. 2.overgeneration
3. 3.missing_tool

Each example follows a RAGTruth-like format:

• —query: user query
• —context: tool output
• —output: final model answer
• —hallucination_labels: span-level hallucination labels with character offsets

Additional fields such as tools, tool_call, source_id, split, and corruption_type are included for reproducibility and analysis.

Splits

The dataset contains train, validation, and test splits. Splits are separated by source_id to avoid leakage between clean and corrupted variants of the same original dialogue.

Evaluation setup

Clean examples are negative at sentence level. The three corrupted types are positive. The dataset supports both sentence-level binary hallucination detection and span-level hallucination localization.

Required hallucination types

The dataset contains the three hallucination categories required by the assignment:

• —tool_contradiction: the answer contradicts the tool output;
• —overgeneration: the answer adds unsupported information not present in the tool output;
• —missing_tool: the answer suggests an action requiring a tool that is not available.

The main dataset contains all examples with a corruption_type column. In addition, per-type dataset configurations are uploaded for each hallucination category.

Limitations

The dataset is automatically corrupted and may contain artifacts of the corruption procedure. This is expected for the assignment, because the task requires automatic hallucination injection. The dataset is suitable as a controlled benchmark for this project, but real-world generalization should be evaluated separately.