Ali-Bhai/toolace-ragtruth-style-hallucinations

ToolACE RAGTruth-style Tool Hallucination Dataset

This dataset was built from ToolACE tool-use dialogues and converted into a RAGTruth-style format for hallucination detection in tool calling.

Task

Given:

• —query: user query
• —context: tool response
• —output: final assistant answer

the goal is to classify whether the answer is grounded in the tool output or belongs to one of three hallucination types.

Labels

• —clean
• —tool_output_conflict
• —overgeneration
• —missing_tool_action_recommendation

Synthetic hallucinated examples contain RAGTruth-style character-span labels in hallucination_labels.

Splits

The default files are the mixed clean + synthetic splits:

Split label counts

{
  "train": {
    "overgeneration": 276,
    "clean": 834,
    "missing_tool_action_recommendation": 276,
    "tool_output_conflict": 276
  },
  "validation": {
    "clean": 103,
    "overgeneration": 37,
    "tool_output_conflict": 37,
    "missing_tool_action_recommendation": 37
  },
  "test": {
    "clean": 96,
    "missing_tool_action_recommendation": 37,
    "tool_output_conflict": 37,
    "overgeneration": 37
  }
}

Loading

from datasets import load_dataset

dataset = load_dataset("Ali-Bhai/toolace-ragtruth-style-hallucinations")
print(dataset)

If automatic split detection is not available in the runtime, use:

from datasets import load_dataset

dataset = load_dataset(
    "json",
    data_files={
        "train": "hf://datasets/Ali-Bhai/toolace-ragtruth-style-hallucinations/train.jsonl",
        "validation": "hf://datasets/Ali-Bhai/toolace-ragtruth-style-hallucinations/validation.jsonl",
        "test": "hf://datasets/Ali-Bhai/toolace-ragtruth-style-hallucinations/test.jsonl",
    },
)

Source and construction

The clean records were extracted from ToolACE dialogues with exact tool-call/tool-output matching. Synthetic hallucinations were injected using three controlled policies:

1. 1.numeric/date tool-output conflict,
2. 2.natural unsupported overgeneration,
3. 3.recommendation of an unavailable tool.

See metadata/dataset_manifest.json for the full file manifest.