LockOnN/OpenVisTool-42K

OpenVisTool-42K

OpenVisTool-42K contains 42,048 outcome-valid, tool-use-gain-filtered visual tool-use trajectories across Chart, GUI Grounding, Table, Web-to-HTML, and Visual Search. Each example preserves the teacher's reasoning, function calls, tool observations, and final answer in the ms-swift agent format. All trajectories were synthesized using Qwen3.5-Plus as the teacher model. The teacher generated the reasoning traces, function calls, and final answers, while tool observations were obtained by executing the corresponding tools. This dataset accompanies OpenVisTool: An Open Recipe for Synthesizing Instructive Visual Tool-Use Trajectories.

Record schema

Every line is one JSON object:

{
  "id": "chart-00000000",
  "domain": "Chart",
  "tools": "[{\"type\":\"function\",...}]",
  "messages": [
    {"role": "system", "content": "..."},
    {"role": "user", "content": "<image>\n..."},
    {"role": "assistant", "content": "<think>...</think>"},
    {"role": "tool_call", "content": "{\"name\":\"crop\",...}"},
    {"role": "tool_response", "content": "<image>\n..."}
  ],
  "images": ["images/chart/ab/ab...ef.jpg"]
}

• —id: stable ID derived from the row index in the published domain shard.
• —domain: one of Chart, GUI Grounding, Table, Web-to-HTML, or Visual Search.
• —tools: a JSON-encoded string containing the function schemas expected by ms-swift. A decoded copy is available as metadata/tools.json.
• —messages: ordered agent messages. Roles are system, user, assistant, tool_call, and tool_response.
• —images: paths relative to the snapshot root. Entries align, in order, with every <image> token across messages. Duplicate entries are intentional when the same image is viewed more than once.

Paths such as /mnt/data/example.png inside messages are runtime paths in the agent's sandbox. They are not paths to the downloaded snapshot and should not be rewritten during training.

Source composition and provenance

Only source images and queries were used as task input; pre-existing source reasoning traces were not used as supervision. The teacher-generated trajectories were selected by outcome validity and measured tool-use gain.

Licenses are those declared by the upstream dataset cards. Source-specific terms continue to apply where an upstream dataset aggregates other datasets.

Tool-produced crops, masks, bounding-box visualizations, and HTML renderings are packaged alongside original inputs because they are observations in the training trajectories. These derivatives remain subject to any applicable terms of the original image corpus.