violetxi/toolathlon-qwen35-4b-base-pass1-pilot15

Toolathlon Qwen3.5-4B Pass@1 Pilot

Readable trajectories from a 15-task Toolathlon-Verified Pass@1 pilot using Qwen/Qwen3.5-4B in thinking mode. The model was served on 6 B200 GPUs with data parallelism; Toolathlon's Docker environments ran on its remote public evaluation service.

Contents

• —Repository: violetxi/toolathlon-qwen35-4b-base-pass1-pilot15
• —Rows: 15 (one row per task)
• —Reported result: 5/15 (33.33%)
• —messages: a Viewer-friendly list containing only user and assistant roles. Tool calls are rendered inside assistant messages; tool results are rendered as labeled user messages so the full interaction remains readable.
• —Per-row task status, pass/fail, timing, and tool-call counts are included.

Important infrastructure caveat

This pilot is not a clean performance measurement. vLLM used a 65,536-token context window while each request reserved up to 32,768 output tokens. The run recorded 114 HTTP 400 context-length errors once prompts exceeded the remaining input budget. The trajectories are useful for qualitative inspection, but the reported 5/15 Pass@1 should not be used as a canonical model score.

Transformation and safety

The source trajectories mixed user, assistant, and tool records. This dataset converts them to a two-role readable representation and scans/redacts credential-shaped strings. Task workspaces, authentication files, and raw environment archives are not uploaded.