Datasetpublic

Ced-Collab/KiCadData

KiCadData — KiCad (cua-bench) agent trajectories Trajectories, model transcripts and logs of memory-augmented agents (codex CLI and Claude Code) on the 25 KiCad schematic tasks of cua-bench (cua.bench.local.kicad; official score = cua_bench.netlist_compare, the mean of three all-or-nothing checks: components, nets, net names). Companion of the Minecraft dataset Ced-Collab/MCUData. kicad-0915 (2026-09-14/15) Three cells of the KiCad results table (CUA-KiCad README)… See the full description on the dataset page: https://huggingface.co/datasets/Ced-Collab/KiCadData.

sourceHugging Facemitupdated 9d agoView on Hugging Face

0likes639downloads

Dataset Card

KiCadData — KiCad (cua-bench) agent trajectories

Trajectories, model transcripts and logs of memory-augmented agents (codex CLI and Claude Code) on the 25 KiCad schematic tasks of cua-bench (cua.bench.local.kicad; official score = cua_bench.netlist_compare, the mean of three all-or-nothing checks: components, nets, net names). Companion of the Minecraft dataset Ced-Collab/MCUData.

kicad-0915 (2026-09-14/15)

Three cells of the KiCad results table (CUA-KiCad README), run one task at a time with --kicad-project-hint, the LN271->LD271 task patch, all 25 tasks (the two gpt cells through codex CLI 0.153.4, the Opus cell through Claude Code 2.1.257, see below):

run	agent	model	effort	mean_reward (25)	on the 24-task subset*	tasks at 1.0 / 0.0	steps/task	wall
`kicad_vista_gpt56high_hint`	VISTA (`--agent vista`)	gpt-5.6-sol	high	0.400	0.375	7 / 12	51.8	5.9 h
`kicad_mfmcarry3design_gpt6high_hint`	MFM (`--agent mfm-carry --carry-outcomes --design-turn --design-effort high --conventions`)	gpt-6-astra	high	0.440	0.417	8 / 10	45.9	7.9 h
`kicad_vista_opus5high_hint`	VISTA (`--agent vista --cli claude`)	claude-opus-5	high	0.373	0.347	6 / 12	41.5	10.0 h

\* without a6414tw4 ("create an empty netlist": a saved blank schematic scores 1.0), the subset the older 24/25 cells of the table hold. All three runs: n_errors 0. smokes/ holds the one-task (154d0750) smoke of each arm.

The Opus cell is not the codex harness: the same VISTA agent (tools, prompts, GUIDE.md/WORKING.md, the checkpoint handoff) drives claude -p (Claude Code) with the VISTA tools served as an MCP server (mfm/vista/mcp_bridge.py -> mfm/vista/claude_runner.py), frames returned inside the tool results, one CLI process per segment (--resume after a checkpoint), inside prolong_search/claude_sandbox.sh (bwrap: workspace read-write, network limited to api.anthropic.com/claude.ai, cleared environment) on a Claude subscription account. Compare it within the Opus column only. One task (a6dd38a9) hit a harness bug on its first attempt (a read-only bridge copy in a reused config dir, MFM f9e7ab3) and was re-run alone after the pass; the row here is the re-run. The access token expired once mid-run (19:43); the driver (harness/opus_plan.sh) refreshed it and resumed, finished tasks untouched.

Code: gpt cells MFM kicad-results-0913 @ c4df5d9 (pins CUA-KiCad d65e188, which holds gym_codex/run_eval.py, the agents and docs/KICAD_RUNS.md); Opus cell MFM @ f9e7ab3 (CUA-KiCad 2d307ce; f9e7ab3 itself only matters for the a6dd38a9 re-run). cua-lite 65a61c4, cua-bench 0.2.11, the cua-bench datasets from github.com/trycua/cua @ b89e4fc (2026-09-14), desktop image docker.io/trycua/cua-xfce:latest (5d90f09c).

Deviations from the FASRC wave setup (record beside the scores): the runs were made on a lab host without a subuid range, so the cua-xfce container's services (Xtigervnc, noVNC, computer-server) run as root instead of cua and the computer API binds 0.0.0.0 (published on the loopback only); the model account is an OpenAI API key, not a ChatGPT login (Opus: a Claude subscription through Claude Code); HF_HUB_OFFLINE=1 skipped cua-lite's unrelated ScreenSpot-Pro download. See logs/PROGRESS.md and harness/.

Layout

runs/<run>/summary.json, results.jsonl          score and per-task rows (reward, steps, wall_s, error)
runs/<run>/<task>/instruction.txt               the task
runs/<run>/<task>/trajectory.json               steps: action batches, screenshots, agent messages
runs/<run>/<task>/step_NN.png                   the screenshot after each step
runs/<run>/<task>/kicad_project/, graded.net    the saved project and the netlist that was graded
runs/<run>/<task>/codex/                        model transcripts: VISTA app-server segments
                                                (segment_NNN.{prompt.txt,requests.jsonl,stdout.jsonl,stderr.txt}),
                                                MFM exec turns (turn_NNNN.events.jsonl, design/); for the Opus
                                                cell the Claude Code stream-json per segment (segment_NNN.
                                                {command.json,prompt.txt,requests.jsonl,stdout.jsonl,stderr.txt};
                                                image payloads replaced by sha256 stubs, the PNGs are the frames)
runs/<run>/<task>/vista/action_log.jsonl        VISTA: every play/inspect call
runs/<run>/<task>/workspace/                    the agent's files: GUIDE.md/WORKING.md (VISTA); notes/, episodes/,
                                                tools/, frames/ (MFM; previous/ = the carried copies of earlier
                                                tasks' workspaces, dropped here as duplicates)
smokes/<run>/...                                the one-task smokes, same shape
stats/runs.csv                                  one row per (run, task)
logs/                                           run logs, night_plan.log / opus_plan.log (the drivers), smoke logs, PROGRESS.md
harness/                                        launch_kicad.sh, night_plan.sh, opus_plan.sh, rerun_a6dd.sh, the podman
                                                shim, storage.conf, stage_hf_kicad.py (this staging)

Not included: the per-episode codex / Claude Code homes (workspace.codexhome/, workspace.claudehome/: raw rollouts, session files, the credential mount point) — kept locally. Text files are anonymised (<COLLAB>, <HOME>, user); PNGs are untouched.