CoolFace
Datasetpublic

gnucleus-ai/cad-gen-freecad-bench

Parametric CAD Bench — results dataset Run-by-run results for Parametric CAD Bench, a benchmark that measures whether AI agents can author editable FreeCAD models from natural-language part descriptions. 1000 rows, one per (agent, model, task_id, trial) over the gnucleus-ai/cad-bench@v1 task suite. The public leaderboard view of this data lives at cadbench.ai. What's in here data/cad-bench-v1.parquet — the row table. Each row carries the composite + sub-scores… See the full description on the dataset page: https://huggingface.co/datasets/gnucleus-ai/cad-gen-freecad-bench.

sourceHugging Faceapache-2.0updated 1mo agoView on Hugging Face
2likes11kdownloads
Dataset Card

Parametric CAD Bench — results dataset

Run-by-run results for Parametric CAD Bench, a benchmark that measures whether AI agents can author editable FreeCAD models from natural-language part descriptions. 1000 rows, one per (agent, model, task_id, trial) over the `gnucleus-ai/cad-bench@v1` task suite. The public leaderboard view of this data lives at cadbench.ai.

What's in here

  • —`data/cad-bench-v1.parquet` — the row table. Each row carries the composite + sub-scores, token counts, USD cost, timing, and repo-relative paths to the per-trial audit files.
  • —`runs/<agent>/<model>/<task_id>/` — full audit tree for every scored trial:
  result.json          trial metadata (timing, exception, token counts, model info)
  config.json          trial config
  trial.log            high-level trial log
  answer.py            candidate FreeCAD script (when produced)
  answer.FCStd         candidate FreeCAD document (when produced)
  agent.log            agent stdout
  trajectory.json      aggregated agent rollout
  trajectory.jsonl     raw agent tool-call log
  reward.json          full grader breakdown (geometry + spec sub-scores)
  verifier_stdout.txt  verifier stdout (FreeCAD console + grader output)

Reading the dataset

python
from datasets import load_dataset
ds = load_dataset("gnucleus-ai/cad-gen-freecad-bench", split="train")
ds[0]   # one trial row, with scores + audit-path pointers

To pull a specific per-trial artifact:

python
from huggingface_hub import hf_hub_download
hf_hub_download(
    repo_id="gnucleus-ai/cad-gen-freecad-bench",
    repo_type="dataset",
    filename=row["audit_dir_path"] + "/answer.FCStd",
)

…or the whole per-trial tree via snapshot_download(repo_id=..., allow_patterns="<audit_dir_path>/*").

Notes

  • —The held-back reference geometry and the scorer's tolerances are not in this repo — they live inside the task images at root-only paths so the agent can't read them at trial time.
  • —Source CAD designs (the instructions, parameter specs, and reference geometry the bench is built from) are at `gnucleus-ai/cad-gen-freecad`.
gnucleus-ai/cad-gen-freecad-bench · CoolFace