harrrshall/cad-preserve

CAD-Preserve v0.1.0

Executable CAD edits that must preserve the rest of the part. This original synthetic collection contains 80 STEP parts, four parameter cases per part, reference Python programs, geometry graders, 240 controlled-error training pairs and hosted-model evaluation traces.

It is for language-model CAD tool use and verifier research. It contains no robot actions, camera recordings, factory observations or expert-certified manufacturing designs. The collection is a small research/development benchmark; a claim of general frontier-model difficulty or improved model training is not established.

[image]

What is included

• —80 original source parts: 40 train, 12 validation, 28 test, with exact source specifications disjoint across splits.
• —320 revised target solids: the nominal change plus three parameter checks per part.
• —Five procedural families: intersecting bores, clevis gaps, pocket depths, rounded manifolds with oblique/toroidal passages, and curved-roof housings. All 12 curved-roof housing tasks are in test; that family is absent from train/validation.
• —80 executable reference edit programs and full original design specifications for auditing.
• —80 input-only surface inventories (1,841 observed faces): an optional structured representation extracted from the original STEP. These inventories were not supplied to the reported model baselines.
• —240 training pairs on the 40 train parts only: correct versus controlled incorrect code, error explanations, candidate B-reps and measured scores. Forty rejected programs pass nominal geometry but fail parameter variation. These are authored mutations, not observed model errors.
• —30 scheduled model episodes: ten selected test parts per model, one run each, with actions, observed tool outputs, available final submissions and separate failure categories. Provider private reasoning and account credentials are excluded.
• —15 observed first edit proposals, independently rescored, in a separate test-only configuration. Three contained geometric errors and one failed execution before a later correct final submission.

There are 80 source problems, not hundreds of independent domains: parameter targets and mutations are derived from those same five templates. Public target solids and solutions support auditing but make this unsuitable as a permanently hidden test set.

Task contract

Given the original STEP and a change order, produce a complete Python function:

def edit(shape, value):
    # Return one valid CadQuery solid implementing the requested value.
    # The original shape is passed afresh on every invocation.
    ...

Use millimetres and preserve the original coordinate frame. All non-target geometry must stay identical. Resizing a port must preserve an intersecting channel; narrowing a clevis gap must preserve its pin bores; restoring material below a curved roof must follow that roof.

The grader checks solid validity, extra and missing material in both directions, and specified functional points. Per-direction volume tolerance is max(1e-5 mm³, reference_volume × 1e-8). This is an ideal CAD equivalence criterion, not a manufacturing-tolerance standard. Volume IoU is diagnostic only. The final program must pass all four parameter values. The harness reveals only nominal test feedback during an episode.

Measured baselines

• —deepseek-v4-flash-0731: 0/10 scheduled episodes passed all four parameter cases. 2 no submission within turn limit, 8 provider or transport failure.
• —glm-5.3: 9/10 scheduled episodes passed all four parameter cases. 9 all parameters pass, 1 provider or transport failure.
• —qwen-3.8-27b-fp8: 6/10 scheduled episodes passed all four parameter cases. 6 all parameters pass, 3 provider or transport failure, 1 response token limit.

These counts include all scheduled episodes; provider/transport failures are explicitly separated and are not evidence of a model capability gap. Turn, token and CPU limits are also reported separately from incorrect submitted geometry. All 15 final submissions passed. DeepSeek also passed a separately reported diagnostic rerun of one bore task; its primary zero is not evidence it cannot solve these tasks. This is one small exploratory sample, conditional on the supplied tools and scaffold, not a proprietary-frontier ranking. See RESULTS.md and the machine-readable results for each episode and the additional diagnostic runs.

[image]

15 episodes produced a complete edit proposal. 11 first proposals passed all four values; 4 initially failed and later passed in the same episode.

• —glm-5.3: 5/9 first complete proposals passed all four values; 9/9 corresponding final submissions passed.
• —qwen-3.8-27b-fp8: 6/6 first complete proposals passed all four values; 6/6 corresponding final submissions passed.

“First” means the first complete program sent to an explicit test/submission tool, after any inspection and scratch code. This conditional comparison excludes episodes with no such program. It is not a randomized feedback ablation or proof that feedback alone caused improvement. The observed error/correction records and controlled mutations support repair and verifier experiments; their training benefit remains unmeasured.

Load the records

from datasets import load_dataset
from huggingface_hub import snapshot_download

repo = "harrrshall/cad-preserve"
tasks = load_dataset(repo, "tasks", revision="v0.1.0")
pairs = load_dataset(repo, "preferences", revision="v0.1.0")
proposals = load_dataset(repo, "model_proposals", revision="v0.1.0")
root = snapshot_download(repo, repo_type="dataset", revision="v0.1.0", local_dir="cad-preserve")

row = tasks["train"][0]
print(row["prompt"], row["nominal_value"])
# row["input_file"], row["reference_file"], row["gold_files"]
# are paths relative to the downloaded snapshot.

The tasks configuration has prompts, parameter ranges and asset paths, including observation_file for optional input-only face geometry. The preferences configuration has chosen and rejected executable code, source-part IDs, error categories and per-parameter scores. The model_proposals configuration contains observed first complete edit proposals from test episodes, rescored at all four values; keep these out of training when using this test split. Native .step and .brep assets are ordinary OpenCascade-compatible CAD files; Python code and JSONL need no custom Hugging Face loading script. Use local_dir to materialize a working directory before running repository scripts, rather than running them from symlinked Hub-cache blobs. See SCHEMA.md for field semantics and constructing an input without answer leakage.

Verify locally

Use Python 3.12 and the pinned CAD runtime:

python3.12 -m venv .venv
.venv/bin/pip install -r requirements.txt
.venv/bin/python tools/verify_release.py --all

This offline command checks every manifest hash, split integrity and all target solids. Grade your own model's saved Python program without a paid model API using tools/grade_submission.py --task-id TASK_ID --code your_edit.py. For a working reference example:

.venv/bin/python tools/grade_submission.py --task-id bore_resize-000100 --code cases/bore_resize-000100/reference.py

tools/replay_references.py executes all positive edit programs through the OS sandbox. Reference execution and the baseline harness were tested on macOS/Apple Silicon. For Linux, build the supplied image with docker build -t cad-preservation-python:0.1 .; the Linux Docker path is provided but was not executed for this release. The harness fails closed if its sandbox is unavailable.

tools/run_evaluation.py resumes the recorded grid and skips existing results. For a fresh billed episode, set JARVISLABS_API_KEY and call tools/runner.py --model glm-5.3 --case bore_resize-000114 --run-id my-replication. Use a new run ID; the runner refuses to overwrite results. This uses the frozen catalog and scaffold, subject to future provider availability, with an estimated INR 1,000 ceiling for a fresh ledger. Inspect limits and current pricing before paid use. Keep reference answers, generators and target solids outside the evaluated model's filesystem; the supplied harness mounts only the input STEP. Released logs redact local paths and exclude private reasoning and request financial accounting.

Validation and provenance

• —All 80 reference programs passed all 320 parameter executions in the model-code sandbox.
• —204,800 independent scalar-arithmetic occupancy comparisons agreed with the generated B-reps.
• —The grader rejected 640 controlled corruptions across the 80 parts and accepted alternate valid construction paths.
• —All three Hugging Face record configurations were loaded locally and their row counts checked.

All geometry and task text were created for this release. References are authored programs with access to the generating specification; they are not independent expert solutions. No model was fine-tuned and no training improvement was measured. See METHODS.md for construction, prior work, split limitations, grader details and research citations dated August–September 2026. See THIRD_PARTY.md for runtime/source-excerpt attribution.

The dataset assets and original annotations are CC BY 4.0; executable software/reference programs are MIT. Third-party excerpts retain their original licenses. Cite the repository and version using CITATION.cff.