CLARKBENHAM/nist-gdt-pmi-vlm-benchmark

NIST GD&T/PMI VLM Benchmark

This benchmark measures exact-match transcription of geometric dimensioning and tolerancing (GD&T) and product and manufacturing information (PMI) from rendered NIST Fully-Toleranced Test Case drawing pages. A row supplies the page image and target element_id; the expected output is one engineering-significant specification string.

The reported evaluation uses open transcription: image plus element_id.

page_answer_choices is included for anyone who wants to run a candidate-constrained variant. Those candidates are derived from the labels themselves, so any score computed with them is an oracle upper bound, not a vision measurement. No candidate-constrained result is reported here.

Load

from datasets import load_dataset

dataset = load_dataset("CLARKBENHAM/nist-gdt-pmi-vlm-benchmark")
validation = dataset["validation"]
leakage = dataset["leakage"]
image = validation[0]["image"]
element_id = validation[0]["element_id"]

The release contains 300 validation rows and 58 leakage rows. There is no train or test split.

The 58 leakage rows are answerable from the element ID alone, without reading the drawing. DF2 is Datum Feature Symbol B because the DF<n> ordinal maps to the n-th datum letter; all 16 RLE rows share the constant target Represented line element; and ftc10_p1_FN4 is Flag Note Symbol 4. Each row's split_reason records which shortcut applies. They are published so this behavior can be measured — and excluded from headline accuracy so it isn't measured by accident.

Summary of results

Across the 6 observed misses of the locator-plus-crops arm, 3/6 omitted the second row of a stacked FCF, 2/6 were All-Around/scope cases (one omission and one extra nearby BOUNDARY), and 1/6 attached an adjacent FCF to a short note.

The estimated row combines two measured development cohorts rather than 358 direct Flash High predictions: all 42 corrected Flash Low residuals and a frozen 50-item retention sample from the Flash Low-correct stratum. It is shown alongside—not substituted for—the measured headline.

What moved accuracy

Hard30 contains 30 baseline-error-selected development items; /60 combines two correlated renders of those same questions.

Failure-mode ladder by original Flash Low failure class

The class describes the original Flash Low error that selected the item, not what a later High arm still gets wrong. Counts are descriptive; the header preserves the executor boundary rather than implying one causal staircase.

Caveat on the example arms: the six demonstrations are hand-ordered, not random, and were deliberately front-loaded with complete stacked feature control frames because that was the known dominant residual failure on this cohort. Every six-shot figure above is therefore tuned on the same 30 items it is measured on. The completed distinct-page Agy selector-parity audit found no resolved hand-selection advantage. At Gemini 3.7 High, standard, random, and hand-selected means were 27.0, 28.0, and 29.0/30; hand-selected minus random was +1.0 (95% item-cluster bootstrap CI -1.5 to +4.0), while the precommitted hand-selected-versus-standard primary was +2.0 (CI -0.5 to +5.0; Holm-adjusted p=0.75). At Gemini 3.6 Low, measured as transcription given fixed gemini-3.7-flash-low localization, hand-selected minus random was also +1.0 (CI -0.3 to +2.3). The older direct-API figures remain tuned descriptive results, not evidence of a repeatable demonstration-selection effect.

Takeaway: the best observed historical Hard30 setup was a Gemini 3.7 Flash High transcriber with gemini-3.7-flash-low locator-derived crops and six hand-selected examples: 59/60 render evaluations (30/30 raster, 29/30 vector600). In the paired Agy ablation, supplied crops were the only primary effect that survived Holm correction at both models; neither examples nor ownership did.

Shipping CLI agents on Hard30 (exploratory)

Same 30 difficult items, high effort, fixed corrected images, naive wording and typed-strict scorer; 210 runs plus four predeclared network/quota recoveries. Correct successful completions /30 planned (new totals include recoveries):

No run reached 60 minutes. Eight correct Gemini 3.8 first-pass completions exceeded five minutes; every Codex rerun finished within five minutes. Gemini 3.8 completed 29/30; a further correct DT3 answer preceded a terminal error (23 matching outputs; 19 frozen protocol points after that error and three access penalties). Other models completed 30/30. One development cohort, not a general model ranking. Counts and accounting.

Gemini generations, task wording and six examples (completed)

420 Hard30 slots: 360 new attempts plus 60 retained references, pinned Agy 1.2.2, corrected full pages, unrestricted detailed instructions and a 60-minute active limit. Strict correct/eligible, 30 planned per cell (lenient identical):

Examples improved all seven matched model/task comparisons; 15/24 gross wording fixes concerned three serialization conventions. Low thinking reduced mean time by 75%/83% (zero/six examples), with three/two clean matched accuracy regressions. Current six-example audited delivered answers were 28/29 for 3.7 High and 27/29 for Low; these include reviewed protocol failures and are sensitivity results. One development replicate, sequential references, incomplete coverage; no matched older-model pipeline arm. Subscription dollar/quota savings are unknown.

The autonomous agent approached the DAG's accuracy but used more inference work:

The DAG proxy includes both recorded stages; timings exclude crop materialization and scheduling. Fixed token weights are not billed dollars, current prices or quota consumption. This historical comparison uses different CLI versions. Traces mainly show row-ownership and serialization errors, despite identical current task rules in the inspected API, DAG and CLI requests. All 14 DT3 slots used the corrected render. Across 420 slots, 24 were unanswered (17 tool/boundary, 7 external failures); none hit the active timeout. Another 19 delivered answers are diagnostic-only, and one retrieval attempt is excluded. No recovery runs were used to fill denominators.

Full analysis, paired effects, mistake traces and progress.

Detailed final zero-shot development result

Validation accuracy by assembly

Validation accuracy by element type

The headline artifact is the completed Gemini 3.7 Flash Low run through Agy using the final visible-transcription v3 prompt, zero demonstrations, no answer choices or answer-key text, one requested element ID, and one high-resolution full page per request. All 358 requests opened the expected image; there were no provider errors or retries. Ground truth was joined only afterward for scoring. Prompt development used observed failures, so this is development evidence—not a prospective or contamination-free test result.

The exact prompt, condition, predictions, scores, sanitized per-item model inputs, request audit, and transcript audit are in `results/`. The embedded dataset pages are the same 22 high-resolution PNGs used by this run; their hashes are in `data/image_manifest.json`.

The headline is the lower number because it is the cleaner measurement: zero demonstrations, no exact-target exposure, no candidates. The historical few-shot result below is higher partly because 59/360 of its items had their exact target in the prompt.

Historical few-shot development result

The historical v1 artifact used six cross-assembly demonstration pages and standard-resolution images. Its source run had 59/360 items whose exact targets appeared somewhere in its demonstrations. This is not a prompt-only comparison with v3. Its predictions, generated scores, prompt, condition, and sanitized metadata are retained in `results/historical_v1/`.

Supplemental Flash High pipeline development check

These are not replacement headline scores or a completed 358-row Flash High run. The first measured cohort contains all 42 corrected Flash Low validation residuals. The second is a deterministic 50-item validation sample that excludes every explicitly targeted selection cohort; all 50 were correct under the Flash Low headline run. Normal Flash High used one full-page request. The other arm used two requests: a gemini-3.7-flash-high locator sub-agent that sees only the same rendered page plus the requested element ID and returns one bounding box around that printed ID label, then a transcription request over the full page plus quarter- and ninth-page crops derived from that box. The locator receives no target text, answer key, PDF text layer or CAD geometry. Neither arm received demonstrations, answer choices, or answer-key text.

The projections stratify the 300 validation rows into the 42 observed Flash Low residuals and 258 Flash Low-correct rows, using the fresh 50-item sample to estimate retention in the latter stratum. They are development-set estimates, not measured full-corpus scores. The candidate oracle assumes selection of a correct candidate and is not deployable. The leakage split was not rerun; combined figures are arithmetic scenarios that retain its published 58/58 outputs. Full assumptions, exact uncertainty bounds, provider-retry notes, item-level results, and source hashes are in `results/supplemental_flash_high/`.

Metric and evaluator

strict is exact match after the context-aware Layer-2 GD&T parser. It preserves engineering-significant datum compartments, modifiers, scope, stacked rows, zone prefixes, punctuation, and datum order while accepting documented glyph/name and harmless formatting aliases. lenient is a separately labeled legacy-style diagnostic. The vendored evaluator and reproduction instructions are in `evaluator/`.

Composition and limitations

• —The 358 rows are clustered on only 22 drawing pages from FTC assemblies 06–11; rows are not independent observations.
• —Both splits are development data. “Test” is reserved for genuinely unseen future material.
• —Prompt and grading-policy development used observed model failures.
• —Training contamination cannot be ruled out for any evaluated model.
• —The drawings are clean two-times-resolution PDF renders rather than scans or photographs.
• —Two source-only supplemental-geometry relationships were excluded because their requested semantics were not visible in the rendered page.

Provenance and NIST acknowledgement

The source material is NIST's MBE PMI Validation Fully-Toleranced Test Cases FTC-06 through FTC-11. NIST states that the test cases, CAD models, and STEP files may be used without restriction, requests acknowledgement, and prohibits promotional use of the NIST logo. This is an independent derived benchmark; NIST has not reviewed, recommended, or endorsed it. No NIST logo is used.

The published NIST label text was faithfully transcribed, but it did not always align with this benchmark's visible-transcription policy. We applied a small, versioned set of post-hoc label standardizations and track two explicit source/render errata: the FTC-06 page 3 render corrects the NIST source's duplicate DT3 element ID to D23, and FTC-10 page 2 T18 is standardized on the render's Position ⌀0.8|D|EⓂ-FⓂ-GⓂ even though nist_ftc_10_asme1_rb_fsi.pdf gives .08. The immutable FSI text remains in target_text_source, the AP242 semantic tolerance independently supports 0.8, and original per-image provenance remains in the release. The reconciled labels were model-cross-checked but did not receive independent second-human verification.

The original evaluator code is MIT-licensed under `LICENSE`. The drawings and NIST-derived dataset artifacts are not MIT-licensed and follow the terms in `NOTICE`.

Integrity

`SHA256SUMS` covers every public release file other than the checksum manifest itself. `release_manifest.json` freezes the source inputs and evaluator files used for this release.