caiotheodoro/assay-demo

Assay — in your browser

Submit an environment. Get an Environment Card back: a validity verdict where every claim is tied to a probe result, and a list of every probe that could not run and why.

That second half is the point. A card with no findings is not a clean bill of health, and this page refuses to render one as if it were.

Where the audit runs

This is a static Space. Hugging Face serves these files and runs no compute — which is why it is free where a Gradio Space is not. The probe battery still runs, in the visitor's tab, in a WebAssembly CPython (Pyodide 314.0.6) against the vendored assay package in assay.zip. Nothing a visitor pastes is uploaded, because there is nowhere to upload it to.

That works because Assay's audit path is pure standard library. The package declares one runtime dependency, pyyaml, and only assay.costs reaches it — no probe does. So the page needs no micropip, no wheel index and no network beyond its own files.

The seven bundled examples are pre-rendered into the page at build time by the same renderer the button calls, so the page is readable before the runtime loads and stays readable if it never does.

What it cannot check here

• —Single-turn, string-answer environments only. Multi-turn shell environments — where the interesting exploits live — need Docker. harbor/self-graded, the one environment in the corpus that Assay itself misses, cannot be expressed in this format.
• —The difficulty-band probe needs a rollout sampler and always reports NOT_APPLICABLE here, so no submission can reach VALID on this page.
• —verifier: "regex" is refused rather than degraded. assay.safe_regex bounds a submitted pattern by matching it in a subprocess under a wall clock, because Python's engine backtracks and (a+)+$ against 31 characters takes about 100 seconds. WebAssembly has no subprocesses, and falling back to a bare re.search would reinstate that denial of service on the one page it was about.
• —Caps: 200 tasks, 2,000 split items, 20,000 characters per item.

Synthetic data, and not production-validated

The bundled examples are synthetic fixtures with defects planted on purpose. Neither this Space nor the tool behind it is production-validated: it has been measured against defects its own authors planted, which is a lower bar than defects found in the wild. Do not use a VALID verdict here as sign-off on anything that matters. The card says as much, every time, and it stays unsigned until a human signs it.

What it is measured at

On a corpus that is mostly caiotheodoro-authored, with planted ground truth, Assay separates from a policy that flags every environment unread on all four cost profiles — but a large part of that margin is arithmetic and not detection: flagging everything pays a false alarm for every defect class on every environment, so the floor got worse when two classes and two environments were added, and a truthful detector gained nothing from either. Every number, with intervals and that decomposition, is on the dataset: `caiotheodoro/assay-corpus`.

The trained adversarial Challenger is a negative result and is published as one: `caiotheodoro/assay-challenger-grpo`.

Apache-2.0. Version v0.1.0.

The assay package is vendored into this Space at commit e08b73c, not installed from a remote — this checkout has none. That commit is the only way to tell which code produced the card you are looking at.