Datasetpublic

NagaYu/assay-receipts

Assay Receipt Corpus Signed internal receipts from an inference provider that is sometimes cheating, together with the verdict an auditor reached on each one and the ground truth of which model actually served the request. Each row is a real receipt, not a summary statistic: it carries the prompt and output token ids, the JL-projected sketch of the provider's hidden_states, the sign/rank invariants, and an HMAC signature. With the gpt2 weights you can recompute the sketch… See the full description on the dataset page: https://huggingface.co/datasets/NagaYu/assay-receipts.

sourceHugging Facemitupdated 18d agoView on Hugging Face

0likes55downloads

Dataset Card

Assay Receipt Corpus

Signed internal receipts from an inference provider that is sometimes cheating, together with the verdict an auditor reached on each one and the ground truth of which model actually served the request.

Each row is a real receipt, not a summary statistic: it carries the prompt and output token ids, the JL-projected sketch of the provider's hidden_states, the sign/rank invariants, and an HMAC signature. With the gpt2 weights you can recompute the sketch yourself and check whether the published verdict was right.

—Code: github.com/NagaYu/assay
—Demo: NagaYu/assay-demo
—Model bundle: NagaYu/assay-tiny-pair

What is being detected

A provider advertises model M (gpt2, 12 layers) and may quietly serve a cheaper M' (distilgpt2, 6 layers, or an INT8 copy of M). Every response ships a receipt; a verifier recomputes one secretly-chosen layer and compares. The corpus covers six provider behaviours:

`scenario`	Provider behaviour
`honest`	Always M. The false-positive control.
`size`	Always M' (distilgpt2).
`quant`	Always an INT8 copy of M.
`random`	M' on 10% of requests — what defeats a sampling audit.
`evasive`	Answers the auditor's canary prompts with M, cheats on everyone else.
`forge`	Decodes with M', then runs M once over M''s own output to build a genuine receipt.

The forge rows are the ones worth studying. Their receipts are authentic — every signature verifies, every hash binds, and the sketch distance is better than honest, because the forger recomputes without the numerical jitter a real serving stack has. A matching sketch proves M was run over the text, not that M produced it. Only the token-level generation check (gen_mismatch on rows where deep_checked is true) separates them.

The two statistics fail in opposite directions on the same rows:

scenario	median `z`	token mismatch rate on deep-checked rows
`honest`	−0.06	0.00%
`size`	+49.33	73.2%
`forge`	−25.07	73.2%

A one-sided test on z flags large values, so a forged receipt — which sits 25 standard deviations on the clean side — is not merely missed, it looks like the most honest traffic in the corpus. The token check sees the same rows at 73% disagreement.

Fields

Field	Meaning
`scenario`, `trial`, `step`, `request_id`	Where the row came from
`served_by`, `is_substituted`	Ground truth — the label a real auditor never sees
`prompt_ids`, `output_ids`	Token ids, enough to recompute the activations
`layers`, `positions`	Which layers and token positions the receipt covers (PRF-derived)
`sketch_f16_b64`, `signs_u8_b64`, `ranks_i16_b64`	The receipt itself, base64 of a `(layers, positions, dim)` array
`sig_alg`, `signature`	HMAC-SHA256 over the receipt's canonical bytes (simulating a TEE attestation)
`layer_checked`, `mean_dist`, `z`, `p`	What the verifier did and concluded
`sign_agreement`, `rank_overlap`	The jitter-invariant features
`deep_checked`, `gen_checked`, `gen_mismatch`	The generation-consistency check, when it ran
`cost_ratio`	Verifier cost as a fraction of a full recompute
`audit_log_e`	Cumulative log e-value of the sequential audit after this request

protocol.json carries the sketch parameters, the calibrated tolerance band, and the demo HMAC key. The key is published on purpose — without it the signatures cannot be checked. It is a demo key for a research prototype, not a secret and not a TEE key.

Verifying it yourself

bash

pip install git+https://github.com/NagaYu/assay.git
huggingface-cli download NagaYu/assay-receipts --repo-type dataset --local-dir assay-receipts

python -m benchmarks.verify_receipts \
  --receipts assay-receipts/receipts.jsonl \
  --protocol assay-receipts/protocol.json \
  --model gpt2 --limit 100

This rebuilds each Receipt, checks the signature, recomputes the sketch on the published layer, and compares against the published z. Note that recomputing on a different layer legitimately gives a different answer — which layer gets checked is the verifier's secret draw, and not reproducing it is the point of that draw.

Suggested uses

—Score your own detector against is_substituted without running any models.
—Study where the sketch statistic fails: sort the forge rows by z and note they are more consistent with M than the honest rows.
—Replay the sequential audit with a different e-value or a different α.
—Check how much evidence a single request carries under each behaviour.

Limitations

Small models (gpt2 / distilgpt2) on CPU, short greedy generations, and a simulated nondeterminism model (~3e-4 relative per-layer noise) rather than measured GPU nondeterminism. The provider's signing key is not protected by a TEE. Substitution detection comes from sketch consistency and the generation check, not from the signature. See the repository README for the full caveats.

Citation

bibtex

@software{assay2026,
  title  = {Assay: per-request internal receipts for model-substitution detection},
  author = {Nagao, Yuta},
  year   = {2026},
  url    = {https://github.com/NagaYu/assay},
  note   = {Research prototype}
}