NagaYu/clearance-bench

Clearance Benchmark

A fully synthetic enterprise corpus for measuring how well a retrieval system contains information flow: heavily overlapping ACLs, sensitivity levels, time-limited grants, a grant/revoke timeline, and evaluation queries.

Generated by `clearance.synth` with seed 7. No real documents, no real access-control lists, and no real identities. Regenerating with the same seed reproduces this dataset byte for byte.

Why this exists

Query-time ACL filtering protects the index. It does not protect the summaries, agent memories and caches a RAG stack derives from that index. This dataset supports measuring three distinct leak channels — direct, existence-inference, and derivative — plus revocation lag, recall under high selectivity, and index memory.

Configs

access_events holds only grant rows by design. Revocations are replayed at evaluation time from revocation_cases, so each system under test is measured against the same withdrawal on a directory it has not already mutated. Rebuilding a directory per condition is what keeps a later condition from "passing" merely because an earlier one already expired the grant.

Access semantics

A principal may read a chunk when all of these hold:

1. 1.allow_tokens ∩ principal_tokens ≠ ∅ (OR over capabilities),
2. 2.principal.level >= min_level (no read-up),
3. 3.not_before <= t < not_after (embargo window),
4. 4.the capability itself has not expired or been revoked at time t.

Canaries

Every chunk carries a unique CLR-<doc>-<ordinal> marker inside its body text. Markers are stripped before embedding, so they never influence retrieval — they exist only so leak detection through derived text is exact rather than heuristic.

Intended use and limits

Built to measure defensive performance of index designs on synthetic data. The probe queries target material the asking principal may not read; they exist to test whether a defence holds and are meaningful only because the generator knows the ground truth of a corpus it fabricated. This dataset is not an attack toolkit and carries no claim about any deployed system.

Embedding-inversion attacks (recovering source text from vectors) are a real, documented risk for any shared vector store. They are referenced here as prior work and deliberately not implemented.

Text is template-generated and is not a language-modelling benchmark. It exists to give retrieval something topically clustered to retrieve.