cnuland/llm-d-sc-sensitivity-v3-prior
llm-d-sc-sensitivity-v3-prior
sensitivity classifier for llm-d semantic classification. Labels: PUBLIC, INTERNAL, CONFIDENTIAL, REGULATED, NEVER_EGRESS.
Architecture: sequence-classification head (requires a runtime that reads logits), base BAAI/bge-base-en-v1.5.
Accuracy
Read the real-traffic row first.
Per-tier recall — read this before the accuracy
For an egress or capacity gate the errors are not interchangeable: missing a NEVER_EGRESS prompt leaks a live credential, while over-flagging a PUBLIC one costs a cheap round trip. Aggregate accuracy is carried by the largest class and hides this.
Run-to-run variance. PUBLISHED SEED IS THE FIRST ONE RUN (11), not the best. Both seeds:
seed 11 entsec 0.8274 seed 22 entsec 0.8331 seed 33 entsec 0.8359 seed 44 entsec 0.8289 MEDIAN OF 4 SEEDS: entsec 0.8310, against 0.7808 for the originally shipped v2 -- +5.02 points.
CORRECTION TO AN EARLIER VERSION OF THIS CARD. It described the prior-matching gain as '+1.27, nine times the noise floor', citing a floor of 0.0014. That figure was the MINIMUM within-configuration seed spread ever observed on this signal, not a floor, and it was measured on a different configuration (MiniLM) than the one it was applied to. Measured properly on four seeds of THIS config, sigma is about 0.0041 and the effect is +1.34 points, i.e. 3.0 sigma rather than 9.6x. The effect is real; the near-certainty implied by the original phrasing was not supported.
A noise floor is a property of a CONFIGURATION -- encoder, corpus, loss -- not of a signal. Two-seed spreads also systematically UNDERSTATE variance: this config's spread grew from 0.0057 at n=2 to 0.0085 at n=4, because the range is a biased estimator at small n.
THREE CHANGES, EACH MEASURED SEPARATELY, NONE OF THEM AN ARCHITECTURE.
- Tier-escalated class weighting REMOVED (+1.91). It was introduced as a safety measure and had never been audited against the safety metric that justified it; swept over 0.0/0.5/1.0 the effect is monotone and it RAISES over-block at every gate while buying under a point of containment.
- bge-base-en-v1.5 instead of all-MiniLM-L6-v2 (+4.53 at fixed schedule and corpus). Costs 31.2 ms p50 against MiniLM's 8.38 ms, measured in one interleaved process.
- Training prior resampled toward the evaluation prior (+1.27). Geometric interpolation, CONFIDENTIAL 14.4% -> 9.4%, INTERNAL 35.1% -> 43.7%.
WHY THE THIRD CHANGE IS THE ONE TO TRUST. A matched-containment control overturned five separate interventions in this project -- logit adjustment, span-max, a three-model ensemble, and two encoder swaps -- each of which looked like a win at argmax and turned out to be a different point on the same curve. Prior matching is the only intervention that control has CONFIRMED: it wins all nine gate/containment cells. At the NEVER_EGRESS gate and 95% containment it blocks 4.95% of legitimate traffic against the previous model's 8.19%.
IT IS NOT EVAL-FITTING. The obvious worry is that resampling toward an eval's class balance fits that eval's sampling design. Tested on the one gate in this project with two independent evals: the prior-matched model lost on BOTH (-0.68 matched, -0.21 independent) rather than gaining on the matched one, which is the signature eval-fitting would leave. It also gives a scope rule -- the gain scales with mismatch magnitude (2.50x -> +1.27, 1.57x -> +0.42, 1.26x -> -0.68) because resampling always discards rows.
THE TRADE, STATED. Real-traffic accuracy is 0.8662 median against 0.8838 for the non-resampled model: -1.76. This model is tuned toward enterprise-like text. If you serve consumer traffic, use llm-d-sc-sensitivity-v3-bge instead.
SCOPE LIMIT. entsec is a SYNTHETIC enterprise eval, 95.8% distinguishable from real assistant traffic. It exists because real consumer traffic is ~93% PUBLIC and cannot exercise the tiers that gate egress at all. There is no real enterprise corpus here to validate against; that is the largest open risk in this work and no further modelling reduces it.
The eval has a measured ceiling
Gold labels were audited by blind paired adjudication in two strata — the rows this model got wrong, and a sample of the rows it got right — with the judge shown two candidate labels in random order and no indication of provenance. Roughly 4.9% of the gold labels are themselves wrong, so a PERFECT classifier scored against this eval would reach about 0.95, not 1.0.
Read the real-traffic accuracy against that ceiling, not against 100%. Auditing only a model's mistakes would move the number up artificially; sampling the correct rows too is what makes the estimate honest, and it revealed that on ~3.3% of "correct" rows the model agreed with a bad label — meaning measured accuracy is very slightly overstated.
How the eval was built
Real-traffic rows come from WildChat-1M (ungated real assistant traffic). Each prompt was labelled independently by three models (claude-opus-5, claude-sonnet-5, claude-fable-5-1) from the task rubric alone -- no labeller ever saw a proposed label, so agreement is evidence rather than assent. Only unanimous rows are scored.
Those three agree unanimously on roughly 70-74% of real prompts. The remaining prompts are published as a contested split rather than discarded: they measure how much real traffic this taxonomy does not resolve, which no single accuracy figure can express.
Training data
744 rows from sensitivity-prior-sqrt (v2+real+enterprise+real-contested, resampled), mixing jury-labelled real traffic (register and class prior) with rubric-grounded synthetic data (coverage of tiers that are rare in real traffic). Training prior: None. Held-out eval prompts are excluded by content hash.
Latency
CPU single-request: p50 16.29 ms, p99 30.57 ms (Apple M-series, single thread). llm-d-sc serves the classifier on CPU, so model size trades directly against per-replica throughput.
Limitations
- WildChat is consumer traffic. For
sensitivityit is ~93%PUBLICand cannot measure the tiers that gate egress; the enterprise row above covers those. - Labels come from LLM jurors, not human annotators. The rubric was validated by reproducing the project's hand-authored gold labels (complexity 0.9875, cost 1.000, sensitivity 1.000) before use.
- Not independently reproduced.