NagaYu/isotope-calibration-qwen2.5-1.5b-instruct

Isotope calibration — Qwen2.5-1.5B-Instruct

This repository contains no trained weights. It is a measured calibration profile for `Qwen/Qwen2.5-1.5B-Instruct`: the per-layer constants that Isotope's information-flow propagation depends on, plus the tool policy calibrated against them. Nothing here changes the base model's behaviour. It lives in the model namespace because it is checkpoint-specific data that has to travel with the checkpoint it describes.

• —💻 Code: https://github.com/NagaYu/isotope
• —🤗 Benchmark: https://huggingface.co/datasets/NagaYu/isotope-bench
• —🤗 Demo: https://huggingface.co/spaces/NagaYu/isotope

Why a checkpoint needs calibrating at all

Isotope propagates a provenance distribution through the attention graph:

U[l+1][t] = (1 − α[l][t])·U[l][t] + α[l][t]·Σⱼ W[l][t,j]·U[l][j]

Two of the inputs are properties of the specific model, and getting them wrong produces a number that looks fine and means nothing.

α — the attention branch's share of the residual stream. Textbook attention-rollout hard-codes 0.5. Measured here:

Mixing 2.3× too hard at every layer drains provenance geometrically. On this model, running the textbook configuration reports 0.000 influence for injections that plainly succeeded.

The layer-0 attention sink. It absorbs a large share of every attention row, and it is labelled trusted — so the drained mass lands there and vanishes:

That last ratio is the load-bearing one. Weighting provenance transport by ‖v‖ rather than by attention probability neutralises the sink because its value vectors are small — which is the same reason it functions as a no-op in the network. If this ratio came back near 1.0 on some checkpoint, value weighting would not work there and the readout would need re-validating before being used to gate anything.

Files

Calibrated policy

Thresholds are on the contested share — untrusted / (untrusted + user_instruction + ε) — so 0.5 reads as "untrusted data contributed more to this decision than the user did". Selected on a 7-task dev split only:

Content arguments (body, answer) are deliberately ungated: a body quoted from a retrieved document measured 0.55–0.60 untrusted on benign dev tasks. That is exactly the influence a block-granularity monitor cannot distinguish from an attack, and leaving it ungated is where the utility advantage comes from.

Use

from isotope import IsotopeGuard, DocumentSpec

guard = IsotopeGuard(model_name="Qwen/Qwen2.5-1.5B-Instruct", device="cpu").load()
result = guard.run(
    instruction="Look up the Q3 invoice total and email it to alice@corp.example.",
    documents=[DocumentSpec("invoices", untrusted_page_text)],
)
for a in result.actions:
    print(a.tool, a.verdict, round(a.u_authorize, 3), a.top_source)

The defaults in isotope already match isotope_policy.json, so this profile is primarily an artefact for auditing — it records what was measured, when, and on which checkpoint.

To calibrate a different model:

python benchmarks/calibrate.py --model <checkpoint> --device cpu

Requirements this profile implies

• —attn_implementation="eager". FlashAttention and SDPA never materialise the attention matrix, so output_attentions=True cannot work with them.
• —Hookable self_attn.v_proj, self_attn.o_proj and input_layernorm on each decoder layer.

Measured behaviour on the benchmark

26 tasks, five conditions, all measured on this checkpoint (dataset):

Separation at the decision token: attack 0.646 vs clean benign 0.069 — a 9.4× gap at AUC 1.00. Deployed tracker overhead: 8.4% of model wall-clock.

Limitations

• —Measured on one prompt shape (system prompt + user turn + retrieved document + tool-call scaffold) and one checkpoint. α varies with context composition; treat these as representative, not exact.
• —The thresholds were calibrated on 7 dev tasks. Re-validate on your own traffic before trusting them.
• —Isotope does not eliminate prompt injection. It bounds and audits untrusted influence over authorization decisions under a stated propagation model. See SECURITY.md.

Citation

@software{isotope2026,
  title  = {Isotope: quantitative information-flow tracking inside the forward pass},
  author = {NagaYu},
  year   = {2026},
  url    = {https://github.com/NagaYu/isotope},
  license = {MIT}
}