jub-aer/ConsistencyBench-interpretability

ConsistencyBench-Interpretability Extension

White-box mechanistic analysis of logical inconsistency using Qwen/Qwen2.5-1.5B-Instruct (local, full activation access) as a dedicated interpretability testbed, distinct from the 17-model black-box leaderboard.

Contents

• —layer_probe_results.csv - per-layer logistic-regression probe accuracy for decoding "will this response be inconsistent?" directly from residual-stream activations
• —activation_patching.csv - literal patching results: copying a consistent run's activation into an inconsistent run and checking whether the output flips
• —steering_dose_response.csv - IR at increasing diff-in-means steering strength (a causal dose-response curve, not just a correlational probe)
• —hint_sensitivity.csv - flip rate and hint-induced inconsistency under misleading epistemic pressure (across the 6-model API subset used for this analysis)
• —metadata.json - best probe layer, probe accuracy, model config

Method Summary

1. 1.Train a linear probe at every layer to decode inconsistency from the residual stream at the final prompt token (Alain & Bengio, 2017 methodology)
2. 2.Take the best layer's probe direction (and the diff-in-means direction) as the "shortcut direction", in place of a pretrained SAE, which does not exist for this checkpoint
3. 3.Run two causal tests: literal activation patching between matched consistent/ inconsistent pairs, and a steering dose-response sweep with a specificity check on unrelated control questions