ceselder/cot-oracle-v4-checkpoints

CoT Oracle v4 (Qwen3-8B LoRA)

A chain-of-thought activation oracle: a LoRA fine-tune of Qwen3-8B that reads the model's own internal activations at sentence boundaries during chain-of-thought reasoning and answers natural-language questions about what was computed.

This is a continuation of the Activation Oracles line of work (Karvonen et al., 2024), extended to operate over structured CoT trajectories rather than single-position activations.

Model Description

An activation oracle is a language model fine-tuned to accept its own internal activations as additional input and answer questions about them. The oracle is the same model as the source -- Qwen3-8B reads Qwen3-8B's activations.

CoT Oracle v4 specializes in reading activations extracted at sentence boundary positions during chain-of-thought reasoning. Given activations from 3 layers (25%, 50%, 75% depth) at each sentence boundary, the oracle can:

• —Classify the reasoning domain (math, science, logic, commonsense, reading comprehension, multi-domain, medical)
• —Predict whether the CoT reached the correct answer
• —Detect decorative reasoning (steps that don't contribute to the answer)
• —Predict surrounding token context from arbitrary positions

Key Properties

• —The oracle reads activations, not text. It has no access to the CoT tokens themselves.
• —Activations are collected with LoRA disabled (pure base model representations).
• —Activations are injected via norm-matched addition at layer 1, preserving the scale of the residual stream.
• —The oracle generates with LoRA enabled (the trained adapter interprets the injected activations).

Training

Base Checkpoint

Training continues from `adamkarvonen/checkpoints_latentqa_cls_past_lens_addition_Qwen3-8B`, an activation oracle pretrained on ~1M examples of context prediction, classification, and past-lens tasks.

LoRA Configuration

Training Tasks

Six tasks mixed together and shuffled into a single training run:

Total: ~221K training examples, 1 epoch.

Sentence-structured tasks (2-6) extract activations at 3 layers per boundary position:

• —Layer 9 (25% depth)
• —Layer 18 (50% depth)
• —Layer 27 (75% depth)

Each sentence-structured example is duplicated: once with all 3 layers interleaved, once with only L50%. This teaches the oracle to work with both multi-layer and single-layer inputs.

Training Details

Activation Injection

Activations are injected at layer 1 via norm-matched addition:

h' = h + ||h|| * (v / ||v||)

where h is the original hidden state and v is the collected activation vector. This preserves the norm of the residual stream while adding directional information from the source activations.

The placeholder token is " ?" (token ID 937). For multi-layer inputs, per-layer placeholder tokens are used: " @" (L25%), " ?" (L50%), " #" (L75%), cycling in that order.

Corpus

The training corpus consists of CoT traces generated by Qwen3-8B across 12 reasoning benchmarks: MATH, GSM8K, GPQA, BBH, ARC, StrategyQA, DROP, LogiQA, MMLU-Pro, CommonsenseQA, AQUA-RAT, and MedQA. CoTs were generated via OpenRouter API.

Evaluation Results

Evaluated on held-out data using exact string match:

Key observations:

• —Domain classification reaches 98-100% accuracy -- the oracle reliably identifies the reasoning domain from activations alone.
• —Correctness prediction reaches 90% -- the oracle can tell whether the model's reasoning led to the right answer without seeing the answer.
• —Decorative detection is noisy (bounces between 50-71%) due to limited eval data (74 unique both-correct entries).
• —Context prediction stays low (7-11%) under exact string match but this is expected -- the pretrained AO checkpoint already handles this task and exact match is a harsh metric for free-text prediction.
• —Summary remains at 0% (labels were all identical in training data -- known issue).

Experiment tracking: wandb `cot_oracle` project, run `cot_oracle_v4_mixed`

Usage

Requirements

This model requires the activation_oracles library for the activation collection and injection infrastructure.

git clone https://github.com/adamkarvonen/activation_oracles
cd activation_oracles && pip install -e .

Loading the Model

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

# Load base model
model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3-8B",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-8B")

# Load oracle adapter
model = PeftModel.from_pretrained(model, "ceselder/cot-oracle-v4-8b")

Collecting Activations

Activations must be collected from the base model (LoRA disabled) at the target layers:

import torch

# Layers at 25%, 50%, 75% depth of Qwen3-8B (36 layers)
LAYERS = [9, 18, 27]

# 1. Prepare input: question + CoT response
messages = [{"role": "user", "content": question}]
prompt = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True,
    enable_thinking=True,
)
full_text = prompt + cot_response

# 2. Find sentence boundary positions in token space
input_ids = tokenizer(full_text, return_tensors="pt")["input_ids"]
# boundary_positions = [...] (token indices at sentence boundaries)

# 3. Collect activations with LoRA DISABLED
with model.disable_adapter():
    activations = {}  # {layer: {position: tensor}}
    # Use hooks on model.model.layers[layer] to capture hidden states
    # at each boundary position for each layer

Running the Oracle

# 4. Build oracle prompt with placeholder tokens
# For multi-layer: interleave " @", " ?", " #" per boundary
n_boundaries = len(boundary_positions)
placeholders = " @ ? #" * n_boundaries  # 3 tokens per boundary

layer_str = ", ".join(str(l) for l in LAYERS)
oracle_prompt = f"Layer: {layer_str}\n{placeholders.strip()} \n"
oracle_prompt += "What domain of reasoning is this? Answer with one word: math, science, logic, commonsense, reading, multi_domain, or medical."

# 5. Format as chat and tokenize
messages = [{"role": "user", "content": oracle_prompt}]
formatted = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True,
    enable_thinking=False,
)

# 6. Inject activations via norm-matched addition at layer 1
# At each placeholder position, add the corresponding activation:
# positions cycle through [L25_s1, L50_s1, L75_s1, L25_s2, L50_s2, L75_s2, ...]
# Injection: h' = h + ||h|| * (v / ||v||)

# 7. Generate with LoRA ENABLED (default state)
output = model.generate(input_ids, max_new_tokens=64)

For complete working code, see the cot-oracle repository, particularly src/signs_of_life/ao_lib.py for the injection mechanism and src/train_mixed.py for the full training pipeline.

Intended Use

This model is a research artifact for studying chain-of-thought interpretability. Intended uses include:

• —Investigating what information is encoded in CoT activations at different stages of reasoning
• —Detecting unfaithful chain-of-thought (reasoning that doesn't match the model's actual computation)
• —Building tools for mechanistic understanding of language model reasoning

Limitations

• —Same-model only: The oracle can only read activations from Qwen3-8B. It will not work with other models.
• —Exact match eval is harsh: Tasks like context prediction and summary show low scores under exact string match, but the model often produces semantically reasonable outputs.
• —Decorative detection is undertrained: Only ~500 unique training examples; results are noisy.
• —Summary task is broken: All 200 training labels were identical, so the model learned nothing useful for this task.
• —No uncertainty calibration: The oracle is confidently wrong sometimes, consistent with findings from Karvonen et al., 2024.

Citation

@misc{cot-oracle-v4,
  title={CoT Oracle: Detecting Unfaithful Chain-of-Thought via Activation Trajectories},
  author={Celeste Deschamps-Helaere},
  year={2026},
  url={https://github.com/ceselder/cot-oracle}
}

Related Work

@article{karvonen2024activation,
  title={Activation Oracles},
  author={Karvonen, Adam and others},
  journal={arXiv preprint arXiv:2512.15674},
  year={2024}
}

@article{bogdan2025thought,
  title={Thought Anchors: Causal Importance of CoT Sentences},
  author={Bogdan, Paul and others},
  journal={arXiv preprint arXiv:2506.19143},
  year={2025}
}

@article{macar2025thought,
  title={Thought Branches: Studying CoT through Trajectory Distribution},
  author={Macar, Uzay and Bogdan, Paul and others},
  journal={arXiv preprint arXiv:2510.27484},
  year={2025}
}

Links

• —Code: github.com/ceselder/cot-oracle
• —Training data: huggingface.co/datasets/ceselder/cot-oracle-data
• —Base AO checkpoint: adamkarvonen/checkpoints_latentqa_cls_past_lens_addition_Qwen3-8B
• —Activation Oracles repo: github.com/adamkarvonen/activation_oracles
• —Experiment tracking: wandb cot_oracle project, run cot_oracle_v4_mixed