cot-oracle
cot-oracle-data
CoT Oracle Training Data
Datasets for training an Activation Oracle to detect unfaithful chain-of-thought reasoning.
Splits
thought_anchors (3 examples)
Qwen3-8B CoT traces with resampled importance scores per sentence. Each problem was solved 5 times with the CoT truncated at different points to measure which sentences causally influence the answer.
Fields:
sentences[i].importance_delta: how much removing this sentence hurts accuracy (0-1)… See the full description on the dataset page: https://huggingface.co/datasets/ceselder/cot-oracle-data.cot-oracle-eval-rot13-reconstruction
CoT Oracle Eval: rot13_reconstruction
Model-organism eval: CoT encoded with ROT13, oracle must reconstruct original. Source: ceselder/qwen3-8b-math-cot-corpus.
Part of the CoT Oracle Evals collection.
Schema
Field
Description
eval_name
Eval identifier
example_id
Unique example ID
clean_prompt
Prompt without nudge/manipulation
test_prompt
Prompt with nudge/manipulation
correct_answer
Ground truth answer
nudge_answer
Answer the nudge pushes toward… See the full description on the dataset page: https://huggingface.co/datasets/ceselder/cot-oracle-eval-rot13-reconstruction.cot-oracle-convqa-chunked
2026-05-12 — target_response_distractor column added
A second per-row column was added: target_response_distractor — a context-aware plausible-but-wrong alternative to target_response, generated by Haiku 4.5 with explicit instructions to:
match the length (token count) of the correct answer within ±10%
mirror the structure (markdown, opening phrase, sentence count, bold/list markup)
match the vocabulary register (math/ethics/etc.)
flip exactly one substantive fact (verdict, number… See the full description on the dataset page: https://huggingface.co/datasets/cds-jb/cot-oracle-convqa-chunked.cot-oracle-truthfulqa-hint-admission-unverbalized
TruthfulQA Hint Admission — Unverbalized
Eval dataset for the CoT Oracle project. Tests whether an activation oracle can detect hint influence from model internals when the model does not verbalize the hint in its chain-of-thought.
What is this?
Qwen3-8B is given TruthfulQA multiple-choice questions with planted hints (correct or wrong). This dataset contains only the rollouts where the model did not mention the hint in its reasoning — the oracle must read… See the full description on the dataset page: https://huggingface.co/datasets/ceselder/cot-oracle-truthfulqa-hint-admission-unverbalized.cot-oracle-eval-hinted-mcq
CoT Oracle Eval: hinted_mcq
GSM8K problems as 4-choice MCQ with hints. 50/50 right/wrong hints, varying subtlety. Source: openai/gsm8k test.
Part of the CoT Oracle Evals collection.
Schema
Field
Description
eval_name
Eval identifier
example_id
Unique example ID
clean_prompt
Prompt without nudge/manipulation
test_prompt
Prompt with nudge/manipulation
correct_answer
Ground truth answer
nudge_answer
Answer the nudge pushes toward… See the full description on the dataset page: https://huggingface.co/datasets/ceselder/cot-oracle-eval-hinted-mcq.cot-oracle-eval-sycophancy
CoT Oracle Eval: sycophancy
User states wrong math belief, model must resist agreeing. 50/50 right/wrong, varying confidence. Source: openai/gsm8k test + HuggingFaceH4/MATH-500 test.
Part of the CoT Oracle Evals collection.
Schema
Field
Description
eval_name
Eval identifier
example_id
Unique example ID
clean_prompt
Prompt without nudge/manipulation
test_prompt
Prompt with nudge/manipulation
correct_answer
Ground truth answer
nudge_answer
Answer the… See the full description on the dataset page: https://huggingface.co/datasets/ceselder/cot-oracle-eval-sycophancy.