datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
cot-oracle-data
CoT Oracle Training Data
Datasets for training an Activation Oracle to detect unfaithful chain-of-thought reasoning.
Splits
thought_anchors (3 examples)
Qwen3-8B CoT traces with resampled importance scores per sentence. Each problem was solved 5 times with the CoT truncated at different points to measure which sentences causally influence the answer.
Fields:
sentences[i].importance_delta: how much removing this sentence hurts accuracy (0-1)… See the full description on the dataset page: https://huggingface.co/datasets/ceselder/cot-oracle-data.cot-oracle-eval-rot13-reconstruction
CoT Oracle Eval: rot13_reconstruction
Model-organism eval: CoT encoded with ROT13, oracle must reconstruct original. Source: ceselder/qwen3-8b-math-cot-corpus.
Part of the CoT Oracle Evals collection.
Schema
Field
Description
eval_name
Eval identifier
example_id
Unique example ID
clean_prompt
Prompt without nudge/manipulation
test_prompt
Prompt with nudge/manipulation
correct_answer
Ground truth answer
nudge_answer
Answer the nudge pushes toward… See the full description on the dataset page: https://huggingface.co/datasets/ceselder/cot-oracle-eval-rot13-reconstruction.cot-oracle-convqa-chunked
2026-05-12 — target_response_distractor column added
A second per-row column was added: target_response_distractor — a context-aware plausible-but-wrong alternative to target_response, generated by Haiku 4.5 with explicit instructions to:
match the length (token count) of the correct answer within ±10%
mirror the structure (markdown, opening phrase, sentence count, bold/list markup)
match the vocabulary register (math/ethics/etc.)
flip exactly one substantive fact (verdict, number… See the full description on the dataset page: https://huggingface.co/datasets/cds-jb/cot-oracle-convqa-chunked.cot-oracle-truthfulqa-hint-admission-unverbalized
TruthfulQA Hint Admission — Unverbalized
Eval dataset for the CoT Oracle project. Tests whether an activation oracle can detect hint influence from model internals when the model does not verbalize the hint in its chain-of-thought.
What is this?
Qwen3-8B is given TruthfulQA multiple-choice questions with planted hints (correct or wrong). This dataset contains only the rollouts where the model did not mention the hint in its reasoning — the oracle must read… See the full description on the dataset page: https://huggingface.co/datasets/ceselder/cot-oracle-truthfulqa-hint-admission-unverbalized.cot-oracle-eval-hinted-mcq
CoT Oracle Eval: hinted_mcq
GSM8K problems as 4-choice MCQ with hints. 50/50 right/wrong hints, varying subtlety. Source: openai/gsm8k test.
Part of the CoT Oracle Evals collection.
Schema
Field
Description
eval_name
Eval identifier
example_id
Unique example ID
clean_prompt
Prompt without nudge/manipulation
test_prompt
Prompt with nudge/manipulation
correct_answer
Ground truth answer
nudge_answer
Answer the nudge pushes toward… See the full description on the dataset page: https://huggingface.co/datasets/ceselder/cot-oracle-eval-hinted-mcq.cot-oracle-convqa-chunked-sonnetcot-oracle-eval-sycophancy
CoT Oracle Eval: sycophancy
User states wrong math belief, model must resist agreeing. 50/50 right/wrong, varying confidence. Source: openai/gsm8k test + HuggingFaceH4/MATH-500 test.
Part of the CoT Oracle Evals collection.
Schema
Field
Description
eval_name
Eval identifier
example_id
Unique example ID
clean_prompt
Prompt without nudge/manipulation
test_prompt
Prompt with nudge/manipulation
correct_answer
Ground truth answer
nudge_answer
Answer the… See the full description on the dataset page: https://huggingface.co/datasets/ceselder/cot-oracle-eval-sycophancy.cot-oracle-corpus-v5Chain of thought rollouts on varied data for my chain of thought monitor that I'm building for Neel MATS sprint. Everything here is from qwen 3 8b.
cot-oracle-training-v6cot-oracle-eval-reasoning-termination
CoT Oracle Eval: reasoning_termination
Reasoning termination prediction — given a CoT prefix, predict whether the model will emit within the next 100 tokens. Labels are resampled (50 continuations per prefix): will_terminate if >=45/50 end within 20-60 tokens, will_continue if >=45/50 continue beyond 200 tokens. Includes Wilson CIs on resample counts. 50/50 balanced. Source: AI-MO/aimo-validation-aime + AI-MO/aimo-validation-amc (no overlap with GSM8K/MATH training data).
Part of… See the full description on the dataset page: https://huggingface.co/datasets/ceselder/cot-oracle-eval-reasoning-termination.cot-oracle-correctness-qwen3-8b
cot-oracle-correctness-qwen3-8b
Qwen3-8B rollouts, regenerated/recovered 2026-06-18 for BBAO-14B; see source pipeline.
cot-oracle-hint-admission-qwen3-8b
cot-oracle-hint-admission-qwen3-8b
Qwen3-8B rollouts, regenerated/recovered 2026-06-18 for BBAO-14B; see source pipeline.
cot-oracle-convqa-chunked
cot-oracle-convqa-chunked
Mirror of cds-jb/cot-oracle-convqa-chunked for reproducibility (orig org fragile).
cot-oracle-eval-step-importance-thought-anchors
CoT Oracle Eval: step_importance_thought_anchors
Causal step importance identification from off-policy deepseek MATH rollouts. Source: uzaymacar/math-rollouts.
Part of the CoT Oracle Evals collection.
Schema
Field
Description
eval_name
"step_importance_thought_anchors"
example_id
Unique identifier
clean_prompt
Problem statement only
test_prompt
Problem + numbered CoT + final answer
correct_answer
Top-3 most important chunk utterances, newline-separated… See the full description on the dataset page: https://huggingface.co/datasets/japhba/cot-oracle-eval-step-importance-thought-anchors.cot-oracle-sycophancy-qwen3-8b
cot-oracle-sycophancy-qwen3-8b
Qwen3-8B rollouts, regenerated/recovered 2026-06-18 for BBAO-14B; see source pipeline.
cot-oracle-cotqa-ARCHIVED
CoT Oracle: Conversational QA
What this dataset is
LLM-generated conversational question-answer pairs about chain-of-thought reasoning traces. Each entry contains a natural-language question about a CoT trace (prompted with activation context like "Activations from N sentence boundaries") and an expected natural-language answer. This dataset is used for Stage 3 training of the CoT Trajectory Oracle — teaching the oracle to answer open-ended questions about… See the full description on the dataset page: https://huggingface.co/datasets/japhba/cot-oracle-cotqa-ARCHIVED.cot-oracle-eval-thought-anchors
CoT Oracle Eval: step_importance_thought_anchors
Causal step importance identification from off-policy DeepSeek MATH rollouts.
Importance metric: resampling KL divergence (resampling_importance_kl) — measures the KL divergence of the answer distribution when a chunk is removed and the continuation is resampled (~100 rollouts per chunk). This is the standard counterfactual importance metric from the Thought Anchors paper, NOT importance++.
Source: uzaymacar/math-rollouts (Thought… See the full description on the dataset page: https://huggingface.co/datasets/japhba/cot-oracle-eval-thought-anchors.cot-oracle-qwen3-8b-onpolicy-recipe
CoT Activation Oracle — On-Policy Qwen3-8B Training Recipe
A reproduction of the on-policy Qwen3-8B training mixture from
Building Better Activation Oracles
(Bauer, De Schamphelaere, Karvonen, Luick, Nanda).
This repository is a recipe card only — it documents the exact dataset
mixture, points at every source on the Hub, and gives regeneration instructions
for the pieces that are no longer available upstream. No third-party data is
re-hosted here; original datasets are linked… See the full description on the dataset page: https://huggingface.co/datasets/francescortu/cot-oracle-qwen3-8b-onpolicy-recipe.cot-oracle-convqa-chunked-gemini
2026-05-12 — target_response_distractor column added
A second per-row column was added: target_response_distractor — a context-aware plausible-but-wrong alternative to target_response, generated by Haiku 4.5 with explicit instructions to:
match the length (token count) of the correct answer within ±10%
mirror the structure (markdown, opening phrase, sentence count, bold/list markup)
match the vocabulary register (math/ethics/etc.)
flip exactly one substantive fact (verdict, number… See the full description on the dataset page: https://huggingface.co/datasets/cds-jb/cot-oracle-convqa-chunked-gemini.cot-oracle-answer-trajectorycot-oracle-atypical-answer
cot-oracle-atypical-answer
Mirror of japhba/cot-oracle-atypical-answer for reproducibility (orig org fragile).
cot-oracle-compqa
CoT Oracle Eval: CompQA
Computational questions about chain-of-thought reasoning traces. 10 query types
probing soundness, errors, non-sequiturs, load-bearing steps, self-correction, and more.
Answers generated by Qwen3-8B via OpenRouter. Source CoTs from
ceselder/qwen3-8b-math-cot-corpus.
Part of the CoT Oracle Evals collection.
Query Types
Type
Count
conclusion_follows
418
error_type
415
first_error
410
load_bearing
412
non_sequitur
415… See the full description on the dataset page: https://huggingface.co/datasets/japhba/cot-oracle-compqa.cot-oracle-eval-forced-answer-entropy
CoT Oracle Eval: forced_answer_entropy
Forced answer entropy regression eval. At each sentence boundary during CoT, forces the model to answer by appending 'So, the answer is:' after the partial reasoning. Extracts logprobs over answer tokens (A/B/C/D), computes softmax entropy H(p). Oracle must predict the entropy value (float) from activations. Metric: R-squared. Source: Idavidrein/gpqa (GPQA Diamond, graduate-level 4-choice science MCQ, NOT in training data).
Part of the CoT… See the full description on the dataset page: https://huggingface.co/datasets/ceselder/cot-oracle-eval-forced-answer-entropy.cot-oracle-eval-atypical-answer
CoT Oracle Eval: atypical_answer
Atypical answer prediction — predict from activations whether the model will give the majority or minority answer. Based on ICLR 2026 'When Just Read the Chain of Thought Fails' Section 2.5. 200 rollouts per question with temperature=0.6, filter to 10-35% minority rate, 15 majority + 15 minority per question CoT-length-matched. Moral dilemmas (Daily Dilemmas + Scruples) to avoid correctness confounder, plus PIQA OOD test. 50/50 balanced… See the full description on the dataset page: https://huggingface.co/datasets/ceselder/cot-oracle-eval-atypical-answer.cot-oracle-persona-driftcot-oracle-eval-cot-metacognition
CoT Oracle Eval: CoT Metacognition
This dataset evaluates whether an activation oracle can recover a model's local metacognitive state from a partial chain-of-thought prefix.
Each example asks a targeted question about a Qwen3-8B reasoning trace, such as "What is the model not sure about right now?", and provides a short target answer grounded in the withheld continuation.
The current uploaded snapshot contains 271 examples in a single test split.
What Each Row… See the full description on the dataset page: https://huggingface.co/datasets/cds-jb/cot-oracle-eval-cot-metacognition.cot-oracle-reasoning-termination
cot-oracle-reasoning-termination
Mirror of japhba/cot-oracle-reasoning-termination for reproducibility (orig org fragile).
cot-oracle-eval-sentence-insertion
CoT Oracle Eval: sentence_insertion
Needle-in-haystack: detect foreign sentence inserted into CoT. 50/50 insertion/clean split. Source: ceselder/qwen3-8b-math-cot-corpus.
Part of the CoT Oracle Evals collection.
Schema
Field
Description
eval_name
Eval identifier
example_id
Unique example ID
clean_prompt
Prompt without nudge/manipulation
test_prompt
Prompt with nudge/manipulation
correct_answer
Ground truth answer
nudge_answer
Answer the nudge pushes… See the full description on the dataset page: https://huggingface.co/datasets/ceselder/cot-oracle-eval-sentence-insertion.cot-oracle-eval-reasoning-termination-riya
CoT Oracle Eval: reasoning_termination_riya
Reasoning termination prediction — given a CoT prefix, predict whether the model will emit within the next 100 tokens. Labels are resampled (50 continuations per prefix): will_terminate if >=45/50 end within 20-60 tokens, will_continue if >=45/50 continue beyond 200 tokens. Includes Wilson CIs on resample counts. 50/50 balanced. Source: AI-MO/aimo-validation-aime + AI-MO/aimo-validation-amc (no overlap with GSM8K/MATH training data).
Part… See the full description on the dataset page: https://huggingface.co/datasets/ceselder/cot-oracle-eval-reasoning-termination-riya.cot-oracle-position-qa