jablonkagroup/corral-oss-trace-logprobs

Corral – OSS-120B Trace Logprobs

<div align="center">

[image]

     

Token-level log-probabilities for GPT-Oss-120B evaluation runs across all 8 Corral environments

</div>

📋 Dataset Summary

This dataset is part of the Corral collection accompanying the paper *AI scientists produce results without reasoning scientifically*. It contains the token-level log-probabilities recorded during the evaluation runs of GPT-Oss-120B across all 8 Corral environments.

Each configuration (config) of this dataset corresponds to a unique combination of environment, scope (difficulty level), and granularity (tasks or subtasks). For example, a config such as afm_level_1_subtasks contains the logprob records for the AFM Experiment Execution environment at scope level 1, broken down at the subtask granularity. The full set of configs spans the Cartesian product of the 8 environments, their respective scope levels, and the tasks/subtasks split.

This resource is designed for process-level analysis, interpretability research, and auditing of scientific-agent reasoning — not for general-purpose model pre-training.

🎯 Supported Uses

• —🔍 Auditing token-level confidence and uncertainty in scientific-agent completions
• —📊 Studying the relationship between model certainty and task success
• —🔁 Reproducing and extending the log-probability analyses reported in the paper
• —📐 Meta-evaluation and calibration studies of frontier LLMs on scientific tasks

🧪 About Corral

*Corral* is a framework for the science of agents and agents for science. It provides a microservice architecture that decouples agents from environments via a client–server design (REST API), ensuring flexibility, reproducibility, and robust isolation.

• —🌍 Environments define the task space, available tools, and observable feedback — from chemistry labs to HPC clusters.
• —🤖 Agents are modular LLM-based entities supporting scaffolds such as ReAct, ToolCalling, LLMPlanner, and Reflection.
• —📝 Tasks define problems to solve, complete with scoring functions. Tasks can be chained into TaskGroups for complex multi-stage challenges.

Corral currently ships 8 environments, 97 tools, 115 tasks, and 786 subtasks spanning chemistry, physics, and materials science.

🌍 Environments

🗂️ Dataset Structure

Configs

Each config name encodes {environment}_{scope}_{granularity}, where:

• —environment is a short identifier for one of the 8 Corral environments (e.g., afm, circuit_inference, spectroscopic, retrosynthesis, ml_property, molecular_simulation, adsorption).
• —scope is the difficulty level (e.g., level_1, level_2, level_3).
• —granularity is either tasks or subtasks.

Data Splits

All configs expose a single train split.

Data Instances

Each row corresponds to one token-level log-probability record from a GPT-Oss-120B completion produced during an agent evaluation run.

🏗️ Dataset Creation

Curation Rationale

This dataset was created as part of Corral to enable process-level analysis of LLM-based scientific agents, specifically to study how token-level confidence relates to scientific reasoning quality and task outcomes.

Source Data

Records are derived from agent evaluation runs on Corral benchmark tasks, capturing the log-probabilities returned by GPT-Oss-120B for each generated token across all 8 environments and their scope levels.

🔗 Relation to Other Corral Artifacts

This dataset is one component of the broader Corral release and is best interpreted together with the matching task definitions, execution traces, reports, aggregate results, and reasoning annotations available in the *Corral* collection.

📄 Citation

@article{ríos-garcía2026ai,
  title   = {AI scientists produce results without reasoning scientifically},
  author  = {Martiño Ríos-García and Nawaf Alampara and Chandan Gupta and Indrajeet Mandal and Sajid Mannan and Ali Asghar Aghajani and N. M. Anoop Krishnan and Kevin Maik Jablonka},
  year    = {2026},
  journal = {arXiv preprint arXiv: 2604.18805}
}

📜 License

This dataset is released under the MIT License.

Changelog

2026-04-22

• —Initial release of the dataset card.