jablonkagroup/corral-QAs-reports

Corral – QA Reports

<div align="center">

[image]

     

Model completions for question-answer evaluations probing factual knowledge and reasoning across Corral environments

</div>

📋 Dataset Summary

This dataset is part of the Corral collection accompanying the paper *AI scientists produce results without reasoning scientifically*. It contains the model completions and reports for the question-answer evaluations used to test the factual knowledge and reasoning ability of models across Corral environments.

The dataset is organized into 50 configurations, with one configuration for each available combination of environment, model, and evaluation dimension (knowledge or reasoning). For example, a config encodes the reports generated by one model on either the knowledge-focused or reasoning-focused QA set for a given Corral environment.

These completions correspond to the QA items used in the Item Response Theory (IRT) analyses reported in the Corral study, where the underlying knowledge and reasoning QAs serve as indicators for the latent knowledge and reasoning factors. This resource is intended for evaluation, psychometric modeling, and analysis of scientific-agent capabilities rather than for general-purpose model pre-training.

🎯 Supported Uses

• —🧠 Analyzing model completions on factual-knowledge and scientific-reasoning QAs across Corral environments
• —📊 Reproducing and extending the IRT analyses reported in the paper
• —📐 Studying latent knowledge and reasoning factors through model response behavior
• —🔁 Building meta-evaluation datasets for model comparison, reporting, and capability analysis

🧪 About Corral

*Corral* is a framework for the science of agents and agents for science. It provides a microservice architecture that decouples agents from environments via a client–server design (REST API), ensuring flexibility, reproducibility, and robust isolation.

• —🌍 Environments define the task space, available tools, and observable feedback — from chemistry labs to HPC clusters.
• —🤖 Agents are modular LLM-based entities supporting scaffolds such as ReAct, ToolCalling, LLMPlanner, and Reflection.
• —📝 Tasks define problems to solve, complete with scoring functions. Tasks can be chained into TaskGroups for complex multi-stage challenges.

Corral currently ships 8 environments, 97 tools, 115 tasks, and 786 subtasks spanning chemistry, physics, and materials science.

🌍 Environments

🗂️ Dataset Structure

Configs

Each config name encodes {environment}_{model}_{dimension}, where:

• —environment is a short identifier for one of the 8 Corral environments (e.g., afm, circuit_inference, spectroscopic, retrosynthesis, ml_property, molecular_simulation, adsorption).
• —model identifies the model whose completions are included in that configuration.
• —dimension is either knowledge or reasoning.

This yields 50 total configs, one for each available environment-model pair and knowledge/reasoning combination.

Data Splits

All configs expose a single train split.

Data Instances

Each row corresponds to one model completion/report for a specific question-answer item, associated with a particular Corral environment, model, and evaluation dimension: knowledge or reasoning.

🏗️ Dataset Creation

Curation Rationale

This dataset was created as part of Corral to measure scientific-agent capabilities beyond end-task success by collecting model outputs on targeted QA items that separate factual knowledge from reasoning ability and support IRT-based analysis.

Source Data

The underlying QAs were derived from the task content, domain knowledge, and reasoning demands of the Corral benchmark environments. The dataset released here contains the corresponding model completions/reports on those QAs, organized by environment, model, and evaluation dimension, and used in the analyses of latent knowledge and reasoning factors.

🔗 Relation to Other Corral Artifacts

This dataset is one component of the broader Corral release and is best interpreted together with the matching task definitions, execution traces, reports, aggregate results, and reasoning annotations available in the *Corral* collection.

📄 Citation

@article{ríos-garcía2026ai,
  title   = {AI scientists produce results without reasoning scientifically},
  author  = {Martiño Ríos-García and Nawaf Alampara and Chandan Gupta and Indrajeet Mandal and Sajid Mannan and Ali Asghar Aghajani and N. M. Anoop Krishnan and Kevin Maik Jablonka},
  year    = {2026},
  journal = {arXiv preprint arXiv: 2604.18805}
}

📜 License

This dataset is released under the MIT License.

Changelog

2026-04-22

• —Initial release of the dataset card.