jablonkagroup/corral-QAs

Corral – QAs

<div align="center">

[image]

     

Question-answer pairs probing factual knowledge and reasoning across all 8 Corral environments

</div>

📋 Dataset Summary

This dataset is part of the Corral collection accompanying the paper *AI scientists produce results without reasoning scientifically*. It contains the question-answer pairs (QAs) used to test the factual knowledge and reasoning ability of models across all 8 Corral environments.

The dataset is organized into 16 configurations, corresponding to the Cartesian product of the 8 environments and the two evaluation dimensions: knowledge and reasoning. For example, a config such as afm_knowledge contains QAs targeting factual knowledge in the AFM Experiment Execution environment, whereas afm_reasoning contains QAs designed to test reasoning over the same domain.

These QAs are the items used in the Item Response Theory (IRT) analyses reported in the Corral study, where they serve as indicators for the latent knowledge and reasoning factors. This resource is intended for evaluation, psychometric modeling, and analysis of scientific-agent capabilities rather than for general-purpose model pre-training.

🎯 Supported Uses

• —🧠 Evaluating factual knowledge and scientific reasoning across Corral environments
• —📊 Reproducing and extending the IRT analyses reported in the paper
• —📐 Studying latent knowledge and reasoning factors in scientific-agent benchmarks
• —🔁 Building meta-evaluation datasets for model comparison and capability analysis

🧪 About Corral

*Corral* is a framework for the science of agents and agents for science. It provides a microservice architecture that decouples agents from environments via a client–server design (REST API), ensuring flexibility, reproducibility, and robust isolation.

• —🌍 Environments define the task space, available tools, and observable feedback — from chemistry labs to HPC clusters.
• —🤖 Agents are modular LLM-based entities supporting scaffolds such as ReAct, ToolCalling, LLMPlanner, and Reflection.
• —📝 Tasks define problems to solve, complete with scoring functions. Tasks can be chained into TaskGroups for complex multi-stage challenges.

Corral currently ships 8 environments, 97 tools, 115 tasks, and 786 subtasks spanning chemistry, physics, and materials science.

🌍 Environments

🗂️ Dataset Structure

Configs

Each config name encodes {environment}_{dimension}, where:

• —environment is a short identifier for one of the 8 Corral environments (e.g., afm, circuit_inference, spectroscopic, retrosynthesis, ml_property, molecular_simulation, adsorption).
• —dimension is either knowledge or reasoning.

This yields 16 total configs, one for each environment and knowledge/reasoning combination.

Data Splits

All configs expose a single train split.

Data Instances

Each row corresponds to one question-answer item associated with a specific Corral environment and one of the two evaluation dimensions: knowledge or reasoning.

🏗️ Dataset Creation

Curation Rationale

This dataset was created as part of Corral to measure scientific-agent capabilities beyond end-task success, separating factual knowledge from reasoning ability through targeted QA items that can be modeled with IRT.

Source Data

The QAs were derived from the task content, domain knowledge, and reasoning demands of the Corral benchmark environments. They were constructed to probe environment-specific factual understanding and multi-step reasoning, and were then used in IRT modeling as indicators of the latent knowledge and reasoning factors.

🔗 Relation to Other Corral Artifacts

This dataset is one component of the broader Corral release and is best interpreted together with the matching task definitions, execution traces, reports, aggregate results, and reasoning annotations available in the *Corral* collection.

📄 Citation

@article{ríos-garcía2026ai,
  title   = {AI scientists produce results without reasoning scientifically},
  author  = {Martiño Ríos-García and Nawaf Alampara and Chandan Gupta and Indrajeet Mandal and Sajid Mannan and Ali Asghar Aghajani and N. M. Anoop Krishnan and Kevin Maik Jablonka},
  year    = {2026},
  journal = {arXiv preprint arXiv: 2604.18805}
}

📜 License

This dataset is released under the MIT License.

Changelog

2026-04-22

• —Initial release of the dataset card.