corral-intervention-reports

# *Corral* – Intervention Ablation Reports <div align="center">  [](https://lamalab-org.github.io/corral/) [](https://lamalab-org.github.io/corral/docs/) [](https://github.com/lamalab-org/corral) [](https://opensource.org/licenses/MIT) [](https://arxiv.org/abs/2604.18805) [](https://huggingface.co/datasets/jablonkagroup/corral-intervention-reports) *Intervention ablation reports for multiple LLM agents across all 8 Corral environments* </div> --- ## 📋 Dataset Summary This dataset is part of the *Corral* collection accompanying the paper [*AI scientists produce results without reasoning scientifically*](https://arxiv.org/abs/2604.18805). It contains the **final evaluation reports** of the **intervention ablation** study conducted across multiple LLM agents and all **8 Corral environments**. The intervention ablation examines how well agents can leverage provided context. Each report captures the final outcomes of this ablation, including aggregate metrics such as **Pass@k** across the evaluated models and environments. Each configuration (config) of this dataset corresponds to a unique combination of *environment*, *scope* (difficulty level), *model*, and *granularity* (tasks or subtasks). The full set of configs spans the Cartesian product of the 8 environments, their respective scope levels, the evaluated models, and the tasks/subtasks split. ### 🎯 Supported Uses - 🤖 Studying policy transfer difficulty across different LLM agents and environments - 📊 Comparing Pass@k and other aggregate metrics across models under intervention conditions - 🔁 Reproducing and extending the intervention ablation analyses reported in the paper - 📐 Meta-evaluation of how well agents leverage externally supplied trace steps in scientific tasks --- ## 🧪 About *Corral* [*Corral*](https://lamalab-org.github.io/corral/) is a framework for the *science of agents and agents for science*. It provides a microservice architecture that **decouples agents from environments** via a client–server design (REST API), ensuring flexibility, reproducibility, and robust isolation. - 🌍 **Environments** define the task space, available tools, and observable feedback — from chemistry labs to HPC clusters. - 🤖 **Agents** are modular LLM-based entities supporting scaffolds such as ReAct, ToolCalling, LLMPlanner, and Reflection. - 📝 **Tasks** define problems to solve, complete with scoring functions. Tasks can be chained into TaskGroups for complex multi-stage challenges. *Corral* currently ships **8 environments**, **97 tools**, **115 tasks**, and **786 subtasks** spanning chemistry, physics, and materials science. ### 🌍 Environments | Environment | Description | 🔧 Tools | 📝 Tasks/scope | 🔭 Scopes | ⏱️ Avg. trace length | |---|---|:---:|:---:|:---:|:---:| | 🧫 **Inorganic Qualitative Analysis** | Identify unknown cations in solution through systematic wet-lab procedures (reagent addition, flame tests, pH measurement, centrifugation, etc.). Observations are computed from thermodynamic data. Three scopes progressively increase the number of candidate ions. | 14 | 10 | 3 | 39.4 | | ⚡ **Circuit Inference** | Recover the topology and component values of a hidden resistor network from pairwise resistance measurements. Tools provide series/parallel calculations, delta-wye transforms, and circuit validation. | 9 | 6 | 1 | 15.0 | | 🔭 **Spectroscopic Structure Elucidation** | Determine the molecular structure of an unknown compound by requesting and interpreting spectroscopic data (MS, NMR, HSQC, IR) alongside reference databases for chemical shifts and isotope distributions. | 16 | 20 | 2 | 15.1 | | 🧬 **Retrosynthetic Planning** | Design multi-step synthetic routes to target molecules under cost, step-count, and commercial-availability constraints, using a template catalogue and functional-group detection tools. | 15 | 8 | 3 | 25.5 | | 🤖 **ML-based Property Prediction** | Assemble a complete ML pipeline to predict formation energies of material polymorphs using data from the Materials Project, covering feature engineering, XGBoost training, and cross-validation. | 14 | 3 | 1 | 16.6 | | 🔬 **AFM Experiment Execution** | Analyze and interpret atomic force microscopy data for nanoscale surface characterization, including topographical and mechanical property measurements. | 6 | 1 | 4 | 26.3 | | ⚛️ **Molecular Simulation** | Design and execute molecular dynamics simulations with LAMMPS to predict materials properties, covering the full workflow from crystal structure retrieval to force-field queries and log analysis. | 8 | 2–3 | 2 | 30.4 | | 🏗️ **Adsorption Surface Construction** | Build adsorbate–slab configurations from bulk crystal structures for heterogeneous catalysis studies, integrating Materials Project retrieval, slab generation, and adsorption-site enumeration. | 15 | 3 | 1 | 19.6 | --- ## 🗂️ Dataset Structure ### Configs The dataset exposes one config per intervention environment: `catalyst`, `md`, `ml`, `resistor`, `retrosynthesis`, `spectra`, and `wetlab`. ### Data Splits All configs expose a single `train` split. ### Data Instances Each row corresponds to one **intervention ablation report** for a specific model, environment, scope level, and task (or subtask). Reports include final evaluation outcomes such as Pass@k and other aggregate metrics recorded after the agent attempted to continue execution from steps sampled from a different agent's trace. --- ## 🏗️ Dataset Creation ### Curation Rationale This dataset was created as part of *Corral* to quantify how difficult it is for different LLM agents to leverage an context, concretely, by injecting steps from another agent's trace at the start of a new episode and measuring whether the agent can successfully continue. This ablation isolates the contribution of policy structure and environment complexity to agent performance. ### Source Data Reports are derived from intervention ablation evaluation runs on *Corral* benchmark tasks, spanning multiple evaluated models across all 8 environments and their scope levels. Each run was seeded with steps from a separately generated trace before the agent continued autonomously. --- ## 🔗 Relation to Other Corral Artifacts This dataset is one component of the broader *Corral* release and is best interpreted together with the matching task definitions, execution traces, logprobs, aggregate results, and reasoning annotations available in the [*Corral* collection](https://huggingface.co/collections/jablonkagroup/corral). --- ## 📄 Citation ```bibtex @article{ríos-garcía2026ai, title = {AI scientists produce results without reasoning scientifically}, author = {Martiño Ríos-García and Nawaf Alampara and Chandan Gupta and Indrajeet Mandal and Sajid Mannan and Ali Asghar Aghajani and N. M. Anoop Krishnan and Kevin Maik Jablonka}, year = {2026}, journal = {arXiv preprint arXiv: 2604.18805} } ``` ## 📜 License This dataset is released under the [MIT License](https://opensource.org/licenses/MIT). ## Changelog ### 2026-04-22 - Initial release of the dataset card.