SciSkillBench: CASCADE Benchmark for Evaluating LLM Agents on Scientific Tasks

This benchmark dataset SciSkillBench accompanies the paper "CASCADE: Cumulative Agentic Skill Creation through Autonomous Development and Evolution".
SciSkillBench is designed to evaluate large language model (LLM) agents on executing materials science and chemistry research tasks. It encompasses two primary types of tasks:
1. Data-oriented tasks: including data-retrieval, data-analysis, data-management, and data-processing problems.
2. Computation-oriented tasks: including simulation problems, and specialized models and toolkits problems.
Contents
- JSON files: Each JSON file contains benchmark questions and ground-truth answers. The structure includes the following keys:
user_query, sources, input_type, output_type, answer,
absolute_tolerance, unit, solution_code_or_process,
reference_link, official_documentation, notes
- Solution code: Reference implementations are provided in corresponding *_solution_code/ directories.
Access
This archive is password-protected to prevent data contamination. The password can be found in the README of the associated GitHub repository. Please DO NOT redistribute the unzipped data files or related data online.