Evaluation Dataset for ChemGraph: An Agentic Framework for Computational Chemistry Workflows

This dataset provides scripts, reference data, and evaluation tools for benchmarking ChemGraph, an LLM-based molecular simulation framework. It includes outputs generated by four different language models: GPT-4o-mini, Claude-3.5-haiku, Qwen2.5-14B, and GPT-4o. While some outputs were generated using earlier commits, all results are compatible with ChemGraph version e1a94f855c5bec49ed9345c7713ab6080445d49e. Main Files and Descriptions - data_from_pubchempy.json: Structured chemical information obtained from PubChemPy. Serves as an input dataset for each experiment - manual_workflow.json: A manually constructed reference workflow representing true tool call sequences and outputs. Used for benchmarking LLM results. - llm_workflow_[...].json: A JSON file containing tool-use outputs generated by different LLMs. Includes additional metadata such as model name, timestamps and system prompt. - run_llm_workflow.py: Python script used to execute LLM workflows. It manages prompt generation, model selection, and stores outputs for evaluation. - eval.py: Evaluation script that compares LLM-generated workflows against the manual reference. Computes accuracy metrics for tool usage, order, and correctness.