Evaluation Dataset for ChemGraph: An Agentic Framework for Computational Chemistry Workflows

This dataset provides scripts, reference data, and evaluation tools for benchmarking ChemGraph, an LLM-based molecular simulation framework. It includes outputs generated by four different language models: GPT-4o-mini, Claude-3.5-haiku, Qwen2.5-14B, and GPT-4o. While some outputs were generated using earlier commits, all results are compatible with ChemGraph version e1a94f855c5bec49ed9345c7713ab6080445d49e. Main Files and Descriptions - data_from_pubchempy.json: Structured chemical information obtained from PubChemPy. Serves as an input dataset for each experiment - manual_workflow.json: A manually constructed reference workflow representing true tool call sequences and outputs. Used for benchmarking LLM results. - llm_workflow_[...].json: A JSON file containing tool-use outputs generated by different LLMs. Includes additional metadata such as model name, timestamps and system prompt. ** Update history: August 29th, 2025: Expanded benchmark from 260 to 360. Reran all evaluations. Added GPT-4o multi-agent evaluation.本数据集提供了用于基准测试ChemGraph（一款基于大语言模型（Large Language Model）的分子模拟框架）的脚本、参考数据与评估工具。其中包含四款大语言模型生成的输出结果：GPT-4o-mini、Claude-3.5-haiku、Qwen2.5-14B及GPT-4o。尽管部分输出基于早期代码提交版本生成，但所有结果均兼容ChemGraph版本e1a94f855c5bec49ed9345c7713ab6080445d49e。 主要文件及说明 - data_from_pubchempy.json：通过PubChemPy获取的结构化化学信息，作为各项实验的输入数据集。 - manual_workflow.json：手动构建的参考工作流，代表真实的工具调用序列与输出结果，用于基准测试大语言模型的生成结果。 - llm_workflow_[...].json：包含不同大语言模型生成的工具调用输出的JSON文件，附带模型名称、时间戳与系统提示词等额外元数据。 ** 更新日志 2025年8月29日： 将基准测试规模从260拓展至360；重新运行全部评估流程；新增GPT-4o多智能体评估。