m&m’s

m&m’s数据集是由华盛顿大学和艾伦人工智能研究所合作创建，专注于多步骤多模态任务的评估。该数据集包含4427个任务，涉及文本、图像和音频三种模态，以及1-3个工具的组合，包括多模态机器学习模型、公共API和图像处理模块。数据集的创建过程包括工具图的构建、输入实例的采样、用户查询的生成、规则基础的计划生成以及人工验证。m&m’s数据集主要用于研究大型语言模型在多步骤多模态任务中的规划能力，旨在解决如何有效利用工具来完成复杂任务的问题。

The M&M’s Dataset was co-created by the University of Washington and the Allen Institute for AI, focusing on the evaluation of multi-step multimodal tasks. This dataset contains 4,427 tasks covering three modalities: text, image, and audio, as well as combinations of 1 to 3 tools including multimodal machine learning models, public APIs, and image processing modules. The dataset’s creation process includes the construction of tool graphs, sampling of input instances, generation of user queries, rule-based plan generation, and manual verification. The M&M’s Dataset is primarily used to study the planning capabilities of large language models (LLMs) in multi-step multimodal tasks, aiming to address the problem of how to effectively utilize tools to complete complex tasks.