Jize1/GTA

GTA是一个用于评估基于LLM的代理在现实场景中使用工具能力的基准数据集。该数据集包含229个人工编写的查询，涉及感知、操作、逻辑和创造力等多个类别的工具，并且每个查询都附有真实的图像文件作为查询上下文。数据集的结构包括工具信息、图像路径、对话序列和真实答案。数据集的创建过程包括查询构建和工具链构建两个步骤。

GTA is a benchmark to evaluate the tool-use capability of LLM-based agents in real-world scenarios. It features three main aspects: real user queries, real deployed tools, and real multimodal inputs. The dataset contains 229 human-written queries with implicit tool-use, requiring the LLM to reason the suitable tools and plan the solution steps. It provides an evaluation platform equipped with tools across perception, operation, logic, and creativity categories to evaluate the agents actual task execution performance. Each query is attached with authentic image files, such as spatial scenes, web page screenshots, tables, code snippets, and printed/handwritten materials, used as the query contexts to align with real-world scenarios closely.