AgentSearch/AgentSearchBench-Tasks

--- license: mit language: - en tags: - agent - search - retrieval - reranking - benchmarking size_categories: - 1K<n<10K configs: - config_name: single-agent_task_query data_files: - split: validation path: single-agent_task_query/validation-00000-of-00001.parquet - split: test path: single-agent_task_query/test-00000-of-00001.parquet - config_name: multi-agent_task_query data_files: - split: validation path: multi-agent_task_query/validation-00000-of-00001.parquet - split: test path: multi-agent_task_query/test-00000-of-00001.parquet - config_name: task_description data_files: - split: validation path: task_description/validation-00000-of-00001.parquet - split: test path: task_description/test-00000-of-00001.parquet --- # AgentSearchBench Tasks **AgentSearchBench** is a large-scale benchmark for AI agent search, built from nearly 10,000 real-world agents sourced from the [GPT Store](https://chatgpt.com/gpts), [Google Cloud Marketplace](https://cloud.google.com/marketplace), and [AgentAI Platform](https://agent.ai/). 🌐 [Project Page](https://bingo-w.github.io/AgentSearchBench) • 💻 [Codebase](https://github.com/Bingo-W/AgentSearchBench) --- ## Overview This repository contains the **benchmark tasks** for AgentSearchBench. Agent search is framed as both a retrieval and reranking problem, where relevance is grounded in real execution performance rather than textual similarity alone. Tasks are generated by: 1. Creating concrete, executable queries from agent documentation. 2. Grouping and abstracting these into broader high-level task descriptions. Agent relevance is assessed by executing candidate agents on each task and evaluating outputs via an LLM Judge, with human alignment validation. --- ## Dataset Statistics | Split | Total | Task Description | Single-Agent Task Query | Multi-Agent Task Query | |------------|-------|-----------------|------------------------|------------------------| | Validation | 3,211 | 259 | 2,452 | 500 | | Test | 798 | 65 | 633 | 100 | --- ## Configurations This dataset contains three configurations, each representing a different query type: ### `single-agent_task_query` Concrete, executable task queries designed to be solved by a **single agent**. Queries are derived directly from agent documentation. ### `multi-agent_task_query` Executable task queries that require the **combination of multiple agents** to complete the task. ### `task_description` Higher-level, abstract task descriptions obtained by grouping and abstracting single-agent task queries. Useful for evaluating agent search under more realistic, open-ended user intents. --- ## Data Fields - `id`: Unique identifier for each task. - `task`: Task content. - `labels`: Binary retrieval labels. - `ranking_labels`: Graded ranking labels. - `ref_agents`: Reference agents used to generate the task. - `ref_subtasks`: Associated subtasks (multi-agent task query and task description). - `rubric`: Subtask selection rubric (task description only). --- ## Usage ```python from datasets import load_dataset # Single-agent task queries ds = load_dataset("AgentSearch/AgentSearchBench-Tasks", "single-agent_task_query") # Multi-agent task queries ds = load_dataset("AgentSearch/AgentSearchBench-Tasks", "multi-agent_task_query") # High-level task descriptions ds = load_dataset("AgentSearch/AgentSearchBench-Tasks", "task_description") ``` --- ## Related Datasets | Dataset | Description | |---------|-------------| | [AgentSearchBench-Agents](https://huggingface.co/datasets/AgentSearch/AgentSearchBench-Agents) | The AgentBase dataset: 9,759 real-world AI agents with metadata | | [AgentSearchBench-Responses](https://huggingface.co/datasets/AgentSearch/AgentSearchBench-Responses) | 60K+ raw agent execution responses from the validation set | --- ## Citation ```bibtex @article{} ```license: MIT许可证 language: - 英语 tags: - AI智能体（AI Agent） - 搜索 - 检索（retrieval） - 重排序（reranking） - 基准测试（benchmarking） size_categories: - 1000 < 样本量 < 10000 configs: - config_name: 单智能体任务查询（single-agent_task_query） data_files: - split: 验证集（validation） path: single-agent_task_query/validation-00000-of-00001.parquet - split: 测试集（test） path: single-agent_task_query/test-00000-of-00001.parquet - config_name: 多智能体任务查询（multi-agent_task_query） data_files: - split: 验证集（validation） path: multi-agent_task_query/validation-00000-of-00001.parquet - split: 测试集（test） path: multi-agent_task_query/test-00000-of-00001.parquet - config_name: 任务描述（task_description） data_files: - split: 验证集（validation） path: task_description/validation-00000-of-00001.parquet - split: 测试集（test） path: task_description/test-00000-of-00001.parquet # AgentSearchBench 基准任务 **AgentSearchBench** 是一款面向AI智能体（AI Agent）搜索的大规模基准测试集，构建自近10000个真实AI智能体，数据来源包括[GPT商店](https://chatgpt.com/gpts)、[谷歌云市场](https://cloud.google.com/marketplace)以及[AgentAI平台](https://agent.ai/)。 🌐 [项目主页](https://bingo-w.github.io/AgentSearchBench) • 💻 [代码仓库](https://github.com/Bingo-W/AgentSearchBench) --- ## 概述 本仓库包含AgentSearchBench的**基准任务**。AI智能体搜索被同时建模为检索与重排序任务，其相关性评判基于真实执行性能，而非仅依靠文本相似度。 任务生成流程如下： 1. 从智能体文档中生成具体可执行的查询； 2. 将这些查询分组并抽象为更宽泛的高层任务描述。 AI智能体相关性通过在每个任务上执行候选智能体，并借助大语言模型（LLM）评判器评估输出结果来确定，同时辅以人工对齐验证。 --- ## 数据集统计 | 数据集划分 | 总样本数 | 任务描述 | 单智能体任务查询 | 多智能体任务查询 | |------------|---------|---------|------------------|------------------| | 验证集 | 3,211 | 259 | 2,452 | 500 | | 测试集 | 798 | 65 | 633 | 100 | --- ## 数据集配置 本数据集包含三种配置，分别对应不同的查询类型： ### `single-agent_task_query`（单智能体任务查询） 专为单个AI智能体设计的可执行具体任务查询，查询直接源自智能体文档。 ### `multi-agent_task_query`（多智能体任务查询） 需要结合多个AI智能体才能完成的可执行任务查询。 ### `task_description`（任务描述） 通过分组并抽象单智能体任务查询得到的高层抽象任务描述，适用于在更贴近现实的开放式用户意图场景下评估AI智能体搜索性能。 --- ## 数据字段 - `id`：每个任务的唯一标识符。 - `task`：任务内容。 - `labels`：二元检索标签。 - `ranking_labels`：分级排序标签。 - `ref_agents`：用于生成该任务的参考智能体。 - `ref_subtasks`：关联子任务（仅适用于多智能体任务查询与任务描述配置）。 - `rubric`：子任务选择准则（仅适用于任务描述配置）。 --- ## 使用方法 python from datasets import load_dataset # 加载单智能体任务查询数据集 ds = load_dataset("AgentSearch/AgentSearchBench-Tasks", "single-agent_task_query") # 加载多智能体任务查询数据集 ds = load_dataset("AgentSearch/AgentSearchBench-Tasks", "multi-agent_task_query") # 加载高层任务描述数据集 ds = load_dataset("AgentSearch/AgentSearchBench-Tasks", "task_description") --- ## 相关数据集 | 数据集 | 描述 | |---------|-------------| | [AgentSearchBench-Agents](https://huggingface.co/datasets/AgentSearch/AgentSearchBench-Agents) | AgentBase数据集：包含9759个带元数据的真实AI智能体 | | [AgentSearchBench-Responses](https://huggingface.co/datasets/AgentSearch/AgentSearchBench-Responses) | 验证集上的60000+条原始智能体执行响应 | --- ## 引用格式 bibtex @article{}