thaki-AI/SkillRetBench

--- license: apache-2.0 language: - en - ko tags: - skill-retrieval - information-retrieval - agents - benchmark - retrieval pretty_name: SkillRetBench size_categories: - n<1K --- # SkillRetBench ## Dataset summary **SkillRetBench** is a benchmark for **query-to-skill retrieval**: given a natural-language user request, systems must rank or select the correct procedural skill document(s) from a fixed library. The corpus is derived from a real-world production-style agent skill library (**501** skills). The benchmark includes **1,250** queries with gold skill identifiers and optional distractors, spanning five evaluation settings (single skill, multi-skill composition, distractors, outdated/redundant, budget-constrained). Packaged **baseline experiment results** (BM25, dense, hybrid, LLM-style, and SADO-style runs) report standard IR metrics (Recall@k, nDCG@k, MRR, MAP) per setting. SkillRetBench accompanies the **SEFO** framework (Self-Evolving Federated Orchestration) and targets research on skill-aware orchestration, large skill libraries, and retrieval under composition and noise. ## Dataset structure The release consists of three JSON artifacts (also listed in `croissant.json`): | File | Description | |------|-------------| | `skill_corpus.json` | Full skill index: metadata, corpus statistics, and the `skills` array (one object per skill). | | `skillretbench_queries.json` | Benchmark definition: `meta` (counts, seed, generator) and `queries` array. | | `baseline_results.json` | Aggregated baseline metrics: per-method, per-setting IR scores and a markdown summary table. | ### `skill_corpus.json` - **`meta`**: `generated_at` (ISO 8601), `skills_root`, `output_path`, `skill_file_count`. - **`statistics`**: `total_skills`, `skills_per_category`, description length stats, `token_count_distribution`, composition graph stats (`skills_with_composition_references`, `total_composition_edges`). - **`skills`** (array, length 501): each element includes: - `skill_id` (string, canonical id) - `skill_name` (string) - `description` (string, short routing description) - `trigger_phrases` (string array) - `anti_triggers` (string array) - `korean_triggers` (string array; may be empty) - `category` (string) - `full_text` (string, full SKILL.md-derived body) - `token_count` (integer) - `composable_skills` (string array; referenced skill ids) - `parse_warnings` (string array) ### `skillretbench_queries.json` - **`meta`**: `benchmark`, `generator`, `corpus_meta`, `total_queries` (1250), `counts_by_setting`, `rng_seed` (42). - **`queries`** (array, length 1250): each element includes: - `query_id` (string, e.g. `SS-0001`) - `setting` (enum-like string): `single_skill` | `multi_skill_composition` | `distractor` | `outdated_redundant` | `budget_constrained` - `query` (string; English and/or Korean natural language) - `gold_skills` (string array; expected skill ids) - `distractor_skills` (string array) - `budget_tokens` (number or `null`) - `difficulty` (string, e.g. `easy`, `medium`) - `source` (string, e.g. `trigger_paraphrase`) **Query counts by setting** | Setting | Count | |---------|------:| | `single_skill` | 400 | | `multi_skill_composition` | 200 | | `distractor` | 300 | | `outdated_redundant` | 150 | | `budget_constrained` | 200 | ### `baseline_results.json` - **`timestamp`**, **`meta`**: `corpus_skills`, `queries_total`, `queries_by_setting`, `dense_backend` (e.g. `jaccard_fallback`). - **`baselines`**: nested object — top-level keys **`BM25`**, **`Dense`**, **`Hybrid`**, **`NaiveLLM`**, **`SADO`**. Under each method, keys match query `setting` names; each leaf holds metrics: `recall@1`, `recall@3`, `recall@5`, `recall@10`, `ndcg@1`, `ndcg@3`, `ndcg@5`, `ndcg@10`, `mrr`, `map` (floats). - **`summary_table`**: pre-rendered markdown table (macro-style summary over settings). > **Note:** As documented in the artifact, `NaiveLLM` and `SADO` entries may be **simulated** (no live API) for reproducible baselines; see project script `run_baselines.py`. ## Supported tasks - **skill-retrieval** — Rank or classify skills given a query; evaluate with gold `gold_skills` and IR metrics. - **information-retrieval** — Same retrieval formulation; comparable to passage/tool retrieval with long procedural documents. ## Languages - **English** — Majority of skill text and many queries. - **Korean** — Present in `korean_triggers`, many `query` strings, and mixed Korean/English skill descriptions. ## Dataset creation ### Sources - Skills were built from **Cursor agent `SKILL.md` files** under a single repository’s `.cursor/skills` tree (501 files), parsed into structured records plus full markdown bodies. - Queries were **programmatically generated** by `app.sefo.benchmark.query_generator` with fixed **`rng_seed`: 42** for reproducibility. ### Methodology (high level) 1. **Corpus build** — Parse frontmatter and body; extract triggers, categories, composition edges, and token counts. 2. **Query generation** — Instantiate templates and paraphrases aligned to skills and settings (single vs. multi-skill, distractors, outdated/redundant, token budget hints). 3. **Baselines** — Run packaged retrievers/rankers (lexical, dense with configured fallback, hybrid, simulated LLM/SADO-style) and aggregate metrics per setting. ## Considerations ### Biases - **Corpus domain** — Skills reflect one organization’s automation stack (dev tooling, PM, trading, cloud, etc.); frequency and vocabulary are **not** uniformly representative of all industries or locales. - **Authoring style** — Descriptions follow a consistent “when to use / do not use” pattern; models may exploit **format bias** rather than deep semantics. - **Query generation** — Synthetic and template-driven queries may **over- or under-represent** realistic user phrasing compared to production logs. ### Limitations - **Scale** — 501 skills is medium-scale vs. ecosystems with 10⁵+ tools; retrieval hardness may not transfer linearly. - **Gold labels** — Defined by benchmark construction rules; edge cases and valid alternative skill sets may exist. - **Static snapshot** — Skill text and ids are frozen for the benchmark revision tied to `meta.generated_at` in the JSON files. - **Baseline simulation** — Some methods are noted as simulated in-code; compare only under the same harness and seeds. ## License This dataset is released under the **Apache License 2.0**. See [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0). ## Citation If you use SkillRetBench or this corpus, please cite: ```bibtex @misc{han2026sefo, title = {{SEFO}: Self-Evolving Federated Orchestration with Trusted Skill Governance and Skill Retrieval Benchmark for Recursive Agentic Systems}, author = {Han, Hyojung and {ThakiCloud AI Research}}, year = {2026}, howpublished = {\url{https://github.com/hyojunguy/ai-model-event-stock-analytics}}, note = {Includes SkillRetBench: 501-skill corpus and 1,250-query retrieval benchmark} } ``` Adjust `howpublished` / venue fields when a formal publication DOI is available. ## Dataset repository layout (recommended) After upload to the Hugging Face Hub, a typical layout is: ``` skill_corpus.json skillretbench_queries.json baseline_results.json croissant.json README.md ``` ## Additional documentation - Croissant (ML Commons) metadata: **`croissant.json`** in this folder. - Implementation and regeneration: `backend/app/sefo/benchmark/` in the source repository.许可证：Apache许可证2.0 语言：英语（en）、韩语（ko） 标签：技能检索（skill-retrieval）、信息检索（information-retrieval）、智能体（agents）、基准测试（benchmark）、检索（retrieval） 展示名称：SkillRetBench 规模分类：样本数少于1000（n<1K） # SkillRetBench基准测试集 ## 数据集概述 **SkillRetBench**是面向**查询-技能检索（query-to-skill retrieval）**任务的基准测试集：给定自然语言形式的用户请求，系统需从固定技能库中排序或选出正确的过程性技能文档。其语料库源自真实生产级智能体技能库（共**501**项技能）。该基准测试集包含**1250**条带有黄金技能标识符与可选干扰项的查询，涵盖五种评估场景：单技能、多技能组合、干扰项、过时/冗余、预算约束。打包提供的基线实验结果（包含BM25、密集检索、混合检索、朴素大语言模型（NaiveLLM）风格及SADO风格的运行结果）报告了各场景下的标准信息检索指标：Recall@k、nDCG@k、MRR、MAP。 SkillRetBench配套**SEFO**框架（自我演化联邦编排（Self-Evolving Federated Orchestration）），目标是支持技能感知编排、大型技能库以及组合与噪声场景下的检索相关研究。 ## 数据集结构 本次发布包含三个JSON格式的数据集文件（同时在`croissant.json`中列出）： | 文件名 | 描述 | |------|-------------| | `skill_corpus.json` | 完整技能索引，包含元数据、语料库统计信息及`skills`数组（每项技能对应一个对象） | | `skillretbench_queries.json` | 基准测试集定义，包含`meta`字段（统计信息、随机种子、生成器）及`queries`数组 | | `baseline_results.json` | 聚合后的基线指标，包含各方法、各场景下的信息检索得分及Markdown格式的汇总表格 | ### `skill_corpus.json` - **`meta`**字段：包含`generated_at`（ISO 8601格式时间戳）、`skills_root`、`output_path`、`skill_file_count`。 - **`statistics`**字段：包含`total_skills`（总技能数）、`skills_per_category`（按分类统计的技能数）、描述长度统计、`token_count_distribution`（Token数量分布）、组合图统计信息（`skills_with_composition_references`，即带有组合引用的技能数；`total_composition_edges`，即总组合边数）。 - **`skills`**数组（共501个元素）：每个元素包含以下字段： - `skill_id`：字符串类型，规范技能标识符 - `skill_name`：字符串类型，技能名称 - `description`：字符串类型，简短路由描述 - `trigger_phrases`：字符串数组，触发短语 - `anti_triggers`：字符串数组，反触发短语 - `korean_triggers`：字符串数组，韩语触发短语，可为空 - `category`：字符串类型，技能分类 - `full_text`：字符串类型，源自SKILL.md的完整正文 - `token_count`：整数类型，Token数量 - `composable_skills`：字符串数组，引用的技能标识符 - `parse_warnings`：字符串数组，解析警告信息 ### `skillretbench_queries.json` - **`meta`**字段：包含`benchmark`、`generator`、`corpus_meta`、`total_queries`（总查询数，1250）、`counts_by_setting`（按场景统计的查询数）、`rng_seed`（随机种子，42）。 - **`queries`**数组（共1250个元素）：每个元素包含以下字段： - `query_id`：字符串类型，例如`SS-0001` - `setting`：枚举类字符串，可选值为`single_skill`（单技能）、`multi_skill_composition`（多技能组合）、`distractor`（干扰项）、`outdated_redundant`（过时/冗余）、`budget_constrained`（预算约束） - `query`：字符串类型，可为英语和/或韩语自然语言 - `gold_skills`：字符串数组，预期匹配的技能标识符 - `distractor_skills`：字符串数组，干扰项技能标识符 - `budget_tokens`：数值或`null`，Token预算 - `difficulty`：字符串类型，例如`easy`（简单）、`medium`（中等） - `source`：字符串类型，例如`trigger_paraphrase`（触发短语释义） #### 按场景统计的查询数量 | 评估场景 | 查询数量 | |---------|------:| | 单技能（single_skill） | 400 | | 多技能组合（multi_skill_composition） | 200 | | 干扰项（distractor） | 300 | | 过时/冗余（outdated_redundant） | 150 | | 预算约束（budget_constrained） | 200 | ### `baseline_results.json` - **`timestamp`**与**`meta`**字段：包含`corpus_skills`、`queries_total`、`queries_by_setting`、`dense_backend`（例如`jaccard_fallback`，即Jaccard回退策略）。 - **`baselines`**为嵌套对象：顶层键为**`BM25`**、**`Dense`**（密集检索）、**`Hybrid`**（混合检索）、**`NaiveLLM`**（朴素大语言模型）、**`SADO`**。每个方法下的键与查询场景名称对应；每个叶子节点包含以下指标：`recall@1`（召回率@1）、`recall@3`（召回率@3）、`recall@5`（召回率@5）、`recall@10`（召回率@10）、`ndcg@1`（归一化折损累积增益@1）、`ndcg@3`（归一化折损累积增益@3）、`ndcg@5`（归一化折损累积增益@5）、`ndcg@10`（归一化折损累积增益@10）、`mrr`（平均倒数排名）、`map`（平均精度），均为浮点型数值。 - **`summary_table`**：预渲染的Markdown表格，用于各场景的宏风格汇总。 > **注意**：如数据集文件中所述，`NaiveLLM`与`SADO`的运行结果可能为**模拟结果**（未调用真实API），以保证基线实验的可复现性；详情请参考项目脚本`run_baselines.py`。 ## 支持任务 - **技能检索（skill-retrieval）**：根据给定查询对技能进行排序或分类；通过黄金标签`gold_skills`与信息检索指标进行评估。 - **信息检索（information-retrieval）**：采用相同的检索范式；可类比长过程性文档的段落/工具检索任务。 ## 支持语言 - **英语**：技能文本与多数查询的主要语言。 - **韩语**：出现在`korean_triggers`字段、多数查询字符串及混合韩英双语的技能描述中。 ## 数据集构建 ### 数据来源 - 技能源自在单个代码仓库的`.cursor/skills`目录下的**Cursor智能体`SKILL.md`文件**（共501个文件），已解析为结构化记录与完整Markdown正文。 - 查询由`app.sefo.benchmark.query_generator`通过**程序化方式生成**，并使用固定随机种子`rng_seed: 42`以保证可复现性。 ### 构建方法（概要） 1. **语料库构建**：解析Markdown文件的前置元数据与正文；提取触发短语、分类、组合边及Token数量。 2. **查询生成**：生成与技能及场景匹配的模板与释义（涵盖单技能、多技能组合、干扰项、过时/冗余场景及Token预算提示）。 3. **基线实验**：运行打包的检索器/排序器（词法检索、带配置回退的密集检索、混合检索、模拟大语言模型/SADO风格方法），并按场景聚合指标。 ## 注意事项 ### 偏差问题 - **语料库领域偏差**：技能集仅反映单个组织的自动化技术栈（开发工具、项目管理、交易、云服务等）；词汇与词频无法均匀代表所有行业或地区。 - **编写风格偏差**：技能描述遵循统一的“何时使用/何时不使用”范式；模型可能会利用**格式偏差**而非深层语义进行检索。 - **查询生成偏差**：与真实生产日志中的用户表述相比，模板驱动的合成查询可能**过度或不足代表**真实用户用语。 ### 局限性 - **规模局限性**：相较于拥有10万+工具的生态系统，501项技能属于中等规模；检索难度可能无法线性迁移至更大规模的场景。 - **黄金标签局限性**：黄金标签由基准测试集构建规则定义；可能存在边缘案例与合法的替代技能集。 - **静态快照局限性**：技能文本与标识符已被冻结，对应JSON文件中`meta.generated_at`所标记的基准测试版本。 - **基线模拟局限性**：部分方法在代码中被标记为模拟结果；仅可在相同的实验框架与随机种子下进行对比。 ## 许可证 本数据集采用**Apache许可证2.0**发布，详情请参阅[Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0)。 ## 引用格式 若您使用SkillRetBench或该语料库，请引用以下文献： bibtex @misc{han2026sefo, title = {{SEFO}: Self-Evolving Federated Orchestration with Trusted Skill Governance and Skill Retrieval Benchmark for Recursive Agentic Systems}, author = {Han, Hyojung and {ThakiCloud AI Research}}, year = {2026}, howpublished = {url{https://github.com/hyojunguy/ai-model-event-stock-analytics}}, note = {注：包含SkillRetBench：501项技能语料库与1250条查询的检索基准测试集} } > 注：若有正式出版的DOI，请调整`howpublished`或会议/期刊字段。 ## 推荐的数据集仓库布局 上传至Hugging Face Hub后，典型的仓库布局如下： skill_corpus.json skillretbench_queries.json baseline_results.json croissant.json README.md ## 补充文档 - Croissant（ML Commons）元数据：本文件夹下的**`croissant.json`**文件。 - 实现与重新生成代码：源代码仓库中的`backend/app/sefo/benchmark/`目录。