pyxnpyx/SkillsGaurd_sft_dataset

# Agent Skills Security Audit Dataset **Hugging Face**: [pyxnpyx/SkillsGaurd_sft_dataset](https://huggingface.co/datasets/pyxnpyx/SkillsGaurd_sft_dataset) **File**: `skills_sft_dataset.json` **Total Records**: 3,902 --- ## Dataset Overview This dataset is designed for training and evaluating AI models in Agent Skills security auditing. Each record pairs a vulnerable skill document with a comprehensive security audit report. The dataset is built upon the vulnerability taxonomy established in **"Agent Skills in the Wild: An Empirical Study of Security Vulnerabilities at Scale"** (Liu et al., 2026), which identified **14 distinct vulnerability patterns** across **4 categories** from an analysis of 31,132 real-world skills. --- ## Data Structure ```json { "instruction": "You are a cybersecurity expert auditing Agent Skills...", "input": "Metadata + SKILL.md documentation + attached scripts", "output": "# Security Audit Report\n\n## Overall Assessment\n...", "metadata": { "skill_name": "string", "category": "string", "vuln_code": "string", "vuln_description": "string", "is_malicious": "boolean", "has_scripts": "boolean", "skill_type": "string", "processed_time": "timestamp" } } ``` --- ## Vulnerability Taxonomy Based on the empirical study [1], vulnerabilities are classified into **4 categories** with **14 patterns**: | Category | Code | Pattern | Severity | |----------|------|---------|----------| | **Prompt Injection** | P1 | Instruction Override | High | | | P2 | Hidden Instructions | High | | | P3 | Exfiltration Commands | High | | | P4 | Behavior Manipulation | Medium | | **Data Exfiltration** | E1 | External Data Transmission | Medium | | | E2 | Environment Variable Harvesting | High | | | E3 | File System Enumeration | Medium | | | E4 | Context Leakage | High | | **Privilege Escalation** | PE1 | Excessive Permission Requests | Low | | | PE2 | Sudo/Root Execution | Medium | | | PE3 | Credential Access | High | | **Supply Chain** | SC1 | Unpinned Dependencies | Low | | | SC2 | External Script Fetching | High | | | SC3 | Obfuscated Code | High | --- ## Data Generation Process ### Stage 1: Vulnerable Skill Generation (Gemini 2.5) Skills are generated based on the 14 vulnerability patterns from the taxonomy: - Each skill appears legitimate while containing a specific hidden vulnerability - Includes YAML frontmatter, comprehensive Markdown documentation, and executable scripts - Vulnerabilities are subtle but detectable through careful analysis ### Stage 2: Security Audit Generation (GPT-4o-mini) Each generated skill is analyzed using the study's four-dimensional risk framework: | Dimension | Focus | |-----------|-------| | Prompt Injection | Instruction override, hidden commands, behavioral manipulation | | Data Exfiltration | External transmission, credential theft, file enumeration | | Privilege Escalation | Excessive permissions, sudo execution, credential access | | Supply Chain | Unlocked dependencies, external code execution, obfuscation | The audit report includes: - Overall assessment and risk classification - Per-dimension analysis with evidence - Risk propagation scenarios - Actionable remediation recommendations --- ## Vulnerability Distribution | vuln_code | Count | Percentage | Category | |-----------|-------|------------|----------| | E1 | 340 | 8.71% | Data Exfiltration | | PE1 | 324 | 8.30% | Privilege Escalation | | P3 | 320 | 8.20% | Prompt Injection | | E3 | 309 | 7.92% | Data Exfiltration | | P1 | 308 | 7.89% | Prompt Injection | | E4 | 306 | 7.84% | Data Exfiltration | | E2 | 306 | 7.84% | Data Exfiltration | | P4 | 303 | 7.77% | Prompt Injection | | P2 | 246 | 6.30% | Prompt Injection | | SC2 | 243 | 6.23% | Supply Chain | | PE2 | 230 | 5.89% | Privilege Escalation | | PE3 | 215 | 5.51% | Privilege Escalation | | SC3 | 190 | 4.87% | Supply Chain | | SC1 | 164 | 4.20% | Supply Chain | | BENIGN | 98 | 2.51% | Safe Skills | --- ## Model Fine-tuning Results The dataset was used to fine-tune GPT-4o-mini for security auditing. Evaluation against held-out test samples: | Severity | Sample Count | Base Model | Fine-tuned Model | Improvement | |----------|--------------|------------|------------------|-------------| | CRITICAL | 19 | 47.4% | **94.7%** | +47.3 pp | | HIGH | 36 | 30.6% | **61.1%** | +30.5 pp | | MEDIUM | 21 | 38.1% | **85.7%** | +47.6 pp | | LOW | 19 | 21.1% | **68.4%** | +47.3 pp | | SAFE | 65 | 33.8% | **53.8%** | +20.0 pp | **Key Findings:** - Critical vulnerability detection improved by nearly 50 percentage points - Fine-tuned model outperforms both base model and reference (GPT-5.4) - Significant improvement across all severity levels demonstrates effective task-specific learning --- ## Quick Start ```python from datasets import load_dataset # Load the dataset dataset = load_dataset("pyxnpyx/SkillsGaurd_sft_dataset", split="train") # Access a sample sample = dataset[0] print(f"Vulnerability: {sample['metadata']['vuln_code']}") print(f"Category: {sample['metadata']['category']}") print(f"Audit Report: {sample['output'][:500]}...") # Split for training train_data = dataset.select(range(3500)) eval_data = dataset.select(range(3500, 3902)) ``` --- ## ⚠️ 重要声明 本数据集由 AI 模型生成（Gemini 2.5 生成技能，GPT-4o-mini 生成审计报告），未经人工逐条校验，可能存在幻觉、标签噪声和语义偏差。请谨慎使用，建议配合人工抽检验证。 --- ## Citation If you use this dataset, please cite both the original study and the dataset: ```bibtex @inproceedings{liu2026skills, title = {Agent Skills in the Wild: An Empirical Study of Security Vulnerabilities at Scale}, author = {Liu, Yi and Wang, Weizhe and Feng, Ruitao and Zhang, Yao and Xu, Guangquan and Deng, Gelei and Li, Yuekang and Zhang, Leo}, year = {2026} } @misc{skillsguard-sft-2026, title = {SkillsGuard SFT Dataset: Agent Skills Security Audit Training Data}, author = {SkillsGuard Team}, year = {2026}, publisher = {Hugging Face}, howpublished = {\url{https://huggingface.co/datasets/pyxnpyx/SkillsGaurd_sft_dataset}} } ``` --- ## License MIT License --- **Version**: 1.0 | **Last Updated**: March 22, 2026 **Reference**: [1] Liu et al., "Agent Skills in the Wild: An Empirical Study of Security Vulnerabilities at Scale," 2026.# AI智能体（AI Agent）技能安全审计数据集 **Hugging Face**: [pyxnpyx/SkillsGaurd_sft_dataset](https://huggingface.co/datasets/pyxnpyx/SkillsGaurd_sft_dataset) **文件**: `skills_sft_dataset.json` **总记录数**: 3902 --- ## 数据集概览 本数据集专为AI智能体技能安全审计领域的模型训练与评估而设计，每条数据均将存在安全漏洞的技能文档与完整的安全审计报告进行配对。 本数据集基于《野外AI智能体技能：大规模安全漏洞实证研究》（*Agent Skills in the Wild: An Empirical Study of Security Vulnerabilities at Scale*，Liu等人，2026）中提出的漏洞分类体系构建。该研究通过分析31132个真实AI智能体技能，识别出4大类共14种独立的漏洞模式。 --- ## 数据结构 json { "instruction": "你是一名审计AI智能体技能的网络安全专家...", "input": "元数据 + SKILL.md 文档 + 附加脚本", "output": "# 安全审计报告 ## 整体评估 ...", "metadata": { "skill_name": "字符串", "category": "字符串", "vuln_code": "字符串", "vuln_description": "字符串", "is_malicious": "布尔值", "has_scripts": "布尔值", "skill_type": "字符串", "processed_time": "时间戳" } } --- ## 漏洞分类体系 基于上述实证研究[1]，漏洞被划分为4大类共14种模式： | 漏洞类别 | 漏洞代码 | 漏洞模式 | 危害等级 | |----------|----------|----------|----------| | **提示词注入（Prompt Injection）** | P1 | 指令覆盖 | 高危 | | | P2 | 隐藏指令 | 高危 | | | P3 | 外泄指令 | 高危 | | | P4 | 行为操纵 | 中危 | | **数据外泄（Data Exfiltration）** | E1 | 外部数据传输 | 中危 | | | E2 | 环境变量采集 | 高危 | | | E3 | 文件系统枚举 | 中危 | | | E4 | 上下文泄露 | 高危 | | **权限提升（Privilege Escalation）** | PE1 | 过度权限申请 | 低危 | | | PE2 | Sudo/Root 执行 | 中危 | | | PE3 | 凭证获取 | 高危 | | **供应链攻击（Supply Chain）** | SC1 | 未固定依赖 | 低危 | | | SC2 | 外部脚本拉取 | 高危 | | | SC3 | 代码混淆 | 高危 | --- ## 数据生成流程 ### 阶段1：漏洞技能生成（基于Gemini 2.5） 技能基于漏洞分类体系中的14种漏洞模式生成： - 每一项技能在外观上均具备合法性，但隐藏有特定的安全漏洞 - 包含YAML前置元数据、完整的Markdown文档以及可执行脚本 - 漏洞较为隐蔽，但通过细致分析即可被检测发现 ### 阶段2：安全审计报告生成（基于GPT-4o-mini） 每一项生成的技能均采用该研究提出的四维风险框架进行分析： | 审计维度 | 关注方向 | |-----------|----------| | 提示词注入 | 指令覆盖、隐藏指令、行为操纵 | | 数据外泄 | 外部数据传输、凭证窃取、文件枚举 | | 权限提升 | 过度权限申请、Sudo执行、凭证获取 | | 供应链攻击 | 未固定依赖、外部代码执行、代码混淆 | 审计报告应包含： - 整体评估与风险分类 - 各维度的详细分析及佐证证据 - 风险传播场景分析 - 可落地的修复建议 --- ## 漏洞分布情况 | 漏洞代码 | 数量 | 占比 | 漏洞类别 | |----------|------|------|----------| | E1 | 340 | 8.71% | 数据外泄 | | PE1 | 324 | 8.30% | 权限提升 | | P3 | 320 | 8.20% | 提示词注入 | | E3 | 309 | 7.92% | 数据外泄 | | P1 | 308 | 7.89% | 提示词注入 | | E4 | 306 | 7.84% | 数据外泄 | | E2 | 306 | 7.84% | 数据外泄 | | P4 | 303 | 7.77% | 提示词注入 | | P2 | 246 | 6.30% | 提示词注入 | | SC2 | 243 | 6.23% | 供应链攻击 | | PE2 | 230 | 5.89% | 权限提升 | | PE3 | 215 | 5.51% | 权限提升 | | SC3 | 190 | 4.87% | 供应链攻击 | | SC1 | 164 | 4.20% | 供应链攻击 | | BENIGN | 98 | 2.51% | 良性（BENIGN） | --- ## 模型微调效果 本数据集被用于对GPT-4o-mini进行安全审计任务的微调。针对预留测试样本的评估结果如下： | 危害等级 | 样本数量 | 基准模型 | 微调后模型 | 性能提升 | |----------|----------|----------|------------|----------| | 极高危（CRITICAL） | 19 | 47.4% | **94.7%** | +47.3个百分点 | | 高危（HIGH） | 36 | 30.6% | **61.1%** | +30.5个百分点 | | 中危（MEDIUM） | 21 | 38.1% | **85.7%** | +47.6个百分点 | | 低危（LOW） | 19 | 21.1% | **68.4%** | +47.3个百分点 | | 安全（SAFE） | 65 | 33.8% | **53.8%** | +20.0个百分点 | **核心发现：** - 极高危漏洞检测率提升近50个百分点 - 微调后模型性能优于基准模型及参照模型（GPT-5.4） - 各危害等级均实现显著性能提升，证明模型完成了针对该任务的专属学习 --- ## 快速上手 python from datasets import load_dataset # 加载数据集 dataset = load_dataset("pyxnpyx/SkillsGaurd_sft_dataset", split="train") # 访问单条样本 sample = dataset[0] print(f"漏洞代码：{sample['metadata']['vuln_code']}") print(f"漏洞类别：{sample['metadata']['category']}") print(f"审计报告：{sample['output'][:500]}...") # 拆分训练集与验证集 train_data = dataset.select(range(3500)) eval_data = dataset.select(range(3500, 3902)) --- ## ⚠️ 重要声明 本数据集由AI模型生成（Gemini 2.5生成技能，GPT-4o-mini生成审计报告），未经人工逐条校验，可能存在幻觉、标签噪声和语义偏差。请谨慎使用，建议配合人工抽检验证。 --- ## 引用格式 若使用本数据集，请同时引用原始研究与本数据集： bibtex @inproceedings{liu2026skills, title = {Agent Skills in the Wild: An Empirical Study of Security Vulnerabilities at Scale}, author = {Liu, Yi and Wang, Weizhe and Feng, Ruitao and Zhang, Yao and Xu, Guangquan and Deng, Gelei and Li, Yuekang and Zhang, Leo}, year = {2026} } @misc{skillsguard-sft-2026, title = {SkillsGuard SFT Dataset: Agent Skills Security Audit Training Data}, author = {SkillsGuard Team}, year = {2026}, publisher = {Hugging Face}, howpublished = {url{https://huggingface.co/datasets/pyxnpyx/SkillsGaurd_sft_dataset}} } --- ## 许可证 MIT许可证 --- **版本**: 1.0 | **最后更新**: 2026年3月22日 **参考文献**: [1] Liu等人，《野外AI智能体技能：大规模安全漏洞实证研究》，2026年。