pyxnpyx/SkillsGaurd_sft_dataset

# Agent Skills Security Audit Dataset

**Hugging Face**: [pyxnpyx/SkillsGaurd_sft_dataset](https://huggingface.co/datasets/pyxnpyx/SkillsGaurd_sft_dataset)
**File**: `skills_sft_dataset.json`
**Total Records**: 3,902

---

## Dataset Overview

This dataset is designed for training and evaluating AI models in Agent Skills security auditing. Each record pairs a vulnerable skill document with a comprehensive security audit report.

The dataset is built upon the vulnerability taxonomy established in **"Agent Skills in the Wild: An Empirical Study of Security Vulnerabilities at Scale"** (Liu et al., 2026), which identified **14 distinct vulnerability patterns** across **4 categories** from an analysis of 31,132 real-world skills.

---

## Data Structure

```json
{
"instruction": "You are a cybersecurity expert auditing Agent Skills...",
"input": "Metadata + SKILL.md documentation + attached scripts",
"output": "# Security Audit Report\n\n## Overall Assessment\n...",
"metadata": {
"skill_name": "string",
"category": "string",
"vuln_code": "string",
"vuln_description": "string",
"is_malicious": "boolean",
"has_scripts": "boolean",
"skill_type": "string",
"processed_time": "timestamp"
}
}
```

---

## Vulnerability Taxonomy

Based on the empirical study [1], vulnerabilities are classified into **4 categories** with **14 patterns**:

| Category | Code | Pattern | Severity |
|----------|------|---------|----------|
| **Prompt Injection** | P1 | Instruction Override | High |
| | P2 | Hidden Instructions | High |
| | P3 | Exfiltration Commands | High |
| | P4 | Behavior Manipulation | Medium |
| **Data Exfiltration** | E1 | External Data Transmission | Medium |
| | E2 | Environment Variable Harvesting | High |
| | E3 | File System Enumeration | Medium |
| | E4 | Context Leakage | High |
| **Privilege Escalation** | PE1 | Excessive Permission Requests | Low |
| | PE2 | Sudo/Root Execution | Medium |
| | PE3 | Credential Access | High |
| **Supply Chain** | SC1 | Unpinned Dependencies | Low |
| | SC2 | External Script Fetching | High |
| | SC3 | Obfuscated Code | High |

---

## Data Generation Process

### Stage 1: Vulnerable Skill Generation (Gemini 2.5)

Skills are generated based on the 14 vulnerability patterns from the taxonomy:

- Each skill appears legitimate while containing a specific hidden vulnerability
- Includes YAML frontmatter, comprehensive Markdown documentation, and executable scripts
- Vulnerabilities are subtle but detectable through careful analysis

### Stage 2: Security Audit Generation (GPT-4o-mini)

Each generated skill is analyzed using the study's four-dimensional risk framework:

| Dimension | Focus |
|-----------|-------|
| Prompt Injection | Instruction override, hidden commands, behavioral manipulation |
| Data Exfiltration | External transmission, credential theft, file enumeration |
| Privilege Escalation | Excessive permissions, sudo execution, credential access |
| Supply Chain | Unlocked dependencies, external code execution, obfuscation |

The audit report includes:
- Overall assessment and risk classification
- Per-dimension analysis with evidence
- Risk propagation scenarios
- Actionable remediation recommendations

---

## Vulnerability Distribution

| vuln_code | Count | Percentage | Category |
|-----------|-------|------------|----------|
| E1 | 340 | 8.71% | Data Exfiltration |
| PE1 | 324 | 8.30% | Privilege Escalation |
| P3 | 320 | 8.20% | Prompt Injection |
| E3 | 309 | 7.92% | Data Exfiltration |
| P1 | 308 | 7.89% | Prompt Injection |
| E4 | 306 | 7.84% | Data Exfiltration |
| E2 | 306 | 7.84% | Data Exfiltration |
| P4 | 303 | 7.77% | Prompt Injection |
| P2 | 246 | 6.30% | Prompt Injection |
| SC2 | 243 | 6.23% | Supply Chain |
| PE2 | 230 | 5.89% | Privilege Escalation |
| PE3 | 215 | 5.51% | Privilege Escalation |
| SC3 | 190 | 4.87% | Supply Chain |
| SC1 | 164 | 4.20% | Supply Chain |
| BENIGN | 98 | 2.51% | Safe Skills |

---

## Model Fine-tuning Results

The dataset was used to fine-tune GPT-4o-mini for security auditing. Evaluation against held-out test samples:

| Severity | Sample Count | Base Model | Fine-tuned Model | Improvement |
|----------|--------------|------------|------------------|-------------|
| CRITICAL | 19 | 47.4% | **94.7%** | +47.3 pp |
| HIGH | 36 | 30.6% | **61.1%** | +30.5 pp |
| MEDIUM | 21 | 38.1% | **85.7%** | +47.6 pp |
| LOW | 19 | 21.1% | **68.4%** | +47.3 pp |
| SAFE | 65 | 33.8% | **53.8%** | +20.0 pp |

**Key Findings:**
- Critical vulnerability detection improved by nearly 50 percentage points
- Fine-tuned model outperforms both base model and reference (GPT-5.4)
- Significant improvement across all severity levels demonstrates effective task-specific learning

---

## Quick Start

```python
from datasets import load_dataset

# Load the dataset
dataset = load_dataset("pyxnpyx/SkillsGaurd_sft_dataset", split="train")

# Access a sample
sample = dataset[0]
print(f"Vulnerability: {sample['metadata']['vuln_code']}")
print(f"Category: {sample['metadata']['category']}")
print(f"Audit Report: {sample['output'][:500]}...")

# Split for training
train_data = dataset.select(range(3500))
eval_data = dataset.select(range(3500, 3902))
```

---
## ⚠️ 重要声明

本数据集由 AI 模型生成（Gemini 2.5 生成技能，GPT-4o-mini 生成审计报告），未经人工逐条校验，可能存在幻觉、标签噪声和语义偏差。请谨慎使用，建议配合人工抽检验证。

---

## Citation

If you use this dataset, please cite both the original study and the dataset:

```bibtex
@inproceedings{liu2026skills,
title = {Agent Skills in the Wild: An Empirical Study of Security Vulnerabilities at Scale},
author = {Liu, Yi and Wang, Weizhe and Feng, Ruitao and Zhang, Yao and Xu, Guangquan and Deng, Gelei and Li, Yuekang and Zhang, Leo},
year = {2026}
}

@misc{skillsguard-sft-2026,
title = {SkillsGuard SFT Dataset: Agent Skills Security Audit Training Data},
author = {SkillsGuard Team},
year = {2026},
publisher = {Hugging Face},
howpublished = {\url{https://huggingface.co/datasets/pyxnpyx/SkillsGaurd_sft_dataset}}
}
```

---

## License

MIT License

---

**Version**: 1.0 | **Last Updated**: March 22, 2026
**Reference**: [1] Liu et al., "Agent Skills in the Wild: An Empirical Study of Security Vulnerabilities at Scale," 2026.