zenith6888/SkillsBench-1650

---
language:
- en
license: cc-by-4.0
size_categories:
- 1K<n<10K
task_categories:
- text-classification
tags:
- security
- malware-detection
- ai-agent
- supply-chain-security
- claude-code
- skills
source_datasets:
- FayeZC/AgentSkills-138k
pretty_name: SkillBench-1650
---

# SkillBench-1650

A benchmark dataset for evaluating safety detection systems on AI agent skill packages. Contains 1,500 benign and 150 malicious Claude Code skill samples, designed for multi-dimensional security scoring research.

## Dataset Summary

| Split | Samples | Description |
|---|---|---|
| `benign` | 1,500 | Real-world skills sampled from open-source repositories |
| `malicious` | 150 | Adversarial payloads injected into real skill hosts |
| **Total** | **1,650** | |

Benign samples are sourced from [FayeZC/SkillMD-138k](https://huggingface.co/datasets/FayeZC/SkillMD-138k). Malicious samples are synthetically constructed by injecting attack payloads into real benign skill hosts from the same source, preserving realistic file structure and content length.

## Motivation

Existing skill security tools perform binary (safe/unsafe) classification. This dataset supports research on **continuous risk scoring** across multiple threat dimensions, enabling finer-grained safety assessment of AI agent skill ecosystems.

## Data Fields

| Field | Type | Description |
|---|---|---|
| `content_hash` | string | SHA-256 prefix of the SKILL.md content |
| `repo` | string | Source GitHub repository |
| `path` | string | Original file path in the repository |
| `stars` | int | Repository star count (0 for malicious samples) |
| `source` | string | Data source identifier |
| `html_url` | string | GitHub URL of the original file |
| `content` | string | Full SKILL.md content |
| `lines` | int | Line count |
| `words` | int | Word count |
| `label` | string | `benign` or `malicious` |
| `attack_type` | string | Attack category (see below), `none` for benign |
| `has_scripts` | bool | Whether the skill includes companion scripts |
| `script_content` | string | Concatenated companion script contents |
| `script_files` | string | JSON list of companion script filenames |
| `script_type` | string | `python`, `shell`, `javascript`, `ruby`, `mixed`, `binary`, or `none` |
| `difficulty` | string | Detection difficulty: `easy`, `medium`, `hard`, or `none` |

## Attack Taxonomy

The malicious split covers 8 attack types across 3 difficulty levels:

| Attack Type | Count | Description |
|---|---|---|
| `plaintext_cmd` | 19 | Unobfuscated malicious commands disguised as troubleshooting steps or deployment scripts |
| `encoded_payload` | 19 | Commands hidden behind encoding layers (base64, hex, rot13, XOR, zlib) |
| `remote_script` | 19 | Downloads and executes code from external URLs (curl\|bash, pip typosquatting, crontab persistence) |
| `steganography` | 18 | Payloads hidden in legitimate data structures (theme configs, translation dicts, test fixtures) |
| `rsa_modular` | 19 | Commands encrypted with textbook RSA, decrypted at runtime via `pow(c, d, n)` |
| `ascii_smuggle` | 19 | Legitimate URL strings secretly used as XOR keys to decrypt hidden payloads |
| `binary_exec` | 19 | Malicious logic in opaque binary files (.bin, .so), invisible to static text analysis |
| `data_exfil` | 18 | Direct exfiltration of sensitive files via HTTP, DNS, or socket connections |

### Difficulty Levels

| Level | Count | Description |
|---|---|---|
| `easy` | 50 | Detectable by basic pattern matching (e.g., `curl \| bash`, `base64 -d`) |
| `medium` | 52 | Blends into code structure, requires contextual understanding |
| `hard` | 48 | Semantically disguised with plausible cover stories, requires deep analysis or taint tracking |

## Benign Split Statistics

- **With companion scripts**: 285 / 1,500 (19%)
- **Script type distribution**: Python (162), JavaScript (52), Shell (39), Mixed (28), Ruby (3), None (1,216)
- **Average content length**: 7,721 characters / 253 lines

## Construction Methodology

### Benign Samples

1. Started from the [FayeZC/AgentSkills-134k](https://huggingface.co/datasets/FayeZC/SkillMD-138k) dataset (138K Claude Code SKILL.md files)
2. Scanned all 2,500 unique repositories via GitHub Git Tree API to identify skills with companion scripts (Python, Shell, JavaScript, etc.)
3. Sampled 285 skills with scripts and 1,215 without, downloading full script content from GitHub
4. Companion scripts are stored concatenated with `--- filename ---` separators

### Malicious Samples

1. Selected 50 benign skills (3,000-15,000 chars) as injection hosts
2. Designed 8 attack types with 3 difficulty tiers each
3. Injected payloads at controlled positions (early/middle/late) into the host's SKILL.md or companion scripts
4. Preserved original file structure: hosts with scripts retain all original files
5. Script-injection payloads are only paired with hosts that have real scripts; MD-injection payloads use any host

### Design Principles

- **Realism**: Payloads use plausible cover stories (deployment validators, crash reporters, font loaders) rather than obviously malicious naming
- **Host preservation**: Original skill structure is preserved; only the injection point is modified
- **Semantic coherence**: Hard-difficulty payloads maintain semantic consistency between their cover story and behavior
- **Domain safety**: All URLs use `example.com` (IANA reserved) to prevent accidental connections

## Intended Use

- Evaluating static analysis tools for AI agent skill security
- Benchmarking multi-dimensional risk scoring frameworks
- Studying the effectiveness of pattern matching, statistical analysis, and taint tracking against different attack types and difficulty levels
- Research on supply-chain security in AI agent ecosystems

## Limitations

- Malicious samples are synthetically injected, not collected from real-world attacks
- Binary attack samples contain dummy ELF files, not actual malicious executables
- All URLs use reserved domains; real attacks would use convincing typosquatted domains
- The dataset focuses on Claude Code skills; generalization to other agent frameworks is not validated
- Benign samples may contain unintentional security issues not captured by the `benign` label

## Ethical Considerations

This dataset is intended solely for **defensive security research**. The malicious payloads are non-functional (all target `example.com`) and designed to advance detection capabilities, not to enable attacks. Researchers should handle malicious samples responsibly and not repurpose the attack patterns for offensive use.

## Citation

```bibtex
@dataset{skillbench1650,
title={SkillBench-1650: A Benchmark for AI Agent Skill Safety Detection},
author={Xinze Chen},
year={2026},
url={https://huggingface.co/datasets/edwardchen/SkillBench-1650},
note={Benign samples sourced from FayeZC/AgentSkills-134k}
}
```

## License

CC-BY-4.0 — free to use, share, and adapt with attribution.