SKILLMINE Dataset

Overview SKILLMINE is the empirical dataset of AI agent skills. This dataset provides a comprehensive foundation for analyzing the current landscape of AI agent capabilities, ecosystem patterns, and cross-platform skill distribution. Data Sources The dataset covers five major distribution channels in the AI agent skills ecosystem, each with distinct platform characteristics. Data Collection Methodology Crawler Infrastructure We developed a modular crawler framework to accommodate technical interface differences across platforms: API-based platforms: For marketplaces providing open REST/GraphQL interfaces, we batch-retrieved skill metadata and download links via official APIs, strictly adhering to Rate-Limit headers to control request frequency. Web-only platforms: For platforms offering only web interfaces, we employed Headless Chrome (Puppeteer) for dynamic page rendering, combined with XPath and CSS selectors to extract structured metadata. User behavior simulation was used to trigger lazy-loaded content. Git-hosted repositories: For Git-based Skills (e.g., ClawHub), we obtained minimized snapshots via git clone --depth=1, significantly reducing bandwidth overhead while preserving essential metadata. Data Schema For each Skill entry, we downloaded the complete release package (typically ZIP format), containing: SKILL.md — skill definition and documentation scripts/ — executable scripts (optional) references/ — reference materials (optional) assets/ — resource files (optional) All raw packages are stored on a local server with directory indexing by platform name and skill ID. Metadata Fields To enable unified cross-platform analysis, we extracted the following lightweight metadata for each Skill: Metadata Fields Field Description _id Unique identifier for each Skill record name Skill name description Functional description author Skill author repo_url Original repository URL local_path Local ZIP package storage path last_updated Last update timestamp source_platform Source marketplace All metadata is stored in Redis for efficient retrieval and statistical analysis.