VuduVations/itsm-change-management-benchmark

---
dataset_info:
features:
- name: scenario_id
dtype: string
- name: category
dtype: string
- name: incidents
dtype: string
- name: cmdb_items
dtype: string
- name: risk_factors
dtype: string
- name: gold_standard_rfc
dtype: string
license: mit
task_categories:
- text-generation
- text2text-generation
language:
- en
tags:
- itil
- itsm
- change-management
- incident-management
- cmdb
- servicenow
- benchmark
- enterprise-ai
- rfc-generation
pretty_name: ITSM Change Management Benchmark
size_categories:
- n<1K
---

# ITSM Change Management Benchmark

The first public dataset for evaluating AI agents on IT Service Management (ITSM) tasks, specifically ITIL Change Management RFC generation.

## Dataset Description

This dataset contains structured ITSM data across three realistic enterprise scenarios, designed to benchmark AI agents that generate or evaluate Request for Change (RFC) documents against ITIL v4 standards.

### Scenarios

| Scenario | Category | Incidents | CMDB Items | Risk Factors |
|----------|----------|-----------|------------|--------------|
| Database Migration (PostgreSQL 16) | Infrastructure | 5 | 23 | 8 |
| Security Patch (Log4Shell) | Security | 5 | 20 | 10 |
| Cost Optimization (Auto-Scaling) | Infrastructure | 5 | 25 | 9 |

### Data Contents

**Incidents** — Structured incident records with:
- Unique ID, title, severity (P1-P4), category
- Detailed description with technical specifics
- Affected configuration items (CI references)
- Resolution details and MTTR (Mean Time To Repair)

**CMDB Items** — Configuration Management Database entries with:
- CI identifier, type classification, description
- Business criticality rating (Critical/High/Medium/Low)
- Infrastructure details (specifications, versions, counts)

**Scenario Metadata** — Context for each change scenario:
- Affected services, estimated cost, business value
- Risk factors with specific technical and organizational risks
- Rollback plans and testing status
- Timeline and deployment strategy

**Gold Standard RFCs** — Complete, multi-iteration RFC outputs showing:
- 6-dimension scoring (quality, compliance, risk, business value, technical readiness, stakeholder confidence)
- Executive summaries with CAB approval probability
- Critical issues identified per iteration
- Improvement recommendations with effort estimates
- Change category assessments

## Intended Use

### Benchmarking AI Agents
Evaluate whether an AI agent can:
1. Generate a complete, ITIL-compliant RFC from incident and CMDB data
2. Identify critical issues and risks
3. Iteratively improve the RFC based on feedback
4. Produce CAB-ready documentation

### Evaluation Metrics
Compare agent output against gold standard RFCs on:
- 6-dimension score correlation
- Critical issue identification (precision/recall)
- ITIL section completeness
- Iteration improvement rate

### Training Data
Use as few-shot examples or fine-tuning data for:
- ITSM document generation models
- RFC quality evaluation models
- Risk assessment classifiers

## Dataset Structure

```
data/
├── incidents.json # 15 incident records (5 per scenario)
├── cmdb.json # 68 CMDB items across 3 scenarios
└── scenarios.json # 3 scenario definitions with metadata
```

## Usage

```python
from datasets import load_dataset

dataset = load_dataset("VuduVations/itsm-change-management-benchmark")
```

Or load individual files:

```python
import json

with open("data/incidents.json") as f:
incidents = json.load(f)

# Get database migration incidents
db_incidents = incidents["db-migration"]
print(f"{len(db_incidents)} incidents")
print(db_incidents[0]["title"])
```

## Citation

```bibtex
@dataset{vuduvations2024itsm,
title={ITSM Change Management Benchmark},
author={Vuduvations},
year={2025},
url={https://huggingface.co/datasets/VuduVations/itsm-change-management-benchmark},
license={MIT}
}
```

## Related

- [ITIL Reflexion Agent](https://github.com/vuduvations/itil-reflexion-agent) — Open-source LangGraph agent that uses this dataset
- [Technical Paper](https://github.com/vuduvations/itil-reflexion-agent/blob/main/docs/technical-paper.pdf)
- [Vuduvations](https://vuduvations.io)

## License

MIT — free to use for research, evaluation, training, and commercial applications.

---
数据集信息:
特征:
- 名称: scenario_id，数据类型: 字符串
- 名称: category，数据类型: 字符串
- 名称: incidents，数据类型: 字符串
- 名称: cmdb_items，数据类型: 字符串
- 名称: risk_factors，数据类型: 字符串
- 名称: gold_standard_rfc，数据类型: 字符串
许可证: MIT
任务类别:
- 文本生成
- 文本到文本生成
语言:
- 英语
标签:
- ITIL
- ITSM
- 变更管理
- 事件管理
- CMDB
- ServiceNow
- 基准测试
- 企业人工智能
- RFC生成
美观名称: ITSM变更管理基准数据集
样本规模类别:
- 样本数少于1000
---

# ITSM变更管理基准数据集

首个用于评估人工智能体（AI Agent）执行信息技术服务管理（IT Service Management, ITSM）任务的公开数据集，专注于ITIL变更管理流程中的变更请求（Request for Change, RFC）生成任务。

## 数据集说明

本数据集涵盖三个贴合实际的企业级场景下的结构化ITSM数据，旨在为生成或评估符合ITIL第四版（ITIL v4）标准的变更请求（RFC）文档的AI智能体提供基准测试依据。

### 场景

| 场景 | 类别 | 事件数 | CMDB条目数 | 风险因素数 |
|----------|----------|-----------|------------|--------------|
| 数据库迁移（PostgreSQL 16） | 基础设施 | 5 | 23 | 8 |
| 安全补丁（Log4Shell漏洞） | 安全 | 5 | 20 | 10 |
| 成本优化（自动扩缩容） | 基础设施 | 5 | 25 | 9 |

### 数据内容

**事件（Incidents）** — 结构化事件记录，包含以下内容：
- 唯一标识符、标题、严重级别（P1-P4）、类别
- 包含技术细节的详细描述
- 受影响的配置项（Configuration Item, CI）引用
- 解决详情与平均修复时间（Mean Time To Repair, MTTR）

**CMDB条目（CMDB Items）** — 配置管理数据库（Configuration Management Database, CMDB）条目，包含：
- CI标识符、类型分类、描述
- 业务关键度评级（关键/高/中/低）
- 基础设施详情（规格、版本、数量）

**场景元数据** — 每个变更场景的上下文信息：
- 受影响服务、预估成本、业务价值
- 包含具体技术与组织风险的风险因素
- 回滚计划与测试状态
- 时间线与部署策略

**标准RFC（Gold Standard RFCs）** — 完整的多迭代RFC输出，包含：
- 六维评分（质量、合规性、风险、业务价值、技术就绪度、干系人认可度）
- 包含变更咨询委员会（Change Advisory Board, CAB）批准概率的执行摘要
- 每一轮迭代中识别出的关键问题
- 附带工作量估算的改进建议
- 变更类别评估

## 预期用途

### 基准测试AI智能体
评估AI智能体是否能够：
1. 基于事件与CMDB数据生成符合ITIL标准的完整RFC文档
2. 识别关键问题与风险
3. 根据反馈迭代优化RFC文档
4. 生成可提交CAB审批的文档

### 评估指标
从以下维度对比智能体输出与标准RFC：
- 六维评分相关性
- 关键问题识别精确率与召回率
- ITIL章节完整性
- 迭代改进速率

### 训练数据
可作为少样本示例或微调数据，用于：
- ITSM文档生成模型
- RFC质量评估模型
- 风险评估分类器

## 数据集结构

data/
├── incidents.json # 15条事件记录（每个场景5条）
├── cmdb.json # 3个场景共68条CMDB条目
└── scenarios.json # 3个带元数据的场景定义

## 使用方法

python
from datasets import load_dataset

dataset = load_dataset("VuduVations/itsm-change-management-benchmark")

或单独加载文件：

python
import json

with open("data/incidents.json") as f:
incidents = json.load(f)

# 获取数据库迁移场景的事件
db_incidents = incidents["db-migration"]
print(f"{len(db_incidents)} 条事件")
print(db_incidents[0]["title"])

## 引用格式

bibtex
@dataset{vuduvations2024itsm,
title={ITSM Change Management Benchmark},
author={Vuduvations},
year={2025},
url={https://huggingface.co/datasets/VuduVations/itsm-change-management-benchmark},
license={MIT}
}

## 相关资源

- [ITIL Reflexion Agent](https://github.com/vuduvations/itil-reflexion-agent) — 使用本数据集的开源LangGraph AI智能体
- [技术白皮书](https://github.com/vuduvations/itil-reflexion-agent/blob/main/docs/technical-paper.pdf)
- [Vuduvations](https://vuduvations.io)

## 许可证

MIT许可证——可免费用于研究、评估、训练与商业应用。