ArtificialAnalysis/ITBench-AA

ITBench-AA数据集是Artificial Analysis发布的公共场景，源自IBM的ITBench基准测试，用于ITBench-AA排行榜。该数据集包含SRE（站点可靠性工程）子集（sre配置），每个数据行代表一个Kubernetes事件场景，并附有预期的贡献因素实体。在评估中，代理可以访问受影响集群的离线快照（包括警报、事件、跟踪和拓扑信息），必须识别导致故障的实体（如Deployment、Pod、ConfigMap等）。sre配置仅包含40个公开场景（标记为source_split: public），这些场景是从总59个场景中选出的，其余私有/保留场景未包含在内。数据集主要用于问题回答任务，涉及SRE、Kubernetes、根因分析、代理和IT运营等标签。The ITBench-AA dataset is a public scenario set released by Artificial Analysis, originating from IBM's ITBench benchmark suite, and is designed for the ITBench-AA leaderboard. This dataset includes the SRE (Site Reliability Engineering) subset (sre configuration), where each data row represents a Kubernetes event scenario paired with expected contributing factor entities. During evaluation, AI agents can access offline snapshots of the affected clusters, which include alerts, events, traces and topology information, and are required to identify the entities that caused the failure, such as Deployment, Pod, ConfigMap and other Kubernetes resources. The sre configuration only contains 40 public scenarios marked as source_split: public, which are selected from a total of 59 scenarios, while the remaining private or reserved scenarios are not included. This dataset is primarily used for question answering tasks, with associated tags including SRE, Kubernetes, Root Cause Analysis, AI Agent and IT Operations.