NIKA: Benchmarking AI Agents for Network Incident Diagnosis and Troubleshooting

NIKA: Benchmarking AI Agents for Network Incident Diagnosis and Troubleshooting

The NIKA benchmark releases a large-scale dataset of AI agents’ behaviors for network troubleshooting, comprising over 900 reasoning traces. The dataset is collected by applying three large language models (LLMs) (GPT-5, GPT-5-mini, and GPT-OSS:20B) to troubleshoot network incidents. A network incident is defined as the occurrence of one or more network issues under a specific traffic load within a given network scenario. The NIKA benchmark covers 54 distinct network issues across five network scenarios, including data center and campus networks. These network incidents are further categorized into five classes, as described below.

Issue category
# Issue
# Trace
Representative Issue
Key Signals

Link failures
6
113
Link flap
flap event logs; packet drops

End-host failures

10
180
Conflicting VPN memberships
Overlapping subnets; VPN servers unreachable

Network node errors 
8
96
Number of MPLS labels hit limit
Error logs; packet drops

Misconfigurations 
14
247
BGP ASN mismatch
BGP session fails; ASN mismatch detected

Resource contention
6
162
Microbursts on interface
Reduced throughput; queue buildup

Network under attack
10
108
Service DoS
Surge in HTTP connections; CPU/RAM usage spikes

Total
54
906
-
-

The dataset hierarchy is organized as follows: issue_category/issue_name/incident_id/*, where each incident corresponds to one complete troubleshooting trace.

conversation_diagnosis_agent.log records the full trajectory of the troubleshooting/diagnosis agent.

conversation_submission_agent.log records the trajectory of the agent responsible for submitting tasks to the framework.

ground_truth.json contains the ground-truth labels that the agent is expected to submit.

submission.json contains the actual outputs produced by the agent.

The evaluation and benchmarking framework developed to evaluate LLM-based agents and to generate this dataset is available at:

https://github.com/sands-lab/nika