CIM-QA: A Benchmark Question-Answering Dataset for Agent Evaluation over the CIM Ontology and Time-Series Data

CIM-QA is a comprehensive, YAML-based evaluation benchmark designed to assess the performance of the CIMon agent in navigating and querying Common Information Model (CIM) ontologies. The dataset evaluates the agent's ability to translate natural language questions into complex graph traversals and time-series data retrievals. It is structured around distinct graph patterns, or templates, which range in structural complexity from 0 to 4 "hops" (property traversals). These templates require the resolution of various entity identifiers (e.g., name, rdfs:label, mRID, significantMrid, EIC code) as well as the application of literal constraints and filters (e.g., rated voltage, start dates). Dataset Composition: The dataset comprises two main components spanning varying levels of complexity: Knowledge Graph (KG) Dataset: 2,508 templates and 5,204 unique parameterised questions requiring 0 to 4 hops across the CIM knowledge graph. Time-Series Dataset: 6 templates and 60 questions that evaluate the agent's ability to fuse CIM model data with dynamic time-series observations retrieved from the Cognite Data Fusion platform. Technical Structure: Each YAML entry in the dataset operates as a structured evaluation template encompassing: Natural Language Questions: Up to five varied question formulations per structural pattern. Parameters & Filters: Defined graph paths, CIM types, and literal constraints required to successfully construct the query. Parameterised SPARQL Templates & Bindings: The underlying SPARQL query patterns and specific IRI mappings for each question. Expected Agent Trajectories: Ordered groups of reference steps (e.g., iri_discovery, sparql_query, Retrieve Time Series, Retrieve Datapoints) that define the expected tool calls and input arguments. This includes expected SPARQL result sets for deterministic accuracy evaluation of the CIMon agent. Dependencies & Base Models: This repository provides the evaluation dataset only. The underlying power system grid models over which the queries are executed are not included in this upload and must be obtained separately to run the evaluations: Nordic44: Available at https://github.com/statnett/Nordic44. Telemark-120: Available via the CIM4NoUtility project at https://github.com/3lbits/CIM4NoUtility.