Build2SPARQL: A Large-Scale Text-to-SPARQL Benchmark Dataset for Building Knowledge Graph Querying

Build2SPARQL is a benchmark dataset of 6,136 executable SPARQL queries and 30,680 natural-language (NL) questions derived from 201 building knowledge graphs (KGs) in the Brick and ASHRAE Standard 223P schemas. It is designed to support research on text-to-SPARQL translation, retrieval-augmented building query systems, and knowledge graph completeness assessment for building automation systems. The central design principle of Build2SPARQL is KG-grounded correctness: SPARQL queries are produced entirely by graph-traversal code that mines structural patterns directly from each source building graph and validates every query by Python triple-walking before any NL generation occurs. Large language models (LLMs) are used solely to generate the NL questions downstream of a verified query. The pipeline mines six structurally distinct query patterns derived from the complete SPARQL operator space applicable to named-node KGs: linear property chains (5,009 queries, 81.6%), branching Y-shape queries (815, 13.3%), COUNT aggregations (120, 2.0%), attribute-filtered chains (100, 1.6%), UNION disjunctions (51, 0.8%), and OPTIONAL extensions (41, 0.7%). Each SPARQL query is paired with five natural-language questions spanning a vocabulary register spectrum — Q1 Technical (ontology class names and property URIs verbatim), Q2 Systems-engineering (engineering domain paraphrase), Q3 Operational (information need without schema terms), Q4 Instance-grounded (names a specific equipment instance from the building's validation results, e.g., VAV-1-01), and Q5 Plain English (no ontology terms, no equipment identifiers) — yielding 30,680 NL questions in total. Mean pairwise lexical self-similarity across the five registers is 0.170, confirming vocabulary diversity at scale. 79% of Q4 questions successfully embed a named equipment instance drawn from validation-walk results. The 201 source buildings span three corpora. 120 synthetic buildings are generated by BuildGraph (DOI: 10.5281/zenodo.20015123), covering 8 DOE commercial prototype archetypes across 3 ASHRAE 90.1 energy code vintages (pre-1980, 2004, 2013) with 5 instances per archetype-vintage combination. 45 real Mortar open testbed buildings (bldg1–bldg44 and smc) are anonymised North American commercial facilities in Brick schema format, available at https://brickschema.org/ttl/mortar/ (Fierro et al., ACM Transactions on Sensor Networks, 2019, DOI: 10.1145/3366375). 14 buildings modelled in ASHRAE Standard 223P are drawn from the Open223 community models repository (https://github.com/open223/models.open223.info), contributed by NIST (Heat Pump Laboratory, Intelligent Buildings Agents Laboratory), Lawrence Berkeley National Laboratory, the National Renewable Energy Laboratory, Pacific Northwest National Laboratory, and ASHRAE Guideline 36-2021 Appendix A system archetypes. The remaining 22 buildings are Brick schema canonical reference examples from https://github.com/BrickSchema/Brick, including metering hierarchies, chiller plant, controller, ASHRAE Guideline 36 Brick examples, and the Rice University and UC Berkeley Soda Hall buildings (Balaji et al., BuildSys 2016, DOI: 10.1145/2993422.2993577). Source building graph files (.ttl) are not redistributed in this deposit; links to all four sources are provided in README.md. This deposit contains one .json file per building (201 files), split.json (reference train/test split), and README.md (schema documentation and usage guide). Each .json file contains a list of query records, each with a sparql_query, pattern_type, retrieved_information, and five NL questions under questions, with fields question_number, question_text, and source. The corpus field on each building record identifies its provenance (buildgraph, mortar, or brick_reference), and pipeline_version is set to 3 for all paper-scope records. The dataset includes a reference train/test split (split.json) partitioned at the building level: 173 train buildings (5,268 queries) and 28 test buildings (868 queries), with all 15 BuildGraph hospital buildings held out in the test set as an out-of-distribution probe. A retrieval-augmented few-shot evaluation across three open-weight LLMs (Gemma 4 4B, Llama 3.1 8B, Qwen 2.5 32B) using this split shows that retrieved examples raise Exact Match F1 from 0.2–20% (zero-shot) to 56–65% (3-shot retrieved), with Row Matching F1 reaching 0.66 for Qwen 2.5 32B. More details can be found in the paper (TBD)