Replication and Discussion Data: User study for Copilot for constraint-driven generation of architectural design evaluation and suggestions

Conducted a user study to test the developed copilot, which automates compliance checking and generates design suggestions based on the check results. The user study involved architects and BIM specialists performing different compliance checks with and without the copilot’s assistance to assess its performance, as well as participating in discussions regarding the interaction and workflow integration implemented for the copilot.

The raw data contains transcripts of the discussions held during the user study as text files, along with the data for each task conducted by the architects during the study. It includes:

Ground truth for the compliance checks to be performed (.JSON)
Evaluation results from the copilot (.JSON)
Evaluation results from the architects (.csv)
A Python script to compare the copilot’s and architects’ results to the ground truth

The processed data contains a report of the user study by architectural company as text data. It summarizes the most relevant findings from the discussions and contextualizes the architectural offices. It also includes an overview of the results obtained by the copilot and the architects for all tasks in the user study, presented as tabular data and plots.