Beyond the Artifact: Analyzing Human-LLM Interaction in Attribute-Driven Design

This replication package contains the artifacts and experimental framework for an empirical study analyzing the collaboration between humans and Large Language Models (LLMs) in software architecture design. The study utilizes the Attribute-Driven Design (ADD) methodology applied to a distributed storage system (Google Drive) case study.

Repository and Data Structure

The artifacts are organized to support a longitudinal analysis of architectural design decisions across nine stages. The data collection includes:

LLM-generated Drafts: Initial architectural outputs produced by Gemini 3.0 based on structured prompts.

Expert Refinements: Definitive versions of the artifacts reviewed and corrected by humans to mitigate errors or hallucinations.

Golden Solution: A benchmark reference solution produced by a specialist to calculate semantic alignment and performance metrics.

Automated Acquisition and Exclusion Criteria

To ensure reproducibility while maintaining the anonymity required for double-blind peer review, this package includes a Repository Management script (Block 2).

Technical Implementation

The script automates the concurrent acquisition of 31 repositories from the anonymous.4open.science platform. It utilizes a direct ZIP API acquisition strategy to bypass server rate-limiting, ensuring the structural integrity of folders and subfolders during external evaluation.

Scope and Filtering

The dataset includes the reference solution and 30 distinct student-pair repositories (Groups 01–33). Three groups were excluded from the final analysis:

Groups 09 and 23: Excluded due to the absence of explicit consent for data usage in research. Participation was voluntary, and choosing "NO" had no impact on evaluation.

Group 12: Excluded because the required architectural artifacts were not delivered, rendering the data incomplete for longitudinal metrics.