Replication Package: Writing Better Software Explanations: A Guideline-Based Approach

Description This replication package accompanies the study Writing Better Software Explanations: A Guideline-Based Approach, accepted at the 2026 IEEE 34th International Requirements Engineering Conference (RE). The study investigates how high-quality software explanations can be systematically supported through a guideline-driven approach and how such a guideline can be operationalized in a Large Language Model (LLM)-based formulation assistant. The research follows a sequential mixed-method design comprising (i) elicitation of candidate explanation quality properties from literature and user interviews, (ii) prioritization through a quantitative survey, (iii) derivation of a formulation guideline, and (iv) operationalization and evaluation of the guideline in a web-based LLM-supported tool. The tool supports developers in generating, checking, and iteratively revising explanations while maintaining human control over content. The package includes: data/survey-1-property-prioritization-results.xlsx: Anonymized dataset of the guideline prioritization survey containing the survey instrument, participant demographics, software experience information, prior experience with software-related explanation needs, item-level usefulness ratings for candidate explanation quality properties, and optional free-text comments. data/developer-study-manual-vs-tool-explanations.xlsx: Developer study dataset including explanation needs, app metadata, factual grounding content, participant identifiers for the manual and tool-supported conditions, recorded formulation times, paired manual and tool-supported explanations, and aggregated user-study rating information for the evaluated explanation pairs. data/survey-2-user-perceived-quality-results.xlsx: Anonymized online user evaluation dataset comparing perceived quality of manual and tool-supported explanations, including the survey instrument, participant demographics, software familiarity information, paired satisfaction ratings, optional free-text comments, confidence ratings, and perceived importance ratings. analysis/: Analysis script, expected output, and cleaned CSV input files for reproducing the main reported quantitative results and inspecting property-usefulness ratings. The reproduced results include formulation-time statistics, Wilcoxon tests, a log-time mixed-effects model, perceived-satisfaction medians and interquartile ranges (IQRs), preference counts derived from paired ratings, and participant-level and item-level summaries. source-code/: Full implementation of the web-based guideline-driven formulation tool, including prompt design, interface logic, guideline configuration, quality-check mechanisms, quick-fix functionality, Flesch readability computation, and OpenAI API integration. The source-code README documents setup, API-key configuration, and the tool behavior when no API key is configured. prompt-templates.md: Summary of the generation, quality-check, and quick-fix prompts used by the tool. tool-interface-screenshot.png: Screenshot of the web-based formulation support tool. A detailed README is included with folder structure, file descriptions, system requirements, installation instructions, usage instructions, reproduction steps, expected runtime, reuse instructions, privacy notes, and contact information. This artifact supports research on software explainability, human-AI collaboration, requirements communication, and LLM-assisted writing support. It enables inspection and reuse of the datasets, reproduction of the main reported quantitative analyses, secondary analysis of explanation quality perceptions, and experimentation with guideline-driven LLM support approaches. Authors Martin Obaidi, Jean-Carl Kremser, Hannah Deters, Jakob Droste, Marc Herrmann, Kurt Schneider Citation If you use this replication package, the accompanying tool, the analysis scripts, or the prompt templates, please cite the accompanying paper: Obaidi, M., Kremser, J.-C., Deters, H., Droste, J., Herrmann, M., Schneider, K. (2026).Writing Better Software Explanations: A Guideline-Based Approach.In: 2026 IEEE 34th International Requirements Engineering Conference (RE). Please also cite this replication package: Obaidi, M., Kremser, J.-C., Deters, H., Droste, J., Herrmann, M., Schneider, K. (2026).Replication Package: Writing Better Software Explanations: A Guideline-Based Approach.Zenodo. DOI: 10.5281/zenodo.20418966. Contact Martin Obaidi ([e-mail]) License Unless otherwise noted, datasets, documentation, prompt templates, screenshots, expected outputs, and study materials are licensed under CC BY 4.0. The source code in source-code/ and the executable analysis scripts in analysis/ are licensed under the MIT License.