Replication Package: Writing Better Software Explanations: A Guideline-Based Approach

Description This replication package accompanies the study Writing Better Software Explanations: A Guideline-Based Approach, accepted at the 2026 IEEE 34th International Requirements Engineering Conference (RE). The study investigates how high-quality software explanations can be systematically supported through a guideline-driven approach and how such a guideline can be operationalized in a Large Language Model (LLM)-based formulation assistant. The research follows a sequential mixed-method design comprising (i) elicitation of candidate explanation quality properties from literature and user interviews, (ii) prioritization through a quantitative survey, (iii) derivation of a formulation guideline, and (iv) operationalization and evaluation of the guideline in a web-based LLM-supported tool. The tool supports developers in generating, checking, and iteratively revising explanations while maintaining human control over content. In this package, an explanation need denotes a user-reported question, uncertainty, or problem related to software behavior. Explanation content denotes factual grounding information provided to the developer, such as known causes, constraints, or possible solutions. The resulting explanation is a user-facing natural-language response formulated from this need and grounding content. The package includes: README.md: Detailed overview of the artifact, including an annotated folder structure, file descriptions, DOI information, system requirements, installation instructions, usage examples, expected tool behavior, troubleshooting notes, reproduction steps, expected runtime, reuse instructions, privacy notes, license notes, and contact information. data/survey-1-property-prioritization-results.xlsx: Anonymized dataset of the guideline prioritization survey containing the survey instrument, participant demographics, software experience information, prior experience with software-related explanation needs, item-level usefulness ratings for candidate explanation quality properties, and optional free-text comments. data/developer-study-manual-vs-tool-explanations.xlsx: Developer study dataset including explanation needs, app metadata, factual grounding content, participant identifiers for the manual and tool-supported conditions, recorded formulation times, paired manual and tool-supported explanations, and aggregated user-study rating information for the evaluated explanation pairs. data/survey-2-user-perceived-quality-results.xlsx: Anonymized online user evaluation dataset comparing perceived quality of manual and tool-supported explanations, including the survey instrument, participant demographics, software familiarity information, paired satisfaction ratings, optional free-text comments, confidence ratings, and perceived importance ratings. analysis/: Analysis script, expected output, and cleaned CSV input files for reproducing the main reported quantitative results and inspecting property-usefulness ratings. The reproduced results include formulation-time statistics, Wilcoxon tests, a log-time mixed-effects model, perceived-satisfaction medians and interquartile ranges (IQRs), preference counts derived from paired ratings, and participant-level and item-level summaries. source-code/: Full implementation of the web-based guideline-driven formulation tool, including prompt design, interface logic, guideline configuration, quality-check mechanisms, quick-fix functionality, Flesch readability computation, and OpenAI API integration. The source-code README documents local setup, optional Docker-based setup, API-key configuration, expected UI behavior, and the tool behavior when no API key is configured. prompt-templates.md: Summary of the generation, quality-check, and quick-fix prompts used by the tool. tool-interface-screenshot.png: Screenshot of the web-based formulation support tool. The quantitative analyses can be reproduced with the script in analysis/reproduce_main_results.py. The web-based tool can be inspected and run locally via the documented Node.js setup. An optional, locally tested Dockerfile is also provided for building and serving the client-side tool interface. A valid OpenAI API key is required only for LLM-based tool functions such as generation, rephrasing, quality checks, and quick fixes; the tool can still be installed, built, and opened without a key. This artifact supports research on software explainability, human-AI collaboration, requirements communication, and LLM-assisted writing support. It enables inspection and reuse of the datasets, reproduction of the main reported quantitative analyses, secondary analysis of explanation quality perceptions, and experimentation with guideline-driven LLM support approaches. Artifact DOI The recommended citation uses the Zenodo Concept DOI 10.5281/zenodo.20418966. This Concept DOI remains stable across artifact updates and resolves to the latest version of the replication package. Zenodo may additionally display version-specific DOIs for individual uploaded versions. Authors Martin Obaidi, Jean-Carl Kremser, Hannah Deters, Jakob Droste, Marc Herrmann, Kurt Schneider Citation If you use this replication package, the accompanying tool, the analysis scripts, or the prompt templates, please cite the accompanying paper: Obaidi, M., Kremser, J.-C., Deters, H., Droste, J., Herrmann, M., Schneider, K. (2026).Writing Better Software Explanations: A Guideline-Based Approach.In: 2026 IEEE 34th International Requirements Engineering Conference (RE). Please also cite this replication package: Obaidi, M., Kremser, J.-C., Deters, H., Droste, J., Herrmann, M., Schneider, K. (2026).Replication Package: Writing Better Software Explanations: A Guideline-Based Approach.Zenodo. DOI: 10.5281/zenodo.20418966. Contact Martin Obaidi ([e-mail]) License Unless otherwise noted, datasets, documentation, prompt templates, screenshots, expected outputs, and study materials are licensed under CC BY 4.0. The source code in source-code/ and the executable analysis scripts in analysis/ are licensed under the MIT License. The root LICENSE.md explains this split-license model, and the separate source-code/LICENSE.md file contains the full MIT License text for the tool source code. If Zenodo displays a single license in its metadata, the file-level license information in LICENSE.md is authoritative for this split-license replication package.