Model Generation from Requirements with LLMs: an Exploratory Study - Replication Package

This is a replication package for the paper "Model Generation from Requirements with LLMs: an Exploratory Study", by Sallam Abualhaija, Chetan Arora, and Alessio Ferrari. Abstract: Complementing natural language (NL) requirements with graphical models can improve stakeholders’ communication and provide directions for system design. However, creating models from requirements involves manual effort. The advent of generative large language models (LLMs), ChatGPT being a notable example, offers promising avenues for automated assistance in model generation. This paper investigates the reliability of ChatGPT in generating sequence diagrams from NL requirements. Specifically, we conduct a qualitative study examining the sequence diagrams generated by ChatGPT for 28 requirements documents of various types and from different domains. Our study aims to uncover potential issues that emerge in the models generated by ChatGPT, thereby hindering its applicability in practice. Observations have systematically been captured through evaluation logs, and categorized through thematic analysis. Our results indicate that, although the models generally conform to the standard and exhibit a reasonable level of understandability, their correctness with respect to the specified requirements often presents challenges. This issue is particularly pronounced in the presence of requirements smells, such as ambiguity and inconsistency. The insights derived from this study can influence the practical utilization of LLMs in the RE process, and open the door to novel RE-specific prompting strategies targeting effective model generation. The replication package consists of the following folders: logs: includes the evaluation logs produced by each evaluator original-documents: includes the original requirements documents used for the evaluation RQ1 - quantitative analysis: includes the analysis made on the scores given to each model and model variant. It includes five files: - results.csv: numerical results of the evaluation for each criterion- analysis-results.Rmd: R file used to perform the quantitative analysis (requires R Studio to be executed)- analysis-results.html: html file produced by analysis-results.Rmd- cross-check.csv: file with the cross-checking of the two assessors applied to a subset of the models- symmary_results.xlsx: final output of the quantitative results in terms of Wilcoxon signed rank tests RQ2 - thematic analysis: includes the codebook produced by the thematic analysis of the issues in generating models with ChatGPT本数据集为Sallam Abualhaija、Chetan Arora与Alessio Ferrari合著的论文《基于大语言模型（Large Language Model，LLM）的需求建模生成：一项探索性研究》的复现包。 摘要：使用图形模型补充自然语言（Natural Language，以下简称NL）需求，可提升涉众沟通效率，并为系统设计提供指引。然而，从需求中构建模型需投入大量手动工作量。生成式大语言模型（Large Language Models，LLMs，其中ChatGPT为典型代表）为建模生成的自动化辅助提供了极具前景的路径。本文探究了ChatGPT从NL需求生成序列图（sequence diagrams）的可靠性。具体而言，本研究针对28份不同类型、不同领域的需求文档开展质性研究，对ChatGPT生成的序列图进行分析。本研究旨在揭示ChatGPT生成的模型中存在的潜在问题，进而明确其在实际应用中的局限性。研究人员通过评估日志系统性记录观测结果，并采用主题分析法对其进行分类。研究结果显示，尽管生成的模型总体上符合规范，且具备合理的可理解性，但在与指定需求的匹配正确性方面往往难以保证。当需求存在歧义、不一致等需求异味（requirements smells）时，这类问题尤为突出。本研究得出的见解，可为大语言模型在需求工程（Requirements Engineering，以下简称RE）流程中的实际应用提供参考，并为面向高效建模生成的专属RE提示策略研发开辟新方向。 本复现包包含以下文件夹： - logs：包含每位评估人员生成的评估日志 - original-documents：包含本次评估所用的原始需求文档 - RQ1 - 定量分析：包含针对各模型及模型变体得分的分析内容，共包含5个文件： - results.csv：各评估准则的量化评估结果 - analysis-results.Rmd：用于执行定量分析的R脚本文件（需通过R Studio运行） - analysis-results.html：由analysis-results.Rmd生成的HTML分析报告 - cross-check.csv：针对部分模型开展两名评估人员交叉核对的结果文件 - symmary_results.xlsx：基于威尔科克森符号秩检验（Wilcoxon Signed-Rank Test）得出的定量结果最终输出文件 - RQ2 - 主题分析：包含针对使用ChatGPT生成模型时出现的问题开展主题分析所形成的编码手册。