An Empirical Study of Prompt Engineering in LLMs for User Story Generation

Large Language Models (LLMs) have been increasingly used to support Requirements Engineering (RE) activities, particularly for generating user stories. However, there are still uncertainties about how these models behave in this process when different LLMs are applied under the same prompting instructions for elicitation, especially regarding the level of consistency in the generated outputs (aiming to minimize hallucination) and its relationship with the quality perceived by specialists, thus creating an exploratory margin to assess the potential of these models as support for agile practices, where predictability and artifact quality are critical factors. In this context, this work analyzes the semantic consistency of user stories generated by three LLMs (ChatGPT 5 Instant, DeepSeek, and Gemini 2.5 Pro) across four domains, following the 3Cs structure for user story formulation. Textual consistency—also treated as semantic consistency throughout the study—was automatically evaluated using text embeddings and cosine similarity between stories generated in different executions of the same protocol. In parallel, a controlled subset of the artifacts was selected for qualitative evaluation across two domains, assessed by two Software Engineering specialists using criteria inspired by the Quality User Story (QUS) framework. The results indicate moderate to high levels of semantic consistency (approximately ranging from 0.59 to 0.73 in average similarity), with variations across models and domains. Overall, greater stability was observed in the outputs generated by ChatGPT and DeepSeek, while Gemini showed higher variation across executions, reflecting lower semantic uniformity. In the qualitative evaluation, the stories were considered suitable as a starting point but had recurring weaknesses, particularly in the criteria of atomicity and testability. Finally, the results show that semantic similarity across generations and the quality perceived by specialists do not present a direct relationship, indicating that these dimensions capture distinct aspects of model behavior and reinforcing the importance of considering multiple perspectives when evaluating the use of LLMs in RE.大语言模型（Large Language Models，LLMs）正日益被用于支撑需求工程（Requirements Engineering，RE）活动，尤其在用户故事生成方面。然而，当在相同的提示指令下应用不同大语言模型开展需求获取时，此类模型在该流程中的表现仍存在诸多不确定性，尤其是在生成输出的一致性水平（旨在最大限度降低幻觉现象）及其与专家感知质量之间的关联层面，这为评估这类模型作为敏捷实践支撑工具的潜力留出了探索空间——毕竟在敏捷实践中，可预测性与工件质量均为核心要素。 在此背景下，本研究针对三款大语言模型（ChatGPT 5 Instant、DeepSeek及Gemini 2.5 Pro）在四个领域中生成的用户故事展开语义一致性分析，所有生成流程均遵循用户故事编写的3C结构。本研究中将文本一致性亦视作语义一致性，通过文本嵌入与同一协议多次执行所生成故事间的余弦相似度，对其开展自动化评估。 与此同时，研究选取了受控的工件子集，针对两个领域开展定性评估：由两名软件工程专家依据源自优质用户故事（Quality User Story，QUS）框架的评估准则完成测评。 研究结果显示，生成内容的语义一致性处于中等至较高水平，平均相似度大致介于0.59至0.73之间，且不同模型与领域间存在显著差异。总体而言，ChatGPT与DeepSeek生成的输出稳定性更强，而Gemini在多次执行中的结果波动更为明显，体现出更低的语义统一性。 在定性评估环节，生成的用户故事虽可作为开发起点，但仍存在多处共性短板，尤其是在原子性与可测试性两项评估准则方面。 最终研究结果表明，不同轮次生成内容的语义相似度与专家感知的质量之间并无直接关联，这说明这两个维度分别捕捉了模型行为的不同侧面，同时也印证了在评估大语言模型在需求工程中的应用时，需兼顾多维度视角的重要性。