An Empirical Study of Prompt Engineering in LLMs for User Story Generation

Large Language Models (LLMs) have increasingly been used to support Requirements Engineering activities, particularly in the generation of user stories; however, questions remain regarding the stability of these outputs when the same elicitation process is repeated, as well as how such stability relates to expert-perceived quality. This paper investigates the semantic consistency of user stories generated by three LLMs (ChatGPT 5 Instant, DeepSeek, and Gemini 2.5 Pro) across four domains by repeatedly applying a protocol based on the 3Cs model. Consistency was measured using text embeddings and cosine similarity, focusing on comparable stories generated under the same identifiers across repetitions. Additionally, a controlled subset of the generated artifacts was evaluated by two Requirements Engineering specialists in two domains using criteria inspired by the Quality User Story (QUS) framework. The results show moderate to high semantic consistency, with variations across models and domains: ChatGPT and DeepSeek exhibited more predictable behavior, while Gemini demonstrated a more exploratory profile, generating a higher number of stories but with lower repeatability. Expert evaluation indicated that the generated stories are useful as an initial drafting mechanism, although recurring issues were identified in terms of atomicity and testability. Overall, the findings suggest that semantic consistency and perceived quality capture distinct aspects of LLM-generated artifacts, reinforcing the need to consider both as complementary dimensions in evaluation.