Quality Evaluation of Software Functional Requirements Generated by LLMs: A Systematic Mapping Study

Abstract. Research Context: Large Language Models (LLMs) have beenincreasingly applied in software engineering, especially in the generation offunctional requirements for information systems. Scientific and/or PracticalProblem: However, the quality of these requirements still raises concerns,such as ambiguities, incompleteness, and dependency on prompts. ProposedSolution and/or Analysis: This study conducts a systematic mapping toanalyze how the literature has evaluated requirements generated by LLMs.Related IS Theory: The work is aligned with the sociotechnical perspective,considering information systems requirements as both technical and socialartifacts. Research Method: A systematic mapping was conducted, reviewing1,875 studies and selecting 51 primary studies published between 2020 and2025. Summary of Results: The findings indicate a diversity of evaluationpractices, combining traditional criteria and NLP metrics, with advantagesin automation and standardization, but limitations regarding the absence ofbenchmarks and validation in industrial contexts, especially in the developmentof information systems. Contributions and Impact to IS area: The study con-tributes to academia by consolidating evaluation approaches and identifyinggaps, and to industry by supporting the understanding of risks and opportu-nities in the use of LLMs in the requirements engineering of information systems.摘要。研究背景：大语言模型（Large Language Models，LLMs）在软件工程领域的应用日益广泛，尤其常用于生成信息系统的功能需求。科学与/或实际问题：然而这类模型生成的需求质量仍存在诸多值得关注的问题，例如表述模糊、内容不完整，且高度依赖提示词（prompt）。拟提出的解决方案与/或分析：本研究开展系统映射研究（systematic mapping），分析现有文献对大语言模型生成的需求的评估方式。相关信息系统理论：本研究契合社会技术视角，将信息系统需求视为兼具技术与社会属性的人工制品。研究方法：本研究开展系统映射研究，共检索1875项相关研究，筛选出2020年至2025年间发表的51项核心研究。结果总结：研究结果显示，当前评估实践呈现多样化特征，融合了传统评估标准与自然语言处理（Natural Language Processing，NLP）指标，在自动化与标准化层面具备优势，但也存在缺乏基准测试集、未在工业场景（尤其是信息系统开发场景）中开展验证等局限。对信息系统领域的贡献与影响：本研究一方面通过整合现有评估方法并明确研究缺口，为学术界提供了参考；另一方面助力工业界理解在信息系统需求工程中应用大语言模型的风险与机遇。