Included Primary Studies of "Metrics to Estimate Model Comprehension Quality: Insights from a Systematic Literature Review"

Conceptual models are an effective and unparalleled means to communicate complicated information with a broad variety of stakeholders in a short period of time. However, in practice, conceptual models often vary in clarity, employed features, communicated content, and overall quality. This potentially impacts model comprehension to a point where models are factually useless. To counter this, guidelines to create “good” conceptual models have been suggested. However, these guidelines are often abstract, hard to operationalize in different modeling languages, partly overlap, or even contradict one another. In addition, no comparative study of proposed guidelines exists so far. This is issue is exacerbated as no established metrics to measure or estimate model comprehension for a given conceptual model exist. In this paper, we present the results of a literature survey investigating 109 publications in the field and discuss metrics to measure model comprehension, their quantification, and their empirical substantiation. Results show that albeit several concrete quantifiable metrics and guidelines have been proposed, concrete evaluative recommendations are largely missing. Moreover, some suggested guidelines are contradictory, and few metrics exist that allow instantiating common frameworks for model quality in a specific way.

概念模型（Conceptual Models）是在短时间内与各类利益相关方高效沟通复杂信息的无可替代的手段。然而在实际应用中，各类概念模型在清晰度、采用的特征、传递内容以及整体质量上往往存在显著差异，这可能会损害模型理解度（Model Comprehension），极端情况下甚至会令概念模型完全失去实用价值。为应对这一问题，学界已提出多项用于构建“优质”概念模型的指南，但这些指南往往较为抽象，难以在不同建模语言中实现可操作化，部分内容存在重叠，甚至彼此矛盾。此外，截至目前尚无针对所提出的各类指南的对比研究。而由于尚未建立可针对特定概念模型衡量或评估其模型理解度的成熟指标体系，这一问题进一步加剧。本研究针对该领域的109篇学术文献开展了系统性文献综述，本文将呈现该综述的研究结果，并探讨用于衡量模型理解度的指标、其量化方法以及实证验证依据。研究结果表明，尽管已有多项具体可量化的指标与指南被提出，但目前仍缺乏切实可行的评估建议。此外，部分所提出的指南存在矛盾，且极少有指标能够以特定方式为模型质量的通用框架完成实例化。