Sampling in Software Engineering Research Supplementary Material

Representative sampling appears rare in empirical software engineering research. Not all studies need representative samples, but a general lack of representative sampling undermines a scientific field. This article therefore reports a systematic review of the state of sampling in recent, high-quality software engineering research. The key findings are: (1) random sampling is rare; (2) sophisticated sampling strategies are very rare; (3) sampling, representativeness and randomness often appear misunderstood. These findings suggest that \textit{software engineering research has a generalizability crisis}. To address these problems, this paper synthesizes existing knowledge of sampling into a succinct primer and proposes extensive guidelines for improving the conduct, presentation and evaluation of sampling in software engineering research. It is further recommended that while researchers should strive for more representative samples, disparaging non-probability sampling is generally capricious and particularly misguided for predominately qualitative research.实证软件工程研究中，代表性抽样（representative sampling）的应用似乎较为罕见。并非所有研究均需代表性抽样，但整体上代表性抽样的匮乏会削弱整个科学领域的研究可信度。因此，本文对近期高质量软件工程研究中的抽样现状展开了系统性综述（systematic review）。核心研究结论如下：(1) 随机抽样极为罕见；(2) 复杂抽样策略更是屈指可数；(3) 抽样、代表性与随机性的概念常被误解。上述结论表明，软件工程研究面临可推广性危机。为解决上述问题，本文将现有抽样相关知识整合为一份简洁的入门指南，并提出了一系列完善软件工程研究中抽样的实施、呈现与评估的详细准则。此外，本文还建议：尽管研究者应尽力获取更具代表性的样本，但贬低非概率抽样（non-probability sampling）的做法通常是轻率的，尤其在以定性研究为主的场景中更是欠妥。