Reproducibility and Replicability of Web Measurement Studies

Web measurement studies can shed light on not yet fully understood phenomena and thus are essential for analyzing how the modern Web works. This often requires building new and adjusting existing crawling setups, which has led to a wide variety of analysis tools for different (but related) aspects. If these efforts are not sufficiently documented, the reproducibility and replicability of the measurements may suffer---two properties that are crucial to sustainable research. In this paper, we survey 117 recent research papers to derive best practices for Web-based measurement studies and specify criteria that need to be met in practice. When applying these criteria to the surveyed papers, we find that the experimental setup and other aspects essential to reproducing and replicating results are often missing. We underline the criticality of this finding by performing a large-scale Web measurement study on4.5 million pages with 24 different measurement setups to demonstrate the influence of the individual criteria. Our experiments show that slight differences in the experimental setup directly affect the overall results and must be documented accurately and carefully.

网络测量研究（Web measurement studies）能够阐明尚未被完全认知的现象，因此对于解析现代互联网的运行机制至关重要。此类研究往往需要搭建全新的爬取配置并调整已有的爬取方案，由此衍生出针对各类（但相互关联的）网络测量维度的多样化分析工具。倘若这些工作未得到充分的文档记录，测量结果的可再现性（reproducibility）与可复制性（replicability）都将大打折扣——而这两项特性对于可持续开展科研工作而言至关重要。本文针对117篇近期发表的相关研究论文展开调研，旨在提炼网络测量研究的最佳实践规范，并明确实际开展研究时需满足的各项标准。将上述标准应用于本次调研的论文后，我们发现，实验配置以及其他对研究结果再现与复制不可或缺的关键环节，往往存在缺失情况。为凸显这一发现的重要性，我们针对450万个网页开展了一项大规模网络测量研究，共使用24种不同的测量配置，以此验证各项标准的实际影响。我们的实验结果表明，实验配置中的细微差异会直接影响整体研究结论，因此必须对其进行精准且细致的文档记录。