Best practices for evaluating IRFL approaches - Supplemental material

Information retrieval fault localization (IRFL) is a popular research field and many IRFL approaches have been proposed recently.Unfortunately, the evaluation of some of these IRFL approaches is often too simplistic, which can cause an overestimation of performance of these approaches.For this reason, we discuss evaluation pitfalls and problems in our paper "Best practices for evaluating IRFL approaches".To support our claim of the poor status quo of current evaluation practices in some research papers, we have performed a literature survey on 135 papers.This repository contains the data that we have extracted from these papers and our evaluation script.信息检索故障定位（Information Retrieval Fault Localization，IRFL）是当前主流的研究方向，近年来已有诸多IRFL方法被提出。遗憾的是，部分此类方法的评估流程往往过于简化，这可能导致对其性能的高估。为此，我们在论文《Best practices for evaluating IRFL approaches》中探讨了相关评估陷阱与现存问题。为佐证我们关于部分研究论文中当前评估实践现状欠佳的观点，我们针对135篇论文开展了系统性文献调研。本仓库包含我们从上述论文中提取的数据集以及我们的评估脚本。