Supplementary Material for "ECSER: Evaluating Classifiers in Software Engineering Research"

This supplementary material for the article titled  "ECSER: Evaluating Classifiers in Software Engineering Research" includes

ECSER-ExploratoryStudy.csv: The annotated meta-data of the papers that have been published in ICSE between 2019 and 2021.

ECSER_ROCplots+StatTest.ipynb:  A python notebook that compares classifiers adn checks the statistical significance of the comparison results.

ECSER_SummaryOfReplicationSteps.pdf: This table presents a summary of ECSER steps for the two original studies and our applications on ECSER.

ECSER_RE: The directory that holds the datasets and code for the replication of Hay et al. [1] and additional runs on the new data sets.

ECSER_FF: The directory that holds the code and data for the replication of Alshammari et al. [2]

README.md presents the structure of the supplementary materials.

requirements.txt lists the dependencies needed to run the code

In the ECSER_RE directory, the code for multiple classifiers that are compared are kept in the "Classifiers" directory. The public data sets are shared in the Datasets directory. "ECSER_RE_Compare_Classifiers.ipynb" python notebook includes the code that runs each classifier. The results are presented in "ECSER_RE_results-Promise-vs-all.csv".

In the ECSER_FF directory, the data sets are presented directly under the main directory. The notebook "ECSER-FF-Compare_Classifiers.ipynb" compares the classifiers of the original study and the results are kept under the "ECSER_FF_results" directory.

Tobias Hey, Jan Keim, Anne Koziolek, and Walter F. Tichy. 2020. SupplementaryMaterial of "NoRBERT: Transfer Learning for Requirements Classification". https://doi.org/10.5281/zenodo.3874137

Abdulrahman Alshammari, Christopher Morris, Michael Hilton, and JonathanBell. 2021. FlakeFlagger: Predicting Flakiness Without Rerunning Tests. In43rdIEEE/ACM International Conference on Software Engineering, ICSE 2021, Madrid,Spain, 22-30 May 2021. IEEE, 1572–1584. https://doi.org/10.1109/ICSE43902.2021.00140