A comparison of citation sources for reference and citations based search in systematic literature reviews

Context: In software engineering, snowball sampling has been used as a supplementary and primary search strategy. The current guidelines recommend the use of Google Scholar (GS) for snowball sampling. However, the use of GS presents several challenges when using it as a source for citations and references. Objective: To compare the effectiveness and usefulness of two leading citation databases (GS and Scopus) for use in snowball sampling search.Method: We relied on a published study that has used snowball sampling as a search strategy and GS as the citation source. We used its primary studies to compute precision and recall for Scopus. Results: In this particular case, Scopus was highly effective with 95% recall and had better precision of 5.1%compared to GS’s 2.8%. On average, one would read 15 extra papers in GS than Scopus to identify an additional relevant paper. Moreover, Scopus supports batch downloading of both citations to a paper and the papers’ references, has better quality metadata, and does better source filtering. Conclusion: This study suggests that Scopus seems to be more effective and useful for snowball sampling than GS for systematic secondary studies attempting to identify peer-reviewed literature.