List of documents and patterns identified by multivocal literature review of software engineering patterns for machine learning applications

We performed a multivocal literature review of both academic and gray literature to collect software engineering for machine learning (ML) application systems and software design. For the academic literature, we chose Engineering Village. For the gray literature, we used a Google search on August 16, 2019. We retrieved 32 scholarly documents and 48 gray literature documents. We vetted whether each document should be included in our review using the following criteria: Documents written in English addressing concrete software-engineering patterns or practices to design ML application systems and software should be included. Documents focusing on design of ML techniques and algorithms should be excluded. This process identified 19 scholarly documents and 19 gray documents. Although 69 patterns related to the design of ML application systems were initially identified, 33 remained after the vetting process. Finally, industrial ML developers reviewed the 33 candidates from the viewpoint of practical usefulness. They identified only 15 ML patterns.