Artifacts: A Multivocal Literature Review on the Benefits and Limitations of Automated Machine Learning Tools

Context. Rapid advancements in Artificial Intelligence (AI) and Machine Learning (ML) are revolutionizing software engineering in every application domain, driving unprecedented transformations and fostering innovation. However, despite these advances, several organizations are experiencing friction in the adoption of ML-based technologies, mainly due to the current shortage of ML professionals. In this context, Automated Machine Learning (AutoML) techniques have been presented as a promising solution to democratize ML adoption, even in the absence of specialized people.

Objective. Our research aims to provide an overview of the evidence on the benefits and limitations of using AutoML tools.

Method. We conducted a Multivocal Literature Review (MLR), which allowed us to identify 54 sources from the academic literature and 108 sources from the grey literature reporting on AutoML benefits and limitations. We extracted explicitly reported benefits and limitations from the papers and applied the thematic analysis method for synthesis.

Results. Overall, we identified 18 reported benefits and 25 limitations. Concerning the benefits, we highlight that AutoML tools can help streamline the core steps of ML workflows, namely data preparation, feature engineering, model construction, and hyperparameter tuning—with concrete benefits on model performance, efficiency, and scalability. In addition, AutoML empowers both novice and experienced data scientists, promoting ML accessibility. On the other hand, we highlight several limitations that may represent obstacles to the widespread adoption of AutoML. For instance, AutoML tools may introduce barriers to transparency and interoperability, exhibit limited flexibility for complex scenarios, and offer inconsistent coverage of the ML workflow.

Conclusions. The effectiveness of AutoML in facilitating the adoption of machine learning by users may vary depending on the specific tool and the context in which it is used. As of today, AutoML tools are used to increase human expertise rather than replace it, and, as such, they require skilled users.

研究背景：人工智能（AI）与机器学习（ML）的快速发展正全方位变革各应用领域的软件工程，带来前所未有的变革并推动创新。然而，尽管取得了这些进展，诸多组织在采用基于机器学习的技术时仍面临阻碍，主要原因在于当前机器学习专业人才短缺。在此背景下，自动化机器学习（AutoML）技术被视为推动机器学习应用普及的可行方案，即便在缺乏专业技术人员的场景中也能发挥作用。
研究目标：本研究旨在梳理有关自动化机器学习工具优势与局限性的相关研究证据。
研究方法：本研究采用多视角文献综述（MLR）方法，共检索到54篇学术文献以及108篇报告自动化机器学习工具优势与局限性的灰色文献。研究人员从文献中提取明确阐述的优势与局限性内容，并通过主题分析法完成综合归纳。
研究结果：本研究共梳理出18项已被报道的优势以及25项局限性。就优势而言，自动化机器学习工具可帮助优化机器学习工作流的核心环节，具体包括数据预处理、特征工程、模型构建以及超参数调优，在模型性能、运行效率与可扩展性方面均有切实增益。此外，自动化机器学习工具可赋能新手与资深数据科学家，提升机器学习技术的可及性。另一方面，本研究也发现多项可能阻碍自动化机器学习大规模应用的局限性：例如，自动化机器学习工具可能带来透明度与互操作性壁垒，在复杂场景下灵活性不足，且对机器学习工作流的覆盖范围并不统一。
研究结论：自动化机器学习在助力用户采用机器学习技术的效果，取决于具体工具以及使用场景。截至目前，自动化机器学习工具的作用在于提升人类专业能力而非替代人类，因此其使用仍需要具备专业技能的用户。