Data and scripts for "An Exploratory Study on Machine Learning Model Management"

AbstractEffective model management is crucial for ensuring performance and reliability in Machine Learning (ML) systems, given the dynamic nature of data and operational environments. However, standard practices are lacking, often resulting in ad hoc approaches. To address this, our research provides a clear definition of ML model management activities, processes, and techniques. Analyzing 227 ML repositories, we propose a taxonomy of 16 model management activities and identify 12 unique challenges. We highlight documentation and bug fixing as two of the most critical model management activities. Additionally, our findings indicate a significant shift towards automation of the ML pipeline, emphasizing the adoptions of tools for data, model, and documentation versioning. To offer practical guidance, we conducted a survey with industry practitioners and academic researchers to understand how model management challenges can be addressed. Our contributions include a detailed taxonomy of model management activities, a mapping of challenges to these activities, practitioner-informed solutions for challenge mitigation, and a publicly available dataset of model management activities and challenges. This work aims to equip ML developers with knowledge and best practices essential for the robust management of ML models.摘要：鉴于数据与运行环境的动态特性，高效的模型管理对于保障机器学习（Machine Learning，ML）系统的性能与可靠性至关重要。然而当前缺乏标准的实践流程，往往导致临时拼凑的解决方案。为解决这一问题，本研究明确界定了机器学习模型管理（Model Management）的活动、流程与技术。通过分析227个机器学习代码仓库（ML Repositories），本研究提出了包含16项模型管理活动的分类体系，并识别出12项独特的挑战。本研究指出，文档编制与缺陷修复是两项最为关键的模型管理活动。此外，研究结果显示，机器学习流水线（ML Pipeline）的自动化已成为重要发展趋势，业内对数据、模型与文档版本控制工具的采用显著提升。为提供实用的指导建议，本研究面向行业从业者与学术研究人员开展了调研，以探究模型管理挑战的应对路径。本研究的贡献包括：构建了详细的模型管理活动分类体系，建立了挑战与对应活动的映射关系，提出了基于从业者经验的挑战应对方案，以及公开了包含模型管理活动与挑战的数据集。本研究旨在帮助机器学习开发者掌握保障机器学习模型稳健运行所需的知识与最佳实践。