Supporting data for "Rapid development of cloud-native intelligent data pipelines for scientific data streams using the HASTE Toolkit"

This paper introduces the HASTE Toolkit, a proof-of-concept cloud-native software toolkit capable of partitioning data streams in order to prioritize usage of limited resources. This in turn enables more efficient data-intensive experiments. We propose a model that introduces automated, autonomous decision making in data pipelines, such that a stream of data can be partitioned into a tiered or ordered data hierarchy. Importantly, the partitioning is performed online and based on data content rather than a priori metadata. At the core of the model are interestingness functions and policies. Interestingness functions assign a quantitative measure of interestingness to a single data object in the stream, an interestingness score. Based on this score, a policy guides decisions on how to prioritize computational resource usage for a given object. The HASTE Toolkit is a collection of tools to adapt data stream processing to this pipeline model. The result is smart data pipelines capable of effective or even optimal use of e.g. storage, compute and network bandwidth, to support experiments involving rapid processing of scientific data characterized by large individual data object sizes. We demonstrate the proposed model and our toolkit through two microscopy imaging case studies, each with their own interestingness functions, policies, and data hierarchies. The first deals with a high content screening experiment, where images are analyzed in an on-premise container cloud with the goal of prioritizing the images for storage and subsequent computation. The second considers edge processing of images for upload into the public cloud for a real-time control loop for a transmission electron microscope.本研究介绍了HASTE工具包（HASTE Toolkit），这是一款概念验证型云原生（cloud-native）软件工具集，可对数据流（data streams）进行分区操作，以优先分配有限的计算资源，进而提升数据密集型实验的运行效率。我们提出了一种可在数据流水线（data pipelines）中实现自动化自主决策的模型，能够将数据流划分为分层或有序的数据层级结构。值得注意的是，该分区操作在线完成，且基于数据内容而非先验元数据（a priori metadata）。模型的核心为兴趣度函数（interestingness functions）与决策策略（policies）：兴趣度函数会为数据流中的单个数据对象赋予量化的兴趣度指标，即兴趣度评分；基于该评分，决策策略将指导如何为特定数据对象优先分配计算资源。HASTE工具包是一组可将数据流处理适配至该流水线模型的工具集合，最终可构建智能数据流水线，能够高效乃至最优地利用存储、计算与网络带宽等资源，以支持针对单个数据对象体量较大的科学数据开展快速处理的相关实验。我们通过两项显微成像案例研究验证了所提模型与工具包的有效性，两项案例分别拥有专属的兴趣度函数、决策策略与数据层级结构。第一项为高内涵筛选（high content screening）实验，该实验在本地容器云环境中对图像进行分析，目标是为图像的存储与后续计算分配优先级。第二项案例则聚焦于图像的边缘处理（edge processing），将处理后的图像上传至公有云（public cloud），为透射电子显微镜（transmission electron microscope）的实时控制回路（real-time control loop）提供支持。