HEP-Frame: Improving the efficiency of pipelined data transformation & filtering for scientific analyses

Software to analyse very large sets of experimental data often relies on a pipeline of irregular computational tasks with decisions to remove irrelevant data from further processing. A user-centred framework was designed and deployed, HEP-Frame, which aids domain experts to develop applications for scientific data analyses and to monitor and control their efficient execution. The key feature of HEP-Frame is the performance portability of the code across different heterogeneous platforms, due to a novel adaptive multi-layer scheduler, seamlessly integrated into the tool, an approach not available in competing frameworks. The multi-layer scheduler transparently allocates parallel data/tasks across the available heterogeneous resources, dynamically balances threads among data input and computational tasks, adaptively reorders in run-time the parallel execution of the pipeline stages for each data stream, respecting data dependencies, and efficiently manages the execution of library functions in accelerators. Each layer implements a specific scheduling strategy: one balances the execution of the computational stages of the pipeline, distributing the execution of the stages of the same or different dataset elements among the available computing threads; another controls the order of the pipeline stages execution, so that most data is filtered out earlier and later stages execute the computationally heavy tasks; yet another adaptively balances the automatically created threads among data input and the computational tasks, taking into account the requirements of each application. Simulated data analyses from sensors in the ATLAS Experiment at CERN evaluated the scheduler efficiency, on dual multicore Xeon servers with and without accelerators, and on servers with the many-core Intel KNL. Experimental results show significant improved performance of these data analyses due to HEP-Frame features and the codes scaled well on multiple servers. Results also show the improved HEP-Frame scheduler performance over the key competitor, the HEFT list scheduler. The best overall performance improvement over a real fine tuned sequential data analysis was impressive in both homogeneous and heterogeneous multicore servers and in many-core servers: 81x faster in the homogeneous 24+24 core Skylake server, 86x faster in the heterogeneous 12+12 core Ivy Bridge server with the Kepler GPU, and 252x faster in the 64-core KNL server.用于分析超大规模实验数据集的软件，通常依赖于包含决策环节的非规整计算流水线，该环节需从后续处理流程中剔除无关数据。本研究设计并部署了面向用户的HEP-Frame框架，旨在帮助领域专家开发科学数据分析应用，并对其高效执行过程进行监控与管控。HEP-Frame的核心特性在于代码在各类异构平台上的性能可移植性，这得益于框架无缝集成的新型自适应多层调度器——目前同类竞品框架均未采用该方案。该多层调度器可在可用异构资源间透明分配并行数据与任务，动态平衡数据输入与计算任务间的线程资源，在运行时依据数据依赖关系，针对各数据流自适应调整计算流水线各阶段的并行执行顺序，并高效管理计算加速器中的库函数执行流程。该调度器的各层级分别实现专属调度策略：其一用于平衡计算流水线各阶段的执行，将同一份或不同数据集元素的阶段任务分配至可用计算线程；其二用于管控流水线阶段的执行顺序，确保大部分数据可在早期环节被过滤，将计算密集型任务留至后期阶段执行；其三则会结合各应用的具体需求，在数据输入与计算任务间自适应平衡自动创建的线程资源。研究依托欧洲核子研究中心（CERN）ATLAS探测器实验的传感器模拟数据分析任务，分别在搭载与未搭载计算加速器的双路多核至强（Xeon）服务器，以及搭载英特尔KNL众核处理器的服务器上，对该调度器的效率进行了验证。实验结果表明，依托HEP-Frame的特性，相关数据分析任务的性能得到显著提升，且代码在多台服务器上均可实现良好的扩展性。同时，相较于主流竞品HEFT列表调度器，HEP-Frame的调度器性能也得到优化。相较于经过精细调优的真实串行数据分析任务，HEP-Frame在各类平台上的整体性能提升效果十分显著：在配备24+24核心的Skylake至强同构服务器上提速81倍，在搭载开普勒GPU（Kepler GPU）的12+12核心Ivy Bridge异构服务器上提速86倍，在64核心的KNL众核服务器上提速252倍。