Kale: A System for Enabling Human-in-the-loop Interactivity in HPC Workflows

Scientific problem-solving frequently requires interactive, iterative exploration and analysis. Web-based interactive electronic notebook interfaces such as Jupyter offer an important mechanism for scientists to capture analyses in a reproducible narrative context. An increasing number of science gateway environments are providing support for Jupyter Notebooks as a means to enable custom, ad-hoc analyses on scientific data. However, Jupyter Notebooks alone are not enough to fulfill the needs of scientific researchers today. Scientists are producing and consuming large amounts of data, and require significant computational resources to process and analyze that data, causing scientific workflows to become increasingly asynchronous in nature as processing is off-loaded to remote resources. Many scientific researchers turn to HPC systems for processing, but the traditional asynchronous batch-queue environment used in HPC for such computationally intensive tasks is largely separate from interactive Notebook-based workflows, producing a fragmented workflow for scientists that does not facilitate rapid scientific inquiry. We introduce our system “Kale” that enables Jupyter Notebooks to seamlessly interface with HPC workflows, leveraging distributed computational resources for iterative human-in-the-loop scientific exploration.

科学问题求解往往需要交互式、迭代式的探索与分析。诸如Jupyter笔记本（Jupyter Notebook）这类基于网页的交互式电子笔记本界面，为科学家提供了重要途径，使其可在可复现的叙述式语境中留存分析过程。越来越多的科学网关环境正提供对Jupyter笔记本的支持，以此实现针对科学数据的定制化、即席分析。然而，仅靠Jupyter笔记本已无法满足当代科学研究者的需求。科学家们正生成并使用海量数据，需要大量计算资源来处理与分析这些数据；随着计算任务被卸载至远程资源，科学工作流的本质正变得愈发异步化。许多科学研究者会借助高性能计算（HPC）系统完成计算任务，但HPC中用于处理这类计算密集型任务的传统异步批处理队列环境，大多与交互式笔记本类工作流相互割裂，这使得科学家的工作流呈现碎片化状态，无法助力快速开展科学探究。我们推出了名为"Kale"的系统，该系统可让Jupyter笔记本与HPC工作流实现无缝对接，借助分布式计算资源开展迭代式的人在回路科学探索。