Kale: A System for Enabling Human-in-the-loop Interactivity in HPC Workflows

Scientific problem-solving
frequently requires interactive, iterative exploration and analysis.
Web-based interactive electronic notebook interfaces such as Jupyter
offer an important mechanism for scientists to capture analyses in a
reproducible narrative context. An increasing number of science gateway
environments are providing support for Jupyter Notebooks as a means to enable
custom, ad-hoc analyses on scientific data. However, Jupyter Notebooks alone
are not enough to fulfill the needs of scientific researchers today.
Scientists are producing and consuming large amounts of data, and require
significant computational resources to process and analyze that data, causing
scientific workflows to become increasingly asynchronous in nature as
processing is off-loaded to remote resources. Many scientific researchers turn
to HPC systems for processing, but the traditional asynchronous batch-queue
environment used in HPC for such computationally intensive tasks is largely
separate from interactive Notebook-based workflows, producing a fragmented
workflow for scientists that does not facilitate rapid scientific inquiry. We
introduce our system “Kale” that enables Jupyter Notebooks to seamlessly
interface with HPC workflows, leveraging distributed computational resources for
iterative human-in-the-loop scientific exploration.