Open Source Scientific Cloud Pipelines to Support Multiscale Hydrologic Modeling Studies

Computational hydrology and real world decision-making increasingly rely on simulation-based, multi-scenario analyses. Enabling scientists to align their research with national-scale efforts is necessary to facilitate knowledge transfer and sharing between operational applications and those focused on local or regional water issues. Leveraging existing large-domain datasets with new and innovative modeling practices is vital for improving operational prediction systems. The scale of these large-domain datasets presents significant challenges when applying them at smaller spatial scales, specifically data collection, pre-processing, post-processing, and reproducibly disseminating findings. Given these challenges, we propose a cloud-based data processing and modeling pipeline, leveraging existing open source tools and cloud technologies, to support common hydrologic data analysis and modeling procedures. Through this work we establish a scalable and flexible pattern for enabling efficient data processing and modeling in the cloud using workflows containing both publicly accessible and privately maintained cloud stores. By leveraging modern cloud computing technologies such as Kubernetes, Dask, Argo, and Analysis Ready Cloud Optimized data, we establish a computationally scalable solution that can be deployed for specific scientific studies, research projects, or communities. We present an approach for using large-domain meteorological and hydrologic modeling datasets for local and regional applications using the NOAA National Water Model, the NOAA NextGen Hydrological Modeling Framework, and Parflow. We discuss how this approach can be used to advance our collective understanding of hydrologic processes, creating reusable workflows, and operating on large-scale data in the cloud.