Demo: Co-Scheduling HPC and BigData jobs using Apache Mesos

Scientific computing, by many measures, is becoming more complicated and heterogeneous. At the application level, cloud-native and data-intensive computing based on MapReduce and its descendants require computing resources that go beyond the traditional batch queuing model. Computing resources themselves have become more heterogeneous as hybrid architectures with graphical processing units (GPUs) and coprocessors (Xeon PHI) are providing alternatives to CPU based parallelizing techniques for traditional High-Performance Computing (HPC). Science gateways have traditionally provided simplifying interfaces for end users that hide the complications of using complicated resources. Thus the core reason for science gateways to exist is becoming more important, while the challenges for building science gateway middleware increases. In this demonstration, we showcase developments in the Apache Airavata science gateway framework that address these challenges by leveraging the Apache Mesos ecosystem to schedule both HPC and BigData jobs; we use XSEDE’s next generation JetStream, Comet, Bridges and Wrangler clusters. Each of these clusters provide unique capabilities; this demonstration will highlight how gateways can take advantage of these capabilities through a unified approach to middleware.