Evaluating Big Data Frameworks To Simplify Distributed Task Execution In Apache Airavata

Apache Airavata powers users to run scientific applications on remote computing resources. Airavata integrates with science gateways to moderate computational jobs on diverse resources. Some of the job may require to execute multiple tasks. Newer technology trends such as Cloud Computing, BigData and Internet of Things are yielding open source software frameworks which are built for scale and reliability. In this paper we share our experiences in exploring the use of such frameworks and adapt for science gateway task management needs implemented by Apache Airavata.