Nephele Big Data Experiment

The big data experiments use HiBench for benchmarking, which generates data using a number of data generators. These generators generate text data, web-data that follows a Zipfian distribution, numeric data for k-means clustering following a uniform or Gaussian distribution. Each of those benchmark instances consists of ten micro benchmarks. Some of those micro benchmarks provide multiple implementations. These differ in language of implementation and framework. The frameworks includes Hadoop and Spark. The Spark-based implementations feature code in Java, Python, and Scala. Below is a list of evaluated workloads and data: 1. Sort – This micro benchmark sorts text data, which is generated by RandomTextWriter. 2. WordCount – This micro-benchmark counts the occurrences of words in the input data, which is generated by RandomTextWriter. 3. TeraSort – This micro benchmark sorts a large data set generated by TeraGen. This micro benchmark has been used to push the boundaries of sorting very large data sets. 4. Sleep – This micro benchmark sleeps for a set amount of time to test the scheduler. 5. SQL (Scan, Join, Aggregate) – This benchmark tests typical SQL commands using Hive queries. The benchmark operates on generated hyperlinks. 6. PageRank – This benchmark computes the PageRank on generated hyperlinks. 7. Nutch indexing – Nutch is an Apache search engine and implements an indexing algorithm. The benchmark operates on generated web sites with words and hyperlinks. 8. Bayesian classification – This benchmarks uses Spark MLLib and Mahout to perform Naive Bayesian classification. The data consists of generated text documents. 9. K-means clustering – In this benchmark the k-means clustering algorithm in Spark MLLib/Mahout is used to cluster generated input data. 10. Enhanced DFSIO – This benchmark tests the HDFS throughput by generating a large number of tasks that simultaneously write and read data from HDFS.