Designing a single cell RNA sequencing benchmark dataset to compare protocols and analysis methods

Single cell RNA sequencing (scRNA-seq) technology has undergone rapid development in recent years and brings new challenges in data processing and analysis. This has led to an explosion of tailored analysis methods for scRNA-seq to address various biological questions. However, the current lack of gold-standard benchmarking datasets makes it difficult for researchers to evaluate the performance of the many methods available in a systematic manner. Here, we designed and generated a cross-platform benchmark dataset that has in-built truth in various forms and varying levels of biological noise. We used this dataset to compare different protocols and data analysis methods. We found that different protocols have different data quality and ERCC spike-in works independently to endogenous RNA. We found significant differences in the results from the methods compared and we associated the results with data characteristics to identify methods that perform well in different situations. Our dataset and analysis provide a valuable resource for algorithm selection in different biological settings. our experiment utilized the 5 human lung adenocarcinoma cell lines H2228, H1975, A549, H838 and HCC827. For the single cell designs, the five cell lines were mixed equally and processed by 10X chromium, referred to as sc_10X_5cl in analysis that follows.