Empirical assessment of analysis workflows for differential expression analysis using RNA-Seq

Background: RNA-Seq is supplanting microarrays as the preferred method of transcriptome-wide identification of differentially expressed genes. However, RNA-Seq analysis is still rapidly evolving, with a large number of tools available for each of the three major processing steps: read alignment, expression modeling, and statistical determination of differentially expressed genes. Although some studies have benchmarked these tools against gold standard gene expression sets, few have evaluated their performance in concert with one another. Additionally, there is a general lack of testing of such tools on real-world, biologically relevant datasets, which often possess qualities not reflected in tightly controlled reference RNA samples or synthetic datasetsResults: Here we evaluate ten combinatorial implementations of several of the most commonly used analysis tools (RSEM, TopHat2, STAR, htseq, Cufflinks, DESeq2, edgeR, EBseq, and Cuffdiff) for their impact on differential gene expression analysis by RNA-Seq. A test dataset was generated from highly purified human classical and nonclassical monocyte subsets from a clinical cohort, allowing us to evaluate analysis workflow performance using four previously published microarray and BeadChip analyses of the same cell populations as reference datasets. We find that the choice of methodologies leads to wide variation in number of genes called significant, as well as precision and recall. In general, recall is correlated with the number of significant genes identified, whereas precision is inversely correlated with both recall and the number of significant genes identified. Additionally, we report that the choice of statistical analysis approach and read aligner exhibited stronger impacts on recall, precision, and F1 score than the choice of software for expression modeling.Conclusions: There is wide variation in the performance of RNA-Seq workflows to identify differentially expressed genes. Different workflows lead to a precision/recall tradeoff, and the ultimate choice of workflow should take into consideration how the results will be used in subsequent applications. Our analyses highlight the performance characteristics of these workflows, and the data generated in this study could also serve as a useful resource for future development of software for RNA-Seq analysis.