Appendix for "Is JavaScript Call Graph Extraction Solved Yet? A Comparative Study of Static and Dynamic Tools"

Abstract

The popularity and wide adoption of JavaScript both at the client and server-side makes its code analysis more essential than ever before. Most of the algorithms for vulnerability analysis, coding issue detection, or type inference rely on the call graph representation of the underlying program. Luckily, there are quite a few tools to get this job done already. However, their performance in vitro and especially in vivo has not yet been extensively compared and evaluated.

In this paper, we systematically compare five static and two dynamic approaches for building JavaScript call graphs on 26 WebKit SunSpider benchmark programs and two static and two dynamic methods on 12 real-world Node.js modules. The tools under examination using static techniques were npm call graph, IBM WALA, Google Closure Compiler, Approximate Call Graph, and Type Analyzer for JavaScript. We performed dynamic analyzes relying on the nodejs-cg tool (a customized Node.js runtime) and the NodeProf instrumentation and profiling framework.

We provide a quantitative evaluation of the results, and a result quality analysis based on 941 manually validated call edges. On the SunSpider programs, which do not take any inputs, so dynamic extraction could be complete, all the static tools also performed well. For example, TAJS found 93% of all edges while having a 97% precision compared to the precise dynamic call graph. When it comes to real-world Node.js modules, our evaluation shows that static tools struggle with parsing the code and fail to detect a significant amount of call edges that dynamic approaches can capture. Nonetheless, a significant number of edges not detected by dynamic approaches are also reported. Among these, however, there are also edges that are real, but for some reason the unit tests did not execute the branches in which these calls were included.