jm1

jm1', the cleaned version by Shepperd et al., described as jm1' here.

jm1'', the cleaned version by Shepperd et al., described as jm1'' here.

Title/Topic: JM1/software defect prediction

Sources:

Creators: NASA, then the NASA Metrics Data Program

Contacts:

Mike Chapman, Galaxy Global Corporation ([e-mail]) [phone];

Pat Callis, NASA, NASA project manager for MDP ([e-mail]) [phone]

Past usage:

How Good is Your Blind Spot Sampling Policy?; 2003; Tim Menzies and Justin S. Di Stefano; 2004 IEEE Conference on High Assurance Software Engineering (http://menzies.us/pdf/03blind.pdf).

Results:

Very simple learners (ROCKY) perform as well in this domain as more sophisticated methods (e.g. J48, model trees, model trees) for predicting detects

Many learners have very low false alarm rates.

Probability of detection (PD) rises with effort and rarely rises above it.

High PDs are associated with high PFs (probability of failure)

PD, PF, effort can change significantly while accuracy remains essentially stable

With two notable exceptions, detectors learned from one data set (e.g. KC2) have nearly they same properties when applied to another (e.g. PC2, KC2). Exceptions:

LinesOfCode measures generate wider inter-data-set variances;

Precision's inter-data-set variances vary wildly

"Assessing Predictors of Software Defects", T. Menzies and J. DiStefano and A. Orrego and R. Chapman, 2004, Proceedings, workshop on Predictive Software Models, Chicago, Available from http://menzies.us/pdf/04psm.pdf

Results:

From JM1, Naive Bayes generated PDs of 25% with PF of 20%

Naive Bayes out-performs J48 for defect detection

When learning on more and more data, little improvement is seen after processing 300 examples.

PDs are much higher from data collected below the sub-sub-system level.

Accuracy is a surprisingly uninformative measure of success for a defect detector. Two detectors with the same accuracy can have widely varying PDs and PFs.

Relevant information:

JM1 is written in "C" and is a real-time predictive ground system: Uses simulations to generate predictions

Data comes from McCabe and Halstead features extractors of source code. These features were defined in the 70s in an attempt to objectively characterize code features that are associated with software quality. The nature of association is under dispute. Notes on McCabe can be found here and notes on Halstead can be found here. These metrics are widely used for defect prediction purposes, a quick tutorial on defect prediction is given here.

Number of instances: 10885

Number of attributes: 22 (5 different lines of code measure, 3 McCabe metrics, 4 base Halstead measures, 8 derived Halstead measures, a branch-count, and 1 goal field).

Attribute Information:

loc : numeric, McCabe's line count of code

v(g) : numeric, McCabe "cyclomatic complexity"

ev(g) : numeric, McCabe "essential complexity"

iv(g) : numeric, McCabe "design complexity"

n : numeric, Halstead total operators + operands

v : numeric, Halstead "volume"

l : numeric, Halstead "program length"

d : numeric, Halstead "difficulty"

i : numeric, Halstead "intelligence"

e : numeric, Halstead "effort"

b : numeric, Halstead

t : numeric, Halstead's time estimator

lOCode : numeric, Halstead's line count

lOComment : numeric, Halstead's count of lines of comments

lOBlank : numeric, Halstead's count of blank lines

lOCodeAndComment: numeric

uniq_Op : numeric, unique operators

uniq_Opnd : numeric, unique operands

total_Op : numeric, total operators

total_Opnd : numeric, total operands

branchCount : numeric, the flow graph

defects : {false,true}, module has/has not one or more reported defects

Missing attributes: none

Class Distribution: the class value (defects) is discrete

false: 2106 = 19.35%

true: 8779 = 80.65%

Comments on the Data

Martin Shepperd | August 19th, 2011 at 10:52 am I've realised this version of the dataset differs from the version available from the NASA MDP website. The following tabulates the differences for all 13 NASA datasets: {{{

Dataset n cases p features CM1 – NASA 505 40

CM1 – Promise 498 21

JM1 – NASA 10878 21

JM1 – Promise 10885 21

KC1 – NASA 2107 21

KC1 – Promise 2109 21

KC3 – NASA 458 40

KC3 – Promise 458 39

MC1 – NASA 9466 39

MC1 – Promise 9466 38

MC2 – NASA 161 40

MC2 – Promise 161 39

MW1 – NASA 403 40

MW1 – Promise 403 37

PC1 – NASA 1107 40

PC1 – Promise 1109 21

PC2 – NASA 5589 40

PC2 – Promise 5589 36

PC3 – NASA 1563 40

PC3 – Promise 1563 37

PC4 – NASA 1458 40

PC4 – Promise 1458 37

PC5- NASA 17186 39

PC5 – Promise 17186 38

KC4- NASA 125 40

KC2 – Promise 522 22 }}}

Therefore researchers are strongly advised to indicate which version of these files they are using!

Martin ShepperdAugust 19th, 2011 at 10:54 am: It is also puzzling how features such as loc can contain values like 1.1 (see feature 1, case 1).Martin ShepperdAugust 20th, 2011 at 7:03 am: Sorry small error — the two versions of this dataset have the following sizes:

{{{

Version, Cases(n), Features(p) JM1 – NASA, 10878, 24 JM1 – Promise, 10885, 22

}}}

Reference

M. Shepperd, Q. Song, Z. Sun and C. Mair, Data Quality: Some Comments on the NASA Software Defect Data Sets (under review), 2011