Have We Overestimated the Value of ROC Analysis Under Varying Class Distributions?

We counsel caution in the application of ROC analysis for prediction of classifier accuracy under varying class distributions. We argue that it is only reasonable to expect ROC analysis to provide accurate error prediction under varying class distributions if the attributes are causally dependent upon the classes and the classes do not contain causally relevant subclasses whose frequency may vary at different rates.