A call to interpret disagreement components during classification assessment DATA.xlsx

This foresight manuscript proposes several ideas concerning how to conduct insightful classification assessment. Authors should report disagreement components that relate to the research question, without anointing results as acceptable or good. This manuscript reviews the citations of the 2011 paper entitled ‘Death to Kappa: Birth of Quantity Disagreement and Allocation Disagreement for Accuracy Assessment’, which gave two recommendations: 1 do not use Kappa and 2 use disagreement components. We analyzed 200 articles that cited the Death to Kappa paper. A quarter of the articles followed both recommendations, another quarter followed only the first recommendation, another quarter followed only the second recommendation, and the last quarter followed neither recommendation. The attempt to replace Kappa with disagreement components has been partially effective, while Kappa continues to haunt several professions. We discuss misguided uses of Percent Correct and Kappa in Remote Sensing and Land Change Modeling. The concepts are general thus relate to additional fields. Authors frequently use arbitrary thresholds of metrics to claim that results are acceptable. However, the notion that results can be acceptable or not is inherently unscientific. Scientists must use a metric that addresses a clear research question in which the scientists have no vested interest in the results.