On the performance of three indices of agreement: an easy-to-use r-code for calculating the Willmott indices

ABSTRACT A key step for any modeling study is to compare model-produced estimates with observed/reliable data. The original index of agreement (also known as original Willmott index) has been widely used to measure how well model-produced estimates simulate observed data. However, in its original version such index may lead the user to erroneously select a predicting model. Therefore, this study compared the sensibility of the original index of agreement with its two newer versions (modified and refined) and provided an easy-to-use R-code capable of calculating these three indices. First, the sensibility of the indices was evaluated through Monte Carlo Experiments. These controlled simulations considered different sorts of errors (systematic, random and systematic + random) and errors magnitude. By using the R-code, we also carried out a case of study in which the indices are expected to indicate that th empirical Thornthwaite’s model produces poor estimates of daily reference evapotranspiration in respect to the standard method Penman-Monteith (FAO56). Our findings indicate that the original index of agreement may indeed erroneously select a predicting model performing poorly. Our results also indicate that the newer versions of this index overcome such problem, producing more rigorous evaluations. Although the refined Willmott index presents the broadest range of possible values, it does not inform the user if a predicting model overestimate or underestimate the simulated data, resulting in no extra information regarding those already provided by the modified version. None of the indices represents the error as linear functions of its magnitude in respect to the observed process.

摘要 任何建模研究的核心环节之一，均需将模型生成的估算结果与观测或可靠实测数据进行比对。原始一致性指数（original index of agreement，亦称原始威尔莫特指数（Willmott index））已被广泛应用于量化模型估算值对实测数据的模拟性能。然而，该指数的原始版本可能会误导使用者错误选择预测模型。为此，本研究对比了原始一致性指数与其两种新版本（修正版与精修版）的敏感性，并提供了一段易用的R代码，可实现这三类指数的计算。首先，本研究通过蒙特卡洛实验（Monte Carlo Experiments）评估了各指数的敏感性。该受控模拟考量了不同类型的误差（系统误差、随机误差以及系统与随机复合误差）及其量级。借助所开发的R代码，本研究还开展了一项案例研究：预期该案例的分析结果将表明，相较于标准方法彭曼-蒙特斯（FAO56）法，经验型桑斯威特（Thornthwaite）模型对日参考蒸散量的估算效果较差。本研究的结果显示，原始一致性指数确实可能错误地筛选出性能不佳的预测模型；而两种新版本指数则解决了上述问题，能够生成更为严谨的模型评估结果。尽管精修版威尔莫特指数的可能取值范围最广，但它无法告知使用者预测模型是高估还是低估了模拟数据，因此并未比修正版提供额外的有效信息。所有指数均未将误差表征为相对于观测过程的量级线性函数。