Fast Robust Correlation for High-Dimensional Data

The product moment covariance matrix is a cornerstone of multivariate data analysis, from which one can derive correlations, principal components, Mahalanobis distances and many other results. Unfortunately, the product moment covariance and the corresponding Pearson correlation are very susceptible to outliers (anomalies) in the data. Several robust estimators of covariance matrices have been developed, but few are suitable for the ultrahigh-dimensional data that are becoming more prevalent nowadays. For that one needs methods whose computation scales well with the dimension, are guaranteed to yield a positive semidefinite matrix, and are sufficiently robust to outliers as well as sufficiently accurate in the statistical sense of low variability. We construct such methods using data transformations. The resulting approach is simple, fast, and widely applicable. We study its robustness by deriving influence functions and breakdown values, and computing the mean squared error on contaminated data. Using these results we select a method that performs well overall. This also allows us to construct a faster version of the DetectDeviatingCells method (Rousseeuw and Van den Bossche 2018) to detect cellwise outliers, which can deal with much higher dimensions. The approach is illustrated on genomic data with 12,600 variables and color video data with 920,000 dimensions. Supplementary materials for this article are available online.

积矩协方差矩阵（product moment covariance matrix）是多元数据分析的基石，借此可推导出相关系数、主成分、马氏距离（Mahalanobis distances）等诸多分析结果。遗憾的是，积矩协方差矩阵及其对应的皮尔逊相关系数（Pearson correlation）极易受到数据中异常值（离群点）的干扰。目前已开发出多种鲁棒协方差矩阵估计方法，但鲜有方法能够适用于如今愈发普及的超高维数据。对此，我们需要满足以下条件的方法：计算复杂度可随维度良好扩展，可保证生成半正定矩阵（positive semidefinite matrix），同时对异常值具备足够强的鲁棒性，且在低变异性的统计准则下具备足够高的精度。我们借助数据变换构建了满足上述要求的方法，所得到的方案简洁高效、适用范围广泛。我们通过推导影响函数（influence functions）与崩溃点（breakdown values），并在污染数据上计算均方误差（mean squared error），来分析该方法的鲁棒性。基于上述结果，我们筛选出了整体表现最优的方法。借此我们还构建了更快版本的离群细胞检测方法（DetectDeviatingCells，Rousseeuw与Van den Bossche，2018），该方法可检测逐单元格异常值，能够处理更高维度的数据。我们通过包含12600个变量的基因组数据与920000维彩色视频数据对所提方法进行了实例演示。本文的补充材料可在线获取。