Sample-wise Combined Missing Effect Model with Penalization

Modern high-dimensional statistical inference often faces the problem of missing data. In recent decades, many studies have focused on this topic and provided strategies including complete-sample analysis and imputation procedures. However, complete-sample analysis discards information of incomplete samples, while imputation procedures have accumulative errors from each single imputation. In this paper, we propose a new method, Sample-wise COmbined missing effect Model with penalization (SCOM), to deal with missing data occurring in predictors. Instead of imputing the predictors, SCOM estimates the combined effect caused by all missing data for each incomplete sample. SCOM makes full use of all available data. It is robust with respect to various missing mechanisms. Theoretical studies show the oracle inequality for the proposed estimator, and the consistency of variable selection and combined missing effect selection. Simulation studies and an application to the Residential Building Data also illustrate the effectiveness of the proposed SCOM.

现代高维统计推断常面临缺失数据问题。近数十年来，诸多研究围绕该主题展开，提出了完整样本分析（complete-sample analysis）与缺失值插补流程等解决方案。然而完整样本分析会舍弃不完整个体的信息，而插补流程则会因单次插补产生累积误差。本文提出一种带惩罚项的样本级联合缺失效应模型（Sample-wise COmbined missing effect Model with penalization，简称SCOM），用于处理预测变量中出现的缺失数据。SCOM并未对预测变量进行插补，而是针对每个不完整个体，估算其所有缺失数据带来的联合效应。该方法可充分利用所有可用数据，对各类缺失机制均具备鲁棒性。理论研究证明了所提估计量的神谕不等式（oracle inequality）性质，以及变量选择与联合缺失效应选择的一致性。仿真实验与住宅建筑数据集（Residential Building Data）的实际应用案例，同样验证了所提SCOM方法的有效性。