One-Step Estimator Paths for Concave Regularization

The statistics literature of the past 15 years has established many favorable properties for sparse diminishing-bias regularization: techniques that can roughly be understood as providing estimation under penalty functions spanning the range of concavity between ℓ₀ and ℓ₁ norms. However, lasso ℓ₁-regularized estimation remains the standard tool for industrial Big Data applications because of its minimal computational cost and the presence of easy-to-apply rules for penalty selection. In response, this article proposes a simple new algorithm framework that requires no more computation than a lasso path: the path of one-step estimators (POSE) does ℓ₁ penalized regression estimation on a grid of decreasing penalties, but adapts coefficient-specific weights to decrease as a function of the coefficient estimated in the previous path step. This provides sparse diminishing-bias regularization at no extra cost over the fastest lasso algorithms. Moreover, our gamma lasso implementation of POSE is accompanied by a reliable heuristic for the fit degrees of freedom, so that standard information criteria can be applied in penalty selection. We also provide novel results on the distance between weighted-ℓ₁ and ℓ₀ penalized predictors; this allows us to build intuition about POSE and other diminishing-bias regularization schemes. The methods and results are illustrated in extensive simulations and in application of logistic regression to evaluating the performance of hockey players. Supplementary materials for this article are available online.

过去15年的统计学文献已确立了稀疏减偏正则化（sparse diminishing-bias regularization）的诸多优良性质——这类技术大致可理解为：在介于L₀范数与L₁范数之间的凹性惩罚函数族下开展估计。然而，套索（Lasso）L₁正则化估计因计算成本极低且存在易于应用的惩罚选择规则，仍是工业界大数据应用的标准工具。为此，本文提出了一种计算复杂度不高于套索路径的简易新型算法框架：一步估计量路径（POSE）在递减惩罚网格上执行L₁惩罚回归估计，并针对各系数赋予随前一路径步骤中估计得到的系数而递减的专属权重。该框架可在不增加最快套索算法额外计算成本的前提下实现稀疏减偏正则化。此外，本文提出的POSE套索实现版本附带了针对拟合自由度的可靠启发式方法，使得标准信息准则可直接用于惩罚项选择。本文还推导了加权L₁与L₀惩罚预测器之间距离的新颖结论，借此可增进对POSE及其他减偏正则化方案的理解。文中方法与研究结果通过大量仿真实验，以及将逻辑回归（logistic regression）应用于评估冰球运动员表现的实例得到了验证。本文的补充材料可在线获取。