High-Performance Kernel Machines With Implicit Distributed Optimization and Randomization

We propose a framework for massive-scale training of kernel-based statistical models, based on combining distributed convex optimization with randomization techniques. Our approach is based on a block-splitting variant of the alternating directions method of multipliers, carefully reconfigured to handle very large random feature matrices under memory constraints, while exploiting hybrid parallelism typically found in modern clusters of multicore machines. Our high-performance implementation supports a variety of statistical learning tasks by enabling several loss functions, regularization schemes, kernels, and layers of randomized approximations for both dense and sparse datasets, in an extensible framework. We evaluate our implementation on large-scale model construction tasks and provide a comparison against existing sequential and parallel libraries. Supplementary materials for this article are available online.

本文提出一种结合分布式凸优化与随机化技术的基于核的统计模型（kernel-based statistical models）大规模训练框架。所提方法基于分块拆分形式的交替方向乘子法（alternating directions method of multipliers），经精心重构以在内存约束下处理超大规模随机特征矩阵，同时利用现代多核机器集群中常见的混合并行性。本高性能实现依托可扩展框架，支持多种统计学习任务，可适配多种损失函数、正则化策略、核函数以及针对稠密与稀疏数据集的多层随机近似方法。我们针对超大规模模型构建任务对本实现进行了评估，并与现有串行及并行库开展了对比实验。本文的补充材料可在线获取。