Industrial Forecasting with Exponentially Smoothed Recurrent Neural Networks

Time series modeling has entered an era of unprecedented growth in the size and complexity of data which require new modeling approaches. While many new general purpose machine learning approaches have emerged (Bayer, 2015; Graves, 2013; Pascanu et al., 2012), they remain poorly understand and irreconcilable with more traditional statistical modeling approaches (Box and Jenkins, 1976; Hamilton, 1994). We present a general class of exponential smoothed recurrent neural networks (RNNs) which are well suited to modeling non-stationary dynamical systems arising in industrial applications. In particular, we analyze their capacity to characterize the non-linear partial autocorrelation structure of time series and directly capture dynamic effects such as seasonality and trends. Application of exponentially smoothed RNNs to forecasting electricity load, weather data, and stock prices highlight the efficacy of exponential smoothing of the hidden state for multi-step time series forecasting. The results also suggest that popular, but more complicated neural network architectures originally designed for speech processing are likely over-engineered for industrial forecasting and light-weight exponentially smoothed architectures, trained in a fraction of the time, capture the salient features while being superior and more robust than simple RNNs and autoregressive models. Additionally uncertainty quantification of Bayesian exponential smoothed RNNs is shown to provide improved coverage. Supplementary materials are available online.

时序建模已然迈入数据规模与复杂度空前增长的时代，此类场景亟需全新的建模方法。尽管诸多新型通用机器学习方法相继问世（Bayer, 2015; Graves, 2013; Pascanu等人, 2012），但这类方法的理论内涵仍未得到充分阐释，且与经典统计建模范式（Box与Jenkins, 1976; Hamilton, 1994）难以兼容。本文提出一类通用的指数平滑循环神经网络（Recurrent Neural Networks, RNNs），其可出色适配工业场景中非平稳动态系统的建模任务。具体而言，本文分析了该架构对时序数据非线性偏自相关结构的刻画能力，以及直接捕捉季节性、趋势性等动态效应的特性。将指数平滑RNNs应用于电力负荷、气象数据与股价预测任务，验证了隐藏状态指数平滑在多步时序预测中的有效性。研究结果同时表明，原本为语音处理设计的主流复杂神经网络架构，用于工业预测任务时大概率存在过度工程化的问题；而仅需少量训练时间的轻量级指数平滑架构，不仅能够捕捉核心特征，还比简单循环神经网络与自回归模型更具优越性与鲁棒性。此外，贝叶斯指数平滑RNNs的不确定性量化方法可有效提升预测覆盖性能。本文补充材料可在线获取。