Industrial Forecasting with Exponentially Smoothed Recurrent Neural Networks

Time series modeling has entered an era of unprecedented growth in the size and complexity of data which require new modeling approaches. While many new general purpose machine learning approaches have emerged (Bayer, 2015; Graves, 2013; Pascanu et al., 2012), they remain poorly understand and irreconcilable with more traditional statistical modeling approaches (Box and Jenkins, 1976; Hamilton, 1994). We present a general class of exponential smoothed recurrent neural networks (RNNs) which are well suited to modeling non-stationary dynamical systems arising in industrial applications. In particular, we analyze their capacity to characterize the non-linear partial autocorrelation structure of time series and directly capture dynamic effects such as seasonality and trends. Application of exponentially smoothed RNNs to forecasting electricity load, weather data, and stock prices highlight the efficacy of exponential smoothing of the hidden state for multi-step time series forecasting. The results also suggest that popular, but more complicated neural network architectures originally designed for speech processing are likely over-engineered for industrial forecasting and light-weight exponentially smoothed architectures, trained in a fraction of the time, capture the salient features while being superior and more robust than simple RNNs and autoregressive models. Additionally uncertainty quantification of Bayesian exponential smoothed RNNs is shown to provide improved coverage. Supplementary materials are available online.