Accelerated Optimal Estimation: Meeting the surface reflectance retrieval performance and quality requirements of future remote imaging spectroscopy missions

Space-based imaging spectrometers designed in recent years aim to monitor Earth's surface for changes in geophysical properties. Most of these instruments will be launched in the coming decade, and all will produce very large volumes of data. Inference to estimate surface properties from radiances is most efficiently done in two stages. First, reflectance is derived from radiance by removing the atmospheric contributions relying on radiative transfer simulations. Second, surface properties are inferred from reflectance by applying specifically designed retrieval algorithms. Recent research indicates that quantifying uncertainties in reflectance retrievals is crucial in order to optimize retrievals of geophysical properties in subsequent analyses. Recently, probabilistic maximum a posteriori inversion methods, such as Optimal Estimation (OE) have been employed to provide reflectance retrievals with posterior uncertainty estimates. However, these probabilistic inversion methods are computationally expensive, and this poses a problem for imaging spectrometers acquiring tens or hundreds of thousands of spectra every second. Furthermore, it has not been clear whether the assumptions inherent in OE --- such as a locally-linear relationship between the estimated state and the measured radiance, or multivariate Gaussian posteriors --- are valid in the context of the surface reflectance retrieval problem. In this paper we present a Bayesian algorithm called Accelerated Optimal Estimation (AOE) that solves the OE problem in the best case using only a few permilles of the computing resources of a reference OE implementation (ROE). We show that the AOE retrieval method converges faster than the ROE method for a set of five test targets. We then perform Markov-chain Monte Carlo (MCMC) retrievals for the same targets, and evaluate OE posterior approximations against the "true" posterior distribution obtained from MCMC. We find that Gaussian approximations in OE accurately describe reflectances conditioned on a particular atmospheric state. We finally demonstrate the capabilities of AOE by retrieving a larger scene with 160,000 pixels with both AOE and ROE. The AOE method produces better-converged posterior estimates and maintains the speed advantage that was demonstrated with the five carefully analyzed test targets.