Automatic image and video enhancement with application to visually impaired people

Images/videos may have poor visual quality due to the relatively low dynamic range of capture/display devices as compared to the human visual system or poor lighting conditions or the lack of experience of people capturing them. We are exploring techniques to perform automatic enhancement of images and videos. The goal is to produce a better visual experience for all viewers. Another motivation is to improve perception for visually impaired patients, in particular, people suffering from Age-related Macular Degeneration. In order to address these problems, we have developed novel techniques for contrast and sharpness enhancement of images and videos. ❧ Our color contrast enhancement technique is inspired from the Retinex theory. We use denoising techniques to estimate the illumination component of the image, while preserving color and white balance. We then enhance only the illumination component using mapping functions. These enhancement parameters are estimated automatically. This enhanced illumination is then combined with the original reflectance to obtain enhanced images with better contrast. ❧ For sharpness enhancement, we use a novel approach based on a hierarchical framework using edge-preserving Non-local means filter. The hierarchical framework is constructed from a single image using a modified version of Laplacian pyramid. We also introduce a new measure to quantify sharpness quality, which allows us to automatically set parameters in order to achieve a preferred sharpness enhancement. ❧ Finally, we propose a novel method based on modularity optimization to perform temporally consistent and robust enhancement of videos. A key aspect of our processes is that it is fully automatic. Our method detects scene changes `on-the-fly' in a video. For every detected cluster, we find a key frame that is most representative of other frames in the sequence and estimate enhancement parameters for only the key frame. We then enhance all frames in that cluster using these enhancement parameters, thus making our method robust. ❧ We compare our enhancement results with existing state-of-the-art approaches and commercial packages and show noticeable improvements. Our image enhancements do not suffer from halo effects, color artifacts, and color shifts; our video enhancement is temporally consistent and does not suffer from flash or flickering artifacts. Validation is challenging, as visual experience is intrinsically subjective. We have conducted extensive tests on real viewers (both normally sighted and simulated visually impaired), and provide a statistical measure of improvement in terms of preference.