Light field reconstruction datasets and algorithms

Abstract [PhD thesis]: Light field is defined as the outgoing radiance at a given point in a given direction. It is the result of the interaction between the incoming light and the surface with a specific material. Multi-view images captured by conventional cameras from multiple viewpoints are 2D projections of the light field. Reconstructing the underlying light field that produces the observed multi-view images is thus an inverse problem. Accurate light field reconstruction of a scene enables 3D understanding of the scene, which is important for many computer vision and machine intelligence problems. Thus, light field reconstruction is the core of many innovative technologies and applications, such as autonomous driving cars and metaverse. However, several challenges limit the practical application of light field reconstruction. Depth estimation is one of the crucial research problems in light field reconstruction, but existing work struggles to efficiently handle occlusions to preserve depth edges. In addition, effective light field representations capable of achieving photo-realistic novel view synthesis are desired to improve on the rendering quality in existing solutions. Novel algorithms dealing with these challenges will facilitate the applications of light field reconstruction in real-world scenarios.
 This thesis addresses these challenges in light field reconstruction from geometric, local, and global levels. At the geometric level, the aim is to reconstruct light field geometry by depth estimation. We construct a novel cost from a new perspective that counts the number of refocused pixels whose deviations from the central-view pixel are less than a small threshold and utilizes that number to select the correct depth. We show that without the use of any explicit occlusion handling methods, the proposed method can inherently preserve edges and produces high-quality depth estimates.
Lastly, we reconstruct a global light field to enable photorealistic view rendering from any point and any view direction by using a novel neural radiance feature field. We propose to use a multiscale tensor decomposition scheme to organize learnable features to represent scenes from coarse to fine scales. We demonstrate many benefits of the proposed multiscale representation, including more accurate scene shape and appearance reconstruction, and faster convergence compared with the single-scale representation. Instead of encoding view direction to model view-dependent effects, we further propose to encode the rendering equation in the feature space by employing an anisotropic spherical Gaussian mixture predicted from the proposed multiscale representation. Based on the proposed methods, we are able to reconstruct the accurate light field of a scene and achieve novel view rendering with high-fidelity view-dependent effects.
This dataset consists of:

links to the open access research data used for light field reconstruction 
source code for depth estimation algorithm (GitHub repository: https://github.com/imkanghan/OAVC)
source code light for field image reconstruction (GitHub repository: https://github.com/imkanghan/IRVAE)
source code for global light field reconstruction (GitHub repository: https://github.com/imkanghan/nrff

The data methods are available in the papers shown in the Related Publications link below and in the author's thesis (in press).