DeepForest: Sensing Into Self-Occluding Volumes of Vegetation With Aerial Imaging

Abstract. Access to below-canopy volumetric vegetation data is crucial for understanding ecosystem dynamics. We address the long-standing limitation of remote sensing to penetrate deep into dense canopy layers. LiDAR and radar are currently considered the primary options for measuring 3D vegetation structures, while cameras can only extract the reflectance and depth of top layers. Using conventional, high-resolution aerial images, our approach allows sensing deep into self-occluding vegetation volumes, such as forests. It is similar in spirit to the imaging process of wide-field microscopy, but can handle much larger scales and strong occlusion. We scan focal stacks by synthetic-aperture imaging with drones and reduce out-of-focus signal contributions using pre-trained 3D convolutional neural networks with mean squared error (MSE) as the loss function. The resulting volumetric reflectance stacks contain low-frequency representations of the vegetation volume. Combining multiple reflectance stacks from various spectral channels provides insights into plant health, growth, and environmental conditions throughout the entire vegetation volume. Compared with simulated ground truth, our correction leads to ~x7 average improvements (min: ~x2, max: ~x12) for forest densities of 220 trees/ha - 1680 trees/ha. In our field experiment, we achieved an MSE of 0.05 when comparing with the top-vegetation layer that was measured with classical multispectral aerial imaging. Movie S1. Animated version of Fig. 8. Raw multispectral (green=GRE, near-infrared=NIR, red=RED, red-edge=REG, and RGB) focal stacks of field experiment (focussing top-down, from highest tree-crown at 20m AGL to surface of forest). For NIR and RED: uncorrected focal stack, corrected reflectance stack, corrected reflectance stack (sensor-mapped), extracted top vegetation layer (corrected and sensor mapped). Movie S2. Animated version of Fig. 9. NDVI stack computed from corrected and sensor-mapped RED and NIR reflectance stacks. Extracted top vegetation layer (COLMAP reconstruction) from NDVI stack (two different color maps). Values above the top vegetation layer are removed from NDVI stack. Cropping and range filtering of NDVI stack. Data S1. Raw data of field experiment. For each spectral channel (see folders: green=GRE, near-infrared=NIR, red=RED, red-edge=REG, and RGB) we provide original aerial images (but cropped to a square resolution, stored as .png in the images folder) and corresponding poses (stored in a .json file in the poses folder). In addition, we provide integral images for each focal stack layer (0.png …439.png in the integrals folder) and a compilation of all layers in one image stack (integrals.tif). For the RGB channel, we provide the integral images in full resolution but cropped to square (in the integrals_full_respolution folder), and in a downsampled resolution that matches the resolution (440x440px) of the integrals of the other spectral channels (in the integrals_440 folder). Data S2. Datasets of uncorrected focal stacks, corrected, and corrected + sensor mapped reflectance stacks (Fig. 8). The .vti files can be viewed in ParaView. The different channels contain the following data: reflectance values (Color, channels_and_opacity, x), COLMAP geometry reconstruction from RED or NIR images (Opacity, opacity). For visualization in ParaView, select color to be the uncorrected focal stack / the corrected + sensor mapped reflectance stack (channels_and_opacity, x) and optionally opacity to be the COLMAP geometry reconstruction (Opacity, opacity). Adapt the transfer function for blending. Data S3. Dataset of NDVI stack (computed from corrected and sensor-mapped RED and NIR reflectance stacks, Fig. 9). The .vti files can be viewed in ParaView. The different channels contain the following data: NDVI values (Color, channels_and_opacity, x), COLMAP geometry reconstruction with point-clouds merged from RED and NIR images (Opacity, opacity). For visualization in ParaView, select color to be the NDVI stack stack (channels_and_opacity, x) and optionally opacity to be the COLMAP geometry reconstruction (Opacity, opacity). Adapt the transfer function for blending. Data S4. Dataset of NDVI stack as in Data S3, but with values above the top vegetation layer removed. The .vti files can be viewed in ParaView. The different channels contain the following data: NDVI values (Color, channels_and_opacity, x) with values above the top vegetation layer = -1.01, binary mask with 0 values above the top vegetation layer and 1 values below and on the top vegetation layer (Opacity, opacity). For visualization in ParaView, select color to be the NDVI stack stack (channels_and_opacity, x) and optionally opacity to be the binary mask (Opacity, opacity). Adapt the transfer function for blending (for Color, channels_and_opacity, x values should be > -1.01). Data S5. Datasets of uncorrected focal stacks, corrected, and corrected + sensor mapped reflectance stacks for green and red edge channels (Fig. S3). The .vti files can be viewed in ParaView. The different channels contain the following data: reflectance values (Color, channels_and_opacity, x), COLMAP geometry reconstruction from REG or GREEN images (Opacity, opacity). For visualization in ParaView, select color to be the uncorrected focal stack / the corrected + sensor mapped reflectance stack (channels_and_opacity, x) and optionally opacity to be the COLMAP geometry reconstruction (Opacity, opacity). Adapt the transfer function for blending. Movie S3. Animated version of Fig. 3 (bottom row) and Fig. 6. Raw focal stack, focal stack before and after correction, ground truth forest plot, and photogrammetric reconstruction. Reflectance values are within the same global range (0..1) and color coded. Volumes are visualized with no full opacity to display intrinsic structures. Data S6. Raw data of simulation. Grid (9x9) of multi-view aerial images (TD_pose_0_rgb.png…TD_pose_80_rgb.png in the images folder) and poses (poses.txt in the poses folder, where the first line is the pose_0 and the last line is pose_80) that were used to compute datasets Data S1-S4. In addition, we provide integral images for each focal stack layer (Layer_1.png …Layer_440.png in the integrals folder) in the and a compilation of all layers in one image stack (integrals.tif). Data S7. Dataset of original ground truth geometry and reflectance (Fig. 6, first row). The .tif file can be viewed in ParaView (TIFF Series Reader). It contains only one channel (TIFF Scalars) that stores the original simulated reflectance values. Data S8. Dataset of uncorrected focal stack (Fig. 6, second row). The .vti file can be viewed in ParaView. The different channels contain the following data: uncorrected reflectance values of focal stack (Color, channels_and_opacity, x), ground truth geometry (Color, channels_and_opacity, y), COLMAP geometry reconstruction (Opacity, opacity). For visualization in ParaView, select color to be the uncorrected focal stack (Color, channels_and_opacity, x) and opacity to be either ground truth geometry (Color, channels_and_opacity, y) or the COLMAP geometry reconstruction (Opacity, opacity). Adapt the transfer function for blending. Data S9. Dataset of corrected reflectance stack (Fig. 6, third row). The .vti file can be viewed in ParaView. The different channels contain the following data: corrected reflectance values of reflectance stack (Color, channels_and_opacity, x), ground truth geometry (Color, channels_and_opacity, y), COLMAP geometry reconstruction (Opacity, opacity). For visualization in ParaView, select color to be the corrected reflectance stack (Color, channels_and_opacity, x) and opacity to be either ground truth geometry (Color, channels_and_opacity, y) or the COLMAP geometry reconstruction. Adapt the transfer function for blending. Data S10. Dataset of original photogrammetry-based (COLMAP) point-cloud reconstruction – including point geometry and reflectance values (Fig. 6, bottom row). The .vtk file can be viewed in ParaView. It contains only one channel (Results) that stores the original reflectance values determined by COMAP. Code S1. Python script used for producing the simulation results for training, inference, and testing using the GAZEBO-based drone simulator (https://github.com/bostelma/gazebo_sim). Code S2. Python code for preprocessing the input and training data, training our models, and final inference (reconstruction). See included readme file for details. Data S11. Pre-trained models for all 440 layers, as used for producing the results in this article.