How Robot Dogs See the Unseeable

Supplementary Materials for How Robot Dogs See the Unseeable By mimicking how insects peer side-to-side, robots can now see through clutter in real-time. Supplementary Movies: Movie S1. Animal and robotic peering enhance scene understanding under limited vision. This movie supplements Fig. 1 in the main text, illustrating dynamic peering motions from locusts, the Florida bush cricket, and ANYbotics' ANYmal robot. It also shows the corresponding images from the robot's perspective during peering, along with the manual focusing process within the synthetic aperture (SA) integral images (here, using a planar synthetic focal surface). Finally, it presents visual reasoning results from a large multimodal model (specifically, ChatGPT-5.0) for both indoor RGB and outdoor near-infrared (NIR) recordings. Movie S2. Various robotic peering motions. This video demonstrates the peering motions implemented on a quadruped robot, including horizontal rotation, horizontal shift, and diagonal shift. These movements are constrained by the robot's equilibrium limits (avoid falling into the ground). For the ANYbotics ANYmal, this yields a maximum vertical SA of approximately 20cm and a horizontal SA of 30cm in our experiments. Supplementary Data: Data S1. Data for Figure 1. This dataset contains the images, poses, and parameters for the RGB and NIR recordings used for Fig. 1 in the main text. Use Software S1 to load this data and compute the SA integral images. Data S2. Data for Figure S3. This dataset contains the images, poses, and parameters for the rotation, horizontal shift, and diagonal shift peering motions used for Fig. S3. Use Software S1 to load this data and compute the SA integral images. Data S3. Data for Figure S4. This dataset contains the images, poses, and parameters for the additional reasoning examples used for Fig S4. Use Software S1 to load this data and compute the SA integral images. Data S4. Data for Figure S5. This dataset contains the images, poses, and parameters for the NIR recordings used for Fig. S5. Note, that the dataset for the RGB recordings is Data S1. Use Software S1 to load this data and compute the SA integral images. Supplementary Software: Software S1. Synthetic Aperture Integrator. This software was used to compute the SA integral images from a given set of conventional input images and the corresponding recording poses. It allows adjusting the focal surface (Fig. S2) interactively and supports occlusion masking (Fig. S6) based on Visible the Difference Vegetation Index (VDVI)15. It requires Microsoft Windows and a reasonably fast GPU. Please note that the following instructions describe only the functionality for viewing the supplementary data. For more detailed information, please contact the corresponding author. To begin, choose a supplementary dataset and copy the images and poses folders, along with the parameters.txt file, into the main SAI directory. By default, the dense scene shown in Fig. S4 is preinstalled. Then, run SAI.exe. Please be aware that loading may take some time, depending on the number of images. Navigate the Virtual Camera using the mouse wheel and left button. Within the menu, you can adjust the Focal Surface parameters. The surface can be shifted into the scene using the z-parameter, where a positive value moves it forward. Additionally, it can be translated (TX, TY), rotated (RX, RY, RZ), and scaled (SX, SY, SZ). Note that the focal surface is based on a unit half-sphere. By selecting large values for SX and SY, the surface will approximate a flat plane, while a positive SZ value scales it in the +z direction. Besides planes, cylindrical and spherical focal surfaces can also be created by setting appropriate values for SX, SY, and SZ. The grid-flag toggles the focal surface grid visualization on or off; the center of this grid indicates the current focus point. To view the individual captured images, select the pinhole aperture option in the Virtual Camera tab and use the Jump to +/- buttons to browse the sequence. To return to the synthetic aperture integration view, simply click the open aperture option. The occlusion mask parameters (T,UB,LB) threshold the occlusion mask values (ranging from -1 to 1). Within this range, high values identify occluder pixels, while low values identify non-occluder pixels. The threshold T serves as the central value for this separation. Pixels with values recognized as occluders are assigned low alpha values, potentially becoming fully transparent (0), whereas non-occluders are assigned high alpha values, potentially becoming fully opaque (1). The LB and UB parameters define lower and upper bounds around T to create a smooth transition. A pixel value below LB is assigned an alpha of 1, and a value above UB is assigned an alpha of 0. For values falling between LB and UB, the alpha value is linearly interpolated between 0 and 1. This entire process is applied to each image individually, resulting in a unique alpha mask for each one. By default, these occlusion mask values are computed as the Visible Difference Vegetation Index (VDVI)15, where a high value indicates vegetation and a low value indicates non-vegetation. VDVI thresholds are relatively low (e.g. T=0.025-0.115).