Convergent Binocular Stereo: Depth Perception for Humanoid Robot Vision

CBS Convergent Binocular Stereo We present an analytical convergent stereo algorithm, named CBS (Convergent Binocular Stereo), intended for active binocular stereo robots. We also propose a natural image convergent stereo dataset, CBS-BM (Convergent Binocular Stereo - BenchMark) Abstract The design of robotic binocular camera systems has been inspired by human vision, as has their use in humanoid robots. One aspect of this inspiration has yet to play a major role, namely that humans determine depth using a convergent binocular imaging geometry with both eyes pointing at the same location. Although some robot heads have the functionality to employ vergence and version movements and thus alter binocular imaging geometry, a method to exploit it for depth computation has not been explored. Even though to an observer their motions and appearance might resemble that of humans, in detail the resemblance to the motions required for actual convergent depth computation is not seen. To bridge this gap, we present Convergent Binocular Stereo (CBS), a stereo algorithm designed to provide a foundation for purposeful binocular computations under the humanoid constraint, intended for active binocular robots. CBS computes horizontal and vertical disparities using a coarse-to-fine refinement strategy with Gabor-filtered responses. We also introduce the Convergent Binocular Stereo BenchMark (CBS-BM), a convergent, natural image dataset, containing 49 scenes with ground-truth horizontal disparity collected on a four-degree-of-freedom robotic system. Our evaluation, a quantitative comparison between parallel and convergent stereo systems, show that CBS is broadly competitive with state-of-the-art parallel methods, even outperforming them in scenes with repeated patterns and in mean horizontal disparity and depth error over all scenes. Although not intended to replace parallel stereo where human-like behavior is unnecessary, CBS enables functionally realistic depth computation for humanoid robotic heads. To Run: Extract the dataset from CBS-BM/CBS-BM.zip Clone the DIJIT head repository for motor-camera calibration code. git clone https://gitlab.nvision.eecs.yorku.ca/robots/dijit-binocular-robotic-head.git To run for an example in the dataset: python3 run_one.py --scene <> --fixation <> --L_parallel_motors "_,_" --R_parallel_motors "_,_" --dataset "_" --calib_file "_" --b <> --motors_calib_L "_" --motors_calib_R "_" python3 run_one.py --scene 5 --fixation 3 --L_parallel_motors "67,116" --R_parallel_motors "88,101" --dataset "CBS-BM/CBS-BM" --calib_file "CBS-BM/CBS-BM/calib.yml" --b 0.115 --motors_calib_L "CBS-BM/CBS-BM/motors/L" --motors_calib_R "CBS-BM/CBS-BM/motors/L" To run your own pair of images: python3 run_one.py --calib_file "_" --b <> --motors_calib_L "_" --motors_calib_R "_" --fundamental "<fundamental matrix in the form 1,2,3,4,5,6,7,8,9>" --im_L "<left image file>" --im_R "<right image file>" Dataset CAS-BM.zip Filetypes:- .png - For all image files.- .csv - To store information on each scene and its fixations.- .npy - To store disparity outputs and ground truth disparities.- .txt - arrays.txt is generated in the output to store estimated fundamental matrices and rotation matrices. Structure: CBS-BM└───0│ │ fixations.csv│ │ gt_disp.npy| | L_parallel_rect.png| | R_parallel_rect.png│ ││ └───fixations│ └───0 │ | L.png│ | R.png│ └───1 │ L.png│ R.png| ...│ └───1│ │ fixations.csv│ │ gt_disp.npy| | L_parallel_rect.png| | R_parallel_rect.png│ ││ └───fixations│ └───0 │ | L.png│ | R.png│ └───1 │ | L.png│ | R.png| ...│ ...|└───motors│ └───L | | └───0 | | <calibration files>| | ...| | └───1 | | <calibration files>| | ... | ...| |│ └───R | | └───0 | | <calibration files>| | ...| | └───1 | | <calibration files>| | ... │ ...| └───calib.yml For each scene:Each scene is indexed by their scene numbers 0, 1, 2, ..., 48. Each scene has a `fixations.csv` which contains information for each fixation includuing the fixation index 0, 1, ... ('fixation'), the motor values for the left ('pan', 'tilt') and right cameras ('pan.1', 'tilt.1'), and the pixel location of the fixation point in the left and right cameras if viewed in the parallel images in the parallel configuration ('L_pix_x', 'L_pix_y','R_pix_x', 'R_pix_y'). Each scene contains the ground truth disparity in `gt_disp.npy` in the parallel camera configuration. Each scene has `L_parallel_rect.png` and `R_parallel_rect.png` images which are the images taken in the parallel configuration, already undistorted using the calibrated camera intrinsic matrices and distortion vectors. Motors:Motor calibration files are given in the motors folder for the left and right cameras marked "L" and "R" respectively. Camera Intrinsics:Calibrated camera intrinsic parameters are detailed in calib.yml. Note that the camera intrinsic matrices are in original sizes whereas the images in the dataset are reduced by 1/2. To use these intrinsics, either upscale the images or downscale the focal length and principal points in these matrices. AnalysisAnalysis and plotting scripts are in the plots folder. To plot the histograms in the manuscript, run `histogram_fig6.py` and `histogram_sup.py`. Results will be put in the `histogram_data` and `histogram_data2` folders. To run the same analysis as in the tables of the manuscript, refer to the README.md file in the `plots/analysis` folder.