The Visual-Inertial Canoe Dataset

If you use this dataset, please cite the IJRR data paper (bibtex is below). We present a dataset collected from a canoe along the Sangamon River in Illinois. The canoe was equipped with a stereo camera, an IMU, and a GPS device, which provide visual data suitable for stereo or monocular applications, inertial measurements, and position data for ground truth. We recorded a canoe trip up and down the river for 44 minutes covering 2.7 km round trip. The dataset adds to those previously recorded in unstructured environments and is unique in that it is recorded on a river, which provides its own set of challenges and constraints that are described in this paper. The data is divided into subsets, which can be downloaded individually. Video previews are available on Youtube: https://www.youtube.com/channel/UCOU9e7xxqmL_s4QX6jsGZSw The information below can also be found in the README files provided in the 527 dataset and each of its subsets. The purpose of this document is to assist researchers in using this dataset. Images ====== Raw --- The raw images are stored in the cam0 and cam1 directories in bmp format. They are bayered images that need to be debayered and undistorted before they are used. The camera parameters for these images can be found in camchain-imucam.yaml. Note that the camera intrinsics describe a 1600x1200 resolution image, so the focal length and center pixel coordinates must be scaled by 0.5 before they are used. The distortion coefficients remain the same even for the scaled images. The camera to imu tranformation matrix is also in this file. cam0/ refers to the left camera, and cam1/ refers to the right camera. Rectified --------- Stereo rectified, undistorted, row-aligned, debayered images are stored in the rectified/ directory in the same way as the raw images except that they are in png format. The params.yaml file contains the projection and rotation matrices necessary to use these images. The resolution of these parameters do not need to be scaled as is necessary for the raw images. params.yml ---------- The stereo rectification parameters. R0,R1,P0,P1, and Q correspond to the outputs of the OpenCV stereoRectify function except that 1s and 2s are replaced by 0s and 1s, respectively. R0: The rectifying rotation matrix of the left camera. R1: The rectifying rotation matrix of the right camera. P0: The projection matrix of the left camera. P1: The projection matrix of the right camera. Q: Disparity to depth mapping matrix T_cam_imu: Transformation matrix for a point in the IMU frame to the left camera frame. camchain-imucam.yaml -------------------- The camera intrinsic and extrinsic parameters and the camera to IMU transformation usable with the raw images. T_cam_imu: Transformation matrix for a point in the IMU frame to the camera frame. distortion_coeffs: lens distortion coefficients using the radial tangential model. intrinsics: focal length x, focal length y, principal point x, principal point y resolution: resolution of calibration. Scale the intrinsics for use with the raw 800x600 images. The distortion coefficients do not change when the image is scaled. T_cn_cnm1: Transformation matrix from the right camera to the left camera. Sensors ------- Here, each message in name.csv is described ###rawimus### time # GPS time in seconds message name # rawimus acceleration_z # m/s^2 IMU uses right-forward-up coordinates -acceleration_y # m/s^2 acceleration_x # m/s^2 angular_rate_z # rad/s IMU uses right-forward-up coordinates -angular_rate_y # rad/s angular_rate_x # rad/s ###IMG### time # GPS time in seconds message name # IMG left image filename right image filename ###inspvas### time # GPS time in seconds message name # inspvas latitude longitude altitude # ellipsoidal height WGS84 in meters north velocity # m/s east velocity # m/s up velocity # m/s roll # right hand rotation about y axis in degrees pitch # right hand rotation about x axis in degrees azimuth # left hand rotation about z axis in degrees clockwise from north ###inscovs### time # GPS time in seconds message name # inscovs position covariance # 9 values xx,xy,xz,yx,yy,yz,zx,zy,zz m^2 attitude covariance # 9 values xx,xy,xz,yx,yy,yz,zx,zy,zz deg^2 velocity covariance # 9 values xx,xy,xz,yx,yy,yz,zx,zy,zz (m/s)^2 ###bestutm### time # GPS time in seconds message name # bestutm utm zone # numerical zone utm character # alphabetical zone northing # m easting # m height # m above mean sea level Camera logs ----------- The files name.cam0 and name.cam1 are text files that correspond to cameras 0 and 1, respectively. The columns are defined by: unused: The first column is all 1s and can be ignored. software frame number: This number increments at the end of every iteration of the software loop. camera frame number: This number is generated by the camera and increments each time the shutter is triggered. The software and camera frame numbers do not have to start at the same value, but if the difference between the initial and final values is not the same, it suggests that frames may have been dropped. camera timestamp: This is the cameras internal timestamp of the frame capture in units of 100 milliseconds. PC timestamp: This is the PC time of arrival of the image. name.kml -------- The kml file is a mapping file that can be read by software such as Google Earth. It contains the recorded GPS trajectory. name.unicsv ----------- This is a csv file of the GPS trajectory in UTM coordinates that can be read by gpsbabel, software for manipulating GPS paths. @article{doi:10.1177/0278364917751842, author = {Martin Miller and Soon-Jo Chung and Seth Hutchinson}, title ={The Visual–Inertial Canoe Dataset}, journal = {The International Journal of Robotics Research}, volume = {37}, number = {1}, pages = {13-20}, year = {2018}, doi = {10.1177/0278364917751842}, URL = {https://doi.org/10.1177/0278364917751842}, eprint = {https://doi.org/10.1177/0278364917751842} }

若使用本数据集，请引用发表于《国际机器人研究杂志》（*The International Journal of Robotics Research, IJRR*）的数据集论文，其BibTeX格式如下。

本数据集采集自美国伊利诺伊州桑加蒙河（Sangamon River）上的一艘独木舟。该独木舟搭载了立体相机（stereo camera）、惯性测量单元（IMU, Inertial Measurement Unit）与全球定位系统（GPS, Global Positioning System）设备，可提供适用于立体或单目视觉任务的视觉数据、惯性测量数据与用作真值的位置数据。

本次航程沿河道往返航行44分钟，总行程2.7公里。本数据集补充了此前在非结构化环境中采集的数据集，其独特之处在于采集场景为河道环境——该场景自带一系列独有的挑战与约束条件，详见本文。

数据集已划分为多个子集，支持单独下载。视频预览可在YouTube平台获取：https://www.youtube.com/channel/UCOU9e7xxqmL_s4QX6jsGZSw。以下信息亦可在527数据集及其各子集附带的README文件中查阅。本文档旨在协助研究人员使用本数据集。

# 图像（Images）
## 原始图像（Raw）
原始图像以BMP格式存储于`cam0`与`cam1`目录中，均为拜耳格式（Bayered）图像，使用前需进行去拜耳化与畸变校正。这些图像的相机参数可在`camchain-imucam.yaml`文件中获取。请注意，相机内参对应的原始图像分辨率为1600×1200，因此在使用前需将焦距与主点像素坐标缩放至原尺寸的0.5倍，而畸变系数在缩放图像后保持不变。该文件中同时包含相机到IMU的变换矩阵。其中，`cam0/`对应左相机，`cam1/`对应右相机。

## 校正后图像（Rectified）
经过立体校正、畸变校正、行对齐与去拜耳化处理的图像存储于`rectified/`目录中，存储方式与原始图像一致，但格式为PNG。`params.yaml`文件包含使用这些图像所需的投影与旋转矩阵，该参数无需像原始图像那样进行缩放。

## params.yaml
该文件包含立体校正参数：`R0`、`R1`、`P0`、`P1`与`Q`对应OpenCV `stereoRectify`函数的输出结果，仅将原输出中的1和2分别替换为0和1。
- `R0`：左相机的校正旋转矩阵
- `R1`：右相机的校正旋转矩阵
- `P0`：左相机的投影矩阵
- `P1`：右相机的投影矩阵
- `Q`：视差到深度的映射矩阵
- `T_cam_imu`：IMU坐标系到左相机坐标系的点变换矩阵

## camchain-imucam.yaml
该文件包含原始图像可用的相机内参、外参以及相机到IMU的变换矩阵：
- `T_cam_imu`：IMU坐标系到相机坐标系的点变换矩阵
- `distortion_coeffs`：采用径向切向模型的镜头畸变系数
- `intrinsics`：焦距x、焦距y、主点x坐标、主点y坐标
- `resolution`：标定分辨率，需针对原始800×600图像对内参进行缩放，而畸变系数在图像缩放后保持不变
- `T_cn_cnm1`：右相机到左相机的变换矩阵

# 传感器数据（Sensors）
本节将逐一说明`name.csv`中各消息的格式：
### rawimus
- `time`：GPS时间，单位为秒
- `message name`：rawimus
- `acceleration_z`：z轴加速度，单位为m/s²，IMU采用右-前-上坐标系
- `-acceleration_y`：y轴加速度（负方向），单位为m/s²
- `acceleration_x`：x轴加速度，单位为m/s²
- `angular_rate_z`：z轴角速度，单位为rad/s，IMU采用右-前-上坐标系
- `-angular_rate_y`：y轴角速度（负方向），单位为rad/s
- `angular_rate_x`：x轴角速度，单位为rad/s

### IMG
- `time`：GPS时间，单位为秒
- `message name`：IMG
- 左图像文件名、右图像文件名

### inspvas
- `time`：GPS时间，单位为秒
- `message name`：inspvas
- `latitude`：纬度
- `longitude`：经度
- `altitude`：椭球高度，采用WGS84坐标系，单位为米
- `north velocity`：北向速度，单位为m/s
- `east velocity`：东向速度，单位为m/s
- `up velocity`：向上速度，单位为m/s
- `roll`：绕y轴的右手旋转角，单位为度
- `pitch`：绕x轴的右手旋转角，单位为度
- `azimuth`：绕z轴的左手旋转角，从正北顺时针计算，单位为度

### inscovs
- `time`：GPS时间，单位为秒
- `message name`：inscovs
- `position covariance`：位置协方差，共9个值，顺序为xx,xy,xz,yx,yy,yz,zx,zy,zz，单位为m²
- `attitude covariance`：姿态协方差，共9个值，顺序同上，单位为deg²
- `velocity covariance`：速度协方差，共9个值，顺序同上，单位为(m/s)²

### bestutm
- `time`：GPS时间，单位为秒
- `message name`：bestutm
- `utm zone`：UTM分区数值编号
- `utm character`：UTM分区字母代号
- `northing`：北坐标，单位为m
- `easting`：东坐标，单位为m
- `height`：平均海平面以上高度，单位为m

# 相机日志（Camera logs）
文件`name.cam0`与`name.cam1`分别对应相机0与相机1的日志文本文件，各列定义如下：
1. 未使用列：首列全为1，可忽略
2. 软件帧编号：每次软件循环迭代结束时递增的编号
3. 相机帧编号：由相机生成的编号，每次快门触发时递增
注：软件帧编号与相机帧编号初始值无需一致，若二者始末差值不匹配，则表明可能存在丢帧情况
4. 相机时间戳：相机内部记录的帧捕获时间戳，单位为100毫秒
5. PC时间戳：图像到达PC端的系统时间

# 其他辅助文件
## name.kml
该KML文件为轨迹映射文件，可通过Google Earth等软件读取，包含记录的GPS航迹。
## name.unicsv
该CSV文件为UTM坐标系下的GPS航迹数据，可通过gpsbabel（一款用于处理GPS路径的软件）读取。

# BibTeX引用格式
@article{doi:10.1177/0278364917751842, author = {Martin Miller and Soon-Jo Chung and Seth Hutchinson}, title ={The Visual–Inertial Canoe Dataset}, journal = {The International Journal of Robotics Research}, volume = {37}, number = {1}, pages = {13-20}, year = {2018}, doi = {10.1177/0278364917751842}, URL = {https://doi.org/10.1177/0278364917751842}, eprint = {https://doi.org/10.1177/0278364917751842} }