AUV-Based Multi-Sensor Dataset: Forward-Looking Camera (FLC) and Forward-Looking Sonar (FLS) Observations in the Red Sea

Context This dataset is the first part of a dataset collection comprised of forward-looking sonar (FLS) and forward-looking camera (FLC) underwater images. The entire data was collected during the years 2021-2023 using 2 underwater vehicles in both the Red Sea and the Mediterranean along the Israeli shoreline, depicting both man-made and natural underwater environments. The data is part of a research project aimed at developing fusion models for improved obstacle detection and navigation in autonomous underwater vehicles. Content This dataset consists of FLC and FLS images and their metadata, collected by the ALICE-AUV. Both sensors were installed in the front payload section in a configuration having aligned fields of view to achieve matching pairs of data. The data was collected to train and evaluate a complete perception and obstacle avoidance framework. A series of diving sessions were performed in the Red Sea, off the coast of Eilat, Israel. The experiments focused on two main sites: A "Sunboat" shipwreck and the Eilat-Ashkelon Pipeline Company (EAPC) pier pillars. The "Sunboat" shipwreck is a 40-meter long vessel resting at a depth of approximately 12 meters, with the surrounding seabed at a depth of 18-24 meters. This dataset contains approximately 8,000 FLC-FLS sample pairs from the first session conducted at the "Sun boat" shipwreck site on September 3, 2023. The data was recorded at depths ranging from 10 to 15 meters. The dataset is organized into separate sessions, each representing a specific dive or experiment. Within each session, the data is further categorized into modalities: camera (FLC images), sonar (FLS images), and navigation (dead reckoning data). The navigation data is derived from a combination of GPS, DVL, and IMU sensors, providing estimated positions when GPS is unavailable. Inside each modality directory, you will find the corresponding data files in PNG format for images and CSV format for navigation data. The file names follow a sequential numbering scheme (e.g., 00001.png, 00002.png, etc.). Each modality directory also contains a CSV file (e.g., camera.csv) that maps each data file to its respective timestamp. Additionally, the samples.json file documents the relationship between uni-modal and multi-modal samples, allowing for easy association of data from different modalities. By providing synchronized and aligned camera and sonar imagery, along with corresponding navigation data, this dataset enables researchers to explore novel algorithms and techniques for multi-modal sensor fusion in the context of autonomous underwater vehicles. Technical Details Sonar: Blueprint Oculus M1200d Operating frequency: 1.2 MHz (low frequency mode) Maximum range: 40 m (set to 20 m for this dataset) Horizontal aperture: 130° Vertical aperture: 20° Number of beams: 512 Angular resolution: 0.6° Beam separation: 0.25° Image resolution: 902x497 pixels Coordinate system: Polar Camera: Allied-Vision Manta G-917 Image dimensions: 3384x2710 pixels (downscaled to 1692x1355 for this dataset) Sensor type: CCD Progressive Sensor bit depth: 12-bit Captured bit depth: 8-bit Camera model: Pinhole with Plumb Bob (Brown–Conrady) distortion coefficients Focal length (fx, fy): (1638.36157, 1641.95202) Principal point (cx, cy): (1705.03529, 1380.27954) Radial distortion coefficients (k1, k2, k3): (-0.124823, 0.048851, 0.000000) Tangential distortion coefficients (p1, p2): (0.000259, -0.002945) Navigation: Data format: CSV Contains fused dead reckoning data based on GPS, DVL, and IMU sensors Columns: timestamp: Unix timestamp (seconds) latitude: Latitude (degrees) longitude: Longitude (degrees) altitude: Altitude (meters) yaw: Yaw angle (degrees) pitch: Pitch angle (degrees) roll: Roll angle (degrees) velocity_x: Velocity along the x-axis (meters per second) velocity_y: Velocity along the y-axis (meters per second) velocity_z: Velocity along the z-axis (meters per second) depth: Depth (meters) Frame rate: 2 Hz for both sonar and camera More datasets from this collection will be uploaded in the future, and a link to access them will be provided on this page. Acknowledgements The data in this repository is part of the DeeperSense project that received funding from the European Commission, Program H2020-ICT-2020-2 ICT-47-2020, Project Number: 101016958.

数据集背景
本数据集为前瞻性声呐（forward-looking sonar, FLS）与前瞻性相机（forward-looking camera, FLC）水下图像数据集合集的第一部分。全部数据采集于2021至2023年间，依托2台水下无人航行器，在红海与地中海的以色列沿岸海域完成，涵盖人工与自然两类水下环境。本数据集隶属于一项旨在研发融合模型、以提升自主水下无人航行器障碍物检测与导航性能的研究项目。

数据集内容
本数据集包含由ALICE-AUV采集的FLC与FLS图像及其元数据。两款传感器均安装于前部有效载荷舱段，采用视场对齐的配置方案，以获取匹配成对的多模态数据。本数据集用于训练与评估完整的感知与避障框架。
数据采集于以色列埃拉特沿岸的红海海域，共开展多组下潜实验，实验聚焦两大目标场景："Sunboat"沉船遗址与埃拉特-阿什凯隆管道公司（Eilat-Ashkelon Pipeline Company, EAPC）码头桩柱。"Sunboat"沉船为一艘长40米的船体，沉没于约12米水深，其周边海床水深为18至24米。
本数据集包含2023年9月3日于"Sunboat"沉船遗址开展的首次下潜实验中采集的约8000对FLC-FLS样本对，数据记录水深介于10至15米之间。
数据集按独立的下潜/实验会话进行组织，每个会话对应一次特定的下潜或实验。每个会话内的数据进一步按传感器模态划分为三类：相机（FLC图像）、声呐（FLS图像）与导航（航位推算（dead reckoning）数据）。导航数据融合了GPS、多普勒测速仪（DVL）与惯性测量单元（IMU）传感器的数据，可在GPS信号失效时提供位置估计值。
每个模态目录下均包含对应的数据文件：图像文件采用PNG格式，导航数据采用CSV格式。文件名采用连续编号规则（例如00001.png、00002.png等）。每个模态目录还包含一个CSV文件（例如camera.csv），用于将各数据文件与其对应的时间戳进行映射。此外，samples.json文件记录了单模态与多模态样本间的关联关系，可便捷实现不同模态数据的匹配关联。
本数据集提供了同步对齐的相机与声呐图像，以及配套的导航数据，可支持研究人员探索适用于自主水下无人航行器场景的多模态传感器融合新算法与新技术。

技术参数
### 声呐
设备型号：Blueprint Oculus M1200d
工作频率：1.2 MHz（低频模式）
最大探测距离：40米（本数据集设置为20米）
水平波束宽度：130°
垂直波束宽度：20°
波束数量：512个
角分辨率：0.6°
波束间距：0.25°
图像分辨率：902×497像素
坐标系：极坐标系

### 相机
设备型号：Allied-Vision Manta G-917
图像尺寸：3384×2710像素（本数据集下采样至1692×1355像素）
传感器类型：CCD逐行扫描传感器
传感器位深度：12位
采集位深度：8位
相机模型：带Brown–Conrady铅垂畸变系数的针孔相机模型
焦距（fx, fy）：(1638.36157, 1641.95202)
主点坐标（cx, cy）：(1705.03529, 1380.27954)
径向畸变系数（k1, k2, k3）：(-0.124823, 0.048851, 0.000000)
切向畸变系数（p1, p2）：(0.000259, -0.002945)

### 导航数据
数据格式：CSV
包含基于GPS、DVL与IMU传感器融合的航位推算数据
字段说明：
timestamp：Unix时间戳（单位：秒）
latitude：纬度（单位：度）
longitude：经度（单位：度）
altitude：海拔高度（单位：米）
yaw：偏航角（单位：度）
pitch：俯仰角（单位：度）
roll：横滚角（单位：度）
velocity_x：x轴方向速度（单位：米/秒）
velocity_y：y轴方向速度（单位：米/秒）
velocity_z：z轴方向速度（单位：米/秒）
depth：水深（单位：米）
帧率：声呐与相机均为2 Hz

后续更新
本数据集合集的其余数据集将于后续上传，相关访问链接将发布于本页面。

致谢
本仓库中的数据隶属于DeeperSense项目，该项目获得了欧盟委员会H2020-ICT-2020-2 ICT-47-2020计划的资助，项目编号：101016958。

搜集汇总

背景与挑战

背景概述

该数据集是红海和地中海沿岸收集的多传感器数据，包含前视摄像头和前视声纳的同步图像及导航数据，用于自主水下车辆的障碍物检测和导航研究。数据集组织清晰，技术细节详尽，适用于多模态传感器融合算法的开发与评估。