ROV-Based Multi-Sensor Dataset: Synchronized Camera and Sonar images taken in the Tropical Waters of the Red Sea, Eilat

Description:

This dataset consists of approximately 46,928 synchronized image pairs collected by the  Blue-ROV2. The images were captured using a machine-vision camera (IDS UI-3260CP-C-HQ)  and a BluePrint Oculus M1200d Forward-Looking Sonar (FLS). Both sensors were installed with the FLS tilted 15 degrees downward to achieve optimal coverage of the terrain and optimal FOV overlap.

The data was collected to train and evaluate a comprehensive perception and obstacle avoidance framework.

Context:

This dataset is the second installment in our collection of synchronized multi-sensor underwater  datasets, aimed at enabling advanced research in multi-modal sensor fusion, obstacle  detection, and navigation for autonomous underwater vehicles (AUVs). The data was collected  using the Blue-ROV2 Remotely Operated Vehicle (ROV) in the tropical waters of the Red Sea,  off the coast of Eilat, Israel. This data captures diverse underwater environments and is part of a  research project focused on developing fusion models for improved obstacle detection and  navigation in AUVs.

Content:

The data encompasses several sites within the tropical waters of the Red Sea, Eilat, including  corals, rocks, shipwrecks, man-made structures, piers, and caves. The ROV platform was  operated by divers, ensuring accurate positioning and coverage. Data was acquired at depths  ranging from 3 to 12 meters at different times from dawn to dusk.

Dataset Composition:

Site

Recording Session

Image Pairs

Description

Tropical Site 1

20221211_092506

20221211_133252

10,915

7,978

Pier, rocks, corals

Tropical Site 2

20221212_095821

20221212_141308

9,900

8,475

Man-made structure, 

rocks, corals

Tropical Site 3

20221213_102542

9,390

Rocks, corals

Total

46,928 

The dataset is organized into separate sessions, each representing a specific dive or  experiment. Within each session, data is further categorized into modalities: camera (FLC  images), sonar (FLS images), and depth. Each modality directory contains the corresponding  data files in PNG format for images and CSV format for depth data.

Each modality directory includes:

A `camera.csv` file for the camera modality that maps each image file to its respective  timestamp.

A `sonar.csv` file for the sonar modality that maps each image file to its respective timestamp.

The depth data in `depth.csv` formatted with `timestamp` and `value`.

Additionally, a `samples.json` file documents the relationship between uni-modal and  multi-modal samples, enabling easy association of data from different modalities.

Technical Details:

Camera: IDS UI-3260CP-C-HQ

Image dimensions: 1936x1216 pixels (downscaled to 968 × 608 for this dataset)

Sensor type: Sony IMX249 1/1.2" CMOS

Lens: Tamron M112FM06

Captured bit depth: 8-bit

Frame rate: 5 Hz

Sonar: BluePrint Oculus M1200d

Operating frequency: 1.2 MHz (low frequency mode)

Maximum range: 40 m (set to 15 m for this dataset)

Horizontal aperture: 130°

Vertical aperture: 20°

Number of beams: 512

Angular resolution: 0.6°

Beam separation: 0.25°

Image resolution: 544x300 pixels

Coordinate system: Polar

Frame rate: 5 Hz

Depth: Blue-Robotics Ping2 Sonar Altimeter and Echosounder

Frequency: 115 kHz

Source Level: 198 dB re 1µPa @ 1m

Beamwidth: 25 degrees

Typical Minimum Range: 0.3 m (1 ft)

Typical Usable Range: 100 m (328 ft)

Range Resolution: 0.5% of range

Depth Rating: 300 m (984 ft)

Data format: CSV

Columns:

timestamp: Unix timestamp (seconds)

value: Depth value (meters)

Sample rate: 5 Hz

Example File Tree Layout:

```${session}/${dataset}/camera/camera.csv00000001.png00000002.png…sonar/sonar.csv00000001.png00000002.png…depth/depth.csvsamples.json``` Example File Content:

 camera.csv```timestamp,filename1644234340.181234,00000001.png1644234343.375667,00000002.png```

 sonar.csv```timestamp,filename1644234340.181234,00000001.png1644234343.375667,00000002.png```

 depth.csv```timestamp,value1644234340.181234,5.41644234343.375667,6.1```

 samples.json

```{    "samples": [        {            "camera": [                0            ],            "depth": [                0            ],            "sonar": [                0            ]        },        {            "camera": [                1            ],            "depth": [                1            ],            "sonar": [                1            ]        }]

```

By providing synchronized and aligned camera, sonar imagery, and depth data, this dataset  enables researchers to explore novel algorithms and techniques for multi-modal sensor fusion in  the context of autonomous underwater vehicles operating in the tropical waters of the Red Sea.

Acknowledgements

The data in this repository is part of the DeeperSense project that received funding from the European Commission, Program H2020-ICT-2020-2 ICT-47-2020, Project Number: 101016958.