Towards a fully automated underwater census for fish assemblages in the Mediterranean Sea

Assessing underwater biodiversity is labour-intensive and costly, but is crucial for measuring the extent of the decline in local fish stock. In most cases, Underwater Visual Census (UVC) is the preferred method, however this can be costly in terms of human effort and is limited by meteorological and logistical factors. Advances in technology allows the utilisation of more autonomous video recording methods (i.e. Remote Operated Vehicles (ROV)) which addresses these limitations. This study used a transect-wise UVC coupled with diver operated videos (DOV). For the video analysis, a comprehensive fully automated pipeline was developed to extract frames from DOV and perform colour correction. This pipeline integrates a YOLO-based model to detect 20 Mediterranean fish species and validate the presence or absence of each species within individual transects. This study was conducted to evaluate the feasibility of using video-based methods for UVC with minimal human-input. The result of automated video analysis were in agreement with manual video counting, validating the autonomous and bias-free procedure for video assessment. In conclusion, utilising a minimal-human-input video method liberates the data acquisition from limiting factors (i.e. meteorological and logistical) and automation of this video analysis significantly reduces the labour and time required. For future fieldwork campaigns, the video data collection protocol needs to be modified to better resemble traditional UVC and enhance this acquisition method. Methods 1. Study area and data collection The training dataset (DATAT ) was gathered in eight different locations in the Mediterranean Sea along the French Riviera, following the same UVC protocol on each site (Harmelin-Vivien et al., 1985). The depth ranged from 1-37m and was carried out during the whole year in 2022 (cold and warm season) to cover the full range of conditions and possibilities of fish occurrences. The experimental dataset (DATAE) was recorded in October 2023 in and around two protected areas, one no-take zone (Cap Roux) and one Natura2000 site (Corniche Varoise), which both have elevated biodiversity. The specific coordinates and meta data can be found in the supplementary material (Table S1). A total of 64 videos, each corresponding to a transect, from 14 sites (8 on seagrass meadows and 6 on rocky substrates) were evaluated and compared. Each site consists of 3 to 6 transects, depending on the availability of video recordings and UVC data from the divers. The videos were obtained with GoPro HERO 9 cameras, mounted on the clipboards (Fig. 1) used by the divers to note the number of fish per species with their respective size category (variable number of categories per species). The videos were recorded with a framerate of 24 frames per second (FPS) and full high definition resolution (1920x1080px). Frames were extracted from these recordings with a framerate of 1 FPS for DATAT and 5 FPS for DATAE. Fish visible for less than 1 second (less than 5 frames) in the videos of DATAE will not be considered in the methodology evaluation as they were unlikely to be actual detections. 2. Image preprocessing The frames were processed in a next step by a marine biology expert to guarantee correctly identified species in the videos. To ensure a good species coverage, 19 different species and an ’Other’ class were labelled manually in the frames resulting in 13,033 images (131 videos in the training set and 47 independent videos in the test set) in DATAT with a total of 68,573 (train = 40,379, test = 28,194) individual fish labels (species breakdown in Table S3) and 8,739 miscellaneous labels such as background and diver. The ’Other’ class includes species (Table S2) that have insufficient occurrences in the test videos (n < 100). Since there was a wide range of conditions in the videos, a preprocessing was applied to both datasets to enhance each image colour range. For this purpose, a pretrained UIEC2-Net model (Y. Wang et al., 2021) was utilised to enhance the images. As the last preprocessing step, images were rescaled to 960x960px.水下生物多样性评估工作强度大、成本高昂，但对于衡量当地鱼类种群的衰退程度至关重要。在多数场景中，水下视觉普查（Underwater Visual Census, UVC）是首选方法，但该方法人力投入成本高，且受气象与后勤因素限制。技术进步使得我们得以采用更多自主式视频记录手段（例如远程操作载具（Remote Operated Vehicles, ROV)），从而解决上述局限。本研究采用样带式水下视觉普查结合潜水员操作视频（diver operated videos, DOV）的方案。在视频分析环节，我们开发了一套完整的全自动流程，用于从DOV中提取帧并完成色彩校正。该流程集成了基于YOLO的模型，可检测20种地中海鱼类，并验证各物种在单个样带中的存在与否。本研究旨在评估低人工投入下基于视频的水下视觉普查方法的可行性。自动视频分析结果与手动视频计数结果一致，验证了该自主且无偏的视频评估流程的有效性。综上，采用低人工投入的视频采集方法可使数据采集摆脱气象与后勤等限制因素，且该视频分析的自动化流程可大幅降低所需人力与时间成本。对于未来的野外作业计划，需对视频数据采集协议进行优化，使其更贴近传统水下视觉普查流程，进一步完善该采集方法。 方法 1. 研究区域与数据采集 训练数据集（DATAT）采集自地中海法国里维埃拉沿岸的8个不同点位，所有站点均遵循统一的水下视觉普查协议（Harmelin-Vivien等，1985）。调查水深范围为1至37米，于2022年全年（涵盖冷暖季节）开展，以覆盖鱼类出现的全部环境条件与场景。 实验数据集（DATAE）于2023年10月在两个保护区及其周边区域录制，分别为禁捕区（Cap Roux）与Natura2000保护地（Corniche Varoise），这两个区域均拥有较高的生物多样性。具体坐标与元数据可参见补充材料（表S1）。本研究共评估并对比了来自14个站点的64段视频（每段对应一个样带），其中8个站点为海草床生境，6个为岩礁底质生境。根据视频录制与潜水员获取的水下视觉普查数据的可用性，每个站点包含3至6个样带。 视频采集采用GoPro HERO 9相机，安装于潜水员用于记录各物种鱼类数量及其体型分类（不同物种的体型分类数量存在差异）的记录板上（图1）。视频录制帧率为24帧每秒（Frames Per Second, FPS），分辨率为全高清（1920×1080px）。从上述录像中提取帧时，训练数据集（DATAT）的提取帧率为1 FPS，实验数据集（DATAE）为5 FPS。在DATAE的视频中，停留时长不足1秒（即少于5帧）的鱼类将不纳入方法学评估，因为这类个体难以被有效识别。 2. 图像预处理 后续步骤中，由海洋生物学专家对提取的帧进行处理，以确保视频中的物种识别准确。为保证物种覆盖的全面性，我们在帧中手动标注了19个不同物种及一个“其他”类别，最终得到训练数据集（DATAT）中的13033张图像（对应训练集内131段视频与测试集内47段独立视频），共计68573个鱼类个体标注（训练集40379个，测试集28194个，物种分类详情见表S3），以及8739个背景、潜水员等杂项标注。“其他”类别包含测试视频中出现次数不足100次的物种（表S2）。鉴于视频中的环境条件差异较大，我们对两个数据集均应用了预处理步骤以优化图像色彩范围。为此，我们采用了预训练的UIEC2-Net模型（Y. Wang等，2021）对图像进行增强。作为最后一步预处理操作，我们将图像缩放至960×960px。