FruitNeRF: A Unified Neural Radiance Field based Fruit Counting Framework

We introduce FruitNeRF, a unified novel fruit counting framework that leverages state-of-the-art view synthesis methods to count any fruit type directly in 3D. Our framework takes an unordered set of posed images captured by a monocular camera and segments fruit in each image. To make our system independent of the fruit type, we employ a foundation model that generates binary segmentation masks for any fruit. Utilizing both modalities, RGB and semantic, we train a semantic neural radiance field. Through uniform volume sampling of the implicit Fruit Field, we obtain fruit-only point clouds. By applying cascaded clustering on the extracted point cloud, our approach achieves precise fruit count. The use of neural radiance fields provides significant advantages over conventional methods such as object tracking or optical flow, as the counting itself is lifted into 3D. Our method prevents double counting fruit and avoids counting irrelevant fruit. We evaluate our methodology using both real-world and synthetic datasets. The real-world dataset consists of three apple trees with manually counted ground truths, a benchmark apple dataset with one row and ground truth fruit location, while the synthetic dataset comprises various fruit types including apple, plum, lemon, pear, peach, and mangoes. Additionally, we assess the performance of fruit counting using the foundation model compared to a U-Net.

我们提出了FruitNeRF这一统一的新型果实计数框架，其依托当前最先进的视图合成技术，可直接在三维空间中完成任意品类果实的计数。该框架可接收单目相机拍摄的无序位姿图像集，并对每张图像中的果实进行分割。为使系统不受果实品类限制，我们采用了一款可生成任意果实二进制分割掩码的基础模型（foundation model）。我们融合RGB与语义两种模态信息，训练得到语义神经辐射场（Semantic Neural Radiance Field）。通过对隐式果实场进行均匀体采样，我们得到仅包含果实的点云。通过对提取出的点云实施级联聚类，我们的方法可实现高精度果实计数。相较于目标跟踪、光流法等传统方法，神经辐射场（Neural Radiance Field）的应用具备显著优势：计数过程可直接升维至三维空间完成，有效避免果实重复计数与无关果实误计数问题。我们分别利用真实世界数据集与合成数据集对所提方法进行性能评估。真实世界数据集包含三棵经人工计数标注的苹果树、一套单排标注果实位置的基准苹果数据集；合成数据集则涵盖苹果、李子、柠檬、梨、桃与芒果等多种果实品类。此外，我们还对比了基础模型与U-Net在果实计数任务中的性能表现。

搜集汇总

背景与挑战

背景概述

FruitNeRF是一个基于神经辐射场的水果计数框架，通过单目相机图像和基础模型生成语义分割，在3D空间中计数水果，避免重复计数和无关计数。数据集包括真实世界的苹果树数据和合成多种水果类型的数据，用于评估框架性能。