deepNIR: Dataset for generating synthetic NIR images and improved fruit detection system using deep learning techniques

In this paper, we present datasets that can be utilised for synthetic near infrared (NIR) image and bounding box level fruit detection system. It is undeniable fact that high-caliber machine learning software frameworks such as Tensorflow or Pytorch and large scale dataset such as ImageNet and COCO, and accelerated GPU hardware support have pushed the limit of machine learning for more than decades. Among these breakthroughs quality dataset is one of important key building blocks that can lead to success in model generalisation and deployment for data-driven deep neural networks. Particularly, synthetic data generation such as generative adversarial networks often requires relatively larger scale data than other supervised approaches. In addition, posing constrains such as geometrical facial constrains in fake face generation or consistent and radiometrically calibrated reflectances from satellite imagery commonly yield better results. We share NIR+RGB dataset that are re-processed from other two public datasets (nirscene and SEN12MS) and our own novel sweetpepper dataset to be able to timely adopt to other following studies. We oversampled from original nirscene dataset at 10, 100, 200, and 400 ratios and total of 127k pair of images. For SEN12MS satellite multispectral dataset, we selected one largest subset; Summer (45k) and All seasons (180k). Our sweetpeppr dataset consists of 1,615 pairs of NIR+RGB images. We demonstrate these NIR+RGB datasets are sufficient to be used for synthetic NIR generation quantitatively and qualitatively. We achieved Frechet Inception Distance (FID) of 11.36, 26.53, and 40.15 for nirscene1, SEN12MS, and sweetpepper dataset respectively. We also release 11 fruits' bounding box annotations that can be exported as various formats using cloud service. 4 newly added fruits [blueberry, cherry, kiwi, and wheat] compounds 11 novel bounding box dastaset together with our previous work in deepFruits project [apple, avocado, capsicum, mango, orange, rockmelon, strawberry]. The total number of bounding box instances is 162k and all bounding box dataset is ready for use from cloud service. For evaluation of these dataset, Yolov5 single stage detector is exploited and reported impressive mean-average-precision, mAP[0.5:0.95] results of [min:0.49, max:0.812]. We hope these dataset is useful and serves as one of baseline for the following up studies.

本研究构建了可用于合成近红外（Near Infrared, NIR）图像与边界框（bounding box）级水果检测系统的数据集。毋庸置疑，数十年来，高性能机器学习软件框架（如TensorFlow、PyTorch）、大规模数据集（如ImageNet与COCO）以及加速图形处理器（Graphics Processing Unit, GPU）硬件支撑，推动机器学习技术实现了长足发展。在这些突破中，高质量数据集是支撑数据驱动的深度神经网络实现模型泛化与部署落地的核心构建模块之一。

尤其值得注意的是，相较于其他监督学习方法，生成式对抗网络（Generative Adversarial Networks）等合成数据生成技术通常需要更大规模的训练数据。此外，在生成虚假人脸时引入面部几何约束，或对卫星图像采用经过辐射定标的反射率一致性处理等约束条件，通常能获得更优的生成效果。

本研究对两份公开数据集（nirscene与SEN12MS）以及自研的新型甜椒数据集进行重新处理，构建了近红外+RGB（NIR+RGB）数据集并公开分享，以供后续相关研究及时使用。我们对原始nirscene数据集分别以10、100、200和400倍的倍率进行过采样，最终得到总计12.7万张图像对。针对SEN12MS卫星多光谱数据集，我们选取了其中两个最大子集：夏季子集（4.5万张）与全季节子集（18万张）。自研甜椒数据集则包含1615对近红外+RGB图像。

我们通过定量与定性实验证明，上述近红外+RGB数据集足以用于合成近红外图像生成任务。针对nirscene1、SEN12MS与甜椒数据集，我们分别得到了11.36、26.53与40.15的弗雷歇初始距离（Frechet Inception Distance, FID）得分。此外，我们还公开了11类水果的边界框标注信息，可通过云服务导出为多种格式。本次新增的蓝莓、樱桃、猕猴桃与小麦4类水果，结合我们在deepFruits项目中已有的苹果、牛油果、辣椒、芒果、橙子、甜瓜与草莓7类标注，共同构成了包含11类水果的新型边界框数据集。该边界框数据集的标注实例总数达16.2万，所有数据均可通过云服务直接获取使用。

为评估上述数据集的性能，我们采用YOLOv5单阶段检测器进行实验，取得了优异的平均精度均值（mean-average-precision, mAP[0.5:0.95]）结果，区间为[最小值0.49，最大值0.812]。我们期望本研究构建的数据集能够为后续相关研究提供助力，并作为基准数据集供参考使用。