Mask R-CNN on NYUv2

Mask R-CNN on NYUv2 This repository mainly contains information from the execution of the Mask R-CNN network [1] on images from the NYUv2 dataset [2] as well as additional metadata. It was created for analyzing the output of Mask R-CNN and post-processing it using contextual information for improving its performance. This work has been carried out by Dr. Jose-Raul Ruiz-Sarmiento (MAPIR group, University of Málaga) and Dr. Shuda Li (AVG group, University of Oxford) in the scope of the European project MoveCare: Multiple-actOrs Virtual Empathic CARgiver for the Elder (Ref: 732158). Concretely, this repository includes: - metadata: + coco_nyu_mapping.txt: Mapping between the categories in COCO dataset and those in NYUv2. + coco_object_categories.txt: Object categories considered in COCO dataset. + nyu_object_categories.txt: Object categories used in NYUv2 dataset. + nyu_scene_categories.txt: Scene categories considered in NYUv2. + objects_and_categories_in_images.txt: For each image in NYUv2, the categories of the appearing objects. - nyu_content: + masks_in_X (Where X is the image index) - Y.png: Where Y is the object index in the image, represents the binary mask of that object. - pixels_labelled.png: Binary mask indicating the labelled pixels in image X. + bboxesX.txt: Where X is the image index, includes the ground truth bounding boxes of the objects in it. Format is: min_x min_y max_x max_y. - preds: + X: Where X is the image index. - Y.png: Where Y is the object index in the image, as detected by Mask R-CNN. Binary image containing the mask of such detected object. + X.txt: Where X is the image index. File containing the objects detected by Mask R-CNN, including: idx class score min_x min_y max_x max_y masks_file, being min_x min_y max_x and max_y bounding box information, while masks_file refers to X/Y.png as described above. + result_X.png: Where X is the image index. Image showing the detections with a socre higher than 0.3. + gt_iou_X: Where X is the image index. - Y: Where Y is the index of the detected object. + Z.png Where Z is the index of the object in the ground truth. Image showing the masks of both objects, Y and Z, for visually checking their overlapping. - Y.txt: Where Y is the index of the detected object. File containing: + The intersection ratio of the object mask Y with the labelled part of the image. + The IoU value for the mask of object Y and those of ground truth objects. References: [1] He, Kaiming, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. "Mask r-cnn." In Proceedings of the IEEE international conference on computer vision, pp. 2961-2969. 2017. [2] Silberman, Nathan, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. "Indoor segmentation and support inference from rgbd images." In European Conference on Computer Vision, pp. 746-760. Springer, Berlin, Heidelberg, 2012.

NYUv2数据集上的Mask R-CNN

本仓库主要包含Mask R-CNN网络[1]在NYUv2数据集[2]的图像上执行得到的相关信息，以及额外的元数据。本项目旨在分析Mask R-CNN的输出结果，并通过上下文信息对其进行后处理以提升模型性能。

本项目由马拉加大学MAPIR研究组的何塞-劳尔·鲁伊斯-萨尔米恩托博士，以及牛津大学AVG研究组的李树达博士，在欧盟项目MoveCare：面向老年人的多智能体虚拟共情护理者（项目编号：732158）的框架下完成。具体而言，本仓库包含以下内容：

- 元数据：
+ coco_nyu_mapping.txt：记录COCO数据集与NYUv2数据集类别对应关系的映射文件。
+ coco_object_categories.txt：COCO数据集所采用的目标类别列表。
+ nyu_object_categories.txt：NYUv2数据集所使用的目标类别列表。
+ nyu_scene_categories.txt：NYUv2数据集涵盖的场景类别列表。
+ objects_and_categories_in_images.txt：针对NYUv2数据集中的每张图像，记录其中出现目标所属类别的文件。

- nyu_content：
+ masks_in_X（其中X为图像索引）- Y.png：其中Y为该图像中的目标索引，代表对应目标的二值掩码。
+ pixels_labelled.png：二值掩码图像，用于标记图像X中的已标注像素区域。
+ bboxesX.txt：其中X为图像索引，包含该图像中目标的真实边界框信息，格式为：min_x min_y max_x max_y。

- 预测结果（preds）：
+ X（其中X为图像索引）- Y.png：其中Y为该图像中的目标索引，为Mask R-CNN检测到的对应目标的二值掩码图像。
+ X.txt：其中X为图像索引，文件内容包含Mask R-CNN检测到的所有目标信息，格式为：idx class score min_x min_y max_x max_y masks_file，其中min_x、min_y、max_x、max_y为边界框信息，masks_file即指代前述的X/Y.png文件。
+ result_X.png：其中X为图像索引，为可视化检测结果的图像，仅展示置信度高于0.3的检测目标。
+ gt_iou_X：其中X为图像索引，Y为检测目标的索引。
* Z.png：其中Z为真实目标的索引，该图像同时展示检测目标Y与真实目标Z的掩码，用于直观对比两者的重叠情况。
* Y.txt：其中Y为检测目标的索引，文件内容包含：
1. 该目标掩码与图像已标注区域的交比；
2. 该目标掩码与所有真实目标掩码的交并比（Intersection over Union, IoU）值。

参考文献：
[1] He, Kaiming, Georgia Gkioxari, Piotr Dollár, Ross Girshick. Mask R-CNN[C]//Proceedings of the IEEE International Conference on Computer Vision. 2017: 2961-2969.
[2] Silberman, Nathan, Derek Hoiem, Pushmeet Kohli, Rob Fergus. 基于RGB-D图像的室内分割与支撑推理[C]//欧洲计算机视觉会议. 柏林, 海德堡: Springer, 2012: 746-760.